From patchwork Wed Nov 15 03:02:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolu Lu X-Patchwork-Id: 165160 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:6358:a59:b0:164:83eb:24d7 with SMTP id 25csp2357472rwb; Tue, 14 Nov 2023 19:09:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IEKR2T6p/+OYiomRd87v1c+QqJz8PtrI7VeafWb4oKtK93Yp9ohUwxVLv/eBPhbhS6hw2VX X-Received: by 2002:a05:6808:1782:b0:3b6:21cc:742b with SMTP id bg2-20020a056808178200b003b621cc742bmr15020695oib.30.1700017756148; Tue, 14 Nov 2023 19:09:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700017756; cv=none; d=google.com; s=arc-20160816; b=LAtOog8Syhv3dIw6X4OPYrnCADXR+wHRoIkk903242wNuwoelyZ8sxFGUxNfMe09Dz aIHt6CJc5X1OHWUd7fzVroFvNVsIq4wLhrZKEcaOIw1u75ruq6yL5XPw8U7xWJ2ct5cv +lLa64f2S/uPyy2JYjHtBDfqkalPoNkBo9VoOPtUzE9Z6I9dTQcEvKTZR/3d1VQdffYi XKDKx7bNa1SeQ2/CtxeUFtGHfbcIOJvkhUsg/Tlv+uwLLnrSa2UVJ8SLVHK6aulpLmbn Irptk1qT5mOyFpgRDTfl1kSdRa0pRcAO/UJkCGan6X2Cc9lQhgrd+xMjvh94EshCTZ7L XQzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ekktdgpsyVgv0iYNducRswoR820PRVnmQYxlWqf2kBg=; fh=DLFwkwrOYM/4/CX+ox6OE1AaCACujz37TA++iS2u/6Y=; b=Cl6eBxVlsXaDRSuV02lA/phBlwEwJGw5+Bz/Dw5K/KxnxTa3GC3tyGnwFHQrdb7eZ9 h8pRsZTK6KycJQxFB0G0qTbBswczltSxk+tFMDdiBPpmShUEY63uXIHgerDstKWWEW9C U4jnDpt/VzWIwr4jf9QwRZhGjChWZrFPWM755malgkNye+4K0Lm9f/UqlSX7YBQP/9+S a+90AJ+FjZp+hPwOxMKqNsnvClSRCzTkU2hqmAusaabqt/D4lvvgaRguLXDD6lNBUohR aYhgkdIGIFmihd/r4o5l8GCR4pxPeKZi79aDC9rjlNSEwb39ExcK6g0yYR99T66oOmpQ Hkhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N3VQ1kBA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id g23-20020a63fa57000000b005b9b68add85si9392926pgk.254.2023.11.14.19.09.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Nov 2023 19:09:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N3VQ1kBA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 4DC1C809A786; Tue, 14 Nov 2023 19:09:09 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234427AbjKODII (ORCPT + 28 others); Tue, 14 Nov 2023 22:08:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234424AbjKODHt (ORCPT ); Tue, 14 Nov 2023 22:07:49 -0500 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E11219F; Tue, 14 Nov 2023 19:07:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1700017648; x=1731553648; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8cX7Oz5UK1iAtPODtWmYwRan6akZgj2D9du4TSJDYN8=; b=N3VQ1kBA1S5t78zboTZPgpmVKKbT7GUVPcGqowtuGwewkamVCsswuHQ1 W4X6HPi/S+7Pw4Sf6OW6N36yZxObANFnvOAnZJLUBrLUoS37Hd8QZ1j+N y4NgGJgFfxYEAuYKsddRXsBfp0dvdIv9O95PPVtp4mKzIvJ4N2ERJz94w N7o9w5MODDES7BjmhfI9d52Leu8Ena0w/sOc1PayMI9PzissZsvo847P9 DH/V8Xt1pMrGS/aTfrJEyofWObP2rmOuGVogvMn8GYjXHFDGMX+LekO+t 2WGRqW4t/K/oC/CN/KQ/4My2zUEajJjoIf7PErNQbDUVBAAt+EjDY5Q7k w==; X-IronPort-AV: E=McAfee;i="6600,9927,10894"; a="394715506" X-IronPort-AV: E=Sophos;i="6.03,303,1694761200"; d="scan'208";a="394715506" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Nov 2023 19:07:22 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10894"; a="1012128925" X-IronPort-AV: E=Sophos;i="6.03,303,1694761200"; d="scan'208";a="1012128925" Received: from allen-box.sh.intel.com ([10.239.159.127]) by fmsmga006.fm.intel.com with ESMTP; 14 Nov 2023 19:07:19 -0800 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Jason Gunthorpe , Kevin Tian , Jean-Philippe Brucker , Nicolin Chen Cc: Yi Liu , Jacob Pan , Yan Zhao , iommu@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v7 11/12] iommu: Consolidate per-device fault data management Date: Wed, 15 Nov 2023 11:02:25 +0800 Message-Id: <20231115030226.16700-12-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231115030226.16700-1-baolu.lu@linux.intel.com> References: <20231115030226.16700-1-baolu.lu@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 14 Nov 2023 19:09:09 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782597818425573619 X-GMAIL-MSGID: 1782597818425573619 The per-device fault data is a data structure that is used to store information about faults that occur on a device. This data is allocated when IOPF is enabled on the device and freed when IOPF is disabled. The data is used in the paths of iopf reporting, handling, responding, and draining. The fault data is protected by two locks: - dev->iommu->lock: This lock is used to protect the allocation and freeing of the fault data. - dev->iommu->fault_parameter->lock: This lock is used to protect the fault data itself. To make the code simpler and easier to maintain, consolidate the lock mechanism into two helper functions. The dev->iommu->fault_parameter->lock lock is also used in iopf_queue_discard_partial() to improve code readability. This does not fix any existing issues, as iopf_queue_discard_partial() is only used in the VT-d driver's prq_event_thread(), which is a single-threaded path that reports the IOPFs. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian Tested-by: Yan Zhao --- include/linux/iommu.h | 3 + drivers/iommu/io-pgfault.c | 122 +++++++++++++++++++++++-------------- 2 files changed, 79 insertions(+), 46 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index d19031c1b0e6..c17d5979d70d 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -597,6 +597,8 @@ struct iommu_device { /** * struct iommu_fault_param - per-device IOMMU fault data * @lock: protect pending faults list + * @users: user counter to manage the lifetime of the data, this field + * is protected by dev->iommu->lock. * @dev: the device that owns this param * @queue: IOPF queue * @queue_list: index into queue->devices @@ -606,6 +608,7 @@ struct iommu_device { */ struct iommu_fault_param { struct mutex lock; + int users; struct device *dev; struct iopf_queue *queue; diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c index 3c119bfa1d4a..b80574323cbc 100644 --- a/drivers/iommu/io-pgfault.c +++ b/drivers/iommu/io-pgfault.c @@ -26,6 +26,49 @@ void iopf_free_group(struct iopf_group *group) } EXPORT_SYMBOL_GPL(iopf_free_group); +/* + * Return the fault parameter of a device if it exists. Otherwise, return NULL. + * On a successful return, the caller takes a reference of this parameter and + * should put it after use by calling iopf_put_dev_fault_param(). + */ +static struct iommu_fault_param *iopf_get_dev_fault_param(struct device *dev) +{ + struct dev_iommu *param = dev->iommu; + struct iommu_fault_param *fault_param; + + if (!param) + return NULL; + + mutex_lock(¶m->lock); + fault_param = param->fault_param; + if (fault_param) + fault_param->users++; + mutex_unlock(¶m->lock); + + return fault_param; +} + +/* Caller must hold a reference of the fault parameter. */ +static void iopf_put_dev_fault_param(struct iommu_fault_param *fault_param) +{ + struct device *dev = fault_param->dev; + struct dev_iommu *param = dev->iommu; + + mutex_lock(¶m->lock); + if (WARN_ON(fault_param->users <= 0 || + fault_param != param->fault_param)) { + mutex_unlock(¶m->lock); + return; + } + + if (--fault_param->users == 0) { + param->fault_param = NULL; + kfree(fault_param); + put_device(dev); + } + mutex_unlock(¶m->lock); +} + /** * iommu_handle_iopf - IO Page Fault handler * @fault: fault event @@ -72,23 +115,14 @@ static int iommu_handle_iopf(struct iommu_fault *fault, struct device *dev) struct iopf_group *group; struct iopf_fault *iopf, *next; struct iommu_domain *domain = NULL; - struct iommu_fault_param *iopf_param; - struct dev_iommu *param = dev->iommu; + struct iommu_fault_param *iopf_param = dev->iommu->fault_param; - lockdep_assert_held(¶m->lock); + lockdep_assert_held(&iopf_param->lock); if (fault->type != IOMMU_FAULT_PAGE_REQ) /* Not a recoverable page fault */ return -EOPNOTSUPP; - /* - * As long as we're holding param->lock, the queue can't be unlinked - * from the device and therefore cannot disappear. - */ - iopf_param = param->fault_param; - if (!iopf_param) - return -ENODEV; - if (!(fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) { iopf = kzalloc(sizeof(*iopf), GFP_KERNEL); if (!iopf) @@ -173,18 +207,15 @@ static int iommu_handle_iopf(struct iommu_fault *fault, struct device *dev) */ int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt) { - struct dev_iommu *param = dev->iommu; + struct iommu_fault_param *fault_param; struct iopf_fault *evt_pending = NULL; - struct iommu_fault_param *fparam; int ret = 0; - if (!param || !evt) + fault_param = iopf_get_dev_fault_param(dev); + if (!fault_param) return -EINVAL; - /* we only report device fault if there is a handler registered */ - mutex_lock(¶m->lock); - fparam = param->fault_param; - + mutex_lock(&fault_param->lock); if (evt->fault.type == IOMMU_FAULT_PAGE_REQ && (evt->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) { evt_pending = kmemdup(evt, sizeof(struct iopf_fault), @@ -193,20 +224,18 @@ int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt) ret = -ENOMEM; goto done_unlock; } - mutex_lock(&fparam->lock); - list_add_tail(&evt_pending->list, &fparam->faults); - mutex_unlock(&fparam->lock); + list_add_tail(&evt_pending->list, &fault_param->faults); } ret = iommu_handle_iopf(&evt->fault, dev); if (ret && evt_pending) { - mutex_lock(&fparam->lock); list_del(&evt_pending->list); - mutex_unlock(&fparam->lock); kfree(evt_pending); } done_unlock: - mutex_unlock(¶m->lock); + mutex_unlock(&fault_param->lock); + iopf_put_dev_fault_param(fault_param); + return ret; } EXPORT_SYMBOL_GPL(iommu_report_device_fault); @@ -218,19 +247,20 @@ int iommu_page_response(struct device *dev, int ret = -EINVAL; struct iopf_fault *evt; struct iommu_fault_page_request *prm; - struct dev_iommu *param = dev->iommu; + struct iommu_fault_param *fault_param; const struct iommu_ops *ops = dev_iommu_ops(dev); bool has_pasid = msg->flags & IOMMU_PAGE_RESP_PASID_VALID; if (!ops->page_response) return -ENODEV; - if (!param || !param->fault_param) - return -EINVAL; + fault_param = iopf_get_dev_fault_param(dev); + if (!fault_param) + return -ENODEV; /* Only send response if there is a fault report pending */ - mutex_lock(¶m->fault_param->lock); - if (list_empty(¶m->fault_param->faults)) { + mutex_lock(&fault_param->lock); + if (list_empty(&fault_param->faults)) { dev_warn_ratelimited(dev, "no pending PRQ, drop response\n"); goto done_unlock; } @@ -238,7 +268,7 @@ int iommu_page_response(struct device *dev, * Check if we have a matching page request pending to respond, * otherwise return -EINVAL */ - list_for_each_entry(evt, ¶m->fault_param->faults, list) { + list_for_each_entry(evt, &fault_param->faults, list) { prm = &evt->fault.prm; if (prm->grpid != msg->grpid) continue; @@ -266,7 +296,9 @@ int iommu_page_response(struct device *dev, } done_unlock: - mutex_unlock(¶m->fault_param->lock); + mutex_unlock(&fault_param->lock); + iopf_put_dev_fault_param(fault_param); + return ret; } EXPORT_SYMBOL_GPL(iommu_page_response); @@ -285,22 +317,15 @@ EXPORT_SYMBOL_GPL(iommu_page_response); */ int iopf_queue_flush_dev(struct device *dev) { - int ret = 0; - struct iommu_fault_param *iopf_param; - struct dev_iommu *param = dev->iommu; + struct iommu_fault_param *iopf_param = iopf_get_dev_fault_param(dev); - if (!param) + if (!iopf_param) return -ENODEV; - mutex_lock(¶m->lock); - iopf_param = param->fault_param; - if (iopf_param) - flush_workqueue(iopf_param->queue->wq); - else - ret = -ENODEV; - mutex_unlock(¶m->lock); + flush_workqueue(iopf_param->queue->wq); + iopf_put_dev_fault_param(iopf_param); - return ret; + return 0; } EXPORT_SYMBOL_GPL(iopf_queue_flush_dev); @@ -349,11 +374,13 @@ int iopf_queue_discard_partial(struct iopf_queue *queue) mutex_lock(&queue->lock); list_for_each_entry(iopf_param, &queue->devices, queue_list) { + mutex_lock(&iopf_param->lock); list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) { list_del(&iopf->list); kfree(iopf); } + mutex_unlock(&iopf_param->lock); } mutex_unlock(&queue->lock); return 0; @@ -392,6 +419,7 @@ int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev) INIT_LIST_HEAD(&fault_param->faults); INIT_LIST_HEAD(&fault_param->partial); fault_param->dev = dev; + fault_param->users = 1; list_add(&fault_param->queue_list, &queue->devices); fault_param->queue = queue; @@ -444,9 +472,11 @@ int iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev) list_for_each_entry_safe(iopf, next, &fault_param->partial, list) kfree(iopf); - param->fault_param = NULL; - kfree(fault_param); - put_device(dev); + if (--fault_param->users == 0) { + param->fault_param = NULL; + kfree(fault_param); + put_device(dev); + } unlock: mutex_unlock(¶m->lock); mutex_unlock(&queue->lock);