From patchwork Mon Oct 31 00:59:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolu Lu X-Patchwork-Id: 13089 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2042453wru; Sun, 30 Oct 2022 18:09:09 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6Hl6/mqgPmb/mEc14+Qg/rQSgEDxGWemLXn6WZWAplvxSKreVvq53/X4x9zOsjWlErTjHW X-Received: by 2002:a17:906:66ce:b0:7ad:d178:c252 with SMTP id k14-20020a17090666ce00b007add178c252mr2743329ejp.158.1667178549325; Sun, 30 Oct 2022 18:09:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667178549; cv=none; d=google.com; s=arc-20160816; b=1LZ6RqpUAndtXOUIjmhoAf7eHoBE1wxOXNel8T/xyPbvWVlc8/GNL6zBwm9pxF4YHF EfKs1K05f6tMy28wgTRHlCuP11gpYi5tu4Bb86diC9Fk5v9Palv7UAopyxZ1mLHyl7ZO 6CC4XkSSMCxfJiosej8dcPMaxcEjvEzj05A8/nI+depZuDVOwKwjSXrLsLIzJckBWQ24 yZOYOPGZmb8PkSqK9nnrmu5IroiTMbLXU5XZPwmPIN7Wo71j4SEHFFAx+hrQZmBt3zt+ qzBF5ES9fU6QJshd1NpIThuD4aQop6zjYLBGz74MwDdNAj9ez+5qQvhSz84qbTOtRmzj 8MDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wVAEcyoWzoxKtQI8ewNsuEVk0nbDG/0Dt/eTc3IA+ag=; b=S7a47ohOM21KZi9VXJW1TBLtJj7nINimh6YbJpW1NzCPHKtF6qMzncLShSkvX88rEM 1NOUQPXVTMHjCeQvQpWyXWGsx62wiFW3R90mYIGdAsFOyi1Rbt1yGBRgRTgyT3RPlrJF GDxC+DyWXDOFBHK04lQieAh0bUcZTgmuUsPN7//heH+Xdl0pY/vc1TnOhTB8slIfmbjm 1avx8n4XxY3lvv0kCC9Sf3lf7Nh6akY1528/UR/JdbVvyh+0nbZwd1fJLu8KWNbDpg5W RxmUghGfOSEjq9Sdcq7YKx06Ecg3WIt63hnf51t9PmL+zKPjf9oe8rR8JqSGdozZLy28 2Erw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="iw2rli/b"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w8-20020a05640234c800b0045c31d9643bsi7304979edc.2.2022.10.30.18.08.45; Sun, 30 Oct 2022 18:09:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="iw2rli/b"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229949AbiJaBHP (ORCPT + 99 others); Sun, 30 Oct 2022 21:07:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229906AbiJaBGm (ORCPT ); Sun, 30 Oct 2022 21:06:42 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9554DBF68 for ; Sun, 30 Oct 2022 18:06:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1667178385; x=1698714385; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DhtYeSNqUAaAkHikDPzpgpLKsdlOqzZmLs+oo9yKc0M=; b=iw2rli/b932T20fiQDZy46OEgVEqe/lqnp5SCML59xexMtHZJ9+oKlFR Pw4YN/id/R+Y6K9XMqDRIZHkyu9rx+JOmfOVoipBbZ6d+nNw0GPsr4m3M CvaZYM6JuifG2Z+HXOS6QF+IkHdLFg6Gnk3s04FpRPYkpK2u264lTbQBO ekn3YOzeGpU583VPsWgwLFVU+IkKTuMls2FU3FwSlSQSZKSrGa8VVLoi5 ruMmN5h1e2syIrd/LY27E5oSep2vjFB/yUP0nxesoLfwBLJSSvX6ynnFC AKMi9JfvLG9yowdt7+in9cpv0tMDYNUUdfIgyJgrJciAeRM2d6Get6Q9R w==; X-IronPort-AV: E=McAfee;i="6500,9779,10516"; a="373001716" X-IronPort-AV: E=Sophos;i="5.95,227,1661842800"; d="scan'208";a="373001716" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Oct 2022 18:06:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10516"; a="584496279" X-IronPort-AV: E=Sophos;i="5.95,227,1661842800"; d="scan'208";a="584496279" Received: from allen-box.sh.intel.com ([10.239.159.48]) by orsmga003.jf.intel.com with ESMTP; 30 Oct 2022 18:06:23 -0700 From: Lu Baolu To: Joerg Roedel Cc: Jason Gunthorpe , Kevin Tian , Will Deacon , Robin Murphy , Jean-Philippe Brucker , iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [RESEND PATCH v14 11/13] iommu: Prepare IOMMU domain for IOPF Date: Mon, 31 Oct 2022 08:59:15 +0800 Message-Id: <20221031005917.45690-12-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221031005917.45690-1-baolu.lu@linux.intel.com> References: <20221031005917.45690-1-baolu.lu@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748163414493122040?= X-GMAIL-MSGID: =?utf-8?q?1748163414493122040?= This adds some mechanisms around the iommu_domain so that the I/O page fault handling framework could route a page fault to the domain and call the fault handler from it. Add pointers to the page fault handler and its private data in struct iommu_domain. The fault handler will be called with the private data as a parameter once a page fault is routed to the domain. Any kernel component which owns an iommu domain could install handler and its private parameter so that the page fault could be further routed and handled. This also prepares the SVA implementation to be the first consumer of the per-domain page fault handling model. The I/O page fault handler for SVA is copied to the SVA file with mmget_not_zero() added before mmap_read_lock(). Suggested-by: Jean-Philippe Brucker Signed-off-by: Lu Baolu Reviewed-by: Jean-Philippe Brucker Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Tested-by: Zhangfei Gao Tested-by: Tony Zhu --- include/linux/iommu.h | 3 ++ drivers/iommu/iommu-sva-lib.h | 8 +++++ drivers/iommu/io-pgfault.c | 7 +++++ drivers/iommu/iommu-sva-lib.c | 58 +++++++++++++++++++++++++++++++++++ drivers/iommu/iommu.c | 4 +++ 5 files changed, 80 insertions(+) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index c337ef1c97bc..7d2648058e43 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -98,6 +98,9 @@ struct iommu_domain { unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */ struct iommu_domain_geometry geometry; struct iommu_dma_cookie *iova_cookie; + enum iommu_page_response_code (*iopf_handler)(struct iommu_fault *fault, + void *data); + void *fault_data; union { struct { iommu_fault_handler_t handler; diff --git a/drivers/iommu/iommu-sva-lib.h b/drivers/iommu/iommu-sva-lib.h index 8909ea1094e3..1b3ace4b5863 100644 --- a/drivers/iommu/iommu-sva-lib.h +++ b/drivers/iommu/iommu-sva-lib.h @@ -26,6 +26,8 @@ int iopf_queue_flush_dev(struct device *dev); struct iopf_queue *iopf_queue_alloc(const char *name); void iopf_queue_free(struct iopf_queue *queue); int iopf_queue_discard_partial(struct iopf_queue *queue); +enum iommu_page_response_code +iommu_sva_handle_iopf(struct iommu_fault *fault, void *data); #else /* CONFIG_IOMMU_SVA */ static inline int iommu_queue_iopf(struct iommu_fault *fault, void *cookie) @@ -63,5 +65,11 @@ static inline int iopf_queue_discard_partial(struct iopf_queue *queue) { return -ENODEV; } + +static inline enum iommu_page_response_code +iommu_sva_handle_iopf(struct iommu_fault *fault, void *data) +{ + return IOMMU_PAGE_RESP_INVALID; +} #endif /* CONFIG_IOMMU_SVA */ #endif /* _IOMMU_SVA_LIB_H */ diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c index 1df8c1dcae77..aee9e033012f 100644 --- a/drivers/iommu/io-pgfault.c +++ b/drivers/iommu/io-pgfault.c @@ -181,6 +181,13 @@ static void iopf_handle_group(struct work_struct *work) * request completes, outstanding faults will have been dealt with by the time * the PASID is freed. * + * Any valid page fault will be eventually routed to an iommu domain and the + * page fault handler installed there will get called. The users of this + * handling framework should guarantee that the iommu domain could only be + * freed after the device has stopped generating page faults (or the iommu + * hardware has been set to block the page faults) and the pending page faults + * have been flushed. + * * Return: 0 on success and <0 on error. */ int iommu_queue_iopf(struct iommu_fault *fault, void *cookie) diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c index e425573a1787..089fd61ff453 100644 --- a/drivers/iommu/iommu-sva-lib.c +++ b/drivers/iommu/iommu-sva-lib.c @@ -180,3 +180,61 @@ u32 iommu_sva_get_pasid(struct iommu_sva *handle) return domain->mm->pasid; } EXPORT_SYMBOL_GPL(iommu_sva_get_pasid); + +/* + * I/O page fault handler for SVA + */ +enum iommu_page_response_code +iommu_sva_handle_iopf(struct iommu_fault *fault, void *data) +{ + vm_fault_t ret; + struct vm_area_struct *vma; + struct mm_struct *mm = data; + unsigned int access_flags = 0; + unsigned int fault_flags = FAULT_FLAG_REMOTE; + struct iommu_fault_page_request *prm = &fault->prm; + enum iommu_page_response_code status = IOMMU_PAGE_RESP_INVALID; + + if (!(prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)) + return status; + + if (!mmget_not_zero(mm)) + return status; + + mmap_read_lock(mm); + + vma = find_extend_vma(mm, prm->addr); + if (!vma) + /* Unmapped area */ + goto out_put_mm; + + if (prm->perm & IOMMU_FAULT_PERM_READ) + access_flags |= VM_READ; + + if (prm->perm & IOMMU_FAULT_PERM_WRITE) { + access_flags |= VM_WRITE; + fault_flags |= FAULT_FLAG_WRITE; + } + + if (prm->perm & IOMMU_FAULT_PERM_EXEC) { + access_flags |= VM_EXEC; + fault_flags |= FAULT_FLAG_INSTRUCTION; + } + + if (!(prm->perm & IOMMU_FAULT_PERM_PRIV)) + fault_flags |= FAULT_FLAG_USER; + + if (access_flags & ~vma->vm_flags) + /* Access fault */ + goto out_put_mm; + + ret = handle_mm_fault(vma, prm->addr, fault_flags, NULL); + status = ret & VM_FAULT_ERROR ? IOMMU_PAGE_RESP_INVALID : + IOMMU_PAGE_RESP_SUCCESS; + +out_put_mm: + mmap_read_unlock(mm); + mmput(mm); + + return status; +} diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index c9da0a1bb3b8..9e0fb18e1b34 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -33,6 +33,8 @@ #include "dma-iommu.h" +#include "iommu-sva-lib.h" + static struct kset *iommu_group_kset; static DEFINE_IDA(iommu_group_ida); @@ -3309,6 +3311,8 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, domain->type = IOMMU_DOMAIN_SVA; mmgrab(mm); domain->mm = mm; + domain->iopf_handler = iommu_sva_handle_iopf; + domain->fault_data = mm; return domain; }