From patchwork Thu Feb 8 08:23:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 198205 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp24609dyd; Thu, 8 Feb 2024 00:35:20 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWO3YCtzTsco0tS/n0dtwHa86DWcSV1bI1a1lYkszKKpDc9Qx+aZZcCG/Zogt85hM/u5FYsKnBug7yZhXn+cX8QWdljaw== X-Google-Smtp-Source: AGHT+IECtupEPGRpk+2MLIo6P5Da7SDLuQA2hbnSE0m5TZa7kzwZoZ8sHCgaOSVL1CZAI43rk20W X-Received: by 2002:a05:6a21:3d88:b0:19c:6f20:599 with SMTP id bj8-20020a056a213d8800b0019c6f200599mr7528690pzc.26.1707381320758; Thu, 08 Feb 2024 00:35:20 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707381320; cv=pass; d=google.com; s=arc-20160816; b=yUu+3r5VWsiqfa5Mnfa2gHgKkAtzvT3Sml/8Y0HVyy5Pt78XjOPdDnquWUizY8s1K5 zB0ovAQSDcb95CmUUZOc3ElGh/n36bi2moVtTGHEsa0JDfs0J87wPoFvXJf1RHIQTL6a T0MrrSGN4GAAEaO8MbQiwBGxBdV7p4nmEDbJe5GrbaBjkfW5cxf/w/Ig3XAfGe9TSh8g 6uvtMbCAtbWdotnNs707Ke/k378R4qYcACcqEDk/zfLRrIhETnn7RtEdIh2IaIWmSTHL rdxr5V1cDIkr65nQecUZSmoHRbLF9HLpPQ212RZdmXslgDN7ctBfO6DW2in0OtyBt+4J 4dtw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=hEB1CXROg93Lt3NNA5HTA5yY8ekY1dwbxNXlaCIMRsI=; fh=Am9LoEjxl6cwcp1TZaHvNW4XNuu0sIei1D0nUGc5nCY=; b=AMSZlNkFov9Y3EG5Q2WiZJJqvSveICSwyDjkfskOMmlfnVNhw208SULRGytSbhb4MC 8rLEj4BroKy5wQUmg/dOYuG9IYnc6cMD29q5GORUY/3+x3glvA1yuQFXoggi82qh+uI1 hGWciA7t2OwAU1KFBT7+6BZhAFWNEEzmnswTRlKPqncyVicqC4sebOovWqPzurOGDo/S KdNymYgOqqbSVqvpGulXQYQRV4zYoE4GQGvmba/PlsnTt3U31qqcUYcb/FnQEDOI/btS YPj3x0BT8lZWD8NIdKSsMUEaq1v7hWI4ZfsvWef8nJn3780I6UfkBNNCDgDgA74+iiXp ORRg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=czmkOdku; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-57605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-57605-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCUWCpilvR1VujhRcfDiG7hWPiGgZaCRROMdAZesDYZtLdr7+6qReuv+yH3IKh6Y/AoUebzBOwmLwsiNxOXCs+Eow9IuRg== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id o22-20020a637e56000000b005897813624fsi3316284pgn.476.2024.02.08.00.35.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Feb 2024 00:35:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-57605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=czmkOdku; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-57605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-57605-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id BA394288C6B for ; Thu, 8 Feb 2024 08:24:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 263E26BFD4; Thu, 8 Feb 2024 08:23:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="czmkOdku" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D24536A329; Thu, 8 Feb 2024 08:23:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707380593; cv=none; b=t78vCwJRujQK0ncVkreRYH79zZhg1MiJiCOaZbAXzL3/RSxpBkv1qgNRxB3BU2f+Eh4K3GH+hbhRTMOsh9P0k3rxnoE3UsQcdUpKPUiirkYjoKzJIJ3/mIXcYNqwcIlAuCMTuiXO86V/ueEt7rcAoxfOqKC7ZOdbE8REyTZTejs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707380593; c=relaxed/simple; bh=/GnRCRW90B1W9TXPxSmAiTyrVQIgkymW5d5jQqKtU2w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HRx2Y4TbT8jB79v3oAGqg4Ch19yy2V4X8uXAhYw841vajyGNyIPyQcLIifZUrh/u6mDhXSZS/uIoUj24UelqHwwKvO+MClJethb2ZdLFZPLUBBr/LIVzMqr41SMK2Kid5xDHaXpHNjdPjtX4ClaodAOaaBoVCtzhf3+hZVd8Zbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=czmkOdku; arc=none smtp.client-ip=198.175.65.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707380593; x=1738916593; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/GnRCRW90B1W9TXPxSmAiTyrVQIgkymW5d5jQqKtU2w=; b=czmkOdku5B648SBN8BjBHOuxZhgu1ZiA1HUu/x8u9y/s/N+zTb3PxDrN pQUcxuPzBHy36k2Ptd4Jgm0YQs4QzxIVoXmauGv/fHGWsO4MgzJVVi8+Z VrpEzfC3POufnPRZ2tOLU91jWZNViBzi12i+NE6oQGHTEMOVZpqIzOAlK Y6pujMSOq2i523/u93Dx6JUnlo7mUjxJPW+UiTltAC5N9ILp3P48bl3it HsqVvgpdfk8JyLfP3IC6+hBts37wuVcZr6R1SpLpsdXtl1ajOUk9z5de/ L9Drd+TgQKwW8VbiaQounN1BQFAiUJpslnb06/I0VLD71zyOtC9Oei9LN g==; X-IronPort-AV: E=McAfee;i="6600,9927,10977"; a="5036302" X-IronPort-AV: E=Sophos;i="6.05,253,1701158400"; d="scan'208";a="5036302" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2024 00:23:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,253,1701158400"; d="scan'208";a="6252090" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orviesa003.jf.intel.com with ESMTP; 08 Feb 2024 00:23:10 -0800 From: Yi Liu To: joro@8bytes.org, jgg@nvidia.com, kevin.tian@intel.com, baolu.lu@linux.intel.com Cc: alex.williamson@redhat.com, robin.murphy@arm.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com, joao.m.martins@oracle.com Subject: [PATCH rc 1/8] iommu/vt-d: Track nested domains in parent Date: Thu, 8 Feb 2024 00:23:00 -0800 Message-Id: <20240208082307.15759-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240208082307.15759-1-yi.l.liu@intel.com> References: <20240208082307.15759-1-yi.l.liu@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790318856205575807 X-GMAIL-MSGID: 1790319075835758477 Today the parent domain (s2_domain) is unaware of which DID's are used by and which devices are attached to nested domains (s1_domain) nested on it. This leads to a problem that some operations (flush iotlb/devtlb and enable dirty tracking) on parent domain only apply to DID's and devices directly tracked in the parent domain hence are incomplete. This tracks the nested domains in list in parent domain. With this, operations on parent domain can loop the nested domains and refer to the devices and iommu_array to ensure the operations on parent domain take effect on all the affected devices and iommus. Signed-off-by: Yi Liu --- drivers/iommu/intel/iommu.c | 18 ++++++++++++++---- drivers/iommu/intel/iommu.h | 6 ++++++ drivers/iommu/intel/nested.c | 10 ++++++++++ 3 files changed, 30 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 6fb5f6fceea1..e393c62776f3 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -3883,6 +3883,7 @@ intel_iommu_domain_alloc_user(struct device *dev, u32 flags, bool dirty_tracking = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING; bool nested_parent = flags & IOMMU_HWPT_ALLOC_NEST_PARENT; struct intel_iommu *iommu = info->iommu; + struct dmar_domain *dmar_domain; struct iommu_domain *domain; /* Must be NESTING domain */ @@ -3908,11 +3909,16 @@ intel_iommu_domain_alloc_user(struct device *dev, u32 flags, if (!domain) return ERR_PTR(-ENOMEM); - if (nested_parent) - to_dmar_domain(domain)->nested_parent = true; + dmar_domain = to_dmar_domain(domain); + + if (nested_parent) { + dmar_domain->nested_parent = true; + INIT_LIST_HEAD(&dmar_domain->s1_domains); + spin_lock_init(&dmar_domain->s1_lock); + } if (dirty_tracking) { - if (to_dmar_domain(domain)->use_first_level) { + if (dmar_domain->use_first_level) { iommu_domain_free(domain); return ERR_PTR(-EOPNOTSUPP); } @@ -3924,8 +3930,12 @@ intel_iommu_domain_alloc_user(struct device *dev, u32 flags, static void intel_iommu_domain_free(struct iommu_domain *domain) { + struct dmar_domain *dmar_domain = to_dmar_domain(domain); + + WARN_ON(dmar_domain->nested_parent && + !list_empty(&dmar_domain->s1_domains)); if (domain != &si_domain->domain) - domain_exit(to_dmar_domain(domain)); + domain_exit(dmar_domain); } int prepare_domain_attach_device(struct iommu_domain *domain, diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index d02f916d8e59..9b27edb73aa9 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -627,6 +627,10 @@ struct dmar_domain { int agaw; /* maximum mapped address */ u64 max_addr; + /* Protect the s1_domains list */ + spinlock_t s1_lock; + /* Track s1_domains nested on this domain */ + struct list_head s1_domains; }; /* Nested user domain */ @@ -637,6 +641,8 @@ struct dmar_domain { unsigned long s1_pgtbl; /* page table attributes */ struct iommu_hwpt_vtd_s1 s1_cfg; + /* link to parent domain siblings */ + struct list_head s2_link; }; }; diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index f26c7f1c46cc..42d4d222e70f 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -70,6 +70,12 @@ static int intel_nested_attach_dev(struct iommu_domain *domain, static void intel_nested_domain_free(struct iommu_domain *domain) { + struct dmar_domain *dmar_domain = to_dmar_domain(domain); + struct dmar_domain *s2_domain = dmar_domain->s2_domain; + + spin_lock(&s2_domain->s1_lock); + list_del(&dmar_domain->s2_link); + spin_unlock(&s2_domain->s1_lock); kfree(to_dmar_domain(domain)); } @@ -201,5 +207,9 @@ struct iommu_domain *intel_nested_domain_alloc(struct iommu_domain *parent, spin_lock_init(&domain->lock); xa_init(&domain->iommu_array); + spin_lock(&s2_domain->s1_lock); + list_add(&domain->s2_link, &s2_domain->s1_domains); + spin_unlock(&s2_domain->s1_lock); + return &domain->domain; }