From patchwork Wed May 10 20:50:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Shavit X-Patchwork-Id: 92295 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3926824vqo; Wed, 10 May 2023 14:20:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7t4BJ33yFYh1MgHgJldidiVvnetMDZPANoMViGzky5UGyYWgGeZaY8Lyd9YWFqdeCwz+uR X-Received: by 2002:a05:6a00:a19:b0:625:efa4:4c01 with SMTP id p25-20020a056a000a1900b00625efa44c01mr25634647pfh.3.1683753648956; Wed, 10 May 2023 14:20:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683753648; cv=none; d=google.com; s=arc-20160816; b=gksTCjzim6Sk0J2U2MoLHMTo2+XL8eCaW1StWuCPcB3r0wkZkq7yfi1+VHGZGBbKUk PVUCvrrGTbtTC0ytvz4mFGwapZiIMMm7bJuDuTe2Vu3IZBj866LX0IamYbE2Q2YF6S2Z 7wYF+Is6GMeF+Ii5YHS9TO6UpG+QICOkShI6OPGduBv+oV2t/jX7MftDVteYqZ/7g41p TYAmNEch2R3WocvAz8nVeHAoe08SD8886JMNsIvvfydxGQlrrmxp2O+6v6ZN/Mc4Kmmp tm77Gxil4afB1vVGtS8ls9lSRFTJ65oo2sDupsKDDTMbFpvfDoP9KXp5TDgj4dAtQfas GWPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=f05oUNHXSveqV1U+rh8Ih41HYQKZ7A3pZBc66Z2rnxc=; b=rqYOk//s5hsq96xi/FG0PuJQQl8VUQHWEIDRcHOn5whjas131/jnLPWKbvayLLLeC7 +TVxk4H4VA/eLtz1HIEG7ZQtXLRY5KycVwk7Edi+vAmZ62EPGZeOQh1v7qsTh7928fu7 mo/Wh28T2GsS9ngkhI+Tr8GbaEdRI2eHTxkYjEDmAeFCgiHNZuRE0rYFSAm0DgGOew0k gImqKlX12L4V5m88hnIk59jrC7AwaMf8cVwNRXpMboaXMkd441Is11jS4xf0tLy4HAfe mUWPBPA2ZAHQdIdNudOG6rRREmmSWSsU6JdLEPUoyfTBFyXbzQR3YI+BUkOShRD8EHWC NEUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=2LKwWhQd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a14-20020aa78e8e000000b0064378422f5fsi5820590pfr.169.2023.05.10.14.20.36; Wed, 10 May 2023 14:20:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=2LKwWhQd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237062AbjEJUwm (ORCPT + 99 others); Wed, 10 May 2023 16:52:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237167AbjEJUwE (ORCPT ); Wed, 10 May 2023 16:52:04 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED5BA6E87 for ; Wed, 10 May 2023 13:51:29 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-b9dcfade347so14099852276.2 for ; Wed, 10 May 2023 13:51:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683751865; x=1686343865; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=f05oUNHXSveqV1U+rh8Ih41HYQKZ7A3pZBc66Z2rnxc=; b=2LKwWhQdomE3dGvwT9OTXPmELcrXLWDV9UEd9UhdeIO1/5OG0hKScpfNWUQ8/CeIgz DISbEAg+eutvYWLtWR7jtbUS+JTOhPxz7ovrMXKkffZVQZxyP/tfWAP7Wni4GjMBSucG 3KjS7yPmR+d4Vwja0z44raVRH9tIsGvMkYUEnkM2Bhuw1vJGKAVcMPYgSIJ4xJaD7/5r nZird9ArtHDDSN2l6f1ctcYYn6l3yEbpqmQ4cRB3EsV0xJiqlQYlF66P/mPMEeqpvImp kTmq5gMuvtKlkgeUfLBHfPDh+GDQxOpCMDjkGplFuz6ovkBUhvtveOubNUnRrkJlSNCi FN+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683751865; x=1686343865; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=f05oUNHXSveqV1U+rh8Ih41HYQKZ7A3pZBc66Z2rnxc=; b=cCsWYlAW6HpFH0K+wg0v5SwnVmWYz3Z6fnrtPZu0zgnK35v7ni9jKC18LogjNpQQOQ fT6I3Q6qNaW2bzH4qAPn0xvPu80L811Si+Gr/xj83XtpP6++Jgsj8bZ1/sC7m/ofxiRa zAcljFL5MLt2nQyx1Rf7FG+KSGFYsyvytdYz7dgQqmDeKGEoRIbDiFA59/ZpH9ep/rl3 0iu5vVqQZUPzRulOr8RYjhZUtcXR+eIo9gg6yID6pfqYZpwFOdqNdS0fYvXth2upggOO vEzB1vqI0NrlOeYqeY/RNdyJ15ZIwZXQnbPEzvpA41wzBPx3fkgvjRDGcKReRzpyQcH2 4uhA== X-Gm-Message-State: AC+VfDyi4DWJ6Q2Wjwzytm1fWmFQz1xRcYe0GqVw7pX23J2Kf1IiOdC/ +4WmfaTzsDRriRnJDG/QOPXybH7SO1rr X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:2b1f:8d06:7923:f154]) (user=mshavit job=sendgmr) by 2002:a25:9f02:0:b0:b8f:53e1:64b with SMTP id n2-20020a259f02000000b00b8f53e1064bmr8607903ybq.13.1683751865516; Wed, 10 May 2023 13:51:05 -0700 (PDT) Date: Thu, 11 May 2023 04:50:48 +0800 In-Reply-To: <20230510205054.2667898-1-mshavit@google.com> Mime-Version: 1.0 References: <20230510205054.2667898-1-mshavit@google.com> X-Mailer: git-send-email 2.40.1.521.gf1e218fcd8-goog Message-ID: <20230510205054.2667898-2-mshavit@google.com> Subject: [PATCH v1 1/5] iommu/arm-smmu-v3: Move cdtable to arm_smmu_master From: Michael Shavit To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Michael Shavit , jean-philippe@linaro.org, nicolinc@nvidia.com, jgg@nvidia.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765543666044880081?= X-GMAIL-MSGID: =?utf-8?q?1765543666044880081?= In the arm-smmu-v3 driver, stage 1 domains represent the entire STE. Iommu domains are conceptually closer to a single address space (with some exceptions), which are better represented by a single ContextDescriptor for stage 1 domains. With this change, the CD table is now owned by the arm_smmu_master instead of the attached domain. Each stage 1 domain now represents a single context. For now, SVA still piggy-backs on the primary arm_smmu_domain for other state management (such as the devices list). Signed-off-by: Michael Shavit --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 35 +++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 185 ++++++++---------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 29 +-- 3 files changed, 127 insertions(+), 122 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index a5a63b1c947eb..a721461b355c6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -35,6 +35,9 @@ struct arm_smmu_bond { #define sva_to_bond(handle) \ container_of(handle, struct arm_smmu_bond, sva) +#define cd_to_domain(cd) \ + container_of(cd, struct arm_smmu_domain, cd) + static DEFINE_MUTEX(sva_lock); /* @@ -45,10 +48,12 @@ static struct arm_smmu_ctx_desc * arm_smmu_share_asid(struct mm_struct *mm, u16 asid) { int ret; + unsigned long flags; u32 new_asid; struct arm_smmu_ctx_desc *cd; struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain; + struct arm_smmu_master *master; cd = xa_load(&arm_smmu_asid_xa, asid); if (!cd) @@ -62,7 +67,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return cd; } - smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu_domain = cd_to_domain(cd); smmu = smmu_domain->smmu; ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, @@ -80,7 +85,11 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) * be some overlap between use of both ASIDs, until we invalidate the * TLB. */ - arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + arm_smmu_write_ctx_desc(master, 0, cd); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); /* Invalidate TLB entries previously associated with that context */ arm_smmu_tlb_inv_asid(smmu, asid); @@ -211,6 +220,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_master *master; + unsigned long flags; mutex_lock(&sva_lock); if (smmu_mn->cleared) { @@ -222,7 +233,11 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. */ - arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &quiet_cd); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + arm_smmu_write_ctx_desc(master, mm->pasid, &quiet_cd); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); @@ -244,7 +259,8 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { /* Allocate or get existing MMU notifier for this {domain, mm} pair */ static struct arm_smmu_mmu_notifier * -arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, +arm_smmu_mmu_notifier_get(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, struct mm_struct *mm) { int ret; @@ -279,7 +295,7 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, goto err_free_cd; } - ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + ret = arm_smmu_write_ctx_desc(master, mm->pasid, cd); if (ret) goto err_put_notifier; @@ -294,7 +310,8 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, return ERR_PTR(ret); } -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +static void arm_smmu_mmu_notifier_put(struct arm_smmu_master *master, + struct arm_smmu_mmu_notifier *smmu_mn) { struct mm_struct *mm = smmu_mn->mn.mm; struct arm_smmu_ctx_desc *cd = smmu_mn->cd; @@ -304,7 +321,7 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) return; list_del(&smmu_mn->list); - arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + arm_smmu_write_ctx_desc(master, mm->pasid, NULL); /* * If we went through clear(), we've already invalidated, and no @@ -348,7 +365,7 @@ __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) bond->sva.dev = dev; refcount_set(&bond->refs, 1); - bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + bond->smmu_mn = arm_smmu_mmu_notifier_get(master, smmu_domain, mm); if (IS_ERR(bond->smmu_mn)) { ret = PTR_ERR(bond->smmu_mn); goto err_free_bond; @@ -527,7 +544,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, if (!WARN_ON(!bond) && refcount_dec_and_test(&bond->refs)) { list_del(&bond->list); - arm_smmu_mmu_notifier_put(bond->smmu_mn); + arm_smmu_mmu_notifier_put(master, bond->smmu_mn); kfree(bond); } mutex_unlock(&sva_lock); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3fd83fb757227..cee3efff3c9fa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -965,14 +965,12 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } -static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, +static void arm_smmu_sync_cd(struct arm_smmu_master *master, int ssid, bool leaf) { size_t i; - unsigned long flags; - struct arm_smmu_master *master; struct arm_smmu_cmdq_batch cmds; - struct arm_smmu_device *smmu = smmu_domain->smmu; + struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_cmdq_ent cmd = { .opcode = CMDQ_OP_CFGI_CD, .cfgi = { @@ -981,16 +979,17 @@ static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, }, }; - cmds.num = 0; + /* + * There's nothing to sync if the STE isn't valid yet. + */ + if (!master->domain) + return; - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - for (i = 0; i < master->num_streams; i++) { - cmd.cfgi.sid = master->streams[i].id; - arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); - } + cmds.num = 0; + for (i = 0; i < master->num_streams; i++) { + cmd.cfgi.sid = master->streams[i].id; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_cmdq_batch_submit(smmu, &cmds); } @@ -1020,16 +1019,16 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, +static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) { __le64 *l1ptr; unsigned int idx; struct arm_smmu_l1_ctx_desc *l1_desc; - struct arm_smmu_device *smmu = smmu_domain->smmu; - struct arm_smmu_ctx_desc_cfg *cdcfg = &smmu_domain->s1_cfg.cdcfg; + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_ctx_desc_cfg *cdcfg = &master->s1_cfg.cdcfg; - if (smmu_domain->s1_cfg.s1fmt == STRTAB_STE_0_S1FMT_LINEAR) + if (master->s1_cfg.s1fmt == STRTAB_STE_0_S1FMT_LINEAR) return cdcfg->cdtab + ssid * CTXDESC_CD_DWORDS; idx = ssid >> CTXDESC_SPLIT; @@ -1041,13 +1040,13 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, l1ptr = cdcfg->cdtab + idx * CTXDESC_L1_DESC_DWORDS; arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); /* An invalid L1CD can be cached */ - arm_smmu_sync_cd(smmu_domain, ssid, false); + arm_smmu_sync_cd(master, ssid, false); } idx = ssid & (CTXDESC_L2_ENTRIES - 1); return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; } -int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, +int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { /* @@ -1059,16 +1058,16 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, * CD, then invalidate the old entry and mappings. * (4) Quiesce the context without clearing the valid bit. Disable * translation, and ignore any translation fault. - * (5) Remove a secondary CD. + * (5) Remove a CD. */ u64 val; bool cd_live; __le64 *cdptr; - if (WARN_ON(ssid >= (1 << smmu_domain->s1_cfg.s1cdmax))) + if (WARN_ON(ssid >= (1 << master->s1_cfg.s1cdmax))) return -E2BIG; - cdptr = arm_smmu_get_cd_ptr(smmu_domain, ssid); + cdptr = arm_smmu_get_cd_ptr(master, ssid); if (!cdptr) return -ENOMEM; @@ -1096,7 +1095,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, * order. Ensure that it observes valid values before reading * V=1. */ - arm_smmu_sync_cd(smmu_domain, ssid, true); + arm_smmu_sync_cd(master, ssid, true); val = cd->tcr | #ifdef __BIG_ENDIAN @@ -1108,7 +1107,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | CTXDESC_CD_0_V; - if (smmu_domain->stall_enabled) + if (master->stall_enabled) val |= CTXDESC_CD_0_S; } @@ -1122,19 +1121,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, * without first making the structure invalid. */ WRITE_ONCE(cdptr[0], cpu_to_le64(val)); - arm_smmu_sync_cd(smmu_domain, ssid, true); + arm_smmu_sync_cd(master, ssid, true); return 0; } -static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain) +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) { int ret; size_t l1size; size_t max_contexts; - struct arm_smmu_device *smmu = smmu_domain->smmu; - struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_s1_cfg *cfg = &master->s1_cfg; struct arm_smmu_ctx_desc_cfg *cdcfg = &cfg->cdcfg; + cfg->s1cdmax = master->ssid_bits; max_contexts = 1 << cfg->s1cdmax; if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_CDTAB) || @@ -1175,12 +1175,12 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain) return ret; } -static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) +static void arm_smmu_free_cd_tables(struct arm_smmu_master *master) { int i; size_t size, l1size; - struct arm_smmu_device *smmu = smmu_domain->smmu; - struct arm_smmu_ctx_desc_cfg *cdcfg = &smmu_domain->s1_cfg.cdcfg; + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_ctx_desc_cfg *cdcfg = &master->s1_cfg.cdcfg; if (cdcfg->l1_desc) { size = CTXDESC_L2_ENTRIES * (CTXDESC_CD_DWORDS << 3); @@ -1272,30 +1272,25 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, struct arm_smmu_device *smmu = NULL; struct arm_smmu_s1_cfg *s1_cfg = NULL; struct arm_smmu_s2_cfg *s2_cfg = NULL; - struct arm_smmu_domain *smmu_domain = NULL; struct arm_smmu_cmdq_ent prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, .prefetch = { .sid = sid, }, }; + struct iommu_domain *domain = NULL; if (master) { - smmu_domain = master->domain; smmu = master->smmu; - } - - if (smmu_domain) { - switch (smmu_domain->stage) { - case ARM_SMMU_DOMAIN_S1: - s1_cfg = &smmu_domain->s1_cfg; - break; - case ARM_SMMU_DOMAIN_S2: - case ARM_SMMU_DOMAIN_NESTED: - s2_cfg = &smmu_domain->s2_cfg; - break; - default: - break; + if (master->domain) + domain = &master->domain->domain; + } + if (domain) { + if (domain->type != IOMMU_DOMAIN_IDENTITY) { + if (master->s2_cfg) + s2_cfg = master->s2_cfg; + else + s1_cfg = &master->s1_cfg; } } @@ -1319,8 +1314,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, val = STRTAB_STE_0_V; /* Bypass/fault */ - if (!smmu_domain || !(s1_cfg || s2_cfg)) { - if (!smmu_domain && disable_bypass) + if (!(s1_cfg || s2_cfg)) { + if (disable_bypass) val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT); else val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_BYPASS); @@ -1863,7 +1858,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) * careful, 007. */ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); } else { cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; @@ -1946,7 +1941,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { cmd.opcode = smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ? CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; + cmd.tlbi.asid = smmu_domain->cd.asid; } else { cmd.opcode = CMDQ_OP_TLBI_S2_IPA; cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; @@ -2071,13 +2066,9 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) /* Free the CD and ASID, if we allocated them */ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; - /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); - if (cfg->cdcfg.cdtab) - arm_smmu_free_cd_tables(smmu_domain); - arm_smmu_free_asid(&cfg->cd); + arm_smmu_free_asid(&smmu_domain->cd); mutex_unlock(&arm_smmu_asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2088,67 +2079,45 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) kfree(smmu_domain); } -static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, - struct arm_smmu_master *master, +static int arm_smmu_domain_finalise_cd(struct arm_smmu_domain *smmu_domain, struct io_pgtable_cfg *pgtbl_cfg) { int ret; u32 asid; struct arm_smmu_device *smmu = smmu_domain->smmu; - struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; - refcount_set(&cfg->cd.refs, 1); + refcount_set(&cd->refs, 1); /* Prevent SVA from modifying the ASID until it is written to the CD */ mutex_lock(&arm_smmu_asid_lock); - ret = xa_alloc(&arm_smmu_asid_xa, &asid, &cfg->cd, + ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) goto out_unlock; - cfg->s1cdmax = master->ssid_bits; - - smmu_domain->stall_enabled = master->stall_enabled; - ret = arm_smmu_alloc_cd_tables(smmu_domain); - if (ret) - goto out_free_asid; - - cfg->cd.asid = (u16)asid; - cfg->cd.ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr; - cfg->cd.tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | + cd->asid = (u16)asid; + cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr; + cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - cfg->cd.mair = pgtbl_cfg->arm_lpae_s1_cfg.mair; - - /* - * Note that this will end up calling arm_smmu_sync_cd() before - * the master has been added to the devices list for this domain. - * This isn't an issue because the STE hasn't been installed yet. - */ - ret = arm_smmu_write_ctx_desc(smmu_domain, 0, &cfg->cd); - if (ret) - goto out_free_cd_tables; + cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair; mutex_unlock(&arm_smmu_asid_lock); return 0; -out_free_cd_tables: - arm_smmu_free_cd_tables(smmu_domain); -out_free_asid: - arm_smmu_free_asid(&cfg->cd); out_unlock: mutex_unlock(&arm_smmu_asid_lock); return ret; } static int arm_smmu_domain_finalise_s2(struct arm_smmu_domain *smmu_domain, - struct arm_smmu_master *master, struct io_pgtable_cfg *pgtbl_cfg) { int vmid; @@ -2173,8 +2142,7 @@ static int arm_smmu_domain_finalise_s2(struct arm_smmu_domain *smmu_domain, return 0; } -static int arm_smmu_domain_finalise(struct iommu_domain *domain, - struct arm_smmu_master *master) +static int arm_smmu_domain_finalise(struct iommu_domain *domain) { int ret; unsigned long ias, oas; @@ -2182,7 +2150,6 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain, struct io_pgtable_cfg pgtbl_cfg; struct io_pgtable_ops *pgtbl_ops; int (*finalise_stage_fn)(struct arm_smmu_domain *, - struct arm_smmu_master *, struct io_pgtable_cfg *); struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_device *smmu = smmu_domain->smmu; @@ -2204,7 +2171,7 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain, ias = min_t(unsigned long, ias, VA_BITS); oas = smmu->ias; fmt = ARM_64_LPAE_S1; - finalise_stage_fn = arm_smmu_domain_finalise_s1; + finalise_stage_fn = arm_smmu_domain_finalise_cd; break; case ARM_SMMU_DOMAIN_NESTED: case ARM_SMMU_DOMAIN_S2: @@ -2234,7 +2201,7 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain, domain->geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1; domain->geometry.force_aperture = true; - ret = finalise_stage_fn(smmu_domain, master, &pgtbl_cfg); + ret = finalise_stage_fn(smmu_domain, &pgtbl_cfg); if (ret < 0) { free_io_pgtable_ops(pgtbl_ops); return ret; @@ -2402,6 +2369,13 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) master->domain = NULL; master->ats_enabled = false; + master->s2_cfg = NULL; + /* + * Note that this will end up calling arm_smmu_sync_cd() even though + * we're about to destroy the entire STE anyways. This is ok because + * arm_smmu_sync_cd will exit early now that we've set domain to NULL; + */ + arm_smmu_write_ctx_desc(master, 0, NULL); arm_smmu_install_ste_for_dev(master); } @@ -2436,7 +2410,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (!smmu_domain->smmu) { smmu_domain->smmu = smmu; - ret = arm_smmu_domain_finalise(domain, master); + ret = arm_smmu_domain_finalise(domain); if (ret) { smmu_domain->smmu = NULL; goto out_unlock; @@ -2444,17 +2418,21 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } else if (smmu_domain->smmu != smmu) { ret = -EINVAL; goto out_unlock; - } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 && - master->ssid_bits != smmu_domain->s1_cfg.s1cdmax) { - ret = -EINVAL; - goto out_unlock; - } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 && - smmu_domain->stall_enabled != master->stall_enabled) { - ret = -EINVAL; - goto out_unlock; } - master->domain = smmu_domain; + /* + * Note that this will end up calling arm_smmu_sync_cd() before + * the master has been added to the devices list for this domain. + * This isn't an issue because the STE hasn't been installed yet. + */ + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + ret = arm_smmu_write_ctx_desc(master, 0, &smmu_domain->cd); + if (ret) + goto out_unlock; + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 || + smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED) { + master->s2_cfg = &smmu_domain->s2_cfg; + } /* * The SMMU does not support enabling ATS with bypass. When the STE is @@ -2466,6 +2444,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (smmu_domain->stage != ARM_SMMU_DOMAIN_BYPASS) master->ats_enabled = arm_smmu_ats_supported(master); + master->domain = smmu_domain; arm_smmu_install_ste_for_dev(master); spin_lock_irqsave(&smmu_domain->devices_lock, flags); @@ -2703,6 +2682,13 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; + ret = arm_smmu_alloc_cd_tables(master); + if (ret) { + arm_smmu_disable_pasid(master); + arm_smmu_remove_master(master); + goto err_free_master; + } + return &smmu->iommu; err_free_master: @@ -2718,6 +2704,7 @@ static void arm_smmu_release_device(struct device *dev) if (WARN_ON(arm_smmu_master_sva_enabled(master))) iopf_queue_remove_device(master->smmu->evtq.iopf, dev); arm_smmu_detach_dev(master); + arm_smmu_free_cd_tables(master); arm_smmu_disable_pasid(master); arm_smmu_remove_master(master); kfree(master); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index b574c58a34876..0b87c74bdf46e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -593,7 +593,6 @@ struct arm_smmu_ctx_desc_cfg { struct arm_smmu_s1_cfg { struct arm_smmu_ctx_desc_cfg cdcfg; - struct arm_smmu_ctx_desc cd; u8 s1fmt; u8 s1cdmax; }; @@ -689,6 +688,8 @@ struct arm_smmu_master { struct arm_smmu_domain *domain; struct list_head domain_head; struct arm_smmu_stream *streams; + struct arm_smmu_s1_cfg s1_cfg; + struct arm_smmu_s2_cfg *s2_cfg; unsigned int num_streams; bool ats_enabled; bool stall_enabled; @@ -707,25 +708,25 @@ enum arm_smmu_domain_stage { }; struct arm_smmu_domain { - struct arm_smmu_device *smmu; - struct mutex init_mutex; /* Protects smmu pointer */ + struct arm_smmu_device *smmu; + /* Protects smmu pointer */ + struct mutex init_mutex; - struct io_pgtable_ops *pgtbl_ops; - bool stall_enabled; - atomic_t nr_ats_masters; + struct io_pgtable_ops *pgtbl_ops; + atomic_t nr_ats_masters; - enum arm_smmu_domain_stage stage; + enum arm_smmu_domain_stage stage; union { - struct arm_smmu_s1_cfg s1_cfg; - struct arm_smmu_s2_cfg s2_cfg; + struct arm_smmu_ctx_desc cd; + struct arm_smmu_s2_cfg s2_cfg; }; - struct iommu_domain domain; + struct iommu_domain domain; - struct list_head devices; - spinlock_t devices_lock; + struct list_head devices; + spinlock_t devices_lock; - struct list_head mmu_notifiers; + struct list_head mmu_notifiers; }; static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) @@ -737,7 +738,7 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; -int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, +int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, From patchwork Wed May 10 20:50:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Shavit X-Patchwork-Id: 92300 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3928919vqo; Wed, 10 May 2023 14:25:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7vPNznpWBfxjLsKsbGNbjzrzC1WPloJ1PS9ZqSrt7yf20K6oRGHAeXMp+5ckK/Xll8Z3P5 X-Received: by 2002:a17:903:44c:b0:1ab:27e7:ad76 with SMTP id iw12-20020a170903044c00b001ab27e7ad76mr18254141plb.45.1683753944395; Wed, 10 May 2023 14:25:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683753944; cv=none; d=google.com; s=arc-20160816; b=HAXEtXZHvnLVqZsubfn3+uO3z942sK88RMO5EVtrT596jUrj9aM1EdFj3c4h41shSU qbd1QcPH56n21SQYwU2PMmmS/rNsiPyZ33zEUdsX7FVX1utg5HuBXjms/Tnijhxg/IO6 jAY0JUxy87ASfK0Z3wo7pekYays/MZeAbHQ6zVosTzFewpo2PkdAmGuUqzu09Y9nteWb ZceYru0Sz3+VyLEZSQpq1GcQSnzzD/HDvkhFm62utWHxstuwRcKgU7ouaUMJD5S/XfSu Da5fikWtUDO1kMFqBwh5OxVtntr3bNW2MyqLlTidPLD/NjegVyx8/eFaYa8dLCv3K8sa gm5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=JD2T39BHpHecI5MHeo7qoTsLquM2+wzDMKb634oTpbY=; b=oGbEoU2hxR6FEY5lIZqHl+CfqB1aRNMXE5+eLa3nhmjHQ+1I+u040+iVmmz1qcMLvG M7nVT5BqXyTWB8rR5mRWTKp3IAhqnCRNmodaRoXcsrNkGxHhmCFSkvOwsfBsG5v8HJKM /Gd7mR+6O+DjoMsYWeyidkFg9FzGn6sV6KJCj48JUWjk5xfNzQAQWZ45iNaBSxBL4JID MoqEtQRSYxvd5WIB0xd0OrFZdlVIn2RBzOre2xvOIxQM9jfdpD19x107ZAIowKMnYANW anStulyPLrQjFE+Q5GZDbL915JEpjrP+qPgWxfAy3jIA3JRDAMNk8EqINAJ3VA+LDRiW JXcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=rGEDpjfK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p9-20020a17090a680900b002479bbf3246si17932101pjj.124.2023.05.10.14.25.30; Wed, 10 May 2023 14:25:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=rGEDpjfK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237310AbjEJUwy (ORCPT + 99 others); Wed, 10 May 2023 16:52:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237182AbjEJUwN (ORCPT ); Wed, 10 May 2023 16:52:13 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 902CE7DAA for ; Wed, 10 May 2023 13:51:40 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-ba6386e6f7aso2797910276.3 for ; Wed, 10 May 2023 13:51:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683751870; x=1686343870; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JD2T39BHpHecI5MHeo7qoTsLquM2+wzDMKb634oTpbY=; b=rGEDpjfKVDSXnegP2Yoh1oS4RKGCm1S/1oh0YrY6TInsUUuT/TatwSrlR5ReWbSpVR 3CEPJJWurPhFZiuXAtH3qfoSNnpYU5t7ee5qT488zgJ7RlD97QkBhrvYViqMbX+qpdw4 bHWVtApzrNDQgPDj41iH5pfZsgCbPXXC/7XLzg+tUpY3lRKKv4NwXsRJGhBgg9bkNd6T dM7JDKt8kuPXk1NdgoJKuDBgvuKJFO7r2pfBj7HS+RnLwVuOiBsVt23rHj38lxD07exT Z1ASuFFRLZkAO7HaK6Taom7jnJaqOj5GAj9Pe7YYwcfnbrwtoaxSXgzy79zc/lPDAd0c ILuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683751870; x=1686343870; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JD2T39BHpHecI5MHeo7qoTsLquM2+wzDMKb634oTpbY=; b=eUhmaEHPfh68+5gWmqAeN7prMXWwesKiVLY7vHtn+bNrd9TmKDtHVqj6rUi1dAikjE pPos5zKrsQkd1vizdFyd+2ehcFiNsrTfa/SXYk960CmGVKuxsPkydAFElACQTEpdd4Dn MdrkJZGcO0tCC+66/HHRQhWUUUP+XKGW4EbPyVR1HID3TlhnQy6LPz/fzgZBc1TNbYLk +19buO61TQMOwSQzDkFEfL/lV99xD8szNR68j2nH8fA3/FAJ/Gqg86wP7Nkesf/4RG7x Lyjq4gp1uSJY0ihu2ue2yNztxxhFTSRVAe8q4Yn1K1/s0DqOjTlRCQpcJmFlVlD6TgtO 1Ryw== X-Gm-Message-State: AC+VfDytE4vv1MK4LB1T19f7KcP25EmQqfpbK9qEDR348W0AllUIDoZB E1XIQzPUFyx4rlfTPcKgYyel4tDsApVG X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:2b1f:8d06:7923:f154]) (user=mshavit job=sendgmr) by 2002:a25:bbc9:0:b0:b95:518b:4921 with SMTP id c9-20020a25bbc9000000b00b95518b4921mr8254206ybk.12.1683751870775; Wed, 10 May 2023 13:51:10 -0700 (PDT) Date: Thu, 11 May 2023 04:50:49 +0800 In-Reply-To: <20230510205054.2667898-1-mshavit@google.com> Mime-Version: 1.0 References: <20230510205054.2667898-1-mshavit@google.com> X-Mailer: git-send-email 2.40.1.521.gf1e218fcd8-goog Message-ID: <20230510205054.2667898-3-mshavit@google.com> Subject: [PATCH v1 2/5] iommu/arm-smmu-v3: Add has_stage1 field From: Michael Shavit To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Michael Shavit , jean-philippe@linaro.org, nicolinc@nvidia.com, jgg@nvidia.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765543976097735973?= X-GMAIL-MSGID: =?utf-8?q?1765543976097735973?= Inferring the state of the STE based on attached domains becomes tricker when multiple domains can be attached to a master on different PASIDs. The new field allows the smmu driver to directly query the state of the STE (S1 present, S2 present, neither) instead of inferring it from the attached domain. Signed-off-by: Michael Shavit --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 22 +++++++-------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 8 insertions(+), 15 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index cee3efff3c9fa..7b4399b5ba68b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -980,9 +980,9 @@ static void arm_smmu_sync_cd(struct arm_smmu_master *master, }; /* - * There's nothing to sync if the STE isn't valid yet. + * There's nothing to sync if stage 1 hasn't been installed yet. */ - if (!master->domain) + if (!master->has_stage1) return; cmds.num = 0; @@ -1278,20 +1278,11 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, .sid = sid, }, }; - struct iommu_domain *domain = NULL; - if (master) { smmu = master->smmu; - if (master->domain) - domain = &master->domain->domain; - } - if (domain) { - if (domain->type != IOMMU_DOMAIN_IDENTITY) { - if (master->s2_cfg) - s2_cfg = master->s2_cfg; - else - s1_cfg = &master->s1_cfg; - } + if (master->has_stage1) + s1_cfg = &master->s1_cfg; + s2_cfg = master->s2_cfg; } if (val & STRTAB_STE_0_V) { @@ -2367,9 +2358,9 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) list_del(&master->domain_head); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - master->domain = NULL; master->ats_enabled = false; master->s2_cfg = NULL; + master->has_stage1 = false; /* * Note that this will end up calling arm_smmu_sync_cd() even though * we're about to destroy the entire STE anyways. This is ok because @@ -2426,6 +2417,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) * This isn't an issue because the STE hasn't been installed yet. */ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + master->has_stage1 = true; ret = arm_smmu_write_ctx_desc(master, 0, &smmu_domain->cd); if (ret) goto out_unlock; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 0b87c74bdf46e..d715794572b13 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -689,6 +689,7 @@ struct arm_smmu_master { struct list_head domain_head; struct arm_smmu_stream *streams; struct arm_smmu_s1_cfg s1_cfg; + bool has_stage1; struct arm_smmu_s2_cfg *s2_cfg; unsigned int num_streams; bool ats_enabled; From patchwork Wed May 10 20:50:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Shavit X-Patchwork-Id: 92299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3928808vqo; Wed, 10 May 2023 14:25:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5El6Y6qlRaqFRkMSKc9LEmImB/2P9Q9Y8a/+vCq67UNLpNeSk4Bbgk4w/wgTh/tMcyNhX7 X-Received: by 2002:a17:902:7481:b0:1a9:7e26:d72 with SMTP id h1-20020a170902748100b001a97e260d72mr19496749pll.9.1683753927914; Wed, 10 May 2023 14:25:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683753927; cv=none; d=google.com; s=arc-20160816; b=gN0BDdQ/UNoBl4y3MCbg1CYnbPHwDmA/U2JjTkDiT/sVnzdLS7CSnB1JxRqpEUC56j uxAcEXW4NMbKcBv0gtdRnXwyHxMgmYSZ2haWsvYgtjX0Q8T7DGovSpTyQbs8PnmS2n8U KS+TB5rF8SWhEV7vhlNrcUfg65jNa8xK//Ai8n444wOk5yDhJFK1V2QKw0Q7Lke9FYIb LKyUq9EVoy3VGuRDajYHpNKt+e8UH59KV+UIKW/ORbZXAmoS+QFtfgydd3c+65/9auYa hqXkH/Z+//cmxZRU6LmIjzu5iAFn2t9IMqMOcSDq4QtPjC7nAXDjiXssNc/3KfPEwcLv 9Deg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=DMTIUjk5Ind647HSgdl65AQXCGi9ZS5fYo432OHDtSQ=; b=V2GfdJVeaSSwsCB5x/wws0pWA166SDC9/+Zj6SlwmdnvkDidhjGanZSSk/Qkv203R5 VdwTkyP6kKf4BhBo4/swnbg4sU04knaSiF3fE/HX2yH39VgeGG+itlxsuMfayXI/at19 eKmzWQoKfNCJL6wO6dh+xL59A8xb/3qKrV5xIVEtYZWY6S2IMq+8YXxsY+HS/NkNJiMl WdN4Sarg1F1drx49VjeCdpa/XkKGYiWVl9zY5/DYnWdLN+xaD5DWa0LxJydPmSD3E+rY HCynUATeFFhihFO0ocZLbNT+MWgnZlMLleUQpnwgHNwt5NAKI9ijxrbSFQLlWcIW08D5 J1kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=KPCdBXKK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c1-20020a170902b68100b001a6e9ae0c0bsi4883527pls.577.2023.05.10.14.25.15; Wed, 10 May 2023 14:25:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=KPCdBXKK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237345AbjEJUxM (ORCPT + 99 others); Wed, 10 May 2023 16:53:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237163AbjEJUw1 (ORCPT ); Wed, 10 May 2023 16:52:27 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C64E81707 for ; Wed, 10 May 2023 13:51:50 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-24dfc3c668dso4118420a91.1 for ; Wed, 10 May 2023 13:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683751876; x=1686343876; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DMTIUjk5Ind647HSgdl65AQXCGi9ZS5fYo432OHDtSQ=; b=KPCdBXKKrCR73vDaoJkOuz1eCOKkI8LYLClUu3ZrD6Rg2/zc1ZSSm1qiKat0BbRArq FzmRUM7rp0v+9GCIQNvhfqNy6gVsr5Z+UcEeGdXWHGtGkuO8UqQD+KxCtxreNbiahZ4y WRs9ZnTK8MAfnni7I6rJo6ZgF2pU32hZo8ijPo0ne33mYgduNBv5MO3HGN+bs9Od89QK i2LHTDL9KXS/yzLi1erv9ZaYyyIhnsYBtHV6e5vK7hSE63IfP6ma2qN7HjsasosoGTdT Q5gvPvvoQSCTowCCHHUlbYR2j7v0iItW60tukvbIeG92nL1GMCHNrDjF6xU3rK/ctXeO aY9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683751876; x=1686343876; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DMTIUjk5Ind647HSgdl65AQXCGi9ZS5fYo432OHDtSQ=; b=ZvSCU5qI+yoyK/GGjt0bQLbXs5pjTLm3pj5B0fijCHBydOUraDiS9qdmHczfv7cjIA meHHZXcjkrSaqA9Q28fqV7u7+TAy+4f6vWlcdV+yptT21S1WVeWazgET7EYJQ2lWMMkS xc/k7frV5rX0h0njZOwpBBjdxJMgd44V9wP/NAhe0WbkluzeKFZBkOAd/IB3hdPWSImG 9BBBP3byIaaR8gb5A0j5kqjTSjU7D9y40oqaT01NDhtiFtkbdYYg3Rt4B4eeQHfDte1q 9ViWJgPF2be0v1KdIXQVSGsfhALlurIxNZWpLLIIfAOXKOFN5W25GRSXj4By+V5YjMxp EAMg== X-Gm-Message-State: AC+VfDwcgUMOZmDG9J37kmLMebALa/5Z1RpS8GTN5cNhstQGZlt2aepr H4mdYEl6tK+jUlnwVl3XOyUVS1I/4s+U X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:2b1f:8d06:7923:f154]) (user=mshavit job=sendgmr) by 2002:a17:90a:db0f:b0:24d:f863:458c with SMTP id g15-20020a17090adb0f00b0024df863458cmr5910104pjv.1.1683751876335; Wed, 10 May 2023 13:51:16 -0700 (PDT) Date: Thu, 11 May 2023 04:50:50 +0800 In-Reply-To: <20230510205054.2667898-1-mshavit@google.com> Mime-Version: 1.0 References: <20230510205054.2667898-1-mshavit@google.com> X-Mailer: git-send-email 2.40.1.521.gf1e218fcd8-goog Message-ID: <20230510205054.2667898-4-mshavit@google.com> Subject: [PATCH v1 3/5] iommu/arm-smmu-v3: Simplify arm_smmu_enable_ats From: Michael Shavit To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Michael Shavit , jean-philippe@linaro.org, nicolinc@nvidia.com, jgg@nvidia.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765543958659928837?= X-GMAIL-MSGID: =?utf-8?q?1765543958659928837?= arm_smmu_enable_ats's call to inv_domain would trigger an invalidation for all masters that a domain is attached to everytime it's attached to another ATS-enabled master. It doesn't seem like those invalidations are necessary, and it's easier to reason about arm_smmu_enable_ats if it only issues invalidation commands for the current master. Signed-off-by: Michael Shavit --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7b4399b5ba68b..47dda287a4736 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2275,7 +2275,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master) pdev = to_pci_dev(master->dev); atomic_inc(&smmu_domain->nr_ats_masters); - arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); + arm_smmu_atc_inv_master(master); if (pci_enable_ats(pdev, stu)) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } From patchwork Wed May 10 20:50:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Shavit X-Patchwork-Id: 92301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3929025vqo; Wed, 10 May 2023 14:25:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6K1PXfQiFKXPfWC66AHZLxTjXEXqt3FxG8fOBmpw6ZQu8KNlRhkJ3UBMXWS7j0+RBoQn5e X-Received: by 2002:a05:6a00:1811:b0:63b:4313:f8c4 with SMTP id y17-20020a056a00181100b0063b4313f8c4mr25494373pfa.9.1683753957620; Wed, 10 May 2023 14:25:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683753957; cv=none; d=google.com; s=arc-20160816; b=zSwi3e2vQ+KdYyzFcScaeGlgZuEij+N7xJG9aYEKJdVSoDyRsBgZUKVBtJwWbx+/Ks BxhOl0w62qt9CS2R6vFikC1MBO6oFu88Ul0UvDQAWVXJX6wlI3vXCHnwvPj1VzMGKAAP EHo3vGNwRekpAEcw8gX/nMLVA7XG4+MhnONDihI2X+tcYB/h0G0jnuz9gyDRFj83HIHL F0oGtbTEMlUtvLil5LxM9gVFA7PsoDEyoauhWqstSJB/S1wrqLwdQb/lcnrYYX0LMGww iwjDTNBVS6BqPguUB3emwiEF35uXPSRh/35+Lb3NUlT/wgb8NnqAhNfJnkPuFmGQNOge nkXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4ydbIuDbfoHmFVfOIEtd0qpW93a6R3EVtBtLPsHyXDc=; b=ybZ00MCNApnzz3pwqn3UXyO+mOX6a4RNLl3bbTzao9giYhWsAv1FybzffJ4NKqopne PiFNhZLrwch9/gYD2r7fG8dnu8CNEqQfVBXMKgSqj+P/Urqab26UigqTRbjPPnUkgTYx BKRuvuUDRfQxeV6P8sv69OeCPADqmMKM5yMPf6RhhyPJp+TsNVk9pQHWaH4yFZA8FZmN UVQCa6Q1EClHVF66cqw6Ek5m2pUad55hDT5VRLPuX2YtbnLvrlezNse3JTnr0FPayuaV 2qudLfczKs7Zb5PiPE2aGFlBqBvf9MWM8La17xlq7C+HN+GX+XQa50e/Rpp4pzynvVUp zdrw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=e4JtwHGi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t67-20020a625f46000000b00643a251968fsi5911800pfb.350.2023.05.10.14.25.45; Wed, 10 May 2023 14:25:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=e4JtwHGi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236971AbjEJUxW (ORCPT + 99 others); Wed, 10 May 2023 16:53:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236940AbjEJUwa (ORCPT ); Wed, 10 May 2023 16:52:30 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD8FC2691 for ; Wed, 10 May 2023 13:51:53 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-52c219bf675so4144463a12.1 for ; Wed, 10 May 2023 13:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683751882; x=1686343882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4ydbIuDbfoHmFVfOIEtd0qpW93a6R3EVtBtLPsHyXDc=; b=e4JtwHGiJHq7PmJTlD5WKq6S+5A77DPe5iCNOUZU56syCxRB7l63NwGg/OLGtwFgsQ Qe+cP/9GKpMkvMOZGwyN2gGyWlpjcUVr86r5vqBlxoCMoPGsbdrTxk0ta8EHe5GNvfm/ IS/VuzvPd2qCMHdRQVR6/o1XRjslR4/Fkwi/yad75djwGWxySp6RJcIpxEMBs+C38Erj yYtgPUkLI3X+dLe5LYkq0uJlJmTtdtBoTj2X49Fyk7XEopSvSou6aP/QtG52tPJmlHMk hgQCARC94lLNFrqC+4GUoFkhyXIV/v6egX+ji/fUdxp9XdvOVjwvMD4HWE1D5Lc79YSO fapg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683751882; x=1686343882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4ydbIuDbfoHmFVfOIEtd0qpW93a6R3EVtBtLPsHyXDc=; b=MMmds02keKbtZEZrgfqmxiYROnGmGIvKrLe/bmZXDTZ2VO8DfzEAiW1+67BygBLzVs pJtq4PA3Phx3ORnoBDH3kJsScT50wFlhYlTQH1CvwDdsNcegakRKJ27xca3ODGk+CGCf fMGziWue5tb51rtdzT9eiO/2xeXHpKDbGHZctb+kFVIGCUKv3WAu1cE2QVLc0M8v9Aq2 Ds6s7NvCS+H2rF8lIJ6zSjzvkxeh0/TnAkHNYOsxME057+PFDxSBNHB7yVokMAaF2eiI hwrrPwwp4mPhrDNv6C5SYN8sy+T7cfRhafpVU6DfTbIWRocQsvOY1IUAnthSrBLaxF5b B6LA== X-Gm-Message-State: AC+VfDwjZU9OVNBK6+H+Qj3NOietsg5gH2eh8PFJt6u8YV++0v3mzk/K Lz6GAXmfJ6K/9X188+nvL6F6HiW93C2o X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:2b1f:8d06:7923:f154]) (user=mshavit job=sendgmr) by 2002:a63:80c8:0:b0:52c:bf5b:69ee with SMTP id j191-20020a6380c8000000b0052cbf5b69eemr4642357pgd.4.1683751882168; Wed, 10 May 2023 13:51:22 -0700 (PDT) Date: Thu, 11 May 2023 04:50:51 +0800 In-Reply-To: <20230510205054.2667898-1-mshavit@google.com> Mime-Version: 1.0 References: <20230510205054.2667898-1-mshavit@google.com> X-Mailer: git-send-email 2.40.1.521.gf1e218fcd8-goog Message-ID: <20230510205054.2667898-5-mshavit@google.com> Subject: [PATCH v1 4/5] iommu/arm-smmu-v3: Keep track of attached ssids From: Michael Shavit To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Michael Shavit , jean-philippe@linaro.org, nicolinc@nvidia.com, jgg@nvidia.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765543990082900039?= X-GMAIL-MSGID: =?utf-8?q?1765543990082900039?= The arm-smmu-v3 driver keeps track of all masters that a domain is attached so that it can re-write their STEs when the domain's ASID is upated by SVA. This tracking is also used to invalidate ATCs on all masters that a domain is attached. This change introduces a new data structures to track all the CD entries that a domain is attached to. This change is a pre-requisite to allow domain attachment on non 0 SSIDs. Signed-off-by: Michael Shavit --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 31 ++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 90 ++++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 17 ++-- 3 files changed, 89 insertions(+), 49 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index a721461b355c6..2eb066c0f3f99 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -53,7 +53,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) struct arm_smmu_ctx_desc *cd; struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain; - struct arm_smmu_master *master; + struct arm_smmu_attached_domain *attached_domain; cd = xa_load(&arm_smmu_asid_xa, asid); if (!cd) @@ -85,11 +85,11 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) * be some overlap between use of both ASIDs, until we invalidate the * TLB. */ - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - arm_smmu_write_ctx_desc(master, 0, cd); + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_for_each_entry(attached_domain, &smmu_domain->attached_domains, domain_head) { + arm_smmu_write_ctx_desc(attached_domain->master, attached_domain->ssid, cd); } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); /* Invalidate TLB entries previously associated with that context */ arm_smmu_tlb_inv_asid(smmu, asid); @@ -213,14 +213,14 @@ static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) arm_smmu_tlb_inv_range_asid(start, size, smmu_mn->cd->asid, PAGE_SIZE, false, smmu_domain); - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, start, size); + arm_smmu_atc_inv_domain_ssid(smmu_domain, mm->pasid, start, size); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - struct arm_smmu_master *master; + struct arm_smmu_attached_domain *attached_domain; unsigned long flags; mutex_lock(&sva_lock); @@ -233,14 +233,19 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. */ - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - arm_smmu_write_ctx_desc(master, mm->pasid, &quiet_cd); + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_for_each_entry(attached_domain, &smmu_domain->attached_domains, domain_head) { + /* + * SVA domains piggyback on the attached_domain with SSID 0. + */ + if (attached_domain->ssid == 0) + arm_smmu_write_ctx_desc(attached_domain->master, + mm->pasid, &quiet_cd); } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + arm_smmu_atc_inv_domain_ssid(smmu_domain, mm->pasid, 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -329,7 +334,7 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_master *master, */ if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + arm_smmu_atc_inv_domain_ssid(smmu_domain, mm->pasid, 0, 0); } /* Frees smmu_mn */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 47dda287a4736..81f49a86c1266 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1703,7 +1703,16 @@ static irqreturn_t arm_smmu_combined_irq_handler(int irq, void *dev) } static void -arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, +arm_smmu_atc_inv_cmd_set_ssid(int ssid, struct arm_smmu_cmdq_ent *cmd) +{ + *cmd = (struct arm_smmu_cmdq_ent) { + .substream_valid = !!ssid, + .atc.ssid = ssid, + }; +} + +static void +arm_smmu_atc_inv_to_cmd(unsigned long iova, size_t size, struct arm_smmu_cmdq_ent *cmd) { size_t log2_span; @@ -1728,8 +1737,8 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, */ *cmd = (struct arm_smmu_cmdq_ent) { .opcode = CMDQ_OP_ATC_INV, - .substream_valid = !!ssid, - .atc.ssid = ssid, + .substream_valid = false, + .atc.ssid = 0, }; if (!size) { @@ -1775,8 +1784,7 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) struct arm_smmu_cmdq_ent cmd; struct arm_smmu_cmdq_batch cmds; - arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); - + arm_smmu_atc_inv_to_cmd(0, 0, &cmd); cmds.num = 0; for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; @@ -1786,13 +1794,21 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size) +/* + * If ssid is 0, the domain is invalidated on all SSIDs that it is attached to. + * Otherwise, the domain is specifically invalidated on the provided SSID only. + * This second functionality is provided specifically for SVA which wants to + * invalidate domains on SSIDs that aren't recorded in the master's + * attached_domains list. + */ +int arm_smmu_atc_inv_domain_ssid(struct arm_smmu_domain *smmu_domain, int ssid, + unsigned long iova, size_t size) { int i; unsigned long flags; struct arm_smmu_cmdq_ent cmd; struct arm_smmu_master *master; + struct arm_smmu_attached_domain *attached_domain; struct arm_smmu_cmdq_batch cmds; if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS)) @@ -1815,25 +1831,35 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!atomic_read(&smmu_domain->nr_ats_masters)) return 0; - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); + arm_smmu_atc_inv_to_cmd(iova, size, &cmd); + if (ssid != 0) + arm_smmu_atc_inv_cmd_set_ssid(ssid, &cmd); cmds.num = 0; - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_for_each_entry(attached_domain, &smmu_domain->attached_domains, domain_head) { + master = attached_domain->master; if (!master->ats_enabled) continue; - + if (ssid == 0) + arm_smmu_atc_inv_cmd_set_ssid(attached_domain->ssid, &cmd); for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); } } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); } +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size) +{ + return arm_smmu_atc_inv_domain_ssid(smmu_domain, 0, iova, size); +} + /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { @@ -1855,7 +1881,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } - arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, @@ -1943,7 +1969,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, * Unfortunately, this can't be leaf-only since we may have * zapped an entire table. */ - arm_smmu_atc_inv_domain(smmu_domain, 0, iova, size); + arm_smmu_atc_inv_domain(smmu_domain, iova, size); } void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, @@ -2023,8 +2049,8 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) return NULL; mutex_init(&smmu_domain->init_mutex); - INIT_LIST_HEAD(&smmu_domain->devices); - spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->attached_domains); + spin_lock_init(&smmu_domain->attached_domains_lock); INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; @@ -2259,12 +2285,12 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master) return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)); } -static void arm_smmu_enable_ats(struct arm_smmu_master *master) +static void arm_smmu_enable_ats(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) { size_t stu; struct pci_dev *pdev; struct arm_smmu_device *smmu = master->smmu; - struct arm_smmu_domain *smmu_domain = master->domain; /* Don't enable ATS at the endpoint if it's not enabled in the STE */ if (!master->ats_enabled) @@ -2280,10 +2306,9 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } -static void arm_smmu_disable_ats(struct arm_smmu_master *master) +static void arm_smmu_disable_ats(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_domain *smmu_domain = master->domain; - if (!master->ats_enabled) return; @@ -2347,20 +2372,21 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) static void arm_smmu_detach_dev(struct arm_smmu_master *master) { unsigned long flags; - struct arm_smmu_domain *smmu_domain = master->domain; + struct arm_smmu_domain *smmu_domain = master->attached_domain.domain; if (!smmu_domain) return; - arm_smmu_disable_ats(master); + arm_smmu_disable_ats(master, smmu_domain); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del(&master->domain_head); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_del(&master->attached_domain.domain_head); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); master->ats_enabled = false; master->s2_cfg = NULL; master->has_stage1 = false; + master->attached_domain.domain = NULL; /* * Note that this will end up calling arm_smmu_sync_cd() even though * we're about to destroy the entire STE anyways. This is ok because @@ -2436,14 +2462,16 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (smmu_domain->stage != ARM_SMMU_DOMAIN_BYPASS) master->ats_enabled = arm_smmu_ats_supported(master); - master->domain = smmu_domain; + master->attached_domain.master = master; + master->attached_domain.domain = smmu_domain; + master->attached_domain.ssid = 0; arm_smmu_install_ste_for_dev(master); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_add(&master->domain_head, &smmu_domain->devices); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_add(&master->attached_domain.domain_head, &smmu_domain->attached_domains); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); - arm_smmu_enable_ats(master); + arm_smmu_enable_ats(master, smmu_domain); out_unlock: mutex_unlock(&smmu_domain->init_mutex); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index d715794572b13..35700534a0b4a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -681,11 +681,18 @@ struct arm_smmu_stream { struct rb_node node; }; +struct arm_smmu_attached_domain { + struct list_head domain_head; + struct arm_smmu_master *master; + int ssid; + struct arm_smmu_domain *domain; +}; + /* SMMU private data for each master */ struct arm_smmu_master { struct arm_smmu_device *smmu; struct device *dev; - struct arm_smmu_domain *domain; + struct arm_smmu_attached_domain attached_domain; struct list_head domain_head; struct arm_smmu_stream *streams; struct arm_smmu_s1_cfg s1_cfg; @@ -724,8 +731,8 @@ struct arm_smmu_domain { struct iommu_domain domain; - struct list_head devices; - spinlock_t devices_lock; + struct list_head attached_domains; + spinlock_t attached_domains_lock; struct list_head mmu_notifiers; }; @@ -746,8 +753,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size); +int arm_smmu_atc_inv_domain_ssid(struct arm_smmu_domain *smmu_domain, int ssid, + unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); From patchwork Wed May 10 20:50:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Shavit X-Patchwork-Id: 92298 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3928663vqo; Wed, 10 May 2023 14:25:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Od4b2eGn1yOkHLKvOJPP8hTEB9zTbC7a/ioD3uKxK7+9s6MBTbVm/U79s4lNvNRfQ2B2L X-Received: by 2002:a05:6a00:989:b0:62d:d045:392 with SMTP id u9-20020a056a00098900b0062dd0450392mr28265041pfg.32.1683753906587; Wed, 10 May 2023 14:25:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683753906; cv=none; d=google.com; s=arc-20160816; b=udU1j4gHqHpOd4uScodERHv8wk8H2O/AreGUC2HyGTd4RmApbVEONGVpQsThIUP3DW PnDyNTS6Z5Nl+P/VWNqpxrw/7IAbAPFH6dSvgBPsSSm9qNeOAN61aG+Kf3+bEuIYW2Fx lt1EK0DFK43fLyYj9RGpwNMCFzsbhWpQ2/6FMR994FkbUD0XyhHPVUxOJr0eg15sGnBH AtQxYS2lmq8PCYxBRmJWUixQdxbiLWouKijaC6zdXYmuqSaSqwUnD8y/nYlX6lyEzDhv HxxuOfI1BX0rhWQIufm82I2nyOsmfR5S3XAXx891/8c4AYedVG4/O3SJnofYprRZuYp1 LzoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Crl7UEul6Dcfq8QDpXhaI9Eed6TzM1uxfD1qvdE3PPQ=; b=vEpmujMc8UBrP65JfnbiPWQnpZAjW8/JLWtAp1uDbi4ppRONB099OJOpOtjwdBqqZe OMN00aw8yeR42nK3tSl2hmvjUMaaJ3b5XbJdSTmnM99ScdLenIb7zc5lEjPgj1kpJ/6E RRAUC3LjCox92YyV3hhVxlNBpXqUYX8nxVGsCR0R7v3C5GNfzPTeCSoat+U87wxULBaN 0rCbTeVo36a2QgzIQ117qOpA367UXMp/ul7sIU3RWw2csugWmLZtdFbNr6JP199ANr/1 gjaMY42MEbyFZzQqXfzoixOZOX7CeWcuoHXyUeFBhobGsLnd+G+pcg1oIkzoCCEwzwlm YAAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=E34iKIHb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o8-20020aa79788000000b006436ead4abasi5679230pfp.246.2023.05.10.14.24.53; Wed, 10 May 2023 14:25:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=E34iKIHb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237396AbjEJUxg (ORCPT + 99 others); Wed, 10 May 2023 16:53:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237056AbjEJUwl (ORCPT ); Wed, 10 May 2023 16:52:41 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C47AA93CF for ; Wed, 10 May 2023 13:52:03 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-b9a7766d1f2so9330135276.3 for ; Wed, 10 May 2023 13:52:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683751887; x=1686343887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Crl7UEul6Dcfq8QDpXhaI9Eed6TzM1uxfD1qvdE3PPQ=; b=E34iKIHbcw2iwAefkui6Y8fb8t4HLsOQjNqt1book/JiKc5MTJgfjTRLyefr2cQ5Tl ttxCxp801cxl+gipPupYLagjsf9cbJi+iujzmWCnorMJI/TqEFnHwO9qxkgkGpt8hOqh /6N+osvQn3gDLvcqOrZhZCyASDH4oP30isSwbDcvXYFxg/tcc3S6sQmkMTLTm2nrl2pw gPXVwqKrkdzby3WLUXuFz7nJKDgVC29y7MqRLTuO0Zoy3QXctrcHxQOicB/jM3nGgix5 pYOESBEWT/Isgxy0bnA5DVDmFRQCtPV301ZOAI+DwBAAqT6Kvrg2cgYImPalkXd+OqCj a+Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683751887; x=1686343887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Crl7UEul6Dcfq8QDpXhaI9Eed6TzM1uxfD1qvdE3PPQ=; b=JQ7SaBKdWyvHA95D5z4GjMiYkrVYlnejH0VsVKTBKIA689pcjEKbJSk1igFDr28RgP Urb9sp9Vp6tdYobsUjyxs89rWOPHCxsTbF4HBwsIDEQ3wryiiu9jlCTcOU0IjyQSPdvq A7SzMW2QnzifLqBszmh1M9qyleehFVvgONRxxVRTgI+KeNcdIAICbEAk82gcOgW9QHd0 DGO9KKA4U/N/HB/aEioankWjlRcGC35z/RACztggJk9B5kAp81r6af2vLBoM/92rqqmB LWvomHxgjjiZM7Bf4daENcCPz+qrW8cTEzJcsbLoKep8vu8XwcfodMYmeIY5+/9l2QYp 6WvQ== X-Gm-Message-State: AC+VfDyuFceLPAbG2qHzWJx+TiIr8/VShjQFzluSuQgOuMLO+56c3jSj B1aiyNz6bXYvlX9j9uZRdayZNrS4u8eK X-Received: from mshavit.ntc.corp.google.com ([2401:fa00:95:20c:2b1f:8d06:7923:f154]) (user=mshavit job=sendgmr) by 2002:a25:328e:0:b0:ba0:f92f:84e7 with SMTP id y136-20020a25328e000000b00ba0f92f84e7mr11905705yby.7.1683751887458; Wed, 10 May 2023 13:51:27 -0700 (PDT) Date: Thu, 11 May 2023 04:50:52 +0800 In-Reply-To: <20230510205054.2667898-1-mshavit@google.com> Mime-Version: 1.0 References: <20230510205054.2667898-1-mshavit@google.com> X-Mailer: git-send-email 2.40.1.521.gf1e218fcd8-goog Message-ID: <20230510205054.2667898-6-mshavit@google.com> Subject: [PATCH v1 5/5] iommu/arm-smmu-v3: Implement set_dev_pasid From: Michael Shavit To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Michael Shavit , jean-philippe@linaro.org, nicolinc@nvidia.com, jgg@nvidia.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765543936379682320?= X-GMAIL-MSGID: =?utf-8?q?1765543936379682320?= This change enables the use of the iommu_attach_dev_pasid API for UNMANAGED domains. The primary use-case is to allow in-kernel users of the iommu API to manage domains with PASID. This change also allows for future support of pasid in the DMA api. Signed-off-by: Michael Shavit --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 165 +++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 147 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 81f49a86c1266..468a7a30ffe7b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2353,6 +2353,11 @@ static int arm_smmu_enable_pasid(struct arm_smmu_master *master) return 0; } +static bool arm_smmu_master_has_pasid_domains(struct arm_smmu_master *master) +{ + return master->nr_attached_pasid_domains > 0; +} + static void arm_smmu_disable_pasid(struct arm_smmu_master *master) { struct pci_dev *pdev; @@ -2396,6 +2401,33 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) arm_smmu_install_ste_for_dev(master); } +/* + * Once attached for the first time, a domain can no longer be attached to any + * master with a distinct upstream SMMU. + */ +static int arm_smmu_prepare_domain_for_smmu(struct device *dev, + struct arm_smmu_device *smmu, + struct arm_smmu_domain *smmu_domain) +{ + int ret = 0; + + mutex_lock(&smmu_domain->init_mutex); + if (!smmu_domain->smmu) { + smmu_domain->smmu = smmu; + ret = arm_smmu_domain_finalise(&smmu_domain->domain); + if (ret) { + smmu_domain->smmu = NULL; + goto out_unlock; + } + } else if (smmu_domain->smmu != smmu) { + ret = -EINVAL; + goto out_unlock; + } +out_unlock: + mutex_unlock(&smmu_domain->init_mutex); + return ret; +} + static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) { int ret = 0; @@ -2411,6 +2443,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; + ret = arm_smmu_prepare_domain_for_smmu(dev, smmu, smmu_domain); + if (ret) + return ret; + /* * Checking that SVA is disabled ensures that this device isn't bound to * any mm, and can be safely detached from its old domain. Bonds cannot @@ -2421,22 +2457,18 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -EBUSY; } - arm_smmu_detach_dev(master); - - mutex_lock(&smmu_domain->init_mutex); - - if (!smmu_domain->smmu) { - smmu_domain->smmu = smmu; - ret = arm_smmu_domain_finalise(domain); - if (ret) { - smmu_domain->smmu = NULL; - goto out_unlock; - } - } else if (smmu_domain->smmu != smmu) { - ret = -EINVAL; - goto out_unlock; + /* + * Attaching a bypass or stage 2 domain would break any domains attached + * with pasid. Attaching an S1 domain should be feasible but requires + * more complicated logic to handle. + */ + if (arm_smmu_master_has_pasid_domains(master)) { + dev_err(dev, "cannot attach - domain attached with pasid\n"); + return -EBUSY; } + arm_smmu_detach_dev(master); + /* * Note that this will end up calling arm_smmu_sync_cd() before * the master has been added to the devices list for this domain. @@ -2446,7 +2478,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master->has_stage1 = true; ret = arm_smmu_write_ctx_desc(master, 0, &smmu_domain->cd); if (ret) - goto out_unlock; + return ret; } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 || smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED) { master->s2_cfg = &smmu_domain->s2_cfg; @@ -2473,11 +2505,74 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_enable_ats(master, smmu_domain); -out_unlock: - mutex_unlock(&smmu_domain->init_mutex); return ret; } +static int arm_smmu_set_dev_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t pasid) +{ + int ret = 0; + unsigned long flags; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_attached_domain *attached_domain; + struct arm_smmu_master *master; + + if (!fwspec) + return -ENOENT; + + master = dev_iommu_priv_get(dev); + smmu = master->smmu; + + ret = arm_smmu_prepare_domain_for_smmu(dev, smmu, smmu_domain); + if (ret) + return ret; + + if (pasid == 0) { + dev_err(dev, "pasid 0 is reserved for the device's primary domain\n"); + return -ENODEV; + } + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) { + dev_err(dev, "set_dev_pasid only supports stage 1 domains\n"); + return -EINVAL; + } + + if (!master->has_stage1 || master->s2_cfg) + return -EBUSY; + + attached_domain = kzalloc(sizeof(*attached_domain), GFP_KERNEL); + if (!attached_domain) + return -ENOMEM; + + attached_domain->master = master; + attached_domain->domain = smmu_domain; + attached_domain->ssid = pasid; + + master->nr_attached_pasid_domains += 1; + /* + * arm_smmu_share_asid may update the cd's asid value and write the + * ctx_desc for every attached_domains in the list. There's a potential + * race here regardless of whether we first the write the ctx_desc or + * first insert into the domain's list. Grabbing the asic_lock prevents + * SVA from changing the cd's ASID while the cd is being attached. + */ + mutex_lock(&arm_smmu_asid_lock); + ret = arm_smmu_write_ctx_desc(master, pasid, &smmu_domain->cd); + if (ret) { + mutex_unlock(&arm_smmu_asid_lock); + kfree(attached_domain); + } + + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_add(&attached_domain->domain_head, &smmu_domain->attached_domains); + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); + mutex_unlock(&arm_smmu_asid_lock); + + return 0; +} + static int arm_smmu_map_pages(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t pgsize, size_t pgcount, int prot, gfp_t gfp, size_t *mapped) @@ -2723,6 +2818,15 @@ static void arm_smmu_release_device(struct device *dev) if (WARN_ON(arm_smmu_master_sva_enabled(master))) iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + if (WARN_ON(master->nr_attached_pasid_domains != 0)) { + /* + * TODO: Do we need to handle this case? + * This requires a mechanism to obtain all the pasid domains + * that this master is attached to so that we can clean up the + * domain's attached_domain list. + */ + } + arm_smmu_detach_dev(master); arm_smmu_free_cd_tables(master); arm_smmu_disable_pasid(master); @@ -2858,12 +2962,34 @@ static int arm_smmu_def_domain_type(struct device *dev) static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) { struct iommu_domain *domain; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_domain *smmu_domain; + struct arm_smmu_attached_domain *attached_domain; + unsigned long flags; - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + if (!master || pasid == 0) + return; + + domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON(IS_ERR(domain)) || !domain) return; + if (domain->type == IOMMU_DOMAIN_SVA) + return arm_smmu_sva_remove_dev_pasid(domain, dev, pasid); - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid); + smmu_domain = to_smmu_domain(domain); + mutex_lock(&arm_smmu_asid_lock); + spin_lock_irqsave(&smmu_domain->attached_domains_lock, flags); + list_for_each_entry(attached_domain, &smmu_domain->attached_domains, domain_head) { + if (attached_domain->master != master || + attached_domain->ssid != pasid) + continue; + list_del(&attached_domain->domain_head); + break; + } + spin_unlock_irqrestore(&smmu_domain->attached_domains_lock, flags); + arm_smmu_write_ctx_desc(master, pasid, NULL); + master->nr_attached_pasid_domains -= 1; + mutex_unlock(&arm_smmu_asid_lock); } static struct iommu_ops arm_smmu_ops = { @@ -2883,6 +3009,7 @@ static struct iommu_ops arm_smmu_ops = { .owner = THIS_MODULE, .default_domain_ops = &(const struct iommu_domain_ops) { .attach_dev = arm_smmu_attach_dev, + .set_dev_pasid = arm_smmu_set_dev_pasid, .map_pages = arm_smmu_map_pages, .unmap_pages = arm_smmu_unmap_pages, .flush_iotlb_all = arm_smmu_flush_iotlb_all, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 35700534a0b4a..9ef63ea381f4b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -705,6 +705,7 @@ struct arm_smmu_master { bool iopf_enabled; struct list_head bonds; unsigned int ssid_bits; + unsigned int nr_attached_pasid_domains; }; /* SMMU private data for an IOMMU domain */