From patchwork Tue Feb 28 04:21:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ganapatrao Kulkarni X-Patchwork-Id: 62261 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2809671wrd; Mon, 27 Feb 2023 20:41:36 -0800 (PST) X-Google-Smtp-Source: AK7set+KiIenrB+p4PvwlKZiqXZ1KOU5oLgDJ8fD85epkCQR7E9OfDfzhEdhGYzz/csk0SnHARGZ X-Received: by 2002:a05:6402:5249:b0:4af:70a5:560b with SMTP id t9-20020a056402524900b004af70a5560bmr13839441edd.9.1677559296758; Mon, 27 Feb 2023 20:41:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1677559296; cv=pass; d=google.com; s=arc-20160816; b=XvwPmkMiAEBKuJJc+SfOxWxP2khGdwfo8hI5LRyEPpW8TDeJSvK9lQQhka5itOJSGT F55mSmjntZR6efgQGOmbi+Iv9fMauo1fqTHSkmIWS9I6u0SMKlTw+UvXNwzNA0eMLdrC AUlPYme3I2c6mhkZWbkrMxNc9O60TWwL7+VOVeqB2YXd4e65LhTPu1X965EJyBW6dNLE exo5Fhd8NP+zfiRqO0q3JSRSWca+QGxSJJ3gjcZSxolVg3iXE/lCQxPNUwrZfmyPOPAW WLUplIA73ysDKKnsa2xLrwMZxLEHGNJ3EFmdyOjhz7VhXxx96NkLNS+9NXDKpXfS5VU5 m/0g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SdyHm+N58UlbZO7Gmi6onc7rUlBRHFd8CHGNB8G2p8I=; b=j71ZvVtmR7EsLoAtTsykSQVMYnJrUt4VAYEXinzr4PFfDIimTVbh4HZtkEMSIpgdcG 8WKHoyAveE20z21Uh0wVISTdRfZxKT9eFQquFFGWonW6bLe21SFiPc/2S2kDWxcZlop3 ECerghHAZ8KKrJ4aeC6pJJposkjvex+AwmotyoYh+WTKoLJU8OaQ9w+9O3Lo2HQSEkmy uDIOXmZRNO3KAtiQkI+SOg1045T89QIBiLuDCfwNEJkJK+yDb9d2C+8QU/Nl3OWQvpe2 qtGCB3dJnYDdkvCP69VoHQPpT2/aN9iX2kr16Ko/OzZ9a04qwNzQ5mEMcrCOrWOBGSIp 6KKA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@os.amperecomputing.com header.s=selector2 header.b=W9DjsTcy; arc=pass (i=1 spf=pass spfdomain=os.amperecomputing.com dkim=pass dkdomain=os.amperecomputing.com dmarc=pass fromdomain=os.amperecomputing.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amperecomputing.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t13-20020a056402020d00b004acc42357cfsi2665269edv.364.2023.02.27.20.41.11; Mon, 27 Feb 2023 20:41:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@os.amperecomputing.com header.s=selector2 header.b=W9DjsTcy; arc=pass (i=1 spf=pass spfdomain=os.amperecomputing.com dkim=pass dkdomain=os.amperecomputing.com dmarc=pass fromdomain=os.amperecomputing.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amperecomputing.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229787AbjB1EWB (ORCPT + 99 others); Mon, 27 Feb 2023 23:22:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229543AbjB1EV7 (ORCPT ); Mon, 27 Feb 2023 23:21:59 -0500 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2116.outbound.protection.outlook.com [40.107.244.116]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DEA7265BD; Mon, 27 Feb 2023 20:21:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HhCOoGoS062MXqexnIUacdbTttlCWhWuxEcI7rQP7lJseuM4hzEoyUFvWNa6UmkDR7vZaHCas0mJf98UHKJx3UnrCqUH0WWck3hVrBeArC34jQmaemKNGYitkGbcNT9PZqGY8t80XNEWKNFLH+tUPRHb5XKYO606pU08wa0OU5l+0lbDCuBW7arF2hwBGk2hyGcXJFBq0seQdUTA+k6bn+7Bz47bEz/MAtgOY7HurDkeRzn0CkqNnB3xkTd9CJd5BDgVyd1tC63unj3mo+BXX3ftasMd1DflSB8izClMfII0/UjV+5Q/8WFjYgf9W1n3yUBtww9tFBK4Pyv6nsUBhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SdyHm+N58UlbZO7Gmi6onc7rUlBRHFd8CHGNB8G2p8I=; b=SyJus1RS3EsbK0pKT0fJbdSHy3d+xBK5Gaw35PiMQnIJYZineVnvU/4kU/FYMMySZJ0kNQzxvy59bVHnuuCCICQWXqdpP7i5a7EXsucCE9pMWGJeDu2AaTFzCAUi7WgSIWdSmogI86+81jRC+5X28JuEha4vhJzb86I2G6HcoonBlM0Zri174U5Hm18RmhiFlC8QPqUl3I5XMKrnLMe9X+KXQ/DFvpmmCmdFgn6o8NMrxDL6LZxKCx2H2ZMvEnpbbGtDK8s6qo89LMO9VzC6vDrLsEc6GROaoAotkNnHN9ZKjOzVQ6ewIjBGqJval9GSAQeAoABIVS3f3X7GWo/Z3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SdyHm+N58UlbZO7Gmi6onc7rUlBRHFd8CHGNB8G2p8I=; b=W9DjsTcyIqbem95FUShNmVvei6ZCT0alGebxHZxeWu3KIHcLJxiAlp0r7NNOcrbmgJQSSeAsaZDLudBQ8NuAWoeDlsQ2UtfWhU7BxGE5H/I3C847IWeS2OIzAEY/hfhDAAKagxhyjaIwOCAFnOBFfssQOI9C5sz7yXF8hk2/ge8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) by DM6PR01MB4700.prod.exchangelabs.com (2603:10b6:5:65::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.25; Tue, 28 Feb 2023 04:21:53 +0000 Received: from DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a]) by DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a%6]) with mapi id 15.20.6134.029; Tue, 28 Feb 2023 04:21:53 +0000 From: Ganapatrao Kulkarni To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, joro@8bytes.org, bhelgaas@google.com, robin.murphy@arm.com, will@kernel.org Cc: jean-philippe@linaro.org, sathyanarayanan.kuppuswamy@linux.intel.com, darren@os.amperecomputing.com, scott@os.amperecomputing.com, gankulkarni@os.amperecomputing.com Subject: [PATCH v2 1/2] PCI/ATS: Add a helper function to configure ATS STU of a PF Date: Mon, 27 Feb 2023 20:21:36 -0800 Message-Id: <20230228042137.1941024-2-gankulkarni@os.amperecomputing.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230228042137.1941024-1-gankulkarni@os.amperecomputing.com> References: <20230228042137.1941024-1-gankulkarni@os.amperecomputing.com> X-ClientProxiedBy: CH0PR04CA0109.namprd04.prod.outlook.com (2603:10b6:610:75::24) To DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM8PR01MB6824:EE_|DM6PR01MB4700:EE_ X-MS-Office365-Filtering-Correlation-Id: a962cd65-22f7-48e2-c64b-08db19435232 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Q9dpGATL/ylPETjt2LQFOPFoQBqo5TSnCRFM/UwjMeFaVy4aHLgZnK0teURnX7GKGGpvA0GFX5PdA16ZGJ73ChKoPdMcPc5kx2vYdF5Qqp21ksRvv7WURmFAzFFucKzHDw+/V4EXSk1YqDe/2lnAggyL+SvPf3dKDizl/2seSvqPAcuyRQ4Y/1nANNBg6GuhV2ELL/Z4qdPl0BrDIdQntGJq7/xS/31FQV2qhMzy1D+YnhO5IgFQ7fD8W9cDJJkzs4W52jLsQ8K4sTMIKTZa6NlM5HWmtkH1+TOyRB4Okg4Lt0wJpQ2+6S8E8+5w1HeXmowja1n/AmG+9jubXuuvkn+S14v51zqZp+8luQHmwtEVgt6fg6wG21QQku/0e7J2H2M1z9w4V4ovNk5lIJbdoCsJzYya52McCCRrnezJAT6EyIFVINwsx0kbn+Bf5mcTsRt91KiGVZvh8DdmavD2krtY1l1QNWDpeGedRdEgzkITXxyI/qKmqUcR3SNZNzNswFdTxT9e5isMRxw8LxC/MoPJz2G/02+Q3y3PBRBct8fq/1covXN9Xyn/6NuaCliIrAl36qxVWMyb4pH9PLj4S+O8Q9hQpJPbFNui7XPeJ50RUUL2HNDVVjcoMnCo/5ort5V2NZE/gzAQTK0djx61w1ibU1DyYexq6x72A2t9Fz6qx8V4ORDQl2QRQ5IhPZm/7HUkDBYmUVJZ+Bx6uHqVPA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR01MB6824.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(346002)(366004)(39850400004)(396003)(376002)(136003)(451199018)(86362001)(316002)(6486002)(186003)(8676002)(7416002)(66946007)(41300700001)(66556008)(6512007)(4326008)(8936002)(478600001)(1076003)(2616005)(6666004)(26005)(5660300002)(107886003)(6506007)(66476007)(52116002)(38350700002)(2906002)(38100700002)(83380400001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /j9GOjpUcozFwNSp2aA4l2zd4PSOXc/7YKj1/RGG8uGE32RVVBs8OWYlBFFId4sPnvojP0QfYj8n/IN6n/Gspe3fOjA28Kq53BEq2fKX4Aw2W87b90ZBMGhD/ENKvwt0wwe3hRC2mY0JFEM7K1Ag2iQDAWZ4cYMy9K/Klwcb/WjD5BD++QH/Y2dAP4+pqrZkhxQDFgUjSRMsobBXFtKKwNiXW4NK2af81mWM6iQt5w+WPgZn0IWeW7tfnTl1o/q2T6ZJAcPuMD+n3A7Dmn1UmdvKOhbVHjzzi+vkZZKRP/2FCfJ9Q7EkX648qqhu5BQjWXXtMM6F21IySpKuuwyLMDRPoahhhDQPm5XQgZOlGu8anB0VkFYRQp2r+8pdtmpWre5jBzeeNIh6m9Kee+6w3lmFoldR8DJL/s7SeZ45EBIPbQbfy3k34qlCZXpDTeN9YTq7riYgLBFK5L5uJOv1egLALDNF0ZRUbFbcoN9omKIedVnTUpU74dhTrUMH2AKif0WEEjm2pkX0HZxrLXPj5xl+ZpKT4qBzMbf2NXUEL7kSF7UTNsAyGufb0FjMxKS7rqC3zXKVblwCbpnIdNHVRZ/yHSZVQ2Ol8n1JLqpM78fLVpUzJ9Slg4d50WcyABic4gWqJUngInDGOoE3mYPb6GgukyImRD8YLN+AjnketimYnCTiysoDvh97sbRdPA2B8Ta6+BjS01VwqxnfkOBB9R5ivWhGK2E5ao+kDORsOVXiXmux2ShbGiKfCAFcZp0zdEb2haOprb3sjwbmbuxcRt2El7tAGs+8BGDYN8qcJQynrjWBMPVlV1Iy52D1DORBWhcwR/CbxojgHpIt41MEDZhx0JrqbGlfcvuJxrH4Q95gpJnBi3CDlZIDWgPuAs1FJH95kycaSFW0YSQkY3umdf3EQrElJdxX8eSDZ43VDrxel5uQcoHS5nTYdXeM8p1nQIrd+/MamDRy5FMDLFLOD86Z3ZlfRIksx6SHwehYKGp3ZTja5QHgLCXKZIpdbEryLXnVVjzBgDcKEDyWpwsBNJGVtsZDAvcsG/GZPqfFQwUT2DtwUhzTYN3sXrxJZ/7njoC2COrfQ6nKz5wUp42OldEW5jW/BHLKM0ksGQ6iwJvRQJBVxCtvQv2HMJ1XGr4DKTcRGAzoZ49lhyJOeQo6FewnjLCchdseDYUGwC09hBF6kICdV1DcnWQWABRXkivUTFaSOBFlWSdYDdYLEI1bPbI3Wv+LxOdHqTS5wdWAXI8wTy3hXPQbcmwrzhKqN2o5DGDDc9PKycDkJh+8DcL2vGGTiap/IFUJxZ2ITWwjpaHbXYMsmj9IRqjrXvaIqUHZ9gPAT7zar25En3Sk+00C7y1YqRfz08tM7Nxq3D2xMcA6+8Fpuu1V/g5DD3SiB1RnS1XyyW9XfG6Mu78YV34jjXj+1atSUSNf1zNhZLpPc52zSzWYJEFY4+pdTtgL74t1sNc+BHgTfES/y1xIpeeZajBXTOxg+8BJxdHD6Oy0OIGedvpD6N2X5aDHfjCZT8rtpW/sSxvENHvFIxvdG8kvxrDbQks8R2yPzVhbBrMi0ORnjMt/DEwHnllhMF/94LYcE7Pqmb+q5ZxnqrhoZqnG3gfB8QiAl2pbyfuGw4QxlqA= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: a962cd65-22f7-48e2-c64b-08db19435232 X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2023 04:21:53.6869 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qtkrA5WqKV9fGSlsuWS2l1Ta4B29MZsGp44AH1lB0cWlUvvgxM63ItBZfB7prfKQC0w6ouaMzyyKRHYsqf4C0IwipgBk5hKHQR/5N2oCgeBE+bMc3h6b84OOU7kiZvKw X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR01MB4700 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759048416967071707?= X-GMAIL-MSGID: =?utf-8?q?1759048416967071707?= As per PCI specification (PCI Express Base Specification Revision 6.0, Section 10.5) both PF and VFs of a PCI EP are permitted to be enabled independently for ATS capability, however the STU(Smallest Translation Unit) is shared between PF and VFs. For VFs, it is hardwired to Zero and the associated PF's value applies to VFs. In the current code, the STU is being configured while enabling the PF ATS. Hence, it is not able to enable ATS for VFs, if it is not enabled on the associated PF already. Adding a function pci_ats_stu_configure(), which can be called to configure the STU during PF enumeration. Latter enumerations of VFs can successfully enable ATS independently. Signed-off-by: Ganapatrao Kulkarni Acked-by: Bjorn Helgaas --- drivers/pci/ats.c | 33 +++++++++++++++++++++++++++++++-- include/linux/pci-ats.h | 3 +++ 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index f9cc2e10b676..1611bfa1d5da 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -46,6 +46,35 @@ bool pci_ats_supported(struct pci_dev *dev) } EXPORT_SYMBOL_GPL(pci_ats_supported); +/** + * pci_ats_stu_configure - Configure STU of a PF. + * @dev: the PCI device + * @ps: the IOMMU page shift + * + * Returns 0 on success, or negative on failure. + */ +int pci_ats_stu_configure(struct pci_dev *dev, int ps) +{ + u16 ctrl; + + if (dev->ats_enabled || dev->is_virtfn) + return 0; + + if (!pci_ats_supported(dev)) + return -EINVAL; + + if (ps < PCI_ATS_MIN_STU) + return -EINVAL; + + dev->ats_stu = ps; + pci_read_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, &ctrl); + ctrl |= PCI_ATS_CTRL_STU(dev->ats_stu - PCI_ATS_MIN_STU); + pci_write_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, ctrl); + + return 0; +} +EXPORT_SYMBOL_GPL(pci_ats_stu_configure); + /** * pci_enable_ats - enable the ATS capability * @dev: the PCI device @@ -68,8 +97,8 @@ int pci_enable_ats(struct pci_dev *dev, int ps) return -EINVAL; /* - * Note that enabling ATS on a VF fails unless it's already enabled - * with the same STU on the PF. + * Note that enabling ATS on a VF fails unless it's already + * configured with the same STU on the PF. */ ctrl = PCI_ATS_CTRL_ENABLE; if (dev->is_virtfn) { diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index df54cd5b15db..7d62a92aaf23 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -8,6 +8,7 @@ /* Address Translation Service */ bool pci_ats_supported(struct pci_dev *dev); int pci_enable_ats(struct pci_dev *dev, int ps); +int pci_ats_stu_configure(struct pci_dev *dev, int ps); void pci_disable_ats(struct pci_dev *dev); int pci_ats_queue_depth(struct pci_dev *dev); int pci_ats_page_aligned(struct pci_dev *dev); @@ -16,6 +17,8 @@ static inline bool pci_ats_supported(struct pci_dev *d) { return false; } static inline int pci_enable_ats(struct pci_dev *d, int ps) { return -ENODEV; } +static inline int pci_ats_stu_configure(struct pci_dev *d, int ps) +{ return -ENODEV; } static inline void pci_disable_ats(struct pci_dev *d) { } static inline int pci_ats_queue_depth(struct pci_dev *d) { return -ENODEV; } From patchwork Tue Feb 28 04:21:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ganapatrao Kulkarni X-Patchwork-Id: 62262 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2810699wrd; Mon, 27 Feb 2023 20:45:00 -0800 (PST) X-Google-Smtp-Source: AK7set8YYuxi+EBrj0gWOEVmHZd//SsPQRGhFhVBn/unCEvrf1LaVtWyfdBFQ9wIpz3XJ5wWHpyJ X-Received: by 2002:a17:906:85cb:b0:8b2:3e72:1022 with SMTP id i11-20020a17090685cb00b008b23e721022mr992134ejy.29.1677559500761; Mon, 27 Feb 2023 20:45:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1677559500; cv=pass; d=google.com; s=arc-20160816; b=JrtP4khF/eLYCB1CuO2rJmXpff5+kbloH0PvmWveQtbJ1cKowik5AHO0+c/iUZKQ0Z BvnpnqH79bbtFuhdf1XTbIXzaD8m4AP9U0smaHY7CbvT60HrCOt30fW11+LFaNWnssaG HCxx/r9DV4MaeIBCm+DxqdGnYGG9NLJIaSUOcUDha5pWxmaeYX/5709kEhs+gKpUyVe9 Eku0FQJzo5rRCpurc3g3k9yu+JT9Qyzqy81muVFCQnTMJw855NsFsabhzpE/7KqZ02ru f5m8mn1Bdc+xkKy8dAEOyB4WjGKM8c1YYAPVhp+UX0uuwhwOPAuMgj3tJvUoWa6ntRSo LWkg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zAt2Jdw+BFCu/M780DOdKTlmXUiMtgpBo93yRj4aDVQ=; b=SxDK5KjTV4pdZM8eom+oDDpQzq6B/2Y55KGNr1l8YxfveztOCSPNFjsMrWPNgALNHn yhfGIiWHOxL+tsbje+n58oUWxTST6sPiysg484NL+3+W8zrUXq1hRu4Ooz5etmqlJrXX i4nX4UpazcUYUAN12h0UAXxlZZZxyPBPPrH7gE94bznPmAUcj82bUbP48zrVxtMUk7KZ IGSCAarFm3FXYfcrk5z7s3V0/P82UP08AFVMV4IUrLs2UMEXl+XBWUts/+bJO9498BZz nHCmG3wbloEuFrIhL5MLoJGeSz/cZBdiZzAEbCmU6qTFd2njK5KxEQhK1heENMJzDgPl 954Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@os.amperecomputing.com header.s=selector2 header.b=bIJ4aXpa; arc=pass (i=1 spf=pass spfdomain=os.amperecomputing.com dkim=pass dkdomain=os.amperecomputing.com dmarc=pass fromdomain=os.amperecomputing.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amperecomputing.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id de1-20020a1709069bc100b008e6266cd19csi9489026ejc.935.2023.02.27.20.44.38; Mon, 27 Feb 2023 20:45:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@os.amperecomputing.com header.s=selector2 header.b=bIJ4aXpa; arc=pass (i=1 spf=pass spfdomain=os.amperecomputing.com dkim=pass dkdomain=os.amperecomputing.com dmarc=pass fromdomain=os.amperecomputing.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amperecomputing.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230059AbjB1EWJ (ORCPT + 99 others); Mon, 27 Feb 2023 23:22:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229660AbjB1EWA (ORCPT ); Mon, 27 Feb 2023 23:22:00 -0500 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2130.outbound.protection.outlook.com [40.107.237.130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D6D926847; Mon, 27 Feb 2023 20:21:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K25u0kyDiART4BZYuzP8dB5e6UmBaa9gqVZdlg+QG3hWfV6gvKC0DhTUdNWdaLTN0Wae2KSyJzIRDt2qw2rR6GNWvRkYhJVMDUQi6U6g+QjdjdVGaiA7I7p1KsuOf2gEvSWFgwPWcP2xKg/FiefEyqQ0rsMA883vsXGkeLj8ab2BEWri2cdwHOTLfQQWalxrqOQOahexi6+mAqgh+3fQsCjICRmYeGRw4892EK6OaE4hZUrH8VNM4HMC6pHdAVLN3NCzkgkcJafSiMH/hTYiOACNH7Fu0lIWtYZo6XM5tdWplsaqOfYmzsQvGgXe6RQbOH9WArkRLzvrn+YVfZR4Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zAt2Jdw+BFCu/M780DOdKTlmXUiMtgpBo93yRj4aDVQ=; b=F6Z40zQkhBYvvkGLM6/54Da7ZdbwkHbO5KjEziYth++UJyO6HFqVY/rhRRhsePPjJyjAdYL1ClpsMa+Q732g6OA5R+esZ+SIFs+FO2fAYCA9RG7pm70QgPTb8hljSzvW8D6vVUHMZdA4VWrwMfqAUuyLTRkP7D6Sniw+hYB+edKFy9MlZljESJrnQ4yvF79bL/urTEqVwGfs58lBgMJ9qoqYfIGFi7aqIiEImHeHch+aK9+7imeItIsUDX+lhtWWUBOEHSQW4OZBCFDZ0SeYd6NKW2UlfKrtg2CLGLJ3Crg3aP5pyEW4EmqM1+hrFAIv8/U7jsF3w+KAHEaNOmZVUA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zAt2Jdw+BFCu/M780DOdKTlmXUiMtgpBo93yRj4aDVQ=; b=bIJ4aXpa1CYEUHux2GLyheShEA0viOWStNR66RBitab3R4zy7yocq2CjMFG6GQPIpB4iqHPyr9WRUGjbSAp4reAgXWC0sEksfjwVH9O+F/eYM2fitLwgsQbZ+85kpw1JznWjDGAZ1jptxhn10ruz1Gf/m5bALYWoKzYGVQ0OqOk= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) by MN2PR01MB5422.prod.exchangelabs.com (2603:10b6:208:112::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.19; Tue, 28 Feb 2023 04:21:55 +0000 Received: from DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a]) by DM8PR01MB6824.prod.exchangelabs.com ([fe80::fc8b:5e5e:a850:7f0a%6]) with mapi id 15.20.6134.029; Tue, 28 Feb 2023 04:21:54 +0000 From: Ganapatrao Kulkarni To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, joro@8bytes.org, bhelgaas@google.com, robin.murphy@arm.com, will@kernel.org Cc: jean-philippe@linaro.org, sathyanarayanan.kuppuswamy@linux.intel.com, darren@os.amperecomputing.com, scott@os.amperecomputing.com, gankulkarni@os.amperecomputing.com Subject: [PATCH v2 2/2] iommu/arm-smmu-v3: Configure STU of a PF if ATS is not enabled. Date: Mon, 27 Feb 2023 20:21:37 -0800 Message-Id: <20230228042137.1941024-3-gankulkarni@os.amperecomputing.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20230228042137.1941024-1-gankulkarni@os.amperecomputing.com> References: <20230228042137.1941024-1-gankulkarni@os.amperecomputing.com> X-ClientProxiedBy: CH0PR04CA0109.namprd04.prod.outlook.com (2603:10b6:610:75::24) To DM8PR01MB6824.prod.exchangelabs.com (2603:10b6:8:23::24) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM8PR01MB6824:EE_|MN2PR01MB5422:EE_ X-MS-Office365-Filtering-Correlation-Id: 44c0fe17-16a2-4c15-5fc2-08db194352ec X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PBLiy04fbeHDYfkru8NSzeZf8QrFMJHsfLWgyXlh9h5BahmSfdy+AqZjvifzkYd6vT0IucQ4ZHuB470el2oQ+WlZIFVm3jg8wubtLD6Q9uKPsSrL28IzISexj3RfiPRZBc6KLJjOeU4X2pRI4rcgI7H009R0rtjNeXbMvltoKB4p4R6VJAIP905Z3T38lJw+n+Aco6t/AFnVgQ8leniAQuJXNowLbT7T1GB1887Se+fPA1T9gNhGRhJt7n1ANp2qjca8vvXm2Cm6KWKieEk40KFkMTu3YweVLnLfBvjeyOqSJF/j4Acy2JiUqa3QhqO/5QHZnFxFXnAr5JA3vn4IOxutat7dc1A0hiSxMfm7wY6kuB7c3QF5w8QzV6XGjuOrIxYkJoDgnNMGIyx9oTV06l7SXGVk2KRQm1P+89B0z34zvHELSHjhjWG7+i7rvs4FkNu87PifbjnBxxhBHYnW3AXiSXiMQH+8vAW6QosViuI/LbzrlTq9DKnFJ8TMDCMN9JR4jKn6hou6OICrErI9AyzKdqFNAFFSwf4AuPzvuzh6a7SMnfslog8JCb9IjAiBpR0VUBBoP+IiWKZ00EEizP13k7Pwzq+ixT2lVlXxSmCDCcAQ6nkPclHiAe/Nu1JJ0IJvih6bOK9ZuDRpFSBiNUUiAV/Ju8tdb6x2nO6lJGcHp2QGn3uRh8bhBWLvECOSgBscdV2wP5Tp2/Z1n1MDUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM8PR01MB6824.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39850400004)(396003)(136003)(346002)(376002)(451199018)(6486002)(107886003)(83380400001)(7416002)(2906002)(5660300002)(1076003)(41300700001)(6506007)(478600001)(86362001)(52116002)(6666004)(4326008)(8936002)(38350700002)(38100700002)(66476007)(6512007)(66556008)(316002)(26005)(2616005)(186003)(8676002)(66946007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NHYHiG2t5zoe8fCtwZF/4PY3Rrng3s0OrEd3xiW1vT7Vv35wLR+wDlwDfG4I5LlENUDjxSDYq/m1MXmkYSy2xr9uq5Ie5hMD/3VGoCMe98XbxS1SH+qao1WrbB5j8V7emUdenCE8HeSVk/KjyCE0o0Ojb+1I57w8gKkeOkNSft5lf1IAO4KY8ifK+M65M1hmB8DH8NC8l0qPzvACWQaG2N0V/+BUNkke3BLWL5C5OVRySAVYAHYdy087y77ADoLaoC8fnv7VH3KX+r0da8UF/8bhldl6m/2X4Cjvp1L6Z9P1z2jZunDxjdZQ+ibbq3cH94i5BoQH5SNYR/7t1XOfkINbvnu2ijVEGcR1ppe97xjcUR43zOFwuqKMuloht9TZZmxu2so+Zp7Vv4RBzL2g9D/o6JA3aw5lBPDidZ3anEBMUto5kCVkD1/PGzSyaQO9YgEaZYx9necFuoxicoFPt264ZhuOV0rkUbufR5CBCrbNE1YEDmxrw1vz57JkL8bGCUdKy+7GHBQz13om7Mc5tKSo43PzCsnLS2L7EeLUUBVUEulBubnQo6QSMP253V9t36BIbkx3TyUYTGlvQyPCrOziCvK+ODd1QHtNHcO455NK82RaW33gl4HWG+maGfBzm2c9j0YGwHWqPUKAXsuQ3MJNhYGGdYV7iHR1mvszYX0PBkd/srb3kab/JN7d1CNQS2a5jYerNX4rXELSbtoDC3RiP7G8PQhLMeV1GNCPvyyer8Du63hlrx0FJEqtxIz8p3C1tsF2HImvqSsz9K9aft8TwtxoDH3WQpAfrdmcDYnW4o/cxUUgNIxkFaIMtEsHK12gx0zUc9qmQqTaNWdInFhDErV9ExXLLbQoOUvIZxBH3fa7EdUdiejwO/clfiIbl/0RVJ0YaEBuP9KmOFFme39dQN4D5XAiPGU6CaHqgXftvcM1ScurS6RkLScJtDyu+3KGLfB7IKBPZ1Kpv8pbk3vNcnqvTZzjFzfKwKFMGQoNHV7uXzMunyjnLAHe1Iv6zW96fYG4Rhc+JlgG36Z2VJ2uC5K7EFJ2q5i7louOXbXMAX2/6UKYGPizFgGO5ylHsJ8LJFUdoiaB2A7ozGjb+J/G4A+jQPYMwiXtwP2d8VmfenjICG13zsUlFlBpQAtthLxJq9c1Xu33b6bEuwOo5DTpnDl0HoPV1hPKZC2jEMvlsVZPRqqqnyLLDqxUYFHxQypcrtz5fN0S4jvHZroLIFDLJmF8/CCybiFPLaTf0JKOT+deYoXM1Hj2PXc62eobSmxDIx46dHERTFd1RM0zZT8tduMILRiQDf0ITSW5Koc3IGZiw+NzJzLcYuTd3DizNZWa5f1RHi/L9GjGHeJT0Zi5zQBbozqaJNUKKI9rL3k9eSPEl4+KyCVzI2ZWjjcg+AsPr1+oehlPpWWLsunDPKGVrvv/AYE8vTmuz6KELHl4tb0auQjvB8H9hb0Z6Ckuu2FsJKsoAjIubg9jjT/4Vuxz2Sv9pNLmz40fIhHJzVd6/kFfVpJ+hZYAPpu3EnavlJj445fcntSqgK8RbgehOyyjAp1Dej6lwm0LITjALbOabmESHa9ZS/2kgAfN/OGcyDFa5ctviosjmtzPLN7paMm4JHhd+O/DcMQgPbQRALA= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 44c0fe17-16a2-4c15-5fc2-08db194352ec X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2023 04:21:54.8889 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bVkl9GROhYaqbxgKC0MP6/sh8896988AxNGtsZLLTTs7HbitWG9rgDxKa3qk/8AlHpkP/rzNxPVGKq9Z+rqSxSTsc3dU9Li/SIPTd1qRpkKaYg2BdnGmg95XV9kIV5Uo X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR01MB5422 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759048630963670776?= X-GMAIL-MSGID: =?utf-8?q?1759048630963670776?= When the host kernel is booted with iommu passthrough mode, PF and VFs are enumerated with iommu/smmu domain set to bypass mode. In bypass mode, ATS is not enabled on all VFs and associated PF. When VFs are attached to a VM, the corresponding iommu domain is set to SMMU translation mode and smmu-v3 driver try to enable the ATS and fails due to invalid STU of a PF. Adding a fix to configure STU of a PF while enumerating in passthrough mode. Signed-off-by: Ganapatrao Kulkarni --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 25 ++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f2425b0f0cd6..b218ef0bf001 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2292,6 +2292,23 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master) return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)); } +static void arm_smmu_ats_stu_init(struct arm_smmu_master *master) +{ + size_t stu; + struct pci_dev *pdev; + struct arm_smmu_device *smmu = master->smmu; + + if (master->ats_enabled) + return; + + /* Smallest Translation Unit: log2 of the smallest supported granule */ + stu = __ffs(smmu->pgsize_bitmap); + pdev = to_pci_dev(master->dev); + + if (pci_ats_stu_configure(pdev, stu)) + dev_err(master->dev, "Failed to configure ATS STU (%zu)\n", stu); +} + static void arm_smmu_enable_ats(struct arm_smmu_master *master) { size_t stu; @@ -2404,6 +2421,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master; + bool ats_supported; if (!fwspec) return -ENOENT; @@ -2446,9 +2464,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } master->domain = smmu_domain; + ats_supported = arm_smmu_ats_supported(master); if (smmu_domain->stage != ARM_SMMU_DOMAIN_BYPASS) - master->ats_enabled = arm_smmu_ats_supported(master); + master->ats_enabled = ats_supported; arm_smmu_install_ste_for_dev(master); @@ -2458,6 +2477,10 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_enable_ats(master); + /* Configure ATS STU of a PF in passthrough */ + if (!master->ats_enabled && ats_supported) + arm_smmu_ats_stu_init(master); + out_unlock: mutex_unlock(&smmu_domain->init_mutex); return ret;