From patchwork Thu Jul 20 08:39:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 123163 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3018552vqt; Thu, 20 Jul 2023 03:27:51 -0700 (PDT) X-Google-Smtp-Source: APBJJlGzI0V4XA/2yMYBhy6KV+vsuV6YF0qswV+YkpD3dPqvu2Hio7bzKQjebEWODmV0xRgt6SOT X-Received: by 2002:a05:620a:4541:b0:765:22d4:b267 with SMTP id u1-20020a05620a454100b0076522d4b267mr32362683qkp.52.1689848871589; Thu, 20 Jul 2023 03:27:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1689848871; cv=pass; d=google.com; s=arc-20160816; b=c7grNGKYvUMu4i/y1LUVRZERNHvFKUo1UPBKSKTT72kgA89NcVzZ0/c1C6T7JppsnU 7sCL9m9g3Nk9K7SqxKp1fWKlt88hKZUMgfL5tocOqcQlArRfXlbDg84WxnB58JpCN+L+ rUm++9u09jh/bQIxFw/bi/kA+eW4409YArW8kstLQ0scH8Qxz0o9CsAjnBX+DZtSCjnb gjXFNRmejjnoYPPOh+rKONIoLkKU8sG3b+Jke1ZihNzW7JDvj1we5yr3eCKxFlJuD9/M iokGvFurTWQpeiW43REcTfjgRiuR8QayoKJVy90BbHqbX6k+hNXlZCvTbE8qDjndGgTk PWXA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ppehoSrCgyLFyaJH/ZGUibIGa4K8lHOftxZoQu5p944=; fh=1zIfrf4XUV5VVq2TK3KbcYs/ElPK9ezytjx3LhRAR54=; b=lYRYCtwymu/1qLy5vu/1yb4zUoo0p1mc4jTR+SeW+yomrt3Z7UXtVUckAtYP2e3RKQ Qr4TNY6O7yRd34RQV4kKn3QH3rZLv9AkQK2hV+5/IZ9GuZlfYAibHjWYbZXmgZuK+F8x DkVHvngINK414Qw1Ab6b/iPKTCqjQKTFa2RgXw4lRxrQq5bS0Mgt6YRTdRb3dt2a6JKX BmdsLei5NQjfJk3jFUPXpigd+g3T1xgPZuMdOPFpjtwCIspXLVnjxLu3VFpUZjnwsYbR kAdDsM8RHNB9EVxe/1Is+SC2LRhEYoGDNjmds2aKV7M37IMhSnY89of3vfzFsXczLXUQ 7dOw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=bTkPNhib; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lk4-20020a17090308c400b001b9d03d0bdesi713295plb.79.2023.07.20.03.27.38; Thu, 20 Jul 2023 03:27:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=bTkPNhib; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231627AbjGTJAK (ORCPT + 99 others); Thu, 20 Jul 2023 05:00:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229601AbjGTIjs (ORCPT ); Thu, 20 Jul 2023 04:39:48 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2081.outbound.protection.outlook.com [40.107.244.81]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E647D26B3; Thu, 20 Jul 2023 01:39:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nvSBmcOJMZUws4Z6ukb23Fh6o92px4nFt2UYG9ZhBJM2NKuwqXLKaHmGeNk0BxbnmZ8l4bQjX+Ms39xtuNGgPsELtlfNbjkCHpMdbKQkuXiL8/5UWqhPsbcCrnkeFupIW9YV7BNfuaJdeCJSVaBuUzfYhyrJJHl9SWQ6whLkZg2nJ7AMohtn443pd4tFnI6HPxK1hqLs8T1EIEC8OGmTyysFN15yu+mbZ3K9nmyMWyzJfla7SxPuuqexffB80x1v6B8NZfEA+bHpAtN8xTfNSeBmeiwCh80i+6j1iTAH9afX15FMM4i+iOwvdA+jZJ4IIajSsdUcQufHpxOvxtwBAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ppehoSrCgyLFyaJH/ZGUibIGa4K8lHOftxZoQu5p944=; b=hlpV1y2aptActkbla/Wo20ejvQYkw5zPb/y4EWi6rsD4HCJeWxbbSxgxoaTDbviyLbzQbjpaUf/z9EQffGIhufGNQprnbsUbbVGH80ZoEDQxKDBctWjFeTVYDKr97DEoha5gG1TIWce686y8GYUoCjIbWSXw5RPDV+g3lXO3fHxlhqL+5Y43lGvRlJBBo/FbsNcyBIIzIE5LXrZRwY9Px7xUz8lQJ2pvJ/o3d9Iz5invOinrMtmeEW9uTb8ykpG1cdi8fUH4HeqCpd6KVLBVSNUuBnbf9+0l+FxqheqGd4zTRhVOFaeRY3is12AyNjt8hQpyQSLspj1EKMRk4Cpucw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ppehoSrCgyLFyaJH/ZGUibIGa4K8lHOftxZoQu5p944=; b=bTkPNhibWjcJI2W+dadrFK6opdVBohDuMRqf/o/60Y2E6Np1/xohYJNtpj9ePv2X0XsnbCY6CnrL7wUN5/EJ20MIpYvbMn9q2QKuj3IQS/I66D0MmpAQLQV7L6ocNexTkieQ81NeVd97LBQOfpvVu/4c7BMTorotvfg1eknwkPB3fKDL/naZKzC3JTxjSvgNo/Xd1OE6wevDju1jRnKDIu2k8GMXuUDfZkNq/vGj6V/Uck3twj8RL4Ks7b+FhZo/gpI/IOAgfrE+WJqfjknaVOlh5E+9tYx3gOJxATOyWYTWF+oQQ40hfwr0PZypexlH3bb2dEvuSzvVqmd5BRpk5Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH8PR12MB7025.namprd12.prod.outlook.com (2603:10b6:510:1bc::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Thu, 20 Jul 2023 08:39:45 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74%7]) with mapi id 15.20.6609.025; Thu, 20 Jul 2023 08:39:45 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: ajd@linux.ibm.com, catalin.marinas@arm.com, fbarrat@linux.ibm.com, iommu@lists.linux.dev, jgg@ziepe.ca, jhubbard@nvidia.com, kevin.tian@intel.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au, nicolinc@nvidia.com, npiggin@gmail.com, robin.murphy@arm.com, seanjc@google.com, will@kernel.org, x86@kernel.org, zhi.wang.linux@gmail.com, sj@kernel.org, Alistair Popple Subject: [PATCH v3 1/5] arm64/smmu: Use TLBI ASID when invalidating entire range Date: Thu, 20 Jul 2023 18:39:23 +1000 Message-Id: <082390057ec33969c81d49d35aa3024d7082b0bd.1689842332.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: X-ClientProxiedBy: SY2PR01CA0037.ausprd01.prod.outlook.com (2603:10c6:1:15::25) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH8PR12MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 71a9a0f8-e298-4619-4932-08db88fcde88 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sVneCM/Dg9K30ylAW9TjIHNkuAsJ203YUZ7CqFkNHefMCTPdPuDCbA4YbzfTu0TtSxzvGupAkt5UD1ipoKHA8r/Zzngusdu2FqhYyGMNwfNvtXOS4rw052nGXRa6UNQev/m0IGXkhAgoRHYzcl+l9c/YqLIYogfz5qHJ43WsxrEdJqwh62qFAnDPxdeXhNOSQ5iuSATUXXliBo/wp6UqonTzxNu63hX6bR16PsTatvU2wXdT21WehheEX93VUqwNfDVbgYu7vGg+VqSZUrHtWbfAbJ28LhR/ZBTvsN/SVz2laNVVHuBmxWWQXi+AOysfan1pIhUzQtAvTZWzvL6ieY7/vapsBn2R+Twi7ixmz31m0UXfxKvBq0SIg/Ld3sX5tN97UpX6YbkA/8KS7z16RTH+hMhXLbavOJAc1srVjD9saeBeY8lv42RLiChF5NQBqzwRdu/oRECQEUE3egSZqlLb5D6bZ+sTmNBZpk4p6V+Q9aoXvCjt4LFpzIejfHkj+W12UZikkXLQef+48L1uqxIraG59Xh3c0dgPxy4vcfLGpqLrRqXi4ZMRYXcBjCpm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(136003)(396003)(376002)(346002)(39860400002)(451199021)(36756003)(86362001)(66899021)(38100700002)(186003)(7416002)(478600001)(6506007)(6666004)(26005)(41300700001)(66476007)(66946007)(4326008)(66556008)(6916009)(5660300002)(316002)(2616005)(8936002)(8676002)(83380400001)(6512007)(107886003)(6486002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wHqiF3NjiI1Qkb+tt4ASongrf64ybb7hwVxTMrmfxWbMKjtOa2gzUg/zOxyCE9yWQdGIM2GzB1EeaZjb7QFk2GbrHJ1ZZi2My7vxoeIAFbrTWSnKMPvdTPmMQxptRc+6JvAjyhjZsUNXT2/XxawZr3XBRBds9oWQT7D7XMRuBE3vhzWscKw/zMN//uxEhFDa7ea47crbdM4L8W6x9w6bj+JZqxnbvFaeEq+40Vvw83trLfg36tfXip4X8dVfYeil40bpW1J3d0R+ql9Xwf3YFv4oiIVmkVqNl9i6xXWOu8DeYSnmuG3+CP5adYj9RShjZPwM+F3sGmUX4YHvpVpnNz3yr9tcWAYojt+DlhiBkYblPjju2Mae9zbHarWvCpeQzKlM4MCYh3rlNJkQSA44eD4hnPcWNXc949BBZO5BKRKCqeV2S9GyLLTI0ghXvTT8vm2pydaXHBmg3EcBHAVnniuVDMTyXLVRepDwLOk6S2IbrgPbaig1LacV+aNqqB5Vm/Y/d0LlM/BGWg3KZlj0lzzHaIGQS8lYxH8RnuT+U3SUX90WjfLl6vDCXbzsoZEc1P9Ujy3g/Aeo1msDc8p75ziFSlfrJHUCP8yZlP8gnhBIuwMVNyqYfytuiJKTRRXu7urCgPnwTvHewCE+QfYtmY4xASg5G1vfx6cOlwjtAowOyIBBEZLABnUSm5XYz+rrTCrMfuAW4x0+KYhrgiPK4F0JcWOjlvIP4AzFOzzv7/4jolQEjpXXLD4vFYK6jKROZ2qA4BFri1LTPc+AXG1fCQbnWG8QeijB02PNrG4shPdOmiUjHGw/rpJ59cn5d9uSHj+b9Hvyem05ckGhBWw4Te56m7ppztOGayCQBVSuKcCYdMvjvXzzsugpaP8gsfxsUUDxbckNee55QwmnAdlCtza7/yVFACGRXHHhHymAl56/qMtOa4LL4MuqQQDnXdCCOM8Rn7QOLlEj3+w4XMq3rFChlkCORVnijr4ZBOvFZF1BJxJGmZtraPANgMwfpEX6PzpIzN5xhvhCP+sduDXub48z4hV1m4E2xYGWv39HtmZOjpNHLMDtOyxAIgmSXX9HfzUSFRZrzyKxdAvhYD8N12VRhKX6GBYEmUEuMVzrkSTWR2oC7gC2obSInYRS/2hcaDCP/U2TgfPnwEz0xIybeTrveoALFkhYsBzxBrYsUASTsOvRZuzfI4vVJIU4iaSFdmbiCq2BeTYrr/VtQ+5KCiGWIO2aQzzhNritYO+20Hkd1ACnCsHCJZsuoT+WKkK/ad2jez2RCA0iCeOl0se1qu27vHqoaOqQo6zcr88Yu7RrYxKA48lARIY2atKMGpmEX6whW67Ofr0sGdR2gPnK9boiIrBu93yYoalnrYfdDTMMZfidYH9wfZwqjV1g8dOUofZ0eHOjmpLwit0HMmLgAJBumOovkESuPu2dyiEOq9AztKM3ym2tZqewmXXQDl8HBC6Cxe5Ni6baHRXYcGxS59r9v7fhPTVK2IzjwZpTmen0fubLGbrZkGx/vHVOieO0d0URNim14XkWf6QEsZGCIu0R40kKsrPjbG4I1BBHI0YNIUduI2oY98exf6aQhAV4 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 71a9a0f8-e298-4619-4932-08db88fcde88 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 08:39:45.0950 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: y2+mhUVdw3pCz9hk3flwamV5KGK7+RznL33Ef24k9BsQxxw/OyhIqEiwNWShtqX//oMjIhuPWSa1MgCQx7kicg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7025 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771934970432516753 X-GMAIL-MSGID: 1771934970432516753 The ARM SMMU has a specific command for invalidating the TLB for an entire ASID. Currently this is used for the IO_PGTABLE API but not for ATS when called from the MMU notifier. The current implementation of notifiers does not attempt to invalidate such a large address range, instead walking each VMA and invalidating each range individually during mmap removal. However in future SMMU TLB invalidations are going to be sent as part of the normal flush_tlb_*() kernel calls. To better deal with that add handling to use TLBI ASID when invalidating the entire address space. Signed-off-by: Alistair Popple Reviewed-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index a5a63b1..2a19784 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -200,10 +200,20 @@ static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, * range. So do a simple translation here by calculating size correctly. */ size = end - start; + if (size == ULONG_MAX) + size = 0; + + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) { + if (!size) + arm_smmu_tlb_inv_asid(smmu_domain->smmu, + smmu_mn->cd->asid); + else + arm_smmu_tlb_inv_range_asid(start, size, + smmu_mn->cd->asid, + PAGE_SIZE, false, + smmu_domain); + } - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) - arm_smmu_tlb_inv_range_asid(start, size, smmu_mn->cd->asid, - PAGE_SIZE, false, smmu_domain); arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, start, size); } From patchwork Thu Jul 20 08:39:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 123123 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp2993350vqt; Thu, 20 Jul 2023 02:28:58 -0700 (PDT) X-Google-Smtp-Source: APBJJlGJSxKCTAy1MXJCH6z/0PyJkNxFEllS1OmCukXeq19NjuZIpM4yKRdsvrGICvlXQi4+2hfk X-Received: by 2002:aa7:cd63:0:b0:521:775b:ec8b with SMTP id ca3-20020aa7cd63000000b00521775bec8bmr1660139edb.25.1689845338141; Thu, 20 Jul 2023 02:28:58 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1689845338; cv=pass; d=google.com; s=arc-20160816; b=XqsCMWsnsRd6VH4fSIavpjZBBymKmyli5z+MvIJ3Kyr7tR8HP3bhwzvmwEvXi34avW /XTAw2619d8RLAup5k79cM1Hg15YFtEgPWqZViIMGO+/FbUd23/+1zTRqCB2mKVfMSqK 2XKE7Khk23V6+EpymYjSpNvChkjiYEm1fVjYf8HV7n/gjuTaI76W/3JJ5rM2zA/Us+3k Yp758LlhbGTQnLmvHAIFp2MTFXVFlziUvls4CqIz6QsLTGzIzZUx0YjOKHJhgzFIRyqm KFPsjrCMlaTTV2Tdw8lWH6fguSFNcj2Ggx4u144N70JC1hqF7FaxAbWWKMLPeolVFYyO OAOw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=D/cCYj8Xd7PDxmlOmxc7ZGo63wZ++XgwVNU3j4KQgOc=; fh=1zIfrf4XUV5VVq2TK3KbcYs/ElPK9ezytjx3LhRAR54=; b=MFGPOqbY41ecD/75NM8l+oDScEkOYZgEYIFX6ify9xv3bwgn/QTJgEo2YKUEkBetVa AyUdP3mrFKuLZCDe5LrFMIG2+oCqxSREBeVjZ6PPoEBXgm51YKdwcGshbYcD+4sD2eXA EWua7HI5VbxpnCyUyxMQflSxUFjhORfbLMAyrJgsMqK9IGff2s9gByrfCXu+KzYZVVYH JXZKyvYZFkqO2aiaIuoYtkdTeJtQXlK4NndaMUUU2jq/CWHAorgXynER17dvjDLFg8nq bDDBUtaBRpKqP55wu9p0h6kFG17lvkG/1ds6wGkGh2l/VhKvmL1xFiY0bZqm09lEzDmf RXUg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=PlVmjHQs; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o20-20020aa7c514000000b0051df73ddeffsi553390edq.358.2023.07.20.02.28.34; Thu, 20 Jul 2023 02:28:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=PlVmjHQs; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229456AbjGTI5M (ORCPT + 99 others); Thu, 20 Jul 2023 04:57:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229536AbjGTIj4 (ORCPT ); Thu, 20 Jul 2023 04:39:56 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2081.outbound.protection.outlook.com [40.107.244.81]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D66AF26B6; Thu, 20 Jul 2023 01:39:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ECvaaA3ySZDw5vHoY+JC9P8vXLYkaBk8MF4sYWcdmjMjl866XZlPdPmXXspCr2Fs64JXZuKEw/QfmJd4IZTj/gF9LCrNhCH3KTXtH/iRpa/Cr4x5K61RLU6Ql3bopGPA5J6gc1btkf60ZZQc2FJ0EZvrHQVLP+qpnX5gTQ+bkcoQSHkXxc8ppnhgo7MDtVvDDed6hlZ3CI+3bEnARdUniHs6L2arz98HuPg1V9eTm7/WjqeT13e+evf0C1PiS3UkFJYkVya0GyeIW/ln1eSszVyo15PUaQpqFPmdcUXOt1uR+6HT5HRzZLyspsXvCANqUTj5cdRE5DqEae75NhUQXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D/cCYj8Xd7PDxmlOmxc7ZGo63wZ++XgwVNU3j4KQgOc=; b=HOwjuVjj3bdkRRRf0+MyuOjBoIv6ui061C9eMFXHos//+uKfc4kQTUswHKR/xaKiZgGKv91Dqr7cToZCVhUKnaCuGGOfHqGHyp6zOgGX2G+OO2BVBDUCV0aCiKDv8fSGrPPJKJWJX+UKdWFZB7SANkAKfdp0I9brzdvN6/oe8Wl+oKlcH2j7DlwEpOpamGQhovIOBBYkHX9JHeSuO7Fb87EBZeU1kL6zJH8ONsIEKxK8gtta3vpJsWYHAa3MEmOLGAD1Ta2LlKYnWSvqqNcdADr7SxUwdUOL+EDT0sZB44ZaWvutveWbf05h3qLFBOfpEFmjnE+nE/coKbrTTwMp/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/cCYj8Xd7PDxmlOmxc7ZGo63wZ++XgwVNU3j4KQgOc=; b=PlVmjHQsWvDa8c0m9WSwFzKwG7DCZ4Z290hPqi2PT5toTVVL+SGoB8m8A+DdY+UGmG191MmxMkTc9dVPDcqSNsuve3OZWnF62qlYJL8WL7ZlD9ohDSaPtdmiPMcv9IG2+XQEV3R/yrB38DUqsWwag8DPey4DBspxKGleN92Zs48XrAnAM2Pu0srKAfLZHirUsbNiBr+6p24bvAVFJILEeH0X8LYoQOO/C41y4gL11bvtuTMA2RoOFwcb9v7WinQmdIg80Lm5X7xqI+0hXwyIdjvZCsJl2qczjiW1i0n9Jv7Tyc8EzGFamsZZHkN1nd6rxfizEvOGlgaeS4zKxKgqVw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH8PR12MB7025.namprd12.prod.outlook.com (2603:10b6:510:1bc::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Thu, 20 Jul 2023 08:39:52 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74%7]) with mapi id 15.20.6609.025; Thu, 20 Jul 2023 08:39:52 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: ajd@linux.ibm.com, catalin.marinas@arm.com, fbarrat@linux.ibm.com, iommu@lists.linux.dev, jgg@ziepe.ca, jhubbard@nvidia.com, kevin.tian@intel.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au, nicolinc@nvidia.com, npiggin@gmail.com, robin.murphy@arm.com, seanjc@google.com, will@kernel.org, x86@kernel.org, zhi.wang.linux@gmail.com, sj@kernel.org, Alistair Popple Subject: [PATCH v3 2/5] mmu_notifiers: Fixup comment in mmu_interval_read_begin() Date: Thu, 20 Jul 2023 18:39:24 +1000 Message-Id: <06fa82756e4d6458895962a7743cc7f162658a54.1689842332.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: X-ClientProxiedBy: SY5P282CA0076.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:201::6) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH8PR12MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 1b834c26-30b5-4686-5146-08db88fce2b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NKFgrMSkEbwVZAmEFXKbnVft15qBKTzeInwePEcG1hrohOUbz6IiSkRTPGx9PLj1Y8p2la6CZoX+RYqMweJwM8eEWc5lPu31BfApk1Rqov10/F0xgEia8id4Mv9WkWcKlAr4cUGw9xyZxn8UA5LIJWtxxIDMyXgmBJmR7ugeUGIcA9fvda6z13SHnJE29cOEZePN2hAzVqkZhAz9Hu154e3T5IYyxznDhB4lP8GPbde58T2NCOBIzo6c00DEvAkzOmhB+iOVavCxksdbmLPjmZC1qhywVgVfceFP6K4Arur0Bv1OK6g2udVgvo7pBAZ+1QMsbJv0Q0fGk8iS+5xYkI5NbmF/+RN8eiQ+7WitilRrpyzJND75e+1b1+yg4aAOK7GfpdgRUTmSquC5HeCWAsZaLpEbkoKV87Po/V2TzBXzqnOpXJjA7+ZF9Pa6EQodp1YDMG/vA/UGhXDgMJMcRjAqUpErNHiCQS47Ajl8KqAhqDpVU12isoLV2xjdAoqtYUtyhOEOY8H4/y4ONMTFfo6UZUvrJF+Yn4dfEP6hfS8Cwkpe9NHXBShpqNSgtaOt X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(136003)(396003)(376002)(346002)(39860400002)(451199021)(36756003)(86362001)(38100700002)(186003)(7416002)(478600001)(6506007)(6666004)(26005)(41300700001)(66476007)(66946007)(4326008)(66556008)(6916009)(5660300002)(316002)(2616005)(8936002)(8676002)(83380400001)(6512007)(107886003)(6486002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qCUN7qdsU7Ongh1mMS76qCO72xN+/RqD5YHQAwUkrveGbNzXZ7WuMZEi/RF8xZwgBd3ng2KbxVo+SWsqgRhv2fze2oIr6mO639CtOmCxRLgFo01AJ6DknYsC6IwswUZWHPAXkWT5XFQvL8DvnBFkDQb4iSnk+/ZgOh+YfpF6+5VPoPnKA5K6gtCWmBjj65s9zwHTbFzk06WS6P0Ec1VpWpUTpDBDlOONZv+OK2OsWhH2x8cmtF1gyJsuQGIHgV++Rp7Svia15rjEvT/F1GbW8dQUWk9oLixC+5R2mINHLfG0EPywjbjqu6x1OUiO9I+Cz3DH0o1goI+ZlOed71HHr40+RvA6Y0A7TAbGw6k73m3zrbDGc5rG/1hRkubpDuysJfnYby4c7Y9ooeTatqBJJOQTmRvecy2RY7qEQB3DJitBojNpvAHmv0Naj5gpWGxWQdI3ioOJcCgrELdPdDg7Mk8rUra/MyrUMqf3qldeLoe+J/H6EqI/Yye7H/cNy1lNDOS0vkpjceaUjeG1SHTjNbgW1xoU7b3iJceZ8wbGd4VDUZD+TW6qjTha88Wd7gVCewwQ2uecV8ysQQHbO9NeIIVanr2kFtvoBw3rJCm7w+qYvG6MKDPI+FtuN6x756A/7x0ntL6xLOVSVjhzEG5MfES+YZW2MxQ+kCI3CUARcCvS+igFf5kbT4AV+Cnnaf8EnW5KMHgS/5O3NVwTOC5RC/6CPtJKjNJQBfbjSo4kK8dGcgeEWwokQ2lFAbWfRtc8k2yQCcBhkp5BoGmTAv9Hx4Ljaysbp89Hy0g9LA+L8XqA7yBTdCfD5Pmf7YG81fo34B4iVmMbTy+dkPGYcbnYprEkXHHpsyNgvd7mNpTE15274Xfosb8+oqszrN4OxufdiReB2d7vkaGMs1mzLofRsPWVpH5AiZUL5uCR8HgUrH3s4bklc86ur2VDBzEVnexOwMvbVAq/oNofAD1d/FppOpMK3mYhy03+jNwcalo7q/EoGVwrN2dF5nvmPN67Qxq/OyxHOjuiEh5whqGkWDnvvAajdmMXPSC5lC43owilhQhGLA5HVlITKSKgGGeMT0rvm0iuXLMqEOQcUxe3iig6WqLJrfAyD42tpKYkYcdGwufxSWYTVgSHmx0C72HaBS5LdQnbhi6n2zlzZwntmYk5LKYSI9rfUsM7nI7L+itvBa6HRwYYctzLj/Qyo8aicfCyiDj+zMT4TpXKgeRB5sWdH062c1/wccQHaYBI8HJ52CUc6nUfoiyumZhg75RMwVi7iUhdXUVEAsBwhMCpt5qkf4wLweCPALlcp+UIwOJxiY24VlmuUHmhvxPn637597fJmCvumnCfz2BRD873WMieAiw0hnOy9aC2J3BIrCNZbkpyfA7nANbDLTNUJPob55a+f7uS2ekOBb3DcCF2AjbVdtHQiwUw9xsevW0yxr48z0MZ3MEsXhMoBF+fkcerSRv8v4ViY6qbeVJChUjMiAzD8A5vxkY2W1YfdJBoqN3zvpW4X68fFb8rydn8BPV3Lr5sOLP4Ienwsjap6/ydsWRy5XnIVoa4FOTLCsPixPjk5Qj3J8NhgQNTZK0t/NVbGX8r X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1b834c26-30b5-4686-5146-08db88fce2b4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 08:39:52.0953 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tInQ6FJ2kzkNrjSLGbp/KoHQPBAJiteOD2LGi/O4vC0tNaD3WT7sJxlDmWe5wvnMCUeuRbjF3GBNIzxUnNJUaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7025 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771931265759694843 X-GMAIL-MSGID: 1771931265759694843 The comment in mmu_interval_read_begin() refers to a function that doesn't exist and uses the wrong call-back name. The op for mmu interval notifiers is mmu_interval_notifier_ops->invalidate() so fix the comment up to reflect that. Signed-off-by: Alistair Popple Reviewed-by: Jason Gunthorpe --- mm/mmu_notifier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 50c0dde..b7ad155 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -199,7 +199,7 @@ mmu_interval_read_begin(struct mmu_interval_notifier *interval_sub) * invalidate_start/end and is colliding. * * The locking looks broadly like this: - * mn_tree_invalidate_start(): mmu_interval_read_begin(): + * mn_itree_inv_start(): mmu_interval_read_begin(): * spin_lock * seq = READ_ONCE(interval_sub->invalidate_seq); * seq == subs->invalidate_seq @@ -207,7 +207,7 @@ mmu_interval_read_begin(struct mmu_interval_notifier *interval_sub) * spin_lock * seq = ++subscriptions->invalidate_seq * spin_unlock - * op->invalidate_range(): + * op->invalidate(): * user_lock * mmu_interval_set_seq() * interval_sub->invalidate_seq = seq From patchwork Thu Jul 20 08:39:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 123107 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp2980988vqt; Thu, 20 Jul 2023 02:01:30 -0700 (PDT) X-Google-Smtp-Source: APBJJlFT+wCDAIIeWAuFmR+O1XpZwaGQ78AnR4KIXUfgV/gy6socazFT/o8tLLeXJAd7h46mqQic X-Received: by 2002:a05:6512:e8c:b0:4fb:c740:326 with SMTP id bi12-20020a0565120e8c00b004fbc7400326mr1882582lfb.55.1689843690117; Thu, 20 Jul 2023 02:01:30 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1689843690; cv=pass; d=google.com; s=arc-20160816; b=xX8CcFcuNLNQQ0sK5npDgn7/gaFDwdschlMW4pNZGYklht0F48lN5k8fSCNYxK7Zbc Yn/AsYLov/L1MEtpccpp5VNAsLCNAsnxH4gRgyk6tfLTWooljZMI18gIU+IO6/rK56o+ OrStIJFIk3TuQxniMsbelVz0WNGHEYNeaJrVHjGKYkUmSFSIyEoCKqIOdw+LiYBYK+7D iPw1VHuW8gv5IQkxYG+0KbYYTitF3cRB5rV7o0qbFB04KaUt9xp6dhpd/wjP/zYZh9Bg oZUnQwbVCchtbO/nD0z8NoaPX9HrxXNBr4CwCnrnfdamjvxMiGzR8DvXwXk4hr+gDnyG bWlg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kyTjkl32gx9e58egMAR68Ym72eF/grtnHpGANsDW46c=; fh=1zIfrf4XUV5VVq2TK3KbcYs/ElPK9ezytjx3LhRAR54=; b=0A1iz0g5/yrPfcmcwtRFy+hQc734aZP2yRsN1US8kwplFyPMoOYzsx4bhPZJ2IHDCo WJ4vTSuiChSVYbMBCy/idi5g/EeXpGqsB0YZRb6tvp/a4M5v5En2MTX/PHsNQSWQ+4ap AOmUadSpLxTYDUYw7eWq/H54ggn223OLAsE2mnGXapqF5PyDQwFX20rpx7cQIB+MSV+/ dtFzfPAP0XGDPrpoz3WuCUUzm0SMaDTdakH7tqWKtQIGd2kOlnnvj++pXvNxCgjTRl6e vE1Q3i/mLWrE3aZTGYh9DivMCUglUpE1pNCn3jrtDyzFs2WpMgw+ncGEV+vfDBjWxOVq xfSg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=pzkVvYJz; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f19-20020a056402069300b0051e16fd4d9asi501630edy.223.2023.07.20.02.01.02; Thu, 20 Jul 2023 02:01:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=pzkVvYJz; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231434AbjGTI7M (ORCPT + 99 others); Thu, 20 Jul 2023 04:59:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229450AbjGTIkD (ORCPT ); Thu, 20 Jul 2023 04:40:03 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2055.outbound.protection.outlook.com [40.107.244.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 057C526B7; Thu, 20 Jul 2023 01:40:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=N0Urp2JJb8Bsyj5XPSrYP1ZnUykYXDayVn9hFknzDkJs1mRr1Cv1LYNrGqmo2iqx/Gl5lcr1I9EIbb77tutGMNi3DyVwT+UDYj0XGcCnmXCjc5E5x4cm/kbkOPzMzvseV3+nrePAvTI3Ib0KrSyZaFN9enQTd60bxzigRumWtBQMYc3McdO0k+sC7Ac+8NDfITwNPgC/ebngsuwEhiZ3q9zBj7SV/zizHr6mFdsIQnvkDaitKXzPTgq7qPbMFYY+SDMD65tVt6fRCqiLUvraeyx+dyJQmlITpuGi7oabVtM1N7VlSscqgEjWSgF+4FWb1hnwTWxckFx3Cjt6Ysq/kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kyTjkl32gx9e58egMAR68Ym72eF/grtnHpGANsDW46c=; b=MIEHN7NSHcczrXuQ7uBNCBg/rwRO8xNzNvqQXMz3OTIof2EdelROairMuKzjEou7VIpkBGyt2HBqpQPB2jSXaJkabdUWwcsDhmGjX4zLkwOoUhtjfIiqkCZo7WS2aIu8e47U9EKmT1vzKn4GiiwzPg8ggJvIAbk5mndYbiFAk3ZQrNcFnhZygmduNZRLJt9pg4cekXbPry54Nswiucg9cEKDcHi2Gk79ZG7Z3gyKedtYBJpR2lDFndk8/101EbVRRDLG6Z1yctT3sejJGqGMln/bK58M5x5+Y5K4F41lXmcMyllyyadgtw5eMm0sFGAV/Ec38Cm8slJLC2jL0msrYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kyTjkl32gx9e58egMAR68Ym72eF/grtnHpGANsDW46c=; b=pzkVvYJzcqb+S4IieNcTuiq3SqCpx4UQZyGpRX8v1G+xz4RmfCmwH7iDq3+w9j8MuKdr8t21+zyJ/uNedHAlPFUmYOjY0S9/0OU2S2dIK0WL3ty0Uq/Lo69sE95g5yJD92OK55S6kWbD4sIp42Ova6gcY62WKbIRpRYjVP2QN9UJXNavSkceTLV5G67FU5za3mnPi6g88hhIpWYOiDIti9hQ9zYkwvGk8QL4j05Db+HoCI7udkLo+ok1rB47ihBUwvWrrkiFSNSt4bOcEiWtxcP+GtMbXOiXNjCdxmAq1K37x7QDKYbLibBl6C0YQNKDf5i2k9ORCA6eSQNqS5cRsA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH8PR12MB7025.namprd12.prod.outlook.com (2603:10b6:510:1bc::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Thu, 20 Jul 2023 08:40:00 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74%7]) with mapi id 15.20.6609.025; Thu, 20 Jul 2023 08:39:59 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: ajd@linux.ibm.com, catalin.marinas@arm.com, fbarrat@linux.ibm.com, iommu@lists.linux.dev, jgg@ziepe.ca, jhubbard@nvidia.com, kevin.tian@intel.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au, nicolinc@nvidia.com, npiggin@gmail.com, robin.murphy@arm.com, seanjc@google.com, will@kernel.org, x86@kernel.org, zhi.wang.linux@gmail.com, sj@kernel.org, Alistair Popple Subject: [PATCH v3 3/5] mmu_notifiers: Call invalidate_range() when invalidating TLBs Date: Thu, 20 Jul 2023 18:39:25 +1000 Message-Id: <86a0bf86394f1765fcbf9890bbabb154ba8dd980.1689842332.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: X-ClientProxiedBy: SY5P282CA0027.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:202::13) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH8PR12MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e63b837-3c96-45cc-262f-08db88fce669 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6tot57rTTfJnVhe6fp8DjkoKozXl1dpztfU9J9WUI4tb58AlpmEJDnzBjdJTh7XwmkkxhqauuOMRSVeTGHE0u4WLGmzGzN0TKJIm89mDKuRVV86Idy6ionB+dDlBS+pU7Zy/KZpvVgDGJ3PoDFg0oayuxdjRPrcNBE9DNDP4FQY3ZO9b79GxAGxv858lixRqbuZABj/BX+4vOG2OJJDUH+ib2kUxP+moHiNh1saqvDTgwXKSxzm1xmUdAJ6wIbr6fGT4/CKnreT1a+/+5E7BOxq0jazFHfmbEYqBOFdIPxhnRGtJQxbySNIhFbLaHaliMI3Im0DcTnK2gybJbuodV1ovkazXbEksQrfXmyegLzFShwwCzeEsVsrJ7YhxDcc4D6PQhVZJP/Ym8rcTK1diCT4T+kqTkQs+8zytOajTtyYamd5Uvnx0hvcL+PCxtcAd86WCYCD3hoYW15bsFHypnpt9IVzwATN6jScqzbE8INGYLaspenukshNNUgebDKSUhqA1CjsN7j9VsL+O+Km0wR8Ujo+F9FZZsaAO+PVMhK3nihEHVbwu0B394P9Utnv7G1QWNduuaa6uFfB68uJDlHsuBXjCq4f9jQiLTtxIHJNKcOHT0mQ92568KXjIAcnZ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(136003)(396003)(376002)(346002)(39860400002)(451199021)(36756003)(86362001)(38100700002)(186003)(7416002)(478600001)(6506007)(6666004)(26005)(41300700001)(66476007)(66946007)(4326008)(66556008)(6916009)(5660300002)(316002)(2616005)(8936002)(8676002)(83380400001)(6512007)(107886003)(6486002)(2906002)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /7HaGLIpXgQ8vHorvsI0YgpPv4wQ8MA9xwOxLIDX6vRO3AJJ5IP6DXoxBtzLuVgMm61Q9VrHrMkGsWk2aw0q3fLr3wsdyWR2JcYaye8AY00ryRNrtrDuJCic9g8UV9Ojdzyi0+N0iVEKYZsQ3BeIU3hrHs2uYXwzCYtD5wx51CYaj2IcIUXOWnguc6wM7lByDcH85bZdfMFaewNrtzJEZaJO1N+Qiw+1NTQHYGYeLzgriK0UNRaHjlT5t42yBGr0MfgSsd+yIf6Qri/53qMs3x9vLl6PD2RoFGTjSuNsQzvHtwqs7l81n8qh+WsDClLA+CtN6CQAFxjdrBnQFnF7nS9VdyGFPIGaWOHtfJecC42yiVLyBqXoKpqopr+YNQtH9+R9XTOexYo9zNEVb1pINZ/sowfxKj/m44Lv8CdSLj+aRiZyA0G0bpCaHXalKk4ZbTEgG43X+ydXYp10mFHAlj0VGsOrDc4LcpHmL4fEXqIvLo2ebJT8S74hCFGiuvjQtOg3ISi3y73H/4HENFLpA/wUZkTFhtOYkwCXgdf750AKk1TNkroJ/r2f7BzN7BsPOmr24aHY1EyOkBiL2kDLyQUwXpPEg5bYzKLpnKtmwi2rV4MsJ0TV6wXHPvecq88FuDEQrvIB8jmuQ4PYHj//Duy6n8Zbj/eJy4e/t6Kls1l08kl24TNp07XaElkqL6crq8HT8/K/2omkcQCIk8UhQKmjqyrjIhJO4/LQ86IWs2Un7eRQbABSJQO6wj1NErKsmELStDJ+WY4pMTBDgRriWaF6rJanVBktJKvenMbbevh2sXgnlm2NQ+i4FBITBhUQ80ujk21LVk/HXydHkciuwpOXvbQDOSwsmZouLrNFNjGHJ338K/nReuo0lSlCAnt8IVfxPksL5RsMFeHXQAv6O0xVXMibn0+KWGkeVt9aEFxFhvtaIi3I8V8p082faEUT6MVt4yJ9921Rft1NgmRdaJhVRND1wzUhBfgLy4kYeGv5sn5tpALRsi3KotP1WvgbZxJidmZgD2l89vKGVEomMfQImettCy1vj9zsyR064n3ywsyVSn9CiklXSRQgchWtSINi4ix4Rf/ctMASPWJb5NShhebNwW/sSV4FL1loPbTTQZ/3tM7OGvm43NvIeaSSpaegrcZ2DfdGcYTSyGHVfY0vQYaCvLw1gaOgyqGdbaVEXdaKBcdx0lHxUoXF3810Efrg4XkziuQDNZhO+vkZOEFnGJhQGLF8ezwMwbbMLvR6qmSA4uvnjzb5EhcPSmTvzwWeXgevT1+dn/xcaiWSpySNZyuMV+QDQGQ1bkoweOWGXk7MNpknimL2NpYwDKyzXayhrfPksb4Ci1ht6VdaEija8g9H+6kMOCL+8KJw4dlnis6PgPUX4g+m/2zMgwY7cfeImCTUhHli4Xzroer6StstUquviHMBghqnB1J8OQ37Gcgv2RhsiqbPxTNEB+k96hAvgTRHZTWpIK8oaqbemnSbm9zv7E6kJ5VCExHfFXtrJ+57AE+hcaSww8F54N3usj4zpUBrtuL137jSAjLzZ4PoGmPU8oOd2RTK2qfe1OY3ydf/SmFROUCaowe/rF43 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e63b837-3c96-45cc-262f-08db88fce669 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 08:39:59.8060 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I0WbJ6hjFPqz9JkPDEgkJtq+Z0pPoQ2oupaF0DF15p5eatu+NmOCruQ/QR1C1X0PufRN7RqADUfhi5NZ13wzJw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7025 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771929537478765420 X-GMAIL-MSGID: 1771929537478765420 The invalidate_range() is going to become an architecture specific mmu notifier used to keep the TLB of secondary MMUs such as an IOMMU in sync with the CPU page tables. Currently it is called from separate code paths to the main CPU TLB invalidations. This can lead to a secondary TLB not getting invalidated when required and makes it hard to reason about when exactly the secondary TLB is invalidated. To fix this move the notifier call to the architecture specific TLB maintenance functions for architectures that have secondary MMUs requiring explicit software invalidations. This fixes a SMMU bug on ARM64. On ARM64 PTE permission upgrades require a TLB invalidation. This invalidation is done by the architecutre specific ptep_set_access_flags() which calls flush_tlb_page() if required. However this doesn't call the notifier resulting in infinite faults being generated by devices using the SMMU if it has previously cached a read-only PTE in it's TLB. Moving the invalidations into the TLB invalidation functions ensures all invalidations happen at the same time as the CPU invalidation. The architecture specific flush_tlb_all() routines do not call the notifier as none of the IOMMUs require this. Signed-off-by: Alistair Popple Suggested-by: Jason Gunthorpe Tested-by: SeongJae Park Acked-by: Catalin Marinas Reviewed-by: Jason Gunthorpe --- arch/arm64/include/asm/tlbflush.h | 5 +++++ arch/powerpc/include/asm/book3s/64/tlbflush.h | 1 + arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 1 + arch/powerpc/mm/book3s64/radix_tlb.c | 6 ++++++ arch/x86/include/asm/tlbflush.h | 2 ++ arch/x86/mm/tlb.c | 2 ++ include/asm-generic/tlb.h | 1 - 7 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 3456866..a99349d 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include @@ -252,6 +253,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm) __tlbi(aside1is, asid); __tlbi_user(aside1is, asid); dsb(ish); + mmu_notifier_invalidate_range(mm, 0, -1UL); } static inline void __flush_tlb_page_nosync(struct mm_struct *mm, @@ -263,6 +265,8 @@ static inline void __flush_tlb_page_nosync(struct mm_struct *mm, addr = __TLBI_VADDR(uaddr, ASID(mm)); __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); + mmu_notifier_invalidate_range(mm, uaddr & PAGE_MASK, + (uaddr & PAGE_MASK) + PAGE_SIZE); } static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, @@ -396,6 +400,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, scale++; } dsb(ish); + mmu_notifier_invalidate_range(vma->vm_mm, start, end); } static inline void flush_tlb_range(struct vm_area_struct *vma, diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0d0c144..dca0477 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -5,6 +5,7 @@ #define MMU_NO_CONTEXT ~0UL #include +#include #include #include diff --git a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c index 5e31955..f3fb49f 100644 --- a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c +++ b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c @@ -39,6 +39,7 @@ void radix__flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long st radix__flush_tlb_pwc_range_psize(vma->vm_mm, start, end, psize); else radix__flush_tlb_range_psize(vma->vm_mm, start, end, psize); + mmu_notifier_invalidate_range(vma->vm_mm, start, end); } void radix__huge_ptep_modify_prot_commit(struct vm_area_struct *vma, diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0bd4866..9724b26 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -752,6 +752,8 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd return radix__local_flush_hugetlb_page(vma, vmaddr); #endif radix__local_flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); + mmu_notifier_invalidate_range(vma->vm_mm, vmaddr, + vmaddr + mmu_virtual_psize); } EXPORT_SYMBOL(radix__local_flush_tlb_page); @@ -987,6 +989,7 @@ void radix__flush_tlb_mm(struct mm_struct *mm) } } preempt_enable(); + mmu_notifier_invalidate_range(mm, 0, -1UL); } EXPORT_SYMBOL(radix__flush_tlb_mm); @@ -1020,6 +1023,7 @@ static void __flush_all_mm(struct mm_struct *mm, bool fullmm) _tlbiel_pid_multicast(mm, pid, RIC_FLUSH_ALL); } preempt_enable(); + mmu_notifier_invalidate_range(mm, 0, -1UL); } void radix__flush_all_mm(struct mm_struct *mm) @@ -1228,6 +1232,7 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm, } out: preempt_enable(); + mmu_notifier_invalidate_range(mm, start, end); } void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -1392,6 +1397,7 @@ static void __radix__flush_tlb_range_psize(struct mm_struct *mm, } out: preempt_enable(); + mmu_notifier_invalidate_range(mm, start, end); } void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 837e4a5..0a54323 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -3,6 +3,7 @@ #define _ASM_X86_TLBFLUSH_H #include +#include #include #include @@ -282,6 +283,7 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); + mmu_notifier_invalidate_range(mm, 0, -1UL); } static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 267acf2..93b2f81 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -1036,6 +1037,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, put_flush_tlb_info(); put_cpu(); + mmu_notifier_invalidate_range(mm, start, end); } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b466172..bc32a22 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -456,7 +456,6 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) return; tlb_flush(tlb); - mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); __tlb_reset_range(tlb); } From patchwork Thu Jul 20 08:39:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 123127 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp2993775vqt; Thu, 20 Jul 2023 02:30:01 -0700 (PDT) X-Google-Smtp-Source: APBJJlGIfHzFmOg5N9cAi51Uj1mipT6wdnkzdjduJ34wp9/z9QJkQwipkKVcfPiewrFCT9uwbZck X-Received: by 2002:a05:6402:6d6:b0:51d:9682:e30c with SMTP id n22-20020a05640206d600b0051d9682e30cmr1797801edy.5.1689845401424; Thu, 20 Jul 2023 02:30:01 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1689845401; cv=pass; d=google.com; s=arc-20160816; b=CZ7++ZTr3J1en0CfaTPCIYv0pKY3eIU27bS7QLVGuRMr4R1g9kbWdmvpqhSnqTtgB3 IEv0JUGv65MoISaBuTeCRXXKLwBnwcbtfSAzdftF3zzZpP5zF3LbKjtZhl72FtUMP7rZ OjMTB6lO5dYbJkrOFhkcJm+nlRs6+c0c/nvVm0W1F93t4drT1I6LIJTuW8JtrWhSyp65 22YUjmHb6K9iuuYzKAdQCiGJ5/79JkopILJQ1gAXsygK/AqTGVKnnaWCWqzyrmMgdMxW uLRX93+/cARKjsKifhz4mfYdy5B34gBVIwXMnpHsIhkPOnRzhNVQ/rDheSZORLOrYiu1 2WKw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eVZWmJyawWp7qJVj6fsL5exDFyKRe6K0X+0eHRtAuyU=; fh=1zIfrf4XUV5VVq2TK3KbcYs/ElPK9ezytjx3LhRAR54=; b=tplApBPer0KeD4jJ1J37DDJhTgIoqhIdRS9kaNd+rFHgcS3eLqgpGkXRhmRm0ne8RE FaTROYufWuUlCIKpO9KcB4ydNPem6iSpJYUZJfXsMNNedUM497oFPbLkEeu17LFhhfci CedsKGHecqeLqhjX0EjE7b1UjtT9mvCSJ+uQcP2VK5eeR041lFvrVFA7VcGm17mPfi9U MtS9nvea3irtKkDAKuINeMcagja+s/wOSEFGGm1u3dGX8ep+2BeDW+K6AIexBfiAUTQJ qwWyMYJKxMXJ9vqtIyxaaGyFT8lqBuzZ1b9TRu3Yso+rDOEeIxclU52VMsvulmRVZHgT Oq2w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=pPpfBbTa; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d22-20020a056402079600b0051e012dfcf1si579117edy.292.2023.07.20.02.29.36; Thu, 20 Jul 2023 02:30:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=pPpfBbTa; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231916AbjGTJAv (ORCPT + 99 others); Thu, 20 Jul 2023 05:00:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229569AbjGTIkL (ORCPT ); Thu, 20 Jul 2023 04:40:11 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2053.outbound.protection.outlook.com [40.107.244.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5426226B8; Thu, 20 Jul 2023 01:40:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QIqKsAIVFtQNsYnJT5dHOtbgCS3X8vryi6jhBYSo6vWJ7Dpyxhh/nt7j4esY7jzVbe3jseoPSQ0gieOM9ulH2loBvgaDbXFfwXV1I4Zgj2pXlwroyKx8AS/AcINgL4hqU3UewWm4n5FJdym3BJ4ZgHH7tYHzjlOfslmpNxHCYhIkMV4iLlzf4XVnGLbYBpj2OPgc7ouDFte4gsVdq4aenzwBdUVPerw4ixJ+7L33IKd6L69/uJegGm6PO9aTJ5Rpe9CThTZGd2D4GZ992+k2fAN1zt60xwf5FYcymVt5HQ1LxO605ho0hW5Oez3i2IYGP07CwpMuTQpM5VBBqTdm/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eVZWmJyawWp7qJVj6fsL5exDFyKRe6K0X+0eHRtAuyU=; b=TEjfkrAgM6hAHJiAjkTMEOXyCaOTO45FhGJQDQOh+xUp+zoT7h7/0Upuo+ZiydWjMV4zUgC+unnM/hGjnem2uqg3v0IgwEiIOQz21kt3vBaT3L0WfoGsD31GP5rRnEVQ/iorXlZNy41rXLp5ytDVyKL5pT+yD2cfR6kN5xLdlamQXXayf/M+eOrDqYSK0sj5ajTApVhJ+YcSqscljtU0jrtHVVE8YfBgmOy4uK7mpfSCoZ0NjK/uz1y+8vLEqeM6ny+GXcq1OpX+5sFELkQb81gFDQYttvQ6+pO6wHO6YxRX7dSPOuuCtW+znqGQ183gy4JqkhEWrOZjJlMPn2l3jA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eVZWmJyawWp7qJVj6fsL5exDFyKRe6K0X+0eHRtAuyU=; b=pPpfBbTaF/OpRBTQ9IMZoTWEXW3uExYK0DT0PohZxWvWh2O6jjMyCY/XggarCm0fMX+n70+rp2iyezVxvVu7YQxrJSPqs9KD5Kl+Zlw9e9xkVi4yyOICgnXVc5v5x3Ko3NTxKtt/8N8JDiOR8RPwiq7pRPeo5RhFOxlfh7I1dnBrscNTtNtQjlEMVxozuV/JduP8ZO8Lt0vtsxuEtrSV4WbFdiZishNNPD1gV35FqzwnqZbziLPqfFb8iRXlMpSilZ2Di5G62M30PS9WcGF2sxS7VYxBLOGcI6iNgD/xMNaYvl4vx/e7NEOB2e6dQ4ckORz6I7dvbhSb+R/0sWErbA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH8PR12MB7025.namprd12.prod.outlook.com (2603:10b6:510:1bc::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Thu, 20 Jul 2023 08:40:07 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74%7]) with mapi id 15.20.6609.025; Thu, 20 Jul 2023 08:40:07 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: ajd@linux.ibm.com, catalin.marinas@arm.com, fbarrat@linux.ibm.com, iommu@lists.linux.dev, jgg@ziepe.ca, jhubbard@nvidia.com, kevin.tian@intel.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au, nicolinc@nvidia.com, npiggin@gmail.com, robin.murphy@arm.com, seanjc@google.com, will@kernel.org, x86@kernel.org, zhi.wang.linux@gmail.com, sj@kernel.org, Alistair Popple Subject: [PATCH v3 4/5] mmu_notifiers: Don't invalidate secondary TLBs as part of mmu_notifier_invalidate_range_end() Date: Thu, 20 Jul 2023 18:39:26 +1000 Message-Id: <141e786b68527b1db9fc5a3259066c360448e7a4.1689842332.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0132.ausprd01.prod.outlook.com (2603:10c6:10:1b9::10) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH8PR12MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 01842850-9864-4354-6da3-08db88fceb7c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZsxigUjQwEWxP12RqI7Nij/BMznIbynTr4xQGb+1dl5S5z17z56Z5DOu31OxsYQP8rxBxJkiTcgWClcEmEshn2ey5m6wxJ2vQ4TH0RhrhmadrppiLY46MB3Z6/fW3/kdQBPjpytircqmKSSSK+0pyu8js5zxHD+YUQfSdrjhK+K3tzomeEjzhnPRZp4yr4ZG6GuzAFN6EWBaFewsLTGHpQE72WerEc6p8EcnwgytxUW060glBSr86u9dX4ZeqxML3Q1EHNnUbPq6E06S27Neh+GGQoi081Z9TOgc3f1WPFY8M9oPAjJgRleieeW31SWr592WK6U8VvO98FXW+HHIkCRk8NVkAJTspZsvXWLmt2kd7EuDSz4oIfZoKjuBGPqP+9H3a0CgyT4HvwWHXj2MxN1bLqTptaG25kKz6F2ydJKy+NL+uuUyNBEke/MuL34E0tWqG3PrALOoay29LlT6Doa77ruxOJvmYhSVymNg0gNrBDYGPkPQNZYg6guvoKRqe2g1QKiFcIm5PZ9jw1NGa+4DF9PpJzJW0ZSclde4VrM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(136003)(396003)(376002)(346002)(39860400002)(451199021)(36756003)(86362001)(38100700002)(186003)(7416002)(478600001)(6506007)(6666004)(26005)(41300700001)(66476007)(66946007)(4326008)(66556008)(6916009)(5660300002)(316002)(2616005)(8936002)(8676002)(83380400001)(6512007)(107886003)(6486002)(30864003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: nQ3Qq5TqscrekbPzQjL4/+Gh5arDy7luatp0n6hlOVki06Imbh7MnTnxZZ2n8s4rZXPFlBLwiaYOY8IDQ5HUXfnOHtSJZ+FLQtRph4ePfG0iHnPUUyWMxa2ILDcbRM0HcLaMbNCWe5vD60D8a7N0ImyJJwFpG7xfbX/DelydMGRfFwv9YK5Y1zckMXLyG/zki7m7g7yEQLaYntO86qr2yeHEPqJP816RvloxAoOtchUUVB8dDE2Z3KDBPgLHjUuM+wQL71k9VC82vR1Qul1t3cCwfGtOwUGMIaNXuLpHSJeq9M5elGFy+dm5LuCNMzSLADwlXYLqPGHRQ/TDypKOkcEp3jSACMPOYqVIyr8Yb28j5LW6yZbFd9giu+7laeOv6a/sxUr4eI51OEgEuR3qRNP6N/U1B4tOnJU8wYoi7gYNPl48o/bhYSl4ZP9Vjhwb7TLdJwRR8PyBFKyQT8X19bGJz4adWuuKhTVd/4mGzmDLLK5aAW0KpMNn9/UF08EWC4rDqc+20zLb6yBinlcfRs5k8WQomOjjTI0Pn4WzfqHf4VQZDPTK64+VonIUuyaSOpSsjmFoz2k2ZV6diUZ85iWV4SYDcwnHkwjv7+tlPYsPwSZomjlwXdm+rd+9YBW3palRAUgau+GaERy4QzjZG9cix1gt9N2ERsm5UCPMqysXwf29Y8FIP+lk7cSzuPNz+6670Kviv4cALVQGziBMT9Bl6IY8535rBtJv3PbsjxKGeb+X0LybpgP/T0kLuZfFX62hCnux5/ml7+ZjQjk0ardXmc+Xw0wA1NuMrSB3hUas2g0Cn4gj27LZLVlD5TgMfowEimg8E5FQWSkoKO9tgfgHCaCx3+8IK8I640mpoglbjpFyO3mQenNmw9kytiDqcGv/6qSN4QjArQpxEVJ5i3tist9FQ2bDYgl1IvFynYIrQ2cNfLaxMumxBA9AfGINDOxG1lw5uLZs5brCPOfLIAWhS/1JRQaVX2A3dQV7Fr8S+/SWDQgtZ77pX6sNcgkScplkOW4+2JOal/hb6C/79jBYjfcD+ezKqxu1S4CodL6tVmUrC+RZNwcHsmpB44UJKnQosxt7klfZvogWVnPblgs1QEjWeijIldVNiiOMu2flA/OdUp8EOXpOLcWTwQCJSEtJnvgt4ZIH4wZkHGaTK5Pzc21dF3++b/YRho7xNsVZgZZkkVJ49bGX8i5a/wuH6eaGfbWBZQsSE0xZlt4xqaVVwIViAnCN8pe2p+COZq9zHyv0eUQXWsBm4s1DDUvOf6PeXI2Zhvc8dujJ/DiLQ3yvd8RGnzhaWjUx/k1tfQDJ78N/GQ9lB8N++592EvSR7Bal/KTkM8jRk313P3dYcxoHbJ19UdWnMGfzCKbjwGknuqpNGi5YuSBeLAdRhrnyNCUibrQh0WH0NsefBcjicYVals/3fLkj5CrlEmVPHoQVAf4O/ft/DBFQMsnYGIJnPWKOUWeswweOKU7UOWU9OlMpxhMAfwCJOut17AoRtCUy+F+iP0DtGcGg78jmSK3ca/UWpz1nWKKWzhoksvwhEbcg5BzZcJWBMhRQgD1BeGlEC+VxgsjgSMHD44UrhDn2 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 01842850-9864-4354-6da3-08db88fceb7c X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 08:40:06.9797 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5736zlc0rImE8FuDAZpZEPCmQgTpNlfn7N6ulUCe5VJ/klTgBpK/7NrPIxESovVvmpi3MIycm9JQF8zC0/suXA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7025 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771931331668603960 X-GMAIL-MSGID: 1771931331668603960 Secondary TLBs are now invalidated from the architecture specific TLB invalidation functions. Therefore there is no need to explicitly notify or invalidate as part of the range end functions. This means we can remove mmu_notifier_invalidate_range_end_only() and some of the ptep_*_notify() functions. Signed-off-by: Alistair Popple Reviewed-by: Jason Gunthorpe --- include/linux/mmu_notifier.h | 56 +------------------------------------ kernel/events/uprobes.c | 2 +- mm/huge_memory.c | 25 ++--------------- mm/hugetlb.c | 1 +- mm/memory.c | 8 +---- mm/migrate_device.c | 9 +----- mm/mmu_notifier.c | 25 ++--------------- mm/rmap.c | 40 +-------------------------- 8 files changed, 14 insertions(+), 152 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 64a3e05..f2e9edc 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -395,8 +395,7 @@ extern int __mmu_notifier_test_young(struct mm_struct *mm, extern void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); -extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r, - bool only_end); +extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range(struct mm_struct *mm, unsigned long start, unsigned long end); extern bool @@ -481,14 +480,7 @@ mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) might_sleep(); if (mm_has_notifiers(range->mm)) - __mmu_notifier_invalidate_range_end(range, false); -} - -static inline void -mmu_notifier_invalidate_range_only_end(struct mmu_notifier_range *range) -{ - if (mm_has_notifiers(range->mm)) - __mmu_notifier_invalidate_range_end(range, true); + __mmu_notifier_invalidate_range_end(range); } static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, @@ -582,45 +574,6 @@ static inline void mmu_notifier_range_init_owner( __young; \ }) -#define ptep_clear_flush_notify(__vma, __address, __ptep) \ -({ \ - unsigned long ___addr = __address & PAGE_MASK; \ - struct mm_struct *___mm = (__vma)->vm_mm; \ - pte_t ___pte; \ - \ - ___pte = ptep_clear_flush(__vma, __address, __ptep); \ - mmu_notifier_invalidate_range(___mm, ___addr, \ - ___addr + PAGE_SIZE); \ - \ - ___pte; \ -}) - -#define pmdp_huge_clear_flush_notify(__vma, __haddr, __pmd) \ -({ \ - unsigned long ___haddr = __haddr & HPAGE_PMD_MASK; \ - struct mm_struct *___mm = (__vma)->vm_mm; \ - pmd_t ___pmd; \ - \ - ___pmd = pmdp_huge_clear_flush(__vma, __haddr, __pmd); \ - mmu_notifier_invalidate_range(___mm, ___haddr, \ - ___haddr + HPAGE_PMD_SIZE); \ - \ - ___pmd; \ -}) - -#define pudp_huge_clear_flush_notify(__vma, __haddr, __pud) \ -({ \ - unsigned long ___haddr = __haddr & HPAGE_PUD_MASK; \ - struct mm_struct *___mm = (__vma)->vm_mm; \ - pud_t ___pud; \ - \ - ___pud = pudp_huge_clear_flush(__vma, __haddr, __pud); \ - mmu_notifier_invalidate_range(___mm, ___haddr, \ - ___haddr + HPAGE_PUD_SIZE); \ - \ - ___pud; \ -}) - /* * set_pte_at_notify() sets the pte _after_ running the notifier. * This is safe to start by updating the secondary MMUs, because the primary MMU @@ -711,11 +664,6 @@ void mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) { } -static inline void -mmu_notifier_invalidate_range_only_end(struct mmu_notifier_range *range) -{ -} - static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, unsigned long start, unsigned long end) { diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f0ac5b8..3048589 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -193,7 +193,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, } flush_cache_page(vma, addr, pte_pfn(ptep_get(pvmw.pte))); - ptep_clear_flush_notify(vma, addr, pvmw.pte); + ptep_clear_flush(vma, addr, pvmw.pte); if (new_page) set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 762be2f..3ece117 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2003,7 +2003,7 @@ static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud, count_vm_event(THP_SPLIT_PUD); - pudp_huge_clear_flush_notify(vma, haddr, pud); + pudp_huge_clear_flush(vma, haddr, pud); } void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, @@ -2023,11 +2023,7 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, out: spin_unlock(ptl); - /* - * No need to double call mmu_notifier->invalidate_range() callback as - * the above pudp_huge_clear_flush_notify() did already call it. - */ - mmu_notifier_invalidate_range_only_end(&range); + mmu_notifier_invalidate_range_end(&range); } #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ @@ -2094,7 +2090,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, count_vm_event(THP_SPLIT_PMD); if (!vma_is_anonymous(vma)) { - old_pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); + old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd); /* * We are going to unmap this huge page. So * just go ahead and zap it @@ -2304,20 +2300,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, out: spin_unlock(ptl); - /* - * No need to double call mmu_notifier->invalidate_range() callback. - * They are 3 cases to consider inside __split_huge_pmd_locked(): - * 1) pmdp_huge_clear_flush_notify() call invalidate_range() obvious - * 2) __split_huge_zero_page_pmd() read only zero page and any write - * fault will trigger a flush_notify before pointing to a new page - * (it is fine if the secondary mmu keeps pointing to the old zero - * page in the meantime) - * 3) Split a huge pmd into pte pointing to the same page. No need - * to invalidate secondary tlb entry they are all still valid. - * any further changes to individual pte will notify. So no need - * to call mmu_notifier->invalidate_range() - */ - mmu_notifier_invalidate_range_only_end(&range); + mmu_notifier_invalidate_range_end(&range); } void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index dc1ec19..9c6e431 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5715,7 +5715,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - mmu_notifier_invalidate_range(mm, range.start, range.end); page_remove_rmap(&old_folio->page, vma, true); hugepage_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) diff --git a/mm/memory.c b/mm/memory.c index ad79039..8dca544 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3158,7 +3158,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * that left a window where the new PTE could be loaded into * some TLBs while the old PTE remains in others. */ - ptep_clear_flush_notify(vma, vmf->address, vmf->pte); + ptep_clear_flush(vma, vmf->address, vmf->pte); folio_add_new_anon_rmap(new_folio, vma, vmf->address); folio_add_lru_vma(new_folio, vma); /* @@ -3204,11 +3204,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); } - /* - * No need to double call mmu_notifier->invalidate_range() callback as - * the above ptep_clear_flush_notify() did already call it. - */ - mmu_notifier_invalidate_range_only_end(&range); + mmu_notifier_invalidate_range_end(&range); if (new_folio) folio_put(new_folio); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index e29626e..6c556b5 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -658,7 +658,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, if (flush) { flush_cache_page(vma, addr, pte_pfn(orig_pte)); - ptep_clear_flush_notify(vma, addr, ptep); + ptep_clear_flush(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, entry); update_mmu_cache(vma, addr, ptep); } else { @@ -763,13 +763,8 @@ static void __migrate_device_pages(unsigned long *src_pfns, src_pfns[i] &= ~MIGRATE_PFN_MIGRATE; } - /* - * No need to double call mmu_notifier->invalidate_range() callback as - * the above ptep_clear_flush_notify() inside migrate_vma_insert_page() - * did already call it. - */ if (notified) - mmu_notifier_invalidate_range_only_end(&range); + mmu_notifier_invalidate_range_end(&range); } /** diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index b7ad155..453a156 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -551,7 +551,7 @@ int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) static void mn_hlist_invalidate_end(struct mmu_notifier_subscriptions *subscriptions, - struct mmu_notifier_range *range, bool only_end) + struct mmu_notifier_range *range) { struct mmu_notifier *subscription; int id; @@ -559,24 +559,6 @@ mn_hlist_invalidate_end(struct mmu_notifier_subscriptions *subscriptions, id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(subscription, &subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { - /* - * Call invalidate_range here too to avoid the need for the - * subsystem of having to register an invalidate_range_end - * call-back when there is invalidate_range already. Usually a - * subsystem registers either invalidate_range_start()/end() or - * invalidate_range(), so this will be no additional overhead - * (besides the pointer check). - * - * We skip call to invalidate_range() if we know it is safe ie - * call site use mmu_notifier_invalidate_range_only_end() which - * is safe to do when we know that a call to invalidate_range() - * already happen under page table lock. - */ - if (!only_end && subscription->ops->invalidate_range) - subscription->ops->invalidate_range(subscription, - range->mm, - range->start, - range->end); if (subscription->ops->invalidate_range_end) { if (!mmu_notifier_range_blockable(range)) non_block_start(); @@ -589,8 +571,7 @@ mn_hlist_invalidate_end(struct mmu_notifier_subscriptions *subscriptions, srcu_read_unlock(&srcu, id); } -void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range, - bool only_end) +void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) { struct mmu_notifier_subscriptions *subscriptions = range->mm->notifier_subscriptions; @@ -600,7 +581,7 @@ void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range, mn_itree_inv_end(subscriptions); if (!hlist_empty(&subscriptions->list)) - mn_hlist_invalidate_end(subscriptions, range, only_end); + mn_hlist_invalidate_end(subscriptions, range); lock_map_release(&__mmu_notifier_invalidate_range_start_map); } diff --git a/mm/rmap.c b/mm/rmap.c index 1355bf6..51ec8aa 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -985,13 +985,6 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) #endif } - /* - * No need to call mmu_notifier_invalidate_range() as we are - * downgrading page table protection not changing it to point - * to a new page. - * - * See Documentation/mm/mmu_notifier.rst - */ if (ret) cleaned++; } @@ -1549,8 +1542,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); flush_tlb_range(vma, range.start, range.end); - mmu_notifier_invalidate_range(mm, - range.start, range.end); /* * The ref count of the PMD page was * dropped which is part of the way map @@ -1623,9 +1614,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * copied pages. */ dec_mm_counter(mm, mm_counter(&folio->page)); - /* We have to invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, address, - address + PAGE_SIZE); } else if (folio_test_anon(folio)) { swp_entry_t entry = { .val = page_private(subpage) }; pte_t swp_pte; @@ -1637,9 +1625,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, folio_test_swapcache(folio))) { WARN_ON_ONCE(1); ret = false; - /* We have to invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, address, - address + PAGE_SIZE); page_vma_mapped_walk_done(&pvmw); break; } @@ -1670,9 +1655,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ if (ref_count == 1 + map_count && !folio_test_dirty(folio)) { - /* Invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, - address, address + PAGE_SIZE); dec_mm_counter(mm, MM_ANONPAGES); goto discard; } @@ -1727,9 +1709,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (pte_uffd_wp(pteval)) swp_pte = pte_swp_mkuffd_wp(swp_pte); set_pte_at(mm, address, pvmw.pte, swp_pte); - /* Invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, address, - address + PAGE_SIZE); } else { /* * This is a locked file-backed folio, @@ -1745,13 +1724,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - /* - * No need to call mmu_notifier_invalidate_range() it has be - * done above for all cases requiring it to happen under page - * table lock before mmu_notifier_invalidate_range_end() - * - * See Documentation/mm/mmu_notifier.rst - */ page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); @@ -1930,8 +1902,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); flush_tlb_range(vma, range.start, range.end); - mmu_notifier_invalidate_range(mm, - range.start, range.end); /* * The ref count of the PMD page was @@ -2036,9 +2006,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * copied pages. */ dec_mm_counter(mm, mm_counter(&folio->page)); - /* We have to invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, address, - address + PAGE_SIZE); } else { swp_entry_t entry; pte_t swp_pte; @@ -2102,13 +2069,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - /* - * No need to call mmu_notifier_invalidate_range() it has be - * done above for all cases requiring it to happen under page - * table lock before mmu_notifier_invalidate_range_end() - * - * See Documentation/mm/mmu_notifier.rst - */ page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); From patchwork Thu Jul 20 08:39:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 123140 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp2998074vqt; Thu, 20 Jul 2023 02:40:43 -0700 (PDT) X-Google-Smtp-Source: APBJJlEGiAJNpPYensDxENgbezvd8JrBaiWm4SkBLHkG/PTw1FsUWo/F3mvkt3w3YCEDspdP7VUA X-Received: by 2002:a17:902:e5c4:b0:1b8:1c4f:4f8e with SMTP id u4-20020a170902e5c400b001b81c4f4f8emr25992642plf.53.1689846042935; Thu, 20 Jul 2023 02:40:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1689846042; cv=pass; d=google.com; s=arc-20160816; b=greKMbQZYwS/SLDALi9BvXC4ntOGO0OjQOIyxom/6LQI7JP1knP/OLKNRImCrZfEZS pcKVu7SKWjOJmNWZtvcIzaoetUjqF/2aFRkAu5OaixmkaiDA9DxZjgVdElEGvbI001md smU3GCP6ma4B6foQjkPl6xVK0Ae73ttDaBIEqazpE44jEHNaPTAw+Qf5eEbpi/4ys7vQ UX133MjGAy0TJlrSipJJQNsFrDqj1ZdtdfqmolGPrJi+YoRwB0JI6XcOHCGDfxF7HRmx Ui+vfekc4S99BgtPf19/iGzbOWehT5Er7EfIKliZtx39bySVMHAvg9V7nOWg8RJJJvSa zDcg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MH28H+CAnzGmPruk0vGr9QVHzZtmLOQboUMLWND1YGU=; fh=aO66KPFddCbqAtBrwMAFStkAjzS5rFFXWC0YjxwFscw=; b=qQBL5hCo+CkQis5ixoI9STQZyEU7g/vHz5B2pyzs8YlodZoUFEz7icVOfp++kkph8U Uob01WWNmfAsjMd4W+wKMyJdN5vhqme7hiK1rpYR3qXvGb+zv87elwd43V38nO8iOGul h4bElM26hryhGPRudUdBrOmOM5fRHPjAMXa+tB5LryOgr7xjJJvTzEyzs7it6mjSqed9 ozKrs4gbA4GTkBUc40S1cAvst5hnkM0nA3NB6kg/QtWCqNUOOKF6i6FcJTBe/3slHAlS 7jsjrhWZh+zX03GePdYT5CckmAcoTwEZ8r9MT5Jr5IxFAEsrshSb+7C/9PoI19m1y48m BV6g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=mE3VgSkr; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t14-20020a17090340ce00b001aaed82c2afsi626541pld.171.2023.07.20.02.40.30; Thu, 20 Jul 2023 02:40:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=mE3VgSkr; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231213AbjGTI60 (ORCPT + 99 others); Thu, 20 Jul 2023 04:58:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229567AbjGTIkS (ORCPT ); Thu, 20 Jul 2023 04:40:18 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2076.outbound.protection.outlook.com [40.107.244.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6C4CFD; Thu, 20 Jul 2023 01:40:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cp0sD6v7udW5RFf4udneb36JIpoguD3JuCQrUBGAQNIHTLbYnkKuiyD2ESJ1nwulGkS24XP8HtG9BYXL8/2Ahhc6d6gyj/kQ2JkDg34i2osgwllOBsDAmJ1pKok2y1c/pHqIGxUnIYSx6OTGnVkCj4hgEOqBaUnQECcgvsHDYjWOHQD8RKTJP9+vgDP6r6K8BD0PnMqZFPumGRR+sZFcx+FkEGH/8pJ5foVUtbWlJ1lzSu78FwETLmMDsV9WGhlqt112S+/Ak43d9k5cKFNJLOwc+QH0kAbU8qAZkTCaryKtsdHm1+vVI5r2yu1dTrBU56jXVZRafOMW7TfquWn2YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MH28H+CAnzGmPruk0vGr9QVHzZtmLOQboUMLWND1YGU=; b=BoFDiv66r18UgHbxA5bntoevMyj4UhZtdW/8yCALPOVbgQD480tny8FDpz4Dz1WhJW62opsOnYxxU1EsuF/agYqqGZxFTbGDseJRA/4NnEtJFED5PxmtmP2sVajINsbQcPBsixU1qfblww2+JhKSWO7PWhkxyxAx/T1yCCGcLNsoI0xLqK5T+lQRsIwHQxd8BaprSdxumCTlNvYy1kmrW612IPaoj3DJWhwxeGcK2+EyKXMwtjTxJDPBkww6y+wOZ+xdYREMohKUz0gfaopRHaO3M4vYiIfCyW/+y6JYBJe8Ld0ea+W0GODAymTb5XIUh5hKXhR2bia+I/eEp56eHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MH28H+CAnzGmPruk0vGr9QVHzZtmLOQboUMLWND1YGU=; b=mE3VgSkr4Zq62VrFLqFvZqijVQhldVPKcC92Z0aSLYdsNNtBktvQs6cQlPEg/Qdr0FDzk+7paWIeM7kryTPPRTsvUflZBF/1s0zNyJ9XS/IPftLa7MZF/dZMTnArn8jUbLArqzRpDZk7EOuO1Xb3yu9hJgrQ/cHwhffyUwpXzZF5KRo8Iz8MPj7022b3fQOB33X/bSJfLALSljSWQ92xAeu/Ys39HwPH4wbuJ6Q/Et2kTgWA67ZPi26td3jjfh9h7Fm239977wsIlxL3YZI6Eydzjx6g1FFxbnOUwoWDwN0K4wE9tItz1hcu0TamHbtG9b0RQHEHxS68VlleDpAWlQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH8PR12MB7025.namprd12.prod.outlook.com (2603:10b6:510:1bc::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.24; Thu, 20 Jul 2023 08:40:14 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::cd5e:7e33:c2c9:fb74%7]) with mapi id 15.20.6609.025; Thu, 20 Jul 2023 08:40:14 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: ajd@linux.ibm.com, catalin.marinas@arm.com, fbarrat@linux.ibm.com, iommu@lists.linux.dev, jgg@ziepe.ca, jhubbard@nvidia.com, kevin.tian@intel.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, mpe@ellerman.id.au, nicolinc@nvidia.com, npiggin@gmail.com, robin.murphy@arm.com, seanjc@google.com, will@kernel.org, x86@kernel.org, zhi.wang.linux@gmail.com, sj@kernel.org, Alistair Popple , Jason Gunthorpe Subject: [PATCH v3 5/5] mmu_notifiers: Rename invalidate_range notifier Date: Thu, 20 Jul 2023 18:39:27 +1000 Message-Id: <3cbd2a644d56d503b47cfc35868d547f924f880e.1689842332.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0156.ausprd01.prod.outlook.com (2603:10c6:10:d::24) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH8PR12MB7025:EE_ X-MS-Office365-Filtering-Correlation-Id: 05cbfbc1-c0ff-4054-614e-08db88fcefb4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: q1RUZK5rBSPAC260poeKsGZQ67ToNdg7OP+9XBJ6wA39XWmMHi0W3R0vXPnUfYs4jKzzOxx+1kNld+Dxc/BSykJDe1EU4fE7nVGhkjMZJ02jAvf3B/g4/AOwaWoKFmPdDS98SbNh0Uuzq3wNtBKmr8i8lxq+Hjkg/5ibYLg4K6i1iXCaLJSC7HBZIxhSJ4HkAh1+AmpvEPC+fdPaGhzA+znYoOc9Yqehj1B1RKRKTsEQ9aw93AvZb8/rzOwejZvW8RM/n9XkL9JczjsH4nQtnoclbvLSYUzwrTmOTPQNeni0Xio+vILv4t0xE0M8mcBDp+v++6MbRphxqzqQwP+Zz6+XkZr6F4NpJF6fWCISwJIkFRw8lkgJoCL2JZUrmlu8IfVx1Drlnnv5OcR6owGwooICIL0zKlhb1/ntaaaaQV5s6M4UyR7y26hxh2BgUhv4BZKe8cnN6Tmc6FdzcQREYwI1f9JZ6mc5whT91EtsjR6QaTpxVfsh8+bku5uPmF7wsXc7me1BZ8YxJOBw1eV+oY9F6spRQVhv/0Jh7ROghCUBa5RwWGuqD7udsii+7jXyRdninviN8zv5FMusKQLKQgirzieEKqYjjTxByTpSx60= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(136003)(396003)(376002)(346002)(39860400002)(451199021)(36756003)(86362001)(38100700002)(186003)(7416002)(478600001)(6506007)(6666004)(26005)(41300700001)(66476007)(66946007)(4326008)(66556008)(6916009)(5660300002)(316002)(2616005)(8936002)(8676002)(83380400001)(6512007)(107886003)(6486002)(30864003)(54906003)(2906002)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Lu9/xGhECsxFSuPSVe21S4nShrycFRbZjy4S51HrMBfAgHUO2hFR+v33/n7Ihk3vfip4UkLaZKXwf0KnCDXHKG4icj08TVmSl42R0+5cTz96zkdnY/EYaTYB3goY1DnRfLUtMWrfTTeILj1UnsiN9ClL3N02SGYqU15ladFrpfr2KGMJUoV19vFE+sQs6f8NQJt31IqXgemQx7hRZp4ZgOchEvqGhpYlZwWdScNR7OkAkVb5R0/K9168hTO6awdUb1ZKYQoGyFZ3yZjNNNmo3SFXQuW8FgKQgN7P9T26EhxCihx1Dep9edpNxMSIt2edRaTOrk0Cu7wU/T8FtYNMtGzTj4zxU/w0rSOXvnLQTfRlccybKUyPaf/kd4pj/JYW0fjV5DIUYS2FG7SWCkjjtG0ylT7PTdbVT02+x13VnhUApIrjDWa+ZeLVM7DWygBuKnAKss3yAeDJQ2t+50CvWR97bemW+dVlw03ObCYRmtom8x0+F6k6xFSnYgD3XFFaZOIHn8xLsnntYfa+wl0i1uPc8/5hjgbjCaRHNy/KK0Dr/SOpPHQfxPc/9hhBNz8DcWaZ9csjrXd28CyOpKd6cOjyd/RIztc19gYDAZDJ6xbjgfZ5qFjnsPLmvfFo2/WTcSrWdS4I8T8VHQ17F4Un9Wp/1I+aJ/A76xPtYZYQ/owUFGUi9Kx2a5OAzs7ktbGRl4xPrRD8006DNwodDeuQbszZ7MDO9gSS7aQRdyT75OHT/4bsDaJbVufmdY6U2uMq2Cv/6x9mapH2a+rqn+fHqFrX8d5Qnv9mQO3URXkM3JppnG/MxxFDgBzsjQjMGV9FNrJ1Zqd1kq5W6yjz6bHjPP1awv+c02hrlixNu9iBqsvVV5tvhKP4tP2OcziEJ4cfgEWpfDP68by2QVtTI/25TyVWcFRxKDW9jZ1JdMX/Furt3DsntPd8h/woEahsQgZ0CoQk2o3eS9ExVBJ7MyLNyWuwGOEFsE10eJiO3G0dXCVaIe8sxv/HdHyBDZ0OVSTf+dc0on7xRfvtR+tCLOBHA7wXtdMUvQrAsudsxcM/kLGK/1lFoelnJljJ/YOCqsCXg94v1wFijvh/aN8Qg2vr6A9tHobHdHKkN5hspc1yFw7dKCS/Yt3QdbJjBbRlgw78+wV6NoRSrHKAcqdA+X9iv9nHgvqbBc3CyrWbDUeJaVB8thL5Fg4zWfJ2wQfm0NfciKTZZ3Qn3divMnuZH9yBCaZLSMb99DvRfDHrvLJjx8FEGVhFguPCtIxksfK04Ke03AV5O9LB7Q4dkuF2pbOT54xOuRGnjdDhNUVjkDjwtVDT91xX7HmDvk4uzxMdPJ/1Kw4mXm6IbQyqUUXYJU+VqlKO8nZ0uIVH5N7dh98GAD7RkMPW9981OkIXRe1pOrukn5JMDnBj/8IbN4UbnfHDA8nUMmEWzhTVwbytwIGBW80zroEDko62oUuP/R/BY542/1+IlMf6sdScpSzSH2E+TLkxl2pJ5Krl9jd+ewR3z0kNyAe5FURiW6ezi459z3x9NiKir4XMBfH6hztfYVVcVTaZxP3Y7yOaw0mT2rILsQwojRpB6KmldCvtiJjWw4Nt X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 05cbfbc1-c0ff-4054-614e-08db88fcefb4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 08:40:14.0687 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tix7EeaDS6l53ammRLs09uuO6zUxjam7jToWa7NELS3yf9S5uR7dpmf3vtRf5ORNgZt1wd3vx0Q3+/eMZ8QQ9A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7025 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771932004334474246 X-GMAIL-MSGID: 1771932004334474246 There are two main use cases for mmu notifiers. One is by KVM which uses mmu_notifier_invalidate_range_start()/end() to manage a software TLB. The other is to manage hardware TLBs which need to use the invalidate_range() callback because HW can establish new TLB entries at any time. Hence using start/end() can lead to memory corruption as these callbacks happen too soon/late during page unmap. mmu notifier users should therefore either use the start()/end() callbacks or the invalidate_range() callbacks. To make this usage clearer rename the invalidate_range() callback to arch_invalidate_secondary_tlbs() and update documention. Signed-off-by: Alistair Popple Suggested-by: Jason Gunthorpe Acked-by: Catalin Marinas Reviewed-by: Jason Gunthorpe --- arch/arm64/include/asm/tlbflush.h | 6 +- arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 10 ++-- arch/x86/include/asm/tlbflush.h | 2 +- arch/x86/mm/tlb.c | 2 +- drivers/iommu/amd/iommu_v2.c | 10 ++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 13 ++--- drivers/iommu/intel/svm.c | 8 +-- drivers/misc/ocxl/link.c | 8 +-- include/linux/mmu_notifier.h | 48 +++++++++--------- mm/huge_memory.c | 4 +- mm/hugetlb.c | 7 +-- mm/mmu_notifier.c | 20 ++++++-- 13 files changed, 76 insertions(+), 64 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index a99349d..84a05a0 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -253,7 +253,7 @@ static inline void flush_tlb_mm(struct mm_struct *mm) __tlbi(aside1is, asid); __tlbi_user(aside1is, asid); dsb(ish); - mmu_notifier_invalidate_range(mm, 0, -1UL); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } static inline void __flush_tlb_page_nosync(struct mm_struct *mm, @@ -265,7 +265,7 @@ static inline void __flush_tlb_page_nosync(struct mm_struct *mm, addr = __TLBI_VADDR(uaddr, ASID(mm)); __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); - mmu_notifier_invalidate_range(mm, uaddr & PAGE_MASK, + mmu_notifier_arch_invalidate_secondary_tlbs(mm, uaddr & PAGE_MASK, (uaddr & PAGE_MASK) + PAGE_SIZE); } @@ -400,7 +400,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, scale++; } dsb(ish); - mmu_notifier_invalidate_range(vma->vm_mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } static inline void flush_tlb_range(struct vm_area_struct *vma, diff --git a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c index f3fb49f..17075c7 100644 --- a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c +++ b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c @@ -39,7 +39,7 @@ void radix__flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long st radix__flush_tlb_pwc_range_psize(vma->vm_mm, start, end, psize); else radix__flush_tlb_range_psize(vma->vm_mm, start, end, psize); - mmu_notifier_invalidate_range(vma->vm_mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } void radix__huge_ptep_modify_prot_commit(struct vm_area_struct *vma, diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 9724b26..64c11a4 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -752,7 +752,7 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd return radix__local_flush_hugetlb_page(vma, vmaddr); #endif radix__local_flush_tlb_page_psize(vma->vm_mm, vmaddr, mmu_virtual_psize); - mmu_notifier_invalidate_range(vma->vm_mm, vmaddr, + mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, vmaddr, vmaddr + mmu_virtual_psize); } EXPORT_SYMBOL(radix__local_flush_tlb_page); @@ -989,7 +989,7 @@ void radix__flush_tlb_mm(struct mm_struct *mm) } } preempt_enable(); - mmu_notifier_invalidate_range(mm, 0, -1UL); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } EXPORT_SYMBOL(radix__flush_tlb_mm); @@ -1023,7 +1023,7 @@ static void __flush_all_mm(struct mm_struct *mm, bool fullmm) _tlbiel_pid_multicast(mm, pid, RIC_FLUSH_ALL); } preempt_enable(); - mmu_notifier_invalidate_range(mm, 0, -1UL); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } void radix__flush_all_mm(struct mm_struct *mm) @@ -1232,7 +1232,7 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm, } out: preempt_enable(); - mmu_notifier_invalidate_range(mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -1397,7 +1397,7 @@ static void __radix__flush_tlb_range_psize(struct mm_struct *mm, } out: preempt_enable(); - mmu_notifier_invalidate_range(mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 0a54323..6ab42ca 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -283,7 +283,7 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); - mmu_notifier_invalidate_range(mm, 0, -1UL); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 93b2f81..2d25391 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1037,7 +1037,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, put_flush_tlb_info(); put_cpu(); - mmu_notifier_invalidate_range(mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c index 261352a..2596466 100644 --- a/drivers/iommu/amd/iommu_v2.c +++ b/drivers/iommu/amd/iommu_v2.c @@ -355,9 +355,9 @@ static struct pasid_state *mn_to_state(struct mmu_notifier *mn) return container_of(mn, struct pasid_state, mn); } -static void mn_invalidate_range(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, unsigned long end) +static void mn_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) { struct pasid_state *pasid_state; struct device_state *dev_state; @@ -391,8 +391,8 @@ static void mn_release(struct mmu_notifier *mn, struct mm_struct *mm) } static const struct mmu_notifier_ops iommu_mn = { - .release = mn_release, - .invalidate_range = mn_invalidate_range, + .release = mn_release, + .arch_invalidate_secondary_tlbs = mn_arch_invalidate_secondary_tlbs, }; static void set_pri_tag_status(struct pasid_state *pasid_state, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 2a19784..dbc812a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -186,9 +186,10 @@ static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) } } -static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, unsigned long end) +static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; @@ -247,9 +248,9 @@ static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) } static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { - .invalidate_range = arm_smmu_mm_invalidate_range, - .release = arm_smmu_mm_release, - .free_notifier = arm_smmu_mmu_notifier_free, + .arch_invalidate_secondary_tlbs = arm_smmu_mm_arch_invalidate_secondary_tlbs, + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, }; /* Allocate or get existing MMU notifier for this {domain, mm} pair */ diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index e95b339..8f6d680 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -219,9 +219,9 @@ static void intel_flush_svm_range(struct intel_svm *svm, unsigned long address, } /* Pages have been freed at this point */ -static void intel_invalidate_range(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, unsigned long end) +static void intel_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) { struct intel_svm *svm = container_of(mn, struct intel_svm, notifier); @@ -256,7 +256,7 @@ static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) static const struct mmu_notifier_ops intel_mmuops = { .release = intel_mm_release, - .invalidate_range = intel_invalidate_range, + .arch_invalidate_secondary_tlbs = intel_arch_invalidate_secondary_tlbs, }; static DEFINE_MUTEX(pasid_mutex); diff --git a/drivers/misc/ocxl/link.c b/drivers/misc/ocxl/link.c index 4cf4c55..c06c699 100644 --- a/drivers/misc/ocxl/link.c +++ b/drivers/misc/ocxl/link.c @@ -491,9 +491,9 @@ void ocxl_link_release(struct pci_dev *dev, void *link_handle) } EXPORT_SYMBOL_GPL(ocxl_link_release); -static void invalidate_range(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, unsigned long end) +static void arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) { struct pe_data *pe_data = container_of(mn, struct pe_data, mmu_notifier); struct ocxl_link *link = pe_data->link; @@ -509,7 +509,7 @@ static void invalidate_range(struct mmu_notifier *mn, } static const struct mmu_notifier_ops ocxl_mmu_notifier_ops = { - .invalidate_range = invalidate_range, + .arch_invalidate_secondary_tlbs = arch_invalidate_secondary_tlbs, }; static u64 calculate_cfg_state(bool kernel) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index f2e9edc..6e3c857 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -187,27 +187,27 @@ struct mmu_notifier_ops { const struct mmu_notifier_range *range); /* - * invalidate_range() is either called between - * invalidate_range_start() and invalidate_range_end() when the - * VM has to free pages that where unmapped, but before the - * pages are actually freed, or outside of _start()/_end() when - * a (remote) TLB is necessary. + * arch_invalidate_secondary_tlbs() is used to manage a non-CPU TLB + * which shares page-tables with the CPU. The + * invalidate_range_start()/end() callbacks should not be implemented as + * invalidate_secondary_tlbs() already catches the points in time when + * an external TLB needs to be flushed. * - * If invalidate_range() is used to manage a non-CPU TLB with - * shared page-tables, it not necessary to implement the - * invalidate_range_start()/end() notifiers, as - * invalidate_range() already catches the points in time when an - * external TLB range needs to be flushed. For more in depth - * discussion on this see Documentation/mm/mmu_notifier.rst + * This requires arch_invalidate_secondary_tlbs() to be called while + * holding the ptl spin-lock and therefore this callback is not allowed + * to sleep. * - * Note that this function might be called with just a sub-range - * of what was passed to invalidate_range_start()/end(), if - * called between those functions. + * This is called by architecture code whenever invalidating a TLB + * entry. It is assumed that any secondary TLB has the same rules for + * when invalidations are required. If this is not the case architecture + * code will need to call this explicitly when required for secondary + * TLB invalidation. */ - void (*invalidate_range)(struct mmu_notifier *subscription, - struct mm_struct *mm, - unsigned long start, - unsigned long end); + void (*arch_invalidate_secondary_tlbs)( + struct mmu_notifier *subscription, + struct mm_struct *mm, + unsigned long start, + unsigned long end); /* * These callbacks are used with the get/put interface to manage the @@ -396,8 +396,8 @@ extern void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); -extern void __mmu_notifier_invalidate_range(struct mm_struct *mm, - unsigned long start, unsigned long end); +extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, + unsigned long start, unsigned long end); extern bool mmu_notifier_range_update_to_read_only(const struct mmu_notifier_range *range); @@ -483,11 +483,11 @@ mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) __mmu_notifier_invalidate_range_end(range); } -static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, - unsigned long start, unsigned long end) +static inline void mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, + unsigned long start, unsigned long end) { if (mm_has_notifiers(mm)) - __mmu_notifier_invalidate_range(mm, start, end); + __mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } static inline void mmu_notifier_subscriptions_init(struct mm_struct *mm) @@ -664,7 +664,7 @@ void mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) { } -static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, +static inline void mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, unsigned long start, unsigned long end) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3ece117..e0420de 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2120,8 +2120,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, if (is_huge_zero_pmd(*pmd)) { /* * FIXME: Do we want to invalidate secondary mmu by calling - * mmu_notifier_invalidate_range() see comments below inside - * __split_huge_pmd() ? + * mmu_notifier_arch_invalidate_secondary_tlbs() see comments below + * inside __split_huge_pmd() ? * * We are going from a zero huge page write protected to zero * small page also write protected so it does not seems useful diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9c6e431..e0028cb 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6676,8 +6676,9 @@ long hugetlb_change_protection(struct vm_area_struct *vma, else flush_hugetlb_tlb_range(vma, start, end); /* - * No need to call mmu_notifier_invalidate_range() we are downgrading - * page table protection not changing it to point to a new page. + * No need to call mmu_notifier_arch_invalidate_secondary_tlbs() we are + * downgrading page table protection not changing it to point to a new + * page. * * See Documentation/mm/mmu_notifier.rst */ @@ -7321,7 +7322,7 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, i_mmap_unlock_write(vma->vm_file->f_mapping); hugetlb_vma_unlock_write(vma); /* - * No need to call mmu_notifier_invalidate_range(), see + * No need to call mmu_notifier_arch_invalidate_secondary_tlbs(), see * Documentation/mm/mmu_notifier.rst. */ mmu_notifier_invalidate_range_end(&range); diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 453a156..63c8eb7 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -585,8 +585,8 @@ void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) lock_map_release(&__mmu_notifier_invalidate_range_start_map); } -void __mmu_notifier_invalidate_range(struct mm_struct *mm, - unsigned long start, unsigned long end) +void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, + unsigned long start, unsigned long end) { struct mmu_notifier *subscription; int id; @@ -595,9 +595,10 @@ void __mmu_notifier_invalidate_range(struct mm_struct *mm, hlist_for_each_entry_rcu(subscription, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { - if (subscription->ops->invalidate_range) - subscription->ops->invalidate_range(subscription, mm, - start, end); + if (subscription->ops->arch_invalidate_secondary_tlbs) + subscription->ops->arch_invalidate_secondary_tlbs( + subscription, mm, + start, end); } srcu_read_unlock(&srcu, id); } @@ -616,6 +617,15 @@ int __mmu_notifier_register(struct mmu_notifier *subscription, mmap_assert_write_locked(mm); BUG_ON(atomic_read(&mm->mm_users) <= 0); + /* + * Subsystems should only register for invalidate_secondary_tlbs() or + * invalidate_range_start()/end() callbacks, not both. + */ + if (WARN_ON_ONCE(subscription->ops->arch_invalidate_secondary_tlbs && + (subscription->ops->invalidate_range_start || + subscription->ops->invalidate_range_end))) + return -EINVAL; + if (!mm->notifier_subscriptions) { /* * kmalloc cannot be called under mm_take_all_locks(), but we