From patchwork Wed Nov 9 16:51:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17694 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447317wru; Wed, 9 Nov 2022 08:57:25 -0800 (PST) X-Google-Smtp-Source: AMsMyM6WWwWXgUpBmb/vQbixx9WAp3otSETMoPUvcV5RcK68jVhyeSAGhsAeOYrtdmXp4iqsi8vy X-Received: by 2002:a17:906:1112:b0:7ad:d11e:a2c2 with SMTP id h18-20020a170906111200b007add11ea2c2mr1407937eja.473.1668013044971; Wed, 09 Nov 2022 08:57:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013044; cv=none; d=google.com; s=arc-20160816; b=f6hXa71uNUuh15FyuflXGTDBWgZchMgcUfT2jFCG6CDmhsFp/DuwthItrNtqai6UQr aOLGu3nZs70/sDUerhRHDI8s5Q7zVxqusRXK4imNaD2uR1R8auQbLpEZANAFpbDBCT6V W/NKy6J0Pe9kwQlAm8TS9UxGggIz/YPpUGFEU5kNj2mV9K6y9+joNGnOqBeWnwtVCzCT 6wuDzmyaLKM4fxMBMbU/emBtrV7lghGFLcPaHmkOhgo10HE5+d5VY4OInWKKJcBA8ApA 7s/Lz1JuqgqbSlq2DAFl9zZYJIs9eL5+M786/GjzjjxuFXg5A9Oeb5LnJt4lsOoatD1A H8yQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/gQxZVfxZ0XVnQZf+76lcdkLcCWBm7/MmNe6nzWlntI=; b=Q/3PB+ys+9z9IGCDDIM5wmw3kPvqg66Opz/3rgnVB1wSczq4ANTD4UOEMruIi6kChy g2T7eb8cUZoK1Niad52NsYUGUNidnB5Ra1l+HIPAX6p9bkHhq41WbjhUsu9AaUNUH9yc 6hESVKaPWH4k6m//hxOLbczuFWSz6SSTzONaTYCHdUHayfTQRkRjQ037V/mzcyqMDl4u +Qxfl0cdtRvog+phN3FKIRY3IQpzTRgImAms2YklcxV/yesWu7z6KyqYuCVCxdMWLlY6 XBzGN75SDgTz0euj0CWP4/wewRskdAWGp5OnaRlWFgas6YvJuOw+oa4xKCmlxHJU8wGK SGsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QpTntwL+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w26-20020a170906b19a00b007a4feae7ae7si12396214ejy.575.2022.11.09.08.57.01; Wed, 09 Nov 2022 08:57:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QpTntwL+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230366AbiKIQwK (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229835AbiKIQwJ (ORCPT ); Wed, 9 Nov 2022 11:52:09 -0500 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74D7EF5AA for ; Wed, 9 Nov 2022 08:52:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012728; x=1699548728; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YRI2YivWJDaeR3A5b46DIJCCsip9GmvTxTmH9XsxWxQ=; b=QpTntwL+FVQNTknom4ZIYddUTiI+tOIcetkLbYYUZP8o1WhDjKJ6EkHl 5olebQlHyFG8z+P33xcJGBZPzNkUsaS5xSrxK8G/dKRqQ4auSDWaPlWB4 anPB5wUNJYXC3zy0Qlhj7avVVjhxmNU3RZLMi9CxS2olm+NGwcVZCEjJg K4j5uExCozC1hMZc3g+plmpcJtOvKLx/0o6vCGco8oGDqoBvGgQVYzU9P Vtq1Y3yhN2LvHRrU4vk1QuugtnxOYbsMFBykbjPjnRLVEWL2cD6My2e4Y 2/a4c5J3i4mllwrby5yK10iORBaI5lQ6B6+r9H+7bDVMTs9oOwGhqgVD2 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="294405263" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="294405263" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:07 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="700426422" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="700426422" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:03 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6269610946C; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 01/16] x86/mm: Fix CR3_ADDR_MASK Date: Wed, 9 Nov 2022 19:51:25 +0300 Message-Id: <20221109165140.9137-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038446708117314?= X-GMAIL-MSGID: =?utf-8?q?1749038446708117314?= The mask must not include bits above physical address mask. These bits are reserved and can be used for other things. Bits 61 and 62 are used for Linear Address Masking. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/processor-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index 02c2cbda4a74..a7f3d9100adb 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -35,7 +35,7 @@ */ #ifdef CONFIG_X86_64 /* Mask off the address space ID and SME encryption bits. */ -#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull) +#define CR3_ADDR_MASK __sme_clr(PHYSICAL_PAGE_MASK) #define CR3_PCID_MASK 0xFFFull #define CR3_NOFLUSH BIT_ULL(63) From patchwork Wed Nov 9 16:51:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17693 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447303wru; Wed, 9 Nov 2022 08:57:22 -0800 (PST) X-Google-Smtp-Source: AMsMyM6ZbCpyiJuwIs0k4g2dBFPwE/sR5w/tFiFh9kP0lyiMENGUDbzEbwjxrPK/HqxKu/m6qJzU X-Received: by 2002:a17:907:75fc:b0:7aa:e0b3:270 with SMTP id jz28-20020a17090775fc00b007aae0b30270mr55715482ejc.680.1668013042243; Wed, 09 Nov 2022 08:57:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013042; cv=none; d=google.com; s=arc-20160816; b=KWHMLQ3pqCJej2Kfr3aCVnPsBkd/Ubm8kgZh1hzYewXx8uljSyh8dsbROJotAgzEzK bolkatIOUTfMdzPjOpiegZNeOF5gIO14v6IUXr+XvEpI8RpiknaLZGaakC88NygEbkor t9lvhXk9N+Njsdct4gpDumu3GOobkaTETDCm2bkfq4wCiQQjBdf91QCPJvnX996uGF+/ y1tiVrtbFcnJjXpbTsl3Fkso3yyoOrzFUfZvyZU9PCkDnTebP1UWmWL6q7jEbcmfnJCc GK1QLWsfq1+7eKdDJrHIjXa+Z2v7vG9lk1Jzo1XLAvMusg2m2Uj8m8qmR4U0BjQ27Qrv lu0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ua/d+KkLZv0PaTb7Hq4TesrbnvHM12E/lwcQfI1JPRY=; b=V8PZ2KUKl2TjXwfX9mHXZC2XDpmYwDI2ituW8o7Iv2d3jkQU/k2n2rLVRmylGZu1au SyGg9+3vnKD/ugzJfg4M2XlXvwa626PmLwIbQxyb2uTlme2mzlRckE1qPimSpiKnkGlu veG+KiEoQ76E4eASNPsoZeGpSjw3MMzf9uQhxZh4WEy1BOLyTTT8BwNArKmdzkaUWlWe NplMe1sp07Q9msdJKwxqhjNlCRXHwfay/CKdV2M/LJ0kNWSVGp9nwsSRDBXJLiaaURGk UvhMnycQEYpYAeWw0ovbwB+8zckpCgRa+2Gdo9occ3sjZOY8XEZy6d/ou7JR30w/jQzH bHiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Bq1/t/8K"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jv10-20020a170907768a00b007ae286f26b1si11477473ejc.620.2022.11.09.08.56.58; Wed, 09 Nov 2022 08:57:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Bq1/t/8K"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231277AbiKIQwN (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230146AbiKIQwK (ORCPT ); Wed, 9 Nov 2022 11:52:10 -0500 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68B161116C for ; Wed, 9 Nov 2022 08:52:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012729; x=1699548729; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7qeZVouD+LQqNvYdTdPAJZv4Un9wdAzppe5kV3GbELE=; b=Bq1/t/8K4e8jOpLMqy0ba0hZ4c8OW+QLTeg80JNpE9fmsO+eHOP8+8UZ ri5ln/IY3aKOEmCogI1lAjMoeBuKq5cAsxPFODL9MdawxIcYYD76R7IF+ P8ivRCm/YiXeOGO2un/V5GRrtnGB/Ezw536dBMX68yyTU9yfbsThWpN/I 1eOJUuPfSaMKOXX5r5SuBSwa+8br/gqnNdUj07oxDfGSovIwUKwNsu1NG ABeOIr1hqbgEDMTboFIVxSrScZX9cu9uawwKyqJCa9lr7FQznTySAOyZB K6V7BA5Z30C4YD6OKw2RKXwhXfswD1/7xF3cfmFr+FPtH8m+SIzdeuMaR Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="294405264" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="294405264" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:07 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="700426421" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="700426421" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:03 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6D74B10946E; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 02/16] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Wed, 9 Nov 2022 19:51:26 +0300 Message-Id: <20221109165140.9137-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038443962084025?= X-GMAIL-MSGID: =?utf-8?q?1749038443962084025?= Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 3 files changed, 9 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index b71f4f2ecdd5..265805e71806 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,6 +308,7 @@ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index a7f3d9100adb..d8cccadc83a6 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -28,6 +28,8 @@ * On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * + * On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. + * * All of the remaining bits indicate the physical address of the top-level * paging structure. * diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Wed Nov 9 16:51:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17696 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447372wru; Wed, 9 Nov 2022 08:57:32 -0800 (PST) X-Google-Smtp-Source: AMsMyM5ZPGuyvV5u5GbqdHBnqc24CtDlWnlZVTwcp65+aVdDg8OTwHMo/10d0ZPxYI+8wotgSvZN X-Received: by 2002:a05:6402:5296:b0:461:b6e5:ea63 with SMTP id en22-20020a056402529600b00461b6e5ea63mr59987219edb.248.1668013052570; Wed, 09 Nov 2022 08:57:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013052; cv=none; d=google.com; s=arc-20160816; b=nxuPqoX4V5pyOoJIumON6HcbboirpV832N3cQIdXTLKsiXtluAWue2Ni0pRNExfnJb Mgvy02ini+vIVp0wYETfw0mdz1Uj/X8WSiSY5gK4YIJrQW7YkUmVnHlbI6U+FdFpbkKT W3jJv0IIzy3Qb4bSM/outLP2UPYZ2iUV8yLFOZeLbYdSh6SAYwRGlSIcOm4Len1YuKak 1Y4KjHOkA1gpEncGMJk4TaxKwcW1UgopVJjH0Mz1Y2iDjWYqzuxUxJGwkGCIQOcs+kLn 1A3mO94ACIE+qrEi+vfinbYQvRsOauG3gl0HkaXJKi0v1MObtF9abllEKFdtsABmZ0iN ySiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Vf0I+ghm4a9nuJZflBRcmjC8DD3w9938QfO4oudLTck=; b=PmFMHiHpLb3HvEfp0no5qXenyJkYjkkJpB0jaZXx6UfZuhTuEdpoZ0aoCBkH8T+Lcd aOV8UF4v/St1HQJnUstE8XCXoZf6bq40bEI9WOfVY5krZA62Z2i+AX3rosd2OHynKFWZ 0uQKcL2vHjv8jNr9loLL9/riMRcEJBxDWFTjQInrcv7FaDj2OJZuLjIzuPbbBj9aiXEY xEtiywiUOjCE4xwoBkVPvx4D6/Ah6Zo7J//ZuS4lheyjDhb8UDe5UsD1KIZ4EYP80aXI vl1sLkUN5Sa7HkrGJQ8UuE3/jwAS087sUTRIrDLMHgkzdV3ZJ8DemB2FLzgCIkEkPart LKzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="F/Dd995l"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dp21-20020a170906c15500b007830e41ed56si17789514ejc.431.2022.11.09.08.57.09; Wed, 09 Nov 2022 08:57:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="F/Dd995l"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229470AbiKIQwZ (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229835AbiKIQwM (ORCPT ); Wed, 9 Nov 2022 11:52:12 -0500 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 247AB1B78D for ; Wed, 9 Nov 2022 08:52:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012730; x=1699548730; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P8gMvL5O01u2pAUkWY9sWozOyXZzWwP0vO43zDps+Zk=; b=F/Dd995lCnvgY14qan+EPxooUHPFiTIsYPWTv6sR1B0Z3WJ7yPbaWw26 nWO0k/aRgwo9vkQ7OZfeJrEbg6nYUa+VyA8iGQx5Scig7tmpmmlNYwmuc wFc0lKuJogOiu6v9BrHEcs3zMnFbeZpNtdYvZj6gbutnkTA7Y4DTKP1ZR th1JF2Ie+GeCK76zRmD5i9NJR6fvi1O5SKdUtu74A5Ar1abU2Y9pvkviV 1/RsgiQv9nGPy5OrDNo0OpWrU7XPQxeFrJ9lvnZabxo0c78bI1aQK4/jl PtsCB3S5tG0rwo26LTx6IsSCmuAwCKvwG2IvhswFyapmwdGaw6UkDF6+/ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="294405269" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="294405269" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:08 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="700426425" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="700426425" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:03 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 77EE610946F; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 03/16] mm: Pass down mm_struct to untagged_addr() Date: Wed, 9 Nov 2022 19:51:27 +0300 Message-Id: <20221109165140.9137-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038454487957792?= X-GMAIL-MSGID: =?utf-8?q?1749038454487957792?= Intel Linear Address Masking (LAM) brings per-mm untagging rules. Pass down mm_struct to the untagging helper. It will help to apply untagging policy correctly. In most cases, current->mm is the one to use, but there are some exceptions, such as get_user_page_remote(). Move dummy implementation of untagged_addr() from to . can override the implementation. Moving the dummy header outside helps to avoid header hell if you need to defer mm_struct within the helper. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/include/asm/signal.h | 2 +- arch/arm64/include/asm/uaccess.h | 2 +- arch/arm64/kernel/hw_breakpoint.c | 2 +- arch/arm64/kernel/traps.c | 4 ++-- arch/arm64/mm/fault.c | 10 +++++----- arch/sparc/include/asm/pgtable_64.h | 2 +- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/radeon/radeon_gem.c | 2 +- drivers/infiniband/hw/mlx4/mr.c | 2 +- drivers/media/common/videobuf2/frame_vector.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- drivers/staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 15 +++++++++++++++ lib/strncpy_from_user.c | 2 +- lib/strnlen_user.c | 2 +- mm/gup.c | 6 +++--- mm/madvise.c | 2 +- mm/mempolicy.c | 6 +++--- mm/migrate.c | 2 +- mm/mincore.c | 2 +- mm/mlock.c | 4 ++-- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/msync.c | 2 +- virt/kvm/kvm_main.c | 2 +- 33 files changed, 58 insertions(+), 52 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 9dd08cd339c3..5b24ef93c6b9 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -227,8 +227,8 @@ static inline unsigned long kaslr_offset(void) #define __untagged_addr(addr) \ ((__force __typeof__(addr))sign_extend64((__force u64)(addr), 55)) -#define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ __addr &= __untagged_addr(__addr); \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/arm64/include/asm/signal.h b/arch/arm64/include/asm/signal.h index ef449f5f4ba8..0899c355c398 100644 --- a/arch/arm64/include/asm/signal.h +++ b/arch/arm64/include/asm/signal.h @@ -18,7 +18,7 @@ static inline void __user *arch_untagged_si_addr(void __user *addr, if (sig == SIGTRAP && si_code == TRAP_BRKPT) return addr; - return untagged_addr(addr); + return untagged_addr(current->mm, addr); } #define arch_untagged_si_addr arch_untagged_si_addr diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 5c7b2f9d5913..122d894a4136 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -44,7 +44,7 @@ static inline int access_ok(const void __user *addr, unsigned long size) */ if (IS_ENABLED(CONFIG_ARM64_TAGGED_ADDR_ABI) && (current->flags & PF_KTHREAD || test_thread_flag(TIF_TAGGED_ADDR))) - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); return likely(__access_ok(addr, size)); } diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c index b29a311bb055..d637cee7b771 100644 --- a/arch/arm64/kernel/hw_breakpoint.c +++ b/arch/arm64/kernel/hw_breakpoint.c @@ -715,7 +715,7 @@ static u64 get_distance_from_watchpoint(unsigned long addr, u64 val, u64 wp_low, wp_high; u32 lens, lene; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); lens = __ffs(ctrl->len); lene = __fls(ctrl->len); diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 23d281ed7621..f40f3885b674 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -477,7 +477,7 @@ void arm64_notify_segfault(unsigned long addr) int code; mmap_read_lock(current->mm); - if (find_vma(current->mm, untagged_addr(addr)) == NULL) + if (find_vma(current->mm, untagged_addr(current->mm, addr)) == NULL) code = SEGV_MAPERR; else code = SEGV_ACCERR; @@ -551,7 +551,7 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs) int ret = 0; tagged_address = pt_regs_read_reg(regs, rt); - address = untagged_addr(tagged_address); + address = untagged_addr(current->mm, tagged_address); switch (crm) { case ESR_ELx_SYS64_ISS_CRM_DC_CVAU: /* DC CVAU, gets promoted */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 5b391490e045..b8799e9c7e1b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -454,7 +454,7 @@ static void set_thread_esr(unsigned long address, unsigned long esr) static void do_bad_area(unsigned long far, unsigned long esr, struct pt_regs *regs) { - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); /* * If we are in kernel mode at this point, we have no context to @@ -524,7 +524,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, vm_fault_t fault; unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(mm, far); if (kprobe_page_fault(regs, esr)) return 0; @@ -679,7 +679,7 @@ static int __kprobes do_translation_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); if (is_ttbr0_addr(addr)) return do_page_fault(far, esr, regs); @@ -726,7 +726,7 @@ static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *regs) * UNKNOWN for synchronous external aborts. Mask them out now * so that userspace doesn't see them. */ - siaddr = untagged_addr(far); + siaddr = untagged_addr(current->mm, far); } arm64_notify_die(inf->name, regs, inf->sig, inf->code, siaddr, esr); @@ -816,7 +816,7 @@ static const struct fault_info fault_info[] = { void do_mem_abort(unsigned long far, unsigned long esr, struct pt_regs *regs) { const struct fault_info *inf = esr_to_fault_info(esr); - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); if (!inf->fn(far, esr, regs)) return; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index a779418ceba9..aa996ffe5c8c 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -1052,7 +1052,7 @@ static inline unsigned long __untagged_addr(unsigned long start) return start; } -#define untagged_addr(addr) \ +#define untagged_addr(mm, addr) \ ((__typeof__(addr))(__untagged_addr((unsigned long)(addr)))) static inline bool pte_access_permitted(pte_t pte, bool write) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 84f44f7e4111..36f87d0e03c3 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -1659,7 +1659,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( if (flags & KFD_IOC_ALLOC_MEM_FLAGS_USERPTR) { if (!offset || !*offset) return -EINVAL; - user_addr = untagged_addr(*offset); + user_addr = untagged_addr(current->mm, *offset); } else if (flags & (KFD_IOC_ALLOC_MEM_FLAGS_DOORBELL | KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP)) { bo_type = ttm_bo_type_sg; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 8ef31d687ef3..691dfb3f2c0e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -382,7 +382,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, uint32_t handle; int r; - args->addr = untagged_addr(args->addr); + args->addr = untagged_addr(current->mm, args->addr); if (offset_in_page(args->addr | args->size)) return -EINVAL; diff --git a/drivers/gpu/drm/radeon/radeon_gem.c b/drivers/gpu/drm/radeon/radeon_gem.c index 261fcbae88d7..cba2f4b19838 100644 --- a/drivers/gpu/drm/radeon/radeon_gem.c +++ b/drivers/gpu/drm/radeon/radeon_gem.c @@ -371,7 +371,7 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data, uint32_t handle; int r; - args->addr = untagged_addr(args->addr); + args->addr = untagged_addr(current->mm, args->addr); if (offset_in_page(args->addr | args->size)) return -EINVAL; diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index a40bf58bcdd3..383ac9e40dfa 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -379,7 +379,7 @@ static struct ib_umem *mlx4_get_umem_mr(struct ib_device *device, u64 start, * again */ if (!ib_access_writable(access_flags)) { - unsigned long untagged_start = untagged_addr(start); + unsigned long untagged_start = untagged_addr(current->mm, start); struct vm_area_struct *vma; mmap_read_lock(current->mm); diff --git a/drivers/media/common/videobuf2/frame_vector.c b/drivers/media/common/videobuf2/frame_vector.c index 542dde9d2609..7e62f7a2555d 100644 --- a/drivers/media/common/videobuf2/frame_vector.c +++ b/drivers/media/common/videobuf2/frame_vector.c @@ -47,7 +47,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, if (WARN_ON_ONCE(nr_frames > vec->nr_allocated)) nr_frames = vec->nr_allocated; - start = untagged_addr(start); + start = untagged_addr(mm, start); ret = pin_user_pages_fast(start, nr_frames, FOLL_FORCE | FOLL_WRITE | FOLL_LONGTERM, diff --git a/drivers/media/v4l2-core/videobuf-dma-contig.c b/drivers/media/v4l2-core/videobuf-dma-contig.c index 52312ce2ba05..a1444f8afa05 100644 --- a/drivers/media/v4l2-core/videobuf-dma-contig.c +++ b/drivers/media/v4l2-core/videobuf-dma-contig.c @@ -157,8 +157,8 @@ static void videobuf_dma_contig_user_put(struct videobuf_dma_contig_memory *mem) static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, struct videobuf_buffer *vb) { - unsigned long untagged_baddr = untagged_addr(vb->baddr); struct mm_struct *mm = current->mm; + unsigned long untagged_baddr = untagged_addr(mm, vb->baddr); struct vm_area_struct *vma; unsigned long prev_pfn, this_pfn; unsigned long pages_done, user_address; diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c index a5fd6d38d3c4..4a01bc7c62b6 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c @@ -668,7 +668,7 @@ static int alloc_user_pages(struct hmm_buffer_object *bo, { int page_nr; - userptr = untagged_addr(userptr); + userptr = untagged_addr(current->mm, userptr); /* Handle frame buffer allocated in user space */ mutex_unlock(&bo->mutex); diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c index 27295bda3e0b..5c85445f3a65 100644 --- a/drivers/tee/tee_shm.c +++ b/drivers/tee/tee_shm.c @@ -262,7 +262,7 @@ register_shm_helper(struct tee_context *ctx, unsigned long addr, shm->flags = flags; shm->ctx = ctx; shm->id = id; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); start = rounddown(addr, PAGE_SIZE); shm->offset = addr - start; shm->size = length; diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 23c24fe98c00..74b6aecea8b0 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -573,7 +573,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 8a74cdcc9af0..fa3eea895210 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1685,7 +1685,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + start_vaddr = untagged_addr(mm, svpfn << PAGE_SHIFT); /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8bbcccbc5565..bfac5a166cb8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -95,17 +95,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index afb18f198843..46680189d761 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,21 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(mm, addr) (addr) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/lib/strncpy_from_user.c b/lib/strncpy_from_user.c index 6432b8c3e431..6e1e2aa0c994 100644 --- a/lib/strncpy_from_user.c +++ b/lib/strncpy_from_user.c @@ -121,7 +121,7 @@ long strncpy_from_user(char *dst, const char __user *src, long count) return 0; max_addr = TASK_SIZE_MAX; - src_addr = (unsigned long)untagged_addr(src); + src_addr = (unsigned long)untagged_addr(current->mm, src); if (likely(src_addr < max_addr)) { unsigned long max = max_addr - src_addr; long retval; diff --git a/lib/strnlen_user.c b/lib/strnlen_user.c index feeb935a2299..abc096a68f05 100644 --- a/lib/strnlen_user.c +++ b/lib/strnlen_user.c @@ -97,7 +97,7 @@ long strnlen_user(const char __user *str, long count) return 0; max_addr = TASK_SIZE_MAX; - src_addr = (unsigned long)untagged_addr(str); + src_addr = (unsigned long)untagged_addr(current->mm, str); if (likely(src_addr < max_addr)) { unsigned long max = max_addr - src_addr; long retval; diff --git a/mm/gup.c b/mm/gup.c index ff8b22327e16..cdff87343884 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1168,7 +1168,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1342,7 +1342,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -3027,7 +3027,7 @@ static int internal_get_user_pages_fast(unsigned long start, if (!(gup_flags & FOLL_FAST_ONLY)) might_lock_read(¤t->mm->mmap_lock); - start = untagged_addr(start) & PAGE_MASK; + start = untagged_addr(current->mm, start) & PAGE_MASK; len = nr_pages << PAGE_SHIFT; if (check_add_overflow(start, len, &end)) return 0; diff --git a/mm/madvise.c b/mm/madvise.c index c7105ec6d08c..0a6cb52cd2da 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1392,7 +1392,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); + start = untagged_addr(mm, start); if (!madvise_behavior_valid(behavior)) return -EINVAL; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 61aa9aedb728..98cf4a22149a 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1472,7 +1472,7 @@ static long kernel_mbind(unsigned long start, unsigned long len, int lmode = mode; int err; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); err = sanitize_mpol_flags(&lmode, &mode_flags); if (err) return err; @@ -1496,7 +1496,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le int err = -ENOENT; VMA_ITERATOR(vmi, mm, start); - start = untagged_addr(start); + start = untagged_addr(mm, start); if (start & ~PAGE_MASK) return -EINVAL; /* @@ -1697,7 +1697,7 @@ static int kernel_get_mempolicy(int __user *policy, if (nmask != NULL && maxnode < nr_node_ids) return -EINVAL; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); err = do_get_mempolicy(&pval, &nodes, addr, flags); diff --git a/mm/migrate.c b/mm/migrate.c index dff333593a8a..57eb588bc019 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1802,7 +1802,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); + addr = (unsigned long)untagged_addr(mm, p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) diff --git a/mm/mincore.c b/mm/mincore.c index fa200c14185f..72c55bd9d184 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -236,7 +236,7 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len, unsigned long pages; unsigned char *tmp; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); /* Check the start address: needs to be page-aligned.. */ if (start & ~PAGE_MASK) diff --git a/mm/mlock.c b/mm/mlock.c index 7032f6dd0ce1..d969703c08ff 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -570,7 +570,7 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t fla unsigned long lock_limit; int error = -ENOMEM; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); if (!can_do_mlock()) return -EPERM; @@ -633,7 +633,7 @@ SYSCALL_DEFINE2(munlock, unsigned long, start, size_t, len) { int ret; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); len = PAGE_ALIGN(len + (offset_in_page(start))); start &= PAGE_MASK; diff --git a/mm/mmap.c b/mm/mmap.c index 2def55555e05..a15def6cff4e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2796,7 +2796,7 @@ EXPORT_SYMBOL(vm_munmap); SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len) { - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); return __vm_munmap(addr, len, true); } diff --git a/mm/mprotect.c b/mm/mprotect.c index f006bafe338f..de7351631e21 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -680,7 +680,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, struct mmu_gather tlb; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); - start = untagged_addr(start); + start = untagged_addr(current->mm, start); prot &= ~(PROT_GROWSDOWN|PROT_GROWSUP); if (grows == (PROT_GROWSDOWN|PROT_GROWSUP)) /* can't be both */ diff --git a/mm/mremap.c b/mm/mremap.c index e465ffe279bb..81c857281a52 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -909,7 +909,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, * * See Documentation/arm64/tagged-address-abi.rst for more information. */ - addr = untagged_addr(addr); + addr = untagged_addr(mm, addr); if (flags & ~(MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP)) return ret; diff --git a/mm/msync.c b/mm/msync.c index ac4c9bfea2e7..f941e9bb610f 100644 --- a/mm/msync.c +++ b/mm/msync.c @@ -37,7 +37,7 @@ SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags) int unmapped_error = 0; int error = -EINVAL; - start = untagged_addr(start); + start = untagged_addr(mm, start); if (flags & ~(MS_ASYNC | MS_INVALIDATE | MS_SYNC)) goto out; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 1376a47fedee..d2139906ff91 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1945,7 +1945,7 @@ int __kvm_set_memory_region(struct kvm *kvm, return -EINVAL; /* We can read the guest memory with __xxx_user() later on. */ if ((mem->userspace_addr & (PAGE_SIZE - 1)) || - (mem->userspace_addr != untagged_addr(mem->userspace_addr)) || + (mem->userspace_addr != untagged_addr(kvm->mm, mem->userspace_addr)) || !access_ok((void __user *)(unsigned long)mem->userspace_addr, mem->memory_size)) return -EINVAL; From patchwork Wed Nov 9 16:51:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17687 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446980wru; Wed, 9 Nov 2022 08:56:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf7q3iYTnIA63e7AeTkkbSGJCaUh4gbaDoEYed9uWKgsZDTN45l7HJi5nmEI9CiTPl38Iu7z X-Received: by 2002:a50:9e42:0:b0:466:94e3:5ec9 with SMTP id z60-20020a509e42000000b0046694e35ec9mr10466522ede.173.1668012997023; Wed, 09 Nov 2022 08:56:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012997; cv=none; d=google.com; s=arc-20160816; b=uL6YkiB1muAuCqirqIoeuRBq5dbptmN7Xa/60cwKBFHYqHwPoKaSK8bz5PPda0WU1u gb7jx5kC4mPDAkJyBxLgPsEZm2H7YDkoBu7C0qLiQrYgZX85i5OG36koC9R8gjN3dDtr 19DHA2gOPCj6jmpXPFInnffhXw28Sby5pESRZgxLC02ghwVW1G352AdimJhW01dIX3ol MvPMjvKcLr080SHJe5NjuXFFFk7X5FswG/cgSEhGC9aGbA0odzi1kaYLIDsli3IxPqT7 4qWK9qBpYYXrC1Jok6ygQgEehYQ0grrJxpq8tPipWKRNSu70QPACBo7ish1FmcMo2g+e 9/aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1pL2uQHSeP6a9GVKYdO2WnrTLCtnJPiuh2L2Z9APwz0=; b=De68dPl2Fl3y07BaQdwDW+Vk4pP6jBTNKpwojqJ0Do/YZ1Xsj03j2qiupZRWZ0iHbM L/SIWYgTYfY0hKXVms4dwiFEpTMm0OfI+C6Fo55Sy20pQ4q+b+lhhzfi8wCmn0Cbp7MK kd43B9qTqIzidouRKPaJfcK9wE9kkEgT+H/ZNGmg54tCGkJFLKRjmoN6Zj8uHpwI2kaD BHWlmi8IkXqaB3uSI45MeKOAnQLK4+awc7bXZlNVqh+v1JK1OY+Q/tdQYy4no/iSezz2 xbcePTPZrKc0csNmKQCsXkFS6usdd4kq2wl7qIDNaar2J6n6CwODWeS1SqyE5pWYanFk nQng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lVBKWW3P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w6-20020a056402268600b004635f094565si19293961edd.535.2022.11.09.08.56.13; Wed, 09 Nov 2022 08:56:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lVBKWW3P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231375AbiKIQwU (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231158AbiKIQwL (ORCPT ); Wed, 9 Nov 2022 11:52:11 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE0EB15807 for ; Wed, 9 Nov 2022 08:52:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012729; x=1699548729; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZQtHykR2C3fr+EFMsPuuJe4JLdfmw3cbShtMyPhrvQ8=; b=lVBKWW3PbbavtFLfp+Ycm0/7zXILX+izAOkhx8Jy0kcvYT8gI/D3xcQM SHVccZrcv/gVexPI668z0YT9rOfCvqcQbeUxTse4kaZsn+Z+VbBBxhZ8N OnzrMU4C271jkO68LRMfTm234jyjdJ/vUdjk6VpXBgvA+NPOVoKVvgfWG VBoGLAdxrMUoT/NRgjbYgge+g4c0tmih8M9Slp09ONWVcFtMzXE/CxbDu QrOl0coFxeSDUw1+enxIOy5sPb9J+TcrvYqG/M8Ie2Xw+7YeRODokTvlX GR0Jq1YRGNYAtaKs2A/cVNyIGHy6WTvGcKOidKepHpBsMXIGFm8eNlnfi A==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="291419846" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="291419846" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:09 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="639257856" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="639257856" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:03 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 7FD7F109470; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 04/16] x86/mm: Handle LAM on context switch Date: Wed, 9 Nov 2022 19:51:28 +0300 Message-Id: <20221109165140.9137-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038396277521445?= X-GMAIL-MSGID: =?utf-8?q?1749038396277521445?= Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t. switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu.h | 3 ++ arch/x86/include/asm/mmu_context.h | 24 ++++++++++++++ arch/x86/include/asm/tlbflush.h | 34 +++++++++++++++++++ arch/x86/mm/tlb.c | 53 +++++++++++++++++++++--------- 4 files changed, 98 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..002889ca8978 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -40,6 +40,9 @@ typedef struct { #ifdef CONFIG_X86_64 unsigned short flags; + + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b8d40ddeab00..58ad18cc2fac 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_X86_64 +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return READ_ONCE(mm->context.lam_cr3_mask); +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..662598dea937 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -101,6 +101,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_X86_64 + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -357,6 +367,30 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_X86_64 +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + unsigned long lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_cr3_lam_mask(unsigned long mask) +{ + this_cpu_write(cpu_tlbstate.lam, mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_cr3_lam_mask(u64 mask) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global(unsigned long cr4) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c1e31e9a85d7..9d1e7a5f141c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,26 +154,30 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { + unsigned long cr3 = __sme_pa(pgd) | lam; + if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); + cr3 |= kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); } + + return cr3; } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { - VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } /* @@ -274,15 +278,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +496,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +526,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, + tlbstate_lam_cr3_mask()))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -552,9 +559,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * instruction. */ if (real_prev == next) { + /* Not actually switching mm's */ VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); + /* + * If this races with another thread that enables lam, 'new_lam' + * might not match tlbstate_lam_cr3_mask(). + */ + /* * Even in lazy TLB mode, the CPU should stay set in the * mm_cpumask. The TLB shootdown code can figure out from @@ -622,15 +635,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_cr3_lam_mask(new_lam); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +705,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +717,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +726,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_cr3_lam_mask(0); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1071,8 +1090,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Wed Nov 9 16:51:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17697 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447507wru; Wed, 9 Nov 2022 08:57:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf7m4bxU4Wo9RRDLKMO/m1VMwNzVMGsvizEy7n8ciztRCUVJxWgLWFyYVCiFXLYTuKqDlncP X-Received: by 2002:a17:906:eec1:b0:782:6384:76be with SMTP id wu1-20020a170906eec100b00782638476bemr7484313ejb.756.1668013068392; Wed, 09 Nov 2022 08:57:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013068; cv=none; d=google.com; s=arc-20160816; b=KX4eWg4aaZXZRblD+Aa3AOh3I0LaJW9+ZmnSOOTehXzJqYAzxnEkHk0Z8yPhwRbKv2 SX20OUes4Zvw/luB9AVHm6sJzts3067cPnaIg/ApV+jfSl6unR8Lh/quROvUL6MBH2u0 NSX+OjX0K/+FcjyVFJPWy9bHubY6uMgJRRGIoPpSQzp0uBJqUw9pYUmgcLPrXs7t2v/t 8loDFgDUW0O8DV7dIUuwBd1gCj/XKRjv2uawKf5EVcpwonMl4ZPPmofCtOQvURfZSsTH u+9n531lDzZ6wydzxUwAH/SNxxeaADdMTzhUZT3In4ls4hIgMqk7HFJu4WY+zTD1bUXR vZLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CaCWx47xlYdkL4l6SbDVp2o77hz1rD4ho9/wxPOL32g=; b=RQl44peSK5ckWumx8GChK1ekazgdj4fjY+J4+gvynF7Yo9tMZGyxNhjD/vLTj1A8pV WAn/tEhFxZCAJsXCqQxEPRnLDPPphVlU3wzV+tasnPpgdlR0nf9hNTPK5rg63vvbGEWa WBaYqU1xolJ9o6PHaFr0/MlazQMXfvQzjsqhsDCvZuhaIw6L4AiMzXzv1cNbNDmQwj91 TGvCmXXQTgyOeIi/gnGg9i19r4daGmHXCfB7zXDUU17TDaeZJ4GBxtfxP3AowdIKLkN7 id/DQDeBToHfPIMfginqRcnnlzovHp5i82o0D69ec5qjDK8VlQ9dsNXXXDE9UPP2v1cv QnKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TOLyxEX2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hv11-20020a17090760cb00b007ae9f42f878si896461ejc.354.2022.11.09.08.57.19; Wed, 09 Nov 2022 08:57:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TOLyxEX2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231686AbiKIQwp (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbiKIQwe (ORCPT ); Wed, 9 Nov 2022 11:52:34 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D18D82496A for ; Wed, 9 Nov 2022 08:52:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012737; x=1699548737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=52HdGnbEU41/jn9zck3I6kDWQYrF1ZQm5uJaHCT5mF0=; b=TOLyxEX2jlwxtnzINlHoT6pY5S2ZOiJk7bu6Y+xZz8m90JvbePBBdwtN CO+a4QOQT6GzruTH/CMvxUC1ZmOnIVma92jDCOkiizGXPCPOLykrNc47J xQbipgrPCE/2GUFOqaDYgDdW5DbpFCM2y25ItKUI3ZSkC5A57cEa4xXIB FLlbsFzAt5IO5az4LPF4GOGfF4M+sDreAlUerK9I9ay1C4DP34Ucbe1/a +LArF9aejon7tKtuAzafVyBQj1Yx3b0gx0uWghcn+pSHTxUYZRhQ5dy73 iZpnS2+4IZRwwtnvOIiuAGt6FqJA6Cf8WZOhfLvRUBCv3Kk0+BJnB5alp w==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="291419870" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="291419870" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053276" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053276" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 891EC109471; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 05/16] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Wed, 9 Nov 2022 19:51:29 +0300 Message-Id: <20221109165140.9137-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038471341051004?= X-GMAIL-MSGID: =?utf-8?q?1749038471341051004?= untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) Acked-by: Andy Lutomirski --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 ++++++++ arch/x86/include/asm/uaccess.h | 42 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 +++ 4 files changed, 56 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 002889ca8978..2fdb390040b5 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 58ad18cc2fac..92cf8f51faf5 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..0db6f5451854 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,30 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= (mm)->context.untag_mask | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(mm, ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(mm, __ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) +#else +#define untagged_addr(mm, addr) (addr) +#define untagged_ptr(mm, ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -41,7 +66,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(current->mm, addr), size)); \ }) #include @@ -127,7 +152,13 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -227,7 +258,12 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c21b7347a26d..d1e83ba21130 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM From patchwork Wed Nov 9 16:51:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17681 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446594wru; Wed, 9 Nov 2022 08:55:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf7FkImTh2bXeZ94bguifzBhT4NiS51FJ2ZJfxuD3tdLOiaXFmpw6J5850zN38lKib09uTHy X-Received: by 2002:a17:906:1599:b0:7ae:9813:f930 with SMTP id k25-20020a170906159900b007ae9813f930mr2547951ejd.762.1668012944531; Wed, 09 Nov 2022 08:55:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012944; cv=none; d=google.com; s=arc-20160816; b=EPKxS4gz7zq0yGkvIPGuKMbRs7DDmH3tsNYxrkAwVldqt8k3dGnQ39UPDOxiLuy0fx TLHjioV2fqp6Y5wMDVbTC89NDlDDBu7OW0lqESrae7JhyXbysbVlxlJx9xpxR6Bg7Liv bYG9PLIv5ee4o2ga45Is4bH3+//KdEgPRCRSpcpLtXC4CZ/q3U8PvhoX7dF4A0LQBSl3 m5SzTC6O5U/VPmPuXx/KGYE0Odm1K6aurn6TQc3YkaoQFs2Xawt9lH6DIDzzhfbcIxZW 4++I9ui7ThzNhZEDclYy1mC24GVcEoJ89X84i37ilf4Mf0/p3fdDLY0PK9JoRQOUMkU6 l4Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZsMnpelDH6S8NiPofehi/MISgeT2xNjyCketcrjkirg=; b=masIMan0xaZZKTMPAITN90dBJVbzmB15Zz8ExTB6YDmolTlIz4YqTA0hY34sXmja64 1wHWLrYfeTXNorp7Ea1aMDNx1t5rHRY0tGK7ssLVxr7l5oNj2xKECtOrEB+RkMDXJtVU /hFmufriBZMPNlHpzZJ0bAf15fsMKN7Rv5lucW5MMfkp3F9TMbZnqxSt4u+QW6pJG37r bLQUNF9y6j8X5dYM3ZTTqXmiRRz/7bev5bMub7EVhvnJ8GTAv7aQ2sx7dwXxEeASt3Yg pWOMzHOXocmuc6QR56ajJ7gAmOYFc7zbt/ZMudOTTUa+Caog2Y0iGKpUv3scdmy6XS2+ hc1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TygHs3kS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cx4-20020a05640222a400b0045caa1628c6si13755007edb.216.2022.11.09.08.55.21; Wed, 09 Nov 2022 08:55:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TygHs3kS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230490AbiKIQxD (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230375AbiKIQwg (ORCPT ); Wed, 9 Nov 2022 11:52:36 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6124F25C41 for ; Wed, 9 Nov 2022 08:52:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012739; x=1699548739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PRHpYCdvQPrEk4Dh8NC0NI3aanRW/V6efiDpPBADx4Y=; b=TygHs3kSZ2Sm/FQBkGTn8pSnkhgqhPSSedePkQYDPfPFIwNUTwkzvEUj cz3veemBdi6fyzn88HG7wV0DEcvcurTyE956E+fkJjNB2v1kbD9YMx9Gz lg0JuxdLRX2LJifQL3yVfblD2w1td54BOsK6wFidZnHMqufJr90f1da/z g+S+OeZGwk9D16w1D/fcO70JDCIv6i807GtqEnHEU5R1fMabEJLW30mQn wnYZAbaLFZLAPJAeSKvuaAzx7cym+apIzT+T64H6UlNS9oRkzU0c/7BOD UZJjWI9Y65Nt2NoU6aHOZz1/b4H2LHPHCflsnLwuh0fYCyxa9CN+mggmK A==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553715" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553715" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827259" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827259" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 93A4C109472; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Marc Zyngier Subject: [PATCHv12 06/16] KVM: Serialize tagged address check against tagging enabling Date: Wed, 9 Nov 2022 19:51:30 +0300 Message-Id: <20221109165140.9137-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038341578439437?= X-GMAIL-MSGID: =?utf-8?q?1749038341578439437?= KVM forbids usage of tagged userspace addresses for memslots. It is done by checking if the address stays the same after untagging. It is works fine for ARM TBI, but it the check gets racy for LAM. TBI enabling happens per-thread, so nobody can enable tagging for the thread while the memslot gets added. LAM gets enabled per-process. If it gets enabled after the untagged_addr() check, but before access_ok() check the kernel can wrongly allow tagged userspace_addr. Use mmap lock to protect against parallel LAM enabling. Signed-off-by: Kirill A. Shutemov Reported-by: Rick Edgecombe Cc: Marc Zyngier --- virt/kvm/kvm_main.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d2139906ff91..8399aae16e83 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1943,12 +1943,22 @@ int __kvm_set_memory_region(struct kvm *kvm, return -EINVAL; if (mem->guest_phys_addr & (PAGE_SIZE - 1)) return -EINVAL; + + /* Serialize against tagging enabling */ + if (mmap_read_lock_killable(kvm->mm)) + return -EINTR; + /* We can read the guest memory with __xxx_user() later on. */ if ((mem->userspace_addr & (PAGE_SIZE - 1)) || (mem->userspace_addr != untagged_addr(kvm->mm, mem->userspace_addr)) || !access_ok((void __user *)(unsigned long)mem->userspace_addr, - mem->memory_size)) + mem->memory_size)) { + mmap_read_unlock(kvm->mm); return -EINVAL; + } + + mmap_read_unlock(kvm->mm); + if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM) return -EINVAL; if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr) From patchwork Wed Nov 9 16:51:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17700 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447506wru; Wed, 9 Nov 2022 08:57:48 -0800 (PST) X-Google-Smtp-Source: AMsMyM5Bdhl+l5HfAxD+d4dz6gA7raP1m+wSeJLO0+nzNgWJ9i7CjragKbWnq1ZUUGXff+5WNq/X X-Received: by 2002:aa7:c302:0:b0:461:85d7:b82 with SMTP id l2-20020aa7c302000000b0046185d70b82mr60680431edq.354.1668013068349; Wed, 09 Nov 2022 08:57:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013068; cv=none; d=google.com; s=arc-20160816; b=AwlasMok7NkpfNhN4qPmXxouUZljQJ9A5SBFa/YZn5pIiuZ5OmpBy+MztD5mwIU2n3 gRGyy9KRMkKbCu3su4gL0izX++OcoOB07NWmf+AqMLrzPb2qC75q7XdtnHVTO0Mj5j+P qUQF6CV/2MACT6bKF/8S2pq0zFnAI/1pPaS1YB485HHWzvPJceXQybYPjTAl7SYipIGN 1GE6q9E1EkptHEDLWVf5n8BUE01MnoFKXUd+89g9of+mk86ZYkZ965nWyrVYbfpAagdO qKLj53eNnUbK4X4SjyJvrcfrPosUpTj5z6Z1gIjDZWdgTrkSEIneFySN5IZ47XaRfQPy OzHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/AHPNiprwO0cZDCn9T6ZmQ2fOpYm/X1SoaXHRjW9+Sk=; b=pecGPqyZjfFdeETpwD4EGWah2v/fRlMjq4mheJ6gSEEY/SoS6JNDorIK7vmPjXp9HG ZKTc6Ud0RY4+AozR10wgcVrGJ9auHyY8oLM6vD0GmOcB6eRvCj/ajyLZCdjH6MnRFOAz V1ECChtOuIbQYYCrs+LEvsKT9Ho4lJzdul9VmqbN+iCQp7K1X3N5ajZbSuxBZ/2ccf0K r7675HCO2rt1A9WHyQPTTQzxDMWP6LhOD8mlb4YYA4IkN7haUJh1629OBZEdLQv7wK7B aSTTrR8sianF5fLyjkB3TX0tpsMbLxCvw/J3+goO/1/qqkLNfY1h3EOQAw3vD704FMAH Bwjw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TwHuNu+n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hs34-20020a1709073ea200b007879e9a7451si18561740ejc.816.2022.11.09.08.57.18; Wed, 09 Nov 2022 08:57:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TwHuNu+n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231416AbiKIQwj (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231278AbiKIQwe (ORCPT ); Wed, 9 Nov 2022 11:52:34 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD16F2316D for ; Wed, 9 Nov 2022 08:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012736; x=1699548736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cWyRCW5aKXNWtiucVf5kkixxGLClq+OtEfuUo4LHoho=; b=TwHuNu+nx9tS/f27EK95wtoviGWV1bRv+KDt1AW6HrExNtL0vXUSn8AT LuwxqSchQLDFF+WaZNxEnBsQefToW0k/T6C2EM9MriUSbHgHdPNC/Ufcw wav7qAM7llBsQrtQqsIH+tgE4O0/tt6MdlxSfMHH5zwbcLbBC314IwzTn qhE5dX7K6q9ZGgmHH//uD/vb4QU4NmKAgBJuzp5edtbJ19GVCQ9F5vk2V yx/FXOxKgLCtj1Is1fo0bpofSJ4+Io8WBlHLVXXRBO/TIum99RXdCjbZG 0AAxz+8K9iKO7sjdgZUafqZSey46onlEROVTABCfxKUvtHvIDYtlT2bIC g==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="291419868" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="291419868" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053277" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053277" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 9E70D109473; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 07/16] x86/mm: Provide arch_prctl() interface for LAM Date: Wed, 9 Nov 2022 19:51:31 +0300 Message-Id: <20221109165140.9137-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038471594893602?= X-GMAIL-MSGID: =?utf-8?q?1749038471594893602?= Add a couple of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. - ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko Reviewed-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/uapi/asm/prctl.h | 4 ++ arch/x86/kernel/process_64.c | 71 ++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..a31e27b95b19 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,8 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 6b3418bff326..b8f2558a3aeb 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,66 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +static void enable_lam_func(void *mm) +{ + struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); + unsigned long lam_mask; + unsigned long cr3; + + if (loaded_mm != mm) + return; + + lam_mask = READ_ONCE(loaded_mm->context.lam_cr3_mask); + + /* + * Update CR3 to get LAM active on the CPU. + * + * This might not actually need to update CR3 if a context switch + * happened between updating 'lam_cr3_mask' and running this IPI + * handler. Update it unconditionally for simplicity. + */ + cr3 = __read_cr3(); + cr3 &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57); + cr3 |= lam_mask; + write_cr3(cr3); + set_tlbstate_cr3_lam_mask(lam_mask); +} + +#define LAM_U57_BITS 6 + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + int ret = 0; + + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + /* Already enabled? */ + if (mm->context.lam_cr3_mask) { + ret = -EBUSY; + goto out; + } + + if (!nr_bits) { + ret = -EINVAL; + goto out; + } else if (nr_bits <= LAM_U57_BITS) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + ret = -EINVAL; + goto out; + } + + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); +out: + mmap_write_unlock(mm); + return ret; +} + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -830,7 +890,16 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_GET_MAX_TAG_BITS: + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return put_user(0, (unsigned long __user *)arg2); + else + return put_user(LAM_U57_BITS, (unsigned long __user *)arg2); default: ret = -EINVAL; break; From patchwork Wed Nov 9 16:51:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446589wru; Wed, 9 Nov 2022 08:55:44 -0800 (PST) X-Google-Smtp-Source: AMsMyM5yEywMKqwUqbcNViOKoQg8X6CGT0chyVp6+0LKG5BsQ2lZ2WQR+xakypWWj1sL/00NCKpU X-Received: by 2002:a17:907:2c42:b0:7ad:d798:f2d5 with SMTP id hf2-20020a1709072c4200b007add798f2d5mr48494266ejc.386.1668012944140; Wed, 09 Nov 2022 08:55:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012944; cv=none; d=google.com; s=arc-20160816; b=QoeMroGG2kTU0jWgTLo0+4f07C+8R/XllkpJRMGGLsAhk3JldI777R/adu78Goushr +PbQYIZSF/yluCNArjcfqFx2hxoxE4wUlKBGQ3PuQ/4mmusKnaga1V/9YruUuanXZfwW RvhX+j0pfFknINdbE40++Tv794UL9UeGJVltxroYMdwXJkvANHqH9eSrZp/5GI6aMfBq L0itsRq3/GU1v+bksL6GYLSpnIQM8i7cJIjZPF8BzfInN5CamBSeJ6r7g5A282+BL8Vy peKhE4D7VGLe+zql6EO0aAB6+d6LuRZf3T0pttrhBnyDzBivwwnAmt6ngcZkZJ0Z+g93 RRkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kTxhWdg/9+KWVLyknAx/c07RpH6UpatcZ0+k9lSUfcM=; b=DAueIEWcPaLAkrPFJNtgta99Q461jJcD4qeEtENdeusilae/mj6MJdmZVzsY9mQoyR eiOQSn7JE+0xLln5ER70jH23eKBbi3zttj2REC//SvL1dfezK1K6Zx4WvVyf9DulGecf inDc3VpLFuj/SvCzdXODnn5yzTIgDEVZF7U7AhQWdPJWsolndj+qG7z0N8ccGO2N7DjZ fRTn0PB0BWorKjPO7lB+jQVYbI29/3xAJ9bYHuqjnzvh+yuOfPEmlEfkx1RZPCw+kmfA u+bnMnZAGLRAV+G73L1+QGkcobWGejgyc9LrklbRtTaOMeeQQj5vLm/ykaEFYoqpk9Ip zAew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SaLl3i7T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id go37-20020a1709070da500b0079fe11e92ddsi17869227ejc.906.2022.11.09.08.55.20; Wed, 09 Nov 2022 08:55:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SaLl3i7T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230216AbiKIQwy (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230240AbiKIQwf (ORCPT ); Wed, 9 Nov 2022 11:52:35 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDF0A240A8 for ; Wed, 9 Nov 2022 08:52:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012737; x=1699548737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qQuq8QNGbNiZCBzAMV30JlXBLCHT9H1+EgK3GfrClpQ=; b=SaLl3i7TQAcy6FwgC5q8VE/gJfLmI8zS+1hzKKei6JPArTdZffD2hn5T Ncmj4HZ82vlQrszK0Zhf6HUfNh9phF4MT/6JDZigGj4Yx0VoJ9wwHhRrP WtPkK+y1JbIKV6gyMarq9N41wu13o9UETnLc9uwnE30A3l0wfa1lf41xT Cs6DuD5N9e45c7FNw1WFwE5SnHF9KHu4eTS1bLpQWA6rbDLCrKbRKIXBf l0loWXcSD8rQgA2LwwEql17RXe30OHtW3pP7J/v1Lu5HYM6NnXhfKzo4e OYphM4sNJvH8OidbPRwNEsimPM4yi6z95vbuNvXGXtyo0c598Ck2zgJGR w==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553708" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553708" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827255" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827255" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id A93E0109474; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 08/16] x86/mm: Reduce untagged_addr() overhead until the first LAM user Date: Wed, 9 Nov 2022 19:51:32 +0300 Message-Id: <20221109165140.9137-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038341026111659?= X-GMAIL-MSGID: =?utf-8?q?1749038341026111659?= Use static key to reduce untagged_addr() overhead. The key only gets enabled when the first process enables LAM. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/uaccess.h | 8 ++++++-- arch/x86/kernel/process_64.c | 4 ++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 0db6f5451854..1d2c79246681 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -23,6 +23,8 @@ static inline bool pagefault_disabled(void); #endif #ifdef CONFIG_X86_64 +DECLARE_STATIC_KEY_FALSE(tagged_addr_key); + /* * Mask out tag bits from the address. * @@ -31,8 +33,10 @@ static inline bool pagefault_disabled(void); */ #define untagged_addr(mm, addr) ({ \ u64 __addr = (__force u64)(addr); \ - s64 sign = (s64)__addr >> 63; \ - __addr &= (mm)->context.untag_mask | sign; \ + if (static_branch_likely(&tagged_addr_key)) { \ + s64 sign = (s64)__addr >> 63; \ + __addr &= (mm)->context.untag_mask | sign; \ + } \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index b8f2558a3aeb..457f79be8bc7 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,9 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +DEFINE_STATIC_KEY_FALSE(tagged_addr_key); +EXPORT_SYMBOL_GPL(tagged_addr_key); + static void enable_lam_func(void *mm) { struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); @@ -798,6 +801,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) } on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); + static_branch_enable(&tagged_addr_key); out: mmap_write_unlock(mm); return ret; From patchwork Wed Nov 9 16:51:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17682 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446608wru; Wed, 9 Nov 2022 08:55:46 -0800 (PST) X-Google-Smtp-Source: AMsMyM5/OXzOxH8qg8xUlJb0C7IpD534e8oQ5xjVxF/cm+XKAPwA2xqO5BYvtQcSoWHAXPieKMxv X-Received: by 2002:a17:906:8475:b0:7ad:c04:d3cf with SMTP id hx21-20020a170906847500b007ad0c04d3cfmr55851933ejc.463.1668012945823; Wed, 09 Nov 2022 08:55:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012945; cv=none; d=google.com; s=arc-20160816; b=u1Dv0EyJLCxw/dyMitnlCsC4nt6UnlpYZCsxOxKjIpD/XzPQbvphwNbyTZXE4p7REo 7bNoC5gx4Dxd1uMC4ItkU4qVi5EMEUX1ue3Zk9ApM4KQ9Y1/XAed2UiDQzMFSj6ErT8G 2o4PtIMBhMwgU8T413mf/TuMI4Tin46qYk7ROh08UsxxY1tQUvNl0R3i/danfeDR4CdP aE2vb5OIHz1+cXdlcizQ6ZI2PwanSq/tnIL72xH6eZ7cupeIA0NHQucjGEM7EwtCoN0+ RA1BqR2+PIZ27EKhYTZWLOTByux/i3aTl//m41WKWV4Ll8zFu3sEsqg66nSz+IpdB18H lqWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0y1kPzN8euHz5iTm3VMzL48rX8M92qtUuei7hugn14M=; b=XIOHKaGx/7DZXAr7Iczcjc9wQe0vWrIbwSe+Q4FMtVNX4NE6t2xODIZO8UprZ5LuOA X8oMGL1uos5JxdoI6dUee6ueIRQVulmty9h/B2/Tvwohy+ePlnLMTVhfoLM0rDsN+HiE e22tI8erlRAJ8OJUrXQcQDO9ctvcuJ15dTdhs1v11PJe/u2YIWKhpmNf7iNl/6uZGe6m OQeCC10qXAcQ2xmSvl2k28wpzu0R1Xh/rRpOv+o7zCppIG1GlPG2ITfYCj6L3WMn1BAw utb9O0vLTZyA/WC4XuO1FnPMHqR7QVuSuJdwF+wmWEKjOHL3wuFnbdtvniHSWgzvG0CS 4GJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="EhHKS/TG"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l18-20020a1709062a9200b0077ea290986fsi11397815eje.584.2022.11.09.08.55.22; Wed, 09 Nov 2022 08:55:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="EhHKS/TG"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229872AbiKIQxG (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230388AbiKIQwg (ORCPT ); Wed, 9 Nov 2022 11:52:36 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77C6125E8C for ; Wed, 9 Nov 2022 08:52:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012740; x=1699548740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4KmDSGMvHF7ZrGPOcEVvMUSJNn2iMbIpR8ULBIekJIg=; b=EhHKS/TGcc/denCM5GU2GqQ+iVHXFOOSIxbEw2GDzhykw1+vJY6qb33T 3VdKYyKGEM6KzU82cyJ0h1ssFdB7CotGviKhMA4hoHtwhyjq4/kd+HLDk nAyMQWhRFe2ddxKKWc+kxwa9A4nwsUf2arJpNMIvFRwnmHpBgRi2fghBX yMiWio/tMphzs6c7+xiU3hqtyF5XmaKKsLPpJTuvW8sG/PRDmBFnJkYis dVFjStvEHeSCKNFXG1Tf7zHTYEAbDU0BjD+TZ2FI+JyztIqhG45YoEjpL QgFWcgrypWhDlHWVcch5Yt0+ihp4262ZbCQSrqPmFElAa7ZtGjGA5/3Ft Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553727" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553727" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:17 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827263" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827263" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id B46D6109475; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , "David S. Miller" , Catalin Marinas , Will Deacon Subject: [PATCHv12 09/16] mm: Expose untagging mask in /proc/$PID/status Date: Wed, 9 Nov 2022 19:51:33 +0300 Message-Id: <20221109165140.9137-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038342493405392?= X-GMAIL-MSGID: =?utf-8?q?1749038342493405392?= Add a line in /proc/$PID/status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Cc: "David S. Miller" Cc: Catalin Marinas Cc: Will Deacon Acked-by: Catalin Marinas --- arch/arm64/include/asm/mmu_context.h | 6 ++++++ arch/sparc/include/asm/mmu_context_64.h | 6 ++++++ arch/x86/include/asm/mmu_context.h | 6 ++++++ fs/proc/array.c | 6 ++++++ include/linux/mmu_context.h | 7 +++++++ 5 files changed, 31 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index d3f8b5df0c1f..81ac12d69795 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -278,6 +278,12 @@ void post_ttbr_update_workaround(void); unsigned long arm64_mm_context_get(struct mm_struct *mm); void arm64_mm_context_put(struct mm_struct *mm); +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> 8; +} + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h index 7a8380c63aab..799e797c5cdd 100644 --- a/arch/sparc/include/asm/mmu_context_64.h +++ b/arch/sparc/include/asm/mmu_context_64.h @@ -185,6 +185,12 @@ static inline void finish_arch_post_lock_switch(void) } } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> adi_nbits(); +} + #include #endif /* !(__ASSEMBLY__) */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 92cf8f51faf5..91fdbb47b0ef 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -103,6 +103,12 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..d2a94eafe9a3 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +static inline void task_untag_mask(struct seq_file *m, struct mm_struct *mm) +{ + seq_printf(m, "untag_mask:\t%#lx\n", mm_untag_mask(mm)); +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -443,6 +448,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_mem(m, mm); task_core_dumping(m, task); task_thp_status(m, mm); + task_untag_mask(m, mm); mmput(mm); } task_sig(m, task); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index b9b970f7ab45..14b9c1fa05c4 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -28,4 +28,11 @@ static inline void leave_mm(int cpu) { } # define task_cpu_possible(cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) #endif +#ifndef mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} +#endif + #endif From patchwork Wed Nov 9 16:51:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17699 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447508wru; Wed, 9 Nov 2022 08:57:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf4qFdtk0XVBbCMi2wqz1hfqvLpRBIRekEBqXm7ay5b2yOYTYnqGaW69vXfCxYoLlTU6tryh X-Received: by 2002:a17:907:2179:b0:7ae:6bcd:d0cf with SMTP id rl25-20020a170907217900b007ae6bcdd0cfmr13275730ejb.428.1668013068379; Wed, 09 Nov 2022 08:57:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013068; cv=none; d=google.com; s=arc-20160816; b=RhdyUKUJkrplQRSyxMoeG7yb0E9ZJlnu8kf+MeItTFU5S45ZfZAyppefQ12jfp9X/n YJaPvf8SInX5f1DQy76hJDLspt2pEr+GWZKdHBe4KTG+IriY7/28MhDkeGaGQObGI/gc c+099YASLiyK38a2PzfW4Y+X7xsedOSE6L/Cup+Enaz3o8kT44QLCD/7lJyHNVxu+I5q 5MNcTicbCRMTnbOka0KGQJg5+Dzqc7NcxsjtUTsdNpBb6OOOPtemmjbPJZ9GLBORO3Ag MSwdHfmiZVBmsO0UTMHaYRWncUGnqeUjkIneRTMwXTmvormfoJQzWtrfZmq7cS2wlAtk U5NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v6wuQwTfMALR6lvNsecDGxi6bxXpGHSsmKWj6R9xe/Y=; b=e5QWHNV1Jdy7KIMlfbKWaUG6dG0WyllolxRnZ9FzTTmVM/e8K7yzkrUHdrAals9QL1 eNQ2WVzCmZxVBPFsZVMwWjbiQcEi0+aRNxkZSJ5fPA6ZG2r7RO2A1ooTZn9i5UoK99RL M47vRMUNSamQ2NKJC9Bv4VzjWHVA1J/XOLlKUKNo/oS0JA9l/AEAtnLxD/7QIW1CDxtq jbHz/GrY+hfcUxoKMNeN/Tov/lY/Qt0CNn1cs4Ab8dPxMFySAmqaqPHR+nj7CID10Ug/ R+wKwnkDqcDVED78b5S1NJppUdnwLyrVgoQJ3dUmLfk0EqTBmfknXwUnx70JEub/Zc+C isLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QE7gBA6K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c24-20020a170906171800b0078dc3cb8b48si11594920eje.625.2022.11.09.08.57.25; Wed, 09 Nov 2022 08:57:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QE7gBA6K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231387AbiKIQwv (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230146AbiKIQwf (ORCPT ); Wed, 9 Nov 2022 11:52:35 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E62082497B for ; Wed, 9 Nov 2022 08:52:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012737; x=1699548737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xa2QDdupIWc8VqMdeXFZWdKQnRlx2ren8j3nAoeW+Vo=; b=QE7gBA6K3X0DnPMoL66EMNoxYY/MEk90HlsinTg4ztdhgAo3Smu44mQi F0VcxHgzmvcpHKaFrQxZA9o8Jhn2DtSVdnB25DJ+I8RVitaz65SNSjF58 Gbe4SoX6L+XxQ1mrMJesb9I6XrzjZFZtG7seNqF4Pry/vjGz7ztSax/9c TJ1xMRb4CxrUKbTB7TaDI+7otAicqUBZgoUWXb0iNXzUvkqKKZ7OeF1yN t1dqS9IVsSIsRD9Yzhvj3Cy8Er0O04NKPNAu9KAfqG0mMPHQULKbkIQHF f3HuPLZ2T71+tfIEaMlaKYUONwFsBQzXpf/KimCSQUNQrExgJPSbLG8Qq w==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="291419872" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="291419872" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053279" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053279" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id BE39D109476; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 10/16] iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() Date: Wed, 9 Nov 2022 19:51:34 +0300 Message-Id: <20221109165140.9137-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038471175037145?= X-GMAIL-MSGID: =?utf-8?q?1749038471175037145?= Kernel has few users of pasid_valid() and all but one checks if the process has PASID allocated. The helper takes ioasid_t as the input. Replace the helper with mm_valid_pasid() that takes mm_struct as the argument. The only call that checks PASID that is not tied to mm_struct is open-codded now. This is preparatory patch. It helps avoid ifdeffery: no need to dereference mm->pasid in generic code to check if the process has PASID. Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/traps.c | 6 +++--- drivers/iommu/iommu-sva-lib.c | 4 ++-- include/linux/ioasid.h | 9 --------- include/linux/sched/mm.h | 8 +++++++- 4 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 178015a820f0..3bdc37ae7d1c 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -664,15 +664,15 @@ static bool try_fixup_enqcmd_gp(void) if (!cpu_feature_enabled(X86_FEATURE_ENQCMD)) return false; - pasid = current->mm->pasid; - /* * If the mm has not been allocated a * PASID, the #GP can not be fixed up. */ - if (!pasid_valid(pasid)) + if (!mm_valid_pasid(current->mm)) return false; + pasid = current->mm->pasid; + /* * Did this thread already have its PASID activated? * If so, the #GP must be from something else. diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c index 106506143896..27be6b81e0b5 100644 --- a/drivers/iommu/iommu-sva-lib.c +++ b/drivers/iommu/iommu-sva-lib.c @@ -33,14 +33,14 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { if (mm->pasid < min || mm->pasid >= max) ret = -EOVERFLOW; goto out; } pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); - if (!pasid_valid(pasid)) + if (pasid == INVALID_IOASID) ret = -ENOMEM; else mm_pasid_set(mm, pasid); diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index af1c9d62e642..836ae09e92c2 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -40,10 +40,6 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); void ioasid_unregister_allocator(struct ioasid_allocator_ops *allocator); int ioasid_set_data(ioasid_t ioasid, void *data); -static inline bool pasid_valid(ioasid_t ioasid) -{ - return ioasid != INVALID_IOASID; -} #else /* !CONFIG_IOASID */ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, @@ -74,10 +70,5 @@ static inline int ioasid_set_data(ioasid_t ioasid, void *data) return -ENOTSUPP; } -static inline bool pasid_valid(ioasid_t ioasid) -{ - return false; -} - #endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..b69fe7e8c0ac 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -457,6 +457,11 @@ static inline void mm_pasid_init(struct mm_struct *mm) mm->pasid = INVALID_IOASID; } +static inline bool mm_valid_pasid(struct mm_struct *mm) +{ + return mm->pasid != INVALID_IOASID; +} + /* Associate a PASID with an mm_struct: */ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) { @@ -465,13 +470,14 @@ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) static inline void mm_pasid_drop(struct mm_struct *mm) { - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { ioasid_free(mm->pasid); mm->pasid = INVALID_IOASID; } } #else static inline void mm_pasid_init(struct mm_struct *mm) {} +static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; } static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif From patchwork Wed Nov 9 16:51:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17702 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447556wru; Wed, 9 Nov 2022 08:57:58 -0800 (PST) X-Google-Smtp-Source: AMsMyM7CQlVjIxDasCM+dAHjQG+/KRVWv9hN3zbtt8Q1EOfvyGRHdsdry/gDktAWGGRH7BsHEH+Z X-Received: by 2002:a17:907:7fa9:b0:791:a5c2:e25 with SMTP id qk41-20020a1709077fa900b00791a5c20e25mr58701860ejc.444.1668013078249; Wed, 09 Nov 2022 08:57:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013078; cv=none; d=google.com; s=arc-20160816; b=zGfcl3Yx0eYLin/BNnV3knGRStQsP99YurzHcLX9sFJpFR/DYB4nUt37YIx4f+Wlub KJwZODPfAwRVrO4yA75s9VCLFOhE8dStoS2IBDj9/U35Kp8+MmKdYoQYClW9nIEuowVL u+DlaQjvnRWWRzAStkWh+bwf7TeTH/zyWXo/HHk2kv0nuZL3FgBjtR60d/TYbuFfNt4t 5ujZ0D2g17jAL7iSvaub7zHZq9I7oJ9nHSpS2bKao4bROVqJXmXCPYia63A+6oHyPocf ueBxC4ZFAxlps/FLD6dGitzqTzhqVgE4HHqlwSiea7vGvT1DBZWc2ggu658/SH8HQVkx ZLtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jIYFGKaKOM9SQxW1TIR2cjV4YqG09lSg63SYAD4cHFg=; b=pYE7aIzz4CmvibXDxeA+ivQb/T+mGS64/jL9ccFTqjm4T1xrdMUFfi9+FJnwfxYHwI NSAyngxpYQKHKbcOMb6zah6LyXMSkqoW4kOmOJeAToTrLV3duCNG6WRP5OrI3BDJVroQ qCdDDw3mIy8JKyuADjsGMbAVUWKkeFvS4+PmLBGd+kJMWDTI86G/BSOiNgQsnYBZLxEN VAumvxZlYFU1rZMhpr81u3WC8prydPrRsTXBmbxWaH61fCOYkDlEqKQVhnWAxrQ9ydhN FJEPPKTgfSOsDIwDw3jveQRltTjzHJIoJ9HuzwYh/zaaHPGkpfAh081N3XjHFrPH1PSw iD1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Xde93O8I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f16-20020aa7d850000000b0046107f9520csi14225551eds.210.2022.11.09.08.57.35; Wed, 09 Nov 2022 08:57:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Xde93O8I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231372AbiKIQxM (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231438AbiKIQwg (ORCPT ); Wed, 9 Nov 2022 11:52:36 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A8EF24F29 for ; Wed, 9 Nov 2022 08:52:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012739; x=1699548739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C8gw4fXZss1vNc+3KIpIWYRKDR6wBig/C9HJl6yEnno=; b=Xde93O8IK/yhSafgu/zmtJGBj+RksPHh21nOtb7gQX+cztUmDfQf83E4 u7V9f/gyug3KeLxGIqywc/0rl4oFauCcHAgVjAKaZWCcUrBoPf2wrgWCA t/vhkDiYSV7BptLujJtp3KEqjbv0mxB6/VB+UH1iD120Qa7DmRGLs/g3D xEDoOTvhle1lfauYxMosoZDvnltQx0qo0x9jq4gKA5ZgRUKCsEusxPPth 1C2MiWwforxJ2RasIlSLGjcYahqlh2UvwF21nMn3qcm+OuFIY7t6ZGM7c XdIgNiBS2KXDfBszh94DdXUQKCO2dQO6dfg3gZlW68SxT30fJj2LJ6tf+ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553710" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553710" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827257" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827257" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id C9090109477; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv12 11/16] x86/mm, iommu/sva: Make LAM and SVA mutually exclusive Date: Wed, 9 Nov 2022 19:51:35 +0300 Message-Id: <20221109165140.9137-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038481784575226?= X-GMAIL-MSGID: =?utf-8?q?1749038481784575226?= IOMMU and SVA-capable devices know nothing about LAM and only expect canonical addresses. An attempt to pass down tagged pointer will lead to address translation failure. By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. By using the arch_prctl() userspace takes responsibility to never pass tagged address to the device. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ashok Raj Reviewed-by: Jacob Pan --- arch/x86/include/asm/mmu.h | 6 ++++-- arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 12 ++++++++++++ drivers/iommu/iommu-sva-lib.c | 12 ++++++++++++ include/linux/mmu_context.h | 7 +++++++ 6 files changed, 42 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 2fdb390040b5..1215b0a714c9 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -9,9 +9,11 @@ #include /* Uprobes on this MM assume 32-bit code */ -#define MM_CONTEXT_UPROBE_IA32 BIT(0) +#define MM_CONTEXT_UPROBE_IA32 BIT(0) /* vsyscall page is accessible on this MM */ -#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +/* Allow LAM and SVA coexisting */ +#define MM_CONTEXT_FORCE_TAGGED_SVA BIT(2) /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 91fdbb47b0ef..32483ab3f763 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -114,6 +114,12 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) mm->context.untag_mask = -1UL; } +#define arch_pgtable_dma_compat arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return !mm_lam_cr3_mask(mm) || + (mm->context.flags & MM_CONTEXT_FORCE_TAGGED_SVA); +} #else static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index a31e27b95b19..eb290d89cb32 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -23,5 +23,6 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 457f79be8bc7..b06de18215ce 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -789,6 +789,12 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) goto out; } + if (mm_valid_pasid(mm) && + !(mm->context.flags & MM_CONTEXT_FORCE_TAGGED_SVA)) { + ret = -EBUSY; + goto out; + } + if (!nr_bits) { ret = -EINVAL; goto out; @@ -899,6 +905,12 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_FORCE_TAGGED_SVA: + if (mmap_write_lock_killable(task->mm)) + return -EINTR; + task->mm->context.flags |= MM_CONTEXT_FORCE_TAGGED_SVA; + mmap_write_unlock(task->mm); + return 0; case ARCH_GET_MAX_TAG_BITS: if (!cpu_feature_enabled(X86_FEATURE_LAM)) return put_user(0, (unsigned long __user *)arg2); diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c index 27be6b81e0b5..8934c32e923c 100644 --- a/drivers/iommu/iommu-sva-lib.c +++ b/drivers/iommu/iommu-sva-lib.c @@ -2,6 +2,8 @@ /* * Helpers for IOMMU drivers implementing SVA */ +#include +#include #include #include @@ -31,6 +33,15 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) min == 0 || max < min) return -EINVAL; + /* Serialize against address tagging enabling */ + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (!arch_pgtable_dma_compat(mm)) { + mmap_write_unlock(mm); + return -EBUSY; + } + mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ if (mm_valid_pasid(mm)) { @@ -46,6 +57,7 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mm_pasid_set(mm, pasid); out: mutex_unlock(&iommu_sva_lock); + mmap_write_unlock(mm); return ret; } EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index 14b9c1fa05c4..f2b7a3f04099 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -35,4 +35,11 @@ static inline unsigned long mm_untag_mask(struct mm_struct *mm) } #endif +#ifndef arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return true; +} +#endif + #endif From patchwork Wed Nov 9 16:51:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17683 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446634wru; Wed, 9 Nov 2022 08:55:48 -0800 (PST) X-Google-Smtp-Source: AMsMyM4W9PXuZA6xwKQe6mfzOkt9cO+3vWdrWSRWrQZc5HCMHhvcE5qpng9GqXaVi8S7DmMUmmgD X-Received: by 2002:a05:6402:884:b0:461:1ed:579f with SMTP id e4-20020a056402088400b0046101ed579fmr60573617edy.413.1668012948138; Wed, 09 Nov 2022 08:55:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012948; cv=none; d=google.com; s=arc-20160816; b=ofYb5nFDq+nHecnCwsc9nO1etPUxJ36KoBx6qBuH4ye9+IHV/Y+aAuFIuGYgYHfsGd yb/JaIIUOSDTXTRLW6XOt14bmU+LEp8ub0KZm9M+lz7sXaKu8Ouml5da9DtOSnPiQ7s9 l2KWW8zyzbHRAwgJMgbA4xST3A0leYvtvqJUTGtY3Hraeldqo8lm2lc8rL/mKqFHcvCE Gxa1OmHfEd03A0Div5moEjXB4VgWB1MWEPx9HjB2KHZ3kvA7GJEosNwnYHZpWZL9zZIw RLiiJFqifWqI0Dk4ooHK8+YrMk85mSiVtZqYhuypf35+zq9P4AXZed6DODiT0+on5ShX 1NjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XTsXpEpsDctwvn64LXsgjN942xnSN/j6wbHHE7wF4nQ=; b=SGJHSr5vmH9WXpkfMalKN/YmNCqe/0m07c2PiZvPr4lRVmQ8kQcdrKWDaz8goq6V1e z4F0dXeb220bhCalZRwPfxy7qkJP9C6k7vKsPhu0Y1bCJxpcr/M7ucIxr0gdFtTXodlG y2EctJX30j9c1NCw77nBm7/BLBZ1PusrS31KloSk/DTD6HosOro2kd+b4Ix6tpQhgLD4 lVSuepOad2/8g3Mv0+QazppkEXmjb5tRfGPmsBZ9YDJqBg9j615FN3vV4Smx6h27qLXF Ukmye8Udq9wsNG+sjhNeAk7hZ+zKVWv4rW3Qe8H8WMIZhxuIpMO2Jzxi3lMrIpGhiJCl kxyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dFmiRSht; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m4-20020aa7d344000000b00460e1cc2d76si14561048edr.72.2022.11.09.08.55.25; Wed, 09 Nov 2022 08:55:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dFmiRSht; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231178AbiKIQxJ (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231458AbiKIQwg (ORCPT ); Wed, 9 Nov 2022 11:52:36 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DDDA25E87 for ; Wed, 9 Nov 2022 08:52:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012740; x=1699548740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=A07f05+t0KWtt4ghU/0U2MmLVS25K/SAxXIfuDOarp4=; b=dFmiRShtpj4mLsxULlmRTDEDE6d7CC0Co24v/K6AxwIzG6kTP4OeUWg/ xz1uWPEkylKDdAqzBOrYbNW2WBQuFHtDLDHDK+zC6tWASZq/5CvEU/lx8 NUTkihZgLgVxvICoyGApKHmhIOIxg9Txi7u1HMOfQSGpyBagmmcr1BvLn zD/84FLFiJeRDmm/x0lB7bVv0dcvcAqun9bfAhJLBOodt27SbwK+BdBxz fhLthexhiVJLVkx2y0vUAhMCVR1VV/zQrSZxlMxPRQJ8PJIf+Jg2DDGJh zGLbyMnTQqBKJTZOtm+f++bSF4/Le/EbCyLdzre3W6EsujxwHT1hCBDrq Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553720" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553720" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827261" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827261" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:10 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id D3B9D1094B3; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv12 12/16] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking Date: Wed, 9 Nov 2022 19:51:36 +0300 Message-Id: <20221109165140.9137-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038345337330845?= X-GMAIL-MSGID: =?utf-8?q?1749038345337330845?= From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are some arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: MALLOC - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. TAG_BITS - Max tag bits of LAM_U57 is 6. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 326 +++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..c1a16a9d4f2f 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..900a3a0fb709 --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 + +#define LAM_U57_MASK (0x3fULL << 57) +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + * + * @return: + * 0: Set LAM mode successfully + * others: failed to set LAM + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(LAM_U57_MASK)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +static unsigned long get_default_tag_bits(void) +{ + pid_t pid; + int lam = LAM_NONE; + int ret = 0; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + } else if (pid == 0) { + /* Set LAM mode in child process */ + if (set_lam(LAM_U57_BITS) == 0) + lam = LAM_U57_BITS; + else + lam = LAM_NONE; + exit(lam); + } else { + wait(&ret); + lam = WEXITSTATUS(ret); + } + + return lam; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t set_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + /* Get a random value as metadata */ + metadata = rand(); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + metadata = (src & ~(LAM_U57_MASK)) | ((metadata & 0x3f) << 57); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + * + * @return: + * 0: value on the pointer with metadate and value on original are same + * 1: not same. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)set_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + + +int handle_max_bits(struct testcases *test) +{ + unsigned long exp_bits = get_default_tag_bits(); + unsigned long bits = 0; + + if (exp_bits != LAM_NONE) + exp_bits = LAM_U57_BITS; + + /* Get LAM max tag bits */ + if (syscall(SYS_arch_prctl, ARCH_GET_MAX_TAG_BITS, &bits) == -1) + return 1; + + return (exp_bits != bits); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg = "BITS: Check default tag bits\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + if (tests & FUNC_BITS) + run_test(bits_cases, ARRAY_SIZE(bits_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Wed Nov 9 16:51:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17680 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp446590wru; Wed, 9 Nov 2022 08:55:44 -0800 (PST) X-Google-Smtp-Source: AMsMyM6gGwfvAC2hWkQBZFb2r7W3fBWFglke5+d+JN6DmfFr6izZD84o20ElMh0M2jiwdpn9FQIU X-Received: by 2002:a17:906:2b57:b0:7ad:90dc:eaf2 with SMTP id b23-20020a1709062b5700b007ad90dceaf2mr1406211ejg.526.1668012944151; Wed, 09 Nov 2022 08:55:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668012944; cv=none; d=google.com; s=arc-20160816; b=UQQj799rUcE1ARpxwKBdVBBxecFfOdZ6P9vLWl3eAHCzsRR3fAHqdwFOL/v3ABhgRq jfN0A26ERg8JZCnmurAsOZgvg7uaQ0wEyMi5KAPfE92ve8KLHJ1NfEWqdb6YVSpaS4pi 5csqv3cOiKlFDMsF50/GV/9ZVnO1BTBRSgUAWbFqLHuAtbln5DFchUAeWi8Hb5eSRX0g R9ZkLtCmbpT5vPHaJD8uRTqu0nahNBc7mrc2cxb+88mtsldsmA5zBCEsmWkvWUzjhGmJ C4s0yMwWFsSVCluBPTUs0WexGkOew4EVhGRWLyST/baSVT8nYpOeK5hK4jPSO2zfgZTB xCrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kvClGH7UWdzL+Tdet8/mTebKDV36RRSGh+dWyFD9pTo=; b=zp8pFsURGtXrIrVulVxpzUQVoj5piD9xZv1eCa8uW3pTPuZDN9ogvI52kCMXjr/oQu TTrUuqdhVpzYAu1bPQhR80g4acNrZZvsQyKRLFpaDYMpJTDJ1p9XFMz5ok72jxoQlU6m eZW5yMxxT5b9PxOY4fkRhYx5yrvTAZNHmKGlpy/s8Jjrx73jJqtWeXGraHhRdJwcMV4r uvT2XOr2kScWc0kNC6c5vdhNbnTicrqC9pRWRd4z4sFBGtiTVuDUxWPPi2M3hS9stxBD IvfKHMKyIbzb+GJdtaGJ6W6DvVJE2ILIA6idqhvkXF62ac0H2FHvL2h/cEVje+bbGmKh otyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=neux5cka; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d17-20020aa7c1d1000000b0045945eed10asi14811303edp.5.2022.11.09.08.55.20; Wed, 09 Nov 2022 08:55:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=neux5cka; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229588AbiKIQw6 (ORCPT + 99 others); Wed, 9 Nov 2022 11:52:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229835AbiKIQwg (ORCPT ); Wed, 9 Nov 2022 11:52:36 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A02224F21 for ; Wed, 9 Nov 2022 08:52:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012739; x=1699548739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TuBz7tekTf1BWBxAT1QoET0tgEOR4TTS5pMFs+n0/Gc=; b=neux5ckaZSyKRfNrpmkqhXVxAt0qzML0WRwB/ACa4+f8Zyf6pdZ3WmbH a5Q9LZk1VIk1MxK3cZhcy6muRh8FsLgujGqSljaaLP/fIOwgVfUxd2iLv v3HZOuVSpED18rnLuhVLoR/jdggPwr9L5CqYtMTjs6VFMWOo8aKLs/Dog 7Fx9mxipvvDVUIg2Q4IEodvIFDMbRbXAxV0rTTAC6zL5TuDXgjviWS4Pu cd5CMeOEwJMKopTr1rwVomtx3QfoSCXt7ot2ZT0VoOPdJaXsc9k8vXarQ GTwnnfXkUBryt2pt1TyvxX0uaodZXvkzjZYYAIxEGbpQV5A9huRrbi84h w==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="291419879" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="291419879" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:18 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053288" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053288" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:12 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id DE8FE1094F5; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv12 13/16] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Wed, 9 Nov 2022 19:51:37 +0300 Message-Id: <20221109165140.9137-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038341158666522?= X-GMAIL-MSGID: =?utf-8?q?1749038341158666522?= From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 144 +++++++++++++++++++++++++++++- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 900a3a0fb709..cdc6e40e00e0 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -29,11 +30,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -49,6 +57,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -61,6 +70,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -213,6 +232,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)set_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -241,13 +322,20 @@ static void run_test(struct testcases *test, int count) struct testcases *t = test + i; /* fork a process to run test case */ + tests_cnt++; ret = fork_test(t); + + /* return 3 is not support LA57, the case should be skipped */ + if (ret == 3) { + ksft_test_result_skip(t->msg); + continue; + } + if (ret != 0) ret = (t->expected == ret); else ret = !(t->expected); - tests_cnt++; ksft_test_result(ret, t->msg); } } @@ -268,7 +356,6 @@ static struct testcases malloc_cases[] = { }, }; - static struct testcases bits_cases[] = { { .test_func = handle_max_bits, @@ -276,11 +363,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); printf("\t-h: help\n"); } @@ -320,6 +450,12 @@ int main(int argc, char **argv) if (tests & FUNC_BITS) run_test(bits_cases, ARRAY_SIZE(bits_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Nov 9 16:51:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17703 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447565wru; Wed, 9 Nov 2022 08:58:00 -0800 (PST) X-Google-Smtp-Source: AMsMyM4PJ3za7pf1LISaTX2mU3Nx41SQ+5rg0qrldjshF9hdvogL6j0bxKidi2nA4jibHzMB4uvl X-Received: by 2002:a17:906:9bc4:b0:7aa:9184:d95d with SMTP id de4-20020a1709069bc400b007aa9184d95dmr57645299ejc.349.1668013079894; Wed, 09 Nov 2022 08:57:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013079; cv=none; d=google.com; s=arc-20160816; b=0/jlkSjPsDKPdchVC6kXs1G5PjeaGw6jFLD1x2eHv7bwdIaNsTO+iYdm857ay3gczC jnYfTj3isRQAM54oxpjjU9kMSy1B0N64hzWMfLclKy0KIxCGGF2p7SEsql26UuZqkpeH DptLbc7PoeTkht7MgGxDQaQWUfJClqVkpfGyz/XEpW38tQd8V9CGlvGZTbV326DHs7qb 4aqrzns2uvOqxfFQ5EKYZ94D3s6hFwwZWk9MVXZa6j/COqpSqeXCN5QfpL2XyrLzd9/A iJlCMV+uy3PsnsTooyP4unC4iWb+Sao17i/cyR7VcF5N6+3hnZKHdFLmjMu56/2gQxbI GDiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9KN/Re8kLSvGk9k/yFun3GFKx3YHJN773QAA//jRU1I=; b=JibDUTJXm3rk+xPuSQTib0SqMXMNf/7Ne3nzyJAyaeRrFSLOA2s/7g2PETLWXioYOG DuLsO1anBk2l80BaT7AsS1OMja3c1MT4k4ASAR9o1G9kVRN4qssRVr3aC6ixGnXBrfsa RwtSsRcangSYmGzNlE585z3mDs8oeRoAgP05/sjCvSvTA6m4VMoXhD2RW/QlGLLka5K7 VX2K31aHBSdSqBmgrZ3doEj6etvUwR7UOnOvKEhOsKpNPREDeohNcqtCH6FHjUfDDzHc 2oV5xPBMdP5IXD14XZ+oB+BlMZSiFTdh1xI0Cw17YYIpNqtxltPKWsp+GoXIcyawnO1O w0BA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="d1/zA6F4"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wg5-20020a17090705c500b00772fe97b31esi16682110ejb.901.2022.11.09.08.57.36; Wed, 09 Nov 2022 08:57:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="d1/zA6F4"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230505AbiKIQxQ (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231479AbiKIQwh (ORCPT ); Wed, 9 Nov 2022 11:52:37 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77C1525E89 for ; Wed, 9 Nov 2022 08:52:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012740; x=1699548740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=br6Om71qzm65KaCJtdGB8t7hbJwdX+JrsLtwnbZitNE=; b=d1/zA6F4d9P3IUyUFxjS0riyrlAqUXjTBJqmPvE2+sSI2fReIe2VysPT WI3IVVbJgAm6BGS9UmgJAlBGfH+S8io1ck89GM21Uk5NgHiqWOY0Jaoj/ SCWMGwbccEeSKwlxQsdXEOGczEVhVZ9p8aWbbWOcv3fD9jbZthhkOUMEJ RPY4urbaQQXaW7knxaoj/M21fG3BzZpJ2jBd8KWlS2TV1CP9mHM9nGkgW 19LzvxlZp0D1l8GvaJh4rZ6wfYoANVHY2X67PHvLz66pUrI53RWjWQ61j SrTu4HjgVZ/x410zmX+NOeC8MSTvpnNHQvIXKBBRW3AE+96oXbGPQz6TU Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="298553748" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="298553748" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:19 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="587827269" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="587827269" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:13 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id E99B21094F8; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv12 14/16] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Wed, 9 Nov 2022 19:51:38 +0300 Message-Id: <20221109165140.9137-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038483089692048?= X-GMAIL-MSGID: =?utf-8?q?1749038483089692048?= From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index cdc6e40e00e0..8ea1fcef4c9f 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -32,8 +36,9 @@ #define FUNC_BITS 0x2 #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -42,6 +47,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -51,6 +63,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -294,6 +333,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)set_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -340,6 +658,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -410,7 +744,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); printf("\t-h: help\n"); } @@ -456,6 +790,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Nov 9 16:51:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17701 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447532wru; Wed, 9 Nov 2022 08:57:53 -0800 (PST) X-Google-Smtp-Source: AMsMyM6w98+wo/awW4Qm2xJy3VbjF2yO84BNgke0DmXIMH3nSzNmT58wLq7KLIsiO9mErupHWM/r X-Received: by 2002:aa7:db96:0:b0:463:a60b:bbcb with SMTP id u22-20020aa7db96000000b00463a60bbbcbmr47219759edt.198.1668013073808; Wed, 09 Nov 2022 08:57:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013073; cv=none; d=google.com; s=arc-20160816; b=ElRr4Td5YVE1qq5XL1t9HbfRUt4HGumIANM1k3odIdFa+5s1twm5CNMDqcOUiymWi8 3WS0jA2FezbUEn8I/pNbCSnOeF6LvI/gHPHrvhEhqNIFYxjvlnZMawLiYRUJ2axOao4/ lXpKlxpF5APNXHPyongCBPxjBTgK4awFYIg5D5d6QrQoGEr46Olwsyij9duqnHt6Kt6H Geab8aIbwM9HYIqjm+LO3ryDSMD4AnDrhjumNUwiZkfqJyJXexhh+IWY/8oT7vGVRaWB if2cHjccQjX8ZBgHTmWHLQQ9/soXjPAOpD2yVDkgHbQNRFC4pXfDlVmfLupApefKOhtx pd0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iRoUaAq1wGIcSIS4qkv8NrfubUNx/rtX6/sfQtwGBdE=; b=NmDDTwjgxsbxcw0TBb44zBolHWhiq0A7246NN2JMBbkfHk3IjmoheQTEmm4SfyWi/3 iChgzPBjoDcJ47CMqZaxGVLiPDEQinbSHWbs/s3f82AP8mbF5ajVLhuC45drJAJwSP7X ITYdC0OGt+rrvfdW3pxjpdS0lBtgEbRMY2kM6qP807/fqcV97LNazT//39Q4ZW1TXtr2 WMmNJv9kvVylUxtD7+Z1Bas5qpSh1G47/7mrfU3wibIQGVWft4C/6dhGkBDcPmU4G6V3 Y2ESZ0YUC+QKhfaE5v70s56mSfSD/D04C1vg8Q/6jaGb6kHwvXhweLngbCHOSlpV4iqS sJpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kr8wqSeA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nb5-20020a1709071c8500b007aa6ce58d9asi16595545ejc.712.2022.11.09.08.57.29; Wed, 09 Nov 2022 08:57:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kr8wqSeA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231406AbiKIQxV (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231202AbiKIQwj (ORCPT ); Wed, 9 Nov 2022 11:52:39 -0500 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0224D2648E for ; Wed, 9 Nov 2022 08:52:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012744; x=1699548744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2PBgo8dzetbAgvNxQe4E7GoQrbgUDaybtLdJFJyGEss=; b=kr8wqSeAyjM1Euo183XZzaJ0nimzESMZQqfbS6xmdjKMsOuuDkoAFtr4 vn2/8mJo8E0ihBtMlIHLe0+PCtTRmDNXpaDOFRSTMTehzEhCzEz2IGcRK 9WuiG3WlOOwGC4VKa0Q1WlEtPJ5Uv/cRdNCAPJYRSbR1IY7qgudVLLl9X rQ2rhv8dPqIDDxnj7ulVYgcByWCEwnhiCKoDvS+R4CRMLXtoRVeJ9a1Nz gK7DIyxxHKw+4c3w41T6dbAd+LIya5BPETnILPoLU1ShKbYQCaIGrPuve sR5hDXZalzuu5QbmD8rlfCuye7CcZEA3D5MeXLa4G/vZxc584Fc7FjxgR Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="337768953" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="337768953" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053352" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053352" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:19 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 009D51094F9; Wed, 9 Nov 2022 19:52:00 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv12 15/16] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Wed, 9 Nov 2022 19:51:39 +0300 Message-Id: <20221109165140.9137-16-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038476472614722?= X-GMAIL-MSGID: =?utf-8?q?1749038476472614722?= From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 125 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 8ea1fcef4c9f..cfc9073c0262 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -37,8 +37,9 @@ #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -174,6 +175,28 @@ static unsigned long get_default_tag_bits(void) return lam; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(LAM_U57_MASK)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t set_metadata(uint64_t src, unsigned long lam) { @@ -581,7 +604,7 @@ int do_uring(unsigned long lam) switch (lam) { case LAM_U57_BITS: /* Clear bits 62:57 */ - addr = (addr & ~(0x3fULL << 57)); + addr = (addr & ~(LAM_U57_MASK)); break; } free((void *)addr); @@ -632,6 +655,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -740,11 +829,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n"); printf("\t-h: help\n"); } @@ -764,7 +868,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -778,6 +882,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -793,6 +907,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Nov 9 16:51:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 17689 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447064wru; Wed, 9 Nov 2022 08:56:50 -0800 (PST) X-Google-Smtp-Source: AMsMyM6MJORPk+sP8thZaahrPCJHqDoN+3RmXqtts2Y0F+I0BG86DvndltPf6sL2rYCRsG3Nxb+q X-Received: by 2002:a17:906:f243:b0:79a:101a:7e57 with SMTP id gy3-20020a170906f24300b0079a101a7e57mr1409228ejb.368.1668013010325; Wed, 09 Nov 2022 08:56:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668013010; cv=none; d=google.com; s=arc-20160816; b=EaQLmMTgq2WEglgUsp4hI+8SD6k53Tw8EjsuSQfeQmf4Lrdz2kgbTYLaWcqVZSZMff wRCbFB/tuGwBKH8SrNlAPaHRRMRRHPV23ILyPk80kS343oREzJlMdx++Xq+Hntvc1op+ K5tY7szSXFMMIzZzmhtOSE4qCafEU7NpMZToq85qTgcj6PNtA/VVYOYx/fz/hgemd0i3 sheweNYtApbR+sH5TWJypvffDqHRYTE3kbrBj9Vam26G/gNv3IANTAaFet6icyOjpBMa zkbu0xdSUZpiXDK6vU2IDHeCVGt+YX8miQFsghTUM5iA4Kca4niqMpMv3nRmenZe2D/D DpUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hbjhs9X/9HHvuRP/04rQj+oMyZLrzeRRdNUsgJNE0n4=; b=F/3cQWjzRSA2Jsucs2a4BlWn3qBSlPSKr7nQJSUHnS7OUi0KCAIUIeWfz4sWSVhVfq Ctr41P8qh0DaJL0N554nGe+n3OTeC+vynjT328TlQtGtguhPiyiJd2dts0UHyMbiQXRk SF7k0lg63gKp4Ss840QFffx0Nse/xsH7F21EQJc3TbmRVrOqJkiXGqV2loQsgYs2+tl2 SNf/5xH3tz+2B/3J4q5N7YbJAVuKn1bAJuYvPedJMF87GhuMJG2jGqakQwPaN1BUFuDj Fb0vDPVLYWeVTfIlx4TFGpdZIGq93UlVBvgY7U3laaSjdp+e7XbKC3TpamDPYwYCtT7h pTZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eDdNrHMC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qk44-20020a1709077fac00b007ae0ca417e4si16408134ejc.690.2022.11.09.08.56.26; Wed, 09 Nov 2022 08:56:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eDdNrHMC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229634AbiKIQx1 (ORCPT + 99 others); Wed, 9 Nov 2022 11:53:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231603AbiKIQwj (ORCPT ); Wed, 9 Nov 2022 11:52:39 -0500 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE81C25298 for ; Wed, 9 Nov 2022 08:52:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668012745; x=1699548745; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9+J3Br8bgbOof9jfpl07GNctQC3Rvx7AQwiIVAY7w34=; b=eDdNrHMCMXLbhUoShuftIHO3GmcsXdWqDzSUde9pPg2m+TYA4th8ImZA 712CnQ4Rjho9OxW3f80aDqGwMJ3GNIBZUilA0Sac6oPt0Zud9Fu0LKLdG +YALfTbPqBgl6bZJ5QhZc+7Wbf3iipEoO5793EJQGSEo5DN7S9mbfbqrq tI9iGl+Z52N5PmcJW+B0QRpqqyRkSikXO9p0vPuxkN+lXvboxPVHdi17s s9JSJfquwDy2W4yGvtiPoYikEywQvl/hSNJj/lDZr3/z3iCk1sCpNoHdi 1tOpMwMTb9bXzhkh3OwKvvVdWAZPWsba+1U2UMEQcpFnCL7/5xrLQWYMi g==; X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="337768954" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="337768954" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10526"; a="668053354" X-IronPort-AV: E=Sophos;i="5.96,151,1665471600"; d="scan'208";a="668053354" Received: from dschramm-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.251.219.85]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2022 08:52:19 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 0ACED109523; Wed, 9 Nov 2022 19:52:01 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv12 16/16] selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking Date: Wed, 9 Nov 2022 19:51:40 +0300 Message-Id: <20221109165140.9137-17-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> References: <20221109165140.9137-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749038409977534098?= X-GMAIL-MSGID: =?utf-8?q?1749038409977534098?= From: Weihong Zhang By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. Add new test cases for the new arch_prctl: Before using ARCH_FORCE_TAGGED_SVA, should not allow to enable LAM/SVA coexisting. the test cases should be negative. The test depands on idxd driver and iommu. before test, need add "intel_iommu=on,sm_on" in kernel command line and insmod idxd driver. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 237 +++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index cfc9073c0262..52a876a7ccb8 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -30,6 +30,7 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 /* Specified test function bits */ #define FUNC_MALLOC 0x1 @@ -38,8 +39,9 @@ #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 #define FUNC_INHERITE 0x20 +#define FUNC_PASID 0x40 -#define TEST_MASK 0x3f +#define TEST_MASK 0x7f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -55,11 +57,19 @@ #define URING_QUEUE_SZ 1 #define URING_BLOCK_SZ 2048 +/* Pasid test define */ +#define LAM_CMD_BIT 0x1 +#define PAS_CMD_BIT 0x2 +#define SVA_CMD_BIT 0x4 + +#define PAS_CMD(cmd1, cmd2, cmd3) (((cmd3) << 8) | ((cmd2) << 4) | ((cmd1) << 0)) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ unsigned long lam; uint64_t addr; + uint64_t cmd; int (*test_func)(struct testcases *test); const char *msg; }; @@ -556,7 +566,7 @@ int do_uring(unsigned long lam) struct file_io *fi; struct stat st; int ret = 1; - char path[PATH_MAX]; + char path[PATH_MAX] = {0}; /* get current process path */ if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) @@ -852,6 +862,226 @@ static void cmd_help(void) printf("\t-h: help\n"); } +/* Check for file existence */ +uint8_t file_Exists(const char *fileName) +{ + struct stat buffer; + + uint8_t ret = (stat(fileName, &buffer) == 0); + + return ret; +} + +/* Sysfs idxd files */ +const char *dsa_configs[] = { + "echo 1 > /sys/bus/dsa/devices/dsa0/wq0.1/group_id", + "echo shared > /sys/bus/dsa/devices/dsa0/wq0.1/mode", + "echo 10 > /sys/bus/dsa/devices/dsa0/wq0.1/priority", + "echo 16 > /sys/bus/dsa/devices/dsa0/wq0.1/size", + "echo 15 > /sys/bus/dsa/devices/dsa0/wq0.1/threshold", + "echo user > /sys/bus/dsa/devices/dsa0/wq0.1/type", + "echo MyApp1 > /sys/bus/dsa/devices/dsa0/wq0.1/name", + "echo 1 > /sys/bus/dsa/devices/dsa0/engine0.1/group_id", + "echo dsa0 > /sys/bus/dsa/drivers/idxd/bind", + /* bind files and devices, generated a device file in /dev */ + "echo wq0.1 > /sys/bus/dsa/drivers/user/bind", +}; + +/* DSA device file */ +const char *dsaDeviceFile = "/dev/dsa/wq0.1"; +/* file for io*/ +const char *dsaPasidEnable = "/sys/bus/dsa/devices/dsa0/pasid_enabled"; + +/* + * DSA depends on kernel cmdline "intel_iommu=on,sm_on" + * return pasid_enabled (0: disable 1:enable) + */ +int Check_DSA_Kernel_Setting(void) +{ + char command[256] = ""; + char buf[256] = ""; + char *ptr; + int rv = -1; + + snprintf(command, sizeof(command) - 1, "cat %s", dsaPasidEnable); + + FILE *cmd = popen(command, "r"); + + if (cmd) { + while (fgets(buf, sizeof(buf) - 1, cmd) != NULL); + + pclose(cmd); + rv = strtol(buf, &ptr, 16); + } + + return rv; +} + +/* + * Config DSA's sysfs files as shared DSA's WQ. + * Generated a device file /dev/dsa/wq0.1 + * Return: 0 OK; 1 Failed; 3 Skip(SVA disabled). + */ +int Dsa_Init_Sysfs(void) +{ + uint len = ARRAY_SIZE(dsa_configs); + const char **p = dsa_configs; + + if (file_Exists(dsaDeviceFile) == 1) + return 0; + + /* check the idxd driver */ + if (file_Exists(dsaPasidEnable) != 1) { + printf("Please make sure idxd driver was loaded\n"); + return 3; + } + + /* Check SVA feature */ + if (Check_DSA_Kernel_Setting() != 1) { + printf("Please enable SVA.(Add intel_iommu=on,sm_on in kernel cmdline)\n"); + return 3; + } + + /* Check the idxd device file on /dev/dsa/ */ + for (int i = 0; i < len; i++) { + if (system(p[i])) + return 1; + } + + /* After config, /dev/dsa/wq0.1 should be generated */ + return (file_Exists(dsaDeviceFile) != 1); +} + +/* + * Open DSA device file, triger API: iommu_sva_alloc_pasid + */ +void *allocate_dsa_pasid(void) +{ + int fd; + void *wq; + + fd = open(dsaDeviceFile, O_RDWR); + if (fd < 0) { + perror("open"); + return MAP_FAILED; + } + + wq = mmap(NULL, 0x1000, PROT_WRITE, + MAP_SHARED | MAP_POPULATE, fd, 0); + if (wq == MAP_FAILED) + perror("mmap"); + + return wq; +} + +int set_force_svm(void) +{ + int ret = 0; + + ret = syscall(SYS_arch_prctl, ARCH_FORCE_TAGGED_SVA); + + return ret; +} + +int handle_pasid(struct testcases *test) +{ + uint tmp = test->cmd; + uint runed = 0x0; + int ret = 0; + void *wq = NULL; + + ret = Dsa_Init_Sysfs(); + if (ret != 0) + return ret; + + for (int i = 0; i < 3; i++) { + int err = 0; + + if (tmp & 0x1) { + /* run set lam mode*/ + if ((runed & 0x1) == 0) { + err = set_lam(LAM_U57_BITS); + runed = runed | 0x1; + } else + err = 1; + } else if (tmp & 0x4) { + /* run force svm */ + if ((runed & 0x4) == 0) { + err = set_force_svm(); + runed = runed | 0x4; + } else + err = 1; + } else if (tmp & 0x2) { + /* run allocate pasid */ + if ((runed & 0x2) == 0) { + runed = runed | 0x2; + wq = allocate_dsa_pasid(); + if (wq == MAP_FAILED) + err = 1; + } else + err = 1; + } + + ret = ret + err; + if (ret > 0) + break; + + tmp = tmp >> 4; + } + + if (wq != MAP_FAILED && wq != NULL) + if (munmap(wq, 0x1000)) + printf("munmap failed %d\n", errno); + + if (runed != 0x7) + ret = 1; + + return (ret != 0); +} + +/* + * Pasid test depends on idxd and SVA, kernel should enable iommu and sm. + * command line(intel_iommu=on,sm_on) + */ +static struct testcases pasid_cases[] = { + { + .expected = 1, + .cmd = PAS_CMD(LAM_CMD_BIT, PAS_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute LAM, PASID, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(LAM_CMD_BIT, SVA_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute LAM, SVA, PASID in sequence\n", + }, + { + .expected = 1, + .cmd = PAS_CMD(PAS_CMD_BIT, LAM_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute PASID, LAM, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(PAS_CMD_BIT, SVA_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute PASID, SVA, LAM in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, LAM_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, LAM, PASID in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, PAS_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, PASID, LAM in sequence\n", + }, +}; + int main(int argc, char **argv) { int c = 0; @@ -910,6 +1140,9 @@ int main(int argc, char **argv) if (tests & FUNC_INHERITE) run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + if (tests & FUNC_PASID) + run_test(pasid_cases, ARRAY_SIZE(pasid_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass();