From patchwork Mon Jan 23 22:04:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47437 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837802wrn; Mon, 23 Jan 2023 14:07:22 -0800 (PST) X-Google-Smtp-Source: AMrXdXubws2mLqF5cJYQW+4fm9gW99ysum9+aU1QjkQbEDzoka9Dlb3cOLW0LJs59xjn+REeR9sx X-Received: by 2002:a05:6402:5419:b0:46b:19ab:68d8 with SMTP id ev25-20020a056402541900b0046b19ab68d8mr28100183edb.40.1674511642429; Mon, 23 Jan 2023 14:07:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511642; cv=none; d=google.com; s=arc-20160816; b=PtXINr0fdjpQnlae23hHTWzJh7j4owUiVF+MIkfEpy9r9Rv9xfzXqSkZvSoiQjRlTd jBGoSp1Efbw6WOa1cyU/NwG5rVQh22zdr+H+c/MpGAun1Y9z8vpesAVlFPrNX9OAb4eJ gcpuRb73a/oNV4RP3Mcs6mDsZfTyCMrGFQXsTykC1zWdE4tnXlSRDt+VOjtilUzAfzza iPS/hK2keMnewIoY2x2iEZrM1gMQEbdvXdY94mFRD2ja/fhyjB9uE9Jdc/b1anFASiQb wa2oai8lWRMz2fvs8UyBh0x+VVYHSoAVHNvbTaTJ/R1j0eFZkzQnSKBTWVIprw0MGLpX CQYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CTJZWXrll6yca84fCeKN5uQPJXa8IESHCD+UdxuT9eY=; b=A+k5WUKBV+OLHcN5Yg5w3zg1PVjArV/hnrX9kLpmcinr9P4L9TkLWeEwwqW+ZRgcoh U626svlQwINFWQfcbhZmOfJU/xpgj/mZzeFydEcLstCnrII9FaixXF/13mM7mCPtlM/H e+c9eyg+3fx63jC7A0lPF/PPdcC+bHFXvVYUtjMsanDdnTmEawdDP8hUf7Mty6Q9ILHV jJOtU7yGmybK3V9v9z6O3khi1bYR2MJOT1Cl/fz7nkEJ1hFitvVxx7PRFgL35g45tcFA Wtf32hOBMe7gDawe1QjPVEpocXg9lqtlJe2tyJqiNqhmbJbADWmk90zLLQxEwjrewXgK v5+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="T4v/IL4X"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ec47-20020a0564020d6f00b0049bac85dfb6si614849edb.19.2023.01.23.14.06.58; Mon, 23 Jan 2023 14:07:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="T4v/IL4X"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232698AbjAWWGR (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232592AbjAWWGJ (ORCPT ); Mon, 23 Jan 2023 17:06:09 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED1BF38E85 for ; Mon, 23 Jan 2023 14:05:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511542; x=1706047542; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Xffq36fUS8+sdsJCnvn6O/XkT/sivjc0AK1M915RtWI=; b=T4v/IL4X/UvljuBfJ9jTTj6AeTxiyHehLVAlk3j1VNn2uJzkajsmDk4f J2tgxIcf/YGdWR2G2Ws/7tScGtds5GB4APQrS+VVLL0MJae/47mEAzuoF 1Hh+U4vZkBLrEgH1MdQI8qTUC8PMPldfz4CyLXj0Qq5bFp4GYryQN3pRR QGKvxeX5A3Yz+lKlusYl6YFKCuT1Vfs9atwoulg1KeyxFrGWfBmzsAJfl 6vivMT+WkOGRIYqlkcQmuSG546AwLGs91pVezuaS9YvRCFwDuOlXSyfH3 IbQGiiGLeFSHmwNU2DPIFTn9ChScaOIIawryFJy861loDPXmER29nSQaS A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198136" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198136" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103353" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103353" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:07 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 21F3A10942D; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 01/17] x86/mm: Rework address range check in get_user() and put_user() Date: Tue, 24 Jan 2023 01:04:44 +0300 Message-Id: <20230123220500.21077-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852719696158069?= X-GMAIL-MSGID: =?utf-8?q?1755852719696158069?= The functions get_user() and put_user() check that the target address range resides in the user space portion of the virtual address space. In order to perform this check, the functions compare the end of the range against TASK_SIZE_MAX. For kernels compiled with CONFIG_X86_5LEVEL, this process requires some additional trickery using ALTERNATIVE, as TASK_SIZE_MAX depends on the paging mode in use. Linus suggested that this check could be simplified for 64-bit kernels. It is sufficient to check bit 63 of the address to ensure that the range belongs to user space. Additionally, the use of branches can be avoided by setting the target address to all ones if bit 63 is set. There's no need to check the end of the access range as there's huge gap between end of userspace range and start of the kernel range. The gap consists of canonical hole and unused ranges on both kernel and userspace sides. If an address with bit 63 set is passed down, it will trigger a #GP exception. _ASM_EXTABLE_UA() complains about this. Replace it with plain _ASM_EXTABLE() as it is expected behaviour now. The updated get_user() and put_user() checks are also compatible with Linear Address Masking, which allows user space to encode metadata in the upper bits of pointers and eliminates the need to untag the address before handling it. Signed-off-by: Kirill A. Shutemov Suggested-by: Linus Torvalds Acked-by: Peter Zijlstra (Intel) --- arch/x86/lib/getuser.S | 83 ++++++++++++++++-------------------------- arch/x86/lib/putuser.S | 54 ++++++++++++--------------- 2 files changed, 55 insertions(+), 82 deletions(-) diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b70d98d79a9d..b64a2bd1a1ef 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -37,22 +37,22 @@ #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rdx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rdx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_DX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rax, %rdx + sar $63, %rdx + or %rdx, %rax +.else + cmp $TASK_SIZE_MAX-\size+1, %eax + jae .Lbad_get_user + sbb %edx, %edx /* array_index_mask_nospec() */ + and %edx, %eax +.endif +.endm .text SYM_FUNC_START(__get_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=1 ASM_STAC 1: movzbl (%_ASM_AX),%edx xor %eax,%eax @@ -62,11 +62,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) SYM_FUNC_START(__get_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=2 ASM_STAC 2: movzwl (%_ASM_AX),%edx xor %eax,%eax @@ -76,11 +72,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) SYM_FUNC_START(__get_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=4 ASM_STAC 3: movl (%_ASM_AX),%edx xor %eax,%eax @@ -90,30 +82,17 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) SYM_FUNC_START(__get_user_8) -#ifdef CONFIG_X86_64 - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=8 ASM_STAC +#ifdef CONFIG_X86_64 4: movq (%_ASM_AX),%rdx - xor %eax,%eax - ASM_CLAC - RET #else - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user_8 - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX - ASM_STAC 4: movl (%_ASM_AX),%edx 5: movl 4(%_ASM_AX),%ecx +#endif xor %eax,%eax ASM_CLAC RET -#endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -166,7 +145,7 @@ EXPORT_SYMBOL(__get_user_nocheck_8) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC -bad_get_user: +.Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET @@ -184,23 +163,23 @@ SYM_CODE_END(.Lbad_get_user_8_clac) #endif /* get_user */ - _ASM_EXTABLE_UA(1b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, .Lbad_get_user_clac) + _ASM_EXTABLE(2b, .Lbad_get_user_clac) + _ASM_EXTABLE(3b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) #endif /* __get_user */ - _ASM_EXTABLE_UA(6b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, .Lbad_get_user_clac) + _ASM_EXTABLE(7b, .Lbad_get_user_clac) + _ASM_EXTABLE(8b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 32125224fcca..3062d09a776d 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -33,20 +33,20 @@ * as they get called from within inline assembly. */ -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rbx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rbx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_BX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rcx, %rbx + sar $63, %rbx + or %rbx, %rcx +.else + cmp $TASK_SIZE_MAX-\size+1, %ecx + jae .Lbad_put_user +.endif +.endm .text SYM_FUNC_START(__put_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=1 ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -66,9 +66,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) SYM_FUNC_START(__put_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=2 ASM_STAC 3: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -88,9 +86,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) SYM_FUNC_START(__put_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=4 ASM_STAC 5: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -110,9 +106,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) SYM_FUNC_START(__put_user_8) - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 @@ -144,15 +138,15 @@ SYM_CODE_START_LOCAL(.Lbad_put_user_clac) RET SYM_CODE_END(.Lbad_put_user_clac) - _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, .Lbad_put_user_clac) + _ASM_EXTABLE(2b, .Lbad_put_user_clac) + _ASM_EXTABLE(3b, .Lbad_put_user_clac) + _ASM_EXTABLE(4b, .Lbad_put_user_clac) + _ASM_EXTABLE(5b, .Lbad_put_user_clac) + _ASM_EXTABLE(6b, .Lbad_put_user_clac) + _ASM_EXTABLE(7b, .Lbad_put_user_clac) + _ASM_EXTABLE(9b, .Lbad_put_user_clac) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, .Lbad_put_user_clac) + _ASM_EXTABLE(10b, .Lbad_put_user_clac) #endif From patchwork Mon Jan 23 22:04:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47435 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837641wrn; Mon, 23 Jan 2023 14:07:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXt+dmIM+ghjV+2GzMAOE/wHUVo6PQqw9SA70QX0KWQ10CPMWXYs5uxybmgQbSQCQ3TK5ayX X-Received: by 2002:a17:906:3291:b0:86f:356e:ba43 with SMTP id 17-20020a170906329100b0086f356eba43mr26302113ejw.18.1674511620311; Mon, 23 Jan 2023 14:07:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511620; cv=none; d=google.com; s=arc-20160816; b=CBEiRb6wsJFSrP2n95hfGSUFnRswFNbzkfsYA4S8aHZiV1jI//GlRVb4Cp7YrVEmZ4 3lvhRu2LmkemHiOIam4FqJ2B7xxCBmxNHi+h60JRhYi5ksJzfgZ8OW0iWYi/dNuUWjHP W5IMOU8hL3GOzqupLknGRRaDFNGdiaQjwEM3jCTVbc0t8InTAAQbmt0xg6LZvl1g5SEt eb3DTg1gJXv3QqzfJJVm6WfCYXX8GpeG5R+rJZphcbJCd/gVFVTbw5OOJG6fdWmFJvuO QrhfLAft77fGXN4ueMZVRh05MUG93IIGUNqFr8cIU0tfT5sGlmswxtl5B+QyvD6QCB69 2/BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jalrD7nqZv10CRswVZmChr9ky4Zab9mWxCuyLV858Iw=; b=mPghhmV1Cdhx7Ch7Ougo3ZdTB+f3U6bw8yuiLNHOnQPnijteryICoW6HtK2Aq5nzYx fE34ZZFrRv0g3AsBgCzjGMtbVWaXdgIXcoVtYNzPDRxs2U4ANm/4pCZe1vzGCVbCx8hY v0G5XwFn78+QFX1vzpIgFB5gikoZ6hDN6OB2Zm6D+S85WngoIzOG1sUVCUh620JDRf+f Uuw3NkeEXWQcqXInyJnxB2vIz21547mf5kxq2N/Awo/XVNnJg0niQX8FCd0n/uR22zQ2 Bf6FheZGAjPZ98EIfPtWn8kuuYajyu5LDdF8ecT2WJ1o4h64rBqHeYpIKbZIVGhcP5fk DGuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N929qQ5D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qn21-20020a170907211500b0087787ae09f4si11742209ejb.1006.2023.01.23.14.06.36; Mon, 23 Jan 2023 14:07:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N929qQ5D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232645AbjAWWGG (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232579AbjAWWF4 (ORCPT ); Mon, 23 Jan 2023 17:05:56 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C3DF37F24 for ; Mon, 23 Jan 2023 14:05:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511536; x=1706047536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PE6h3wytuKJk3kXgRyY8lFutpFDe1IptHJikql0qTdY=; b=N929qQ5Dwu4CqLIVdOsHyxBPOIFGT5hyqHqdBRs+yNmMAeKQgfigaEbp LLKTcPtrnOpvH1nLf9LCqIyMbYggUGz8osRx1GiLIrpgOSB1ZGg8wOfS0 3uyM2v++1dIFRTF6XRP7yjC1SwuqG+Q95+AnyWojpIHNJAst/Taxm5rwP P2HOknmubA8EvqjIesYhDqTUjboZWpDYFxycg2W64yUiMXjMScuMPPUQj Ay/IS0QZKOMVKdumrF+ZOM1FvuRAhYA8XG+jbTff6ZzE9cIH1iF7/ILGb +bYAxcbvlc/yeczk6WM/hT4gRX3IGd8883AEUjolT6GV/xz9YtInDL2k4 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198132" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198132" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103346" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103346" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:07 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 2C979109437; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 02/17] x86: Allow atomic MM_CONTEXT flags setting Date: Tue, 24 Jan 2023 01:04:45 +0300 Message-Id: <20230123220500.21077-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852696685785034?= X-GMAIL-MSGID: =?utf-8?q?1755852696685785034?= So far there's no need in atomic setting of MM context flags in mm_context_t::flags. The flags set early in exec and never change after that. LAM enabling requires atomic flag setting. The upcoming flag MM_CONTEXT_FORCE_TAGGED_SVA can be set much later in the process lifetime where multiple threads exist. Convert the field to unsigned long and do MM_CONTEXT_* accesses with __set_bit() and test_bit(). No functional changes. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/mmu.h | 6 +++--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/process_64.c | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 4af81df133ee..aa226f451c52 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -317,7 +317,7 @@ static struct vm_area_struct gate_vma __ro_after_init = { struct vm_area_struct *get_gate_vma(struct mm_struct *mm) { #ifdef CONFIG_COMPAT - if (!mm || !(mm->context.flags & MM_CONTEXT_HAS_VSYSCALL)) + if (!mm || !test_bit(MM_CONTEXT_HAS_VSYSCALL, &mm->context.flags)) return NULL; #endif if (vsyscall_mode == NONE) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..efa3eaee522c 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -9,9 +9,9 @@ #include /* Uprobes on this MM assume 32-bit code */ -#define MM_CONTEXT_UPROBE_IA32 BIT(0) +#define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ -#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +#define MM_CONTEXT_HAS_VSYSCALL 1 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. @@ -39,7 +39,7 @@ typedef struct { #endif #ifdef CONFIG_X86_64 - unsigned short flags; + unsigned long flags; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b8d40ddeab00..53ef591a6166 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -181,7 +181,7 @@ static inline void arch_exit_mmap(struct mm_struct *mm) static inline bool is_64bit_mm(struct mm_struct *mm) { return !IS_ENABLED(CONFIG_IA32_EMULATION) || - !(mm->context.flags & MM_CONTEXT_UPROBE_IA32); + !test_bit(MM_CONTEXT_UPROBE_IA32, &mm->context.flags); } #else static inline bool is_64bit_mm(struct mm_struct *mm) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..8b06034e8c70 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -671,7 +671,7 @@ void set_personality_64bit(void) task_pt_regs(current)->orig_ax = __NR_execve; current_thread_info()->status &= ~TS_COMPAT; if (current->mm) - current->mm->context.flags = MM_CONTEXT_HAS_VSYSCALL; + __set_bit(MM_CONTEXT_HAS_VSYSCALL, ¤t->mm->context.flags); /* TBD: overwrites user setup. Should have two bits. But 64bit processes have always behaved this way, @@ -708,7 +708,7 @@ static void __set_personality_ia32(void) * uprobes applied to this MM need to know this and * cannot use user_64bit_mode() at that time. */ - current->mm->context.flags = MM_CONTEXT_UPROBE_IA32; + __set_bit(MM_CONTEXT_UPROBE_IA32, ¤t->mm->context.flags); } current->personality |= force_personality32; From patchwork Mon Jan 23 22:04:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47433 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837604wrn; Mon, 23 Jan 2023 14:06:54 -0800 (PST) X-Google-Smtp-Source: AMrXdXuHSqnl9ebjX5iKdhgHN2F/tALR2ZHiNuPFjaQezlfPgevU2PiXO9ZvO0SJN6am/Jx4Sk1N X-Received: by 2002:a17:906:4094:b0:854:956:1438 with SMTP id u20-20020a170906409400b0085409561438mr56977729ejj.25.1674511604345; Mon, 23 Jan 2023 14:06:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511604; cv=none; d=google.com; s=arc-20160816; b=aZ9N8fwzqZgcoG0lxqpWop+VZmzNZ1cY08BLGxQuWajxzUtzfJRZb4hOKLdqabRDJl knd/M7zuKdHDFlm4gKO1lxPSomkbOs0Ymx9a3I6VCHkLYVNq2DBir+uQl/Qr6kcDWTZF LSpyyh7eikRkYtTiKyLpnf1C7KFUz1+zrHAI4bsNt9LF8HNIlhTJxYKQaQovCw5hKlwm ywI4GThIWL8psjf0EqF4exslyAbEdHsfwZ3L4U+jwtHlg3FNJ7IphL4jxwvtqypEQ8UY FwMgHIaiQTeCcptgw6JvFojduQFSQWCLlFtnYVbX1s45U4SR8zyZion7Q7ORwT0sdObd sHnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xExjoMkzN/P2NAoPJBM1R3czgTsyLyY3dlPRiGQ5Uxw=; b=W2K8kNJtDJjSDl74wFtC0G/a+Hbzhl1lfXPrCxUSrzdsRK1x4qjoChjWC4G7/1C6op pnOaHfuzVeQZCxP0Ja41egLOKyIcP3gOk5NU4g8GBkvFg/W/KUUniR9xUiZ/g81mjWRS mXm4m6tglfWjlI8554+sIOj6aukygMcGeYTlnEUSoKHjnxOVqSE8RGL0rMKUSo/faFE9 HOPjVYiVvRIbfXK3XuuQe7UfO4AH6pyz3GKzxB1tN0C7i1nUmBE3hu/IoOWBaxXkXY9s /d08sDWIaBs8reglZvTVuYAl5ztNcyW0JRgAb/VqnsavLRIzr4XhWJWF/thXvjI8uxg3 pzxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KCrfa8SE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mq18-20020a170907831200b00873092fd3bdsi6751ejc.746.2023.01.23.14.06.20; Mon, 23 Jan 2023 14:06:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KCrfa8SE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232440AbjAWWFt (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232422AbjAWWFo (ORCPT ); Mon, 23 Jan 2023 17:05:44 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 041956A7E for ; Mon, 23 Jan 2023 14:05:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511529; x=1706047529; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Qaw9/4SUZhSsN7dpC0zn5y1iDL4Dr/tJI+fTv1FbMq0=; b=KCrfa8SER2bO6wuTyxbai8u1H4L6IWaiRWA+cRl6qyRLgX7CqGCD1/BR fcLVs61cW9N8CCL6cIIBqEfhNCr6G9h8LTRiN9gmsb+QlAl68bD9y6ICV 1y07uXHgtvf1a+Q0oU7yXs/gNKIZQDss1/5yAowEf9MX9GimvrZ1lVHDb 7PIV8MfYoeRuLmMhvMlbrYgFnw0DgGOer3DNRgFafxt1VP2b3ch5mulN5 sjjg/IeO5nj56sZ2mz9+S2hgCPvWjPMEOLMGoniA9ec7TM52XymfY/Iz2 Ruf9/kFln5Bg1wlf9REMdxnoZ6b//Bvo/sv1jANdMbzOoKYtVOdiv43Ig A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198114" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198114" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103344" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103344" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:07 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 37E2510943A; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 03/17] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Tue, 24 Jan 2023 01:04:46 +0300 Message-Id: <20230123220500.21077-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852680445636659?= X-GMAIL-MSGID: =?utf-8?q?1755852680445636659?= Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. The new CONFIG_ADDRESS_MASKING option enables the feature support in kernel. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/Kconfig | 11 +++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..211869aa618d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2290,6 +2290,17 @@ config RANDOMIZE_MEMORY_PHYSICAL_PADDING If unsure, leave at the default value. +config ADDRESS_MASKING + bool "Linear Address Masking support" + depends on X86_64 + help + Linear Address Masking (LAM) modifies the checking that is applied + to 64-bit linear addresses, allowing software to use of the + untranslated address bits for metadata. + + The capability can be used for efficient address sanitizers (ASAN) + implementation and for optimizations in JITs. + config HOTPLUG_CPU def_bool y depends on SMP diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 61012476d66e..bc662c80b99d 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -314,6 +314,7 @@ #define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */ #define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index a7f3d9100adb..d8cccadc83a6 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -28,6 +28,8 @@ * On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * + * On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. + * * All of the remaining bits indicate the physical address of the top-level * paging structure. * diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Mon Jan 23 22:04:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47432 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837580wrn; Mon, 23 Jan 2023 14:06:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXvuwsYq/QvgB70dNXiF82agQ1/zp6TzbukC6kyO52/gbTU7cWT7+BhvSoBofXTTvabNfKeG X-Received: by 2002:a05:6402:524f:b0:49d:6503:9743 with SMTP id t15-20020a056402524f00b0049d65039743mr36705998edd.9.1674511611203; Mon, 23 Jan 2023 14:06:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511611; cv=none; d=google.com; s=arc-20160816; b=uHAYLhH2XziOXn/QBoHg5+IGGCRHFfKZTICrnJptsHbsYLj6RT0dXtGU2DKcKSsRY2 VnXE3ap9DR0cohhDsXuNFUorpNjYieYPu35DXyMrSeaYKGytZjKZ5vZI5odGQpS64ehH ywjtFFjQ9I/evXq6Njy5+XGHZsbVvHwH586my6eEItrRYkIiQ2kDVeC6uW436mFT9nw9 Idgk4Gk66/6rW7teQheG7UnsOTbOoy98OBu8wdu7L2AYMZldnXSCJ1pJ7jI8ZzoUabte 1OyjmPRxWsEkEUY/kpKEwbafaqMH/q39VcbJsg3wn2uIwaq6GZcc6y1jkk790XIKBvx+ yHpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=61QxZhcR6pYtOXmEW9HiCsWXSE5tEJMLzhCxDLKuynE=; b=qc6/RGfJuy+qb2cmJf3yp38rDP/+/xueA1GY9VQzpHcfF3fzthnNriYrJ8tpVW8uQC yj2Zn/J8O9oIbI0jUnkWuXOiCxm3XmuNz+lxsHBxDoNwQCyChJWey6JcvuCTArmWxsuj zD3VJlOkWJsTd6FEWbU3nEC0TVXBN6Vx+Y0YgLeJ8o6Y4YJsCBlBj2ML4E00b7yIxQRa HeCf658EFIHbPS6DBI1JEfsnSa2GclQn7LnFvRKzxXTtCZCMUk6suv2zeQNH4EGXGAL7 Ol217m8taKbUDsZPMgSEma93ZnYmuuqXmDCEr2ifDRh4GX+KqsAGNH6q/2IsuhHOV3Q+ emqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=E2CrqYbZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m21-20020a50ef15000000b00486c1b099d5si673670eds.3.2023.01.23.14.06.26; Mon, 23 Jan 2023 14:06:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=E2CrqYbZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232422AbjAWWFx (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232441AbjAWWFo (ORCPT ); Mon, 23 Jan 2023 17:05:44 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3651B38006 for ; Mon, 23 Jan 2023 14:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511531; x=1706047531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qIsUK4Bjkv4k1bnjwa0+NfY23qPYFPmF67/Bd/iUBiw=; b=E2CrqYbZJm8jVZHuja2ZaZWgUwDhIEUwH+MWBP6IWkEeBCQQUuT33ZKA m0ozNcS92AW9k0Mww5QA1Z6qdX5IJT8rJLrue+fuugKiwxUDhnNx1xp7o brJHqQdOW0iBnbdhcgBUlIv7jLGa4Lu+SZyqDrTJirs6FSolOs2zcVCbo z7HDZs4f69QdxzKl86p0I78CKayRHXYmY0SpuEqM36PjhsqjP87iGkQtk nJ/jhHRpyRDbgrQdjnctOAAxhXAz32dUntMeCGRkeBweq31QyV5noJJfl O5JEXC0I58qjnmKfH7CGx5efjc0jfa7insxHxeqlpRiEx2X2yZpCj5Kpd g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198122" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198122" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103343" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103343" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:07 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 4328010943B; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 04/17] x86/mm: Handle LAM on context switch Date: Tue, 24 Jan 2023 01:04:47 +0300 Message-Id: <20230123220500.21077-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852687358827948?= X-GMAIL-MSGID: =?utf-8?q?1755852687358827948?= Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t. switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 5 +++ arch/x86/include/asm/mmu_context.h | 24 ++++++++++++++ arch/x86/include/asm/tlbflush.h | 38 ++++++++++++++++++++- arch/x86/mm/tlb.c | 53 +++++++++++++++++++++--------- 4 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index efa3eaee522c..22fc9fbf1d0a 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -42,6 +42,11 @@ typedef struct { unsigned long flags; #endif +#ifdef CONFIG_ADDRESS_MASKING + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; +#endif + struct mutex lock; void __user *vdso; /* vdso base address */ const struct vdso_image *vdso_image; /* vdso image in use */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 53ef591a6166..a62e70801ea8 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_ADDRESS_MASKING +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return mm->context.lam_cr3_mask; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..e8b47f57bd4a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -2,7 +2,7 @@ #ifndef _ASM_X86_TLBFLUSH_H #define _ASM_X86_TLBFLUSH_H -#include +#include #include #include @@ -12,6 +12,7 @@ #include #include #include +#include void __flush_tlb_all(void); @@ -101,6 +102,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_ADDRESS_MASKING + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -357,6 +368,31 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_ADDRESS_MASKING +static inline u64 tlbstate_lam_cr3_mask(void) +{ + u64 lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ + this_cpu_write(cpu_tlbstate.lam, + mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline u64 tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global(unsigned long cr4) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c1e31e9a85d7..8c330a6d0ece 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,26 +154,30 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { + unsigned long cr3 = __sme_pa(pgd) | lam; + if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); + cr3 |= kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); } + + return cr3; } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { - VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } /* @@ -274,15 +278,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +496,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +526,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, + tlbstate_lam_cr3_mask()))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -552,9 +559,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * instruction. */ if (real_prev == next) { + /* Not actually switching mm's */ VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); + /* + * If this races with another thread that enables lam, 'new_lam' + * might not match tlbstate_lam_cr3_mask(). + */ + /* * Even in lazy TLB mode, the CPU should stay set in the * mm_cpumask. The TLB shootdown code can figure out from @@ -622,15 +635,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +705,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +717,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +726,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_lam_mode(mm); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1071,8 +1090,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Mon Jan 23 22:04:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47438 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837981wrn; Mon, 23 Jan 2023 14:07:48 -0800 (PST) X-Google-Smtp-Source: AK7set9xwLlub5iYj9GEbBDFu0I7ARcJVHXOU/PAMTPUtOjo7xAVcq8EQEtgJBd24PAh3T4Gow94 X-Received: by 2002:a05:6402:ca6:b0:49f:20e9:f985 with SMTP id cn6-20020a0564020ca600b0049f20e9f985mr1288703edb.41.1674511668257; Mon, 23 Jan 2023 14:07:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511668; cv=none; d=google.com; s=arc-20160816; b=hDZtQEZ/8ztWhTdacN38Q8Mqxl3jhD7X19AHBQZ1gvBS5g7fuDSIFmh4u1fsmYivjX JPLcCyOMh8Yj1v91vwsQJJUYM4m0nvyx6Js4fg9B+xy0JiXtYpzTGGD4h22WZJf9Cz4J m4m15hvHhV0rJjeD1ty/AqCl6Ytd7VJP87WUu6hx0LkEHGpssM+s2Tn626yg9yJnGga3 h9oWTxQUGPpLSBiWV4t/CAGVf4MINV0yO/4TOymsQQIUjU0e5ArTK0G7M5DEQh7oZrpZ 5TE1H/O/ABCTNqHUp7dMD4kJCPHdPH01EA4WdcZIIBOiKkss/8D3lL9zyUvOSm+ftxUw 7RFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jBFBgJD/6dl6oTna7icjA39z+Ib28+T9yHYDVp4Iaec=; b=fr8cfSzOqaHzts9UqGRWt6OFaFM9ls1ynOF98lkFDP5lIV9EV1G6KLoPmw5TZjd580 KPbOOpqMA0H1In4ScMMMwOpRXsUrC1dZFDXqpN4T8s5aGqsGmdDEqWnjCNptPnOk2M+/ QIqFLydRwD8/deEKznZnHN+0f2w0ESZATPr/1hh8IekMlFhR66naRdGiMnwN7KASO2Qe Ozdi+A4LyVBrhot/PVe3NMAXLg/0I59ksKT+WB5qbprplx8MHZNTBC9LVmGcWDncBidq DTICHKy1tc7ZTFi3ZbpshznB5gY8qoNN99kiNO/i0geThzcJaqE+gk4Mye1NTY9powB1 sC6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=iqhJUat5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cx9-20020a05640222a900b0049d8416507dsi612995edb.292.2023.01.23.14.07.23; Mon, 23 Jan 2023 14:07:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=iqhJUat5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232680AbjAWWGU (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232618AbjAWWGJ (ORCPT ); Mon, 23 Jan 2023 17:06:09 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECA9238022 for ; Mon, 23 Jan 2023 14:05:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511542; x=1706047542; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=x4eTiqqg8GV8B3k8LuKHAxCNwFiXmamwhK2QFN6bvNg=; b=iqhJUat5fjKouqDJDxjTW7x4Iz1qJSF4gAGEuDEMpLvWYB0F1DoUVAAV rs9lc0iQWfk5xceOkqJkbNslTQ2NJpu1WGf65uMfUwySJoYPLGpBGCTcA JtUa6XvqsaH8axnGvneijGHv3CAosBFSvJpgMPmqjJA6mINzhQWbg3Yya s1w8sLzzHjDMHgKe8EEtSGTLq2PFG1lNAuGcWrPfOdfZUF9AS8DTW92G9 0QIj+R5MCRM9ml2AkD0igji/TaMe9Kot5zI+lXRE0kmWzAOT3aVn5r6NF XtJcFKBY2ptVHgoR5QSkCrWWqZh1MidTYbPlq93PiuIQdJ/Zt2e+Oe/Yo g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198199" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198199" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103437" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103437" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 4E9AA10943C; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 05/17] mm: Introduce untagged_addr_remote() Date: Tue, 24 Jan 2023 01:04:48 +0300 Message-Id: <20230123220500.21077-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852747075165863?= X-GMAIL-MSGID: =?utf-8?q?1755852747075165863?= untagged_addr() removes tags/metadata from the address and brings it to the canonical form. The helper is implemented on arm64 and sparc. Both of them do untagging based on global rules. However, Linear Address Masking (LAM) on x86 introduces per-process settings for untagging. As a result, untagged_addr() is now only suitable for untagging addresses for the current proccess. The new helper untagged_addr_remote() has to be used when the address targets remote process. It requires the mmap lock for target mm to be taken. Export dump_mm() as there's now the first user for it: VFIO can be compiled as module and untagged_addr_remote() triggers dump_mm() via mmap_assert_locked(). Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 9 +++++++-- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 22 ++++++++++++++++++++++ mm/debug.c | 1 + mm/gup.c | 4 ++-- mm/madvise.c | 5 +++-- mm/migrate.c | 11 ++++++----- 9 files changed, 44 insertions(+), 23 deletions(-) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 23c24fe98c00..daf34f957b7b 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -573,7 +573,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr_remote(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..df139a717230 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1692,8 +1692,13 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; - if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) { + ret = mmap_read_lock_killable(mm); + if (ret) + goto out_free; + start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); + mmap_read_unlock(mm); + } /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..6b28eb9c6ea2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -96,17 +96,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index afb18f198843..bfdadf5f8bbb 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,28 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(addr) (addr) +#endif + +#ifndef untagged_addr_remote +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr(addr); \ +}) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..3c1b490c7e2b 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL_GPL(dump_mm); static bool page_init_poisoning __read_mostly = true; diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..e28d787ba8f8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1091,7 +1091,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr_remote(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1265,7 +1265,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr_remote(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; diff --git a/mm/madvise.c b/mm/madvise.c index a56a6d17e201..90cd4a442fd2 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1407,8 +1407,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) return -EINVAL; @@ -1440,6 +1438,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh mmap_read_lock(mm); } + start = untagged_addr_remote(mm, start); + end = start + len; + blk_start_plug(&plug); error = madvise_walk_vmas(mm, start, end, behavior, madvise_vma_behavior); diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..dae5022d94b0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1719,14 +1719,17 @@ static int do_move_pages_to_node(struct mm_struct *mm, * target node * 1 - when it has been queued */ -static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, +static int add_page_for_migration(struct mm_struct *mm, const void __user *p, int node, struct list_head *pagelist, bool migrate_all) { struct vm_area_struct *vma; + unsigned long addr; struct page *page; int err; mmap_read_lock(mm); + addr = (unsigned long)untagged_addr_remote(mm, p); + err = -EFAULT; vma = vma_lookup(mm, addr); if (!vma || !vma_migratable(vma)) @@ -1831,7 +1834,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, for (i = start = 0; i < nr_pages; i++) { const void __user *p; - unsigned long addr; int node; err = -EFAULT; @@ -1839,7 +1841,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) @@ -1867,8 +1868,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, * Errors in the page lookup or isolation are not fatal and we simply * report them via status */ - err = add_page_for_migration(mm, addr, current_node, - &pagelist, flags & MPOL_MF_MOVE_ALL); + err = add_page_for_migration(mm, p, current_node, &pagelist, + flags & MPOL_MF_MOVE_ALL); if (err > 0) { /* The page is successfully queued for migration */ From patchwork Mon Jan 23 22:04:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47440 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838086wrn; Mon, 23 Jan 2023 14:08:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXvSGP/u82g4NK3MltJugisEknrf+QtfZU6XV9K6FDgZLka3mQkxayxHbTxFA7F7oJIFXjXF X-Received: by 2002:a17:907:1c07:b0:86f:2819:2760 with SMTP id nc7-20020a1709071c0700b0086f28192760mr44586086ejc.41.1674511683853; Mon, 23 Jan 2023 14:08:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511683; cv=none; d=google.com; s=arc-20160816; b=z8XeMGNFv02P+VOZsvMkLpEGtJDwVR+FA+xcIN87Tlka9wJT3bid9Hyd1p+rFFUF+v MGtyxsSkKo1+V/JmEWHvopZwwC62VoKv4Xas7hv8dDhQaNJst42lDCAHMCF+axvdjqHo dWBLIcEUFjz+aTU+EJMs+OvV4OvrmbRXBP/ZMICgMhRWIVtAGyUoyFOTId8V3Qn40VFm L9m1MK3pnYmkezl1uMiYv8napvPR5kmapgO7GdqPTn4ytfQIcpdzn+nsEOmlIw4XOPgc R6UAm8yT3Be5wXfGM9SXnOv14Z0ZSdS5OpDyYkBzZu8hM7LW0QCHr4ZyYpE+L9QbyrLg ovQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5+B99BlcEo3PWqjXR7YOZtBFOieLV9Tg+g7OpNNtMIs=; b=UjaFIFGppT3eJT41NDFTz2tMaAQTCYrYDCGyioaQsB2VM653H9iZ6h2zHihcLJGqCV QGylbiQzLMGkkyNn/EjLyaL/ZxIMoCcPB13DZ9ROWWHf3sIkyT+Vt/Eek9NP23tvma6i OLexjY0ZigyQxl3xYEesHtgbd60L56ILn/zm81kI7GPH5bNkJVLJJ3tLRT5Qmz8DS9V4 6JTBiA6bYDNbtwQiB8hFQPdhx/DPKUIVt/i+ziaOvqUVobEsg/pFggKmY1L7fuSFXck4 QCfLfQS/RLtHl/ETmVUdy4YLFff1qpwWQtmNsjr6X7cPrBQIdyO+kYRz+uO/ViDUtNsZ vB9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=JAVVk5MW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v4-20020a17090610c400b008699e402607si27512ejv.524.2023.01.23.14.07.40; Mon, 23 Jan 2023 14:08:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=JAVVk5MW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232770AbjAWWG1 (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232656AbjAWWGL (ORCPT ); Mon, 23 Jan 2023 17:06:11 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A9E237F2A for ; Mon, 23 Jan 2023 14:05:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511543; x=1706047543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=S16hikoH0iYKdwUp/S4gZcDbr0rxYU1DuFHDDckKYQs=; b=JAVVk5MW+KA+Kj2PO7EOcFeceAT4USwEhwI+P27zcBqcy5x3HTz5sIl0 Kq1gKlQN68eMqNqA0goSC1n2YjhrNR1UnYweNLe8QD9fZkS93OzpJaz/X ccNMAC2mg1jk50TzAF/u/xxp135Y5DwdYL2DxspusqR2ahTUGxhpwa2qh ZE3fDC/aLrdIPgeGGVKtwTecndYYQZd4oJxNLehmztRFpNh0O51Pc8GOX fXJShWkw83rl14ENG59AjI84Tc6b2gK9wmoytOHuNZcchOU7M2GpxhBfk ezIwsOwny7O9REGu8H9RwA9hiPNI+LJKx2Mm9OWRhh23wiPy8KRgQqdzr g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198210" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198210" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103439" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103439" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 59C7110943D; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 06/17] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Tue, 24 Jan 2023 01:04:49 +0300 Message-Id: <20230123220500.21077-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852763505229429?= X-GMAIL-MSGID: =?utf-8?q?1755852763505229429?= untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape based on rules of the current thread. It only handles userspace addresses. The untagging mask is stored in per-CPU variable and set on context switching to the task. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 +++++++++ arch/x86/include/asm/tlbflush.h | 10 ++++++++ arch/x86/include/asm/uaccess.h | 39 ++++++++++++++++++++++++++++-- arch/x86/kernel/process.c | 3 +++ arch/x86/mm/init.c | 5 ++++ 6 files changed, 69 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 22fc9fbf1d0a..9cac8c45a647 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -45,6 +45,9 @@ typedef struct { #ifdef CONFIG_ADDRESS_MASKING /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index a62e70801ea8..ca2e2ffdd3c6 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index e8b47f57bd4a..75bfaa421030 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -54,6 +54,15 @@ static inline void cr4_clear_bits(unsigned long mask) local_irq_restore(flags); } +#ifdef CONFIG_ADDRESS_MASKING +DECLARE_PER_CPU(u64, tlbstate_untag_mask); + +static inline u64 current_untag_mask(void) +{ + return this_cpu_read(tlbstate_untag_mask); +} +#endif + #ifndef MODULE /* * 6 because 6 should be plenty and struct tlb_state will fit in two cache @@ -380,6 +389,7 @@ static inline void set_tlbstate_lam_mode(struct mm_struct *mm) { this_cpu_write(cpu_tlbstate.lam, mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); } #else diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..fd9182951084 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,11 +7,13 @@ #include #include #include +#include #include #include #include #include #include +#include #ifdef CONFIG_DEBUG_ATOMIC_SLEEP static inline bool pagefault_disabled(void); @@ -21,6 +23,39 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_ADDRESS_MASKING +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +static inline unsigned long __untagged_addr(unsigned long addr, + unsigned long mask) +{ + long sign = addr >> 63; + + addr &= mask | sign; + return addr; +} + +#define untagged_addr(addr) ({ \ + u64 __addr = (__force u64)(addr); \ + __addr = __untagged_addr(__addr, current_untag_mask()); \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr_remote(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + mmap_assert_locked(mm); \ + __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ + (__force __typeof__(addr))__addr; \ +}) + +#else +#define untagged_addr(addr) (addr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +73,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 40d156a31676..ef6bde1d40d8 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index d3987359d441..be5c7d1c0265 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1044,6 +1044,11 @@ __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = { .cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */ }; +#ifdef CONFIG_ADDRESS_MASKING +DEFINE_PER_CPU(u64, tlbstate_untag_mask); +EXPORT_PER_CPU_SYMBOL(tlbstate_untag_mask); +#endif + void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) { /* entry 0 MUST be WB (hardwired to speed up translations) */ From patchwork Mon Jan 23 22:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47429 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837423wrn; Mon, 23 Jan 2023 14:06:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXsfDb95PYo0O5/sun3IxCWj80HRmvXOsD20/BQpqo8Gq/GPTS52x8+A7Q7/uRwln5Y0ZmBa X-Received: by 2002:a17:906:8d0f:b0:870:b950:18d4 with SMTP id rv15-20020a1709068d0f00b00870b95018d4mr29913073ejc.5.1674511590313; Mon, 23 Jan 2023 14:06:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511590; cv=none; d=google.com; s=arc-20160816; b=rrXky0jqyZi1wS4CS7rro8T62FPu66tQJ7UbYxUNbh7hGkVdQtEYmYnwl20JyK88ZS Luh3lOAaGbC2/6GuDceSjMuTHjWR4tzXk88Kt0NgZ8ahBH81Ir1Pp1F31W6fnn6UQJQx kLr3nfob96kQ5CY2c4lGOJAJp3kDy5T1pkTys+bbFBuNQUzN+WTMEWA4pY3EzLxmTtdK N/Wn6EETdZ1o7eGTMd7w3zGbQMxBa8teEFzIlCPGosZrGfHl3yuxEkitdqR8yCDxoTRf lXLqP4WEvEOc9ZHUglNDdrOzrHHIcI8NY9RoEyTGoIi6R8yPX0VuM4Q6nv6mD5mmhRQl h9Bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HdRYkzOlUG43YjkCu8PsfJrpwpJ/D/N4XwLO2lELylw=; b=t6ChbODLXsQWnXUrkbGwjocR9y/cv87xvjv7SeroVRjIyNUDidPIN3Zbb9Dfh0Z1BL 6Mm+khdVFmLfIsJ9m9QSqYMUFOIHBLggD46nAQv96f0ycECuHG2kdPeYmV9CflX1I9F6 bi86gK0HIy3pTfqamv6j3+KIgO84fgAVDVTTnh94oxRSCnrScL8U2cirPjpTL+kjqo+e hTg4KYmQies3KXN1ysSx12AjatRFeJgngtXjA3oBExzwHZXt5Hrxf8njEKNJA53KJHwP 591BvD1XbV9exAu3/h7zivJCq+4U376K4hqSNDEg7nRLiCwliYO1Lv9Sk2DN/iFTf/XE LbiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IdwpCIgh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fw31-20020a170907501f00b007ae1874c142si32853ejc.446.2023.01.23.14.06.07; Mon, 23 Jan 2023 14:06:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IdwpCIgh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232299AbjAWWFf (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232401AbjAWWFa (ORCPT ); Mon, 23 Jan 2023 17:05:30 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCFA638035 for ; Mon, 23 Jan 2023 14:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511523; x=1706047523; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=focBDCcui25Nna3zw6dN8ZwvWQ0ckdMqYvOG7S60BQo=; b=IdwpCIghC1+YrAHNrn+4T4mrad+Qvpxu5DkiFDxDkxo5TPGqqmNPSYlR bsZEVM9Q8vRKx0lNu7BxtETUTmgThzv7OOkkEvjaSGLcm8kYzdAt9P1M3 m4wWm5Dv3Tze7hn4xFTulzvE8m/TaH0PRX0vShBI0HMYQbj800l+60Wo7 pAlvBslQuvIcEHk4aRbivU+j45hqQYcj/Ul0HjmIgaro1oroLrcPYrGot mAH9Jnk39sRFrbUUdwce80hFysL7du3pXdIxH+PN0Q/zAzQ3i1JmgqIN3 3tkKNBDZfNLxLI2m9E38+abfbn3R4bwzuqlQnKYR0h2cBEbaKXGW8Qs87 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421936" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421936" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878108" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878108" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6779A109478; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 07/17] x86/mm: Reduce untagged_addr() overhead for systems without LAM Date: Tue, 24 Jan 2023 01:04:50 +0300 Message-Id: <20230123220500.21077-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852665271876269?= X-GMAIL-MSGID: =?utf-8?q?1755852665271876269?= Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/include/asm/uaccess.h | 41 +++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c44b56f7ffba..3f0c31044f02 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index fd9182951084..6450a2723bcd 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any branches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign = addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); - addr &= mask | sign; return addr; } #define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ - __addr = __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else From patchwork Mon Jan 23 22:04:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47430 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837455wrn; Mon, 23 Jan 2023 14:06:35 -0800 (PST) X-Google-Smtp-Source: AMrXdXtfGgBfXahGrWm1/5hItzDJmaz7cbs+I6EUA4GFdSO7GiT92gQRiJn/NjPJpc7/eYgP7bAJ X-Received: by 2002:a17:906:3f87:b0:809:c1f4:ea09 with SMTP id b7-20020a1709063f8700b00809c1f4ea09mr25370829ejj.69.1674511594825; Mon, 23 Jan 2023 14:06:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511594; cv=none; d=google.com; s=arc-20160816; b=VQKoSor6WFZNRGzeRtczAPPIxVi1eCApPuR+7hwTDexDEKWbELyhaD9u7Tvhvz8oXi W6TfrFDZ0b/eUF7h2GE6dThHuG7EWGcWJhachogGaPkAKAHlROt9KjIU0CJmyiVxe2kA fvHI1PpHLviTe0HdoRDfuWbA4Fx8BtHmocg5prAM8/RC3AB9dIlnRzUazbXSMu7hCX9/ U0O0mfn9D7Xe2N/1YTgyqBeUheZNR7ICoyqOy5QXfBYK/Eop/UtFPJ5lgPNKwboFWtG6 3D+u9pEMT7AEhXZ8s7Ti2wCm6j5wfA1rcWlc6GPCrPQk+l9Rbr9aMyqpOUBppnHyQUSx DtCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TQF1tdq01mdscIRxQekmBHdkXQlI68Nw30IEMqdgrEQ=; b=Q6ZHgmrEWsV18Jo7pl1RZcGxVv/lW1OJSEA4+R6ghClq25NpAirHdOiw1A38XSXYvp E5A6RkH9LW53JkermDGm1mITLC/Mj1Fqk6Pc9049gJHI7ZKn1ipal/XZWml8PRZsuTh1 +avTPg6jK6r6N4ml3E3NlNc7DezzFGzmB4Zf8HFM5+lVXPoDauSxYyBSHHgGsBYMOJ95 Y/ak2wju+i+rFGNaqHoHeM2fDWXAeuDsWzKc5zyqdapax/kHNyPJRc4P5FljyGEMpp3S CdES4IiHebjKRS2oPXyfo/DiH+oI7/vc15hk6ilmz4Rah6UG9AhqdJ3acu1fyocOmrKI 07PQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="S+EL+mq/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vl14-20020a17090730ce00b0084d1310711dsi35236ejb.315.2023.01.23.14.06.11; Mon, 23 Jan 2023 14:06:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="S+EL+mq/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232360AbjAWWFi (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232284AbjAWWFg (ORCPT ); Mon, 23 Jan 2023 17:05:36 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58394B774 for ; Mon, 23 Jan 2023 14:05:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511525; x=1706047525; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=goQpwLMTaN14VQUbpuKQyEVCPJNPBRQVQcckNmVxg+o=; b=S+EL+mq/K+fGAkpJXFolcKoG+HY7YnlwjOODdz7/rvzwEd8r56GS1rEh 7k0g3QCSOvRJze8l3Itt3UUkMCZk9BHEFxAZYFX5ep5oTUaPJ/DrEQZKv ksUPwa6G6XvO6O0An6Eb4QmLbBcxhH7QxYzifryu71d6nDNA6oYc1guGt RzkfIKYpn7AJE/hT8sKKvh6roVWz+NkozCPvHIja8W5GPT9w3MK7xbJT0 Kmi54TU2MVU2vMEbK/RiQJhG5Yr2KnLkc004XY/GEAKcgM3ASaq+5VBV6 bw9925/qwf8NaaP70KAoXdXxWE/Vo3MU9i6aiim78tVNzpHgjW074AQKv A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421946" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421946" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878110" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878110" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6EBC3109479; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 08/17] x86/mm: Provide arch_prctl() interface for LAM Date: Tue, 24 Jan 2023 01:04:51 +0300 Message-Id: <20230123220500.21077-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852669931728211?= X-GMAIL-MSGID: =?utf-8?q?1755852669931728211?= Add a few of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. - ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/uapi/asm/prctl.h | 4 +++ arch/x86/kernel/process.c | 3 ++ arch/x86/kernel/process_64.c | 55 ++++++++++++++++++++++++++++++- 4 files changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 9cac8c45a647..e80762e998ce 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -12,6 +12,8 @@ #define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ #define MM_CONTEXT_HAS_VSYSCALL 1 +/* Do not allow changing LAM mode */ +#define MM_CONTEXT_LOCK_LAM 2 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..a31e27b95b19 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,8 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index ef6bde1d40d8..cc0677f58f42 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -162,6 +162,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) savesegment(es, p->thread.es); savesegment(ds, p->thread.ds); + + if (p->mm && (clone_flags & (CLONE_VM | CLONE_VFORK)) == CLONE_VM) + set_bit(MM_CONTEXT_LOCK_LAM, &p->mm->context.flags); #else p->thread.sp0 = (unsigned long) (childregs + 1); savesegment(gs, p->thread.gs); diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 8b06034e8c70..88aae519c8f8 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,48 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +#ifdef CONFIG_ADDRESS_MASKING + +#define LAM_U57_BITS 6 + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + /* PTRACE_ARCH_PRCTL */ + if (current->mm != mm) + return -EINVAL; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (test_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags)) { + mmap_write_unlock(mm); + return -EBUSY; + } + + if (!nr_bits) { + mmap_write_unlock(mm); + return -EINVAL; + } else if (nr_bits <= LAM_U57_BITS) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + mmap_write_unlock(mm); + return -EINVAL; + } + + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); + + mmap_write_unlock(mm); + + return 0; +} +#endif + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -830,7 +872,18 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - +#ifdef CONFIG_ADDRESS_MASKING + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_GET_MAX_TAG_BITS: + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return put_user(0, (unsigned long __user *)arg2); + else + return put_user(LAM_U57_BITS, (unsigned long __user *)arg2); +#endif default: ret = -EINVAL; break; From patchwork Mon Jan 23 22:04:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47442 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838639wrn; Mon, 23 Jan 2023 14:09:24 -0800 (PST) X-Google-Smtp-Source: AMrXdXsjbzb4Y6SvN0PTE7Y95fk9vfuabHcdtK2uMBNets6dzaT19Ey+45rYso7sq3pDLRcQ/Msb X-Received: by 2002:a17:906:7f03:b0:86e:eccd:ba61 with SMTP id d3-20020a1709067f0300b0086eeccdba61mr32456761ejr.66.1674511763942; Mon, 23 Jan 2023 14:09:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511763; cv=none; d=google.com; s=arc-20160816; b=f/Jwvuei+czPvZo6et/icl7u9ZO9OzGU2UM7B4cSgayI0aa0S49U6rg75v8y5nSMG+ TpSJAsByh9Vc/pYHldm2yWO8C0BcLO/GENhtepleW/eLZBUn8XW9UiFUFhANHDwccRyG PIR9vAfVUbJjZpb7XDNNXXjhUxqAHXQ72zbVENw3VK8Oy868pKvVzBYRTGRgnqOGaZiP w6afCGIbPuc4B939+TrpYGWjZ/rmca9PKykyVmPObCTML+u/Kbuvg58PHQ5CdC+9E5hK ze1fEpiikCe3OE36Mv3LYcQDs3CfuNL5GinZQMuAZEXvvHniky9hV5i465xucL8/jliW esrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EFiuqEF1AJ35Zpy8Sw/UigXH2q9I0vE41W1sxgJG5sg=; b=bsK8AoUyyIBDt86nEAvh+wRCk8GIY24SDbapm3r+zsKCoyw0G6t19Wf8v0iG0LW85r eGtpyMvsb8EXeFAy1ClbR5BVFO4u7Hq62fSSiOY7DA8BQkPxHj1ftU++ompR1bHQbkWl giF5WX+DNM5vL78nP60gMNcCY+Wzct4/o3SZbxbamxGLkGJE8ZYo93zmooY02fEzewgG BFwMgYBzt+l5WSjUcRDqLKqU8DgVusNf0kkUy1+XjKn7RpdhpTvMkQiEKb4kFJeOMFQX mnTXA9NPdoev290gM0yNTcrdQxJ35vnOJHvVrBxqaZCVFZNZDTt8sqJtRJZjgvTnulIK EQOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RjPuQaSm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l3-20020a170907914300b00877a940e10dsi54401ejs.248.2023.01.23.14.09.00; Mon, 23 Jan 2023 14:09:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RjPuQaSm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232823AbjAWWGd (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232590AbjAWWGO (ORCPT ); Mon, 23 Jan 2023 17:06:14 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BBDA38B6E for ; Mon, 23 Jan 2023 14:05:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511546; x=1706047546; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OVB8ixW2SlvKXCjvUsAuiBNx4ctXMcDQHlcvi2JSoRw=; b=RjPuQaSmG+Vd9dZgYITMB4H1uDXuHB+5JB25CN/eoRX7AMlaNvfhguEp DAcErMugXKiq0Mo/tbNYnT5BSq1s0Elthdbdr27mRNLIb8ZkRiXg3ynPA Dvk7VRyAn3ieKYy7eKXkQQL7drRHiK7uCHSuMPlVZ9xHWTFVXFbcDhdOE FvD9jDjzxXa+F9Of3h8ds7f5uurnStspN9ZGDkfcD6ivtlPEVR2+093HS fSp3XmkiK5dd/K+yPNhUeDbBDblZR1DJmt8XUn0ZqFN+dlWTrdeo25B/G WD+ae0r70R/clqVqRVgyh3O7DkLrH1UdlOQmZxrCzz3SMLSYhVgs68+Ze Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198213" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198213" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103441" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103441" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 79A2B10947A; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Catalin Marinas Subject: [PATCHv15 09/17] mm: Expose untagging mask in /proc/$PID/status Date: Tue, 24 Jan 2023 01:04:52 +0300 Message-Id: <20230123220500.21077-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852847406573169?= X-GMAIL-MSGID: =?utf-8?q?1755852847406573169?= Add a line in /proc/$PID/status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Acked-by: Catalin Marinas Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/arm64/include/asm/mmu_context.h | 6 ++++++ arch/sparc/include/asm/mmu_context_64.h | 6 ++++++ arch/x86/include/asm/mmu_context.h | 6 ++++++ fs/proc/array.c | 6 ++++++ include/linux/mmu_context.h | 7 +++++++ 5 files changed, 31 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 72dbd6400549..56911691bef0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -288,6 +288,12 @@ void post_ttbr_update_workaround(void); unsigned long arm64_mm_context_get(struct mm_struct *mm); void arm64_mm_context_put(struct mm_struct *mm); +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> 8; +} + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h index 7a8380c63aab..799e797c5cdd 100644 --- a/arch/sparc/include/asm/mmu_context_64.h +++ b/arch/sparc/include/asm/mmu_context_64.h @@ -185,6 +185,12 @@ static inline void finish_arch_post_lock_switch(void) } } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> adi_nbits(); +} + #include #endif /* !(__ASSEMBLY__) */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index ca2e2ffdd3c6..43eb6b1357dd 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -103,6 +103,12 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..d2a94eafe9a3 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +static inline void task_untag_mask(struct seq_file *m, struct mm_struct *mm) +{ + seq_printf(m, "untag_mask:\t%#lx\n", mm_untag_mask(mm)); +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -443,6 +448,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_mem(m, mm); task_core_dumping(m, task); task_thp_status(m, mm); + task_untag_mask(m, mm); mmput(mm); } task_sig(m, task); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index b9b970f7ab45..14b9c1fa05c4 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -28,4 +28,11 @@ static inline void leave_mm(int cpu) { } # define task_cpu_possible(cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) #endif +#ifndef mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} +#endif + #endif From patchwork Mon Jan 23 22:04:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47436 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837745wrn; Mon, 23 Jan 2023 14:07:14 -0800 (PST) X-Google-Smtp-Source: AMrXdXtVaVMy7rMcCPfLeZLFlauDp6FECq4owUXutr7uEUfh9eQeFD1UiMTuwvvuIKe5T041k78K X-Received: by 2002:a17:907:c307:b0:816:ef2a:631a with SMTP id tl7-20020a170907c30700b00816ef2a631amr31100832ejc.31.1674511634167; Mon, 23 Jan 2023 14:07:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511634; cv=none; d=google.com; s=arc-20160816; b=ZQZWPMMeK+t+k7cczncQ+cc84dHbaqrAQc2Bn+TGnSFOZTfHicg5tM+Si63t3zpeDk hmtEUdcCyf5z9y30JvUYJhAGst8oP2nZzVS2FeSlaRatldMGYB/4hNyTtGegEUt2CHFW ub4h9Z3lrLTvZzBsORuJPx+/VZbHdDe/8wNpeRRiVkY4a6Yip+1wRnapEMUKNvfIHrns JoU27yHNTno6drkomCL9siAUuWBNx+eOlSDPbv93sQ+hiNsXaco2am6gh0YvMJou10IO 6CKfE9fJ+6ZXydEsOkhTJyu9lFnYa3YnanBFNHU7zuo8Yurtl3DazdXhDKMIlI2bStY/ 5YPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=S9DsTRd50dZdn1Uk8iWIuFqBLWLxK9qmStCBqOt6qmk=; b=dqBCM2tQbLpisPdrHdg6n1XPt1MSy9MWZnYEZkMgmNgBzF4ZIEifcgSw3ckkLx2P54 QePgVo8wRsh1jJDES3SkRiaK+zn5bDBO/TGId0stortJAmp9fM3wdDkWrlAwp4IqoHym q2hq4rf/g9JIE26Dncx+10jWCr/M7U7BVyLs0R7lQ4So3zNeNJ4pHiHpd/GgWACebvV2 xKODVbd0A7QH1HELCzY1OBa2fz1NK+9iXCNRddfAtHstSkgEx/UQ3+YsMhvbKonvF1il CIp8F3dfaqJU2iy9fWNdh/LiYj8Wq5SJcWSlN3uMxAc1p7KPlPTn/6lYPCsbZ4TrCiP3 7LDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=l+7WRgAb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gi37-20020a1709070ca500b0084d2c9af2e9si302ejc.828.2023.01.23.14.06.50; Mon, 23 Jan 2023 14:07:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=l+7WRgAb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232042AbjAWWGJ (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232459AbjAWWF4 (ORCPT ); Mon, 23 Jan 2023 17:05:56 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80D8037F3B for ; Mon, 23 Jan 2023 14:05:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511536; x=1706047536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=q6cZBJye+9S/4/Mk+1QQktyLKF2mTMV32eIG7PWZWnw=; b=l+7WRgAbA4j36oF0Sm4/2JFPGqTavUUSanNln8Z/rhMphRy2fYuWnSIj b1zHurIYtctgU/vaARgimuqoD2vVhQK1b3/XpMa+p4XJDYWsUmc8+2jsZ JrTybwaXrNdV5TwSrfUDPvw3weQ4utmB0Mj/BLi9oCMBUJV2Qxc9/b6gP +OpzFAkq3qE1ihCbGj5GdDzjVNobhrlBDJPwZh/sUPIncW+h3cG60fzqO fE6t9jlOWsePRWQrDihCl+ySy6eIJ+Eoyfnhyq7bxoThKAudxV08p6cYW pBZ7BWwJd6svKwpzi3YK0mx+ue0/7LasZzvUv9oYlxWlhUV+F67VGAjMe A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421971" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421971" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878116" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878116" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 8502610947B; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 10/17] iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() Date: Tue, 24 Jan 2023 01:04:53 +0300 Message-Id: <20230123220500.21077-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852711538948828?= X-GMAIL-MSGID: =?utf-8?q?1755852711538948828?= Kernel has few users of pasid_valid() and all but one checks if the process has PASID allocated. The helper takes ioasid_t as the input. Replace the helper with mm_valid_pasid() that takes mm_struct as the argument. The only call that checks PASID that is not tied to mm_struct is open-codded now. This is preparatory patch. It helps avoid ifdeffery: no need to dereference mm->pasid in generic code to check if the process has PASID. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- arch/x86/kernel/traps.c | 6 +++--- drivers/iommu/iommu-sva.c | 4 ++-- include/linux/ioasid.h | 9 --------- include/linux/sched/mm.h | 8 +++++++- 4 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..8b83d8fbce71 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -671,15 +671,15 @@ static bool try_fixup_enqcmd_gp(void) if (!cpu_feature_enabled(X86_FEATURE_ENQCMD)) return false; - pasid = current->mm->pasid; - /* * If the mm has not been allocated a * PASID, the #GP can not be fixed up. */ - if (!pasid_valid(pasid)) + if (!mm_valid_pasid(current->mm)) return false; + pasid = current->mm->pasid; + /* * Did this thread already have its PASID activated? * If so, the #GP must be from something else. diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 24bf9b2b58aa..4ee2929f0d7a 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -34,14 +34,14 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { if (mm->pasid < min || mm->pasid >= max) ret = -EOVERFLOW; goto out; } pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); - if (!pasid_valid(pasid)) + if (pasid == INVALID_IOASID) ret = -ENOMEM; else mm_pasid_set(mm, pasid); diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index af1c9d62e642..836ae09e92c2 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -40,10 +40,6 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); void ioasid_unregister_allocator(struct ioasid_allocator_ops *allocator); int ioasid_set_data(ioasid_t ioasid, void *data); -static inline bool pasid_valid(ioasid_t ioasid) -{ - return ioasid != INVALID_IOASID; -} #else /* !CONFIG_IOASID */ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, @@ -74,10 +70,5 @@ static inline int ioasid_set_data(ioasid_t ioasid, void *data) return -ENOTSUPP; } -static inline bool pasid_valid(ioasid_t ioasid) -{ - return false; -} - #endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..b69fe7e8c0ac 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -457,6 +457,11 @@ static inline void mm_pasid_init(struct mm_struct *mm) mm->pasid = INVALID_IOASID; } +static inline bool mm_valid_pasid(struct mm_struct *mm) +{ + return mm->pasid != INVALID_IOASID; +} + /* Associate a PASID with an mm_struct: */ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) { @@ -465,13 +470,14 @@ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) static inline void mm_pasid_drop(struct mm_struct *mm) { - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { ioasid_free(mm->pasid); mm->pasid = INVALID_IOASID; } } #else static inline void mm_pasid_init(struct mm_struct *mm) {} +static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; } static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif From patchwork Mon Jan 23 22:04:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47431 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837479wrn; Mon, 23 Jan 2023 14:06:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXvJWOV88javRKYhlSKu2tcrRWoUSprQk3fL90SMD8VOdidRsKBjYEoJfB7ObCSMF4qg+CR4 X-Received: by 2002:a05:6402:138b:b0:48e:a9a1:c584 with SMTP id b11-20020a056402138b00b0048ea9a1c584mr27108177edv.35.1674511597169; Mon, 23 Jan 2023 14:06:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511597; cv=none; d=google.com; s=arc-20160816; b=vlZp5CWi1LHvnw8EB9yPNbvwE2emfhFTopM0xA1S2kjcLZE4SkfUXvpC3NO8E3DJgp 0EQSIEdDg839gDlDLfFGbJ7qnPcEMb3K5/6/HYoJhMMRXRbiZmVGo+H4uMyDmdLpXkBw CxBIDUJX/q7p99EWZqBKo84f2EJl3zqXjum9Ibu7jQPW4UlCxirye6ASQnCBw5zhqQ+t FJB1+mOnlnK/pg6BlGRDD9G8XxX99R55xfClMh/bF8BYv2IuH4oO1x54QR9XuGV8HfIL P/uIXp2t00RLh9efquXBbrPhMRvz+kX+9UkT0faKMZM1/gbGyr8v14NOnWtyl3HYETY2 xiww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LVsKenYT3mFv97dtFWuoiYTKKtzno/tEciQ1BDxQMp4=; b=0mZz5AyFOYhS8ik++MLFMPOWWxt54SEaLAb+WdOCMDNoph2CMIuubvkQJtRQx8+pfH lWq4VTsd9hybexBGERR4wMipLATAIXJVUqTNAZXGYApcWfIVThuRpxYw9fzvltZfDXrV D42EochgXmxO54wpag9zQ1/B5ehzfADGe6EyVs1GcQr4/LSMLrqOtYio7Ovzq5TG98+Z Fmj3fHeLyGNFpdY/S5sV3CPFToR56OWCJqT5zfzKBTJ1gRpBfhUuvZXpxraMprBNpS9w 3N+AcmUEdB2fjgTdMCux6TmAggw6QDWU+0JXZfsEiK6YxtSVCmsAPb2W+490vD8sxg98 kcUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=d5A0cx+m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r2-20020aa7cfc2000000b0048612395228si645721edy.44.2023.01.23.14.06.13; Mon, 23 Jan 2023 14:06:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=d5A0cx+m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232410AbjAWWFr (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232341AbjAWWFo (ORCPT ); Mon, 23 Jan 2023 17:05:44 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B612737B65 for ; Mon, 23 Jan 2023 14:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511530; x=1706047530; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zJfY0kFY/EZxC+6SwRsnbg7CP0YtABdL7GCv8v5SU4Q=; b=d5A0cx+m42J2/3gzG/iagocYhySRHIf7xY2AN2e9um2Y4hSuyhhCMog4 Tmrl1kLuNiNjTDzc17Q1LuEOb27ln3Q/9EAgKYMclxfNiVPyaYrP9Vm5+ ubSHOeFEgnb3uHjNZcXXZRhuNgYxy4/CH/+/ZzhiYaSJdNTQQCHUDhn5/ sjlxf4itQldC3IEWTOL0f+BzdxzuEdXM4kJ58j/e83yUJOq5t/HY2tQfl hTtbxtZg1XncICMavdLTjvLe1REoYkD/B/2GxpFXIOSRzH8O83Y696r2K evfpog5wofD3IW4XvtmTY68e/vHi2GF/slGvvSl2Pe7M37ebqW9Qwku+f g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421954" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421954" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878112" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878112" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 8F35810947C; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 11/17] x86/mm/iommu/sva: Make LAM and SVA mutually exclusive Date: Tue, 24 Jan 2023 01:04:54 +0300 Message-Id: <20230123220500.21077-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852672926695213?= X-GMAIL-MSGID: =?utf-8?q?1755852672926695213?= IOMMU and SVA-capable devices know nothing about LAM and only expect canonical addresses. An attempt to pass down tagged pointer will lead to address translation failure. By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. By using the arch_prctl() userspace takes responsibility to never pass tagged address to the device. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ashok Raj Reviewed-by: Jacob Pan Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 7 +++++++ drivers/iommu/iommu-sva.c | 4 ++++ include/linux/mmu_context.h | 7 +++++++ 6 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index e80762e998ce..0da5c227f490 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -14,6 +14,8 @@ #define MM_CONTEXT_HAS_VSYSCALL 1 /* Do not allow changing LAM mode */ #define MM_CONTEXT_LOCK_LAM 2 +/* Allow LAM and SVA coexisting */ +#define MM_CONTEXT_FORCE_TAGGED_SVA 3 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 43eb6b1357dd..e77a8774cf29 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -114,6 +114,12 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) mm->context.untag_mask = -1UL; } +#define arch_pgtable_dma_compat arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return !mm_lam_cr3_mask(mm) || + test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags); +} #else static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index a31e27b95b19..eb290d89cb32 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -23,5 +23,6 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 88aae519c8f8..eda826a956df 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -756,6 +756,10 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) if (current->mm != mm) return -EINVAL; + if (mm_valid_pasid(mm) && + !test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags)) + return -EINTR; + if (mmap_write_lock_killable(mm)) return -EINTR; @@ -878,6 +882,9 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_FORCE_TAGGED_SVA: + set_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &task->mm->context.flags); + return 0; case ARCH_GET_MAX_TAG_BITS: if (!cpu_feature_enabled(X86_FEATURE_LAM)) return put_user(0, (unsigned long __user *)arg2); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 4ee2929f0d7a..dd76a1a09cf7 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -2,6 +2,7 @@ /* * Helpers for IOMMU drivers implementing SVA */ +#include #include #include #include @@ -32,6 +33,9 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) min == 0 || max < min) return -EINVAL; + if (!arch_pgtable_dma_compat(mm)) + return -EBUSY; + mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ if (mm_valid_pasid(mm)) { diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index 14b9c1fa05c4..f2b7a3f04099 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -35,4 +35,11 @@ static inline unsigned long mm_untag_mask(struct mm_struct *mm) } #endif +#ifndef arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return true; +} +#endif + #endif From patchwork Mon Jan 23 22:04:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47434 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837605wrn; Mon, 23 Jan 2023 14:06:55 -0800 (PST) X-Google-Smtp-Source: AMrXdXs1TtGVZ0zINqzY6bPxpxEIwq9S3DDqeLdRG/S4doOA6hHYMwevHEs8O6ud+YB7yez0DdVo X-Received: by 2002:aa7:c30b:0:b0:47c:4479:d60d with SMTP id l11-20020aa7c30b000000b0047c4479d60dmr27727534edq.20.1674511614816; Mon, 23 Jan 2023 14:06:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511614; cv=none; d=google.com; s=arc-20160816; b=U3zI1rWQ0X7CLewqInhFe/5VEMxRBZOre1drjvdMfaY6XSb+qtbRLAriq8u7kr7bBE DeO4SAP5tuXb4NjoKNjwQW+a4zWro+ZahiLLRkle3+QWpwIhh5SPyv/ydzVoPg6eSVBS Cv8PB9Mv9OgdQO9BGwcwa6GNexlWpkeDPVhE1KjbeKLv0fXsXglH4CcIWlwRADFhSQmc jeIkv8KSas4WYAPvsFgw7wC9wwrCCG8Qgqi7XDjyUwaVCRjUksQkGoz9sjcwP17OkkLc Ykzmf9+HArbCr4WSsEaLInriKK9DWjktnyOVRVBmnbhmSh5UaZtlaYJnOFHzq7soVM1K NwWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0z/p0YZ8d9b0ubykNIIv1oWuyLJjIvasC7sYYDb8pDI=; b=fIiVuYSIDEWzEftRQkwRAjThsSB/UJpad8qD7OcHBl5BNRcGAtgvi2qsmQ0NEf1JaF zR/x8TgAsZE/6C0PVG64l4Ylg2ileJkC69nBA7tEE2PBwCq6e6zeshT/o4wxwVomEqUL ugHgwh1tFPJmpe90T48+GLLevGVIZf/9lEE7ICn/ht4WUJEscXLyVpuwBgWozyI1C4br xgeTg8tPCe5tSf+o8hu3hDZVjlZQvgTrxFUACwpp/ZwuX04vAxIZoHmaDfoSumtt1hcs 0b/KIOyhiczs3o6DUKJKz9juI5vj1JdjwGjLelyqcepMumQbw+ssqXQlZhqEA655Hw3k aWiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ixs4IWcD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j18-20020aa7ca52000000b004763231785asi707775edt.33.2023.01.23.14.06.31; Mon, 23 Jan 2023 14:06:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ixs4IWcD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232490AbjAWWFz (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232450AbjAWWFo (ORCPT ); Mon, 23 Jan 2023 17:05:44 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 364602B2BB for ; Mon, 23 Jan 2023 14:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511531; x=1706047531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fwizFXa+ntJFgk9BjkRzRTV74GE4SUYwl24HZQzzh/A=; b=Ixs4IWcDceLptO5gPnABcNwME3ADawKOob3KblxR5uwaUBPSFYBFOW4h B+If6reiTrjYyibT6b/Bg7624uKIR9A5PspxqSuvuCbTLjIh1jIBO9UQl RfmglmhYFwFesWkx6lBv3Hsf+kXKOPhhYE0GoqmfYb0XBwgR/oH+lgbpn 8vxthaqptN8ppKxkgEdtzsaIaUK7yLS8tPPHFM7hTDYJc/L2x2u6tCXDv akeMIN14GNZZ8S5iLIe7fKXrKJ4zcgD7WK2IOuyXwXrulnq+vJbhlmcRd kpdo4k5iw92lcjbsOvPFpbUJyAY/OKPUsfqm1cjUoBMxsyO8yFbaNp6Z5 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421964" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421964" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:24 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878114" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878114" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 9A6F110947E; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv15 12/17] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking Date: Tue, 24 Jan 2023 01:04:55 +0300 Message-Id: <20230123220500.21077-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852691066062001?= X-GMAIL-MSGID: =?utf-8?q?1755852691066062001?= From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are some arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: MALLOC - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. TAG_BITS - Max tag bits of LAM_U57 is 6. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 326 +++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..c1a16a9d4f2f 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..268c1d2749af --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 + +#define LAM_U57_MASK (0x3fULL << 57) +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + * + * @return: + * 0: Set LAM mode successfully + * others: failed to set LAM + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(LAM_U57_MASK)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +static unsigned long get_default_tag_bits(void) +{ + pid_t pid; + int lam = LAM_NONE; + int ret = 0; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + } else if (pid == 0) { + /* Set LAM mode in child process */ + if (set_lam(LAM_U57_BITS) == 0) + lam = LAM_U57_BITS; + else + lam = LAM_NONE; + exit(lam); + } else { + wait(&ret); + lam = WEXITSTATUS(ret); + } + + return lam; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t set_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + /* Get a random non-zero value as metadata */ + metadata = (rand() % ((1UL << LAM_U57_BITS) - 1) + 1) << 57; + metadata |= (src & ~(LAM_U57_MASK)); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + * + * @return: + * 0: value on the pointer with metadate and value on original are same + * 1: not same. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)set_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + + +int handle_max_bits(struct testcases *test) +{ + unsigned long exp_bits = get_default_tag_bits(); + unsigned long bits = 0; + + if (exp_bits != LAM_NONE) + exp_bits = LAM_U57_BITS; + + /* Get LAM max tag bits */ + if (syscall(SYS_arch_prctl, ARCH_GET_MAX_TAG_BITS, &bits) == -1) + return 1; + + return (exp_bits != bits); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg = "BITS: Check default tag bits\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + if (tests & FUNC_BITS) + run_test(bits_cases, ARRAY_SIZE(bits_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Mon Jan 23 22:04:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47444 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838711wrn; Mon, 23 Jan 2023 14:09:35 -0800 (PST) X-Google-Smtp-Source: AMrXdXt3vJOxuHO/+S9ktbtW2FRv593nlK3iHaotRJKqvM3P2eEPTWZolKfLe4oHO4rcGN3ftGMh X-Received: by 2002:a17:907:c307:b0:816:ef2a:631a with SMTP id tl7-20020a170907c30700b00816ef2a631amr31107760ejc.31.1674511775131; Mon, 23 Jan 2023 14:09:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511775; cv=none; d=google.com; s=arc-20160816; b=eCOkORW6HKrRHRpR65g8+sh7J65qKv3GlNHzkcigDLzlBbb/QKHF+jejkxchkuS731 at+9WpnZ+Wc/5dND0MUnjHm5sFYhjpB4/QixY2xHH+iynVuAxQufvY8FOFNZ+/HdFnFh dhbfwGNmRv5P5UO4MBpJdY8UgO+tPDYwuO5q0yNOTxAj+FSbaGuOLPCtttc2HVWPMlsl swllGSOkifGd9F4Doi/vc8KeGahwvfYGySDSPrLLUW3pS4bGZCOXF3U4ENHbL95A5LHk efIlf+pmnXWpkmmI7F2ejbOYrh/fMj8e1NSoS6JQs3xeft5HftTOq/C9xU0/4TcblA4P baoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=87xuAjkSbldiQtfsWLM746IVTINRnCLSgl55gV1EVKA=; b=jhT2bvmNb1GlQkrufJCwr02zRk86gLLOBS1D7swEzKkT/o6Xe0VD7h+Qd6GePAXE1X mAYxgclZjeHOXPlCp+SdH4SvzIcUW18jaM5sR4Pt4Mh3skahDDcZDqCw3RlqxsG4wGA1 adW896MrMZoJkV3qgC9QYPd7EGau3v0eVdqVrLSSVfCwyPwndiU0GscElcpIcMcJYxAF yerP9GMtWwlA2GrUqeqlA1FnMQb5UpQ4RCBSfxLDU7exNMypQKvKVrDEVNajrTlX5onI fkG03tohTHEqOco3/tg17huEdgS1484FsgeBuM1K8X3PLzld1/ylarv+aBOeDjQ4VRRe QJJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkklNv4n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 30-20020a170906015e00b00857c01c6b3fsi80234ejh.164.2023.01.23.14.09.12; Mon, 23 Jan 2023 14:09:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkklNv4n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232896AbjAWWG5 (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232697AbjAWWGm (ORCPT ); Mon, 23 Jan 2023 17:06:42 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B20C1392B1 for ; Mon, 23 Jan 2023 14:06:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511569; x=1706047569; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zlqOcpbNexZKX1umuAS4btmQO9sdmsrwhAByyp7kByk=; b=SkklNv4n8c8xLl02XE0zoCriSTVLy/wo294kjulAA328MJMJh95RIEwN FP6i/xUMv/lVWhnkZImNdcAFdtwAs2+aaJWGYRB6NLM/WaI9Vf0XX2Adt TGIy9bAn7rXEuLko6j65FZaqa0x55eijreDuoL7AthLOA0NKRuYuWAY43 8XWPpQolYCkcaLSjwMUR8U6IFlcaVjRH9qSdZ2LFOiCmvNUMFzXScKCK+ 1DGshSYz+rH6MyybQRjgSHPpvnw2Uvkf7EFOWgh5ukfTwqdOIkBUzU9/U plpBjNLXNCKLhHGj3A04fq0O8BcPxUo5s2m/AvijxOx5cCseo8VT7Wz1z A==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198220" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198220" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103444" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103444" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id A545B109B82; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv15 13/17] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Tue, 24 Jan 2023 01:04:56 +0300 Message-Id: <20230123220500.21077-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852858987437099?= X-GMAIL-MSGID: =?utf-8?q?1755852858987437099?= From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 144 +++++++++++++++++++++++++++++- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 268c1d2749af..39ebfc511685 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -29,11 +30,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -49,6 +57,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -61,6 +70,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -213,6 +232,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)set_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -241,13 +322,20 @@ static void run_test(struct testcases *test, int count) struct testcases *t = test + i; /* fork a process to run test case */ + tests_cnt++; ret = fork_test(t); + + /* return 3 is not support LA57, the case should be skipped */ + if (ret == 3) { + ksft_test_result_skip(t->msg); + continue; + } + if (ret != 0) ret = (t->expected == ret); else ret = !(t->expected); - tests_cnt++; ksft_test_result(ret, t->msg); } } @@ -268,7 +356,6 @@ static struct testcases malloc_cases[] = { }, }; - static struct testcases bits_cases[] = { { .test_func = handle_max_bits, @@ -276,11 +363,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); printf("\t-h: help\n"); } @@ -320,6 +450,12 @@ int main(int argc, char **argv) if (tests & FUNC_BITS) run_test(bits_cases, ARRAY_SIZE(bits_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Mon Jan 23 22:04:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47443 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838647wrn; Mon, 23 Jan 2023 14:09:25 -0800 (PST) X-Google-Smtp-Source: AMrXdXuL4VtpyCeKlsHz2wc0EfbTgtmvKndGvgzUD1GLfBubtr7y0TLpmru6JPW7LkPawhDGKB+/ X-Received: by 2002:a17:907:9a8c:b0:872:4fc7:fc05 with SMTP id km12-20020a1709079a8c00b008724fc7fc05mr30422184ejc.62.1674511765104; Mon, 23 Jan 2023 14:09:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511765; cv=none; d=google.com; s=arc-20160816; b=ihXy58/4h4jmQ+29Yb+jlnc1wTWIckww0yfooiwuYAndXIbylqUqC3kdULi256uBS3 /uX56plJRa2+dz50nhnXah7qeoKPwC9m1HakC0dDWHtlIg4CWbKFEm0zvy9yf1dBOsoo 8zEAh00WxjWgJshf9tJ77f3GJAig0x7lVCAF6CHzYBBPiI3ozO8TF3H2MB+st2piSuxC J58W3GW2J0ibRNmUP2bIh8VNZtiPywSqTHP/oX2u4kyFhxxay9gVYXHvF1SQVs5w0UbJ PuCc+u428mLgTIE6SGaz10pc7vsgukBfFqNNRcvpgUaAFIFrgk8n7stf9t/AnoIXXY/Q kmww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iELgJ2hkEbcjlsU3NCxIEG5m0tt39V5mR+E0WqyULIo=; b=zgKqSG/y9nDcvCVP3XY5pcqnDpcGtXELAXZXJm2aixP5ziBIubI0kfDiSACgLcmcNo emTmI9uEQEAxi6akcOxoSc5dLVTG6pfKlfjJrJ3rfZp6mS6ohBpz3trV5oXxKh8dQWDT idfzvSyaZC6BHxWHtkI0CdwXwc+HVPIOyUyfyxkoMyLERtT2hNNNWosZ0hgnXvlM+qqv 9xK8kXCKEotVto15U3Ct+mU5l3NrlUX3Q8alA0dFCqkOFACr+96wGPpa7GWdIBFUNvr1 efEY3Kbg/y3bV1qy0oXW2+7Dcr+wowE7STAEk4bqiZGLHdziqrlJb9ejs5uRfeAGh4n7 7p9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UeiehPTN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fx10-20020a1709069e8a00b0084c41b233a4si40878ejc.327.2023.01.23.14.09.01; Mon, 23 Jan 2023 14:09:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UeiehPTN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232660AbjAWWGs (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232602AbjAWWGk (ORCPT ); Mon, 23 Jan 2023 17:06:40 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E9883929C for ; Mon, 23 Jan 2023 14:05:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511556; x=1706047556; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ok9UHKEz94fal/z7/7UibwpH0juB5StoEOfS/K/Cro4=; b=UeiehPTNEcRB8nD8YiGRKVLFhbV2dd5x88inkEW50ky/kRv6auj7YTtF /Td48a3lwA6GyUxlPCBWJlYRpXhyl+TZ/MjqH5A8nXTne8D5gV4aSyHZA +CfLBAE3vhKFoEaVW2Z21KFbYz32SVjVzppOiCiUNPOsOSm8NTYuYbHxE OZDwclJXLekvkrV/6f9wNuU8iatnL+JrLJx80V+qvnUlKxaS2z+pDPJNp 2IgP00fNmbLJS1KXnuDx12rCIV8UfeWjLFCHF2x1zUbqO3O0Fp8gSLpVA ASrza3j1l7bQm8A0zX00RmwvBstBn559wEBK0UqayFH3H4mHlet/1GrNe w==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198225" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198225" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:26 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103447" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103447" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:18 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id B0D67109B83; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv15 14/17] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Tue, 24 Jan 2023 01:04:57 +0300 Message-Id: <20230123220500.21077-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852848737308757?= X-GMAIL-MSGID: =?utf-8?q?1755852848737308757?= From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 39ebfc511685..52750ebd0887 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -32,8 +36,9 @@ #define FUNC_BITS 0x2 #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -42,6 +47,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -51,6 +63,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -294,6 +333,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)set_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -340,6 +658,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -410,7 +744,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); printf("\t-h: help\n"); } @@ -456,6 +790,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Mon Jan 23 22:04:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47439 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838002wrn; Mon, 23 Jan 2023 14:07:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXspEhqZiSp9ar918l5CTk1C7CwrTBnJJRD/GzhneeH7hXNcUJqisVAMeMrc+820gFKvWUY4 X-Received: by 2002:aa7:dc10:0:b0:483:b15a:3206 with SMTP id b16-20020aa7dc10000000b00483b15a3206mr16079902edu.23.1674511671101; Mon, 23 Jan 2023 14:07:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511671; cv=none; d=google.com; s=arc-20160816; b=kbN37Lj+BGAAODRDOHRleGSCTWRzfH2LGNqgbcz2Go/CjCA4N9RljmyuZo4uk/hHFm 7RtC8qVeAnowSrHMUnChVvSpyoVhlwTBkYO01Fl/3MO4hNKzn3Lh7h5UO4w5QhDtxN2R pW9XtueEtTBtHjvW5XKxf9hzRITQtenP85/nJlxzB9r71i1ak0+81LpmV0nmRsMQzzQk 3qCM+HhgDoP7p7mG005I4LYdueYgoyCd2xEEweAZj0eSrKl+JSGpPEnnvrkoHVtgFy95 EzhBW3XqEnk7Ght8proAqeH/0P0vIynvkFdHCuHJAhLXDscYcBOW8yZSoW3woKtLAuw0 jhIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eJY/16d3ayK+bkLmy9h+DZWLLsp2YrqC4AI8wcT4lhs=; b=M9/CfgoY7Ac/CFCxfWo8NZ5ee72lhtH4YSoAaS2mn1lkz2QVJaWUNiQbuCGOVBjV0e yDBCYa2FsKW6nQ05nOJE+mqqatMeKOyNaEwEncSmmLAo6/apwEJ6IR1bD8E7jCSqw/St p+Nt5/IBGYYVxXywa7gxUR37YhocNXzNplzeemKgEFSzigbGudDygy+zmdj/X1t0PBZq m6ougOGBXvK1Uxi7XjV4T7OpKkSjyKUpsBg6CBvU6ol/2T0CEnG0fce9JuLvl31HqNEd jX7iavhGEFn+6KSf6WCGrYw3XD6kWilGLC4d6/1Q7kwA92f6tdEOSfeei0iWskOXYpE6 asOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=EAFaYFgL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f3-20020a05640214c300b0049d221b4b39si638170edx.187.2023.01.23.14.07.26; Mon, 23 Jan 2023 14:07:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=EAFaYFgL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232712AbjAWWGX (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232621AbjAWWGK (ORCPT ); Mon, 23 Jan 2023 17:06:10 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B86338E93 for ; Mon, 23 Jan 2023 14:05:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511543; x=1706047543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+tHZ3UxGi6IqVb4/dqBwv5+3olZFm7fLyt/mIZ12PcU=; b=EAFaYFgLSvCmFg15wpP9kEZi0uLWImaR83btNs81LVdPGPJ0hbMyYm4N zPCw7vdhMzKcI8MshSJYYkpm03PANjmPwCEFyreLyfV5p1yYldd/8Onxh HoltH/0ZXvm0F1bwXGgS/n2cVyQeNaQRYXitlGGkiNfEYuJyjNbr43Wt9 6jRpcKs+ixGgjA/Np5vZJN1RtjmhfnQyb6dKcu5NpDyqFYZ/KcaOlZnOB c7ziYNJR7hO2xWdIQpMvP2bCEBsjn+wCVBRx4XzKOdZFvu87h+pXvmgYM f+0xZoQ7+oaYKjEoAHX1PhkRqCySXt+IlNM/iOM8UxffkOMq4ILqXeUAC g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327422009" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327422009" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878126" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878126" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:26 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id BCB35109B84; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv15 15/17] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Tue, 24 Jan 2023 01:04:58 +0300 Message-Id: <20230123220500.21077-16-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852749817368825?= X-GMAIL-MSGID: =?utf-8?q?1755852749817368825?= From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 125 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 52750ebd0887..ebabd4333b7d 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -37,8 +37,9 @@ #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -174,6 +175,28 @@ static unsigned long get_default_tag_bits(void) return lam; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(LAM_U57_MASK)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t set_metadata(uint64_t src, unsigned long lam) { @@ -581,7 +604,7 @@ int do_uring(unsigned long lam) switch (lam) { case LAM_U57_BITS: /* Clear bits 62:57 */ - addr = (addr & ~(0x3fULL << 57)); + addr = (addr & ~(LAM_U57_MASK)); break; } free((void *)addr); @@ -632,6 +655,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -740,11 +829,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n"); printf("\t-h: help\n"); } @@ -764,7 +868,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -778,6 +882,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -793,6 +907,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Mon Jan 23 22:04:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47441 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838578wrn; Mon, 23 Jan 2023 14:09:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXv9n3QUFVCtMIs+CXMpJLNzLi20KilKZE351GDu8vFHRQBk/34lmbx2+N6Zj/3OI/EBk6ly X-Received: by 2002:a05:6402:43cd:b0:49e:ef4:51ca with SMTP id p13-20020a05640243cd00b0049e0ef451camr37321632edc.5.1674511756675; Mon, 23 Jan 2023 14:09:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511756; cv=none; d=google.com; s=arc-20160816; b=KdPd2TU0U6xd1HY1wUX0WbQyHQHo8fm4GFj22mZKW3XlgYOQrZkn493OueWKH8mYey D0TbSlWCqVXJbkg8sCi0a0qLiscUmPBCooqwV9wFGGnMjVRW9UVxoPFYkCxwGzx9TDr7 1Bjl+abtxBodcBzxJJCX9Y8Nq1keHSG4M7PiaCqVZPwwHrJxStpF0Aphg6YBDlD4HOOf wuGI422fEbBU+W9216SoMY6F0TlUaf1gDbwsq6NozBYMUzcYrnbBjQT6oPxOePPv1W+T N0T3mSriElZssfSRAI0qnXSX2PD5gi492APg57uouAnDM5SHguvHRwhnGKolbaSRpqMB 06zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Z2Xbzao2Trzy/9esyr32LxfHyKql/ACddIcl2fmSKko=; b=hL7OfX78egPeYeN3SF7b+p6T9hCYQwL5u1I6hyvUAANGSJHdNtNN6D3ivy0gmIJB/O i3lw0HKKCeEQO5RZK1sEdX8Wc3AXeKGCDrv2NSA6wHzoAlNXL2E4tGIJm8vs+tKm0CeL QM+NzjNTcAQcbpNz2x/3YfEWyedxykj7IUlZUNd8eH26NjshbQV4jBAwFiU3AT6rwEka PssC4pN27hRsRT5wGHuKE4jnEKQC1Sp/z3wfI7cXFR1AJO73aKm4cRScWY6B8lZaYiA8 dz5XX1Z4irJJCptOIsfw5fzY2D6lvjQpeNBvxsKYaUbS9817keLkiVFFXN4v6TUVgFef I8Mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UomvztrZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 19-20020a508e53000000b00499bf569f4bsi685331edx.340.2023.01.23.14.08.53; Mon, 23 Jan 2023 14:09:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UomvztrZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232806AbjAWWG3 (ORCPT + 99 others); Mon, 23 Jan 2023 17:06:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232691AbjAWWGO (ORCPT ); Mon, 23 Jan 2023 17:06:14 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0155C38E9F for ; Mon, 23 Jan 2023 14:05:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511544; x=1706047544; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=k16PD+aFmTFf1DI58jd2raYmUcnyQu1o21tgV69ulz0=; b=UomvztrZ11CaX7Jw5XU/jpqwhROgWi+XrEJytV4VN05DjV4ksmikfucv 2V8iGKvMDi4bzBmIUms4NPavveXCVtLC+XzSTNwoVjYl7dVVJd4TNTN2F rj7s/LgLbhuCafxS4ZFB29wTilLey4xh6UioIbcPVotfruEbzDkT1mDFK tynhsbrOvC7of6SzwPj2Np7uPA5JAUgt1QJ3e5Ei2D72TY9vEP7jnAwUX 5wC33lwQAroEql8eDFAulWArhB7ncSpi5UANfsNAPc/qS6ksyC+40X+Wp N1pPolu4wwIriRMUJVdi5NZxrt4+n+lvE+6Tj2bQ8aoXLq3ZKHzK0Jjvl g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327422016" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327422016" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878128" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878128" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:26 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id C873D109B85; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv15 16/17] selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking Date: Tue, 24 Jan 2023 01:04:59 +0300 Message-Id: <20230123220500.21077-17-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852839516995640?= X-GMAIL-MSGID: =?utf-8?q?1755852839516995640?= From: Weihong Zhang By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. Add new test cases for the new arch_prctl: Before using ARCH_FORCE_TAGGED_SVA, should not allow to enable LAM/SVA coexisting. the test cases should be negative. The test depands on idxd driver and iommu. before test, need add "intel_iommu=on,sm_on" in kernel command line and insmod idxd driver. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 237 +++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index ebabd4333b7d..a8c91829b616 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -30,6 +30,7 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 /* Specified test function bits */ #define FUNC_MALLOC 0x1 @@ -38,8 +39,9 @@ #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 #define FUNC_INHERITE 0x20 +#define FUNC_PASID 0x40 -#define TEST_MASK 0x3f +#define TEST_MASK 0x7f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -55,11 +57,19 @@ #define URING_QUEUE_SZ 1 #define URING_BLOCK_SZ 2048 +/* Pasid test define */ +#define LAM_CMD_BIT 0x1 +#define PAS_CMD_BIT 0x2 +#define SVA_CMD_BIT 0x4 + +#define PAS_CMD(cmd1, cmd2, cmd3) (((cmd3) << 8) | ((cmd2) << 4) | ((cmd1) << 0)) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ unsigned long lam; uint64_t addr; + uint64_t cmd; int (*test_func)(struct testcases *test); const char *msg; }; @@ -556,7 +566,7 @@ int do_uring(unsigned long lam) struct file_io *fi; struct stat st; int ret = 1; - char path[PATH_MAX]; + char path[PATH_MAX] = {0}; /* get current process path */ if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) @@ -852,6 +862,226 @@ static void cmd_help(void) printf("\t-h: help\n"); } +/* Check for file existence */ +uint8_t file_Exists(const char *fileName) +{ + struct stat buffer; + + uint8_t ret = (stat(fileName, &buffer) == 0); + + return ret; +} + +/* Sysfs idxd files */ +const char *dsa_configs[] = { + "echo 1 > /sys/bus/dsa/devices/dsa0/wq0.1/group_id", + "echo shared > /sys/bus/dsa/devices/dsa0/wq0.1/mode", + "echo 10 > /sys/bus/dsa/devices/dsa0/wq0.1/priority", + "echo 16 > /sys/bus/dsa/devices/dsa0/wq0.1/size", + "echo 15 > /sys/bus/dsa/devices/dsa0/wq0.1/threshold", + "echo user > /sys/bus/dsa/devices/dsa0/wq0.1/type", + "echo MyApp1 > /sys/bus/dsa/devices/dsa0/wq0.1/name", + "echo 1 > /sys/bus/dsa/devices/dsa0/engine0.1/group_id", + "echo dsa0 > /sys/bus/dsa/drivers/idxd/bind", + /* bind files and devices, generated a device file in /dev */ + "echo wq0.1 > /sys/bus/dsa/drivers/user/bind", +}; + +/* DSA device file */ +const char *dsaDeviceFile = "/dev/dsa/wq0.1"; +/* file for io*/ +const char *dsaPasidEnable = "/sys/bus/dsa/devices/dsa0/pasid_enabled"; + +/* + * DSA depends on kernel cmdline "intel_iommu=on,sm_on" + * return pasid_enabled (0: disable 1:enable) + */ +int Check_DSA_Kernel_Setting(void) +{ + char command[256] = ""; + char buf[256] = ""; + char *ptr; + int rv = -1; + + snprintf(command, sizeof(command) - 1, "cat %s", dsaPasidEnable); + + FILE *cmd = popen(command, "r"); + + if (cmd) { + while (fgets(buf, sizeof(buf) - 1, cmd) != NULL); + + pclose(cmd); + rv = strtol(buf, &ptr, 16); + } + + return rv; +} + +/* + * Config DSA's sysfs files as shared DSA's WQ. + * Generated a device file /dev/dsa/wq0.1 + * Return: 0 OK; 1 Failed; 3 Skip(SVA disabled). + */ +int Dsa_Init_Sysfs(void) +{ + uint len = ARRAY_SIZE(dsa_configs); + const char **p = dsa_configs; + + if (file_Exists(dsaDeviceFile) == 1) + return 0; + + /* check the idxd driver */ + if (file_Exists(dsaPasidEnable) != 1) { + printf("Please make sure idxd driver was loaded\n"); + return 3; + } + + /* Check SVA feature */ + if (Check_DSA_Kernel_Setting() != 1) { + printf("Please enable SVA.(Add intel_iommu=on,sm_on in kernel cmdline)\n"); + return 3; + } + + /* Check the idxd device file on /dev/dsa/ */ + for (int i = 0; i < len; i++) { + if (system(p[i])) + return 1; + } + + /* After config, /dev/dsa/wq0.1 should be generated */ + return (file_Exists(dsaDeviceFile) != 1); +} + +/* + * Open DSA device file, triger API: iommu_sva_alloc_pasid + */ +void *allocate_dsa_pasid(void) +{ + int fd; + void *wq; + + fd = open(dsaDeviceFile, O_RDWR); + if (fd < 0) { + perror("open"); + return MAP_FAILED; + } + + wq = mmap(NULL, 0x1000, PROT_WRITE, + MAP_SHARED | MAP_POPULATE, fd, 0); + if (wq == MAP_FAILED) + perror("mmap"); + + return wq; +} + +int set_force_svm(void) +{ + int ret = 0; + + ret = syscall(SYS_arch_prctl, ARCH_FORCE_TAGGED_SVA); + + return ret; +} + +int handle_pasid(struct testcases *test) +{ + uint tmp = test->cmd; + uint runed = 0x0; + int ret = 0; + void *wq = NULL; + + ret = Dsa_Init_Sysfs(); + if (ret != 0) + return ret; + + for (int i = 0; i < 3; i++) { + int err = 0; + + if (tmp & 0x1) { + /* run set lam mode*/ + if ((runed & 0x1) == 0) { + err = set_lam(LAM_U57_BITS); + runed = runed | 0x1; + } else + err = 1; + } else if (tmp & 0x4) { + /* run force svm */ + if ((runed & 0x4) == 0) { + err = set_force_svm(); + runed = runed | 0x4; + } else + err = 1; + } else if (tmp & 0x2) { + /* run allocate pasid */ + if ((runed & 0x2) == 0) { + runed = runed | 0x2; + wq = allocate_dsa_pasid(); + if (wq == MAP_FAILED) + err = 1; + } else + err = 1; + } + + ret = ret + err; + if (ret > 0) + break; + + tmp = tmp >> 4; + } + + if (wq != MAP_FAILED && wq != NULL) + if (munmap(wq, 0x1000)) + printf("munmap failed %d\n", errno); + + if (runed != 0x7) + ret = 1; + + return (ret != 0); +} + +/* + * Pasid test depends on idxd and SVA, kernel should enable iommu and sm. + * command line(intel_iommu=on,sm_on) + */ +static struct testcases pasid_cases[] = { + { + .expected = 1, + .cmd = PAS_CMD(LAM_CMD_BIT, PAS_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute LAM, PASID, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(LAM_CMD_BIT, SVA_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute LAM, SVA, PASID in sequence\n", + }, + { + .expected = 1, + .cmd = PAS_CMD(PAS_CMD_BIT, LAM_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute PASID, LAM, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(PAS_CMD_BIT, SVA_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute PASID, SVA, LAM in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, LAM_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, LAM, PASID in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, PAS_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, PASID, LAM in sequence\n", + }, +}; + int main(int argc, char **argv) { int c = 0; @@ -910,6 +1140,9 @@ int main(int argc, char **argv) if (tests & FUNC_INHERITE) run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + if (tests & FUNC_PASID) + run_test(pasid_cases, ARRAY_SIZE(pasid_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Mon Jan 23 22:05:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47445 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1838874wrn; Mon, 23 Jan 2023 14:10:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXvXquh7uI5pP1HfAIk2Fa8hgRQwwglhn1DUXC30QinwvMaRc5riFQU71MctEROUa5t/qdmX X-Received: by 2002:a17:902:e74f:b0:195:ed7d:67d4 with SMTP id p15-20020a170902e74f00b00195ed7d67d4mr17849187plf.7.1674511800146; Mon, 23 Jan 2023 14:10:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511800; cv=none; d=google.com; s=arc-20160816; b=aj7Gexm3pcjzNw58K+U21Dt+AMJlh6jd2N9slzL2wQbSYEEI8+kGd0yUDfbaPTi8LP TtFILLhYiV4HJxeS/VQAWpw/FKpdhRgnXb6cHiyDnlzl2Q7RY9qyVrypeD7iLJ0nuvlc ePOZbaMAfBG5evFuz+1QYg+Y5I73bEynBNwhbpjyw72X9x4rVNBebfACUAX1DIK89qj+ csYXgeQ+6u2QZSqd50XB5B0GJdV3l+m+TgB4k0RlSN03tlYtcxSzcxkwDkrxer/5ryQw rvukMtmmzTs5mtyN+k+cVMkXEjNRtd635WfLVu98/jYfiuxM1XxacLZIuZWDj/wxkKkp w0TQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iOJ+KgvrCLkVGQ8VGNwPxdk7HoxrkwZ1NIJieW24rag=; b=ChcW7Uvwaye+kedmZxZj4+BgHcZiMFpi2aadryKGu4VGzZtE7Djc20ryYvODwvlXd1 e6e4IZTAuQl3IM5E3s64Dk7SCgZZoFb6q03+NnCQhmRf9/lKSGONx7u+1alUU4pNj4vu J9HIBR2INZmuvF1Z1Kcj/QoUC2O+1uBSG1ccWx16IIpYOVDaguYT+RAhDIwP3efk4asN XSCYhhHjUEMjOlYuZd10/RDwNCHSS8tjZpv8T462p/3okMZr77S9Uj2wKdHdW38PBxlk Hpn7D2pIbRzOAyMbEqWOh0c1t7aNEuHVwL0RNTrJEJUXKtIDfDTm4cVyo0oGzUKYOtPh c/CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P53Rd+Ku; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l10-20020a170903120a00b001891fd4ebf0si528728plh.495.2023.01.23.14.09.47; Mon, 23 Jan 2023 14:10:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P53Rd+Ku; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232947AbjAWWH0 (ORCPT + 99 others); Mon, 23 Jan 2023 17:07:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232946AbjAWWHL (ORCPT ); Mon, 23 Jan 2023 17:07:11 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 625A439BA8 for ; Mon, 23 Jan 2023 14:06:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511598; x=1706047598; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aL4h23Fj04DVJyvMyBqOftpxNoBnsmucH0zYgJ3jJBA=; b=P53Rd+KuO95YzuCKMSB9x0ih1xlUINz4P6XD+ILfTKBjuNi3dqOy2FQc Je54ElTEYwJ5LPYeFaVStpkN0ZzhdDJtdOvhpGxZiM5SbQ9Uz8LxpbMnh 7HDekTKA53O1d8iHw8Y4i/BKTMwKBbWYo1fugaN3vIDuwFWq+B7zDn/OD PPB71PvTV9JDyD/hlm2uTr3ypSyhV5IRjjx8TwBneOWPy89UPJdsQML9O yt1NWxQhhFU6izSjm7he9AvWwWXOoucTBddmnlZq21g1kA1iLV8LG4Cvd Mvn2IQR5pp/J05LxKN8eiBv/kR+2X3wXe1Nfz8j+vuXjFTxa5UB/t1G00 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="326198288" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="326198288" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:35 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="694103572" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="694103572" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:27 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id D3EA2109B86; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 17/17] selftests/x86/lam: Add test cases for LAM vs thread creation Date: Tue, 24 Jan 2023 01:05:00 +0300 Message-Id: <20230123220500.21077-18-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852885242728321?= X-GMAIL-MSGID: =?utf-8?q?1755852885242728321?= LAM enabling is only allowed when the process has single thread. LAM mode is inherited into child thread. Trying to enable LAM after spawning a thread has to fail. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) --- tools/testing/selftests/x86/lam.c | 92 +++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index a8c91829b616..eb0e46905bf9 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE #include #include #include @@ -12,6 +13,7 @@ #include #include #include +#include #include #include @@ -50,6 +52,8 @@ #define PAGE_SIZE (4 << 10) +#define STACK_SIZE 65536 + #define barrier() ({ \ __asm__ __volatile__("" : : : "memory"); \ }) @@ -731,6 +735,75 @@ static int handle_inheritance(struct testcases *test) return 0; } +static int thread_fn_get_lam(void *arg) +{ + return get_lam(); +} + +static int thread_fn_set_lam(void *arg) +{ + struct testcases *test = arg; + + return set_lam(test->lam); +} + +static int handle_thread(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = 0; + pid_t pid; + + /* Set LAM mode in parent process */ + if (!test->later) { + lam = test->lam; + if (set_lam(lam) != 0) + return 1; + } + + pid = clone(thread_fn_get_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, NULL); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + if (test->later) { + if (set_lam(test->lam) != 0) + return 1; + } + + return 0; +} + +static int handle_thread_enable(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = clone(thread_fn_set_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, test); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + return 0; +} static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -846,6 +919,25 @@ static struct testcases inheritance_cases[] = { .test_func = handle_inheritance, .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: LAM_U57, child thread should get LAM mode same as parent\n", + }, + { + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread_enable, + .msg = "THREAD: [NEGATIVE] Enable LAM in child.\n", + }, + { + .expected = 1, + .later = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: [NEGATIVE] Enable LAM in parent after thread created.\n", + }, { .expected = 0, .lam = LAM_U57_BITS,