From patchwork Wed Jan 11 12:37:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41980 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318027wrt; Wed, 11 Jan 2023 05:25:41 -0800 (PST) X-Google-Smtp-Source: AMrXdXvYFwz0KHbBaPEJxz7vVpHlvgQjsJWoX+LsTMmOu00ht1saTcANTGuoUdl5g70wZooMWiM6 X-Received: by 2002:a05:6a21:33a3:b0:b5:ce03:5b4e with SMTP id yy35-20020a056a2133a300b000b5ce035b4emr15998832pzb.57.1673443540760; Wed, 11 Jan 2023 05:25:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443540; cv=none; d=google.com; s=arc-20160816; b=L5YtIlaJNVJ482zVf0NlUS6F45UT8/7bH32oyrF9A/tfcVGHq/wiZK7Ty9xV+8/O7A 2mdb8r4XlxZJavEcgrWBjqn6hXJZv/PFmrf8LBUKtJA+CO3b8WSpfo+JPr6tA/ys6/du RAnbs91dgjfHQvJX3NLC4IbHmVjeCARR2Mkc5luArFr6kjVUOHffyngPVjQYNkkxxRXw pi/5hBjayXJ++DneuNtcyIk6oNzSQ0DFis9NgdiJ8lVTxOHuKssWygZ4lSmUF9QQtQv8 Zx+Dd7EX8P8VxGDCC2DHszv21tjdW9f4PeQ+cSXpl0SMPXYMHw/QbzUOlbMkdO42BTWm oBXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=l43pfHtaJjON3Lp2KRYKsKXr+JNlieBiJ3YoFLjROu4=; b=W8U9wjnmFrXDW9bvrtkHbK7+9VjTocBnQgQis8UEzmxJ/EPVQGJb4SVV3qmWI5mVVN tTgMwRAMax3Zf5KEgxl+jMaQk6/lPUSyQHVdX/uwL7GRG4fd3KFdVvKyyYVVlOU+BLCx zfeB0vkWAGqVVXG5YKUGI4rpLpupbiAN8TARfRB2b721M8O8lo4GTg+9cQwt0WkP8FYg p3nLj+KdkIzLm6LBEa+mPFimc144Zn+V4OlaLamYVOVx44Yv+OkAV6+odRZfz5etvVZB aqN43FZzxtRIF3S8Ttiwyi/RdDSotthx7aSCpG+5gHYZiOS2PGH1FnjEo8rwltWIfoAU qYSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ri5bLSZ0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s11-20020a65584b000000b004a26c35feeesi15234931pgr.859.2023.01.11.05.25.27; Wed, 11 Jan 2023 05:25:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ri5bLSZ0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233530AbjAKNXy (ORCPT + 99 others); Wed, 11 Jan 2023 08:23:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232778AbjAKNXu (ORCPT ); Wed, 11 Jan 2023 08:23:50 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E60DC2632 for ; Wed, 11 Jan 2023 05:23:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443428; x=1704979428; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+rl4a/r+n7WVh8LTw1aFIfnP3cclJ/IXLZLoq8uYoKo=; b=Ri5bLSZ0AYMznAuCyJfV/7O/Vbuiu1h6aUElmQcKfSUSPvDBAZbJX7Nc OQaCAIz/BO2rekpbDuPaFfxfaz6qEIe5OQ7a/XV0iSSNNoOtzyq2iV1rY GN96qTHHHsqVIvfkkRx9TQnbfp1xHss+/v/HUP6NC9u2JNFirys7HFjZX xudbARUW3NmG0JyVsl7vHRAFIt2F+oLji0XtSAoIpZlTwwh4IR07tB8of jIz8AipNdYatoBqHdfff5N0SVnSv1oKiwwSo98N3E43OKuHjHSVPvoFJ2 ZQfkzhoq7NlDV7ngDhEvSswwG0LaJBSgx/jTBrjCT8NSphukmLyg307sl w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872495" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872495" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927404" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927404" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:42 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 0815B109C73; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 01/17] x86/mm: Rework address range check in get_user() and put_user() Date: Wed, 11 Jan 2023 15:37:20 +0300 Message-Id: <20230111123736.20025-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732734238123711?= X-GMAIL-MSGID: =?utf-8?q?1754732734238123711?= The functions get_user() and put_user() check that the target address range resides in the user space portion of the virtual address space. In order to perform this check, the functions compare the end of the range against TASK_SIZE_MAX. For kernels compiled with CONFIG_X86_5LEVEL, this process requires some additional trickery using ALTERNATIVE, as TASK_SIZE_MAX depends on the paging mode in use. Linus suggested that this check could be simplified for 64-bit kernels. It is sufficient to check bit 63 of the address to ensure that the range belongs to user space. Additionally, the use of branches can be avoided by setting the target address to all ones if bit 63 is set. There's no need to check the end of the access range as there's huge gap between end of userspace range and start of the kernel range. The gap consists of canonical hole and unused ranges on both kernel and userspace sides. If an address with bit 63 set is passed down, it will trigger a #GP exception. _ASM_EXTABLE_UA() complains about this. Replace it with plain _ASM_EXTABLE() as it is expected behaviour now. The updated get_user() and put_user() checks are also compatible with Linear Address Masking, which allows user space to encode metadata in the upper bits of pointers and eliminates the need to untag the address before handling it. Signed-off-by: Kirill A. Shutemov Suggested-by: Linus Torvalds --- arch/x86/lib/getuser.S | 83 ++++++++++++++++-------------------------- arch/x86/lib/putuser.S | 54 ++++++++++++--------------- 2 files changed, 55 insertions(+), 82 deletions(-) diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b70d98d79a9d..b64a2bd1a1ef 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -37,22 +37,22 @@ #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rdx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rdx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_DX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rax, %rdx + sar $63, %rdx + or %rdx, %rax +.else + cmp $TASK_SIZE_MAX-\size+1, %eax + jae .Lbad_get_user + sbb %edx, %edx /* array_index_mask_nospec() */ + and %edx, %eax +.endif +.endm .text SYM_FUNC_START(__get_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=1 ASM_STAC 1: movzbl (%_ASM_AX),%edx xor %eax,%eax @@ -62,11 +62,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) SYM_FUNC_START(__get_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=2 ASM_STAC 2: movzwl (%_ASM_AX),%edx xor %eax,%eax @@ -76,11 +72,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) SYM_FUNC_START(__get_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=4 ASM_STAC 3: movl (%_ASM_AX),%edx xor %eax,%eax @@ -90,30 +82,17 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) SYM_FUNC_START(__get_user_8) -#ifdef CONFIG_X86_64 - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX + check_range size=8 ASM_STAC +#ifdef CONFIG_X86_64 4: movq (%_ASM_AX),%rdx - xor %eax,%eax - ASM_CLAC - RET #else - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_DX,%_ASM_AX - jae bad_get_user_8 - sbb %_ASM_DX, %_ASM_DX /* array_index_mask_nospec() */ - and %_ASM_DX, %_ASM_AX - ASM_STAC 4: movl (%_ASM_AX),%edx 5: movl 4(%_ASM_AX),%ecx +#endif xor %eax,%eax ASM_CLAC RET -#endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -166,7 +145,7 @@ EXPORT_SYMBOL(__get_user_nocheck_8) SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC -bad_get_user: +.Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET @@ -184,23 +163,23 @@ SYM_CODE_END(.Lbad_get_user_8_clac) #endif /* get_user */ - _ASM_EXTABLE_UA(1b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, .Lbad_get_user_clac) + _ASM_EXTABLE(2b, .Lbad_get_user_clac) + _ASM_EXTABLE(3b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) #endif /* __get_user */ - _ASM_EXTABLE_UA(6b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_get_user_clac) - _ASM_EXTABLE_UA(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, .Lbad_get_user_clac) + _ASM_EXTABLE(7b, .Lbad_get_user_clac) + _ASM_EXTABLE(8b, .Lbad_get_user_clac) #ifdef CONFIG_X86_64 - _ASM_EXTABLE_UA(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_clac) #else - _ASM_EXTABLE_UA(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE_UA(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 32125224fcca..3062d09a776d 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -33,20 +33,20 @@ * as they get called from within inline assembly. */ -#ifdef CONFIG_X86_5LEVEL -#define LOAD_TASK_SIZE_MINUS_N(n) \ - ALTERNATIVE __stringify(mov $((1 << 47) - 4096 - (n)),%rbx), \ - __stringify(mov $((1 << 56) - 4096 - (n)),%rbx), X86_FEATURE_LA57 -#else -#define LOAD_TASK_SIZE_MINUS_N(n) \ - mov $(TASK_SIZE_MAX - (n)),%_ASM_BX -#endif +.macro check_range size:req +.if IS_ENABLED(CONFIG_X86_64) + mov %rcx, %rbx + sar $63, %rbx + or %rbx, %rcx +.else + cmp $TASK_SIZE_MAX-\size+1, %ecx + jae .Lbad_put_user +.endif +.endm .text SYM_FUNC_START(__put_user_1) - LOAD_TASK_SIZE_MINUS_N(0) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=1 ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -66,9 +66,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) SYM_FUNC_START(__put_user_2) - LOAD_TASK_SIZE_MINUS_N(1) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=2 ASM_STAC 3: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -88,9 +86,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) SYM_FUNC_START(__put_user_4) - LOAD_TASK_SIZE_MINUS_N(3) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=4 ASM_STAC 5: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -110,9 +106,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) SYM_FUNC_START(__put_user_8) - LOAD_TASK_SIZE_MINUS_N(7) - cmp %_ASM_BX,%_ASM_CX - jae .Lbad_put_user + check_range size=8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 @@ -144,15 +138,15 @@ SYM_CODE_START_LOCAL(.Lbad_put_user_clac) RET SYM_CODE_END(.Lbad_put_user_clac) - _ASM_EXTABLE_UA(1b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, .Lbad_put_user_clac) + _ASM_EXTABLE(2b, .Lbad_put_user_clac) + _ASM_EXTABLE(3b, .Lbad_put_user_clac) + _ASM_EXTABLE(4b, .Lbad_put_user_clac) + _ASM_EXTABLE(5b, .Lbad_put_user_clac) + _ASM_EXTABLE(6b, .Lbad_put_user_clac) + _ASM_EXTABLE(7b, .Lbad_put_user_clac) + _ASM_EXTABLE(9b, .Lbad_put_user_clac) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) - _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, .Lbad_put_user_clac) + _ASM_EXTABLE(10b, .Lbad_put_user_clac) #endif From patchwork Wed Jan 11 12:37:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41996 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3319574wrt; Wed, 11 Jan 2023 05:28:17 -0800 (PST) X-Google-Smtp-Source: AMrXdXtL7QydUBMo5U9CQgUsg1Gy6erSQ7f5Fn+TzzcyQG0leq9RCh/9ej83m6p7WLSjPOKUUH9R X-Received: by 2002:a17:903:28d:b0:192:85f3:5b1 with SMTP id j13-20020a170903028d00b0019285f305b1mr59953192plr.59.1673443697591; Wed, 11 Jan 2023 05:28:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443697; cv=none; d=google.com; s=arc-20160816; b=qeDbewrMCrSww2sCavE3LUSVeKe6+qd55MMBJ6WMbt6abhHp9+ra0UUCncmiLeVSW3 6RUnyWIC8hi1MX9V2j6jZHvEfoQS3Des7+vJfDBLBT5GDp9tlLbWf943vDERZOqsJPty 9XMUy4Ts7rIqYRiAV2aibB9Nz9Vb3he7v7OlRmX/cEVR6wISRjy13xSyB06VXfq1ORfh d7QYbTInUb+KtH4+Uvc6UQJW4IUJ9kAg0l004rif1j0reSV39qWLgza2V5BRVopYl/F1 3DjFDef5f5M5FgH15cr80luCCgr+/cFnh/yjhtHrld1sWVlipZlJWGesfpshusmZk2it HPPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gpMcPWw+KawxLiOKdWGlNB1Ju6DNgSGHUJPP1yQBi3Q=; b=ByWRVTtwU2ZIW5EBElKYVNe/qgWuvLb2HC7dahAbotxAaeLnNzsnW8P6AfH1gbNZzW xOfmsgpoGNDZiYrFYoX9aTsEDsH2p+NADW7CML43OA9rAJzKPuH57goUPzy8Ywmyi4d8 caIva9TWkFIyBgZ6sNYB+myaA02UxCh11js4GQHfWMPRZ8xJXOSI5b2VYd/V5psDFDKG WzojM/1utcGIHk80xmlXj6YzWH/n3jcOgBBz3b+6TynhvZ2BQuz3mklSE7FVbhLHyHvi Zu4uqeyNfircP4m9HuQQVjhCJ3BYEatepCPZhCauuS3pd280oreEsEapPiUAmMGmbN1c ciYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dHLiQl2q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b2-20020a170902d50200b00192721d6a97si16003611plg.499.2023.01.11.05.28.04; Wed, 11 Jan 2023 05:28:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dHLiQl2q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232898AbjAKNZH (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234013AbjAKNYB (ORCPT ); Wed, 11 Jan 2023 08:24:01 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7CF32BE8 for ; Wed, 11 Jan 2023 05:24:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443440; x=1704979440; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yOPc1Ih/L2zXkoODIOcmvuNWh1/N6FlnZJa49Np69gU=; b=dHLiQl2q2+gnDrHk+kIybhYO6U1lb350dRSvE0mv/vgY/oajkyp47yIC Pm9C6NoBh7bGrctmFrJ7H9KUiEkrNrgRCTY0b5Yvn4d5WaBmDc7or5OIu MRFxv8xTEoymAGlRux1e24OQVJpkm6eA/7bztzA1ewtB35KplnW1VtOMz OawHzvUmyLaBLZBz2o6u0aJA4KVaFk2JW+gAgz9fKWEMzYrkRM5rfDNEF treLI3LRNdA4UsYOjJK2qdK42s+s5AzUO92Tgyoqt30/pU//gGU2MdO9G r5MTaxD3YTltzRc/CAiJWkWlFWDlsASw3O0nbflsIo62S3iGvL6hTNx5X w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646101" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646101" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:54 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725483" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725483" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:49 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 133EB109C7C; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 02/17] x86: Allow atomic MM_CONTEXT flags setting Date: Wed, 11 Jan 2023 15:37:21 +0300 Message-Id: <20230111123736.20025-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732898704834677?= X-GMAIL-MSGID: =?utf-8?q?1754732898704834677?= So far there's no need in atomic setting of MM context flags in mm_context_t::flags. The flags set early in exec and never change after that. LAM enabling requires atomic flag setting. The upcoming flag MM_CONTEXT_FORCE_TAGGED_SVA can be set much later in the process lifetime where multiple threads exist. Convert the field to unsigned long and do MM_CONTEXT_* accesses with __set_bit() and test_bit(). No functional changes. Signed-off-by: Kirill A. Shutemov --- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/mmu.h | 6 +++--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/process_64.c | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 4af81df133ee..aa226f451c52 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -317,7 +317,7 @@ static struct vm_area_struct gate_vma __ro_after_init = { struct vm_area_struct *get_gate_vma(struct mm_struct *mm) { #ifdef CONFIG_COMPAT - if (!mm || !(mm->context.flags & MM_CONTEXT_HAS_VSYSCALL)) + if (!mm || !test_bit(MM_CONTEXT_HAS_VSYSCALL, &mm->context.flags)) return NULL; #endif if (vsyscall_mode == NONE) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..efa3eaee522c 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -9,9 +9,9 @@ #include /* Uprobes on this MM assume 32-bit code */ -#define MM_CONTEXT_UPROBE_IA32 BIT(0) +#define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ -#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +#define MM_CONTEXT_HAS_VSYSCALL 1 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. @@ -39,7 +39,7 @@ typedef struct { #endif #ifdef CONFIG_X86_64 - unsigned short flags; + unsigned long flags; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b8d40ddeab00..53ef591a6166 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -181,7 +181,7 @@ static inline void arch_exit_mmap(struct mm_struct *mm) static inline bool is_64bit_mm(struct mm_struct *mm) { return !IS_ENABLED(CONFIG_IA32_EMULATION) || - !(mm->context.flags & MM_CONTEXT_UPROBE_IA32); + !test_bit(MM_CONTEXT_UPROBE_IA32, &mm->context.flags); } #else static inline bool is_64bit_mm(struct mm_struct *mm) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..8b06034e8c70 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -671,7 +671,7 @@ void set_personality_64bit(void) task_pt_regs(current)->orig_ax = __NR_execve; current_thread_info()->status &= ~TS_COMPAT; if (current->mm) - current->mm->context.flags = MM_CONTEXT_HAS_VSYSCALL; + __set_bit(MM_CONTEXT_HAS_VSYSCALL, ¤t->mm->context.flags); /* TBD: overwrites user setup. Should have two bits. But 64bit processes have always behaved this way, @@ -708,7 +708,7 @@ static void __set_personality_ia32(void) * uprobes applied to this MM need to know this and * cannot use user_64bit_mode() at that time. */ - current->mm->context.flags = MM_CONTEXT_UPROBE_IA32; + __set_bit(MM_CONTEXT_UPROBE_IA32, ¤t->mm->context.flags); } current->personality |= force_personality32; From patchwork Wed Jan 11 12:37:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41989 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318783wrt; Wed, 11 Jan 2023 05:26:55 -0800 (PST) X-Google-Smtp-Source: AMrXdXswQcmaoHqMKEwMgmxojTuWjglBQQMnQe1ExieaprhCc6njX6z7Raf3yW5Xwar4buhDhT0G X-Received: by 2002:a05:6a00:1d8e:b0:575:6a68:65f8 with SMTP id z14-20020a056a001d8e00b005756a6865f8mr73222178pfw.18.1673443615471; Wed, 11 Jan 2023 05:26:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443615; cv=none; d=google.com; s=arc-20160816; b=Fu5WvgvDPAh6vBmt5/3pmZ7RTbjLmTEJ5E4Ag+WvMO1D7uTkhDgv/ylg2A0fkLq3IO NmyBfRRhPaSu077dxPxcoZMlh2qSVRyJj7/TttGG0lUfo+tISlTtVJl9mBz2kpWF2B6a faw5sADlui7iaukr3p8t9omXrXXsdDM4VnC51wY+Cqy8Ye3tzhh4BRI5LB+eMAaLyIzP G3w9/GVQ5B0Vj3ZzkZOksrvZlWfGfDbh8HJJNmfxpDvLmVUhg7gHJ6r0mSxachfKRQ8V bNmuuEAAxY1jpyaupWMbkNnw7NYgOBaZ+EpkS7CKgab8M/ebBKS+Bw6Z9LVcSgKrjxV8 1/Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gtvQdR/tQF6edmArQpOLj1DCcO+bGxNE9riU66jpwoM=; b=iLgg3Y/ssiur4DfahnbRf11LEWjS2lRK7Qz+hZha6Rre9GTa1XO4VuDh8sW791Dzv7 2xYvBdi3djleC4T6zWSgWWrtWst46YOrPaG0PVMaWGR9AXkcFmiubG7tYz7U6e1uOeEX V7AyR/+jpkLIxyeU2wUevEjZyu06BuElPywBSBH1W0B4SLXY2AmhvOV1ofbgeFE7TdEI TWzHKhf6kKRZPzwUDbeX9qEqSRW3CQdo0rp9gN+mRPoVZA8kEuj50J4wcyPvV2ZbMvk9 i3wm1DIoWT1Pq3li+cPJWxAkfZH3vWFeTFEmf+JEpZlApIbvi5HYHJSHSXyu4OxAnEtd WXKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=oHiLIj3x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e10-20020a056a001a8a00b0056aafb43783si15356222pfv.155.2023.01.11.05.26.42; Wed, 11 Jan 2023 05:26:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=oHiLIj3x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231760AbjAKNZU (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234100AbjAKNYC (ORCPT ); Wed, 11 Jan 2023 08:24:02 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA0BE389F for ; Wed, 11 Jan 2023 05:24:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443441; x=1704979441; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=q9S2rKMrsOnON3J2uWuzPapruoVCfa2fXWWpeKLEJbA=; b=oHiLIj3xjSU9LHsUO39p7ED3DOa/f0E2Zz5ysn/1IcaV9Nc7dSEitHS5 NzS2dpPnkf0tSftwvk24Jxvzb6EDHyqdGYcAbBQcWUHou2IRZsf3d++O5 QMa7CVTvotuxAAr2FUqbUUKM2n92RzCv7XuMB93OleNaj2/BH+tm9Nvr7 LQNQz+/tsWTu4ErGSKQYNfZ93gL0jG3AQK+AGzzAzKxf84Tf+ChO8i//O rWW5PS/ZK4hU+MjNv2HzRyyLcKsk1Z5HOWFwroCfgsu2ezWcrwxt9dA3u i7hi4oRT/uc4PwXzsJZ7BtzlaVOtm/FRnDWskDYHIrrbqH0PboOD1Zx8I w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646109" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646109" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725488" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725488" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 1EC3E109C7D; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 03/17] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Wed, 11 Jan 2023 15:37:22 +0300 Message-Id: <20230111123736.20025-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732812716894624?= X-GMAIL-MSGID: =?utf-8?q?1754732812716894624?= Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. The new CONFIG_ADDRESS_MASKING option enables the feature support in kernel. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/Kconfig | 11 +++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..211869aa618d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2290,6 +2290,17 @@ config RANDOMIZE_MEMORY_PHYSICAL_PADDING If unsure, leave at the default value. +config ADDRESS_MASKING + bool "Linear Address Masking support" + depends on X86_64 + help + Linear Address Masking (LAM) modifies the checking that is applied + to 64-bit linear addresses, allowing software to use of the + untranslated address bits for metadata. + + The capability can be used for efficient address sanitizers (ASAN) + implementation and for optimizations in JITs. + config HOTPLUG_CPU def_bool y depends on SMP diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 61012476d66e..bc662c80b99d 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -314,6 +314,7 @@ #define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */ #define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index a7f3d9100adb..d8cccadc83a6 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -28,6 +28,8 @@ * On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * + * On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. + * * All of the remaining bits indicate the physical address of the top-level * paging structure. * diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Wed Jan 11 12:37:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41984 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318317wrt; Wed, 11 Jan 2023 05:26:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXsFoHXKFnQ2oyYjw1VuFsfFNxnns5j2OvChUsd+CBdbqLpuPgwWscgkybaj1uSTWd0s6YsZ X-Received: by 2002:a17:902:aa8e:b0:189:abdd:400a with SMTP id d14-20020a170902aa8e00b00189abdd400amr73514972plr.15.1673443562169; Wed, 11 Jan 2023 05:26:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443562; cv=none; d=google.com; s=arc-20160816; b=RRyI4RP8CiiF3mDzDVRjpEaBjmkROMfYJD/W9pcmNByml6S8HSV9MczJF2xhlsPws1 2rreXs+CYjpAw7tBAFCJHX6Pf7rTdmPZZPFTPQQ+MAQRAiIH2L2PxsLgknZhEXAFfoma jICcwLPVD1dGZnHe0JOFqlT279DlqJtF5dysMHFI9S0MYS/pnr45D4P0worjMRo3D85/ aoOu8FfH2a4h24pi0La0qigaVlEuT27LN0okfM94ASnkOPl5Otyni/riCVa1Opab9m5P ZDviTjeyl3q4+aHkTejTrf3nCxcl/RvgdGrjDJ4gbwh585fg3kwVkWknKEZd4jDJ2q4s hewA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=u/4HXYv/X9+jKtG9X3etrjRZ+kquIcK8CEU7X0X+F6c=; b=p9E+ospfTdgTau+5jvI1wn/PKdno+SLha+XFfI4OVJOE0I18XxIbBf2aYzcy9guZKQ mcddT9Fp4Na//IGNxhVzH5Aa8QFdM051UqHXFlK10Ezjl+e/4HNh2J+Y+RyyYLko2WZh iVGth2pnuanRwAkx+xNAzxgxPv7/C4hVtNjMRySEdA5hmj0z30ctDQ0QRLx+tqwSnTk1 NFwZTvGlpeLUnES+qDfuQ0vxGLX5svUNmTmXgTKdHVgbL2+cMMYImD+fbMdoxn1jvkfK LuUQ7oi11KJRIxKxxVm+oShRFkKxMEmn7q1YT4kJQzonYGki88SgHgZyV3XRMIuH+ON6 0GkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="k4W/9XHM"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jg17-20020a17090326d100b0019317aff5f1si12533167plb.603.2023.01.11.05.25.49; Wed, 11 Jan 2023 05:26:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="k4W/9XHM"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236087AbjAKNYw (ORCPT + 99 others); Wed, 11 Jan 2023 08:24:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233845AbjAKNYA (ORCPT ); Wed, 11 Jan 2023 08:24:00 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3933425EF for ; Wed, 11 Jan 2023 05:23:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443439; x=1704979439; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EWetv4S/fG+ONbpqRIasNYzww3aPLc98mlPS570ieV4=; b=k4W/9XHMDvBgHivS/KHMAlPo6SAhPf5DLLB3/6tkOkzwf/S8YtNbyoPj bNi8zvcG+9EunJBV9SHCV3cvT3umjSf/afsvDj2P6u+M5t8/BFY41aLkI p0983RJEejnU33VKqW7DBP8iBxn9tWwvPd6V7IVFPQ22QTTjybeRQxZcJ SDKueMl+EJBWeUauvVcVMErxzri8U/Fk9iFBJumkpQzJ9RacbAY+93V9o oub76yBqMeVwWh4YCmCVU4c4EbwG2eDRvKaprF5vmQpeOl4TsBL9duDQX esVyX3oHGYkSLbp0/Ca+OyqJQbiWTArgkLRu6rpDtOENI1INusuafQi4B A==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646063" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646063" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:49 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725455" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725455" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:43 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 2A47F109C81; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 04/17] x86/mm: Handle LAM on context switch Date: Wed, 11 Jan 2023 15:37:23 +0300 Message-Id: <20230111123736.20025-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732756764444279?= X-GMAIL-MSGID: =?utf-8?q?1754732756764444279?= Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t. switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 5 +++ arch/x86/include/asm/mmu_context.h | 24 ++++++++++++++ arch/x86/include/asm/tlbflush.h | 38 ++++++++++++++++++++- arch/x86/mm/tlb.c | 53 +++++++++++++++++++++--------- 4 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index efa3eaee522c..22fc9fbf1d0a 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -42,6 +42,11 @@ typedef struct { unsigned long flags; #endif +#ifdef CONFIG_ADDRESS_MASKING + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; +#endif + struct mutex lock; void __user *vdso; /* vdso base address */ const struct vdso_image *vdso_image; /* vdso image in use */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 53ef591a6166..8388fccc4700 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_ADDRESS_MASKING +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return READ_ONCE(mm->context.lam_cr3_mask); +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..e8b47f57bd4a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -2,7 +2,7 @@ #ifndef _ASM_X86_TLBFLUSH_H #define _ASM_X86_TLBFLUSH_H -#include +#include #include #include @@ -12,6 +12,7 @@ #include #include #include +#include void __flush_tlb_all(void); @@ -101,6 +102,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_ADDRESS_MASKING + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -357,6 +368,31 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_ADDRESS_MASKING +static inline u64 tlbstate_lam_cr3_mask(void) +{ + u64 lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ + this_cpu_write(cpu_tlbstate.lam, + mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline u64 tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global(unsigned long cr4) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c1e31e9a85d7..8c330a6d0ece 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,26 +154,30 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { + unsigned long cr3 = __sme_pa(pgd) | lam; + if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); + cr3 |= kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); } + + return cr3; } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { - VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } /* @@ -274,15 +278,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +496,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +526,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, + tlbstate_lam_cr3_mask()))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -552,9 +559,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * instruction. */ if (real_prev == next) { + /* Not actually switching mm's */ VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); + /* + * If this races with another thread that enables lam, 'new_lam' + * might not match tlbstate_lam_cr3_mask(). + */ + /* * Even in lazy TLB mode, the CPU should stay set in the * mm_cpumask. The TLB shootdown code can figure out from @@ -622,15 +635,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +705,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +717,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +726,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_lam_mode(mm); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1071,8 +1090,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Wed Jan 11 12:37:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41999 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3320196wrt; Wed, 11 Jan 2023 05:29:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXuhwG8XF9Ar3uBh1miFgcTTf79c1yuAJpJP78koYYk8wkVLH0pT8V6KdaecAgKAVCsXDqMi X-Received: by 2002:a17:907:d68e:b0:7c1:37:421c with SMTP id wf14-20020a170907d68e00b007c10037421cmr65305817ejc.32.1673443778004; Wed, 11 Jan 2023 05:29:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443777; cv=none; d=google.com; s=arc-20160816; b=B3X5u3icEMGwuKWWXKgLLV4wdqqaomo2nLHtw+iSGy4IoZrbSplgqqMLKxaXJqwS33 2pDvmo2FUegK84oqAkjN3wdbYPyExJyHLFNuEQaI5YD+9uYpGe4Ylyl5WvwcZOvMA0MM 5X4BXj8wijDi6WFsj+0IgyJfajiQ4ymtnsPlCFLteJyPCHYziJMOq2P1S7ZDd5gH+nk4 M5vIFUbvxfJJ6n6639ID+SkDA42x8Nuqgl0KmTzZTiM+YoxLiLTCnAoUbjBmNO6+sqRD sOHrkEnaCHsnriUysyyyTYlLrXNDxzpFyor2WlcCAD+eQa7DkDUkvE/09ALmcC4RgVA0 xEDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=veekTc4paEUCv01tPg9a3/HHGnDBIKMOOOZayyh7HXU=; b=lFGaKdbVPkCWV+NoYwUutxGrsBI0GzaB6zv0r866g4Trd+BE+nrgdZtoSSOUrbQtkC 4Q8BUvA5Zdx05COFw62pUEb7hOPAn42uFDpKZaRjG41MM4TMopl7eedkESHjfUkaS6ky G891btrvW2H5hPspiZKU8B3aOBizi+uomyOailDVtUs6Gx3+N45ptQxRML8RcLzlRLau N/OoSHwNhni7OnSPuNeplPF8RkdIsNPnQDd5QOg8qDnW7bUvMBMWmyGhDRTAFaNt4X5D RALxzCz0+5VTgDjWk5c6VN9ywIfY32mx8UgmCAOIgWf63PlJePz4MVrgOF+eRvhnj62t 98mA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mcqYhNXl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dt17-20020a170907729100b0084d1d7fbd27si16167228ejc.205.2023.01.11.05.29.14; Wed, 11 Jan 2023 05:29:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mcqYhNXl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236298AbjAKNZ7 (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234212AbjAKNYE (ORCPT ); Wed, 11 Jan 2023 08:24:04 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DA5E389F for ; Wed, 11 Jan 2023 05:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443443; x=1704979443; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fj5bb8VX8rl+0ZAKXFJl1itptKLgoWWul+30bcsj43k=; b=mcqYhNXlYTfPzncNPpjJxUl3IRzWJGqA0QSp9IV0ke4AGhyVWQtLd6vx cctCv1DqVmdQfeAsqsFw1zhubi0o9KBwgASnU/1uCYzRxYgwQKFVKknKy C9C7PgKMxV2b+Hn98OPUNqI5l7LX/tiG51ypOIQbFxLYH0IoR/UcsJevx BsYBBZ/Yjv6raEUi2lOe6DqpCzW/OejJN6m0i7f3j3wBiP1rYERU5fzxf tfIiaUVWorCN7eCXnetRUWQZk6xRah8Cr90Ku57tuO6QeGNR8ypiwNKQG 2feK/oWH6iOJb1SDPUIcwCUswo9u3BYPrAJeJ85vXmzQwMx+JEmaIbO+Z A==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646147" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646147" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725496" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725496" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 35933109C87; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 05/17] mm: Introduce untagged_addr_remote() Date: Wed, 11 Jan 2023 15:37:24 +0300 Message-Id: <20230111123736.20025-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732982924530597?= X-GMAIL-MSGID: =?utf-8?q?1754732982924530597?= untagged_addr() removes tags/metadata from the address and brings it to the canonical form. The helper is implemented on arm64 and sparc. Both of them do untagging based on global rules. However, Linear Address Masking (LAM) on x86 introduces per-process settings for untagging. As a result, untagged_addr() is now only suitable for untagging addresses for the current proccess. The new helper untagged_addr_remote() has to be used when the address targets remote process. It requires the mmap lock for target mm to be taken. Export dump_mm() as there's now the first user for it: VFIO can be compiled as module and untagged_addr_remote() triggers dump_mm() via mmap_assert_locked(). Signed-off-by: Kirill A. Shutemov --- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 9 +++++++-- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 22 ++++++++++++++++++++++ mm/debug.c | 1 + mm/gup.c | 4 ++-- mm/madvise.c | 5 +++-- mm/migrate.c | 11 ++++++----- 9 files changed, 44 insertions(+), 23 deletions(-) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 23c24fe98c00..daf34f957b7b 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -573,7 +573,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr_remote(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..df139a717230 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1692,8 +1692,13 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; - if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) { + ret = mmap_read_lock_killable(mm); + if (ret) + goto out_free; + start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); + mmap_read_unlock(mm); + } /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..6b28eb9c6ea2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -96,17 +96,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index afb18f198843..bfdadf5f8bbb 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,28 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(addr) (addr) +#endif + +#ifndef untagged_addr_remote +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr(addr); \ +}) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..3c1b490c7e2b 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL_GPL(dump_mm); static bool page_init_poisoning __read_mostly = true; diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..e28d787ba8f8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1091,7 +1091,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr_remote(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1265,7 +1265,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr_remote(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; diff --git a/mm/madvise.c b/mm/madvise.c index a56a6d17e201..90cd4a442fd2 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1407,8 +1407,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) return -EINVAL; @@ -1440,6 +1438,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh mmap_read_lock(mm); } + start = untagged_addr_remote(mm, start); + end = start + len; + blk_start_plug(&plug); error = madvise_walk_vmas(mm, start, end, behavior, madvise_vma_behavior); diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..dae5022d94b0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1719,14 +1719,17 @@ static int do_move_pages_to_node(struct mm_struct *mm, * target node * 1 - when it has been queued */ -static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, +static int add_page_for_migration(struct mm_struct *mm, const void __user *p, int node, struct list_head *pagelist, bool migrate_all) { struct vm_area_struct *vma; + unsigned long addr; struct page *page; int err; mmap_read_lock(mm); + addr = (unsigned long)untagged_addr_remote(mm, p); + err = -EFAULT; vma = vma_lookup(mm, addr); if (!vma || !vma_migratable(vma)) @@ -1831,7 +1834,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, for (i = start = 0; i < nr_pages; i++) { const void __user *p; - unsigned long addr; int node; err = -EFAULT; @@ -1839,7 +1841,6 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) @@ -1867,8 +1868,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, * Errors in the page lookup or isolation are not fatal and we simply * report them via status */ - err = add_page_for_migration(mm, addr, current_node, - &pagelist, flags & MPOL_MF_MOVE_ALL); + err = add_page_for_migration(mm, p, current_node, &pagelist, + flags & MPOL_MF_MOVE_ALL); if (err > 0) { /* The page is successfully queued for migration */ From patchwork Wed Jan 11 12:37:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41991 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318828wrt; Wed, 11 Jan 2023 05:27:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXvsjpFQkexRKkMc0SxhDF3yn/VO8a0ZMulbu8N3XdZ53PKYmTR/A8V6d8txtneKEkSG1BcI X-Received: by 2002:a17:907:874c:b0:7c0:9bc2:a7d6 with SMTP id qo12-20020a170907874c00b007c09bc2a7d6mr59338330ejc.38.1673443619881; Wed, 11 Jan 2023 05:26:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443619; cv=none; d=google.com; s=arc-20160816; b=sPo5P/qAU8OerhhhEp0ddX8wvE+XRcBaNJcocCodQkl2lP4mlp4bq8sfTUC/+zAWMp ae/0Fo/s0DnjYD+UrxHr+gMQRfDQdCFADkDM9/sEshUi6UEqPlGncISNIzxLXr0FllDG cMUjH3Se9GKq4OAGI6zByEP7WMO4+BDWoWsMUtTqp3P9zvHdlVm+LtK5v2CZyFP62vFp YffLu++xKtwcn3jqvzzu0sLjcrYnhi/+GEkxiUuPDB4Lcl46S0eWdQfXyPu4PJ8n+4wS uhuO8xMV5jAyUq8kU4OPZcUVtcARRKOEXTsNiTjdMzom47YZrgxFLkyFvliQUw5vqas2 jCQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZcjVo3STBREvzEAFByYMAnPs1e/jyq7z4ZoSrv2nKhk=; b=wNZYdPYPNO5GMg0OTCWJ/QEWBUmGPrJfXUkWCsT/+thtwumuuzY5THSwkVbaVA3QO4 tiZ012LJsw/1wvyOXfappayx5VfVWPoBj3hjBvPPSZGZDbo3e4Cqo8AV6FZy+Y77Asg1 jbte/YguxlR0AaN1LYVaQwmbdPxo+sFGv69pziyBu3CwPWQXNMqA3XICC7LsG4lgb6WP iRHXfgfPHLOcunkHPyeH45pqGf4LMML5/jjHbitoLHuLX6ybAuH5W2YR4s9DbpyVBfI4 MOocRlA0se9mWtjhOAlmL6B87kUGmsYQKGq/Y0v8RMCbZNxXaQ4eZrQ1/tJjgcfnGkmw 4fbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Oy5FJwB+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wu7-20020a170906eec700b007adf38290f1si15453802ejb.642.2023.01.11.05.26.36; Wed, 11 Jan 2023 05:26:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Oy5FJwB+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236011AbjAKNZi (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234247AbjAKNYE (ORCPT ); Wed, 11 Jan 2023 08:24:04 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AAC35F55 for ; Wed, 11 Jan 2023 05:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443443; x=1704979443; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SGiBGTV0pV1LSdKWzpftY7fVqqKe+kxFUGucNy6ovfQ=; b=Oy5FJwB+61G2i6gJr80oroQ8roSJKes7dWld9/8aSOTye5qGxlC3TRmL Ur2f9k2GaGa1crgRs28LsfST7JPDam1tprmYbEwDcCmCXiqVM2sOynNuD s0ln8ugV52Yb1TorUVcvxCG9JJ5La4TI+OERDVdcA3qXPMkBHRnMVEyNg OsLVpvJaeIWwsV4+NhXRV0wMhrpsbHbtXqyUg3IWz1mitveHjvjvWqIf7 79FOlQ/5TnnnnMl3bSDfqc2Cu2x0xH/4gCSPnsjlVq4O9xEx9ZBeW57iK gZcn25AhLCZJT7F+kPUiT1K2z1gVM7F/xGERKT97ke9sAUZ4ucf0ToSxB Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646159" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646159" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:24:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725511" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725511" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:56 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 3DF10109C8B; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 06/17] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Wed, 11 Jan 2023 15:37:25 +0300 Message-Id: <20230111123736.20025-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732817186968491?= X-GMAIL-MSGID: =?utf-8?q?1754732817186968491?= untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape based on rules of the current thread. It only handles userspace addresses. The untagging mask is stored in per-CPU variable and set on context switching to the task. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 +++++++++++ arch/x86/include/asm/tlbflush.h | 10 ++++++++++ arch/x86/include/asm/uaccess.h | 31 ++++++++++++++++++++++++++++-- arch/x86/kernel/process.c | 3 +++ arch/x86/mm/init.c | 5 +++++ 6 files changed, 61 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 22fc9fbf1d0a..9cac8c45a647 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -45,6 +45,9 @@ typedef struct { #ifdef CONFIG_ADDRESS_MASKING /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 8388fccc4700..1d0b743daebb 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index e8b47f57bd4a..75bfaa421030 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -54,6 +54,15 @@ static inline void cr4_clear_bits(unsigned long mask) local_irq_restore(flags); } +#ifdef CONFIG_ADDRESS_MASKING +DECLARE_PER_CPU(u64, tlbstate_untag_mask); + +static inline u64 current_untag_mask(void) +{ + return this_cpu_read(tlbstate_untag_mask); +} +#endif + #ifndef MODULE /* * 6 because 6 should be plenty and struct tlb_state will fit in two cache @@ -380,6 +389,7 @@ static inline void set_tlbstate_lam_mode(struct mm_struct *mm) { this_cpu_write(cpu_tlbstate.lam, mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); } #else diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..32c9dd052e43 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,11 +7,13 @@ #include #include #include +#include #include #include #include #include #include +#include #ifdef CONFIG_DEBUG_ATOMIC_SLEEP static inline bool pagefault_disabled(void); @@ -21,6 +23,31 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_ADDRESS_MASKING +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define __untagged_addr(untag_mask, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= untag_mask | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr(addr) __untagged_addr(current_untag_mask(), addr) + +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + __untagged_addr((mm)->context.untag_mask, addr); \ +}) + +#else +#define untagged_addr(addr) (addr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +65,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 40d156a31676..ef6bde1d40d8 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index d3987359d441..be5c7d1c0265 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1044,6 +1044,11 @@ __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = { .cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */ }; +#ifdef CONFIG_ADDRESS_MASKING +DEFINE_PER_CPU(u64, tlbstate_untag_mask); +EXPORT_PER_CPU_SYMBOL(tlbstate_untag_mask); +#endif + void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) { /* entry 0 MUST be WB (hardwired to speed up translations) */ From patchwork Wed Jan 11 12:37:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41993 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318957wrt; Wed, 11 Jan 2023 05:27:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXtxbxKIOh6WeBrLydlR31X5hX8PuFAl2ST13LCn1z1vETIyeSr2C0DNbr7H/3g1t1IvCq0+ X-Received: by 2002:a05:6a20:4995:b0:ad:9646:b9b with SMTP id fs21-20020a056a20499500b000ad96460b9bmr80448156pzb.49.1673443632870; Wed, 11 Jan 2023 05:27:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443632; cv=none; d=google.com; s=arc-20160816; b=tdll1nDxmwJ1FG2rYQyC7UyRksmZYYDhoevhk5bveSVCeZ1Ccswa8sa2g9wDJqrFda /+JHDWh/YU35HV/+coRtaOHIf61OtwbIUQMJryPZ3laQzHC6+4qScXInL0UbqkgCuN0/ 9UVUrjKh5b20pNA6pePsfGwJtbRj2r7aPtPiBaNo+7DohU1krSklUWXUJkYOWGvTZRy6 jLWVr8UkqB6pbRuRvrRg1STVXxO4u5pGQcVi1sL7Jn6aUKUVE+OoLa7WMXNT1eV3DTsN e7OoLNEUPWgV6ZHR52Ud1E3Goux5dwV40Zau0i7H0phH5JYb3tglswhJNLAR9MbFm18+ vVcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6WRPVos/HMIaBDHmKS6MLy3kevGkb6PumiQBQUSfvn0=; b=PwaLaBQpaaGrwhuhBTX0ltXY79CsUjfkWaUsC4cz1o3OIPm0g4+smp2F1rMbx/0KdU fnhf2CLgzXA89fHsQSZK3wQCVcnSwjosFvQDiV8D9B1l/eRw2B6Kp+K8nR2KGVdZa+JJ KKGK3Jv5c4ODv28/ZHJNjKa33tvFddA97AW8Hj1oiMVbA5kI4DirLUipIxsj2seedZdv ZENrKKNTSvDzCVQILzd/IE9Ht45qPd503sBp98WWu7C1O7GC2lhS/nlRfte7i9mQJUpr JBOyiZ3QFGkWLSiZPKZowAQDdnhLF4dq+Z5x5jTSiD3wJW7HAWQzs1cTr0GVRgwSookb YGvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KCYsuFfF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k71-20020a63844a000000b004b124f7a5eesi8447649pgd.387.2023.01.11.05.26.59; Wed, 11 Jan 2023 05:27:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KCYsuFfF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236589AbjAKNZZ (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234102AbjAKNYC (ORCPT ); Wed, 11 Jan 2023 08:24:02 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD1F45F67 for ; Wed, 11 Jan 2023 05:24:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443441; x=1704979441; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UBKxNmRcAlY5Gk5PHZ4C0nPPjFoKYauY1gpRMXD9SVM=; b=KCYsuFfFqeeGcm9L2zgKaB+zNanW+n10JeSyhhqmK0QJ+SAPh873eTe6 AEFVsmC1zAeihvqsgugH2CN79ZjbEJpzesNVWEINDKpZhRTDLB2CxzWqP Ti6WLEqM8HfTihvvuvPJ8RO1QeMQJHaopH/nyaRxPeRM9eb+tS61oTryT gGLk6XI/9maO1OR3hn//XDJdosub09Gq5MIWkCbyXCm8TJ6MQFzdqh0uC kuJWDzdHFIS13IpmUiJql9hwzdy3LfucA62joJ3VaHLxZCR+lVNdh3TxS LiCy2n4ZRIURMDv86wxmcL1AGl7o0HpqDRk3KUyX77noGe8EQS9qLzgjk g==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646129" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646129" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725492" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725492" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 48E17109CB2; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 07/17] x86/mm: Provide arch_prctl() interface for LAM Date: Wed, 11 Jan 2023 15:37:26 +0300 Message-Id: <20230111123736.20025-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732830605343551?= X-GMAIL-MSGID: =?utf-8?q?1754732830605343551?= Add a few of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. - ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/uapi/asm/prctl.h | 4 +++ arch/x86/kernel/process.c | 3 ++ arch/x86/kernel/process_64.c | 55 ++++++++++++++++++++++++++++++- 4 files changed, 63 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 9cac8c45a647..e80762e998ce 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -12,6 +12,8 @@ #define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ #define MM_CONTEXT_HAS_VSYSCALL 1 +/* Do not allow changing LAM mode */ +#define MM_CONTEXT_LOCK_LAM 2 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..a31e27b95b19 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,8 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index ef6bde1d40d8..cc0677f58f42 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -162,6 +162,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) savesegment(es, p->thread.es); savesegment(ds, p->thread.ds); + + if (p->mm && (clone_flags & (CLONE_VM | CLONE_VFORK)) == CLONE_VM) + set_bit(MM_CONTEXT_LOCK_LAM, &p->mm->context.flags); #else p->thread.sp0 = (unsigned long) (childregs + 1); savesegment(gs, p->thread.gs); diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 8b06034e8c70..88aae519c8f8 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,48 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +#ifdef CONFIG_ADDRESS_MASKING + +#define LAM_U57_BITS 6 + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + /* PTRACE_ARCH_PRCTL */ + if (current->mm != mm) + return -EINVAL; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (test_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags)) { + mmap_write_unlock(mm); + return -EBUSY; + } + + if (!nr_bits) { + mmap_write_unlock(mm); + return -EINVAL; + } else if (nr_bits <= LAM_U57_BITS) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + mmap_write_unlock(mm); + return -EINVAL; + } + + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); + + mmap_write_unlock(mm); + + return 0; +} +#endif + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -830,7 +872,18 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - +#ifdef CONFIG_ADDRESS_MASKING + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_GET_MAX_TAG_BITS: + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return put_user(0, (unsigned long __user *)arg2); + else + return put_user(LAM_U57_BITS, (unsigned long __user *)arg2); +#endif default: ret = -EINVAL; break; From patchwork Wed Jan 11 12:37:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41997 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3319644wrt; Wed, 11 Jan 2023 05:28:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXuAx02gWrKnhLzkvjornVFxR+nNqmw6a7R3q1c69MUe2hxeGRwDFr49kd4ePUrZOAz62A/j X-Received: by 2002:a17:906:bc47:b0:78d:f455:3110 with SMTP id s7-20020a170906bc4700b0078df4553110mr55624741ejv.56.1673443708397; Wed, 11 Jan 2023 05:28:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443708; cv=none; d=google.com; s=arc-20160816; b=P/lg3zYhZPq5ieNvDpOv91CkPxUgwgqCxFnylFUyNdwdMxe/iQXoyJXelqa1ka+9cX sXlIqp5lxJYE4tXywhq6c43nU0RkWVb1ttx3NQlqTn/brSZF/DUZ1PYWVUKb/2G9nGPr ri4GIP33kZTujU9hZKhi9G8jEJNfnXrhS4wHSKdkCZVoNuKbT9LSa9VtJFC86avu1k3I F7losMIi32Q7n8VJ/+tl4MiA5rQF7DqQ2sfGGc51hIhNPlHhQ/qmEHcQyLOhaHbofoSg JLJsc+ioJ7J1GcLSjWrsJsQ9X7Yo253xjeQG2FfyNs8JpnjeK+IrcAzrUUcN3xSotK+p +aMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OEfHSuGHxXrdy3tTJ9locmwSqOVY/2Ji+MT08R/VhuQ=; b=YApX6TSX4wm1T12o0RVWSt94Ddi3HGzpFqTQPDA6Wc6RjxGQDTHpMbNLbZYOudZbD/ uPPsCn7TSdWG44zuhDEs3as3bB3qu2pCZb7a5ZYcUJkytsF1F3f7q9uR3MrXviuXp+Pd tQlXZmo6gd3oCX03x4kSkXQT7mzdg6RfNv3aq5FcjVQFc3MDAoXsTwXK4UDDUl5q6eyB Bz31r1iRBkrT60Ny+b/nGckwCgtBAvB6qMDn9Z1SMSgcO746U0g0aY7caWeI2T2pNjwC D2fs87Hsd8p/AX56N1pJYjiWKSq9EhyrhHEzdn0soVEwCVxi7azHXlf3sZ2cyHTRBR3z WNfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=EQ2ZbevF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sh39-20020a1709076ea700b008361e2a67b2si3274360ejc.0.2023.01.11.05.28.04; Wed, 11 Jan 2023 05:28:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=EQ2ZbevF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229554AbjAKNZp (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234175AbjAKNYD (ORCPT ); Wed, 11 Jan 2023 08:24:03 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3D2F25EF for ; Wed, 11 Jan 2023 05:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443442; x=1704979442; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yKZbUwdFl1OELkYDtSril3bPVHQFFQ9B8sbyyX7BAAY=; b=EQ2ZbevFjOEvN9UyTZaGVsOxOPKZ8o2xgIq4ro1gin8J9WRslxBEv48V AHg6EJg7dqabeKEHImrNfIMJ3WonY2XGS2h1Ohy6Szhz6wNPRZ2F1u/vP 1A3QBu5NoXaHpjPkjxGjrzrb+UpsY90lSPdwYtWuph/DRoe3KNItY9eZZ n2L4RmFBMhnAVJ25nQBYylqaidjCA0Bs3UViqaulQSAWKxeh8vdU8LlIG wBBbHfwD4L5aXf3HcOBi7gADkZr+IQnXaE9PZw8azPkKcNxHHTRBKFnDo jVR2PjadwviMfMd8BkJACq9yecxCTBD4K7HgscHq4kJ1oVJ+cHGmU4lfG w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872549" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872549" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:24:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927431" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927431" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:56 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 53F0E109CB3; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 08/17] x86/mm: Reduce untagged_addr() overhead until the first LAM user Date: Wed, 11 Jan 2023 15:37:27 +0300 Message-Id: <20230111123736.20025-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732909476692893?= X-GMAIL-MSGID: =?utf-8?q?1754732909476692893?= Use static key to reduce untagged_addr() overhead. The key only gets enabled when the first process enables LAM. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/uaccess.h | 8 ++++++-- arch/x86/kernel/process_64.c | 4 ++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 32c9dd052e43..f9f85d596581 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -24,6 +24,8 @@ static inline bool pagefault_disabled(void); #endif #ifdef CONFIG_ADDRESS_MASKING +DECLARE_STATIC_KEY_FALSE(tagged_addr_key); + /* * Mask out tag bits from the address. * @@ -32,8 +34,10 @@ static inline bool pagefault_disabled(void); */ #define __untagged_addr(untag_mask, addr) ({ \ u64 __addr = (__force u64)(addr); \ - s64 sign = (s64)__addr >> 63; \ - __addr &= untag_mask | sign; \ + if (static_branch_likely(&tagged_addr_key)) { \ + s64 sign = (s64)__addr >> 63; \ + __addr &= untag_mask | sign; \ + } \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 88aae519c8f8..1b41c60ebf6e 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -745,6 +745,9 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #ifdef CONFIG_ADDRESS_MASKING +DEFINE_STATIC_KEY_FALSE(tagged_addr_key); +EXPORT_SYMBOL_GPL(tagged_addr_key); + #define LAM_U57_BITS 6 static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) @@ -781,6 +784,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) mmap_write_unlock(mm); + static_branch_enable(&tagged_addr_key); return 0; } #endif From patchwork Wed Jan 11 12:37:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41998 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3319667wrt; Wed, 11 Jan 2023 05:28:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXt4auMoUbcXsNFBo9oC6Dqsxrdr0PflsmMSjACS4LA1ugDvsNmwUJ4/w2IYKjnwWuT7RF87 X-Received: by 2002:a17:902:b18b:b0:192:6198:a51b with SMTP id s11-20020a170902b18b00b001926198a51bmr65558491plr.20.1673443710421; Wed, 11 Jan 2023 05:28:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443710; cv=none; d=google.com; s=arc-20160816; b=cL4b80w8ZmPMeXywXrH8wh8QW6sLFAglGqq4DpDeSOW2FZodHJxtBR1BVKYbNtVkl7 SxmhuNrVAXvB10AUFqvoAzGxR3HpNeNv8YX20VI0NF9+GIfwhPIpyZEEu+jyqoGJLeGZ fHmMZCwZ9uyNHZXpoCgn+YvtOhnWcbMGYqIYkyFmBMuCxrE9B+GEEe30UFSy0BsF02yO iSolzLmUXVONwMATns/xXKvfY3rJWY0SecVcz/F+OeWjeTplTlKKJuJ44W7vH5kq4VdZ v1DbnQOm7ROQ3nKS3rhepbbFoBgRYU8qJXIe4E1bIBvWcok9KnlYSN2CcUlXlOZEamQ4 5PCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JWhcpMTBZsFYvhyMDa+qvhTJQCJxY9DKtAeneQshjaY=; b=QN0JNnyFMpr7i2MorXA/XUdR0xRHlZo9Muto4MdCIBMekcEHi0Qj+tE4P2j5jOziIj cB1hdglcggyfSwenMXyI1WQZ/Fp+JKEwqthcf4qPoNyPLuhy1vWzXFn/CYtKf3aRWdCG FDqP5Z4t75DevwS0qMKHyhy1LaVyEUxdIUc24o1Q9IeXBhUQ3/oujXL251hhpcxbDhNS 91bnQwLuRF4kLieJYWK61ulpFS3QE0TpItwWmylAmO+sKK5t4gABLPwP8t579kEvxsaf k0jsMSbR3i1ePQBOHjGTTzjVjok6UYYgPrO2oQ+s9Ka2wyAN8rj35Yd5WOtIyoF0e/rm 91FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AVpoSwX0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c17-20020a170903235100b00192ff55301csi14671298plh.519.2023.01.11.05.28.17; Wed, 11 Jan 2023 05:28:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AVpoSwX0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238303AbjAKNZt (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234269AbjAKNYE (ORCPT ); Wed, 11 Jan 2023 08:24:04 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A32C3883 for ; Wed, 11 Jan 2023 05:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443443; x=1704979443; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Kszyd7Hv0tB7dUw4wLUAy/XS4ksOrmCTMURWZ8B+Snc=; b=AVpoSwX0IuU7AN2fypET9xIzUc6PvYi9UcRlmdGGiyfCrREaesuDdWbE DVN4NR7nLJlDro3/R3lXFu4InN1BjGcmar2yS1aFwfLVxO9Xqtkma3e+F qXrUNgut5xz6ZMUuETP4MAbphLhYsm5UOj21m2mUbVK69Va18ShutJvqP zlDIHWVIdib0IL82hBZD0dTnNb8rKmjLGwsd7jZq152c6ay2IMIAhAgby +Hso9nFq2J2/HDB1tNbV0K6e9XcdtnT8ROUcRD7UknygG2oiUw7wI5XHO FEnoRgzhxvZIm3ZPcIoSwUxGsNLMb5/6Z9vyd0hkLckSricV04RyJgGQl A==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872557" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872557" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:24:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927441" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927441" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:57 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 5F5FA109CB6; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Catalin Marinas Subject: [PATCHv14 09/17] mm: Expose untagging mask in /proc/$PID/status Date: Wed, 11 Jan 2023 15:37:28 +0300 Message-Id: <20230111123736.20025-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732911765840610?= X-GMAIL-MSGID: =?utf-8?q?1754732911765840610?= Add a line in /proc/$PID/status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Acked-by: Catalin Marinas --- arch/arm64/include/asm/mmu_context.h | 6 ++++++ arch/sparc/include/asm/mmu_context_64.h | 6 ++++++ arch/x86/include/asm/mmu_context.h | 6 ++++++ fs/proc/array.c | 6 ++++++ include/linux/mmu_context.h | 7 +++++++ 5 files changed, 31 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 72dbd6400549..56911691bef0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -288,6 +288,12 @@ void post_ttbr_update_workaround(void); unsigned long arm64_mm_context_get(struct mm_struct *mm); void arm64_mm_context_put(struct mm_struct *mm); +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> 8; +} + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h index 7a8380c63aab..799e797c5cdd 100644 --- a/arch/sparc/include/asm/mmu_context_64.h +++ b/arch/sparc/include/asm/mmu_context_64.h @@ -185,6 +185,12 @@ static inline void finish_arch_post_lock_switch(void) } } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> adi_nbits(); +} + #include #endif /* !(__ASSEMBLY__) */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 1d0b743daebb..4b77c96da07f 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -103,6 +103,12 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..d2a94eafe9a3 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +static inline void task_untag_mask(struct seq_file *m, struct mm_struct *mm) +{ + seq_printf(m, "untag_mask:\t%#lx\n", mm_untag_mask(mm)); +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -443,6 +448,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_mem(m, mm); task_core_dumping(m, task); task_thp_status(m, mm); + task_untag_mask(m, mm); mmput(mm); } task_sig(m, task); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index b9b970f7ab45..14b9c1fa05c4 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -28,4 +28,11 @@ static inline void leave_mm(int cpu) { } # define task_cpu_possible(cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) #endif +#ifndef mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} +#endif + #endif From patchwork Wed Jan 11 12:37:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41994 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3319061wrt; Wed, 11 Jan 2023 05:27:25 -0800 (PST) X-Google-Smtp-Source: AMrXdXtHuwvI+ZePi3PsaGV/PkYLll5f1mH5huAuI6/9IJcYbTVkdwEb4byqPk6A/n8fc62I3QJh X-Received: by 2002:a17:902:f80c:b0:189:340c:20c4 with SMTP id ix12-20020a170902f80c00b00189340c20c4mr74541010plb.66.1673443645555; Wed, 11 Jan 2023 05:27:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443645; cv=none; d=google.com; s=arc-20160816; b=mEMpYebmmoL7L+FsrghfTwIhwph+recyqxxiNwKwL8o4SFI0tyzIAi3PyyIxr9iM3E Co+r/+/tcVw/1KeLGC0PVws5oUpalgnQMAEmvHWYKj5A/DBmEUWhmEZ20Hz3WhGO6iFs 17Gu0iCRfO0GacOi+pKtSb8jwjjpyIh2OjP0cbz4X/HzHzBZG57Skgpfz/v3o7+Z6Qp2 dNjKqlcNSsLAAPL4HIn2aGgd8ts2pg2frayz3fhFNr2ic7G8YwO5ci0vhGt0/MbhpLE7 F/R4Z5ErAW5UtIFtDfxkZMJ3gCylwFQh7qeBmFal8kVWQiP+/biYSnz2wM8y/8KcV0k/ 2l4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RH27/yMEaIKNSxbdZlEBBsW8S4Z6jdHWiIv/A1SgiNc=; b=KuSWCa9a6aLeoegsVSkIyG7Gxp5LGjK3e4XpgG3/GPH9bpIXABZfpLgKOu1lVUOWXK QpfgvEPiYtF9xB7WJskCNlJdWjoQO+xL4LrH/2mmGJbtnm9YSTc26OZgCli7bmCIPp8e 343KGRrlDZQCvlWXlhoRLJHBJB5YUMBXxIhgYGwf20juzQ2iBqMjWZxP1TrqFTt2Plvp X1ileO8aKfqqit2xW3FgtQN/1dLkIr1O6J/jazACOcG4NToZsCBlpftBPoUerMHTMnM7 RRmn7nvmaGIikF/h/4/tkUDuLQmyufKkAVmC7QGT0VUnakpAqv6HnFz2z2ExlJqUyOnw 2wDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gnuA313b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w8-20020a63f508000000b0049f82fa59eesi15410390pgh.6.2023.01.11.05.27.07; Wed, 11 Jan 2023 05:27:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gnuA313b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233772AbjAKNZ3 (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234108AbjAKNYD (ORCPT ); Wed, 11 Jan 2023 08:24:03 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 806FA2ADE for ; Wed, 11 Jan 2023 05:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443442; x=1704979442; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Qz3TmQ0EtndSBYuEcqzm7EswKf58sIKEztNONQoPvAY=; b=gnuA313bAoIBAPuNBdvEdVVSIhZRadNOekdX+BJ8oiAa0PXDkLtdzyEy lmb1cbgkj1tpiRJYggPrfydX6uXpm+3pd4vI5CovA/gnb53+pNEI6tkcp 5D5DgjsyohU8ZJoGrYkrvQXYAe+xAaa64nOX4H47gyV2IOot6Q3KAqyiM 9DhumnokqwEKQmLA4g2lRjNFVj3mLu7xJLqFFTelQrwpuOB32ibl+NSjm keZulxuDNzsC9kDw2zte0Dej+0mU7lcx41hsm56YpIxu44PMfGs4wzuo8 D7a8+FAMTgpg6fWwstSsegshuy+ZHrEajyE7AUHnjPbcjoYSVNVQNu6gF g==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646140" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646140" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725495" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725495" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 68957109CC7; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 10/17] iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() Date: Wed, 11 Jan 2023 15:37:29 +0300 Message-Id: <20230111123736.20025-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732844428438614?= X-GMAIL-MSGID: =?utf-8?q?1754732844428438614?= Kernel has few users of pasid_valid() and all but one checks if the process has PASID allocated. The helper takes ioasid_t as the input. Replace the helper with mm_valid_pasid() that takes mm_struct as the argument. The only call that checks PASID that is not tied to mm_struct is open-codded now. This is preparatory patch. It helps avoid ifdeffery: no need to dereference mm->pasid in generic code to check if the process has PASID. Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/traps.c | 6 +++--- drivers/iommu/iommu-sva.c | 4 ++-- include/linux/ioasid.h | 9 --------- include/linux/sched/mm.h | 8 +++++++- 4 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..8b83d8fbce71 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -671,15 +671,15 @@ static bool try_fixup_enqcmd_gp(void) if (!cpu_feature_enabled(X86_FEATURE_ENQCMD)) return false; - pasid = current->mm->pasid; - /* * If the mm has not been allocated a * PASID, the #GP can not be fixed up. */ - if (!pasid_valid(pasid)) + if (!mm_valid_pasid(current->mm)) return false; + pasid = current->mm->pasid; + /* * Did this thread already have its PASID activated? * If so, the #GP must be from something else. diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 24bf9b2b58aa..4ee2929f0d7a 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -34,14 +34,14 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { if (mm->pasid < min || mm->pasid >= max) ret = -EOVERFLOW; goto out; } pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); - if (!pasid_valid(pasid)) + if (pasid == INVALID_IOASID) ret = -ENOMEM; else mm_pasid_set(mm, pasid); diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index af1c9d62e642..836ae09e92c2 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -40,10 +40,6 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); void ioasid_unregister_allocator(struct ioasid_allocator_ops *allocator); int ioasid_set_data(ioasid_t ioasid, void *data); -static inline bool pasid_valid(ioasid_t ioasid) -{ - return ioasid != INVALID_IOASID; -} #else /* !CONFIG_IOASID */ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, @@ -74,10 +70,5 @@ static inline int ioasid_set_data(ioasid_t ioasid, void *data) return -ENOTSUPP; } -static inline bool pasid_valid(ioasid_t ioasid) -{ - return false; -} - #endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..b69fe7e8c0ac 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -457,6 +457,11 @@ static inline void mm_pasid_init(struct mm_struct *mm) mm->pasid = INVALID_IOASID; } +static inline bool mm_valid_pasid(struct mm_struct *mm) +{ + return mm->pasid != INVALID_IOASID; +} + /* Associate a PASID with an mm_struct: */ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) { @@ -465,13 +470,14 @@ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) static inline void mm_pasid_drop(struct mm_struct *mm) { - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { ioasid_free(mm->pasid); mm->pasid = INVALID_IOASID; } } #else static inline void mm_pasid_init(struct mm_struct *mm) {} +static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; } static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif From patchwork Wed Jan 11 12:37:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41987 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318452wrt; Wed, 11 Jan 2023 05:26:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXtKc6BQgNLb+E5JrmodfvhDSRnJYufHJl/cvV0E33tU1H+dIdXUouOTuyxmvUcoXyL8Kqsp X-Received: by 2002:a05:6a21:2d8e:b0:a7:866d:40b5 with SMTP id ty14-20020a056a212d8e00b000a7866d40b5mr82314477pzb.6.1673443575040; Wed, 11 Jan 2023 05:26:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443575; cv=none; d=google.com; s=arc-20160816; b=qoluKK6zohR5pwStuO1vcmYh6teWR/SKVSTz8Q5B4nHwIXVs5G16rRlKlPdgg2A1qS hpS7pmhlFCm2oKGnx610PGXSYBvwTilkiLK46mw0RRq9+9YXcQXc8x5fFX9tvSvlZLrz cc6bc8avAAB5/jaF46yCtWOqnur7O/s8QgWFCYgQi1e+riXV90VOaC9XU62nbYynK/PR g5GazjiTBzjnqqeH/gIRnbOCOKnDI5Qo+dh3GwZCeMdspLN5COFI51UiFM0KdnWvnHFU lSMdl3+1em4F/l/ug+U+2tkkcj/o3Ugx0UKAxF/Ta7vDmRU8aBNS+DWCN2jEuD+0Yt00 JOdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4X3Yl3VDJMkgc/rTeU7ylnnNNVhA7d2827/V+2fvJYE=; b=TBmDsTSnB7fREpAXOXALPLuNpPllaW5uicEdUu+7yz+eBdJ+Sl4okDyWKIiLaWyqW2 WeYjuxMLREK/ISXjYahcuFM/TVQqFYoeKTYo1F/nMPgPa3rBxV53/rMfddqYWHS2uupD AUiGvQi6nPjJPgCT2ZJD3tjZPmyWrt9M/ZIt37IjyreymcTjlz+FYUDHCd7l4nSxuOql 1ex1ovUybe/fJKF/UtON0EACUAtwHnti48yntsH7zhb8owmJC/Cc6DdvhUuPJLbQhYBj pn5sM8ghoaWmsf+yRdMQJXAdzgDg1zIHGnTF9hUg3R1/NjBQvRD7OtN8GlWptcYyIdVu c1/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YhiyPcOO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k66-20020a632445000000b004a5b8385b68si16223181pgk.412.2023.01.11.05.26.02; Wed, 11 Jan 2023 05:26:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YhiyPcOO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234277AbjAKNZM (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233948AbjAKNYB (ORCPT ); Wed, 11 Jan 2023 08:24:01 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 639832632 for ; Wed, 11 Jan 2023 05:24:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443440; x=1704979440; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C8PPsmcjUUuQBbdSz1HmIC78H3wDV2qhKG6GO472d7A=; b=YhiyPcOO1f+HFK31XrdkWIHvcw7JnzbmEaC2H587bxjlycIvwtlqqtez RivWcW3gKeFDLRjau6+ABxSF8RYtjVpAs/95OiuC8DJ+v4SZ+aDmfsVQT 6P4ow838gKcVtyJ/g4362QGSkea16MoEKMAb3QJL0+RnV8JXJOKLSCTGI ly+Qm1YJFs4KtYGGBsGhi41LJq/FEfOaXxsot5opseBEYe/4ZfD5zHCYM +rHYXj74q6HFVjuQvX4kpMSe4P5K+tHhS+ckdKgEQ3GphKMc0CeoAtmJK eZfpeJjt76oBJLHdvmzJ/55/rOF8u8lLUn/juKPafHDJoYtlHRLupHjzM w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646093" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646093" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:54 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725481" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725481" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:49 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 73247109CE1; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 11/17] x86/mm/iommu/sva: Make LAM and SVA mutually exclusive Date: Wed, 11 Jan 2023 15:37:30 +0300 Message-Id: <20230111123736.20025-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732770053798600?= X-GMAIL-MSGID: =?utf-8?q?1754732770053798600?= IOMMU and SVA-capable devices know nothing about LAM and only expect canonical addresses. An attempt to pass down tagged pointer will lead to address translation failure. By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. By using the arch_prctl() userspace takes responsibility to never pass tagged address to the device. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ashok Raj Reviewed-by: Jacob Pan --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 7 +++++++ drivers/iommu/iommu-sva.c | 4 ++++ include/linux/mmu_context.h | 7 +++++++ 6 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index e80762e998ce..0da5c227f490 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -14,6 +14,8 @@ #define MM_CONTEXT_HAS_VSYSCALL 1 /* Do not allow changing LAM mode */ #define MM_CONTEXT_LOCK_LAM 2 +/* Allow LAM and SVA coexisting */ +#define MM_CONTEXT_FORCE_TAGGED_SVA 3 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 4b77c96da07f..6ffc42dfd59d 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -114,6 +114,12 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) mm->context.untag_mask = -1UL; } +#define arch_pgtable_dma_compat arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return !mm_lam_cr3_mask(mm) || + test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags); +} #else static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index a31e27b95b19..eb290d89cb32 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -23,5 +23,6 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 1b41c60ebf6e..0831d2be190f 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -759,6 +759,10 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) if (current->mm != mm) return -EINVAL; + if (mm_valid_pasid(mm) && + !test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags)) + return -EINTR; + if (mmap_write_lock_killable(mm)) return -EINTR; @@ -882,6 +886,9 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_FORCE_TAGGED_SVA: + set_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &task->mm->context.flags); + return 0; case ARCH_GET_MAX_TAG_BITS: if (!cpu_feature_enabled(X86_FEATURE_LAM)) return put_user(0, (unsigned long __user *)arg2); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 4ee2929f0d7a..dd76a1a09cf7 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -2,6 +2,7 @@ /* * Helpers for IOMMU drivers implementing SVA */ +#include #include #include #include @@ -32,6 +33,9 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) min == 0 || max < min) return -EINVAL; + if (!arch_pgtable_dma_compat(mm)) + return -EBUSY; + mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ if (mm_valid_pasid(mm)) { diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index 14b9c1fa05c4..f2b7a3f04099 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -35,4 +35,11 @@ static inline unsigned long mm_untag_mask(struct mm_struct *mm) } #endif +#ifndef arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return true; +} +#endif + #endif From patchwork Wed Jan 11 12:37:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41982 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318058wrt; Wed, 11 Jan 2023 05:25:44 -0800 (PST) X-Google-Smtp-Source: AMrXdXsSbVc+KgFahTpjBKbXH+PbTVoNgSMufQXKbhE3WQeUN5GXck+RwUqylcdbZO8LP3J/8boH X-Received: by 2002:a05:6a00:1d03:b0:580:149a:5650 with SMTP id a3-20020a056a001d0300b00580149a5650mr73639492pfx.22.1673443543655; Wed, 11 Jan 2023 05:25:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443543; cv=none; d=google.com; s=arc-20160816; b=nk8DoyovVsviqrSy3QW32TxguK7trHaOSvio30ajlax0Es9APkJWeV9ZdeyV2pNghG Xzda1MaqYvxEdM9S1elPf6fQIPpCMKMq+tXB+PD/u3PwWng+d504GigWabhUDKDYc1rQ IAbf89xIvDa63IgM6L5mUIcUkV23RSKvI6vo283fytE7Dp7GNw4/iadWpUC2SV/F7N0K Z7T9JUOZupnVYLmiLgpPfPkfZjiW/ZFQDbHSJHXdudYFNEwC3kxY0YgNttbey5S1rYDs VczdVeIjNiXbVQT2/mCE/qMDb1fFfqP50QDeAbv/qcTgJDzsEhiOOzpOb/wEXpbpJUrW gKQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CcTvrgQKQ/rkXiQxaI8bwSJNlFdXphQIHvzJF4nmf1U=; b=KsaZb9kQ2Pw5VIHt41qCRsb3Su9SxMbY2sf0opg4XlZaTQilrzZsikc3dix9mwe0Ly RcgI/uXmvrYuUZrQRkT4IB2/gWhdmlsDviRoab7gX+rLknzkHpi1pDLufmDJKHwFyc/o 4zVc2luWjqP9LzKzPHoDp+gltCX0GvSwojPjMEIUM/gYfPWKGWMyyAJ2OwoMFNp/M1Df 52wc9LAn2vpyl2Fuiqn85wrxvIKlszaWUVaJzY8Bt94q4gAObCthQP/2QCfy0HWoyPOo nDcPddf2PNnG7wXEvmxkVd1nyuLnMxNGAX+IJQbuZ5XCaH3X6mmx6vE8a537rmzEbasl pRRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dfET4tbz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h12-20020a056a00230c00b00561898445bdsi15421492pfh.273.2023.01.11.05.25.30; Wed, 11 Jan 2023 05:25:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dfET4tbz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233768AbjAKNX7 (ORCPT + 99 others); Wed, 11 Jan 2023 08:23:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232854AbjAKNXu (ORCPT ); Wed, 11 Jan 2023 08:23:50 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C60B2BE8 for ; Wed, 11 Jan 2023 05:23:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443429; x=1704979429; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zY+AHqWaGn5qzJju0QvduZoCFJgbmdkX0N6fUdecUBc=; b=dfET4tbzlTitYHHzlmiAhaTNrXJqniNqymg9Pd1iXq69O+aGPOJGmUoW YxQYGaB0VjKFeT0297gg7kpjZ1ghTeDN7xjd8ovB8q/SQByQ20+OKGjzD xjrd36mWTdMDJrHOY1AQZZ04BVskfHMuVsAKVmOTdTqwAP9+nJryHY6GN eEXXKbcC77yY7v7yiF/qyeORRaM1X6C0jEPdHTO1EL4SLuRQV30GYPcd+ sYwHCGrsmBYin8tehrepVq1dBsG7Tf0/rS+MqlDGfE0N/EwR4Vq6veFLJ aupHyDEVrLnDUJ6wwKgahSfNPLqaDyAwvX5a5+1wBuQiK1kAJqGoiwhpm g==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872504" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872504" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927406" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927406" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:42 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 7EAF8109CE4; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv14 12/17] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking Date: Wed, 11 Jan 2023 15:37:31 +0300 Message-Id: <20230111123736.20025-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732737266435687?= X-GMAIL-MSGID: =?utf-8?q?1754732737266435687?= From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are some arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: MALLOC - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. TAG_BITS - Max tag bits of LAM_U57 is 6. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 326 +++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..c1a16a9d4f2f 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..268c1d2749af --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 + +#define LAM_U57_MASK (0x3fULL << 57) +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + * + * @return: + * 0: Set LAM mode successfully + * others: failed to set LAM + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(LAM_U57_MASK)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +static unsigned long get_default_tag_bits(void) +{ + pid_t pid; + int lam = LAM_NONE; + int ret = 0; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + } else if (pid == 0) { + /* Set LAM mode in child process */ + if (set_lam(LAM_U57_BITS) == 0) + lam = LAM_U57_BITS; + else + lam = LAM_NONE; + exit(lam); + } else { + wait(&ret); + lam = WEXITSTATUS(ret); + } + + return lam; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t set_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + /* Get a random non-zero value as metadata */ + metadata = (rand() % ((1UL << LAM_U57_BITS) - 1) + 1) << 57; + metadata |= (src & ~(LAM_U57_MASK)); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + * + * @return: + * 0: value on the pointer with metadate and value on original are same + * 1: not same. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)set_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + + +int handle_max_bits(struct testcases *test) +{ + unsigned long exp_bits = get_default_tag_bits(); + unsigned long bits = 0; + + if (exp_bits != LAM_NONE) + exp_bits = LAM_U57_BITS; + + /* Get LAM max tag bits */ + if (syscall(SYS_arch_prctl, ARCH_GET_MAX_TAG_BITS, &bits) == -1) + return 1; + + return (exp_bits != bits); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg = "BITS: Check default tag bits\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + if (tests & FUNC_BITS) + run_test(bits_cases, ARRAY_SIZE(bits_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Wed Jan 11 12:37:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41992 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318930wrt; Wed, 11 Jan 2023 05:27:10 -0800 (PST) X-Google-Smtp-Source: AMrXdXvXZOUQfOU+sLX0YSXxadQx9jJh6vdhSXoTtpr/WiWuaZsiNo8o8nr8R8MbtmCNXDHWsT6T X-Received: by 2002:a17:906:92d2:b0:861:3ed5:e029 with SMTP id d18-20020a17090692d200b008613ed5e029mr863749ejx.49.1673443630263; Wed, 11 Jan 2023 05:27:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443630; cv=none; d=google.com; s=arc-20160816; b=JmKiKGy9aNR6nhCeeEsMDkFD6Zw+/16mph1vgkf90BcyZMgdmuN7Bqp3SqagJoPcYD 5gUWhR/j3z6FQI5LwZHSaLGbHEHKQxjfIVNtHvzTP+O8FBB/jZb3EbQQaoTK+2aMgSiI SbIUWKO7LJzn3WqncIlpF/OgHdbBB3JKmYe5DrEabn+51h048QBNPMKpPQUOs2aIUCHU ULKF9M9yvoNmQin8cGl/tBnjmi2qxPC1TPBmwtSTaCmKl/SPxXalIEZYHM2SsbkUSARr 5OZGyvl9atilAoyMHODxKcipKKh1+w1ucA5Om8Xw3T6LgQgqPBY199IvIJY56hxsLuBn 5Nhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vHQ3z/K+R6lu2S//JkDKczmFG2Z/EZNup2Lu4cZ6hXs=; b=WQpEBW97lcxOSTvc4OveM1TULTq8Quzt63VcmJsfmVYiWZJRFcc+CJmDpSNiBho5j1 SB2ytiwtmWX0l/aEVT9rgzMgdyERdOCCsD7hyNm1bpma62fheCM8kkkGcvVwZ98B5uss 8Qi0UNSWgSTSoVlzrv4rimmuOk1+k1teC7n9nhou/ONd/h3frOq9SuuwvdNW4AKYhiaw +Pm0rut39bb47vim6TV/wIpgdnOwxAEHJMjhvLQsLB4E5mYdZ3I3g4kqF+3/5BX8u80i zK2iw0i5F0HCtH/Zzimi6s2u2lV+zgPjoimxYYAs0IjihTNjYwErHy4JiA3YqFE0sgWd P8+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kIF4G8MF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y10-20020a056402270a00b0048e913098c1si17178146edd.398.2023.01.11.05.26.46; Wed, 11 Jan 2023 05:27:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kIF4G8MF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238088AbjAKNZl (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234187AbjAKNYD (ORCPT ); Wed, 11 Jan 2023 08:24:03 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EEC72632 for ; Wed, 11 Jan 2023 05:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443442; x=1704979442; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Xcv6q+YR2BNToaMGTKdVS78DajTVDdfIYocqspXOJIA=; b=kIF4G8MF0PlBoJVg1P5z4yqTKuqvhIuTHi0KUTPCON+rxs1CyxgVnOrI sn5DayeX8dOdyojzZ0q6ydkpZgRWpBS62W5DV97Gh41DGPAMwY0qjrCez ZljMJBQMWvsMXdggthknmNi78o3qery3cS7qL2/bp5tBdVJEuMzlJVIwx 6FGH+0qzDJd785kSfVTSX0QHFohmF9CfcSQmrDXpOZNriMpIjh3BTFJTY nlM+HeBlavp4uHwyYnmZtsTfY8wfhbdE2yp6r1wfh/a/xXhqDFwYGLR88 h6pEkikeUAdTNHtpFxINiPVo0wjSk9t12nOf6ZkDG4ot+ELKTKNAKr5Bx Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646119" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646119" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725486" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725486" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 8A284109CF7; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv14 13/17] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Wed, 11 Jan 2023 15:37:32 +0300 Message-Id: <20230111123736.20025-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732828112241830?= X-GMAIL-MSGID: =?utf-8?q?1754732828112241830?= From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 144 +++++++++++++++++++++++++++++- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 268c1d2749af..39ebfc511685 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -29,11 +30,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -49,6 +57,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -61,6 +70,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -213,6 +232,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)set_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -241,13 +322,20 @@ static void run_test(struct testcases *test, int count) struct testcases *t = test + i; /* fork a process to run test case */ + tests_cnt++; ret = fork_test(t); + + /* return 3 is not support LA57, the case should be skipped */ + if (ret == 3) { + ksft_test_result_skip(t->msg); + continue; + } + if (ret != 0) ret = (t->expected == ret); else ret = !(t->expected); - tests_cnt++; ksft_test_result(ret, t->msg); } } @@ -268,7 +356,6 @@ static struct testcases malloc_cases[] = { }, }; - static struct testcases bits_cases[] = { { .test_func = handle_max_bits, @@ -276,11 +363,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); printf("\t-h: help\n"); } @@ -320,6 +450,12 @@ int main(int argc, char **argv) if (tests & FUNC_BITS) run_test(bits_cases, ARRAY_SIZE(bits_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Jan 11 12:37:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41986 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318413wrt; Wed, 11 Jan 2023 05:26:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXsScXFJbekElGitg1oIdldTkclGd38ot4cYF9fozJdNN1rCLf2nYmLQrLnqHm8oXx9kdjNM X-Received: by 2002:a05:6a00:2997:b0:582:1f25:5b8 with SMTP id cj23-20020a056a00299700b005821f2505b8mr41436079pfb.19.1673443571394; Wed, 11 Jan 2023 05:26:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443571; cv=none; d=google.com; s=arc-20160816; b=mVx/+Ig0oEoYv5/QQ0ySZQklNT0nd3Kg8gUcQuKTxaq9h/P4JuNegV2AAebg85zUtE 6PlRR6hcXIdWM5ZU/DcGozmr8CYdyyUMLLF4JXyWJAGQ3UwHy04hDkNqJ9gEACrjsAut M+SbiT02GB89OufUvu+KY23f9SnfJumzH4FXxcAiJhis7/iBF2ybGLCsvQ51aVlczGJm hc8W1fsEFv6whO8GADz9APmrffUtqGjnl81fLXwPNf4VjG7qEcfB2Oj5/CUftu8TG5XM iZrPUoszo2XFOS9RZRC9uqs+C08R5GeDsNzeTVGuPbmPGWIwIdR+EFsgmzVtr8vAhfXz bwVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uuiBgt0fB2kTjl1Ku+81wfV6gcDi8StWysDGKsByKac=; b=rBUhpNTuVnmCZZiP2W+h8bv2/i4zk8ekrMD24OEAQ6lt8FLgHd0ef8jBpJKfH/221t luAIO7svJDTh4v74BCW8RVjR040x5CnXJ46E2erXW+0PHRK6kt+FPe/KGbo5dYhT1ERq ZrhriviUhC+rK5Ae7X/H6rRidkl0kla0344ybGe+7U+4wHQ8Dkeri/X/okdMx2Td7Hu0 fUZZW3XS0WGpDh/3RC2JpIxMPTY5+oHpjykF56RA3dPoM/lDx2kLUs3deSvXuB8V8cO1 Hidu3vCD18I7h7mTIQaoOyCbHG4vBkN7XBlsnZCOHHal4Anw+zf5xt+QM43mYN1lCVWK Wmxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QIpaVC9h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f21-20020a056a0022d500b00580fb9dfd95si14998430pfj.186.2023.01.11.05.25.59; Wed, 11 Jan 2023 05:26:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QIpaVC9h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236002AbjAKNYr (ORCPT + 99 others); Wed, 11 Jan 2023 08:24:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233744AbjAKNX5 (ORCPT ); Wed, 11 Jan 2023 08:23:57 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 935FB25EF for ; Wed, 11 Jan 2023 05:23:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443436; x=1704979436; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9gNQ5WNG2l+gZLhtnf8anitiVNgYtGVQwEkYU4Rdi08=; b=QIpaVC9hDtK/2dJ6XWKt/my90cTdAFprdh2gjhkeEqqLBz2Gv1jab0oH DXzE5F69lyZOv6T8HA5f9D4AowINTuc0wZSNU8cBdMtiIPLwYyinBgZf6 64RKx4mGjDClKDMavYKal1r+ElEiVUNP761dec5zkNkAjzL5NjY5kvw5Z JCPp0OmBC/PBVCGpGrk/L2DvvlMhBwh6lSTF/vq9NMkEWOY7Q8LEVOpre AjO+vLZZdeuwcbUUMg3eNSk0yE80aesaEOU+bE3ECeR1ZsE2zJIb0VK3E wKR4ZjSgKEHBH/4+A9Crx8rcb8trBOGOZNCEqKj1bW8WSH3HVyBi+M0Yq A==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872531" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872531" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927419" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927419" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 95E44109D23; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv14 14/17] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Wed, 11 Jan 2023 15:37:33 +0300 Message-Id: <20230111123736.20025-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732766320080575?= X-GMAIL-MSGID: =?utf-8?q?1754732766320080575?= From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 39ebfc511685..52750ebd0887 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -32,8 +36,9 @@ #define FUNC_BITS 0x2 #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -42,6 +47,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -51,6 +63,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -294,6 +333,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)set_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -340,6 +658,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -410,7 +744,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); printf("\t-h: help\n"); } @@ -456,6 +790,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Jan 11 12:37:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3319455wrt; Wed, 11 Jan 2023 05:28:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXsF4OE5IKrxkvFjA2JTdU3ddlS8AZVYgR7KP9e2j3pG3H1O5CUPyjQcNq2KFFljP4gDTvzj X-Received: by 2002:a17:902:ce08:b0:189:81dd:6b8b with SMTP id k8-20020a170902ce0800b0018981dd6b8bmr100455113plg.63.1673443684503; Wed, 11 Jan 2023 05:28:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443684; cv=none; d=google.com; s=arc-20160816; b=zK3N+uOnjZo4XxX2OQ7rcIRzQDOw3kG0n1Ob23NYzCqHk4mSlWUBFS8IokCvh4Oe2X WRGdpKNnWvK17fp6RTclNaLQBwL0wgeyrOTeqnEMArmrvmYuvBV8QkWZkntkXEDpVe4x I1PKv8MtVHlcZM9NjJ3/8IGgo7WaBmrnEzwcS958sG6sgkokqQudtDsrGunkYD5CVYeA oGJ5/WZBr8pzGAla+gmXN369ATG+sA5dcHJcZ0oh6yYwcJSKiFktCEMA4h22hZmcQs+y tcPRgTtsZD1cqvf6k84aiU7JoF7UeP6y0d3TOFEyjZZ1nrde6LgzzzP+UbImxq2eTCnF 94Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Iz+2YzFeFq8aEoQ4mjxDQbwLq4DS3oV50HIO+IOed8M=; b=PZNqMt4fcqPLfI9kvKiXlmIRVfIKpbZ+vaLHibY6zp3FkmgsYHuGptsifAJSf85w18 B9onhRh5cj2xbrDAALYJN/y6xuALPaTz/ha4VZlM37L/VGLp/wlxj9lyZ+MtOr/K5C6f 0S0PKt0CqfxBzWsYTggbxNHw1h2v8aG0Txd33ePu+Vqilr3YPSt8l6CDUaOMU0bVQKmL QXSGrlpeXzQLXmCfxEfiHoAFATvEA7iz6zqDVdCeO+Dp/kIyQBpK8Of4wj7ik3jeFz9Y UL68DgryTxxEL1Dk/CrIhqMFnN5mjZiolK+i2Q763CWjhDA2aiRKDZyX5Tg076yygwhP VdPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h90+8BQC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f6-20020a17090274c600b00189ac5a25e8si13648328plt.160.2023.01.11.05.27.51; Wed, 11 Jan 2023 05:28:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h90+8BQC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235892AbjAKNYj (ORCPT + 99 others); Wed, 11 Jan 2023 08:24:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232778AbjAKNX4 (ORCPT ); Wed, 11 Jan 2023 08:23:56 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C70792BE8 for ; Wed, 11 Jan 2023 05:23:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443435; x=1704979435; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/BxgmUMqCgv4xO8mgnvpl691Lp21f1iYYEA3Da/Uyw0=; b=h90+8BQCE33c+xktM6AxsAJj8xVY6TUWRRK7maez5JdsCRi3lTlYdS+R YGmP/wqsf2r+erCRfV2Jvcjb1tCsK89aUdJ6YklvZwJZ/xSNauRLlqiMi mi7MhOjX8zp/lerySx+BTIF2ZgS3hosE/dc1IIgV1M+yf1gX2IL3P/Iv7 +UbCpjZ/SiPllkitzxrjCpVEtqEqF6B+ozzP7KrXJ1xdmFD7dKZ7Vj1eM TAu5KGfhKOkMvVHjPazTE8UQOKf5J9TqA7ZRzxOgpiC5q6hj42aZYy+lZ FDS9RJUnGIZ7Kf7hUMU6oyl5RBGsdokX6bQXPR1YfJR0xftZy62tpfHpM Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872523" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872523" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927417" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927417" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id A05BD109D24; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv14 15/17] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Wed, 11 Jan 2023 15:37:34 +0300 Message-Id: <20230111123736.20025-16-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732885028782209?= X-GMAIL-MSGID: =?utf-8?q?1754732885028782209?= From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 125 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 52750ebd0887..ebabd4333b7d 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -37,8 +37,9 @@ #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -174,6 +175,28 @@ static unsigned long get_default_tag_bits(void) return lam; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(LAM_U57_MASK)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t set_metadata(uint64_t src, unsigned long lam) { @@ -581,7 +604,7 @@ int do_uring(unsigned long lam) switch (lam) { case LAM_U57_BITS: /* Clear bits 62:57 */ - addr = (addr & ~(0x3fULL << 57)); + addr = (addr & ~(LAM_U57_MASK)); break; } free((void *)addr); @@ -632,6 +655,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -740,11 +829,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n"); printf("\t-h: help\n"); } @@ -764,7 +868,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -778,6 +882,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -793,6 +907,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Jan 11 12:37:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41988 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318688wrt; Wed, 11 Jan 2023 05:26:44 -0800 (PST) X-Google-Smtp-Source: AMrXdXtBr4mPjCVpVxOqXDopsV+VzeCN/Ub04XJGicVI3k4a0ZO9QNiSjbxNhDg/PRtxCeSho1Of X-Received: by 2002:a17:90a:380c:b0:227:199c:e66 with SMTP id w12-20020a17090a380c00b00227199c0e66mr2464992pjb.2.1673443603844; Wed, 11 Jan 2023 05:26:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443603; cv=none; d=google.com; s=arc-20160816; b=sxPJGwhaLASDCBl9P00wlpzG2auYkVxT2AX38BzCgkXGqzUhf8KkdSA84Hn9066Pm0 v3NH2lvyOFCfSrY2OGGCUAgiTaBJz4K0BALCE45XyTQmvVqx2FW3e3KTv/1R4jcmjImU 0So/yyM9IAOGEhDSmqUTvgwe/xI3UzZJyrvaIFvESIal6n8KbU8t7T7lieTcA868jSz8 o7rLbsP4xdDjy025uM1BZ5JrM+HK+86ArcNme6Q35V1rUyh+JLtIWqdossrb9WwXjzOE JKDfsBgsffECqGaS/EbFoLi1eSmJv7YmAE74cy+cIBUVahfkBuiLU7e8laq0MfJAp62A mlrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kSVd57pM2rbZ/kZ1M6v3+isQB8js5rLSnrkT38sGcZo=; b=Bua/ARMWFo6+cTslvblmpb7PM1Haf/dJ+UUcM1aMI91RJ5w6KK/P/444OzAkPebTQN HWQGfRuV4302fs2tO+zAgN+AOVhgwPH1olR70tAtscYC4eaDymH8hSBl4yHFBxMt6XTe sBkqbuC9+RcWwPltudilwnnlOOc99mS1GCl9PDT30J4tXvOg8ZmoKqCbRosiWbVtnM+q 2iAYXry26YW27OPQRQaWOn7KDwV+a4fz+PFcD6P0q1F/ce1qvVV6pdlCYIgqo2hqzmIH 4L5InW+RQVUcU+PHMA7/kXpVU80knk6tH+IPU3wHFgCwq7E2ul1ezT05otj3tUJjSMLk pjwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NDO3T8gE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 66-20020a630145000000b0049cd15c93fcsi14701359pgb.625.2023.01.11.05.26.31; Wed, 11 Jan 2023 05:26:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NDO3T8gE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236005AbjAKNZe (ORCPT + 99 others); Wed, 11 Jan 2023 08:25:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234133AbjAKNYD (ORCPT ); Wed, 11 Jan 2023 08:24:03 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80F2D2BE8 for ; Wed, 11 Jan 2023 05:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443442; x=1704979442; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=H8MFSa5pt0FmaVqEcF+CsDPjHHBDTbBE3p2CVIKRTN8=; b=NDO3T8gEDJsumCm9ZASnfcMl6f6ydxp8EvE/NrPwlauO6xn/86mfUamw wqdUs3VVTv3LOuhtS5zfQatuziJqv3uS401vl0mTVu/R7a6eCX+C4BIek XFbr+d2TgcqKF5EG5fo+/J42G77+e3utn8QWKB2xWbZSrB3M1fHFFKL+X MWI2WvJ5HhnbP0ngsLHGtQbR2y/BnCeq8wOHVOV4Eh4oXV12ENJxIwErY Trmao/QulCUMktwuo3/QnNA72GYxIxu5Rx4fNi230RRivxhU19wc2f16u 2cmNcaQegVCi2TFCUQRMS8KBEnq8SrX0rSVpzB8yUTEZoXIuASTUNhLXs Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="324646138" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="324646138" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720725490" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="720725490" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:50 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id AB414109D2A; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv14 16/17] selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking Date: Wed, 11 Jan 2023 15:37:35 +0300 Message-Id: <20230111123736.20025-17-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732800264441916?= X-GMAIL-MSGID: =?utf-8?q?1754732800264441916?= From: Weihong Zhang By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. Add new test cases for the new arch_prctl: Before using ARCH_FORCE_TAGGED_SVA, should not allow to enable LAM/SVA coexisting. the test cases should be negative. The test depands on idxd driver and iommu. before test, need add "intel_iommu=on,sm_on" in kernel command line and insmod idxd driver. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 237 +++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index ebabd4333b7d..a8c91829b616 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -30,6 +30,7 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 /* Specified test function bits */ #define FUNC_MALLOC 0x1 @@ -38,8 +39,9 @@ #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 #define FUNC_INHERITE 0x20 +#define FUNC_PASID 0x40 -#define TEST_MASK 0x3f +#define TEST_MASK 0x7f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -55,11 +57,19 @@ #define URING_QUEUE_SZ 1 #define URING_BLOCK_SZ 2048 +/* Pasid test define */ +#define LAM_CMD_BIT 0x1 +#define PAS_CMD_BIT 0x2 +#define SVA_CMD_BIT 0x4 + +#define PAS_CMD(cmd1, cmd2, cmd3) (((cmd3) << 8) | ((cmd2) << 4) | ((cmd1) << 0)) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ unsigned long lam; uint64_t addr; + uint64_t cmd; int (*test_func)(struct testcases *test); const char *msg; }; @@ -556,7 +566,7 @@ int do_uring(unsigned long lam) struct file_io *fi; struct stat st; int ret = 1; - char path[PATH_MAX]; + char path[PATH_MAX] = {0}; /* get current process path */ if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) @@ -852,6 +862,226 @@ static void cmd_help(void) printf("\t-h: help\n"); } +/* Check for file existence */ +uint8_t file_Exists(const char *fileName) +{ + struct stat buffer; + + uint8_t ret = (stat(fileName, &buffer) == 0); + + return ret; +} + +/* Sysfs idxd files */ +const char *dsa_configs[] = { + "echo 1 > /sys/bus/dsa/devices/dsa0/wq0.1/group_id", + "echo shared > /sys/bus/dsa/devices/dsa0/wq0.1/mode", + "echo 10 > /sys/bus/dsa/devices/dsa0/wq0.1/priority", + "echo 16 > /sys/bus/dsa/devices/dsa0/wq0.1/size", + "echo 15 > /sys/bus/dsa/devices/dsa0/wq0.1/threshold", + "echo user > /sys/bus/dsa/devices/dsa0/wq0.1/type", + "echo MyApp1 > /sys/bus/dsa/devices/dsa0/wq0.1/name", + "echo 1 > /sys/bus/dsa/devices/dsa0/engine0.1/group_id", + "echo dsa0 > /sys/bus/dsa/drivers/idxd/bind", + /* bind files and devices, generated a device file in /dev */ + "echo wq0.1 > /sys/bus/dsa/drivers/user/bind", +}; + +/* DSA device file */ +const char *dsaDeviceFile = "/dev/dsa/wq0.1"; +/* file for io*/ +const char *dsaPasidEnable = "/sys/bus/dsa/devices/dsa0/pasid_enabled"; + +/* + * DSA depends on kernel cmdline "intel_iommu=on,sm_on" + * return pasid_enabled (0: disable 1:enable) + */ +int Check_DSA_Kernel_Setting(void) +{ + char command[256] = ""; + char buf[256] = ""; + char *ptr; + int rv = -1; + + snprintf(command, sizeof(command) - 1, "cat %s", dsaPasidEnable); + + FILE *cmd = popen(command, "r"); + + if (cmd) { + while (fgets(buf, sizeof(buf) - 1, cmd) != NULL); + + pclose(cmd); + rv = strtol(buf, &ptr, 16); + } + + return rv; +} + +/* + * Config DSA's sysfs files as shared DSA's WQ. + * Generated a device file /dev/dsa/wq0.1 + * Return: 0 OK; 1 Failed; 3 Skip(SVA disabled). + */ +int Dsa_Init_Sysfs(void) +{ + uint len = ARRAY_SIZE(dsa_configs); + const char **p = dsa_configs; + + if (file_Exists(dsaDeviceFile) == 1) + return 0; + + /* check the idxd driver */ + if (file_Exists(dsaPasidEnable) != 1) { + printf("Please make sure idxd driver was loaded\n"); + return 3; + } + + /* Check SVA feature */ + if (Check_DSA_Kernel_Setting() != 1) { + printf("Please enable SVA.(Add intel_iommu=on,sm_on in kernel cmdline)\n"); + return 3; + } + + /* Check the idxd device file on /dev/dsa/ */ + for (int i = 0; i < len; i++) { + if (system(p[i])) + return 1; + } + + /* After config, /dev/dsa/wq0.1 should be generated */ + return (file_Exists(dsaDeviceFile) != 1); +} + +/* + * Open DSA device file, triger API: iommu_sva_alloc_pasid + */ +void *allocate_dsa_pasid(void) +{ + int fd; + void *wq; + + fd = open(dsaDeviceFile, O_RDWR); + if (fd < 0) { + perror("open"); + return MAP_FAILED; + } + + wq = mmap(NULL, 0x1000, PROT_WRITE, + MAP_SHARED | MAP_POPULATE, fd, 0); + if (wq == MAP_FAILED) + perror("mmap"); + + return wq; +} + +int set_force_svm(void) +{ + int ret = 0; + + ret = syscall(SYS_arch_prctl, ARCH_FORCE_TAGGED_SVA); + + return ret; +} + +int handle_pasid(struct testcases *test) +{ + uint tmp = test->cmd; + uint runed = 0x0; + int ret = 0; + void *wq = NULL; + + ret = Dsa_Init_Sysfs(); + if (ret != 0) + return ret; + + for (int i = 0; i < 3; i++) { + int err = 0; + + if (tmp & 0x1) { + /* run set lam mode*/ + if ((runed & 0x1) == 0) { + err = set_lam(LAM_U57_BITS); + runed = runed | 0x1; + } else + err = 1; + } else if (tmp & 0x4) { + /* run force svm */ + if ((runed & 0x4) == 0) { + err = set_force_svm(); + runed = runed | 0x4; + } else + err = 1; + } else if (tmp & 0x2) { + /* run allocate pasid */ + if ((runed & 0x2) == 0) { + runed = runed | 0x2; + wq = allocate_dsa_pasid(); + if (wq == MAP_FAILED) + err = 1; + } else + err = 1; + } + + ret = ret + err; + if (ret > 0) + break; + + tmp = tmp >> 4; + } + + if (wq != MAP_FAILED && wq != NULL) + if (munmap(wq, 0x1000)) + printf("munmap failed %d\n", errno); + + if (runed != 0x7) + ret = 1; + + return (ret != 0); +} + +/* + * Pasid test depends on idxd and SVA, kernel should enable iommu and sm. + * command line(intel_iommu=on,sm_on) + */ +static struct testcases pasid_cases[] = { + { + .expected = 1, + .cmd = PAS_CMD(LAM_CMD_BIT, PAS_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute LAM, PASID, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(LAM_CMD_BIT, SVA_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute LAM, SVA, PASID in sequence\n", + }, + { + .expected = 1, + .cmd = PAS_CMD(PAS_CMD_BIT, LAM_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute PASID, LAM, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(PAS_CMD_BIT, SVA_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute PASID, SVA, LAM in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, LAM_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, LAM, PASID in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, PAS_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, PASID, LAM in sequence\n", + }, +}; + int main(int argc, char **argv) { int c = 0; @@ -910,6 +1140,9 @@ int main(int argc, char **argv) if (tests & FUNC_INHERITE) run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + if (tests & FUNC_PASID) + run_test(pasid_cases, ARRAY_SIZE(pasid_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Wed Jan 11 12:37:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 41981 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3318028wrt; Wed, 11 Jan 2023 05:25:41 -0800 (PST) X-Google-Smtp-Source: AMrXdXs1p2p+mIVQp/oDjYgLhDPczf+65gcOCX8PXCXceqwKymzAnf1PHYy922gEWW2tZPDpjG5M X-Received: by 2002:a17:90a:acf:b0:226:ddf6:b7da with SMTP id r15-20020a17090a0acf00b00226ddf6b7damr19212201pje.41.1673443540815; Wed, 11 Jan 2023 05:25:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673443540; cv=none; d=google.com; s=arc-20160816; b=CO9GpfEVgToZ4pmFiaRLjYDKFwdMkMAYvKM6PMfxNMjvv2EQ6KLOB5uC5Q5U7kYOB0 kHfSuSqAxvIIgMajjZ/T73VimmwMEWBaee1txNRVp676WWbM4DuJOh3GDJ9+zg+inVza VA9gW5DK/Wm9Db3SKZBvMPhtRU+Ns+yGKSCD/lCrpyVUP3gBZtFKtkwmM/ZypVoNYL8O vU9vSRGhASpC3lSnKhmUPApI8IrullmaG1zf0PNlIJ7V7CZUEPfXMTs9RaJWKKAT2o5m mjkGwTarMzNrB27bY+Osf/5tzoxbIsheWkIqpIRMagPFyWtQHFFoj1DUxctvObLY3KtR cphQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ndhjY0FXUCMFSa32IGE1nAjZD8csY3IWkZ1zvnSeNd0=; b=G8NSQmBNY1E2n9DNrusRpewVhHPs3M6x3HV3nqBlKM+7xhBFIQSMy4ynivo37tUIO3 PCz0z3XSq42qg58Co/pNqu6NARAK5VqLWnBJaQxkxn7rqKq14HHg4KHg+a/6b5L3oMOW 4CooNzyLCyp/MFOGaJ/EcdDJc/GaqyMOckdnU4QmmlpxGj0quEUa9ly8PopfDfIVb9D4 U1bVnKIZP5TyNnOYUWGbeDl95Ndq2CfyMqc7gzbD5T0Gf8nfvaHpBUt9zlhWH8Nb1+iZ ihGPIFy09xEjSfk6LguKgC2lzEvxqP4a3ddJZRjGiFRmI6xiyE/3hl4YLVVUMZft10Qj JSQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Z2Oc/SBL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lb13-20020a17090b4a4d00b00228ea30ff29si1165321pjb.23.2023.01.11.05.25.27; Wed, 11 Jan 2023 05:25:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Z2Oc/SBL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233245AbjAKNXv (ORCPT + 99 others); Wed, 11 Jan 2023 08:23:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231453AbjAKNXt (ORCPT ); Wed, 11 Jan 2023 08:23:49 -0500 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35D1925EF for ; Wed, 11 Jan 2023 05:23:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673443428; x=1704979428; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nLb+D373AsFdYz+VydkUKZ3nmk+8SeRPO35WqLjs/ys=; b=Z2Oc/SBLrngL79WzeX4pwnFfH0hBLpwUtZbbtFRNYVlTMm4Mupx7SjdX P4kpUonlgMEIcjxyz51OZg3q0S3xJBp/i0XWObctZQYKAfJH6l3ftZLM6 cT6Q78p/h8CyX/lwfKat8NSl5f6t7IIV5ULMXJgEJ2au/2tOWA/luKelj tBoADmj5TOv2fmjrIB56OVENR1mcUA0JgQxk5owN0wmJCHFQrf3t6LNS6 LgvVeShJBtX3LO4VWtGkpqL7HJBHC5+o9/unPSbCLI1e5QfQS5GtzXqBc Iokn6j0r5rGiIIkpwSU0h677BOnkqcYGvwl5Z58mPC48ifhz9gbqtLVw+ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="387872486" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="387872486" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="687927402" X-IronPort-AV: E=Sophos;i="5.96,317,1665471600"; d="scan'208";a="687927402" Received: from bachaue1-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.37.250]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2023 05:23:42 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id B6579109D31; Wed, 11 Jan 2023 15:37:41 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv14 17/17] selftests/x86/lam: Add test cases for LAM vs thread creation Date: Wed, 11 Jan 2023 15:37:36 +0300 Message-Id: <20230111123736.20025-18-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> References: <20230111123736.20025-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754732734052674050?= X-GMAIL-MSGID: =?utf-8?q?1754732734052674050?= LAM enabling is only allowed when the process has single thread. LAM mode is inherited into child thread. Trying to enable LAM after spawning a thread has to fail. Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 92 +++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index a8c91829b616..eb0e46905bf9 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE #include #include #include @@ -12,6 +13,7 @@ #include #include #include +#include #include #include @@ -50,6 +52,8 @@ #define PAGE_SIZE (4 << 10) +#define STACK_SIZE 65536 + #define barrier() ({ \ __asm__ __volatile__("" : : : "memory"); \ }) @@ -731,6 +735,75 @@ static int handle_inheritance(struct testcases *test) return 0; } +static int thread_fn_get_lam(void *arg) +{ + return get_lam(); +} + +static int thread_fn_set_lam(void *arg) +{ + struct testcases *test = arg; + + return set_lam(test->lam); +} + +static int handle_thread(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = 0; + pid_t pid; + + /* Set LAM mode in parent process */ + if (!test->later) { + lam = test->lam; + if (set_lam(lam) != 0) + return 1; + } + + pid = clone(thread_fn_get_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, NULL); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + if (test->later) { + if (set_lam(test->lam) != 0) + return 1; + } + + return 0; +} + +static int handle_thread_enable(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = clone(thread_fn_set_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, test); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + return 0; +} static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -846,6 +919,25 @@ static struct testcases inheritance_cases[] = { .test_func = handle_inheritance, .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: LAM_U57, child thread should get LAM mode same as parent\n", + }, + { + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread_enable, + .msg = "THREAD: [NEGATIVE] Enable LAM in child.\n", + }, + { + .expected = 1, + .later = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: [NEGATIVE] Enable LAM in parent after thread created.\n", + }, { .expected = 0, .lam = LAM_U57_BITS,