From patchwork Mon Jan 23 22:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 47429 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1837423wrn; Mon, 23 Jan 2023 14:06:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXsfDb95PYo0O5/sun3IxCWj80HRmvXOsD20/BQpqo8Gq/GPTS52x8+A7Q7/uRwln5Y0ZmBa X-Received: by 2002:a17:906:8d0f:b0:870:b950:18d4 with SMTP id rv15-20020a1709068d0f00b00870b95018d4mr29913073ejc.5.1674511590313; Mon, 23 Jan 2023 14:06:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674511590; cv=none; d=google.com; s=arc-20160816; b=rrXky0jqyZi1wS4CS7rro8T62FPu66tQJ7UbYxUNbh7hGkVdQtEYmYnwl20JyK88ZS Luh3lOAaGbC2/6GuDceSjMuTHjWR4tzXk88Kt0NgZ8ahBH81Ir1Pp1F31W6fnn6UQJQx kLr3nfob96kQ5CY2c4lGOJAJp3kDy5T1pkTys+bbFBuNQUzN+WTMEWA4pY3EzLxmTtdK N/Wn6EETdZ1o7eGTMd7w3zGbQMxBa8teEFzIlCPGosZrGfHl3yuxEkitdqR8yCDxoTRf lXLqP4WEvEOc9ZHUglNDdrOzrHHIcI8NY9RoEyTGoIi6R8yPX0VuM4Q6nv6mD5mmhRQl h9Bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HdRYkzOlUG43YjkCu8PsfJrpwpJ/D/N4XwLO2lELylw=; b=t6ChbODLXsQWnXUrkbGwjocR9y/cv87xvjv7SeroVRjIyNUDidPIN3Zbb9Dfh0Z1BL 6Mm+khdVFmLfIsJ9m9QSqYMUFOIHBLggD46nAQv96f0ycECuHG2kdPeYmV9CflX1I9F6 bi86gK0HIy3pTfqamv6j3+KIgO84fgAVDVTTnh94oxRSCnrScL8U2cirPjpTL+kjqo+e hTg4KYmQies3KXN1ysSx12AjatRFeJgngtXjA3oBExzwHZXt5Hrxf8njEKNJA53KJHwP 591BvD1XbV9exAu3/h7zivJCq+4U376K4hqSNDEg7nRLiCwliYO1Lv9Sk2DN/iFTf/XE LbiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IdwpCIgh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fw31-20020a170907501f00b007ae1874c142si32853ejc.446.2023.01.23.14.06.07; Mon, 23 Jan 2023 14:06:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IdwpCIgh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232299AbjAWWFf (ORCPT + 99 others); Mon, 23 Jan 2023 17:05:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232401AbjAWWFa (ORCPT ); Mon, 23 Jan 2023 17:05:30 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCFA638035 for ; Mon, 23 Jan 2023 14:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674511523; x=1706047523; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=focBDCcui25Nna3zw6dN8ZwvWQ0ckdMqYvOG7S60BQo=; b=IdwpCIghC1+YrAHNrn+4T4mrad+Qvpxu5DkiFDxDkxo5TPGqqmNPSYlR bsZEVM9Q8vRKx0lNu7BxtETUTmgThzv7OOkkEvjaSGLcm8kYzdAt9P1M3 m4wWm5Dv3Tze7hn4xFTulzvE8m/TaH0PRX0vShBI0HMYQbj800l+60Wo7 pAlvBslQuvIcEHk4aRbivU+j45hqQYcj/Ul0HjmIgaro1oroLrcPYrGot mAH9Jnk39sRFrbUUdwce80hFysL7du3pXdIxH+PN0Q/zAzQ3i1JmgqIN3 3tkKNBDZfNLxLI2m9E38+abfbn3R4bwzuqlQnKYR0h2cBEbaKXGW8Qs87 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="327421936" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="327421936" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10599"; a="661878108" X-IronPort-AV: E=Sophos;i="5.97,240,1669104000"; d="scan'208";a="661878108" Received: from ssauty-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.46.171]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jan 2023 14:05:17 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6779A109478; Tue, 24 Jan 2023 01:05:03 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv15 07/17] x86/mm: Reduce untagged_addr() overhead for systems without LAM Date: Tue, 24 Jan 2023 01:04:50 +0300 Message-Id: <20230123220500.21077-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> References: <20230123220500.21077-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755852665271876269?= X-GMAIL-MSGID: =?utf-8?q?1755852665271876269?= Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/include/asm/uaccess.h | 41 +++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c44b56f7ffba..3f0c31044f02 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index fd9182951084..6450a2723bcd 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any branches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign = addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=r" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); - addr &= mask | sign; return addr; } #define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ - __addr = __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign = addr >> 63; + + mmap_assert_locked(mm); + addr &= (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr = (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr = __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr = (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) #else