From patchwork Fri Nov 11 21:58:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 19014 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp985446wru; Fri, 11 Nov 2022 13:59:55 -0800 (PST) X-Google-Smtp-Source: AA0mqf5AXIEHz1OXQL1K/fJK1cjbqrA9OymZLm+AiPnkRf7socvF29lSk5KZyjiUWs+g46HSgFTt X-Received: by 2002:a17:906:a401:b0:7ad:f398:78e6 with SMTP id l1-20020a170906a40100b007adf39878e6mr3427702ejz.341.1668203995633; Fri, 11 Nov 2022 13:59:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668203995; cv=none; d=google.com; s=arc-20160816; b=LaAXxVvb1B3iCVx/IqtbpV0IicpNkr2USlqTtV6t+Y3edYNLFC8YPkne5JVyEzlR9J SO9EG1FnnxVmUghMfPeg0SKpixZZNUoHviyI0WyEiT5Ry4/C/aUeJiqTA98WZ+PSZfD3 yfhrD+4NGM/2XZBMvKsExC3lty30CE0EivlN4B6hV6EY0KQVm2a+o3ai3j8zq8cEcV2i zR+KIZltLWDaZsuJaKgPQDmbTfl0E31O+4iXZZOAzpuHhEkb2vbHoBMzBoJi5S6B1N9e kkmU51n/0HnlZTGpkIET/efaAF0GvHoVOMikBCSZOKk4Jbmz3nLTkeehgWD8TfnDv0SA 6DrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:cc:subject:to:reply-to:sender:from :dkim-signature:dkim-signature:date; bh=qErqtVV5bJCFQVAImQPlp+Ioi47svNnc5+/U9yTHKlc=; b=UtFcoVX67YTKEBWbYfS8di43ViAOEIzNVfJg4mXnA3BIHVoMJGl3sL5N3WrVjRDQ8K TpW0dLTcFFdaa5Wu2KdaEU6nHO/zVBJG73nabfc16VM79WwP6LdOac+JaI81M18nRSRs khMLeNvNW6ff8HYB/1mNtBvjEAVQotwT3T5ELM2bxV9z9FQTdkXYbqEs+duIJKfPPNO5 LEJwYRfBmL4M4zhVizF78e0XGsUN13/7mT2xGBanIDRH1fo5Gqee5wNHxNoFnxcAEJaH 4tWtwiFuj6qkyIxE6cjC9Hsq0tYtFt91ydUAGWvOgQvXPkOGmoqPXFHdT1fCgjFzPrZ6 XENA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Nm1JMTGk; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=FBh7DAzf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x21-20020a170906299500b007ae545e330csi2644593eje.218.2022.11.11.13.59.31; Fri, 11 Nov 2022 13:59:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Nm1JMTGk; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=FBh7DAzf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234361AbiKKV7K (ORCPT + 99 others); Fri, 11 Nov 2022 16:59:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234157AbiKKV6q (ORCPT ); Fri, 11 Nov 2022 16:58:46 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90E76F5AA; Fri, 11 Nov 2022 13:58:33 -0800 (PST) Date: Fri, 11 Nov 2022 21:58:30 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1668203912; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qErqtVV5bJCFQVAImQPlp+Ioi47svNnc5+/U9yTHKlc=; b=Nm1JMTGkLNnCnhMhjc/FX7IUbkYTYZ+ouK9IpChUPuN4KabvRc2fE6JSzhASSGYe2Oh9FK 68nGBew6gr/+/UubWGfJNqvJQeGcQoC5oRkn6S0Ph3WtSRfd/crgIYfXhEY7QYhcZGPrGK GdbpeDu18N7cZ3JnBxEv5ABOvQ2Wn0OtKggXbsQ6JMvNyyKsoyih7DeR9voaDLW+EsGHuH WA4qXlVS43ZnullrVtfDVtbWDU4vOr4gKd9Djjv5Gji5La5pfoe+AJeBhnTaIRwQBzkJTF +Hd8iQYAabsET7NSBQFjmBWg2AED58Wg9gf6gXknt8sSnbIcReDG4gvMcSXdqQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1668203912; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qErqtVV5bJCFQVAImQPlp+Ioi47svNnc5+/U9yTHKlc=; b=FBh7DAzflbNb66Aeg3Sf/1Jfx9UFAeOR0VEf1eUS60gIQ4yRfsVI776LF54V2S7IduYeSj jED+I+SzK5l8c/DA== From: "tip-bot2 for Kirill A. Shutemov" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/mm] x86/uaccess: Provide untagged_addr() and remove tags before address check Cc: "Kirill A. Shutemov" , Dave Hansen , "Peter Zijlstra (Intel)" , Andy Lutomirski , Alexander Potapenko , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <166820391081.4906.11503973915855356452.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749238672774582861?= X-GMAIL-MSGID: =?utf-8?q?1749238672774582861?= The following commit has been merged into the x86/mm branch of tip: Commit-ID: 5744534bdae403270f86d934a6c5878da6a5a64f Gitweb: https://git.kernel.org/tip/5744534bdae403270f86d934a6c5878da6a5a64f Author: Kirill A. Shutemov AuthorDate: Wed, 09 Nov 2022 19:51:29 +03:00 Committer: Dave Hansen CommitterDate: Fri, 11 Nov 2022 13:28:07 -08:00 x86/uaccess: Provide untagged_addr() and remove tags before address check untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Signed-off-by: Dave Hansen Acked-by: Peter Zijlstra (Intel) Acked-by: Andy Lutomirski Tested-by: Alexander Potapenko Link: https://lore.kernel.org/all/20221109165140.9137-6-kirill.shutemov%40linux.intel.com --- arch/x86/include/asm/mmu.h | 3 ++- arch/x86/include/asm/mmu_context.h | 11 ++++++++- arch/x86/include/asm/uaccess.h | 42 ++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 ++- 4 files changed, 56 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 002889c..2fdb390 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 58ad18c..92cf8f5 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756e..0db6f54 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,30 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= (mm)->context.untag_mask | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(mm, ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(mm, __ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) +#else +#define untagged_addr(mm, addr) (addr) +#define untagged_ptr(mm, ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -41,7 +66,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(current->mm, addr), size)); \ }) #include @@ -127,7 +152,13 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -227,7 +258,12 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c21b734..d1e83ba 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM