From patchwork Fri Oct 13 11:18:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 152505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp1815212vqb; Fri, 13 Oct 2023 04:19:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHLcy/iugOd1Ix10KDBe5e5puAVjnmsWyFpwVTc+7Kxw513cFM7YVzV8xx60huuZEgzclxe X-Received: by 2002:a92:dacc:0:b0:34f:b824:5844 with SMTP id o12-20020a92dacc000000b0034fb8245844mr26196744ilq.3.1697195942019; Fri, 13 Oct 2023 04:19:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697195941; cv=none; d=google.com; s=arc-20160816; b=j9OW//mc+OLEudBWQmgfpG2Razqnr5qMBKg7sDnjdBgZ5fPfRVFGAeLnbnNm0trXbU nlSF+CQr7CKvZGebqwyGeuMV/ePbH9hM5c2VUnrcRuvvKb7xHt9ejc1AFgpB2P2/6iyO QozHD3JU1ob0hD35a+v6vIVLQVlVhitE+dCn7CM9LSCDkMdkCaHE93GpfoJI+bJT3gcL b06IkuVkvKbb9/j9JVVvjjvRaj6aU2G6NOX7GzLqsVnAlk4Gr5GWdV3WjTE/YUrrpLOZ LeFc7tBFOeZLyb51ywXWs7xrxA2Skg7cF7wUXHC52JFFUT0x4ZuaghHEXuqa/hHTAcE6 dgUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=tSqmBMVpyuwFzlCFIFtbk3M96+8ZydCk3RJzpjIhmRQ=; fh=rgFuo364X/bMod36etsfjmew4T/qgY2GNOeHFq2LY2k=; b=pyUsvs3dxA6Iu0s1V/o6N78m7n/TF6TUTSCpPtsxkkA5SgcOdGkAM3G1DEMTzQQgAo ukQfYL4N103DGahVWRq+UpOT7CUTtwx6QJ2V3+os11EWW/sM3TmynCqDZTrYRkSZlfn4 7O8EhERLyrhfUELvD90fO2iz93fPiIH+eG5E1WoWSxcca917Hqs8GY4lebeS/lyATDwo BI3LJkwwbPMf7iK103K+ASo7clVAM62n2wqssASeTJV0cj/3ulfC3FTKlYhIQRp6+Z4N Sc99vPl4eUnV3dbPHhNVOaY/H5r7kBjF1eyZvqdMFf22PPca374pJzTZBuoMTAFS0wd5 l9KQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UlT4DpKT; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id z11-20020a6552cb000000b0055793097dbesi4187822pgp.469.2023.10.13.04.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Oct 2023 04:19:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UlT4DpKT; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id D5FCF83A158F; Fri, 13 Oct 2023 04:18:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230510AbjJMLSf (ORCPT + 19 others); Fri, 13 Oct 2023 07:18:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230039AbjJMLSc (ORCPT ); Fri, 13 Oct 2023 07:18:32 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C1AEBE; Fri, 13 Oct 2023 04:18:30 -0700 (PDT) Date: Fri, 13 Oct 2023 11:18:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1697195908; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tSqmBMVpyuwFzlCFIFtbk3M96+8ZydCk3RJzpjIhmRQ=; b=UlT4DpKTT0P8ICgzCYHghW3Ngt8zmEkGUztIhtBvo6TqCE7khYEU3ouj1feDpbhBQR0h7S deF+x1wG4ambZYG9i7BnBCNpdDAI4YallANvQoCglMcpBInm90TNYm4sAtz0l/c3GTd1NB RONj4MOb8mxZz74jC/F1GXdzDHgwUiddQY8WkJXGTMfAMB/r/GM6IIObpCRol4g6+dMKG6 2u2yRMOdCoKQOWBmCi7rZpfe8cnFY6qltteZ+0S5LXw/GDhkp8jNBbjSU4WHOH1ydMhL8m sMwpqEwbdCwi47y78qBWKyC19IjcvLYzivA2SXryOKTbRE79ALLv25iPBhy3LQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1697195908; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tSqmBMVpyuwFzlCFIFtbk3M96+8ZydCk3RJzpjIhmRQ=; b=2G/GI/DGVTAkul9dLU2x1PsFA4lHuAygxkI+67KsyQfCT/fdQn2O4FKxJWehx+t5Zu+tMr U/wHCMRWsqeF8UAg== From: "tip-bot2 for Brian Gerst" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/entry] x86/entry/64: Convert SYSRET validation tests to C Cc: Brian Gerst , Ingo Molnar , Andy Lutomirski , Borislav Petkov , Denys Vlasenko , "H. Peter Anvin" , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Josh Poimboeuf , Uros Bizjak , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20231011224351.130935-2-brgerst@gmail.com> References: <20231011224351.130935-2-brgerst@gmail.com> MIME-Version: 1.0 Message-ID: <169719590773.3135.13248965487116739403.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 13 Oct 2023 04:18:59 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779500858333767251 X-GMAIL-MSGID: 1779638932139931236 The following commit has been merged into the x86/entry branch of tip: Commit-ID: ca282b486a570a0bfda5c1a4595ace7fa14243bf Gitweb: https://git.kernel.org/tip/ca282b486a570a0bfda5c1a4595ace7fa14243bf Author: Brian Gerst AuthorDate: Wed, 11 Oct 2023 18:43:49 -04:00 Committer: Ingo Molnar CommitterDate: Fri, 13 Oct 2023 13:05:28 +02:00 x86/entry/64: Convert SYSRET validation tests to C No change in functionality expected. Signed-off-by: Brian Gerst Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Josh Poimboeuf Cc: Uros Bizjak Link: https://lore.kernel.org/r/20231011224351.130935-2-brgerst@gmail.com --- arch/x86/entry/common.c | 43 ++++++++++++++++++++++++++- arch/x86/entry/entry_64.S | 53 +-------------------------------- arch/x86/include/asm/syscall.h | 2 +- 3 files changed, 45 insertions(+), 53 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 0551bcb..9021465 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -71,7 +71,8 @@ static __always_inline bool do_syscall_x32(struct pt_regs *regs, int nr) return false; } -__visible noinstr void do_syscall_64(struct pt_regs *regs, int nr) +/* Returns true to return using SYSRET, or false to use IRET */ +__visible noinstr bool do_syscall_64(struct pt_regs *regs, int nr) { add_random_kstack_offset(); nr = syscall_enter_from_user_mode(regs, nr); @@ -85,6 +86,46 @@ __visible noinstr void do_syscall_64(struct pt_regs *regs, int nr) instrumentation_end(); syscall_exit_to_user_mode(regs); + + /* + * Check that the register state is valid for using SYSRET to exit + * to userspace. Otherwise use the slower but fully capable IRET + * exit path. + */ + + /* XEN PV guests always use the IRET path */ + if (cpu_feature_enabled(X86_FEATURE_XENPV)) + return false; + + /* SYSRET requires RCX == RIP and R11 == EFLAGS */ + if (unlikely(regs->cx != regs->ip || regs->r11 != regs->flags)) + return false; + + /* CS and SS must match the values set in MSR_STAR */ + if (unlikely(regs->cs != __USER_CS || regs->ss != __USER_DS)) + return false; + + /* + * On Intel CPUs, SYSRET with non-canonical RCX/RIP will #GP + * in kernel space. This essentially lets the user take over + * the kernel, since userspace controls RSP. + * + * Change top bits to match the most significant bit (47th or 56th bit + * depending on paging mode) in the address. + */ + if (unlikely(!__is_canonical_address(regs->ip, __VIRTUAL_MASK_SHIFT + 1))) + return false; + + /* + * SYSRET cannot restore RF. It can restore TF, but unlike IRET, + * restoring TF results in a trap from userspace immediately after + * SYSRET. + */ + if (unlikely(regs->flags & (X86_EFLAGS_RF | X86_EFLAGS_TF))) + return false; + + /* Use SYSRET to exit to userspace */ + return true; } #endif diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 7574639..1730640 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -126,57 +126,8 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL) * In the Xen PV case we must use iret anyway. */ - ALTERNATIVE "", "jmp swapgs_restore_regs_and_return_to_usermode", \ - X86_FEATURE_XENPV - - movq RCX(%rsp), %rcx - movq RIP(%rsp), %r11 - - cmpq %rcx, %r11 /* SYSRET requires RCX == RIP */ - jne swapgs_restore_regs_and_return_to_usermode - - /* - * On Intel CPUs, SYSRET with non-canonical RCX/RIP will #GP - * in kernel space. This essentially lets the user take over - * the kernel, since userspace controls RSP. - * - * If width of "canonical tail" ever becomes variable, this will need - * to be updated to remain correct on both old and new CPUs. - * - * Change top bits to match most significant bit (47th or 56th bit - * depending on paging mode) in the address. - */ -#ifdef CONFIG_X86_5LEVEL - ALTERNATIVE "shl $(64 - 48), %rcx; sar $(64 - 48), %rcx", \ - "shl $(64 - 57), %rcx; sar $(64 - 57), %rcx", X86_FEATURE_LA57 -#else - shl $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx - sar $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx -#endif - - /* If this changed %rcx, it was not canonical */ - cmpq %rcx, %r11 - jne swapgs_restore_regs_and_return_to_usermode - - cmpq $__USER_CS, CS(%rsp) /* CS must match SYSRET */ - jne swapgs_restore_regs_and_return_to_usermode - - movq R11(%rsp), %r11 - cmpq %r11, EFLAGS(%rsp) /* R11 == RFLAGS */ - jne swapgs_restore_regs_and_return_to_usermode - - /* - * SYSRET cannot restore RF. It can restore TF, but unlike IRET, - * restoring TF results in a trap from userspace immediately after - * SYSRET. - */ - testq $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11 - jnz swapgs_restore_regs_and_return_to_usermode - - /* nothing to check for RSP */ - - cmpq $__USER_DS, SS(%rsp) /* SS must match SYSRET */ - jne swapgs_restore_regs_and_return_to_usermode + ALTERNATIVE "testb %al, %al; jz swapgs_restore_regs_and_return_to_usermode", \ + "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV /* * We win! This label is here just for ease of understanding diff --git a/arch/x86/include/asm/syscall.h b/arch/x86/include/asm/syscall.h index c7e25c9..f44e2f9 100644 --- a/arch/x86/include/asm/syscall.h +++ b/arch/x86/include/asm/syscall.h @@ -126,7 +126,7 @@ static inline int syscall_get_arch(struct task_struct *task) ? AUDIT_ARCH_I386 : AUDIT_ARCH_X86_64; } -void do_syscall_64(struct pt_regs *regs, int nr); +bool do_syscall_64(struct pt_regs *regs, int nr); #endif /* CONFIG_X86_32 */