From patchwork Wed Sep 20 15:04:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 142575 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp4395221vqi; Wed, 20 Sep 2023 13:07:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH9sp6RjLxX4POMtEjQT/k/22bh1STC5Z7n927r0Ah/TwYWBQdKcyjHtVmr4hWjEWVlyJQr X-Received: by 2002:a05:6a20:7d96:b0:130:7803:5843 with SMTP id v22-20020a056a207d9600b0013078035843mr4058665pzj.4.1695240451975; Wed, 20 Sep 2023 13:07:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695240451; cv=none; d=google.com; s=arc-20160816; b=NkDtkZuDLQ6Z7gnSLkPTvHdCrulYFd18V2Um5j6fzieN24jMFbmqdl80WVGrgRZ2Uw YrNCn9HeW8Thr/Yj0qNnBNWfkkTKMqzqW89Dd4RFTvsqVSj9SH9o0GG3hXMWoQjWoKLd xxP3z5kGTSagRzd+RIRH/17fQ98aTP6ldbH0LIjJQx0xhIVtdmIRvwlytLgZUSXWLDvr RlROF+DDl+e/U8DEDrQ7GPYknIZh9hpscjf++jTyFx78xIMASgKwrFOXV059thdbqBGa upUDprudD4NjHM37rNhp4GR9Hxek/svpLSOw6MCtxL6ReCy0CF9xTkIc4WRnyL+fjDlH OAfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=FVknj5SviIXtWZwFEQq1ySMJBkaRNL9MG0K5kDDN86g=; fh=sYoYae35IWAk586EzWcbtIwkBWfX19XFz0RR9yifTtM=; b=CTorVPoIPdMgh7LbqSOcWD2qQCoH1ngN3Xii5A+Z8e8nepCFByhSByoYrXLiC08mlM nS2vSqjKHHyvFt0JBZWb0dED3iKylg7yD5rroq7qdXcN5gtTJa1EqAuY9S0qbkYKNNQO fYfwijZhp1jo9JTfa6Yyl4BM40gnOzb4BX1GS8UwprX06VtSNXbGDyvin5dy1W9Oxhon 8GC2+WKhEevb2Zokwu0RzaExTdDwvzIWpMcpSSPewrkZXs+/shubiyG6QHZsi6YqCzj6 /IANaiIfLLZrXzNyJkc07BwdSE3cdOUQLUB9c60k/xI57cf1Q8tzn2na9AA4NoCMv3V0 Z0DA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XwKTAQ2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id m16-20020a63fd50000000b00563fac86c55si9820176pgj.134.2023.09.20.13.07.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 13:07:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XwKTAQ2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 01B7880D0C96; Wed, 20 Sep 2023 08:05:06 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236309AbjITPEz (ORCPT + 27 others); Wed, 20 Sep 2023 11:04:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235293AbjITPEy (ORCPT ); Wed, 20 Sep 2023 11:04:54 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CFC79F for ; Wed, 20 Sep 2023 08:04:48 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d81a76a11eeso6495624276.3 for ; Wed, 20 Sep 2023 08:04:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695222287; x=1695827087; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=FVknj5SviIXtWZwFEQq1ySMJBkaRNL9MG0K5kDDN86g=; b=XwKTAQ2CBj6YJRlnp5V8FAf+nPuqrZt8Gcu+37SYis8kJIfzFngC42T362jsDjlpGz +++q0tVHYjd0ZQv1/cO0Ta/vN5iNGPviuEmYVitViByj4g73cIakRuDDtZEXfL9F59SG P/LkVkZigELD1J6R3vhRUiV/szeAFrJFlqox9Nrok/Dvv7UAXe21A8US9EEwppwVgKVa RiRyN8oIT9u8b6jI/K/CCMoShtpc8xYQuM24VlhumR+RhSqy9kYTHRYRTnNSEi3YuFSm OqJ8HxpanTLtwVi2kX1aqyJ7XPFWrUnuldNA0NdrYHeOUF/ygc/jVyHU7a2156lxO0aB wRIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695222287; x=1695827087; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FVknj5SviIXtWZwFEQq1ySMJBkaRNL9MG0K5kDDN86g=; b=D/4XOZZQBqlW1g/6F4xoDJEJm3iXnnFSemmFiPmGOgUYqa1exhOpP78xaT4Ruk5ke9 FQCz6VDVc5Jxc786zKV7ZJgbiz8xi+DrNe3FLd8NwfXZD+srm8/4WiJTkJNM2dUd49Y7 3+5o9TFfJAjacx7DJpEc8PnwEeR07dJwD70LynSOEnHkJTCUr35ROoZBqBaTlL5EMOra 7BsqBFfA/6GzsxKsIL4So7USoAqk83uyMx1LUwm+MiplVzv5X+YdM25/z6YYQjN+P84w SeasOGXSGiKkQ0muc9x1ErGZUqqlwu/Mn9Kw8+323Iye3zQ7BcOdz5fnWLShfdbnPDIn Mmuw== X-Gm-Message-State: AOJu0YxDE3S7PUBK3aKYMtijSBMqi5terInOp16doyi+cS11xCE2VUAT 3JyYksKdZDqoQPlDe5sDcWl4ig3HFzji1g== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a25:d692:0:b0:d7e:add7:4de6 with SMTP id n140-20020a25d692000000b00d7eadd74de6mr42351ybg.4.1695222287531; Wed, 20 Sep 2023 08:04:47 -0700 (PDT) Date: Wed, 20 Sep 2023 15:04:43 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230920150443.1789000-1-jackmanb@google.com> Subject: [PATCH v2] x86/entry: Avoid redundant CR3 write on paranoid returns From: Brendan Jackman To: luto@kernel.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, linux-kernel@vger.kernel.org, laijs@linux.alibaba.com, yosryahmed@google.com, reijiw@google.com, oweisse@google.com, peterz@infradead.org, Brendan Jackman X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 20 Sep 2023 08:05:06 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777588452151067765 X-GMAIL-MSGID: 1777588452151067765 From: Lai Jiangshan This path gets used called from: 1. #NMI return. 2. paranoid_exit (i.e. #MCE, #VC, #DB and #DF return) Contrary to the implication in commit 21e94459110252 ("x86/mm: Optimize RESTORE_CR3"), we never modify CR3 in any of these exceptions, except for switching from user to kernel pagetables under PTI. That means that most of the time when returning from an exception that interrupted the kernel no CR3 restore is necessary. Writing CR3 is expensive on some machines, so this commit avoids redundant writes. I said "most of the time" because we might have interrupted the kernel entry before the user->kernel CR3 switch or the exit after the kernel->user switch. In the former case skipping the restore might actually be be fine, but definitely not the latter. So we do still need to check the saved CR3 and restore it if it's a user CR3. To reflect the new behaviour RESTORE_CR3 is given a longer name, and a comment that was describing its behaviour at the call site is removed. We can also simplify the code around the SET_NOFLUSH_BIT invocation as we no longer need to branch to it from above. Signed-off-by: Lai Jiangshan [Rewrote commit message; responded to review comments] Signed-off-by: Brendan Jackman Acked-by: Peter Zijlstra (Intel) --- Notes: V1: https://lore.kernel.org/lkml/20230817121513.1382800-1-jackmanb@google.com/ v1->v2: Rewrote some comments, added a proper commit message, cleaned up the code per tglx's suggestion. I've kept Lai as the Author. If you prefer for the blame to record the last person that touched it then that's also fine though, I can credit Lai as Co-developed-by. arch/x86/entry/calling.h | 25 +++++++++---------------- arch/x86/entry/entry_64.S | 7 +++---- 2 files changed, 12 insertions(+), 20 deletions(-) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index f6907627172b..84b1e32c27a1 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -233,17 +233,18 @@ For 32-bit we have the following conventions - kernel is built with .Ldone_\@: .endm -.macro RESTORE_CR3 scratch_reg:req save_reg:req +.macro RESTORE_CR3_IF_USER scratch_reg:req save_reg:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI - ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID - /* - * KERNEL pages can always resume with NOFLUSH as we do - * explicit flushes. + * If CR3 contained the kernel page tables at the paranoid exception + * entry, then there is nothing to restore as CR3 is not modified while + * handling the exception. */ bt $PTI_USER_PGTABLE_BIT, \save_reg - jnc .Lnoflush_\@ + jnc .Lend_\@ + + ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID /* * Check if there's a pending flush for the user ASID we're @@ -251,20 +252,12 @@ For 32-bit we have the following conventions - kernel is built with */ movq \save_reg, \scratch_reg andq $(0x7FF), \scratch_reg - bt \scratch_reg, THIS_CPU_user_pcid_flush_mask - jnc .Lnoflush_\@ - btr \scratch_reg, THIS_CPU_user_pcid_flush_mask - jmp .Lwrcr3_\@ + jc .Lwrcr3_\@ -.Lnoflush_\@: SET_NOFLUSH_BIT \save_reg .Lwrcr3_\@: - /* - * The CR3 write could be avoided when not changing its value, - * but would require a CR3 read *and* a scratch register. - */ movq \save_reg, %cr3 .Lend_\@: .endm @@ -279,7 +272,7 @@ For 32-bit we have the following conventions - kernel is built with .endm .macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req .endm -.macro RESTORE_CR3 scratch_reg:req save_reg:req +.macro RESTORE_CR3_IF_USER scratch_reg:req save_reg:req .endm #endif diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 43606de22511..ff73767b5d1f 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1019,14 +1019,14 @@ SYM_CODE_START_LOCAL(paranoid_exit) IBRS_EXIT save_reg=%r15 /* - * The order of operations is important. RESTORE_CR3 requires + * The order of operations is important. RESTORE_CR3_IF_USER requires * kernel GSBASE. * * NB to anyone to try to optimize this code: this code does * not execute at all for exceptions from user mode. Those * exceptions go through error_return instead. */ - RESTORE_CR3 scratch_reg=%rax save_reg=%r14 + RESTORE_CR3_IF_USER scratch_reg=%rax save_reg=%r14 /* Handle the three GSBASE cases */ ALTERNATIVE "jmp .Lparanoid_exit_checkgs", "", X86_FEATURE_FSGSBASE @@ -1457,8 +1457,7 @@ end_repeat_nmi: /* Always restore stashed SPEC_CTRL value (see paranoid_entry) */ IBRS_EXIT save_reg=%r15 - /* Always restore stashed CR3 value (see paranoid_entry) */ - RESTORE_CR3 scratch_reg=%r15 save_reg=%r14 + RESTORE_CR3_IF_USER scratch_reg=%r15 save_reg=%r14 /* * The above invocation of paranoid_entry stored the GSBASE