From patchwork Mon Mar 20 16:39:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 72311 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1331458wrt; Mon, 20 Mar 2023 10:12:13 -0700 (PDT) X-Google-Smtp-Source: AK7set8XmNPhVWPWQvvSBLVyd1pAweCwgITilhduoqWBkKq6uVY5uF5XJyEIBU596ZAe5b6Un8wK X-Received: by 2002:a05:6a20:b913:b0:d6:f3dd:5a88 with SMTP id fe19-20020a056a20b91300b000d6f3dd5a88mr15914130pzb.5.1679332333425; Mon, 20 Mar 2023 10:12:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679332333; cv=none; d=google.com; s=arc-20160816; b=QguZ6nWE31cVUXAga6xCVNCEF+31xavnTU3Op6yXTtIOUhcQJ/T8Q/zkdvTsSV60/L CFOAbBuvJyS9CWKpGNQR5GEl2HWEIj6U2blMkZ57I0jlpgcVpdlkTWd6IyEd+QxlTsho I07V3rJVM+/z9zhucXYI1hacSoqrHuJTsCaPcSzKUn7moWS7a6Rln1nBw6s8rATYo89d l1gdQILpD+Q+qmbGqtRofJOpPphRlcs9ccUjtITJ05EBedklsiU5Ey02EiXMIQlzQijl WcbpkK+mB72GDIUg8xneEFgt+yH/quhVwSQq7WajV3LlzIJBhyRXNETScVnmZ2rxEJH3 3Taw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:cc:subject:to:reply-to:sender:from :dkim-signature:dkim-signature:date; bh=k7Csc63oNaUrUCRjkydySx1BB4b/9hMNseImrAKDvjs=; b=sGvyIa5PNvFNpZF49hOpKh1kGvqbq0z1RGtzyWQL0QeTppWsO9BMDIuLhE2NvYDcqP w3R5PT+gOgMtEG8ezSYZjIsKKaxynAyXF8BmYTCLZw+qF/v++KYOdQeH6+t1qbVhFwhI zAbRJd7O+7TEYFR0H8l319nu5uXOjv0YN5ebhLzWS2eAcwVFcZCUch8KQwpX6NdETbZH vu9qbmaRKet0QU97idN8auMNe5TIX7WXCEXzTLl0xqHAtNbyVsmulNnb7oZClaFXxUpc /8dJWdaAvGSelnizJfQjLtHguIp7C82l/5n7IgOtRpnAoLgh0w26KlmqY906pgfH0T4C Tqmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MADAdHv2; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="WA/bh7sq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m10-20020a638c0a000000b0050f93a7aa76si1187722pgd.516.2023.03.20.10.12.00; Mon, 20 Mar 2023 10:12:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MADAdHv2; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="WA/bh7sq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232482AbjCTQsT (ORCPT + 99 others); Mon, 20 Mar 2023 12:48:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231683AbjCTQri (ORCPT ); Mon, 20 Mar 2023 12:47:38 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8B3E35242; Mon, 20 Mar 2023 09:41:07 -0700 (PDT) Date: Mon, 20 Mar 2023 16:39:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1679330367; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=k7Csc63oNaUrUCRjkydySx1BB4b/9hMNseImrAKDvjs=; b=MADAdHv2kjvW3kPVJo6RsiG9fMaA14JcLDI4aPcQXUL9wLHjOVmNs7ZhVkwj41rfkbnE0g ShMmIBd47rQ2KeqRpIzshZO2IxNdmZsV0B/zZ+Tc1b9NrN6IoDkCsLjrQbdiOegrzUp40V UBYX/dhdKpSgIochwSDfRvNNrrD1Zks4pOMAfhPRq6c8YBA25H6Y99pyhWBGISEaVK+B3K fT4vk21V+vLPplTygkdo+yamrAZSg4JxOZfr9BQPjSSpj/1ROLQPGtw8uSOZ9adyMF2jIV 4qwHrYcdxR3i1JI0JpFXwp+svTq7KmXPzs2gURxStaJDLjw19qOaPyT1z3PrKg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1679330367; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=k7Csc63oNaUrUCRjkydySx1BB4b/9hMNseImrAKDvjs=; b=WA/bh7sqhxkP2Ayu17U6mZJt9/VR0ANWmms9LCgNqarMrzXapLHQreNG3L2LqLei/SNW6x hvzrTIYPMKBKpVCA== From: "tip-bot2 for Rick Edgecombe" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/shstk] x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY Cc: "Yu-cheng Yu" , Rick Edgecombe , Dave Hansen , "Borislav Petkov (AMD)" , Kees Cook , "Mike Rapoport (IBM)" , Pengfei Xu , John Allen , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <167933036724.5837.5697853056919150153.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760907580515366358?= X-GMAIL-MSGID: =?utf-8?q?1760907580515366358?= The following commit has been merged into the x86/shstk branch of tip: Commit-ID: c169f96ab134479018710af1e2291022eff5af99 Gitweb: https://git.kernel.org/tip/c169f96ab134479018710af1e2291022eff5af99 Author: Rick Edgecombe AuthorDate: Sat, 18 Mar 2023 17:15:10 -07:00 Committer: Dave Hansen CommitterDate: Mon, 20 Mar 2023 09:01:09 -07:00 x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,SavedDirty=1. When a PTE goes from Write=1,Dirty=1 to Write=0,SavedDirty=1, it could become a transient shadow stack PTE in two cases: 1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary. 2. When _PAGE_DIRTY is replaced with _PAGE_SAVED_DIRTY non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg. In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature. Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Signed-off-by: Dave Hansen Reviewed-by: Borislav Petkov (AMD) Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook Link: https://lore.kernel.org/all/20230319001535.23210-16-rick.p.edgecombe%40intel.com --- arch/x86/include/asm/pgtable.h | 35 +++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 7360783..349fcab 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1192,6 +1192,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pte_t old_pte, new_pte; + + old_pte = READ_ONCE(*ptep); + do { + new_pte = pte_wrprotect(old_pte); + } while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte)); + + return; + } +#endif clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); } @@ -1244,6 +1261,24 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pmd_t old_pmd, new_pmd; + + old_pmd = READ_ONCE(*pmdp); + do { + new_pmd = pmd_wrprotect(old_pmd); + } while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + + return; + } +#endif + clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); }