From patchwork Sat Feb 18 21:14:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 59010 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555607wrn; Sat, 18 Feb 2023 13:21:52 -0800 (PST) X-Google-Smtp-Source: AK7set8nck6jlWes5BrJQndwecsCWHS8dP2XweZQ5FmIe/Zji3rRnexk2sTkaXi9o4DULzs0Z/a4 X-Received: by 2002:a17:90b:1b03:b0:234:bf0:86be with SMTP id nu3-20020a17090b1b0300b002340bf086bemr1132638pjb.27.1676755311962; Sat, 18 Feb 2023 13:21:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676755311; cv=none; d=google.com; s=arc-20160816; b=qYQ09mHjTk/Nm5vZuGfNLmtkXEANkRSKIK5zAz4iTP6DuI7q75qyftxBtRwbAyPoRs xvzhlgsv9nIwcDDJIjraQ5RPaWeLPPva/9WPp3Wd6mqQbRgkdkQJjdpj1sdwr3LsK0jY Aygun6Nn7XyOalWRSjhXjgyo017Qxl6FhUs06+ZRwtl5eDUUaPo42ytS8QY28C0R/8Rv ErW2T0CPZSh4K6gXr+pz6QnCMfsBWFCLQ+/QMZkQ7cR62022k2uJv7boP3y1yg5Dpv7l Dj6/MNtDxeQIUid8YRrqi6XvV7znbesz/PjzM0upUkG7vdXhOJvOFTwPF/G22jUsI/Df vbtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=XSFlVqZd719tgVIR5uAh62h+RJ0ve8ohUB9lJ1MzRl4=; b=Rf1LlRMCqXuMsNAEfEOrFoUx1Pdx7FEETLyhlxplDAVfJ1MBrt894g9Bvr9OJCDw/Q J9WRrNzSapWaaVA0MlUHUn7O3K1R+ZnZpkqpS+oUJE293WvCZPA+rNNYz7sGjnf/uiBg DKIVKgDXEbtk005eAPp0yy1i+jcwqnyIlOMTrCp7ef2JBYRAVuGxPCsmCa5cW9Z/8A4Y J14JDgNAwSXOtxiGOmpoFEN7SjJ4z/ksdjX8QxvNJTcZOaoK7ehFIjR+YYQKBbqaVm7R Qk1ImROhcK/Cn4vLLiSf8te6S+wQXsCG9d8wOHxuFOtSI3ng9+HeY/YWNKFYoaftXKp/ dNJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=E1UiPFrx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l11-20020a65680b000000b004da3b29e1cdsi9569869pgt.116.2023.02.18.13.21.39; Sat, 18 Feb 2023 13:21:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=E1UiPFrx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230255AbjBRVVZ (ORCPT + 99 others); Sat, 18 Feb 2023 16:21:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbjBRVUq (ORCPT ); Sat, 18 Feb 2023 16:20:46 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 984B018B30; Sat, 18 Feb 2023 13:18:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676755096; x=1708291096; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QFt1nd3j6Yjl0jrlYnYinrJNdu94hS5cjLAeUme4VXo=; b=E1UiPFrxhJldmoH6JhfQco+WGw96hbZ90AGvKng9xkYHA+B1+CK+O4PE SLXdb+CvhtHi9E/oLSifMuNYdztRIpqj91U6UkFl0brGFX9JU94ceR5gN X94Br6/eYK3XJB8YDp3H25jqT5wMxRuuixNRE1Gu2vocSWo+Xp5geaNzy Makcg+48lzd2vXp/+pyh/uMR6Pw8RSJHCvzkEfvKtu6FldtJKIk7yiueD fMM9Vhrp7ajaeuiA+D5qBLVB5Y+ln4R9NZjqMKRxSmSFf0yKAzdNA7GT6 SYWWScGGe3O/pelOeFkJRakuSvnDdHhsoS5Gvg26RRr0mb+pmd8Zdpg1z g==; X-IronPort-AV: E=McAfee;i="6500,9779,10625"; a="418427649" X-IronPort-AV: E=Sophos;i="5.97,309,1669104000"; d="scan'208";a="418427649" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2023 13:16:17 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10625"; a="664241695" X-IronPort-AV: E=Sophos;i="5.97,309,1669104000"; d="scan'208";a="664241695" Received: from adityava-mobl1.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.209.80.223]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2023 13:16:17 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v6 26/41] mm: Warn on shadow stack memory in wrong vma Date: Sat, 18 Feb 2023 13:14:18 -0800 Message-Id: <20230218211433.26859-27-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230218211433.26859-1-rick.p.edgecombe@intel.com> References: <20230218211433.26859-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758205378143416436?= X-GMAIL-MSGID: =?utf-8?q?1758205378143416436?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a pte is shadow stack in core mm code, add two arch breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack specific code to be kept in arch/x86. Tested-by: Pengfei Xu Tested-by: John Allen Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v6: - Add arch breakout to remove shstk from core MM code. v5: - Fix typo in commit log v3: - New patch --- arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 12 ++++++++++++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 20d0df494269..f3dc16fc4389 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1687,6 +1687,12 @@ static inline bool arch_has_hw_pte_young(void) return true; } +#define arch_check_zapped_pte arch_check_zapped_pte +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); + +#define arch_check_zapped_pmd arch_check_zapped_pmd +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 98856bcc8102..afab0bc7862b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -906,3 +906,15 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } + +void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(pte)); +} + +void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) +{ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(pmd)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1159b25b0542..22787c86c8f2 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -291,6 +291,20 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_check_zapped_pte +static inline void arch_check_zapped_pte(struct vm_area_struct *vma, + pte_t pte) +{ +} +#endif + +#ifndef arch_check_zapped_pmd +static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, + pmd_t pmd) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a216129e6a7c..842925f7fa9e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1696,6 +1696,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index 6ad031d5cfb0..29e8f043b603 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1377,6 +1377,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);