From patchwork Thu Jan 19 21:22:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45988 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555902wrn; Thu, 19 Jan 2023 13:35:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXta87p8SoCOJVxEhHTN76lUlwuTwn1yvXvH95KgbmrbpndbnioFWy0B61lrfS+QFNQsswMF X-Received: by 2002:a62:870f:0:b0:582:bb80:58d7 with SMTP id i15-20020a62870f000000b00582bb8058d7mr35287030pfe.26.1674164130562; Thu, 19 Jan 2023 13:35:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164130; cv=none; d=google.com; s=arc-20160816; b=ar+dIMdSSGWDl4nEkar0Y6gNFFrvYJmvogGjORsIw8y64JpEntVFIeJxwdmPFd8Ylh rYFEnjJG3OVC19pV8UTylT/AwdO/0nHBinemPAVMdnA0NLT7QryRHexuNx8YElIjyBvG lyHwORQ10H+DfEo0JHuuafJRW5gZWcQTehWRQtDqWyBycpHoQU0ukCDJtItLXCPBLZTg 2JPabIMbMZIGlpMus4ISbM63oWgUPKghBAqZTunknQALpn0gmXhNbGPDURO9nZFgUmes EiCcGi3LcK8fP4S1DYogs3UmKaXlHmmuwBO9Orat4BVh36Dw+SMoJBcq9JQWc2XcPs5U kWSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=raQIy6Fe766NUbCocQ4XfRzcFOdeBSgY+sYH+kVPZqU=; b=SMWixnFrPMcq+jk2d5jkVx2ECLr+NnAxylaZvBIEUY7A+nEG6i1GCgGCJRBkQDUMr/ 6CvGWWlDcq4RUepCvmCU+8t4OImZLurkYVGZFEQd7aBvAND3DNzG9t+SD/uTn4mDhaGJ ZYMU+uLqHJqNfGSHHkLUZcvoNrKW3o9opF3sjLA1eSOmJXzl9XoHi+Yzj/npPWzaKEDE F0Hlc+n5OM5xcxmTOiYJd8+AuX8O/fDgNH4dvKXUayNfSw2d7wVDtXxlzsspVbuE0dRt o9Cfsm+DGTjbhVeAEU0o3PDegHyjwdAsm4yQSYdmat43By4GRTXQ3k1gnW7QccK1Yj+S ofcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=duB0BEml; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d19-20020a056a00199300b00575757288e4si12927932pfl.90.2023.01.19.13.35.18; Thu, 19 Jan 2023 13:35:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=duB0BEml; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230467AbjASVd7 (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229685AbjASVb1 (ORCPT ); Thu, 19 Jan 2023 16:31:27 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 616ACA8393; Thu, 19 Jan 2023 13:25:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163552; x=1705699552; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0xRZR4lTYh6NOOM79Ggd51EdLXNXPAUjGASZ+kAeGKM=; b=duB0BEmlqw+ygPNkvMxSD1o+dnTZME0CnvbBTESXHyjxDrbbKpuGv4am lBKW4UPNwndUCTNGsrlGsEQNmKmAOwOJIWBP8RrPD+brqhcs1zc+tDurJ BuQkU67vPKPOeV1jIHSkWjJy+cLdLcEmc55g2vaVuk0hqnZpdxVwMAl6k 3xMdThUepL4voIWiXJjeSOb/4v/hSnV3lmRuk53yS4i5kK+bCd9uMIJ3f gZofpixT+VuWu3ryIhCgEJ0lDk6lS+E5zT5XTqPNX7V73vP6t9ANk9Tp/ Ctk17W3o+3KqGbzmxo0FF6aQS4NWDTbTCz8QoYIXSGz/chn+yoWYCfNLa A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119409" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119409" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:42 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139038" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139038" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:41 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW Date: Thu, 19 Jan 2023 13:22:49 -0800 Message-Id: <20230119212317.8324-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488327225099092?= X-GMAIL-MSGID: =?utf-8?q?1755488327225099092?= From: Yu-cheng Yu The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this. pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one. However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Do this by using the pte_mkdirty() helper. Since pte_mkdirty() also sets the soft dirty bit, extract a helper that optionally doesn't set _PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move _PAGE_DIRTY to _PAGE_COW can live in one place. Apply the same changes to pmd_modify(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix pte_modify() again, to not lose _PAGE_DIRTY, but still not set _PAGE_SOFT_DIRTY as was fixed in v4. v4: - Fix an issue in soft-dirty test, where pte_modify() would detect _PAGE_COW in pte_dirty() and set the soft dirty bit in pte_mkdirty(). v2: - Update commit log with text and suggestions from (Dave Hansen) - Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen) arch/x86/include/asm/pgtable.h | 64 +++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 6d2f612c04b5..7942eff2af50 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -392,9 +392,19 @@ static inline pte_t pte_mkexec(pte_t pte) return pte_clear_flags(pte, _PAGE_NX); } +static inline pte_t __pte_mkdirty(pte_t pte, bool soft) +{ + pteval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pte_set_flags(pte, dirty); +} + static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pte_mkdirty(pte, true); } static inline pte_t pte_mkyoung(pte_t pte) @@ -503,9 +513,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return pmd_clear_flags(pmd, _PAGE_RW); } +static inline pmd_t __pmd_mkdirty(pmd_t pmd, bool soft) +{ + pmdval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pmd_set_flags(pmd, dirty); +} + static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pmd_mkdirty(pmd, true); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -715,26 +735,54 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { + pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY; pteval_t val = pte_val(pte), oldval = val; + pte_t pte_result; /* * Chop off the NX bit (if present), and add the NX portion of * the newprot (if present): */ - val &= _PAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; + val &= _page_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); - return __pte(val); + + pte_result = __pte(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pte_mkdirty() will do this in + * the case of shadow stack. + */ + if (pte_dirty(pte)) + pte_result = __pte_mkdirty(pte_result, false); + + return pte_result; } static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { + pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY; pmdval_t val = pmd_val(pmd), oldval = val; + pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val &= _hpage_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); + + pmd_result = __pmd(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pmd_mkdirty() will do this in + * the case of shadow stack. + */ + if (pmd_dirty(pmd)) + pmd_result = __pmd_mkdirty(pmd_result, false); + + return pmd_result; } /*