From patchwork Sun Mar 19 00:15:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 71679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp530347wrt; Sat, 18 Mar 2023 18:28:58 -0700 (PDT) X-Google-Smtp-Source: AK7set/Fp53Rbmcu4A6vlv3TFVKij7BbwMru1szj7SsrXadw48FXHrKkcm5W5Oif9+B8QbTDKiUi X-Received: by 2002:a05:6a20:1bdd:b0:d5:7f0b:f2f with SMTP id cv29-20020a056a201bdd00b000d57f0b0f2fmr11198331pzb.26.1679189338038; Sat, 18 Mar 2023 18:28:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679189338; cv=none; d=google.com; s=arc-20160816; b=SfRBuZ0n60HhcbpXiP3SMc4Y0BAqBjP7T2NoxhATWdXzwI9NrDRtRjaGr9gAw81YBv isjfT3YvgjryncQiphRd13FCHmzwQvpWU/t4fpM9lRwcEsEwZH1msRUg9ufo4ATW89Pj TOx0a75Da8xXWpkKlwOpKZkBPwfLMyDw9Dycnh44ppcn2xtgdLzbmKf8S03rqE8kh7lV JppdizNl0Sya8q+2/Vr2c/+l4WgLx2OPhfjQ7yYeBRRxug+BhV/PIPuIP+yXc2teQb9y iU8adxZ9RxVtwAm9FAAa2SYb1jV7LWxhRTTxUhWFLvd5GfmNxTzynEbm+ZQzyPuPmsk3 kHlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=7b/QOAWeKJUMsVXOohTWsZBhj9LBeQOJng5A/JiJyYY=; b=uyH9sCTqMnMxrxppAPJUhS/6dDgvJHMuhBsQl8lTvED8oKEXPnqCJCGcvrwH4EQ/FW RxUGDMNEeTTIqEuk8xGDO/K5SdzyNBsyoUFFQtr8MtinG0zXNemVDzRRwmUDsttkBq9k FuLqi74Y0PE3Wa6hhKLBI5/HnuP43YqPhGJFMJYVAFwHufhAuf9j+CHnDdUKBHnjWV+y QESUfFkOqXSn22y0VJFyMoLOLuwbILUYjy+caOP7FhrtRWqIF5yNBG+bjOI5i7f2IWDp hk59c24Car/HiZH/fhCLL70sqHaSmL8b/VGPzaLE9RxB6JQufHdNAg58qbfQKd5lSTU6 Tc5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fipricKj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q28-20020a635c1c000000b0050b8a7635dfsi6259866pgb.295.2023.03.18.18.28.45; Sat, 18 Mar 2023 18:28:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fipricKj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229828AbjCSAUK (ORCPT + 99 others); Sat, 18 Mar 2023 20:20:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230070AbjCSATb (ORCPT ); Sat, 18 Mar 2023 20:19:31 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE1FC29163; Sat, 18 Mar 2023 17:18:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679185080; x=1710721080; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=u/rJkxTnxXm+UTYwj285Z8LlwIJixTQ+7DbH6nVmNI4=; b=fipricKjTrDO61A0MNC+r5bnaPod7K3uX4LX8QX8hhfE+IY22XBgZSFS M71sHWVJZRRpFzr32Nv/C0rkdDbW4E+U9qvqAAsZiLZZcEYoenHrYGz2L IBNHZZKAJ46v+RI47UMCzbz1zKiTL6Tnr0/o8mnzi0xTXb22CsLG6jqsu /8Aj4JEREB0xZXFaLYobQn1zS+KOg89VpCZ5w4+nMd/p/xlkSh2U6q0St 9xYu1fk1A8OEJxVYPI31k9TTUMMsRQcp7lkbB0wyGdBFUjEL81VKb3o3B +oPVzJxKXYUFgGkPXqLN3VBoa4UOKkX35XhmGawwXlc3ACPTnhWcE5O4W A==; X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="338491186" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="338491186" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10653"; a="749672865" X-IronPort-AV: E=Sophos;i="5.98,272,1673942400"; d="scan'208";a="749672865" Received: from bmahatwo-mobl1.gar.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.135.34.5]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2023 17:16:28 -0700 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com, david@redhat.com, debug@rivosinc.com, szabolcs.nagy@arm.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v8 21/40] mm: Add guard pages around a shadow stack. Date: Sat, 18 Mar 2023 17:15:16 -0700 Message-Id: <20230319001535.23210-22-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230319001535.23210-1-rick.p.edgecombe@intel.com> References: <20230319001535.23210-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760757639076826952?= X-GMAIL-MSGID: =?utf-8?q?1760757639076826952?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP instruction can move the SSP to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack VMAs, such that there will always be a gap between adjacent shadow stacks. Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can be incremented or decremented by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed. The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like: addq $0x80, %rsp However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this: READ_ONCE(ssp); // read+discard top element on stack ssp += nr_to_pop * 8; // move the shadow stack READ_ONCE(ssp-8); // read+discard last popped stack element The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore, a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault. This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stacks to grow, which is unneeded and adds a strange difference to how most regular stacks work. Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook Acked-by: Mike Rapoport (IBM) Tested-by: Pengfei Xu Tested-by: John Allen Tested-by: Kees Cook --- v8: - Update commit log verbiage (Boris) - Move and update comment (Boris, David Hildenbrand) v5: - Fix typo in commit log v4: - Drop references to 32 bit instructions - Switch to generic code to drop __weak (Peterz) v2: - Use __weak instead of #ifdef (Dave Hansen) - Only have start gap on shadow stack (Andy Luto) - Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen) - Improve commit log partly with verbiage from (Dave Hansen) --- include/linux/mm.h | 52 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 097544afb1aa..d09fbe9f43f8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -349,7 +349,36 @@ extern unsigned int kobjsize(const void *objp); #endif /* CONFIG_ARCH_HAS_PKEYS */ #ifdef CONFIG_X86_USER_SHADOW_STACK -# define VM_SHADOW_STACK VM_HIGH_ARCH_5 /* Should not be set with VM_SHARED */ +/* + * This flag should not be set with VM_SHARED because of lack of support + * core mm. It will also get a guard page. This helps userspace protect + * itself from attacks. The reasoning is as follows: + * + * The shadow stack pointer(SSP) is moved by CALL, RET, and INCSSPQ. The + * INCSSP instruction can increment the shadow stack pointer. It is the + * shadow stack analog of an instruction like: + * + * addq $0x80, %rsp + * + * However, there is one important difference between an ADD on %rsp + * and INCSSP. In addition to modifying SSP, INCSSP also reads from the + * memory of the first and last elements that were "popped". It can be + * thought of as acting like this: + * + * READ_ONCE(ssp); // read+discard top element on stack + * ssp += nr_to_pop * 8; // move the shadow stack + * READ_ONCE(ssp-8); // read+discard last popped stack element + * + * The maximum distance INCSSP can move the SSP is 2040 bytes, before + * it would read the memory. Therefore a single page gap will be enough + * to prevent any operation from shifting the SSP to an adjacent stack, + * since it would have to land in the gap at least once, causing a + * fault. + * + * Prevent using INCSSP to move the SSP between shadow stacks by + * having a PAGE_SIZE guard gap. + */ +# define VM_SHADOW_STACK VM_HIGH_ARCH_5 #else # define VM_SHADOW_STACK VM_NONE #endif @@ -3107,15 +3136,26 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) return mtree_load(&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_GROWSDOWN) + return stack_guard_gap; + + /* See reasoning around the VM_SHADOW_STACK definition */ + if (vma->vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { + unsigned long gap = stack_guard_start_gap(vma); unsigned long vm_start = vma->vm_start; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; return vm_start; }