From patchwork Thu Jan 19 21:22:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45987 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555775wrn; Thu, 19 Jan 2023 13:35:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXuTVB1OLr0YWVejBlcCRBIwopY6CMrtAgSOB/TonTgpqtBaxLoARsd09VlfFZdH8uKiGEKd X-Received: by 2002:a62:1b8a:0:b0:56c:318a:f83b with SMTP id b132-20020a621b8a000000b0056c318af83bmr10121568pfb.13.1674164107175; Thu, 19 Jan 2023 13:35:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164107; cv=none; d=google.com; s=arc-20160816; b=mZNzKXEw+9bvVb7N/YkHFiM62AzkKi/bCjPqJSXs+Hf6YsihHA/VN7HgLO+9LX1wvw F342wQ0eN2ngfACEWUZ49gHYzcmgM+9ltOOX9B9eSdev4Qn3OOq5HCPeS49lquxTvPEy QyFFHdZkGFH180HZEDFJNABqEyHzL5BxXo9vZJ7OLemFCIRNlJhNKfTvlYSMCEUqQcZl AT4H7dqMu8uAyp+rUMvS6AgDYyFBl0fW56MuMyznKNx9wg5MWlo6Ydod9n2UkTToIcRl xRe132DXu5vVNstUEG2k5duuTN8GOEx5L0HUoQmV9B0biZ++vdScDBiYp7Sk2vX62FRW wMYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=GnbG2zT0q0TQKJhci012/C8hz/SCKJl3hl5drYijqMc=; b=CiwmboFPhrrY6Y5TQlxlmKOJs3PCDDDnSlJ2QuSp4t5FzGVG1kZgh16Gj3rqkrSyc1 LK8Vqc6F6R6REVSygnWgvmjacomT/XOiQyt0aQiZd77/kqdqjMv2HqwowsQ7M+qAmyJk la5dGRh9HUcRCgqx1psekYlnjNHtP8/UcFQwvIxYMqzo+EtN5PhvGXhQeOImGu+LrNi1 vYCsTsK0lTwK6MCdW4JJZ2TXP5FOm2OmQln463ZwxiTQBowV+ehyTpcLKIKTZmBtRmiS LiyKYwnJjFvecd5SMgJQq1uAyirqiPBx8U7f2lKwwWMeGydzMimW6Y/jdn7xNPJt+JiD 0/xQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QbY7zYo1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u10-20020a62790a000000b0057674a6a662si38750325pfc.138.2023.01.19.13.34.54; Thu, 19 Jan 2023 13:35:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QbY7zYo1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230382AbjASVcF (ORCPT + 99 others); Thu, 19 Jan 2023 16:32:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230332AbjASV3R (ORCPT ); Thu, 19 Jan 2023 16:29:17 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8ECAA6C42; Thu, 19 Jan 2023 13:24:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163449; x=1705699449; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=uhuMbHp6Kbqx+B8M1wEyR7U+E0nR3IzT2DHoy+pPYUg=; b=QbY7zYo1BNvtIiSNYL7sxgn7pbhLVCivi1d9hy52XZ4+wXgpRoBenjSj XU0gwZ4SMmjefa2hlIRROStXjtw9mpu46EMFoLdIJUo017QO4aWFBZSJK 5XmQyXJ++BMprxsArmzQ5na/kr8lmSDmjxCk6GIqK/OdUcI26PjMsRVFu teMt77yl9qOTnctmdFd8mHKqhattIx8YErHbRHBI9rdmyNaUIbaPsuHCv x6XxksjiOXQZB+QZ+n1wtx4bv8+X+8fPEAcoMT9fZGdOpzz74K8Wnqioz JqFkGS2T9prHoD+TJ+cImNOPEK1y4LxHqyDuYuPUP/wTpisxfxupVf4uz w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119163" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119163" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:26 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989138986" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989138986" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:24 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 01/39] Documentation/x86: Add CET shadow stack description Date: Thu, 19 Jan 2023 13:22:39 -0800 Message-Id: <20230119212317.8324-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488302655497155?= X-GMAIL-MSGID: =?utf-8?q?1755488302655497155?= From: Yu-cheng Yu Introduce a new document on Control-flow Enforcement Technology (CET). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Literal format tweaks (Bagas Sanjaya) - Update EOPNOTSUPP text due to unification after comment from (Kees) - Update 32 bit signal support with new behavior - Remove capitalization on shadow stack (Boris) - Fix typo v4: - Drop clearcpuid piece (Boris) - Add some info about 32 bit v3: - Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper) - Clarify which arch_prctl's can take multiple bits. (Kees) - Describe ASLR characteristics of thread shadow stacks. (Kees) - Add exec section. (Andrew Cooper) - Fix some capitalization (Bagas Sanjaya) - Update new location of enablement status proc. - Add info about new user_shstk software capability. - Add more info about what the kernel pushes to the shadow stack on signal. v2: - Updated to new arch_prctl() API - Add bit about new proc status Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 Documentation/x86/shstk.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index c73d133fd37c..8ac64d7de4dc 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -22,6 +22,7 @@ x86-specific Documentation mtrr pat intel-hfi + shstk iommu intel_txt amd-memory-encryption diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst new file mode 100644 index 000000000000..f2e6f323cf68 --- /dev/null +++ b/Documentation/x86/shstk.rst @@ -0,0 +1,166 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================================================== +Control-flow Enforcement Technology (CET) Shadow Stack +====================================================== + +CET Background +============== + +Control-flow Enforcement Technology (CET) is term referring to several +related x86 processor features that provides protection against control +flow hijacking attacks. The HW feature itself can be set up to protect +both applications and the kernel. + +CET introduces shadow stack and indirect branch tracking (IBT). Shadow stack +is a secondary stack allocated from memory and cannot be directly modified by +applications. When executing a CALL instruction, the processor pushes the +return address to both the normal stack and the shadow stack. Upon +function return, the processor pops the shadow stack copy and compares it +to the normal stack copy. If the two differ, the processor raises a +control-protection fault. IBT verifies indirect CALL/JMP targets are intended +as marked by the compiler with 'ENDBR' opcodes. Not all CPU's have both Shadow +Stack and Indirect Branch Tracking. Today in the 64-bit kernel, only userspace +shadow stack and kernel IBT are supported. + +Requirements to use Shadow Stack +================================ + +To use userspace shadow stack you need HW that supports it, a kernel +configured with it and userspace libraries compiled with it. + +The kernel Kconfig option is X86_USER_SHADOW_STACK, and it can be disabled +with the kernel parameter: nousershstk. + +To build a user shadow stack enabled kernel, Binutils v2.29 or LLVM v6 or later +are required. + +At run time, /proc/cpuinfo shows CET features if the processor supports +CET. "user_shstk" means that userspace shadow stack is supported on the current +kernel and HW. + +Application Enabling +==================== + +An application's CET capability is marked in its ELF note and can be verified +from readelf/llvm-readelf output:: + + readelf -n | grep -a SHSTK + properties: x86 feature: SHSTK + +The kernel does not process these applications markers directly. Applications +or loaders must enable CET features using the interface described in section 4. +Typically this would be done in dynamic loader or static runtime objects, as is +the case in GLIBC. + +Enabling arch_prctl()'s +======================= + +Elf features should be enabled by the loader using the below arch_prctl's. They +are only supported in 64 bit user applications. + +arch_prctl(ARCH_SHSTK_ENABLE, unsigned long feature) + Enable a single feature specified in 'feature'. Can only operate on + one feature at a time. + +arch_prctl(ARCH_SHSTK_DISABLE, unsigned long feature) + Disable a single feature specified in 'feature'. Can only operate on + one feature at a time. + +arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) + Lock in features at their current enabled or disabled status. 'features' + is a mask of all features to lock. All bits set are processed, unset bits + are ignored. The mask is ORed with the existing value. So any feature bits + set here cannot be enabled or disabled afterwards. + +The return values are as follows. On success, return 0. On error, errno can +be:: + + -EPERM if any of the passed feature are locked. + -ENOTSUPP if the feature is not supported by the hardware or + kernel. + -EINVAL arguments (non existing feature, etc) + +The feature's bits supported are:: + + ARCH_SHSTK_SHSTK - Shadow stack + ARCH_SHSTK_WRSS - WRSS + +Currently shadow stack and WRSS are supported via this interface. WRSS +can only be enabled with shadow stack, and is automatically disabled +if shadow stack is disabled. + +Proc Status +=========== +To check if an application is actually running with shadow stack, the +user can read the /proc/$PID/status. It will report "wrss" or "shstk" +depending on what is enabled. The lines look like this:: + + x86_Thread_features: shstk wrss + x86_Thread_features_locked: shstk wrss + +Implementation of the Shadow Stack +================================== + +Shadow Stack Size +----------------- + +A task's shadow stack is allocated from memory to a fixed size of +MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to +the maximum size of the normal stack, but capped to 4 GB. However, +a compat-mode application's address space is smaller, each of its thread's +shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB). + +Signal +------ + +By default, the main program and its signal handlers use the same shadow +stack. Because the shadow stack stores only return addresses, a large +shadow stack covers the condition that both the program stack and the +signal alternate stack run out. + +When a signal happens, the old pre-signal state is pushed on the stack. When +shadow stack is enabled, the shadow stack specific state is pushed onto the +shadow stack. Today this is only the old SSP (shadow stack pointer), pushed +in a special format with bit 63 set. On sigreturn this old SSP token is +verified and restored by the kernel. The kernel will also push the normal +restorer address to the shadow stack to help userspace avoid a shadow stack +violation on the sigreturn path that goes through the restorer. + +So the shadow stack signal frame format is as follows:: + + |1...old SSP| - Pointer to old pre-signal ssp in sigframe token format + (bit 63 set to 1) + | ...| - Other state may be added in the future + + +32 bit ABI signals are not supported in shadow stack processes. Linux prevents +32 bit execution while shadow stack is enabled by the allocating shadow stack's +outside of the 32 bit address space. When execution enters 32 bit mode, either +via far call or returning to userspace, a #GP is generated by the hardware +which, will be delivered to the process as a segfault. When transitioning to +userspace the register's state will be as if the userspace ip being returned to +caused the segfault. + +Fork +---- + +The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required +to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a +shadow access triggers a page fault with the shadow stack access bit set +in the page fault error code. + +When a task forks a child, its shadow stack PTEs are copied and both the +parent's and the child's shadow stack PTEs are cleared of the dirty bit. +Upon the next shadow stack access, the resulting shadow stack page fault +is handled by page copy/re-use. + +When a pthread child is created, the kernel allocates a new shadow stack +for the new thread. New shadow stack's behave like mmap() with respect to +ASLR behavior. + +Exec +---- + +On exec, shadow stack features are disabled by the kernel. At which point, +userspace can choose to re-enable, or lock them. From patchwork Thu Jan 19 21:22:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45982 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555616wrn; Thu, 19 Jan 2023 13:34:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXu9FroK47EyKaEeYS67n2VEBZ0+NHcfoBLkXuSFNWtQoFnOh7SMHyezmUWFCq75ReWcgEHH X-Received: by 2002:a17:90a:19d5:b0:223:ed96:e3ca with SMTP id 21-20020a17090a19d500b00223ed96e3camr12528870pjj.28.1674164079765; Thu, 19 Jan 2023 13:34:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164079; cv=none; d=google.com; s=arc-20160816; b=a9VDFVW2+78jhBJZEeWaTBUugJxSTJ/yyatYyzlbdFWMrAT/MyJQt/LTXiIoRso/qi e7ZwtpI2cpHPK2O5EAxmRDD4F1fMQVWaC1MHKZabvHOLUZdPu/e4hqLwoxT+EEmgruuA Kz/s2J3wi1218gDhxj0UqkfRlPptKt6pe2vz+JpG5UlclopZDAFiv2dULYDO94kVyJGB 0Hx8Kw7tU0vES7emW3HmdjXpA2l4jksHVH1QyGUTxV13V63TFN3PUbElFuttog/QW6hQ XaOkJH9aPZv8oXRsj58KPh7MOKthpTqdvLFsw9e4yuqtc3kHW5bMemIqJzwLQrF6t/hG zf3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=3U7z+h3YDF8Sba0ajewGuAx6MSy1V++Xb+AytMN0PuA=; b=itZZRNbfs2ahMhKDkeCsDNaY7TgAB367AFrxro0DdYzl9OoQbCRqCL0SsUWzFtolWg MFC7oC3aulnMCFnAZRg0YA7QVHA9JJaJDb2dPpvBLFYtKEvCgtauq5AsZUB/jNIr1ntu jsxwCTujYpXldyPf8il9mWN43mzWmEmAWMnaYXrrNqiVHaRoPfrvbbZDfRwgP/1RhBm+ p3r328VMUrWTmmbeuk1VwNQhlHD07MGKsXOdJarf77fAKAtD6BON2dcbAQ9jY7U6Wms5 I5BrMYvQehpqs/fPY3xOWVyf83J+FwsxeItiv1mR1jr95nhBXj0TZBTFnSu7AJtJzTgr NZ/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FSil+2cW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gj5-20020a17090b108500b00223ea512658si262170pjb.162.2023.01.19.13.34.27; Thu, 19 Jan 2023 13:34:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FSil+2cW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230403AbjASVc1 (ORCPT + 99 others); Thu, 19 Jan 2023 16:32:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230059AbjASV3e (ORCPT ); Thu, 19 Jan 2023 16:29:34 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 518FBA6C61; Thu, 19 Jan 2023 13:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163455; x=1705699455; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=qg2fCiCJOKwg0sz79Qyr0Ky5ojS7yp2AsnYDlj2uU6w=; b=FSil+2cWSOKTN+AoMyZqvpsvaHJnLkszDDt14KNeMZzBSrHEi2fhvYjl eIPF/9tLED4/SRgw9SvsIr675x805xkQ26bvN29SO5P24DJT4wKZVRz8m jUnVGIEW3u5hXWXbTurAxckhQ8UQAfOvXg7NdI/m9s41mkQwZV6grHLEE 3Gu3nuPQdsS8Qqv51S0IghdfKfsqsP6khbip1nw6M1n7PffhypJOEv8F7 3M16LuGRnjcdwdXJHCu1hJwIfyeaWTIYPweNZHQlbtpkHr8coPP3ITJS2 qVrDSZftODGQcPALKWPuypV8zRoJvRempGjfcPoqM3zFQ/5pN1vS4L4j/ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119192" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119192" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989138990" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989138990" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:26 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 02/39] x86/shstk: Add Kconfig option for shadow stack Date: Thu, 19 Jan 2023 13:22:40 -0800 Message-Id: <20230119212317.8324-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488274387252169?= X-GMAIL-MSGID: =?utf-8?q?1755488274387252169?= From: Yu-cheng Yu Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection. Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create CONFIG_CET to signify that at least one CET feature is configured. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Remove capitalization of shadow stack (Boris) v3: - Add X86_CET (Kees) - Add back WRUSS dependency (Kees) - Fix verbiage (Dave) - Change from promt to bool (Kirill) - Add more to commit log v2: - Remove already wrong kernel size increase info (tlgx) - Change prompt to remove "Intel" (tglx) - Update line about what CPUs are supported (Dave) Yu-cheng v25: - Remove X86_CET and use X86_SHADOW_STACK directly. arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 29 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..d0037181bc15 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1851,6 +1851,11 @@ config CC_HAS_IBT (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \ $(as-instr,endbr64) +config X86_CET + def_bool n + help + CET features configured (Shadow stack or IBT) + config X86_KERNEL_IBT prompt "Indirect Branch Tracking" def_bool y @@ -1858,6 +1863,7 @@ config X86_KERNEL_IBT # https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f depends on !LD_IS_LLD || LLD_VERSION >= 140000 select OBJTOOL + select X86_CET help Build the kernel with support for Indirect Branch Tracking, a hardware support course-grain forward-edge Control Flow Integrity @@ -1952,6 +1958,24 @@ config X86_SGX If unsure, say N. +config X86_USER_SHADOW_STACK + bool "X86 userspace shadow stack" + depends on AS_WRUSS + depends on X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_CET + help + Shadow stack protection is a hardware feature that detects function + return address corruption. This helps mitigate ROP attacks. + Applications must be enabled to use it, and old userspace does not + get protection "for free". + + CPUs supporting shadow stacks were first released in 2020. + + See Documentation/x86/shstk.rst for more information. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Kconfig.assembler b/arch/x86/Kconfig.assembler index 26b8c08e2fc4..00c79dd93651 100644 --- a/arch/x86/Kconfig.assembler +++ b/arch/x86/Kconfig.assembler @@ -19,3 +19,8 @@ config AS_TPAUSE def_bool $(as-instr,tpause %ecx) help Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7 + +config AS_WRUSS + def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) + help + Supported by binutils >= 2.31 and LLVM integrated assembler From patchwork Thu Jan 19 21:22:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45979 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555425wrn; Thu, 19 Jan 2023 13:34:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXs/prTw7oNdmCCIFqszLK2UNzvyLKwxOtFSdQWuKtrUfGjX59X2hFf553kQDJ7ZXTE+SHGt X-Received: by 2002:a05:6a21:788e:b0:ac:82ff:9f9e with SMTP id bf14-20020a056a21788e00b000ac82ff9f9emr47619221pzc.22.1674164053062; Thu, 19 Jan 2023 13:34:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164053; cv=none; d=google.com; s=arc-20160816; b=UIlvse1TIClvQWuoUxqA03ynIfMLGdACt7cuiuKjK8xKPqrmphxVzn+22gF0LFpfVv 2C/EiawwdZnr6WPbC7v2N1qiVZ7idZ/1QCrhWNLxQ30zc3LvdbVfmJn6LbiVRvXE7jsd DWkf5VI6uedvsNQUW50oGapOYYqnjRjKVkfM63gY8XCGFofPvM04bCSiRoEPGiKQgtV3 NnDXmn8DgzBbHxxqGse+b9mZjzQhXq6uIp3OzYtV1FwofG2/3UuYPfyfD3j8hCe50lIC IDQuGN5eWXF+EcP+U7E5G8a8oK97zooxSEQLg9AMAQBviYNuJAEFHJN4Chc7mcyeXctZ /nCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=4jbP/vvwC2xDxOm16xmDL1x5rE+wMcxLrGHW3HbSuJY=; b=Y2kmGiCC4uLp6Pw9A3cM4gLorU3XpHtjwHgZBY2QoBMmuMO7hd/GZxd4OMw5yljLg6 vC+oINeaB+tt08Nw61Y0RHy6YRo9JPjyYc3vupZcUderowx0bpJvyPk4USjVUKubW3cS WKnML8BGdI5rd7BUk/RzM5LnyC51VkQR9+XYHZQqx/O9gWqZ8kFD7jBKRHxFKX8AhYVg 0qls13fVUQI9zgBDgMHbIqGV1uv8AQSBvkKo5oFGGgxKU/ktOzJurqGI07i5bCdO7hWM k7BYsnfHi2+eXOcACcfIEA+WYei+2stWa6arYKCICMr8HIFLqCQDvfZ1VSy6R5BJLQOV MLkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WdBnp6zX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 131-20020a630089000000b0047865b303a5si2159620pga.762.2023.01.19.13.34.00; Thu, 19 Jan 2023 13:34:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WdBnp6zX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230121AbjASVcN (ORCPT + 99 others); Thu, 19 Jan 2023 16:32:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230420AbjASVaF (ORCPT ); Thu, 19 Jan 2023 16:30:05 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFA05A7333; Thu, 19 Jan 2023 13:24:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163490; x=1705699490; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=d2PWSkL27px2D+va1Qwpdf/EaJ+P/H11jw+dYhLJV08=; b=WdBnp6zXjbHCe8tTccNcz7wm7jAaHdnt4PdEVoy/DXH+4nVi4Ob/BP3e Kin5MW/5JVzb7kpMrnI/okMbYwAexXzyaRSAIszyNJHoWWK9Iz5RKYLWZ bsp0Uo24nIxCOXRk2NGyHrwAaek3sE+nt6Coqov0dzmLML6aH/iKnSeLk ZHYZPfZ4iTg5znhJ73Gilj1ZKqNEviAegPDMAEaBAzcN68mmGsADXfSLI egKblWleYsL1Ilo5bDArBJ4DpnRWaKLvdaC+a7J71Y/AXsRwaUljrAqJJ Z0TH8Lr9LGprLUjLcBRIuaqF5O1CgZzt5skGe4odaD7LqiB8dO9RL/48j g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119217" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119217" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989138996" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989138996" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:27 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 03/39] x86/cpufeatures: Add CPU feature flags for shadow stacks Date: Thu, 19 Jan 2023 13:22:41 -0800 Message-Id: <20230119212317.8324-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488246267997023?= X-GMAIL-MSGID: =?utf-8?q?1755488246267997023?= From: Yu-cheng Yu The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]). To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES. The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Drop "shstk" from cpuinfo (Boris) - Remove capitalization on shadow stack (Boris) v3: - Add user specific shadow stack cpu cap (Andrew Cooper) - Drop reviewed-bys from Boris and Kees due to the above change. v2: - Remove IBT reference in commit log (Kees) - Describe xsaves dependency using text from (Dave) v1: - Remove IBT, can be added in a follow on IBT series. arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 7b319acda31a..a8551b6c8041 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -307,6 +307,7 @@ #define X86_FEATURE_SGX_EDECCSSA (11*32+18) /* "" SGX EDECCSSA user leaf function */ #define X86_FEATURE_CALL_DEPTH (11*32+19) /* "" Call depth tracking for RSB stuffing */ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20) /* "" MSR IA32_TSX_CTRL (Intel) implemented */ +#define X86_FEATURE_USER_SHSTK (11*32+21) /* Shadow stack support for user mode applications */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ @@ -373,6 +374,7 @@ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5) /* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* "" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 5dfa4fb76f4b..505f78ddca82 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -99,6 +99,12 @@ # define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31)) #endif +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define DISABLE_USER_SHSTK 0 +#else +#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -114,7 +120,7 @@ #define DISABLED_MASK9 (DISABLE_SGX) #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ - DISABLE_CALL_DEPTH_TRACKING) + DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index d95221117129..c3e4e5246df9 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -79,6 +79,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, {} }; From patchwork Thu Jan 19 21:22:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45976 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555229wrn; Thu, 19 Jan 2023 13:33:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXvOBbquZDlzBDJ9DXU+chm6hsNqp/bJVPXvw6zrO3tbaMga7fCByplRoVUarDclIAoeaaXU X-Received: by 2002:a05:6a20:b289:b0:b8:77a6:5afb with SMTP id ei9-20020a056a20b28900b000b877a65afbmr12206484pzb.28.1674164018315; Thu, 19 Jan 2023 13:33:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164018; cv=none; d=google.com; s=arc-20160816; b=rt4NqGZfIIRN1qghgr8JWgsl5blIlLZGNUyqKeZJcCuDe7dtsBCdJIKjnYR8dM4sxI 4rsyZUKdg6eLIBw3H/QKPbHlq054pL5InmlpQRCkRpbbDLJJTlWRrUeGt0IuNMMNuq2+ Rfl2a88LsdhIHtl8+LyLteieStBHSCrHXERnVWT/etle5w4Gi76exLGx64VIYVbKZ4qM wSrhX4PmFkMlnSlMx/H+9g1QHmRTvdjEFToCpEf+5DmIz/vWe9DO9mBZE/kdPdk4wYzg HujMomqKleNnZNSTJsS0hpLk7lyLAho0d+RfyN4eiFIcD4r1x7UF6QPyURgAjzCUNC4w UAqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=vxp0lIQdGnMd31RbCo3vuJaZKtAwgMarQuDGlKx2nuY=; b=XB9+7JlBafGoYB3fLTwrUMqEWKy0hEIsDR/5X373drG5lwm9Low2ShYJ1gO8vLfUxL fdusKwXkFUcadeSzxkkwAbz2M9AG6Gpda8VfSuljOK52vwFDyKI7y6SEPMiCbHl8lzE4 sY0Hm9385Gp2BVe1A4wEsuCNqbeKJxFNXcHIDSAfIaLqDMwfqt58peq2FgGDEyTG0Ok0 My4jsLamI/SxurPRT5gX6bcndYBGGFlx/r/SHkwaTADJZnA8u/Srj6vjZ3W7xssv3aNv D2HNS+1PquqU80BPxnovE1KO9eogabDN/0LCNMFGJrjhOwHHYIOL3D1YHaU22vCirpSk sUzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=koWs2Bbx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q131-20020a632a89000000b004772ea50c14si39689743pgq.171.2023.01.19.13.33.25; Thu, 19 Jan 2023 13:33:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=koWs2Bbx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229984AbjASVba (ORCPT + 99 others); Thu, 19 Jan 2023 16:31:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230426AbjASVaG (ORCPT ); Thu, 19 Jan 2023 16:30:06 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF5F9A7330; Thu, 19 Jan 2023 13:24:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163490; x=1705699490; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=d7P5HLFSxYXgzr8kgfmXPsFqpuUUbCRpcWe/3LmTQFM=; b=koWs2BbxJ1+kgpMz+9gR6xYGNyqwMNcGteVIpZFV4KtIsbMzEVVAPkIe 0isSzIodvx8up5YkzJrZECmEh8dXAPwe/yw4r4N1G1KTwamA3+2nUBFhO pysxZzatDNSTPPOR0m5Lkz73VHvI+ARK5hu2mU4TbuD9BT8EoJ9uHU4eM GUfXroki3Xr/CY3IzIi0Z+pkSwYULsaYlHiULLpmlMBGi19knOLSnPayP gKKl0HG9j9ieXuVFr7Zx3+JKd5n7hGuyIIsuKbPz/GX9PCyBtjz/lM5YA Lh0PtoJAPpARNHImc1hB9Q5FwJZG7WUhv9xzkpziY6+nJ5T7rV4jPs4tt Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119244" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119244" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:31 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139002" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139002" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:29 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack Date: Thu, 19 Jan 2023 13:22:42 -0800 Message-Id: <20230119212317.8324-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488209844958853?= X-GMAIL-MSGID: =?utf-8?q?1755488209844958853?= From: Yu-cheng Yu Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs. Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case. Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Remove #ifdeffery (Boris) v4: - Add back dedicated command line disable: "nousershtk" (Boris) v3: - Remove stay new line (Boris) - Simplify commit log (Andrew Cooper) v2: - In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support. - Clear MSR_IA32_U_CET as well. (PeterZ) arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index cec654e674ff..80507a5ba0ca 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -599,27 +599,43 @@ __noendbr void ibt_restore(u64 save) static __always_inline void setup_cet(struct cpuinfo_x86 *c) { - u64 msr = CET_ENDBR_EN; + bool user_shstk, kernel_ibt; - if (!HAS_KERNEL_IBT || - !cpu_feature_enabled(X86_FEATURE_IBT)) + if (!IS_ENABLED(CONFIG_X86_CET)) return; - wrmsrl(MSR_IA32_S_CET, msr); + kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT); + user_shstk = cpu_feature_enabled(X86_FEATURE_SHSTK) && + IS_ENABLED(CONFIG_X86_USER_SHADOW_STACK); + + if (!kernel_ibt && !user_shstk) + return; + + if (user_shstk) + set_cpu_cap(c, X86_FEATURE_USER_SHSTK); + + if (kernel_ibt) + wrmsrl(MSR_IA32_S_CET, CET_ENDBR_EN); + else + wrmsrl(MSR_IA32_S_CET, 0); + cr4_set_bits(X86_CR4_CET); - if (!ibt_selftest()) { + if (kernel_ibt && !ibt_selftest()) { pr_err("IBT selftest: Failed!\n"); wrmsrl(MSR_IA32_S_CET, 0); setup_clear_cpu_cap(X86_FEATURE_IBT); - return; } } __noendbr void cet_disable(void) { - if (cpu_feature_enabled(X86_FEATURE_IBT)) - wrmsrl(MSR_IA32_S_CET, 0); + if (!(cpu_feature_enabled(X86_FEATURE_IBT) || + cpu_feature_enabled(X86_FEATURE_SHSTK))) + return; + + wrmsrl(MSR_IA32_S_CET, 0); + wrmsrl(MSR_IA32_U_CET, 0); } /* @@ -1476,6 +1492,9 @@ static void __init cpu_parse_early_param(void) if (cmdline_find_option_bool(boot_command_line, "noxsaves")) setup_clear_cpu_cap(X86_FEATURE_XSAVES); + if (cmdline_find_option_bool(boot_command_line, "nousershstk")) + setup_clear_cpu_cap(X86_FEATURE_USER_SHSTK); + arglen = cmdline_find_option(boot_command_line, "clearcpuid", arg, sizeof(arg)); if (arglen <= 0) return; From patchwork Thu Jan 19 21:22:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45983 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555635wrn; Thu, 19 Jan 2023 13:34:42 -0800 (PST) X-Google-Smtp-Source: AMrXdXsNTB4c3nXorZzsUEZ5rCSVV7l7gvRC1RkdMMVzJdzGcjvRkfBFyET9xs9k5GEGRUfr0I7K X-Received: by 2002:a17:90b:4fc2:b0:229:680:1729 with SMTP id qa2-20020a17090b4fc200b0022906801729mr12375487pjb.10.1674164081708; Thu, 19 Jan 2023 13:34:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164081; cv=none; d=google.com; s=arc-20160816; b=1JJ+/zW7KdTfLau/AY9TEAY//fEHQnIcMRZtUwpvjnt1ZPCZzUyYhNcVU9EYmhP18M gTGmUUkeybR+1AJxAVqS9JoWOFlQGKtSxXp0jOZwXXXgVF8LKUeqKg8tzb50lp9elYQZ +qGtt1Fco+ep1Fhv1b9ZklhpYr+Ohi4CanzZ0wNqvQSI4usCEoV1PA6Nij9Ws7/Wo0Yw lh+u2XvEIgXFt4Rbh7xJk/vi9/ynbm6bNRfcNNTJZV9URNlstRctRiEtYSk5ADmV2VoO cF8LLJC2EPkxGdeiib3bMrryNUaSskcHiGoMcn7zs/n+2AJbEK/rzJdHPIsiWgOIB8gC RKCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=V27cKRY2KCv5vyRkPBcMUXcbgd5W/TlMLH+92/zt3E0=; b=pSZolblDcPlKbF/z5FU59M1C+03lpj5x1Vtcyzfl/ApfTUVjObjzFg68Iw31ImmYVL cqslzMDfPs/to+C1NsrjEjMCiIynLVL1XyjvY/KmJ1OYEbl61hgKNcVsVlngBjZstkbU xPfYM7ZSEug5AVKw4B+teowrHccpnyUv4fL50ckWMBZGYqxKLV9tyaxLkP6l4lSFJymX Gb1vseZYkV0prKFFZp4VSFbbJ6UoXXwGiHo2X16M6o6drsin+OLifqz4GtHMByUR0JSK TsTUa8IOoat3iHon+5EfmXwi9rjK9m897LVIEjegx5LX4aCuF+JZrtOeCXn8svVc7uwU +XNQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mDk7MnRH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h1-20020a17090ac38100b0021924a18b39si321173pjt.96.2023.01.19.13.34.29; Thu, 19 Jan 2023 13:34:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mDk7MnRH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230356AbjASVbp (ORCPT + 99 others); Thu, 19 Jan 2023 16:31:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230450AbjASVaJ (ORCPT ); Thu, 19 Jan 2023 16:30:09 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDA20A731C; Thu, 19 Jan 2023 13:24:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163497; x=1705699497; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=oiH2FubPSeETMvfMbKf2yNJtMKC2vkuWNh3uopEbpLc=; b=mDk7MnRH9jmsahvkEv2R7C6uL5k7daneoP3rdMhBO3QA/18J7gcFsMlF d8QnO6XbOKIhYltzK2bMZX8YmEhBYDRFChz7X+c1/IUqhYYwtudu0z2MT RBOQKPD61fiFvCMHDBA6liSEmuaV65k1EHqXSPNua4KWO7H4h4APYX7p4 a/keJZJl/fe22836XAYfD9tLdaiEiYXKC/X02y3rRET000Mml/SdB24Iy 32RfYuuxR7yirfuIS9QGE742uF4X9trgfi2lByNokP9YClNjm8zaMlk3z Q2TYWo2+351KF+SzfbL+eo26Pmt2V6laG5rZQ04or5+IVbXE9RUGQj2Fr w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119266" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119266" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:32 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139006" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139006" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:31 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states Date: Thu, 19 Jan 2023 13:22:43 -0800 Message-Id: <20230119212317.8324-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488276217815183?= X-GMAIL-MSGID: =?utf-8?q?1755488276217815183?= From: Yu-cheng Yu Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups: * Registers controlling user-mode operation * Registers controlling kernel-mode operation The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size. Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required. Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable. While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks. Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET. To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Move comments from end of lines in cet_user_state struct (Boris) v3: - Add missing "is" in commit log (Boris) - Change to case statement for struct size checking (Boris) - Adjust commas on xfeature_names (Kees, Boris) v2: - Change name to XFEATURE_CET_KERNEL_UNUSED (peterz) KVM refresh: - Reword commit log using some verbiage posted by Dave Hansen - Remove unlikely to be used supervisor cet xsave struct - Clarify that supervisor cet state is not saved by xsave - Remove unused supervisor MSRs arch/x86/include/asm/fpu/types.h | 16 +++++- arch/x86/include/asm/fpu/xstate.h | 6 ++- arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++---------------- 3 files changed, 61 insertions(+), 51 deletions(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index eb7cd1139d97..26abde698fc0 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -115,8 +115,8 @@ enum xfeature { XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, XFEATURE_PASID, - XFEATURE_RSRVD_COMP_11, - XFEATURE_RSRVD_COMP_12, + XFEATURE_CET_USER, + XFEATURE_CET_KERNEL_UNUSED, XFEATURE_RSRVD_COMP_13, XFEATURE_RSRVD_COMP_14, XFEATURE_LBR, @@ -138,6 +138,8 @@ enum xfeature { #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) #define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL_UNUSED) #define XFEATURE_MASK_LBR (1 << XFEATURE_LBR) #define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG) #define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA) @@ -252,6 +254,16 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control-flow Enforcement user states + */ +struct cet_user_state { + /* user control-flow settings */ + u64 user_cet; + /* user shadow stack pointer */ + u64 user_ssp; +}; + /* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth). diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index cd3dd170e23a..d4427b88ee12 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -50,7 +50,8 @@ #define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA /* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER) /* * A supervisor state component may not always contain valuable information, @@ -77,7 +78,8 @@ * Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL) /* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \ diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 714166cc25f2..13a80521dd51 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -39,26 +39,26 @@ */ static const char *xfeature_names[] = { - "x87 floating point registers" , - "SSE registers" , - "AVX registers" , - "MPX bounds registers" , - "MPX CSR" , - "AVX-512 opmask" , - "AVX-512 Hi256" , - "AVX-512 ZMM_Hi256" , - "Processor Trace (unused)" , + "x87 floating point registers", + "SSE registers", + "AVX registers", + "MPX bounds registers", + "MPX CSR", + "AVX-512 opmask", + "AVX-512 Hi256", + "AVX-512 ZMM_Hi256", + "Processor Trace (unused)", "Protection Keys User registers", "PASID state", - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "AMX Tile config" , - "AMX Tile data" , - "unknown xstate feature" , + "Control-flow User registers", + "Control-flow Kernel registers (unused)", + "unknown xstate feature", + "unknown xstate feature", + "unknown xstate feature", + "unknown xstate feature", + "AMX Tile config", + "AMX Tile data", + "unknown xstate feature", }; static unsigned short xsave_cpuid_features[] __initdata = { @@ -73,6 +73,7 @@ static unsigned short xsave_cpuid_features[] __initdata = { [XFEATURE_PT_UNIMPLEMENTED_SO_FAR] = X86_FEATURE_INTEL_PT, [XFEATURE_PKRU] = X86_FEATURE_PKU, [XFEATURE_PASID] = X86_FEATURE_ENQCMD, + [XFEATURE_CET_USER] = X86_FEATURE_SHSTK, [XFEATURE_XTILE_CFG] = X86_FEATURE_AMX_TILE, [XFEATURE_XTILE_DATA] = X86_FEATURE_AMX_TILE, }; @@ -276,6 +277,7 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); print_xstate_feature(XFEATURE_MASK_PASID); + print_xstate_feature(XFEATURE_MASK_CET_USER); print_xstate_feature(XFEATURE_MASK_XTILE_CFG); print_xstate_feature(XFEATURE_MASK_XTILE_DATA); } @@ -344,6 +346,7 @@ static __init void os_xrstor_booting(struct xregs_state *xstate) XFEATURE_MASK_BNDREGS | \ XFEATURE_MASK_BNDCSR | \ XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER | \ XFEATURE_MASK_XTILE) /* @@ -446,14 +449,15 @@ static void __init __xstate_dump_leaves(void) } \ } while (0) -#define XCHECK_SZ(sz, nr, nr_macro, __struct) do { \ - if ((nr == nr_macro) && \ - WARN_ONCE(sz != sizeof(__struct), \ - "%s: struct is %zu bytes, cpu state %d bytes\n", \ - __stringify(nr_macro), sizeof(__struct), sz)) { \ +#define XCHECK_SZ(sz, nr, __struct) ({ \ + if (WARN_ONCE(sz != sizeof(__struct), \ + "[%s]: struct is %zu bytes, cpu state %d bytes\n", \ + xfeature_names[nr], sizeof(__struct), sz)) { \ __xstate_dump_leaves(); \ } \ -} while (0) + true; \ +}) + /** * check_xtile_data_against_struct - Check tile data state size. @@ -527,36 +531,28 @@ static bool __init check_xstate_against_struct(int nr) * Ask the CPU for the size of the state. */ int sz = xfeature_size(nr); + /* * Match each CPU state with the corresponding software * structure. */ - XCHECK_SZ(sz, nr, XFEATURE_YMM, struct ymmh_struct); - XCHECK_SZ(sz, nr, XFEATURE_BNDREGS, struct mpx_bndreg_state); - XCHECK_SZ(sz, nr, XFEATURE_BNDCSR, struct mpx_bndcsr_state); - XCHECK_SZ(sz, nr, XFEATURE_OPMASK, struct avx_512_opmask_state); - XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); - XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); - XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); - XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state); - XCHECK_SZ(sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg); - - /* The tile data size varies between implementations. */ - if (nr == XFEATURE_XTILE_DATA) - check_xtile_data_against_struct(sz); - - /* - * Make *SURE* to add any feature numbers in below if - * there are "holes" in the xsave state component - * numbers. - */ - if ((nr < XFEATURE_YMM) || - (nr >= XFEATURE_MAX) || - (nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) || - ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) { + switch (nr) { + case XFEATURE_YMM: return XCHECK_SZ(sz, nr, struct ymmh_struct); + case XFEATURE_BNDREGS: return XCHECK_SZ(sz, nr, struct mpx_bndreg_state); + case XFEATURE_BNDCSR: return XCHECK_SZ(sz, nr, struct mpx_bndcsr_state); + case XFEATURE_OPMASK: return XCHECK_SZ(sz, nr, struct avx_512_opmask_state); + case XFEATURE_ZMM_Hi256: return XCHECK_SZ(sz, nr, struct avx_512_zmm_uppers_state); + case XFEATURE_Hi16_ZMM: return XCHECK_SZ(sz, nr, struct avx_512_hi16_state); + case XFEATURE_PKRU: return XCHECK_SZ(sz, nr, struct pkru_state); + case XFEATURE_PASID: return XCHECK_SZ(sz, nr, struct ia32_pasid_state); + case XFEATURE_XTILE_CFG: return XCHECK_SZ(sz, nr, struct xtile_cfg); + case XFEATURE_CET_USER: return XCHECK_SZ(sz, nr, struct cet_user_state); + case XFEATURE_XTILE_DATA: check_xtile_data_against_struct(sz); return true; + default: XSTATE_WARN_ON(1, "No structure for xstate: %d\n", nr); return false; } + return true; } From patchwork Thu Jan 19 21:22:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45978 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555371wrn; Thu, 19 Jan 2023 13:34:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXtRU2F+MUiGTtYMODOjrrl9sNFeUtOdZcMUA+yh5+usGhUtWlnwHAjuxTFBGjl0vWdIVG/G X-Received: by 2002:a17:902:6ac3:b0:194:c25a:d824 with SMTP id i3-20020a1709026ac300b00194c25ad824mr5496596plt.26.1674164040153; Thu, 19 Jan 2023 13:34:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164040; cv=none; d=google.com; s=arc-20160816; b=QVCPmOV30CNdTvjrQvBd1m3h5bNdGeS3+FeAO/t1UAOrzc0PUr7yM1gFHb8OyaCpV8 C/nM4pop8cV0AT0/iPa5ZoaWGEFGKPbqnahWBpl4anuiFz9kAKwx7HLoarCbUATZZug5 xXACR0c6wRBbg5I4f5yLDrzYnu+YsREY4u9O3q8SBGaWGEj5eNZGK0ywtTHmyJQ9Z+ql s1fiQNsV4teYv1jEsU3Rc8eL5/Rd/lXG/l5YxbbuLd9dGXJaszfrPRJF/SZllUJXvEjw H5AVyzAoIjG0PX6e0KauRCQvH4UI0XChdGWkOUcrRr3wb7bbIowJBFp4FNSOlsVPf3PZ j0mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=/I2rA7g9gaeqAh7utVe0HX92Ho3+6Ko6l1A1i1JyDLw=; b=F3YVe2PufUZoCOPpO9qNRcsKEQF3au5k/FFyBi6Anf5rZlEe10kLrJhVR+G0IEXw2i Jwdbsn7OiTlWYlN4G8eh/lLZ+wDrj6y9Nfx8UxbrhzC+ZM2gpNURYWJ8SVMRhusPuvtS Gf2MwnBt95y/ZuwvggF/T+NKTQF6G88X/Y+jfUiLcg0FPEnPt3VFO5zTm+E1Z+v78j3y zrcE2b6w0/m3nl3jhnfyfyyPWWPW5v4k0Iyf59WsQh8e57Nl4TiM3+FsEi3HU3VxI4gS KD+PmEaZfR4o7+HwDlm/Rkn+2/9XppQ/36D8kOWw34vs/WtTLUUKdwbikk4C79uX2khm 1pmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CU4D+mLB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t23-20020a170902b21700b001946e6c4fb9si9972308plr.96.2023.01.19.13.33.47; Thu, 19 Jan 2023 13:34:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CU4D+mLB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230376AbjASVby (ORCPT + 99 others); Thu, 19 Jan 2023 16:31:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230506AbjASVar (ORCPT ); Thu, 19 Jan 2023 16:30:47 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39881A7914; Thu, 19 Jan 2023 13:25:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163509; x=1705699509; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=AD5tXA8hUee5SZ1Pd+Ll3tDzu/Wr8phq95vUnNoKkT8=; b=CU4D+mLB7mIoLlEv8ME3Sg6ThixyUMvysX59m+nItF2n7lFvLiCdsEES dHYLyQaCni72Uwvflj2ees3kSSvdzO7xRg3whl8etMdJU5qN8hPVObSrG KC+H+r+UM13KCbgDdEugVWA6Epr8VUfcZbTllZt3bRrvarsykPrl/r7gO 3VyIi2tajAM9Uti60NkBpVq2HJAWX24ur8B/JZ5EPmq21b+jFq+HDbfTy 5LzLBXz98v24PPQLltyjDzinS8+zUT51ku/KWSqpA/yUqYbulpu2i71M6 gZKRMfbqmXK12RrXZvPTxjFblkNcQ3vRhgDOiCPTD6v267AVG5M5n8m0q A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119291" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119291" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:34 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139011" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139011" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:32 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate Date: Thu, 19 Jan 2023 13:22:44 -0800 Message-Id: <20230119212317.8324-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488232201099275?= X-GMAIL-MSGID: =?utf-8?q?1755488232201099275?= Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer. When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is. The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future. Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code. Tested-by: Pengfei Xu Tested-by: John Allen Suggested-by: Thomas Gleixner Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix spelling error (Boris) - Don't export fpregs_lock_and_load() (Boris) v3: - Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees) - Elaborate in comment about helper. (Dave) v2: - Drop optimization of writing directly the buffer, and change API accordingly. - fpregs_lock_and_load() suggested by tglx - Some commit log verbiage from dhansen v1: - New patch. arch/x86/include/asm/fpu/api.h | 9 +++++++++ arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index 503a577814b2..aadc6893dcaa 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -82,6 +82,15 @@ static inline void fpregs_unlock(void) preempt_enable(); } +/* + * FPU state gets lazily restored before returning to userspace. So when in the + * kernel, the valid FPU state may be kept in the buffer. This function will force + * restore all the fpu state to the registers early if needed, and lock them from + * being automatically saved/restored. Then FPU state can be modified safely in the + * registers, before unlocking with fpregs_unlock(). + */ +void fpregs_lock_and_load(void); + #ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent(void); #else diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index dccce58201b7..7317bfd5ea36 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -753,6 +753,24 @@ void switch_fpu_return(void) } EXPORT_SYMBOL_GPL(switch_fpu_return); +void fpregs_lock_and_load(void) +{ + /* + * fpregs_lock() only disables preemption (mostly). So modifying state + * in an interrupt could screw up some in progress fpregs operation, + * but appear to work. Warn about it. + */ + WARN_ON_ONCE(!irq_fpu_usable()); + WARN_ON_ONCE(current->flags & PF_KTHREAD); + + fpregs_lock(); + + fpregs_assert_state_consistent(); + + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + fpregs_restore_userregs(); +} + #ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this From patchwork Thu Jan 19 21:22:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45985 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555701wrn; Thu, 19 Jan 2023 13:34:54 -0800 (PST) X-Google-Smtp-Source: AMrXdXtTDrpcm6CVwCQxEa4JH6QN1KAdK8XDAzL7LC6juLvrQAzGbJORZEu3O+kUkMCKjYiWQbtq X-Received: by 2002:a17:90b:3c90:b0:229:32de:981e with SMTP id pv16-20020a17090b3c9000b0022932de981emr12714664pjb.10.1674164093895; Thu, 19 Jan 2023 13:34:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164093; cv=none; d=google.com; s=arc-20160816; b=i5jWjb7MYOtQIYM35OvPbuEbys+Zcbmi6pinAyyZ8QiogO7SiHokge9Q/iiQnw2NWT cQcHMH98W4zeKNS+47IYoMGxzCSpiHYwd2i/j3jnD+k+svYgCTWF86XWi9OYpKJPgt0W gGqwiO8/Lz2UkcEGPk67K44McNAvtYnA4sGaMFYkAVtSeB50reuhn7V+VLysmViRMCG7 eI9knFzu5330bbNGIkMridXfn6Y1fanOT/6VjMvdgNC7Wqg24zSlSfyknY1cgVk9cWqR UCOE/9RP9XDFw3rM8+0Yh33aF4Z1669VIIIYxk5F/wDCpuNPgVbDL7PHWVivL6t0N2bb YnWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=Zfu5uYQB7EldEQLqi+9Lt3YRP95ugqetP6chtNBuX1o=; b=qI2vwgCmZdFn+O6lujyEnSuyR1nGmk6e3rzJCjybJL7FT392/6CO4mWuiOQsPncUBJ ghKnqIaKXV6QhARMDKhgkM9fsMcbLvFLjzcqWggl4U1F+qqUtf70fcvBu8F+m0jTDwzx Riw/YGjSrmNMCNV63U8DYOWi95wRTEgeJStpYj/QM0Dn/FafWHiXrOUPF0OVClqXgBuT SUaVLurj9rsIx0v0Lv9SpxciiPYI1Sc8VskamRWTjYvGm1lWEyUHEtjDHUesrBzJItCa u16Lf+7IG83+pxw6gmj+eB64DYJfAw1Yl1UiENmmXUgnR4NaI7wI5QTuHypqZpG5dpgh d+OA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OtXfLuNd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b2-20020a6541c2000000b00478cc0a100asi41002422pgq.673.2023.01.19.13.34.41; Thu, 19 Jan 2023 13:34:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OtXfLuNd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230450AbjASVdc (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231127AbjASVax (ORCPT ); Thu, 19 Jan 2023 16:30:53 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C05EA7919; Thu, 19 Jan 2023 13:25:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163513; x=1705699513; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PzST9NKmNucCPEHUcN1NhrjLFoRDDlEf869nc/warJQ=; b=OtXfLuNdP+s5ABXvTSA6h3JcsJJ74kB46UQkIE55VBfg+4EvfX8ahcX+ Q9TK1f8noc1YveWVtuuaCCsFYiG/xqq/ftyG/fRz6+/TgppUtmXHy4rPd tZHAOqiqVRzBfQ3IxMYCnnCzeA582pY3Av5rGCiCAAP0lOqEvsTTiVCP9 i/9NtZHBWdtkTIOCWLodVMmO9b3a4+3Z5tLsQGyyIGc886E7dwckLZlhd 33LX4DTmAt0kfKkTTCDHzFR5AjlC5/NmUDcvNsBZpxKStDll3rACVkkqi Gvlhah6hq8Y8tGkEdAbrGCWsHzS1OM/egyN1sKDv73f/lCzRKYi0JDnwW g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119314" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119314" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:36 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139017" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139017" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:34 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Michael Kerrisk Subject: [PATCH v5 07/39] x86: Add user control-protection fault handler Date: Thu, 19 Jan 2023 13:22:45 -0800 Message-Id: <20230119212317.8324-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488288552254628?= X-GMAIL-MSGID: =?utf-8?q?1755488288552254628?= From: Yu-cheng Yu A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack. There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file cet.c, which is compiled in the case of either of the two CET features supported in the kernel: kernel IBT or user mode shadow stack. Move some static inline functions from traps.c into a header so they can be used in cet.c. Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it. Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable. The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Cc: Michael Kerrisk Reviewed-by: Kees Cook --- v5: - Move to separate file to advoid ifdeffery (Boris) - Improvements to commit log (Boris) - Rename control_protection_err (Boris) - Move comment from end of line in IBT fault handler (Boris) v3: - Shorten user/kernel #CP handler function names (peterz) - Restore CP_ENDBR check to kernel handler (peterz) - Utilize CONFIG_X86_CET (Kees) - Unify "unexpected" warnings (Andrew Cooper) - Use 2d array for error code chars (Andrew Cooper) - Add comment about why to read SSP MSR before enabling interrupts v2: - Integrate with kernel IBT fault handler - Update printed messages. (Dave) - Remove array_index_nospec() usage. (Dave) - Remove IBT messages. (Dave) - Add enclave error code bit processing it case it can get triggered somehow. - Add extra "unknown" in control_protection_err. v1: - Update static asserts for NSIGSEGV arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/traps.h | 12 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 152 +++++++++++++++++++++++ arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 2 +- arch/x86/kernel/traps.c | 87 ------------- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- include/uapi/asm-generic/siginfo.h | 3 +- 17 files changed, 186 insertions(+), 100 deletions(-) create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c index e07f359254c3..9a3c9de5ac5e 100644 --- a/arch/arm/kernel/signal.c +++ b/arch/arm/kernel/signal.c @@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index be279fd48248..4bced22213d5 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -1176,7 +1176,7 @@ void __init minsigstksz_setup(void) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c index 4700f8522d27..bbd542704730 100644 --- a/arch/arm64/kernel/signal32.c +++ b/arch/arm64/kernel/signal32.c @@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/sparc/kernel/signal32.c b/arch/sparc/kernel/signal32.c index dad38960d1a8..82da8a2d769d 100644 --- a/arch/sparc/kernel/signal32.c +++ b/arch/sparc/kernel/signal32.c @@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/sparc/kernel/signal_64.c b/arch/sparc/kernel/signal_64.c index 570e43e6fda5..b4e410976e0d 100644 --- a/arch/sparc/kernel/signal_64.c +++ b/arch/sparc/kernel/signal_64.c @@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 505f78ddca82..652e366b68a0 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -105,6 +105,12 @@ #define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) #endif +#ifdef CONFIG_X86_KERNEL_IBT +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -128,7 +134,7 @@ #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ DISABLE_ENQCMD) #define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 0 #define DISABLED_MASK20 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 21) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 72184b0b2219..69e26f48d027 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF, xenpv_exc_double_fault); #endif /* #CP */ -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_CP, exc_control_protection); #endif diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 47ecfff2c83d..75e0dabf0c45 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -47,4 +47,16 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, struct stack_info *info); #endif +static inline void cond_local_irq_enable(struct pt_regs *regs) +{ + if (regs->flags & X86_EFLAGS_IF) + local_irq_enable(); +} + +static inline void cond_local_irq_disable(struct pt_regs *regs) +{ + if (regs->flags & X86_EFLAGS_IF) + local_irq_disable(); +} + #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index dd61752f4c96..92446f1dedd7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -144,6 +144,8 @@ obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_CALL_THUNKS) += callthunks.o +obj-$(CONFIG_X86_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..33d7d119be26 --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include + +enum cp_error_code { + CP_EC = (1 << 15) - 1, + + CP_RET = 1, + CP_IRET = 2, + CP_ENDBR = 3, + CP_RSTRORSSP = 4, + CP_SETSSBSY = 5, + + CP_ENCL = 1 << 15, +}; + +static const char cp_err[][10] = { + [0] = "unknown", + [1] = "near ret", + [2] = "far/iret", + [3] = "endbranch", + [4] = "rstorssp", + [5] = "setssbsy", +}; + +static const char *cp_err_string(unsigned long error_code) +{ + unsigned int cpec = error_code & CP_EC; + + if (cpec >= ARRAY_SIZE(cp_err)) + cpec = 0; + return cp_err[cpec]; +} + +static void do_unexpected_cp(struct pt_regs *regs, unsigned long error_code) +{ + WARN_ONCE(1, "Unexpected %s #CP, error_code: %s\n", + user_mode(regs) ? "user mode" : "kernel mode", + cp_err_string(error_code)); +} + +static DEFINE_RATELIMIT_STATE(cpf_rate, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + +static void do_user_cp_fault(struct pt_regs *regs, unsigned long error_code) +{ + struct task_struct *tsk; + unsigned long ssp; + + /* + * An exception was just taken from userspace. Since interrupts are disabled + * here, no scheduling should have messed with the registers yet and they + * will be whatever is live in userspace. So read the SSP before enabling + * interrupts so locking the fpregs to do it later is not required. + */ + rdmsrl(MSR_IA32_PL3_SSP, ssp); + + cond_local_irq_enable(regs); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + /* Ratelimit to prevent log spamming. */ + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + __ratelimit(&cpf_rate)) { + pr_emerg("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, ssp, error_code, + cp_err_string(error_code), + error_code & CP_ENCL ? " in enclave" : ""); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); + } + + force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0); + cond_local_irq_disable(regs); +} + +static __ro_after_init bool ibt_fatal = true; + +/* code label defined in asm below */ +extern void ibt_selftest_ip(void); + +static void do_kernel_cp_fault(struct pt_regs *regs, unsigned long error_code) +{ + if ((error_code & CP_EC) != CP_ENDBR) { + do_unexpected_cp(regs, error_code); + return; + } + + if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) { + regs->ax = 0; + return; + } + + pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); + if (!ibt_fatal) { + printk(KERN_DEFAULT CUT_HERE); + __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); + return; + } + BUG(); +} + +/* Must be noinline to ensure uniqueness of ibt_selftest_ip. */ +noinline bool ibt_selftest(void) +{ + unsigned long ret; + + asm (" lea ibt_selftest_ip(%%rip), %%rax\n\t" + ANNOTATE_RETPOLINE_SAFE + " jmp *%%rax\n\t" + "ibt_selftest_ip:\n\t" + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR + " nop\n\t" + + : "=a" (ret) : : "memory"); + + return !ret; +} + +static int __init ibt_setup(char *str) +{ + if (!strcmp(str, "off")) + setup_clear_cpu_cap(X86_FEATURE_IBT); + + if (!strcmp(str, "warn")) + ibt_fatal = false; + + return 1; +} + +__setup("ibt=", ibt_setup); + +DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) +{ + if (user_mode(regs)) { + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + do_user_cp_fault(regs, error_code); + else + do_unexpected_cp(regs, error_code); + } else { + if (cpu_feature_enabled(X86_FEATURE_IBT)) + do_kernel_cp_fault(regs, error_code); + else + do_unexpected_cp(regs, error_code); + } +} diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index a58c6bc1cd68..5074b8420359 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = { ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE), #endif -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET INTG(X86_TRAP_CP, asm_exc_control_protection), #endif diff --git a/arch/x86/kernel/signal_32.c b/arch/x86/kernel/signal_32.c index 9027fc088f97..c12624bc82a3 100644 --- a/arch/x86/kernel/signal_32.c +++ b/arch/x86/kernel/signal_32.c @@ -402,7 +402,7 @@ int ia32_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/kernel/signal_64.c b/arch/x86/kernel/signal_64.c index 13a1e6083837..0e808c72bf7e 100644 --- a/arch/x86/kernel/signal_64.c +++ b/arch/x86/kernel/signal_64.c @@ -403,7 +403,7 @@ void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact) */ static_assert(NSIGILL == 11); static_assert(NSIGFPE == 15); -static_assert(NSIGSEGV == 9); +static_assert(NSIGSEGV == 10); static_assert(NSIGBUS == 5); static_assert(NSIGTRAP == 6); static_assert(NSIGCHLD == 6); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..18fb9d620824 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -77,18 +77,6 @@ DECLARE_BITMAP(system_vectors, NR_VECTORS); -static inline void cond_local_irq_enable(struct pt_regs *regs) -{ - if (regs->flags & X86_EFLAGS_IF) - local_irq_enable(); -} - -static inline void cond_local_irq_disable(struct pt_regs *regs) -{ - if (regs->flags & X86_EFLAGS_IF) - local_irq_disable(); -} - __always_inline int is_valid_bugaddr(unsigned long addr) { if (addr < TASK_SIZE_MAX) @@ -213,81 +201,6 @@ DEFINE_IDTENTRY(exc_overflow) do_error_trap(regs, 0, "overflow", X86_TRAP_OF, SIGSEGV, 0, NULL); } -#ifdef CONFIG_X86_KERNEL_IBT - -static __ro_after_init bool ibt_fatal = true; - -extern void ibt_selftest_ip(void); /* code label defined in asm below */ - -enum cp_error_code { - CP_EC = (1 << 15) - 1, - - CP_RET = 1, - CP_IRET = 2, - CP_ENDBR = 3, - CP_RSTRORSSP = 4, - CP_SETSSBSY = 5, - - CP_ENCL = 1 << 15, -}; - -DEFINE_IDTENTRY_ERRORCODE(exc_control_protection) -{ - if (!cpu_feature_enabled(X86_FEATURE_IBT)) { - pr_err("Unexpected #CP\n"); - BUG(); - } - - if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR)) - return; - - if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) { - regs->ax = 0; - return; - } - - pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); - if (!ibt_fatal) { - printk(KERN_DEFAULT CUT_HERE); - __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); - return; - } - BUG(); -} - -/* Must be noinline to ensure uniqueness of ibt_selftest_ip. */ -noinline bool ibt_selftest(void) -{ - unsigned long ret; - - asm (" lea ibt_selftest_ip(%%rip), %%rax\n\t" - ANNOTATE_RETPOLINE_SAFE - " jmp *%%rax\n\t" - "ibt_selftest_ip:\n\t" - UNWIND_HINT_FUNC - ANNOTATE_NOENDBR - " nop\n\t" - - : "=a" (ret) : : "memory"); - - return !ret; -} - -static int __init ibt_setup(char *str) -{ - if (!strcmp(str, "off")) - setup_clear_cpu_cap(X86_FEATURE_IBT); - - if (!strcmp(str, "warn")) - ibt_fatal = false; - - return 1; -} - -__setup("ibt=", ibt_setup); - -#endif /* CONFIG_X86_KERNEL_IBT */ - #ifdef CONFIG_X86_F00F_BUG void handle_invalid_op(struct pt_regs *regs) #else diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index bb59cc6ddb2d..9c29cd5393cc 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -640,7 +640,7 @@ static struct trap_array_entry trap_array[] = { TRAP_ENTRY(exc_coprocessor_error, false ), TRAP_ENTRY(exc_alignment_check, false ), TRAP_ENTRY(exc_simd_coprocessor_error, false ), -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET TRAP_ENTRY(exc_control_protection, false ), #endif }; diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S index 4a184f6e4e4d..7cdcb4ce6976 100644 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET xen_pv_trap asm_exc_control_protection #endif #ifdef CONFIG_X86_MCE diff --git a/include/uapi/asm-generic/siginfo.h b/include/uapi/asm-generic/siginfo.h index ffbe4cec9f32..0f52d0ac47c5 100644 --- a/include/uapi/asm-generic/siginfo.h +++ b/include/uapi/asm-generic/siginfo.h @@ -242,7 +242,8 @@ typedef struct siginfo { #define SEGV_ADIPERR 7 /* Precise MCD exception */ #define SEGV_MTEAERR 8 /* Asynchronous ARM MTE error */ #define SEGV_MTESERR 9 /* Synchronous ARM MTE exception */ -#define NSIGSEGV 9 +#define SEGV_CPERR 10 /* Control protection fault */ +#define NSIGSEGV 10 /* * SIGBUS si_codes From patchwork Thu Jan 19 21:22:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45981 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555587wrn; Thu, 19 Jan 2023 13:34:36 -0800 (PST) X-Google-Smtp-Source: AMrXdXsvyYFF+UOjMmM3URyzXs6IuxT7snYkwmlOdIrbkdT8NSiQVfPgVbra2A0TudSvwOoa/J6S X-Received: by 2002:a17:902:7601:b0:193:236:3a2b with SMTP id k1-20020a170902760100b0019302363a2bmr10998289pll.3.1674164075774; Thu, 19 Jan 2023 13:34:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164075; cv=none; d=google.com; s=arc-20160816; b=zdJNufrs1bW2Y8f7QphawuRGD+iIE61f95yyB8jfNNyUIlCUGkkNazEWAdD5eR37O2 CYHehXFCa314sq7Sd8Rw+DUZGNkDn8y25jQOmTEZ6rX8jE1RMtaLK2OrazFhMLJGORr6 W7HqeqyRsTbgoaZXXSnofwdctKLTdPJqUjPoL1MIAvLGicnawO882OsQW20jIRJilRD9 AUYNx6gY6I/fvYFXknycnlofO2ZAkYi8eAYkN72mzrYs9GbYa6LHeEy0m7nKN/cLLAA1 jdq4IF1ttuxQ4U4HLuXFy6cmoocJYj+/XVX9hp0YilIb9LLFmYjvc8SbJvVRmOMVbcM+ vEiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=BVe42IZ3bUVWxOXb5SeOdBeUgISVBwg/EiJcXbj5PIQ=; b=ca4Y77rJr9pCpinxBHKEedFWaAkK0GAyZA0SU7uF9XKn74wPjqXNeEprOSHjpMvZjf RnsIkw8w4fXJVSTE+i+GjQ/o7JsqlAyKDosVNJGnOnhlml2QOJrozlKZimFPVel1L8x5 Nf4sE5pfmjIDOnhNWEqF6bk2TkrIXn3/UJneZLqrv9XzlxSsjrKtf58UJR+hfhyyJpdK SjUomVLGdLEZyFSy83K/UvKvyqaVobWyN69E73TG/oBUBfzVePejaj43ELd2xouUHp2j 7uoYBcu9wMW0+GrsSIP6NWQ7jHUItwSdsua1UMXO7JswXpf8Wa4m6wVT1RciOa7AH2vT 0PWg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=K9anBuZf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ba9-20020a170902720900b00192972afbc5si38844158plb.459.2023.01.19.13.34.23; Thu, 19 Jan 2023 13:34:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=K9anBuZf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230260AbjASVdA (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231207AbjASVbN (ORCPT ); Thu, 19 Jan 2023 16:31:13 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EBC0A7930; Thu, 19 Jan 2023 13:25:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163532; x=1705699532; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=3Wysy2UxXgn7XrxNNcTO6OHZIBa9HpH74QVE6RG86iA=; b=K9anBuZfDLYjz7R1vDNkH5UsUFaApob41Pz4+TG6oa6u1DXPC/qwkeWI er7Y8kmQsmkzoaLVWaH0QTTAMeAA0ee3o14dCLdU6owfDlxtl2McQafxh SpkJWqOiFve3Gic/Zp5JNtxHesXokF3GGirwFe6qZW0A2/8n/WApd5y7P I0yEWGhSyjjOzB5f7dfYY6ovPLn9sk9yY+a8gOGSpUUUsyZsYzypTu5E3 /4s4zVGQae6TKvY4rXKVqUyE53X8XLga4EK0fmULdUYWr6X3CuDBz5qOW Ba2nH9pqnzkF4ioIuH0lbZ41OJJw0g4Qtl+h4J4Yna8i+1RD0th4m2vC+ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119341" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119341" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:37 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139021" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139021" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:36 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Christoph Hellwig Subject: [PATCH v5 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages Date: Thu, 19 Jan 2023 13:22:46 -0800 Message-Id: <20230119212317.8324-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488270149892573?= X-GMAIL-MSGID: =?utf-8?q?1755488270149892573?= From: Yu-cheng Yu New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages. In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory. When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: "H. Peter Anvin" Cc: Kees Cook Cc: Thomas Gleixner Cc: Dave Hansen Cc: Christoph Hellwig Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Borislav Petkov Cc: Peter Zijlstra Reviewed-by: Kees Cook --- v5: - Spelling and grammar in commit log (Boris) v3: - Update commit log (Andrew Cooper, Peterz) v2: - Normalize PTE bit descriptions between patches arch/x86/include/asm/pgtable_types.h | 6 +++--- arch/x86/mm/pat/set_memory.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 447d4bee25c4..0646ad00178b 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -192,10 +192,10 @@ enum page_cache_mode { #define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) #define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0) #define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) -#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0|___D| 0|___G) +#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) +#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) -#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_LARGE (__PP|__RW| 0|___A|__NX|___D|_PSE|___G) #define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW| 0|___A| 0|___D|_PSE|___G) #define __PAGE_KERNEL_WP (__PP|__RW| 0|___A|__NX|___D| 0|___G| __WP) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 356758b7d4b4..d41706ad29db 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -2073,7 +2073,7 @@ int set_memory_nx(unsigned long addr, int numpages) int set_memory_ro(unsigned long addr, int numpages) { - return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0); + return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW | _PAGE_DIRTY), 0); } int set_memory_rox(unsigned long addr, int numpages) From patchwork Thu Jan 19 21:22:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45984 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555680wrn; Thu, 19 Jan 2023 13:34:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXs5Yvl4wzufzAZa0v6z9vtUTlIgKr9KxbLqXVl6uSg0rJ7KTzUl0m3h/jeMzLDXUC+jquYX X-Received: by 2002:a05:6a20:b282:b0:a3:a90f:5326 with SMTP id ei2-20020a056a20b28200b000a3a90f5326mr11511400pzb.61.1674164090761; Thu, 19 Jan 2023 13:34:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164090; cv=none; d=google.com; s=arc-20160816; b=RXOak9Lga5uM22W4hJMDeQrqKtYB2LlEd3gLtWwnJXKjirHEjhytL/26kkrIadETK5 jXjXwOopRsvX8KRT9+TlJpPN8B50yCneJQB44LrdV3bCikrtEck00PdSOyDVy5MfokVM pEuCvdi5l/zcrjwRKshYXBijQ9POc9iJxKlN/Y86icwsrPvEU7Ra7Pk2LUGDYENjFWSa doTa2zYO3nHvEiJE7aMr/MUqfbTI2K7wlpndBJSiXAVe4T5UnjndA9mPbbT86sTEmwFq OlVvIg2v1MEOE+FZvbW2B2Bg5EmMWOXxTAWqmCoxF0EsR2ac6/5ESfhdNJiaJjoGfdg8 tY2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=SLtGihyn/x9FzqL6Gl9syvqGmcYAMPQwgzBEC4XQqQQ=; b=zWMPIYhRFsDPrUX5QLSRI+RqdX87HUdBjuENAcWrFXlGqoX5optaTs8lQkNGWn6l40 RUjw6Kq/NYSDYWR03cckc31N97gdBE6u9Kmbdvt9fdv1YwImXm4P8RGDtOScUrQ3XNxa pWky0BDcWXJ03+7rNhBsNBxfvdYiTVZXLzd29ohVZJN0zgYd2r0BfQZPDO0yh41p9YgR SdDSzHPYP0kg0Pw2WqUlnh5fzPEQD2XG1m1UqE8wx0K2OF0cXysNyOAjFtCyB9CAdbBs NTvVukFlxjV+5ID9uXpQ8Jna0++EwJlwOxOVECWDYq+enIsDNRLnUxmfzyngoxmTX7jU vRzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OUOshJsK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 16-20020a631250000000b004c9b3574f66si11564216pgs.62.2023.01.19.13.34.34; Thu, 19 Jan 2023 13:34:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OUOshJsK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230310AbjASVdU (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231210AbjASVbO (ORCPT ); Thu, 19 Jan 2023 16:31:14 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C6D6A7934; Thu, 19 Jan 2023 13:25:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163533; x=1705699533; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=tmt4cSd6A0kDS/7f4GwHJfynyFRJ89yUY8hbbxiN2tQ=; b=OUOshJsKRQlVruHXtVt5HbRVoMPr/ovawrjdFmalLu2IAWaByZXJMSbj jy+iAXt2ZYFsTC6yBbeAUU0lnW3IlUP4/T2KAF8sSau6BHGQhW7AE4Cop Md4ik5uSY4NZdKeKPqTA/k62UfSZar8G2Xc+kAAOXYF4WG5XA4hVSg9tK uk4LbCzlClfvfdz5t8oDQ5CgMCILTNH2zxbtUqn19MF3+YCI9JN+HF/pz n7lovzJVGKk2H/JP30cT0D0qz3mbjFoQhbgxru7qd9UzzsThTKhirRElG yG2qF+0hqy+8Fla+tHmL7pgua+d0HlrbFcI2CPJOFw6oy9ZX5UDwxYyiP w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119364" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119364" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:39 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139027" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139027" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:37 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 09/39] x86/mm: Move pmd_write(), pud_write() up in the file Date: Thu, 19 Jan 2023 13:22:47 -0800 Message-Id: <20230119212317.8324-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488285666034753?= X-GMAIL-MSGID: =?utf-8?q?1755488285666034753?= From: Yu-cheng Yu To prepare the introduction of _PAGE_COW, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes. Tested-by: Pengfei Xu Tested-by: John Allen Reviewed-by: Kees Cook Signed-off-by: Yu-cheng Yu Reviewed-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 0564edd24ffb..b39f16c0d507 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -160,6 +160,18 @@ static inline int pte_write(pte_t pte) return pte_flags(pte) & _PAGE_RW; } +#define pmd_write pmd_write +static inline int pmd_write(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_RW; +} + +#define pud_write pud_write +static inline int pud_write(pud_t pud) +{ + return pud_flags(pud) & _PAGE_RW; +} + static inline int pte_huge(pte_t pte) { return pte_flags(pte) & _PAGE_PSE; @@ -1120,12 +1132,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); -#define pmd_write pmd_write -static inline int pmd_write(pmd_t pmd) -{ - return pmd_flags(pmd) & _PAGE_RW; -} - #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); } -#define pud_write pud_write -static inline int pud_write(pud_t pud) -{ - return pud_flags(pud) & _PAGE_RW; -} - #ifndef pmdp_establish #define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma, From patchwork Thu Jan 19 21:22:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45993 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556277wrn; Thu, 19 Jan 2023 13:36:34 -0800 (PST) X-Google-Smtp-Source: AMrXdXse+VLo+P3mXk8zf1l8X59u2qflmeyfEwZmDDUyk35GYco3x4ndmuRI0/TBKihr9lyRta3u X-Received: by 2002:a17:902:8d94:b0:192:d9dd:167d with SMTP id v20-20020a1709028d9400b00192d9dd167dmr11383945plo.43.1674164194378; Thu, 19 Jan 2023 13:36:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164194; cv=none; d=google.com; s=arc-20160816; b=yMOi0cM+FcjdzmfH/DLKy80eXuc7I0JpJzJKFQIkEQ3gDi8w4fif+a2nraCwT60o2r JAptzmiREkI4DTKBSt8NLtP3o6Jj61U/y9QW+2H7YBTqqqo2NyiIXa7VepfI6Pg+2h8K eewfQjedmiPBtYi2tnhGzDTj/tilcq7EmFQxhwq1zhPFRKXxlt6XVcdI6b+SjojY7uhf spM2XNy/OiB/bjPCXo6B5Qs7Pq7yANn6n4u3Zzg+bncygnGv3yzdHA0xScgCH+PdvMqs H8yNXSL6X4iqVQfS6DOF7/8NsuD60zq35kRd2hWtHi2I9D/EEKbcLqL1qM8eGUQMELea 5SeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=gg1qCyi+VRVbGSEkEF0teOFXegG4SQhXxychn9nztMQ=; b=0dcppTuQzywX3G+iv9qPKGr3yA5wX7o+phg10jChznKKGCTRlwZR2mtBOrIzVkmvA5 pxriELQOjaPFMBr3BdAcz/JDl1P9lkbYKavITsdIvbRgD5fOu+PAa1guccVad09qRGVK Q1WsIS3KQHAi4WlS2a4H1oL/cNwc+E7NbAokPaukv4zPwLtungH4D8q4DZwCaVSzOni/ mx32aS0PKEyuBGGFiy3SbNg6Uo2DHdm2ltyxEs289c8t9+auqlMhFJ5Mnb5R/y2G1cr1 /PokmrYLXPCVsxQ2jU1zM4rjwoOBGFjbHSwibuHiyoWFS+Bo1X6hDtNJoGvoXRPEWgHd 3hIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aRnOtZyM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a13-20020a170902eccd00b00186c5eb0d48si24782948plh.425.2023.01.19.13.36.21; Thu, 19 Jan 2023 13:36:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aRnOtZyM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230350AbjASVdt (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231224AbjASVbQ (ORCPT ); Thu, 19 Jan 2023 16:31:16 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A87BA2977; Thu, 19 Jan 2023 13:25:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163540; x=1705699540; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=FFlwLGFrc3HasCEVoFiq4z4jSgXcdwaUjJhlr8Fvg+4=; b=aRnOtZyMko5ae+FJxv4iVnVL/0VoeR40dnzCJ5SFkI3FMmrUprwC7AQf Dud+YvZWnqA92Z8DId+I28Jtw6gusuFHAp8AAtkxhRd69Em5lrnJ63iW1 XWyukqttdXpCb156zgHveQi797Pc4pJsuZi1E+T2HK1RZN49NHqnH2WCN BAt5qYl+Au7Mmd8Lj5VLR02xLLmU2wDhdNcbf6HEWrMB4L4AXbZOKYLv3 +Unv2ewsKqefz1mlpexTwKK66Q9UF2mCOeuCgcTGJcc18iBrcB3F8e0oc nvGqB08B7RJVGjrni/9ViYP7i6u/VHg3VUxl9x0Ska2LJp6EvYre9THH+ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119389" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119389" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:41 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139032" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139032" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:39 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW Date: Thu, 19 Jan 2023 13:22:48 -0800 Message-Id: <20230119212317.8324-11-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488394602828467?= X-GMAIL-MSGID: =?utf-8?q?1755488394602828467?= Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack. The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity. So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory: (Write=0,Cow=1,Dirty=0): - A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW via fork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This happens in copy_present_pte(). - A R/O page that has been COW'ed. The user page is in a R/O VMA, and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1. - A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write. (Write=0,Cow=0,Dirty=1): - A shadow stack PTE. - A Cow PTE created when a processor without shadow stack support set Dirty=1. There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there. Implement only the infrastructure for _PAGE_COW. Changes to start creating _PAGE_COW PTEs will follow once other pieces are in place. Tested-by: Pengfei Xu Tested-by: John Allen Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix log, comments and whitespace (Boris) - Remove capitalization on shadow stack (Boris) v4: - Teach pte_flags_need_flush() about _PAGE_COW bit - Break apart patch for better bisectability v3: - Add comment around _PAGE_TABLE in response to comment from (Andrew Cooper) - Check for PSE in pmd_shstk (Andrew Cooper) - Get to the point quicker in commit log (Andrew Cooper) - Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz) - Fix comment that implied dirty bit for COW was a specific x86 thing (peterz) - Fix swapping of Write/Dirty (PeterZ) v2: - Update commit log with comments (Dave Hansen) - Add comments in code to explain pte modification code better (Dave) - Clarify info on the meaning of various Write,Cow,Dirty combinations arch/x86/include/asm/pgtable.h | 78 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 134 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index b39f16c0d507..6d2f612c04b5 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -301,6 +301,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +/* + * Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case + * of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the + * Dirty=1,Write=0 will result in the memory being treated as shadow stack + * by the HW. So when creating COW memory, a software bit is used + * _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow() + * take a PTE marked conventionally COW (Dirty=1) and transition it to the + * shadow stack compatible version of COW (Cow=1). + */ +static inline pte_t pte_mkcow(pte_t pte) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pte; + + pte = pte_clear_flags(pte, _PAGE_DIRTY); + return pte_set_flags(pte, _PAGE_COW); +} + +static inline pte_t pte_clear_cow(pte_t pte) +{ + /* + * _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK kernels, since + * the HW dirty bit can be used without creating shadow stack memory. + * See the _PAGE_COW definition for more details. + */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pte; + + /* + * PTE is getting copied-on-write, so it will be dirtied + * if writable, or made shadow stack if shadow stack and + * being copied on access. Set the dirty bit for both + * cases. + */ + pte = pte_set_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_COW); +} + #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp(pte_t pte) { @@ -413,6 +451,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_mkcow(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pmd; + + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_set_flags(pmd, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_clear_cow(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pmd; + + pmd = pmd_set_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_COW); +} + #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp(pmd_t pmd) { @@ -484,6 +542,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pud_t pud_mkcow(pud_t pud) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pud; + + pud = pud_clear_flags(pud, _PAGE_DIRTY); + return pud_set_flags(pud, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pud_t pud_clear_cow(pud_t pud) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return pud; + + pud = pud_set_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_COW); +} + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 0646ad00178b..5c3f942865d9 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -21,7 +21,8 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ -#define _PAGE_BIT_SOFTW4 58 /* available for programmer */ +#define _PAGE_BIT_SOFTW4 57 /* available for programmer */ +#define _PAGE_BIT_SOFTW5 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ #define _PAGE_BIT_PKEY_BIT2 61 /* Protection Keys, bit 3/4 */ @@ -34,6 +35,15 @@ #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +/* + * Indicates a copy-on-write page. + */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_BIT_COW _PAGE_BIT_SOFTW5 /* copy-on-write */ +#else +#define _PAGE_BIT_COW 0 +#endif + /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ #define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL @@ -117,6 +127,40 @@ #define _PAGE_SOFTW4 (_AT(pteval_t, 0)) #endif +/* + * The hardware requires shadow stack to be read-only and Dirty. + * _PAGE_COW is a software-only bit used to separate copy-on-write PTEs + * from shadow stack PTEs: + * + * (Write=0,Cow=1,Dirty=0): + * - A modified, copy-on-write (COW) page. Previously when a typical + * anonymous writable mapping was made COW via fork(), the kernel would + * mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This + * happens in copy_present_pte(). + * - A R/O page that has been COW'ed. The user page is in a R/O VMA, + * and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault + * handler creates a copy of the page and sets the new copy's PTE as + * Write=0 and Cow=1. + * - A shared shadow stack PTE. When a shadow stack page is being shared + * among processes (this happens at fork()), its PTE is made Dirty=0, so + * the next shadow stack access causes a fault, and the page is + * duplicated and Dirty=1 is set again. This is the COW equivalent for + * shadow stack pages, even though it's copy-on-access rather than + * copy-on-write. + * + * (Write=0,Cow=0,Dirty=1): + * - A shadow stack PTE. + * - A Cow PTE created when a processor without shadow stack support set + * Dirty=1. + */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_COW (_AT(pteval_t, 1) << _PAGE_BIT_COW) +#else +#define _PAGE_COW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY | _PAGE_COW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) /* @@ -186,12 +230,17 @@ enum page_cache_mode { #define PAGE_READONLY __pg(__PP| 0|_USR|___A|__NX| 0| 0| 0) #define PAGE_READONLY_EXEC __pg(__PP| 0|_USR|___A| 0| 0| 0| 0) -#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) -#define _KERNPG_TABLE_NOENC (__PP|__RW| 0|___A| 0|___D| 0| 0) -#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) +/* + * Page tables needs to have Write=1 in order for any lower PTEs to be + * writable. This includes shadow stack memory (Write=0, Dirty=1) + */ #define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0) #define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) +#define _KERNPG_TABLE_NOENC (__PP|__RW| 0|___A| 0|___D| 0| 0) +#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC) + +#define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) #define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..9429da70d689 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -273,7 +273,8 @@ static inline bool pte_flags_need_flush(unsigned long oldflags, const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED; const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | - _PAGE_SOFTW3 | _PAGE_SOFTW4; + _PAGE_SOFTW3 | _PAGE_SOFTW4 | + _PAGE_COW; const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT | _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT | _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 | From patchwork Thu Jan 19 21:22:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45988 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555902wrn; Thu, 19 Jan 2023 13:35:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXta87p8SoCOJVxEhHTN76lUlwuTwn1yvXvH95KgbmrbpndbnioFWy0B61lrfS+QFNQsswMF X-Received: by 2002:a62:870f:0:b0:582:bb80:58d7 with SMTP id i15-20020a62870f000000b00582bb8058d7mr35287030pfe.26.1674164130562; Thu, 19 Jan 2023 13:35:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164130; cv=none; d=google.com; s=arc-20160816; b=ar+dIMdSSGWDl4nEkar0Y6gNFFrvYJmvogGjORsIw8y64JpEntVFIeJxwdmPFd8Ylh rYFEnjJG3OVC19pV8UTylT/AwdO/0nHBinemPAVMdnA0NLT7QryRHexuNx8YElIjyBvG lyHwORQ10H+DfEo0JHuuafJRW5gZWcQTehWRQtDqWyBycpHoQU0ukCDJtItLXCPBLZTg 2JPabIMbMZIGlpMus4ISbM63oWgUPKghBAqZTunknQALpn0gmXhNbGPDURO9nZFgUmes EiCcGi3LcK8fP4S1DYogs3UmKaXlHmmuwBO9Orat4BVh36Dw+SMoJBcq9JQWc2XcPs5U kWSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=raQIy6Fe766NUbCocQ4XfRzcFOdeBSgY+sYH+kVPZqU=; b=SMWixnFrPMcq+jk2d5jkVx2ECLr+NnAxylaZvBIEUY7A+nEG6i1GCgGCJRBkQDUMr/ 6CvGWWlDcq4RUepCvmCU+8t4OImZLurkYVGZFEQd7aBvAND3DNzG9t+SD/uTn4mDhaGJ ZYMU+uLqHJqNfGSHHkLUZcvoNrKW3o9opF3sjLA1eSOmJXzl9XoHi+Yzj/npPWzaKEDE F0Hlc+n5OM5xcxmTOiYJd8+AuX8O/fDgNH4dvKXUayNfSw2d7wVDtXxlzsspVbuE0dRt o9Cfsm+DGTjbhVeAEU0o3PDegHyjwdAsm4yQSYdmat43By4GRTXQ3k1gnW7QccK1Yj+S ofcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=duB0BEml; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d19-20020a056a00199300b00575757288e4si12927932pfl.90.2023.01.19.13.35.18; Thu, 19 Jan 2023 13:35:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=duB0BEml; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230467AbjASVd7 (ORCPT + 99 others); Thu, 19 Jan 2023 16:33:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229685AbjASVb1 (ORCPT ); Thu, 19 Jan 2023 16:31:27 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 616ACA8393; Thu, 19 Jan 2023 13:25:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163552; x=1705699552; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0xRZR4lTYh6NOOM79Ggd51EdLXNXPAUjGASZ+kAeGKM=; b=duB0BEmlqw+ygPNkvMxSD1o+dnTZME0CnvbBTESXHyjxDrbbKpuGv4am lBKW4UPNwndUCTNGsrlGsEQNmKmAOwOJIWBP8RrPD+brqhcs1zc+tDurJ BuQkU67vPKPOeV1jIHSkWjJy+cLdLcEmc55g2vaVuk0hqnZpdxVwMAl6k 3xMdThUepL4voIWiXJjeSOb/4v/hSnV3lmRuk53yS4i5kK+bCd9uMIJ3f gZofpixT+VuWu3ryIhCgEJ0lDk6lS+E5zT5XTqPNX7V73vP6t9ANk9Tp/ Ctk17W3o+3KqGbzmxo0FF6aQS4NWDTbTCz8QoYIXSGz/chn+yoWYCfNLa A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119409" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119409" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:42 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139038" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139038" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:41 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW Date: Thu, 19 Jan 2023 13:22:49 -0800 Message-Id: <20230119212317.8324-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488327225099092?= X-GMAIL-MSGID: =?utf-8?q?1755488327225099092?= From: Yu-cheng Yu The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this. pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one. However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Do this by using the pte_mkdirty() helper. Since pte_mkdirty() also sets the soft dirty bit, extract a helper that optionally doesn't set _PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move _PAGE_DIRTY to _PAGE_COW can live in one place. Apply the same changes to pmd_modify(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix pte_modify() again, to not lose _PAGE_DIRTY, but still not set _PAGE_SOFT_DIRTY as was fixed in v4. v4: - Fix an issue in soft-dirty test, where pte_modify() would detect _PAGE_COW in pte_dirty() and set the soft dirty bit in pte_mkdirty(). v2: - Update commit log with text and suggestions from (Dave Hansen) - Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen) arch/x86/include/asm/pgtable.h | 64 +++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 6d2f612c04b5..7942eff2af50 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -392,9 +392,19 @@ static inline pte_t pte_mkexec(pte_t pte) return pte_clear_flags(pte, _PAGE_NX); } +static inline pte_t __pte_mkdirty(pte_t pte, bool soft) +{ + pteval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pte_set_flags(pte, dirty); +} + static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pte_mkdirty(pte, true); } static inline pte_t pte_mkyoung(pte_t pte) @@ -503,9 +513,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return pmd_clear_flags(pmd, _PAGE_RW); } +static inline pmd_t __pmd_mkdirty(pmd_t pmd, bool soft) +{ + pmdval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pmd_set_flags(pmd, dirty); +} + static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pmd_mkdirty(pmd, true); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -715,26 +735,54 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { + pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY; pteval_t val = pte_val(pte), oldval = val; + pte_t pte_result; /* * Chop off the NX bit (if present), and add the NX portion of * the newprot (if present): */ - val &= _PAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; + val &= _page_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); - return __pte(val); + + pte_result = __pte(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pte_mkdirty() will do this in + * the case of shadow stack. + */ + if (pte_dirty(pte)) + pte_result = __pte_mkdirty(pte_result, false); + + return pte_result; } static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { + pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY; pmdval_t val = pmd_val(pmd), oldval = val; + pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val &= _hpage_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); + + pmd_result = __pmd(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pmd_mkdirty() will do this in + * the case of shadow stack. + */ + if (pmd_dirty(pmd)) + pmd_result = __pmd_mkdirty(pmd_result, false); + + return pmd_result; } /* From patchwork Thu Jan 19 21:22:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45989 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555950wrn; Thu, 19 Jan 2023 13:35:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXsmSGWkhUqhBjgHirq05ioFrlq1VsXTrRMM5S2YDETcf5iAhzWVSJvtwN8GqZCDdKOQdP4t X-Received: by 2002:a17:902:8688:b0:193:19c3:4915 with SMTP id g8-20020a170902868800b0019319c34915mr11292262plo.67.1674164137766; Thu, 19 Jan 2023 13:35:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164137; cv=none; d=google.com; s=arc-20160816; b=PNXE/9go4zN5vS4P35kV246GcqRURIh+6OhnhsTYCa5kx5bGD7dPvo/rpb8YqIzREi aju36OUg9j9qAa9pMp+4CZTysLCImHid/K32ZFFIaBh3v37OYUYorKK/618dyR5MD812 8e0Tiw+TKhlITgzy619pccizFD08r21jEAw7US4hLkYx2LyUOdASHEA/n6b88m6X2WRP dTSkG+YDoe2aa6BUaGDwkPXznUQbyfsl5MIIT6QM4a80KO4h37oLhrPjdnc9uGu4qeEF vOcvmIgeCKH8f3tJ2fPjcw4DN8xOS3fcTz4umwgybEAPMDrrddpVhy1NbsQ/2qbXgOuO jlaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=qRKDNK2oMQO5oOmk+unpG3nEq+k+7RxaZmTu69DQeXE=; b=nYMiFgMDFCKhxRRrhE8f+szefI5zENzk/TnMSeyuEPlgAWBk8Hz0m/QJhd6KBxA6zB zA6FMNMMzs3vQwcFUlEXwMBPXkAJUvAnvJQGH2tETedg3yx7voUykmM2W3V1adV0RgPh NasNcD4DT4tXyOqBg+bihpnmsLM+5hIszgm3Ob5CkTdhSeklLPC7/mci6MrrzrzGwULA 5ei6bOpusbaMKrStO/A5HKGkRkhJ4CCoyfeRJpinVe5sH3OS2j4T5bcUvEhOxR71ri6h f0ig6cov0T3Fka5dcBZz73DfzvGTqhufOAN5j29ELYG+LVn0ksTIEC4kwqJs46+o3d0+ N18g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fz7WkrmJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ay8-20020a1709028b8800b00191457548c0si35386029plb.507.2023.01.19.13.35.25; Thu, 19 Jan 2023 13:35:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fz7WkrmJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229834AbjASVeJ (ORCPT + 99 others); Thu, 19 Jan 2023 16:34:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbjASVba (ORCPT ); Thu, 19 Jan 2023 16:31:30 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 403A24B4A8; Thu, 19 Jan 2023 13:25:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163557; x=1705699557; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=BvuOTjZc5NHm9ltMy7F/33kG5ApCivtjPVo72HaZTJE=; b=fz7WkrmJtAuTmeLsTlW4joHvSBAe0DFMq1wvuF7i2ZHA4+9tNpdQI+Hi IbyBliCQxo6m+tsJQNOQVvJTHjMNzbhG2owSPD9JmwHv5juHXlR+lLO1R arxEaRlO6mvpbRgklr8xZgqXiMLbUe0frBhnkahBKvS1FB8Dod/x+0sNn eN+JUGkv04PV/1AL12Fu84KTI0J2+mTnGTxNi5KMlSy30DziPNOwZT3VO f1TQdIA46qokiZWLeN81athol3JyJjMJGH3pgC4DKgGJA1U9Zx6Fvuxdv gIweIq0HE0G033EFZKbSeLP75CnyfwZb4cSwP9fivGjehkDrM7TTLYtZI A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119431" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119431" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:44 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139043" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139043" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:42 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW Date: Thu, 19 Jan 2023 13:22:50 -0800 Message-Id: <20230119212317.8324-13-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488334676704360?= X-GMAIL-MSGID: =?utf-8?q?1755488334676704360?= From: Yu-cheng Yu When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,Cow=1. When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could become a transient shadow stack PTE in two cases: 1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary. 2. When _PAGE_DIRTY is replaced with _PAGE_COW non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg. In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature. Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Commit log verbiage and formatting (Boris) - Remove capitalization on shadow stack (Boris) - Fix i386 warning on recent clang v3: - Remove unnecessary #ifdef (Dave Hansen) v2: - Compile out some code due to clang build error - Clarify commit log (dhansen) - Normalize PTE bit descriptions between patches (dhansen) - Update comment with text from (dhansen) Yu-cheng v30: - Replace (pmdval_t) cast with CONFIG_PGTABLE_LEVELES > 2 (Borislav Petkov). arch/x86/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 7942eff2af50..c5047eb5f406 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1232,6 +1232,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pte_t old_pte, new_pte; + + old_pte = READ_ONCE(*ptep); + do { + new_pte = pte_wrprotect(old_pte); + } while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte)); + + return; + } +#endif clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); } @@ -1284,6 +1301,26 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * If shadow stack is enabled, pmd_wrprotect() moves _PAGE_DIRTY + * to _PAGE_COW (see comments at pmd_wrprotect()). + * When a thread reads a RW=1, Dirty=0 PMD and before changing it + * to RW=0, Dirty=0, another thread could have written to the page + * and the PMD is RW=1, Dirty=1 now. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pmd_t old_pmd, new_pmd; + + old_pmd = READ_ONCE(*pmdp); + do { + new_pmd = pmd_wrprotect(old_pmd); + } while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + + return; + } +#endif + clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); } From patchwork Thu Jan 19 21:22:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45991 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556062wrn; Thu, 19 Jan 2023 13:35:56 -0800 (PST) X-Google-Smtp-Source: AMrXdXvJ9Zi5/Sbn/mY0TogsWJl4YBAVl+IqmJTg//NBqJqj/Wilofv64cWaPTIcYCEbBDxj3x8O X-Received: by 2002:a17:90a:46c8:b0:227:1e67:d58f with SMTP id x8-20020a17090a46c800b002271e67d58fmr36357105pjg.12.1674164156482; Thu, 19 Jan 2023 13:35:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164156; cv=none; d=google.com; s=arc-20160816; b=x/q7hkWaQ6iB/No4V3dHPaM0W2q+R9r4Z/gQHAoSwNPGG9OzF47smj9IK523ZvnVSG PNSVWtGG3XdcmD3URMCHrP8qx+z4cvXm9c6PzbDqRSTEa9LNSnO8AEq9AbAA70Hiviwq k1ixU8ZgZOC+A/wjZfRK5ZBfDnFmhip6SWiP1ejUJkwbRlFIWZbpEQnMrmoIQWwU5f3i 7tTxi4ViNYBsGYIYUVrIquHS5bbc9tyWavpqgi/Fs37F1QT1mB1JQyTpdL/dHjsWMQZH 3C1NQ8Gi5h/E/eNeisqIVe2PKHQzhzcK+XR6TDm4OIngyRBki4Q1RLiJ2r2DFGPTJUHP WoJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=e2Tm+WaYXmy3ls6tYjrnsqh0ZVC1D7WOAXojsWXtuCw=; b=HO5yMjBURh2ET93pZockE6ACm7V4ly5c4WYX1YwZTFM2cxbauCdXqU1a6K1t6Q4hI2 YXH9mUMmCNeVU9EuVPLHTXUlrEiAPz6a5udBzaj8kOCHqdr+y5QzppdrTG2oUsNA6wQI dCCCTj17JS7bUW+4DbaF3xUZ3PG/cyU7W8h4Mn9jOmF9iw+mkQatsA0nnjPcJSjvcn54 2C2fUSGtNvrPmqRebgRgvwRigHYVZ46aiA22nrPRqKact3KGE8OtqRy9981D92s9PeRc fqrlJpopLJWZ6DkMg1J6kPqzYVs6wLIu4Sr9cUJU9qKsUi6LJgBUsluWao1q02iNLLIt I7UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lzfC46M9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w5-20020a17090abc0500b002199a16366csi261165pjr.173.2023.01.19.13.35.43; Thu, 19 Jan 2023 13:35:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lzfC46M9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230501AbjASVew (ORCPT + 99 others); Thu, 19 Jan 2023 16:34:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230222AbjASVco (ORCPT ); Thu, 19 Jan 2023 16:32:44 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C765FA958A; Thu, 19 Jan 2023 13:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163581; x=1705699581; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Wk+41wU25mzgjYiTB7ECzyixSrgPZ+gatfisdIuQPw8=; b=lzfC46M9xHUl6OnbO52fbYafeK348F7gNdfgBOPY6lMt87watvaosTCt JwqOzs9Pc1kWQavGKqk6O9Gr1sCHAW7ZSyE1yT+koTAdnFAN+mixdzIbK OV2SotEU085F/H8I5mH5ZnmVJEUGBcHzn6mqXtPJkrJils/A36WV1R2py lVE/pNa1wnXPd8eCr54tA1TAzdoj2xPxa52MZ9uonG/76Y7rjLtYlQIra MMQRrF9XOJJZg2cRRp9t7ELyQKXeO5bs29lCHKMrrtHHirIL7mr5zkef8 ++vghGjKc5zoh/7wBIEaZdonkXjmvmlzrKFJo5+HrGXslmeZXKceEsM8R w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119455" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119455" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:46 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139047" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139047" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:44 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 13/39] x86/mm: Start actually marking _PAGE_COW Date: Thu, 19 Jan 2023 13:22:51 -0800 Message-Id: <20230119212317.8324-14-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488354808735408?= X-GMAIL-MSGID: =?utf-8?q?1755488354808735408?= The recently introduced _PAGE_COW should be used instead of the HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply the same changes to pmd and pud. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe --- v4: - Break part patch for better bisectability arch/x86/include/asm/pgtable.h | 125 ++++++++++++++++++++++++++++----- 1 file changed, 107 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index c5047eb5f406..e96558abc8ec 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; +} + +static inline bool pte_shstk(pte_t pte) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return false; + + return (pte_flags(pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; } static inline int pte_young(pte_t pte) @@ -134,9 +142,18 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; +} + +static inline bool pmd_shstk(pmd_t pmd) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return false; + + return (pmd_flags(pmd) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); } #define pmd_young pmd_young @@ -145,9 +162,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -157,13 +174,21 @@ static inline int pud_young(pud_t pud) static inline int pte_write(pte_t pte) { - return pte_flags(pte) & _PAGE_RW; + /* + * Shadow stack pages are logically writable, but do not have + * _PAGE_RW. Check for them separately from _PAGE_RW itself. + */ + return (pte_flags(pte) & _PAGE_RW) || pte_shstk(pte); } #define pmd_write pmd_write static inline int pmd_write(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_RW; + /* + * Shadow stack pages are logically writable, but do not have + * _PAGE_RW. Check for them separately from _PAGE_RW itself. + */ + return (pmd_flags(pmd) & _PAGE_RW) || pmd_shstk(pmd); } #define pud_write pud_write @@ -374,7 +399,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte) static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -384,7 +409,16 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { - return pte_clear_flags(pte, _PAGE_RW); + pte = pte_clear_flags(pte, _PAGE_RW); + + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PTE (Write=0,Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pte_dirty(pte)) + pte = pte_mkcow(pte); + return pte; } static inline pte_t pte_mkexec(pte_t pte) @@ -396,6 +430,10 @@ static inline pte_t __pte_mkdirty(pte_t pte, bool soft) { pteval_t dirty = _PAGE_DIRTY; + /* Avoid creating Dirty=1,Write=0 PTEs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pte_write(pte)) + dirty = _PAGE_COW; + if (soft) dirty |= _PAGE_SOFT_DIRTY; @@ -407,6 +445,12 @@ static inline pte_t pte_mkdirty(pte_t pte) return __pte_mkdirty(pte, true); } +static inline pte_t pte_mkwrite_shstk(pte_t pte) +{ + /* pte_clear_cow() also sets Dirty=1 */ + return pte_clear_cow(pte); +} + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -414,7 +458,12 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { - return pte_set_flags(pte, _PAGE_RW); + pte = pte_set_flags(pte, _PAGE_RW); + + if (pte_dirty(pte)) + pte = pte_clear_cow(pte); + + return pte; } static inline pte_t pte_mkhuge(pte_t pte) @@ -505,18 +554,30 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_RW); + pmd = pmd_clear_flags(pmd, _PAGE_RW); + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PMD (RW=0, Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pmd_dirty(pmd)) + pmd = pmd_mkcow(pmd); + return pmd; } static inline pmd_t __pmd_mkdirty(pmd_t pmd, bool soft) { pmdval_t dirty = _PAGE_DIRTY; + /* Avoid creating (HW)Dirty=1, Write=0 PMDs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pmd_write(pmd)) + dirty = _PAGE_COW; + if (soft) dirty |= _PAGE_SOFT_DIRTY; @@ -528,6 +589,11 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd) return __pmd_mkdirty(pmd, true); } +static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) +{ + return pmd_clear_cow(pmd); +} + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -545,7 +611,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_RW); + pmd = pmd_set_flags(pmd, _PAGE_RW); + + if (pmd_dirty(pmd)) + pmd = pmd_clear_cow(pmd); + return pmd; } static inline pud_t pud_set_flags(pud_t pud, pudval_t set) @@ -589,17 +659,32 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { - return pud_clear_flags(pud, _PAGE_RW); + pud = pud_clear_flags(pud, _PAGE_RW); + + /* + * Blindly clearing _PAGE_RW might accidentally create + * a shadow stack PUD (RW=0, Dirty=1). Move the hardware + * dirty value to the software bit. + */ + if (pud_dirty(pud)) + pud = pud_mkcow(pud); + return pud; } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + pudval_t dirty = _PAGE_DIRTY; + + /* Avoid creating (HW)Dirty=1, Write=0 PUDs */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pud_write(pud)) + dirty = _PAGE_COW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -619,7 +704,11 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { - return pud_set_flags(pud, _PAGE_RW); + pud = pud_set_flags(pud, _PAGE_RW); + + if (pud_dirty(pud)) + pud = pud_clear_cow(pud); + return pud; } #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY From patchwork Thu Jan 19 21:22:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45990 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp555964wrn; Thu, 19 Jan 2023 13:35:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXt0QBfTgBJru9bTxjXoV5yj9+5dp4EVTrMWJxV+NkPjpptE/zd1cqcBTPXJrePDBuJDXGy0 X-Received: by 2002:a05:6a20:a598:b0:b3:ed81:9f58 with SMTP id bc24-20020a056a20a59800b000b3ed819f58mr13378218pzb.52.1674164139949; Thu, 19 Jan 2023 13:35:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164139; cv=none; d=google.com; s=arc-20160816; b=Mo87/TKEYePUdZH0fMot1QBmpMK0+J0CkWLzMygAxpaV8GJL3CSV2YXb4LvXnVc002 vt7cJY+beui7+2Q5bsqv0tpY9hz03KfLqQhR8IhF1SjJAr4xMMvgqrB0Uf9yzLIQjUwp dyZf2+/AmY98Gu5zUufOT5+DzlvyNCMlUUv1jM8SLfYgQJHma6Io5ArcPJj6jZmVRWrv Xp5NshA0yLuGYVJH//1v9Z1q+vDm/QJyDrjuv5tVgahYhpR1cBdR7st9000Egob3DqBU mACUX+EfIdpexrqbVv32czdUOHszbM7k3Uec8FkuMERtM3rjYlVEQnTTEjgFaN7WFKn3 nW8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=Cxz0RFzia4yt8BlCv7dHJItKc83W6pPpciB1MrUhgXc=; b=Sl+FVoCFWONSlRelTFZNv/Oq2eNIsbk8Ebd5fJifGyy6Uj90/YLA7+/Ug8fi9D3zK4 OncGd8wTfyehQlFhp34/W+rZKusfsrZAoNk349vgnOYG4I6513O20JAR7xZvFjATUSzQ UGltgin0eSeYC8UNfNP0SML490ZOwhmBb5o3wYsWM+Sp/IDlNOAaBSXC11GBKigTJHCF MqPwCoQFSsUhEPgonbkt03iYwcEQzNm1BnEwHTvOTJPPwWSmf7/pCVzoD/EKJRbuvXNq 6tZ2CVnz7kwF5SmXt+aeWfzES9U+OXDHo8oMw3kmpU1l1iTVg10GoZ5yP0ZRd8oRoZF4 YZgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WhE8+KXR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c12-20020a631c0c000000b004b1eac7d37bsi35815124pgc.131.2023.01.19.13.35.28; Thu, 19 Jan 2023 13:35:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WhE8+KXR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230259AbjASVeQ (ORCPT + 99 others); Thu, 19 Jan 2023 16:34:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230381AbjASVb5 (ORCPT ); Thu, 19 Jan 2023 16:31:57 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08052A9580; Thu, 19 Jan 2023 13:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163581; x=1705699581; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=i0EcBhWDxak7M9Yc3IVuRFqXYd9IgUPiDgl15diefTc=; b=WhE8+KXRZzLkYhRyBiIEwgz12qvjmpYj7S6svL+9/2diq8fFTsAOTfQf 55AzLxK3nGi5Qg/zt/Ef1LFs0S4XouQUzOdJzkdNmT5dh1R29nB8CG4rI NBOsWWfatqsNqfUT9gjeBjPi5L7CG1Rd5tCwC5SVJd9nSOXeMoz090JEO fufhhuWhcX7WfD3/WDzl8AXinHNtMW69yfK/gdolyN516zGoH+T+o6dXo 4qv6CqHikzyVvrAaH85AUEhze5sLvbfjBSdSfbxwqETyo4ePQjrL4JwAw DI6D8ZaW+xFbhV6mfs4TwEE9S4ZrlD69j0/3aR2NDdKE09I6XCD9/roVZ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119476" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119476" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139051" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139051" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:45 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu , Peter Xu Subject: [PATCH v5 14/39] mm: Move VM_UFFD_MINOR_BIT from 37 to 38 Date: Thu, 19 Jan 2023 13:22:52 -0800 Message-Id: <20230119212317.8324-15-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488336777698557?= X-GMAIL-MSGID: =?utf-8?q?1755488336777698557?= From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Future patches will introduce a new VM flag VM_SHADOW_STACK that will be VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake of order, make all VM_HIGH_ARCH_BITs stay together by moving VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be introduced as 37. Tested-by: Pengfei Xu Tested-by: John Allen Reviewed-by: Kees Cook Acked-by: Peter Xu Signed-off-by: Yu-cheng Yu Reviewed-by: Axel Rasmussen Signed-off-by: Rick Edgecombe Cc: Peter Xu Cc: Mike Kravetz --- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7afc86d50442..82a9a4903651 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -366,7 +366,7 @@ extern unsigned int kobjsize(const void *objp); #endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR -# define VM_UFFD_MINOR_BIT 37 +# define VM_UFFD_MINOR_BIT 38 # define VM_UFFD_MINOR BIT(VM_UFFD_MINOR_BIT) /* UFFD minor faults */ #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ # define VM_UFFD_MINOR VM_NONE From patchwork Thu Jan 19 21:22:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45992 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556165wrn; Thu, 19 Jan 2023 13:36:18 -0800 (PST) X-Google-Smtp-Source: AMrXdXsrv1Qrtp8tVDaefCUpMcW6h6W+AZaMkgd08sb0O7RARtZKLu2FxCmtzmvwIPInrvrYoxvu X-Received: by 2002:a05:6a21:170f:b0:af:9dda:b033 with SMTP id nv15-20020a056a21170f00b000af9ddab033mr13080302pzb.37.1674164177881; Thu, 19 Jan 2023 13:36:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164177; cv=none; d=google.com; s=arc-20160816; b=Cz2akeKrsK6smJjfTmsiL/6gSdJYzD6QhXZBaAEE6t+AAK/15NurrhzfL4+nV8+7oI uDoqkkIdR3GA8W3MPbkC7XGrl1eMwyaLOxshR22kr6iVeCLlwHJoG+IHQ5FLXsgxjfcW 8d2gbYQiUY00cfH+AyzfERPnZLr9ckngUvZGLKtlwW6Te4sxqb/Wm+doBrPHuoEZejsS U9+8nU5+ZseSd3Wl6bZ9ysg64hNdvaxeVb4ZG/vMDQ888/Aq+7sFXJWqITTlGjGXNVgQ 6uTI3c4cuI8MXx989z5PuNlYGg/zpcmtwQYZ2tI7gFJGw8VwWRgxYuUD12KJU72vmzoq hwxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=StQkA6ot9FCk/YmAiAeLd46+t6/Gkl8n00cyZWXpPeQ=; b=xRiJcrLyWTj/YN5vWrsR/O6ogNuYbFjrTDCiQDbLJVmVOYQmi0l9AolIQS1srziTYC aTL+/Bb/IFLdO5lL9eSrY9XOnez9Xdqpp+ijWoQ7/L2b4xneZ8jc1d9PPfy2NjLoHa+R yHMBvyeytDAuLGo39OgBRLXv7ddnPYC1H+AkzEsBOT6Avzmo7IEYKyH7PX04Tqo4ikOn SSKE1QjpwSu9PEIct20WuQe8z6ppnlxVz6LlV5Ieu5tuaJUtow0xi1sNpZWuwsjP32Aj iC+sdTY8SGG+w2ytjlLGEWkL2YY/UZOUhB5ooigHk9Ky83hmhvhWUojuNfNXJAFc3NcT FIcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LXsvEtmK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f69-20020a636a48000000b004d21c99e5b7si2375551pgc.316.2023.01.19.13.36.05; Thu, 19 Jan 2023 13:36:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LXsvEtmK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230419AbjASVe5 (ORCPT + 99 others); Thu, 19 Jan 2023 16:34:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230423AbjASVcz (ORCPT ); Thu, 19 Jan 2023 16:32:55 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C73EFA9585; Thu, 19 Jan 2023 13:26:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163581; x=1705699581; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=G9eEBGmU/UnEJ+400J+VwekvQK2O1I9MsdKnAZAqxQw=; b=LXsvEtmKlb3UalVFARwCFI7tkjepU7eCssI9j0x3Yo2MHNki7w8VGscW D6zfdb/x6YKWhbmcqcR9o3w3kiMqjUd08vrg9eXnQO1eSH8lYVxb0Zot5 fiDqjDl58fcwfp9TrWjM+kgNRAVgc7kIkw9LvXDl4SosPLznNfsOR7TRF NoBxNCfj5flKkMn4Lp7hJNZ/CIBCwevvO/wqFt8vyVNKeTGfGcoDkBvgm 7o5/NPMRzqth4GvyHJCAaS3+kynRxkvfS0kfxXjFUY6ItQbsJQqvz99UV qbcSuomH5e3j6HGRQLUbvb0y3XAgjr6ZoP1bWEdR23niquyOGluWR22HK A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119497" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119497" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:49 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139056" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139056" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:47 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 15/39] mm: Introduce VM_SHADOW_STACK for shadow stack memory Date: Thu, 19 Jan 2023 13:22:53 -0800 Message-Id: <20230119212317.8324-16-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488376604578079?= X-GMAIL-MSGID: =?utf-8?q?1755488376604578079?= From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Reviewed-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v3: - Drop arch specific change in arch_vma_name(). The memory can show as anonymous (Kirill) - Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags() (Boris) Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index e224b6d5b642..115843e8cce3 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..982126ffdbae 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui", #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif }; size_t i; diff --git a/include/linux/mm.h b/include/linux/mm.h index 82a9a4903651..824e730b21af 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp); #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +# define VM_SHADOW_STACK VM_HIGH_ARCH_5 +#else +# define VM_SHADOW_STACK VM_NONE +#endif + #if defined(CONFIG_X86) # define VM_PAT VM_ARCH_1 /* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC) From patchwork Thu Jan 19 21:22:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556482wrn; Thu, 19 Jan 2023 13:37:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXuGcySosRKkT3UyCg5YuY6YyWMP6c7fJ9UkExnnUGxn435QSA91+fXYrNGN6e/FJoF51gTy X-Received: by 2002:a17:907:c712:b0:7ba:5085:869 with SMTP id ty18-20020a170907c71200b007ba50850869mr13925139ejc.9.1674164231719; Thu, 19 Jan 2023 13:37:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164231; cv=none; d=google.com; s=arc-20160816; b=FSE+9squ7CpiKFC7DkMMxNw+3RCCmZdOo+Sr8LA2vsxpb37kjZO1GGxc9XyzZlbU1O XdHSNXa1/H5nvr3c6awAqhWx7cx0UmJR/KuKQxsoavLsi+KnY37dMRk7qL4Ervs3WhLy m8vqmxxH3epv+lNC46KwCCqJVI3NcpRmPcZFZdgIGBerMYAatrzWlFX7nYqfwbKe387C 22V3zBb70+1TVSLIT/VROQyjTgKR/yHZ98bHHmH0DgODMr9PuvQeKSbS3h3oDEL4w0fY izBe93P/+tEEE5+LT8YEiGbIawM6k0TJ0A+dv3bUeMa0fWBIrDK5Y0jPmWs/qy07ohW0 ngkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=LjZx3/OxatauvlHEC9jJBaxJxgdku9VSDjOhpZOLExo=; b=S0oEln33p6u6qK9bgA2uuATCXUTBIdDgNp6X1m0JeRLxcVGuQHSyyS3Pm2P8a5d/+R F7LSbJcK/gW1fSXZY0LeVBWUOtl2aX0Aj8iRVUcKQdjDUJAosqSGVpJYnwn2ZdNEAC4f 4QyUWU6QUccd4ZgkKSNN5oJUwXjZsen83t447ESJzFuP4K9B0tJA93NC+7EO/k3VChNq m4UB96+/RpGNx3o0G9gE/XKThpbIiZt31rRkRYTDrygMkY/Qh/svSet0UUA/DdPuh+tA Qqpc4o/Wu2yOw43V4vsdvL9eUgqIvBnDIqRi2dHJeOqLMVNwyCG3hh31gcjpRXd9w4KY YyRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XQ0pPGRR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eb7-20020a0564020d0700b004859d0f61e0si50873727edb.380.2023.01.19.13.36.46; Thu, 19 Jan 2023 13:37:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XQ0pPGRR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230466AbjASVfj (ORCPT + 99 others); Thu, 19 Jan 2023 16:35:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230454AbjASVdc (ORCPT ); Thu, 19 Jan 2023 16:33:32 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C84AA95AA; Thu, 19 Jan 2023 13:26:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163586; x=1705699586; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=W0uo/pMYztKK7cUO1RHzIUdsd0CMMqFeZAfln1MTMss=; b=XQ0pPGRRmwr68scWhZzHMdL4krPzckNoEV5Z0WqASlcVbvxQJjkze8Gx FAlrg4WRd2nrzBvFuLG2sFV1sE+AWhRftH3Q94uhgAYNlDNOXzShi0Cuq ctb5lK1Ic/67aaWW/NZGXb00eqP5SE3ydfVqHhTdLRLGpPdeDptfPzJQS UkJR6bD1c94VrDYjrGeg01RNflXWD+s0CyNJ4GPoM9SE3TLU5MrnOk4YF DeG2aP7A99drVqgu9SmcUxRvmHHyUFQUKTMtsAT2pCxfMBwnNgq/6zK+O DeYWe455qcFr9+SR9h7b+eJqN57SSQT7VWZO1GjjOfd8gfFOwD6M+QmR0 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119524" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119524" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139070" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139070" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:49 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 16/39] x86/mm: Check shadow stack page fault errors Date: Thu, 19 Jan 2023 13:22:54 -0800 Message-Id: <20230119212317.8324-17-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488433423012043?= X-GMAIL-MSGID: =?utf-8?q?1755488433423012043?= From: Yu-cheng Yu The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS). Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features. The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set. This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read. For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes. Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Add description of COW example (Boris) - Replace "permissioned" (Boris) - Remove capitalization of shadow stack (Boris) v4: - Further improve comment talking about FAULT_FLAG_WRITE (Peterz) v3: - Improve comment talking about using FAULT_FLAG_WRITE (Peterz) v2: - Update commit log with verbiage/feedback from Dave Hansen - Clarify reasoning for FAULT_FLAG_WRITE for all shadow stack accesses - Update comments with some verbiage from Dave Hansen arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h index 10b1de500ab1..afa524325e55 100644 --- a/arch/x86/include/asm/trap_pf.h +++ b/arch/x86/include/asm/trap_pf.h @@ -11,6 +11,7 @@ * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code { @@ -20,6 +21,7 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, X86_PF_SGX = 1 << 15, }; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7b0d4ab894c8..070b50c87415 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Shadow stack accesses (PF_SHSTK=1) are only permitted to + * shadow stack VMAs. All other accesses result in an error. + */ + if (error_code & X86_PF_SHSTK) { + if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK))) + return 1; + if (unlikely(!(vma->vm_flags & VM_WRITE))) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ + if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) + return 1; if (unlikely(!(vma->vm_flags & VM_WRITE))) return 1; return 0; @@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * When a page becomes COW it changes from a shadow stack permission + * page (Write=0,Dirty=1) to (Write=0,Dirty=0,CoW=1), which is simply + * read-only to the CPU. When shadow stack is enabled, a RET would + * normally pop the shadow stack by reading it with a "shadow stack + * read" access. However, in the COW case the shadow stack memory does + * not have shadow stack permissions, it is read-only. So it will + * generate a fault. + * + * For conventionally writable pages, a read can be serviced with a + * read only PTE, and COW would not have to happen. But for shadow + * stack, there isn't the concept of read-only shadow stack memory. + * If it is shadow stack permission, it can be modified via CALL and + * RET instructions. So COW needs to happen before any memory can be + * mapped with shadow stack permissions. + * + * Shadow stack accesses (read or write) need to be serviced with + * shadow stack permission memory, so in the case of a shadow stack + * read access, treat it as a WRITE fault so both COW will happen and + * the write fault path will tickle maybe_mkwrite() and map the memory + * shadow stack. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR) From patchwork Thu Jan 19 21:22:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45994 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556458wrn; Thu, 19 Jan 2023 13:37:06 -0800 (PST) X-Google-Smtp-Source: AMrXdXtx1nIehFK0UFoN13HMJlpnJsszr44uCJakCP9ZhqvCHmPwZCF4AmyRkuX7ZAlipFB9dBiG X-Received: by 2002:a05:6a20:e608:b0:b6:64f8:4c8b with SMTP id my8-20020a056a20e60800b000b664f84c8bmr11595219pzb.54.1674164225790; Thu, 19 Jan 2023 13:37:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164225; cv=none; d=google.com; s=arc-20160816; b=0QA95JpFXNooumuZe3UoGhmdqDpad2GzhXsJrnzsIIHcnuGmXgISfPWQ2lyJucvpcM RyraHYprFMZoIZZP7baeQ3l7y8bTC3Wzt5CgqjHS+zqHysw8fKw6vy+2dA3440NbDTHL rnLto0dW2Oc/YkdUnvD2rM77D5yo3bixEggukln3iWHqddCq5puzKDuibDwjgtotxidB qFBgSi1ss6nkTPAToehqL+5vAKSMOYu3mEvaUxTZ9r1QeI/+V4kBoMyTKbEjZ6xlsm5p izDv8nsxQvHMRKd/QhcNSWnqbD+MOS5JT8TtxfvNujP25BJaaA3nyHFJDkHDQkyq4qtb m07g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=F+8Tf9HdQyQPAdeGQwF3qLXqysHCxD0/3GwiYQ1Fj7g=; b=RHOJko0xLPjKQeAH1PRpjXQsm+5Df0sk34JEDMxYs2AK9QAHC7m9ExY1e76FX6baAt tLD0vpLQrOdvAt0kJm0QmII39zUvA+ABjppZ9a1CfBkukH5bhRgxlCPHzI7fe/MEAuQd pv7CXECDyFTprqNVfx5nIBV37KRU3AlJXlh8ggJQi2SCgfqF5GQ3bm/7iSKN+Gc9/dZ2 oZFCsohb0zJHvGL1xXlnRvH2g38G1gmWnJR+sye0vcyYJ5GLclrzXeEMpSBLAbZtpWaC FIQkPMbYm1aIWv/5lLfJY3YhXflG3kLiRShS66sNQ3vtGeJzKGk7B8iYGLJjdjaU4XkJ coaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DCMS9Lq1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y14-20020a634b0e000000b0049026a12769si40730210pga.739.2023.01.19.13.36.52; Thu, 19 Jan 2023 13:37:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DCMS9Lq1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230125AbjASVfu (ORCPT + 99 others); Thu, 19 Jan 2023 16:35:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230285AbjASVdj (ORCPT ); Thu, 19 Jan 2023 16:33:39 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58E37A95BD; Thu, 19 Jan 2023 13:26:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163591; x=1705699591; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=M7Xo1HXic3OeNqgtbqqoeSfgPr5lKe9tQj+QswBwq38=; b=DCMS9Lq1EUd+MJXo33q0ueut6cFPZ861HiL+7iRdUQ57cRC+bbr42uZp CJmtroMT3zAuuwDvJg/SXyk4TNZq5G6wilzeKa4PxZ0k89sw5G9SoQoae H59RzC4+CBG1W0xns9o7Zy9GiCszieUlagvDlL7KVsTGNi/9lC9GMGgqq hUtlrC7aU3XMOCQVfduaxyloAcn+dRafVAp8lOJHrndYXKAIlcSi1ftoS m5oXCRKaH7+jAeJIXMIU1ctkNV3RbLcpJWcI8RubK93yzQade9EAfujjS eFW2Ecp7AzerwjZ6jr8X+1uJbV4LarqxnnL5crsZKEIsjF0gW9JawifQN Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119553" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119553" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:52 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139076" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139076" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:50 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 17/39] x86/mm: Update maybe_mkwrite() for shadow stack Date: Thu, 19 Jan 2023 13:22:55 -0800 Message-Id: <20230119212317.8324-18-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488427467332628?= X-GMAIL-MSGID: =?utf-8?q?1755488427467332628?= From: Yu-cheng Yu When serving a page fault, maybe_mkwrite() makes a PTE writable if there is a write access to it, and its vma has VM_WRITE. Shadow stack accesses to shadow stack vma's are also treated as write accesses by the fault handler. This is because setting shadow stack memory makes it writable via some instructions, so COW has to happen even for shadow stack reads. So maybe_mkwrite() should continue to set VM_WRITE vma's as normally writable, but also set VM_WRITE|VM_SHADOW_STACK vma's as shadow stack. Do this by adding a pte_mkwrite_shstk() and a cross-arch stub. Check for VM_SHADOW_STACK in maybe_mkwrite() and call pte_mkwrite_shstk() accordingly. Apply the same changes to maybe_pmd_mkwrite(). Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v3: - Remove unneeded define for maybe_mkwrite (Peterz) - Switch to cleaner version of maybe_mkwrite() (Peterz) v2: - Change to handle shadow stacks that are VM_WRITE|VM_SHADOW_STACK - Ditch arch specific maybe_mkwrite(), and make the code generic - Move do_anonymous_page() to next patch (Kirill) Yu-cheng v29: - Remove likely()'s. arch/x86/include/asm/pgtable.h | 2 ++ include/linux/mm.h | 13 ++++++++++--- include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 10 +++++++--- 4 files changed, 33 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e96558abc8ec..45b1a8f058fe 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -445,6 +445,7 @@ static inline pte_t pte_mkdirty(pte_t pte) return __pte_mkdirty(pte, true); } +#define pte_mkwrite_shstk pte_mkwrite_shstk static inline pte_t pte_mkwrite_shstk(pte_t pte) { /* pte_clear_cow() also sets Dirty=1 */ @@ -589,6 +590,7 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd) return __pmd_mkdirty(pmd, true); } +#define pmd_mkwrite_shstk pmd_mkwrite_shstk static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) { return pmd_clear_cow(pmd); diff --git a/include/linux/mm.h b/include/linux/mm.h index 824e730b21af..e15d2fc04007 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1106,12 +1106,19 @@ void free_compound_page(struct page *page); * servicing faults for write access. In the normal case, do always want * pte_mkwrite. But get_user_pages can cause write faults for mappings * that do not have writing enabled, when used by access_process_vm. + * + * If a vma is shadow stack (a type of writable memory), mark the pte shadow + * stack. */ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) { - if (likely(vma->vm_flags & VM_WRITE)) - pte = pte_mkwrite(pte); - return pte; + if (!(vma->vm_flags & VM_WRITE)) + return pte; + + if (vma->vm_flags & VM_SHADOW_STACK) + return pte_mkwrite_shstk(pte); + + return pte_mkwrite(pte); } vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1159b25b0542..14a820a45a37 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -532,6 +532,20 @@ static inline pte_t pte_sw_mkyoung(pte_t pte) #define pte_sw_mkyoung pte_sw_mkyoung #endif +#ifndef pte_mkwrite_shstk +static inline pte_t pte_mkwrite_shstk(pte_t pte) +{ + return pte; +} +#endif + +#ifndef pmd_mkwrite_shstk +static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) +{ + return pmd; +} +#endif + #ifndef __HAVE_ARCH_PMDP_SET_WRPROTECT #ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pmdp_set_wrprotect(struct mm_struct *mm, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index abe6cfd92ffa..fbb8beb9265e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -553,9 +553,13 @@ __setup("transparent_hugepage=", setup_transparent_hugepage); pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) { - if (likely(vma->vm_flags & VM_WRITE)) - pmd = pmd_mkwrite(pmd); - return pmd; + if (!(vma->vm_flags & VM_WRITE)) + return pmd; + + if (vma->vm_flags & VM_SHADOW_STACK) + return pmd_mkwrite_shstk(pmd); + + return pmd_mkwrite(pmd); } #ifdef CONFIG_MEMCG From patchwork Thu Jan 19 21:22:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45997 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556723wrn; Thu, 19 Jan 2023 13:37:58 -0800 (PST) X-Google-Smtp-Source: AMrXdXu9mU/yi6ZEjawbxz07vTXQnyVrgTw2Wm5JJTexoc+09+7NEuFVxOLAm9XiVXtbO48vhPx8 X-Received: by 2002:a17:906:e24a:b0:7c1:ee:5bca with SMTP id gq10-20020a170906e24a00b007c100ee5bcamr10612178ejb.73.1674164277908; Thu, 19 Jan 2023 13:37:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164277; cv=none; d=google.com; s=arc-20160816; b=dGeR/Nn5DxLZ6ck0uCVLdWPkpBwgYhJ4eEoAM7/tLzYkxTH4NlcYBHtdJsL27d97/B LvAa1MszUdR4/IyXOHM5Q2WY/rTUIkrl7kqhoxnIhLeYjJABjH0Y3nvG4QJTsvuNAbQO PRQDwc9IpJVkQDStDLiMOuaVpy+facQciaPCxaq3wb0EAPt8ADavpb64IbDy77jpc4KS Y8gWoGpHnBy2ps4mQ2owTqzT7jEMaikWhF6EOHZ/CX98t92tCNeGiWs+PX2kMzSrmehW BOpO4e95PAKoUf8ArCq3gxn4FukEuxV2D3ThsRigoVKPYY0Wx7K9joEg+HtIKFypyhM/ WAYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=cL1+A7sEmRT22WV4GTGRusW0jLfRlihJHMYsiIHNplM=; b=N36ieve2AOEWcQxmlTRyFspljtZre0wFmQ8tYbf/fEa+v5UzPmMkjij4zf5MzVTXDv FVMsyP/u8VtB1BxoFSqU6o2PxlY3e3qooLOJAtzdn1R5r/y4QHlWK2GU3Dptki9i7Zlo DLo4DAZf3SlWOgB2M7ivapOVvrTiPQzos8FSmPaPuzFlr5y1v9YRSXxipGsIi3VGnKb4 V4gdI/t9u+QlnXM0DTc2uX+uVsrljJJDFGdeWIjO4a36fTBDpeMiA5p/3VdFRxqLE2v8 IFErsd1BtvKXWTXquxcYxGTFaBP9lAEE6AP1gAGOlEhkNrBfrz2U+bD6M41l7vX/bIQK ez8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Q1kHlcw7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ae2-20020a17090725c200b007c07d0dbe85si19229747ejc.463.2023.01.19.13.37.34; Thu, 19 Jan 2023 13:37:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Q1kHlcw7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230123AbjASVhB (ORCPT + 99 others); Thu, 19 Jan 2023 16:37:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230482AbjASVeo (ORCPT ); Thu, 19 Jan 2023 16:34:44 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CF5CA3177; Thu, 19 Jan 2023 13:26:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163616; x=1705699616; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=NJhAPcbhUYo6SN1rH0mb7F3rAzE882GPxl5rAvSEEz0=; b=Q1kHlcw7UBkpLpksUL1Cmb/+k3cunnk/ojuV6VYfWRVa/sw7WXi75hi2 0E/S9HmeHUODsoruMi30PA83n5npVPuxc4dGJVYd93kZcPJXb2//ecVgj WQOpsyd+jbOayOmxqZoXpzEfMUB5WmNgfR3t6NBrYqlE5/a21qzsC4EqL 3cgTiF/pQiRm3v5VG4AtabkvduRZPPyvzBPE0VrM2XQbgvqZutVRhu3Qc /qtkgOz73Zbi693mINDHsUXTXOJrSYhgetpZ7XbntgOaZ1jM0u7a3WScr cHG4iFYaW+sQtZtGvl5V0uaK03dTYgCSdBAtriGATqU57Lbiszj/RwsrP g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119581" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119581" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:54 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139082" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139082" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:52 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, David Hildenbrand , Yu-cheng Yu Subject: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk Date: Thu, 19 Jan 2023 13:22:56 -0800 Message-Id: <20230119212317.8324-19-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488482041568344?= X-GMAIL-MSGID: =?utf-8?q?1755488482041568344?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Since shadow stack memory can be changed from userspace, is both VM_SHADOW_STACK and VM_WRITE. But it should not be made conventionally writable (i.e. pte_mkwrite()). So some code that calls pte_mkwrite() needs to be adjusted. One such case is when memory is made writable without an actual write fault. This happens in some mprotect operations, and also prot_numa faults. In both cases code checks whether it should be made (conventionally) writable by calling vma_wants_manual_pte_write_upgrade(). One way to fix this would be have code actually check if memory is also VM_SHADOW_STACK and in that case call pte_mkwrite_shstk(). But since most memory won't be shadow stack, just have simpler logic and skip this optimization by changing vma_wants_manual_pte_write_upgrade() to not return true for VM_SHADOW_STACK_MEMORY. This will simply handle all cases of this type. Cc: David Hildenbrand Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Reviewed-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe --- v5: - Update solution after the recent removal of pte_savedwrite() v4: - Add "why" to comments in code (Peterz) Yu-cheng v25: - Move is_shadow_stack_mapping() to a separate line. Yu-cheng v24: - Change arch_shadow_stack_mapping() to is_shadow_stack_mapping(). include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e15d2fc04007..139a682d243b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2181,7 +2181,7 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma */ if (vma->vm_flags & VM_SHARED) return vma_wants_writenotify(vma, vma->vm_page_prot); - return !!(vma->vm_flags & VM_WRITE); + return (vma->vm_flags & VM_WRITE) && !(vma->vm_flags & VM_SHADOW_STACK); } bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, From patchwork Thu Jan 19 21:22:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45998 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557035wrn; Thu, 19 Jan 2023 13:38:50 -0800 (PST) X-Google-Smtp-Source: AMrXdXuKVD2t922aCOJ/XD3VeIXFYS6FeUHqYzp0mxODiLROcubJb0V3N7NzzWYXwJ6xC/pk/LBI X-Received: by 2002:a50:e610:0:b0:49e:c16:2c0f with SMTP id y16-20020a50e610000000b0049e0c162c0fmr11612423edm.1.1674164329890; Thu, 19 Jan 2023 13:38:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164329; cv=none; d=google.com; s=arc-20160816; b=HjsnhLBVf+ubbtSNexqAZJGt1G1gX+nO0FMweD+quQRV4gxKTl+0/2J46aJCLAbYgI jTFdgq9RTr/5Zqe/fhEvjk6AGmK/8W0SMR8M6Tu61cI29LOYNvTPeZz7yZc68lWkRzZa yXOCGM50xEoRnet2QH7l5uViVe7/Majme8KS0di24Ee0gQ5i+dfFoMg2ncrMx+zUjPVe WZfCaucpNeUMnzXKH6ldHqLtM6rdLJliu/cdBbZgh55cRoj9GonMJGIN4+M49PFSu1ry 7LpgJ7wUmTI1HofGzBl0LlxuUWl257cZ7p7nJLj+zN/CqJIraqQiIjJMRgM2AF1g/CM8 rPoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=1+7FX9/q+7niILOmZEQ+FO8zDLlVkBM+CikBvkIlWYA=; b=VBnDJD8NEbqU2ZjOO49x/EA0PLOQYU3p4cRPzVjPj9AsxG1A3/PsoaFWkC2da7b1Dc JOLwV62bb3Pfij4zQgCP5D+d1S7FRSKCJtmYTvvOjlPMormtlh1uh0dYvXefUH6F0XEO XCARPF5Fnw+TsRBbwtEesPZUsdNtD9j4FVXpya2mOKr0y906Jcch+VAM0WfuVaU05hl+ oBIQ8n+I8Wh73uuN4I+Blj++kiKvSD//vRNOvHZZV7cw+wjpVrjyvs8dWyON9kAnEs8G Izr53nZLKatkwM5yXaxAGzPhsxMVtA3eYt4XlQ8QPj+Z831avOyU0Vpc4x40Wt1t4Rp8 l2iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PMpO9Az+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r28-20020a50c01c000000b00461c50013b8si42206422edb.192.2023.01.19.13.38.24; Thu, 19 Jan 2023 13:38:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PMpO9Az+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231144AbjASVhd (ORCPT + 99 others); Thu, 19 Jan 2023 16:37:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229961AbjASVfb (ORCPT ); Thu, 19 Jan 2023 16:35:31 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A569A259E; Thu, 19 Jan 2023 13:26:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163619; x=1705699619; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mm+/ZDL+/WIl4y++bICVeaKn86uLCgMEEZawTB+3rKc=; b=PMpO9Az+5sFWH+BMRmx3RFkQAGEH25D5e6AsgrAQnR9X/Xwhqeb0MwBL QwvfWbMG6dLGT/VQgH924UKAZ4bsXdQsCKyrJtFLFVQVGUd2Tib9YwI9V 5jgYe43GmeAm8Pr9b6eyarvyax1MrP9uS0mdyT7BSddWokkeXnlZs4yCK kPQsiGPqUOvFqYMNMQjeMk+b4FrTSaqwKnw2uHw9bBJGwCMTBVBR+U0bl zTuH7p5I+PHgIOs+R2fiYA4JcKwdg7uAGbT6cDljmbZlDtAsLM3TyLui7 y6HeE/amBzIp8yNUesrdS3VU1EE0wEJAIodm/6vU7B+dskbDkZndr39Hl g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119608" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119608" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139087" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139087" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:53 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly Date: Thu, 19 Jan 2023 13:22:57 -0800 Message-Id: <20230119212317.8324-20-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488536118264480?= X-GMAIL-MSGID: =?utf-8?q?1755488536118264480?= From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory. In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite() or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places where pte_mkwrite() is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA. - do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(). Teach it about pte_mkwrite_shstk() - When userfaultfd is creating a PTE after userspace handles the fault it calls pte_mkwrite() directly. Teach it about pte_mkwrite_shstk() To make the code cleaner, introduce is_shstk_write() which simplifies checking for VM_WRITE | VM_SHADOW_STACK together. In other cases where pte_mkwrite() is called directly, the VMA will not be VM_SHADOW_STACK, and so shadow stack memory should not be created. - In the case of pte_savedwrite(), shadow stack VMA's are excluded. - In the case of the "dirty_accountable" optimization in mprotect(), shadow stack VMA's won't be VM_SHARED, so it is not necessary. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Fix typo in commit log v3: - Restore do_anonymous_page() that accidetally moved commits (Kirill) - Open code maybe_mkwrite() cases from v2, so the behavior doesn't change to mark that non-writable PTEs dirty. (Nadav) v2: - Updated commit log with comment's from Dave Hansen - Dave also suggested (I understood) to maybe tweak vm_get_page_prot() to avoid having to call maybe_mkwrite(). After playing around with this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be writable ended up mapping the zero page as writable and other surprises. So creating shadow stack memory needs to be done with manual logic like pte_mkwrite(). - Drop change in change_pte_range() because it couldn't actually trigger for shadow stack VMAs. - Clarify reasoning for skipped cases of pte_mkwrite(). Yu-cheng v25: - Apply same changes to do_huge_pmd_numa_page() as to do_numa_page(). arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 7 +++++++ mm/memory.c | 5 ++++- mm/migrate_device.c | 4 +++- mm/userfaultfd.c | 10 +++++++--- 6 files changed, 30 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 45b1a8f058fe..87d3068734ec 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -951,6 +951,9 @@ static inline pgd_t pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) } #endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#define is_shstk_write is_shstk_write +extern bool is_shstk_write(unsigned long vm_flags); + #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index e4f499eb0f29..d103945ba502 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -880,3 +880,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +bool is_shstk_write(unsigned long vm_flags) +{ + return (vm_flags & (VM_SHADOW_STACK | VM_WRITE)) == + (VM_SHADOW_STACK | VM_WRITE); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 14a820a45a37..49ce1f055242 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1578,6 +1578,13 @@ static inline bool arch_has_pfn_modify_check(void) } #endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shstk_write +static inline bool is_shstk_write(unsigned long vm_flags) +{ + return false; +} +#endif + /* * Architecture PAGE_KERNEL_* fallbacks * diff --git a/mm/memory.c b/mm/memory.c index aad226daf41b..5e5107232a26 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4088,7 +4088,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); entry = pte_sw_mkyoung(entry); - if (vma->vm_flags & VM_WRITE) + + if (is_shstk_write(vma->vm_flags)) + entry = pte_mkwrite_shstk(pte_mkdirty(entry)); + else if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 721b2365dbca..53d417683e01 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -645,7 +645,9 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, goto abort; } entry = mk_pte(page, vma->vm_page_prot); - if (vma->vm_flags & VM_WRITE) + if (is_shstk_write(vma->vm_flags)) + entry = pte_mkwrite_shstk(pte_mkdirty(entry)); + else if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 0499907b6f1a..832f0250ca61 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, int ret; pte_t _dst_pte, *dst_pte; bool writable = dst_vma->vm_flags & VM_WRITE; + bool shstk = dst_vma->vm_flags & VM_SHADOW_STACK; bool vm_shared = dst_vma->vm_flags & VM_SHARED; bool page_in_cache = page_mapping(page); spinlock_t *ptl; @@ -84,9 +85,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, writable = false; } - if (writable) - _dst_pte = pte_mkwrite(_dst_pte); - else + if (writable) { + if (shstk) + _dst_pte = pte_mkwrite_shstk(_dst_pte); + else + _dst_pte = pte_mkwrite(_dst_pte); + } else /* * We need this to make sure write bit removed; as mk_pte() * could return a pte with write bit set. From patchwork Thu Jan 19 21:22:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45996 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp556627wrn; Thu, 19 Jan 2023 13:37:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXvAOxEAcSooZ6XlsoSmXncy7R7EJnD7flzaiHrJwzejx6IuuBMjXmrnmBNJI9KPxg0c78n0 X-Received: by 2002:a17:906:78a:b0:86e:7683:4213 with SMTP id l10-20020a170906078a00b0086e76834213mr13626163ejc.42.1674164257154; Thu, 19 Jan 2023 13:37:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164257; cv=none; d=google.com; s=arc-20160816; b=GRMwxZgnVX+yyQKiZh/I7ikHW/ZvJ1sR7KfrY20BcATxmIpV7oDMkmtgsYUTKgDjAR L7GeL/uTUF+RhQeh4wSU8nCv8LGaJGVSE9EjA8ZzYGIc+moeyl5SWdt+5OGYSD+IA/6b xqPu61I6go/Kom+6QwHuBhs5oI/BM+VJ/NGMjokbehkpVo3oUn3Bxa6s9caeVCUeGSel WQiG1x4BjQDhSYEk/OdCCKCfVXQAU5b6lZXlxqsA6rw5EYKOWNpfuhCpYwm++3izDn3o RRWchcVhlpNaYCUJmcgH2asLGSrzWZCkxmsO2SsuXv/uwZ/RCAWGckw/S8kio0Mrdr35 NDNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=YWVE+Ni8aNDtLo7tgyffJpW9kI97N/rM06lnRUBkkSk=; b=J4YBoN1OUPaTwIvPYErca8bWf/2kWVe8fmoxitE3qArjCUyrOnqi9SRIjOsxgZCSgp zQMa6C822sJ5hs6CsMLQ3c8lHMSiOLQeEEYlpBHapH8+tT3vpu6gZmqzzwgVOfsrZbqb gSUQIXZc99vpAFOchRgvN4zBoKRxblEdSz1opttSkIvAI/MsshoTf35B2uHWYzto7iXW qjbStTwbLkWkR+RFnTICTxrdEdTi4UExFTbtQJKB3UojS0XjVrQtlaYLgvSiRSJOGzX3 thLbMciYwqOEg8iT9xUR3GogBHfYWmiQGRCcKDIi9XvdgzVzOEwz3JSWHTRUMZ5joU31 nPiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="no/aVTuC"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l19-20020a056402255300b0049e475c817asi8025924edb.479.2023.01.19.13.37.13; Thu, 19 Jan 2023 13:37:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="no/aVTuC"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230063AbjASVgs (ORCPT + 99 others); Thu, 19 Jan 2023 16:36:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230498AbjASVer (ORCPT ); Thu, 19 Jan 2023 16:34:47 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A4F94A1E0; Thu, 19 Jan 2023 13:26:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163619; x=1705699619; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=+3N34QEERZNcvQNJAARc1au4ZPtrpENbiq/H3U3EHgE=; b=no/aVTuC4lT7oAPeDGhAvjU3xIb4ro4QSJgI/FApFptenkK5ErT3tJes t0B9M7xQTQKWWoE2o85bZzhNAvPnCaL0S3x1CKxS2QPUcLBF19XVwpboM itPhwqs+c5T44F4eE5Wg9YX0BLyX7kS0w85WaeemNnlXPmSslNmcam5th jsb8mnf1YFOTPcRcFmXDRJfjKBo9ZxXVMgaZfna0qQin4B2of9PqpCFeT q8mdcD8EsnjR0UxI25EnNBqdUwnhbgFA5uv0TORZY4Bbhwq5HnoReLj71 Frx85Juh8Uai6OsyfBqIBUxrHXEK/el/JRLbHE6aFW3lR2U9RUCqTTOej w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119634" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119634" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:57 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139093" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139093" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:55 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 20/39] mm: Add guard pages around a shadow stack. Date: Thu, 19 Jan 2023 13:22:58 -0800 Message-Id: <20230119212317.8324-21-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488459825931922?= X-GMAIL-MSGID: =?utf-8?q?1755488459825931922?= From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks. Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed. The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like: addq $0x80, %rsp However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this: READ_ONCE(ssp); // read+discard top element on stack ssp += nr_to_pop * 8; // move the shadow stack READ_ONCE(ssp-8); // read+discard last popped stack element The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault. This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v5: - Fix typo in commit log v4: - Drop references to 32 bit instructions - Switch to generic code to drop __weak (Peterz) v2: - Use __weak instead of #ifdef (Dave Hansen) - Only have start gap on shadow stack (Andy Luto) - Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen) - Improve commit log partly with verbiage from (Dave Hansen) Yu-cheng v25: - Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c. include/linux/mm.h | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 139a682d243b..3f980d4823ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2987,15 +2987,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) return mtree_load(&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_GROWSDOWN) + return stack_guard_gap; + + /* + * Shadow stack pointer is moved by CALL, RET, and INCSSPQ. + * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB + * and touches the first and the last element in the range, which + * triggers a page fault if the range is not in a shadow stack. + * Because of this, creating 4-KB guard pages around a shadow + * stack prevents these instructions from going beyond. + * + * Creation of VM_SHADOW_STACK is tightly controlled, so a vma + * can't be both VM_GROWSDOWN and VM_SHADOW_STACK + */ + if (vma->vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { + unsigned long gap = stack_guard_start_gap(vma); unsigned long vm_start = vma->vm_start; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; return vm_start; } From patchwork Thu Jan 19 21:22:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 45999 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557092wrn; Thu, 19 Jan 2023 13:39:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXuBguXdeihNN4n8NxhlVdCrJr33b2m22s6UBRwi1RErH3j2hXi/OpfMvaXDheh32KtQ0Xg9 X-Received: by 2002:a05:6402:cf:b0:45c:835c:1ecc with SMTP id i15-20020a05640200cf00b0045c835c1eccmr23832019edu.26.1674164340143; Thu, 19 Jan 2023 13:39:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164340; cv=none; d=google.com; s=arc-20160816; b=S2y6FMP/AMmzfRDWFstGL2eLQc1qLUdnYLXXZ/qXdpJzK6bg6DrSLbEbKBpA1QruAg KSuMSGO5A3AuuzH4+qER0lZy+tFWvJVd6ShKAA61b92PfK8ugIjlqxAOOLKc7+mpynNy U03GGsAOgYeY28hQw9QO6n1MMc0TxfBYVIutBKrFpb5Je1Dfb430lApuvPHy1DqDxg2v /lohc429ALkdRUfbDXTL1Hf7Errc+hnnOa35q4vbiM1N6kpm+7bNiIEfEhf29dkUtjl3 KH2pUiV2qKcpc2FZteo4nZx9VsOyN+p9XDLZMYfOLAgQ1HZG1rE9L2A3i32PAJWgLVrt sYSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=uNNpN6cdmenOslkS6PIT2EBjNmLCOw87BxzclT8zLwY=; b=sMfMUZOo6QLDDBEAMLSp+LPRExggq1DkwocY0NvgJCi7xGBU/A9EClwcEuvM6gH4aq cyyd3G/ALJvNAZXfYPKwlvZdzc1gCR2SlgsPey13AlEXzPS8qlnh93IN0H2UTe7AZzh0 xaBSQjRFIg/gpqMe1lxK+vQ3bKp1LbqIyDD3qZCkCkECF4FhpbuddES3eNH6wIQhkdeT fG6aIiJn+9Jm0P8nBHXh66TOmyffA3B9Z0KFSInEg3Z8JMXWbpWr/kUyu3IZgn5Nvn0T M2Bkn8vDfDCg8hMHpfupro8HDGHqkSvt0San7tM6oYyc/Co+zIFxSiEo3d09MgKUuJdx ZL2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lCqPhsz5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c10-20020aa7df0a000000b00497779c45dfsi16184961edy.538.2023.01.19.13.38.34; Thu, 19 Jan 2023 13:39:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lCqPhsz5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230140AbjASVho (ORCPT + 99 others); Thu, 19 Jan 2023 16:37:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230500AbjASVfb (ORCPT ); Thu, 19 Jan 2023 16:35:31 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AA74A1024; Thu, 19 Jan 2023 13:27:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163626; x=1705699626; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Sqmbg6aViWSIuyH7MnUz0YCJC/oOzGtcIb7+8J/qX00=; b=lCqPhsz5Ligt4SU1XQzj7I8rtmx+ohVt4v+2Yjeefot2JPgXHGv7447r ecS8rKgKl6iQM9rH95Qxn9sWY5fLnOaOc7wC1P7N+ghAIjEpsCCPpWyMZ rX+GhueS+wgKabuxRQ0qpkFI0hHowWe8XfMhWD717+OopZTzF8FKa1PsO YJuV6bz0BQv+1MndopB2/1EkDKIU00OTPxp48kUcpLz8gUDjJQMC8jw21 GdeXpI7iej9QBGB7QwxXVbPC61lfc8TEDACLISeYTrHsubc9AWgM99Pe5 fLQe/7Wpw68nwphVrmYo8jJKhHZgFoFflVYNfU01d4TyInY3s8aQ6h2fj g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119660" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119660" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:58 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139099" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139099" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:57 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 21/39] mm/mmap: Add shadow stack pages to memory accounting Date: Thu, 19 Jan 2023 13:22:59 -0800 Message-Id: <20230119212317.8324-22-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488547028486467?= X-GMAIL-MSGID: =?utf-8?q?1755488547028486467?= From: Yu-cheng Yu The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Account shadow stack pages to stack memory. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook --- v3: - Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill) v2: - Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK. Yu-cheng v26: - Remove redundant #ifdef CONFIG_MMU. Yu-cheng v25: - Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping(). mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 425a9349e610..9f85596cce31 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; } From patchwork Thu Jan 19 21:23:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46000 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557175wrn; Thu, 19 Jan 2023 13:39:12 -0800 (PST) X-Google-Smtp-Source: AMrXdXtoQW5InZvLX9d7gngSCEsMGfDwMQrXpQZvhaE+B5iixcNs8AWbcIuiwqx5uI3tJ1gOwJUD X-Received: by 2002:a17:907:c70f:b0:871:d04e:1df with SMTP id ty15-20020a170907c70f00b00871d04e01dfmr13724741ejc.29.1674164352572; Thu, 19 Jan 2023 13:39:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164352; cv=none; d=google.com; s=arc-20160816; b=CfS70p+/bv94kAShsSL7YasueOsxgwsM1ZIrJOzZcT6kqEmv61v+rK9lvhdgBNkJ3S EwVrCz4iKU1Q92pkJ2iqhIF6/FL+WZzrgBMVMyo2a7tpLwJZQfSDfhdE43WUUxSoI/HD oWYd3SWlppmp0cjLl2Dzi/2H8DRLEj3TAvScmkgXN05iP2GOybLGUshIXMC0g/m20qyv OCAkwYrYBJ/p+dLxpDJ2f8HS2RW8481imwaF6kyiSKK8f9bMcc33tVq2tBn/jS0QsKdj NlL6qAIjcw1ECOT9/FBvjstT4g2nq0KKiVpctz1TEKcclHOD63XElgcNU+hi2VNC7bMc BrwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=+CXWRCPnpy0btc6mfBj/1TwmGopDVclXbZtiYlkdzbE=; b=OwdJoSNXYJeyWcr3fUrcnkcc77FV63G8rRN/Z2tvNHBbdNpG1Xkl5fgFu8lVMqXHSu YHXc0MUrJJ+b0QDacJMq0keUNWbxav/I1XU2U8IczMayZmR2LQKsO9kgoigyGwayet4P ciCHveWrgDMRfE2iQezZYAPu1JvGFt7jibYQY3xqi+4dKGAvh05FBDRHpJR3U6oHV2Ve 0+ZUz28tJSMI/PPWbkNad7gJoOOhKxKtr+WmMIy1nlx3vDkcCRVCafG5g3VFRyiZcBJ5 1hyvEoszVg4AA3KeU9Bg1btPR9rJVmzkP1pK+RT3z8FuwQ3qwqp4TxlG9xJVNtcUE4lS Bw2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="lk9s9lz/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gn5-20020a1709070d0500b007c0aa685133si21808530ejc.34.2023.01.19.13.38.48; Thu, 19 Jan 2023 13:39:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="lk9s9lz/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230458AbjASVhy (ORCPT + 99 others); Thu, 19 Jan 2023 16:37:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230518AbjASVfe (ORCPT ); Thu, 19 Jan 2023 16:35:34 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F7E7A317D; Thu, 19 Jan 2023 13:27:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163633; x=1705699633; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=yYO9z3rz8CNjUY1wgWFKyzY7eNICGcV4jFfyGKxF5+o=; b=lk9s9lz/qn9AJQXFxfGT0MsCWqsSypdwhQh1mhhiOyJO5mNoZP2k1IO6 n/S6AmLAZLhG65GbtSb+qJGPn8/zISvjTBDOVlaL4PGVkYFw8CQ5kq0b2 FjNnXJqibnd+tC7fvjmYvhbKuMmc8JZfyKfcKesjIwaC7rZDJ8ZptDNiH pi2E6jn7HeWAiYskJpk8q+cjK5x6iyuZ3mfn7nXeGX/yJgGWOI73BLqUZ 5AVJ5Lymc2pS3Lw/Smowo91KDhSZsPHuckSg4wAgb40scwkFj+PRYWHog tmbrLNszIlq+bf5hLqfRmozQfAeJrz6CPXjwm/DaOfYXFzMxRcVa5gHO1 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119688" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119688" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:00 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139105" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139105" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:58 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 22/39] mm: Re-introduce vm_flags to do_mmap() Date: Thu, 19 Jan 2023 13:23:00 -0800 Message-Id: <20230119212317.8324-23-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488560340917312?= X-GMAIL-MSGID: =?utf-8?q?1755488560340917312?= From: Yu-cheng Yu There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by: commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()"). There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Reviewed-by: Peter Collingbourne Reviewed-by: Kees Cook Reviewed-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe Cc: Andrew Morton Cc: Oleg Nesterov Cc: linux-mm@kvack.org --- fs/aio.c | 2 +- include/linux/mm.h | 3 ++- ipc/shm.c | 2 +- mm/mmap.c | 10 +++++----- mm/nommu.c | 4 ++-- mm/util.c | 2 +- 6 files changed, 12 insertions(+), 11 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 562916d85cba..279c75ec6a05 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -554,7 +554,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size, PROT_READ | PROT_WRITE, - MAP_SHARED, 0, &unused, NULL); + MAP_SHARED, 0, 0, &unused, NULL); mmap_write_unlock(mm); if (IS_ERR((void *)ctx->mmap_base)) { ctx->mmap_size = 0; diff --git a/include/linux/mm.h b/include/linux/mm.h index 3f980d4823ad..6e1796ee7e1a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2897,7 +2897,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf); extern unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, - unsigned long pgoff, unsigned long *populate, struct list_head *uf); + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, + struct list_head *uf); extern int do_mas_munmap(struct ma_state *mas, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool downgrade); diff --git a/ipc/shm.c b/ipc/shm.c index bd2fcc4d454e..1c5476bfec8b 100644 --- a/ipc/shm.c +++ b/ipc/shm.c @@ -1662,7 +1662,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr = do_mmap(file, addr, size, prot, flags, 0, &populate, NULL); + addr = do_mmap(file, addr, size, prot, flags, 0, 0, &populate, NULL); *raddr = addr; err = 0; if (IS_ERR_VALUE(addr)) diff --git a/mm/mmap.c b/mm/mmap.c index 9f85596cce31..350bf156fcae 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1238,11 +1238,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode, */ unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, - unsigned long flags, unsigned long pgoff, - unsigned long *populate, struct list_head *uf) + unsigned long flags, vm_flags_t vm_flags, + unsigned long pgoff, unsigned long *populate, + struct list_head *uf) { struct mm_struct *mm = current->mm; - vm_flags_t vm_flags; int pkey = 0; validate_mm(mm); @@ -1303,7 +1303,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, * to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags = calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | + vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; if (flags & MAP_LOCKED) @@ -2877,7 +2877,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, file = get_file(vma->vm_file); ret = do_mmap(vma->vm_file, start, size, - prot, flags, pgoff, &populate, NULL); + prot, flags, 0, pgoff, &populate, NULL); fput(file); out: mmap_write_unlock(mm); diff --git a/mm/nommu.c b/mm/nommu.c index 5b83938ecb67..3642a3e01265 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1042,6 +1042,7 @@ unsigned long do_mmap(struct file *file, unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf) @@ -1049,7 +1050,6 @@ unsigned long do_mmap(struct file *file, struct vm_area_struct *vma; struct vm_region *region; struct rb_node *rb; - vm_flags_t vm_flags; unsigned long capabilities, result; int ret; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); @@ -1069,7 +1069,7 @@ unsigned long do_mmap(struct file *file, /* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags = determine_vm_flags(file, prot, flags, capabilities); + vm_flags |= determine_vm_flags(file, prot, flags, capabilities); /* we're going to need to record the mapping */ diff --git a/mm/util.c b/mm/util.c index b56c92fb910f..77867bf9959a 100644 --- a/mm/util.c +++ b/mm/util.c @@ -517,7 +517,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, if (!ret) { if (mmap_write_lock_killable(mm)) return -EINTR; - ret = do_mmap(file, addr, len, prot, flag, pgoff, &populate, + ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate, &uf); mmap_write_unlock(mm); userfaultfd_unmap_complete(mm, &uf); From patchwork Thu Jan 19 21:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46001 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557275wrn; Thu, 19 Jan 2023 13:39:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXua5vr5a1foTjkwezxM4Z1/KJL5vekTrGeJf58KkR4qUbd0xmz5jyGsY0ZIc/e7eVm7X3B2 X-Received: by 2002:a05:6402:1748:b0:497:c96b:4ded with SMTP id v8-20020a056402174800b00497c96b4dedmr11879881edx.34.1674164367852; Thu, 19 Jan 2023 13:39:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164367; cv=none; d=google.com; s=arc-20160816; b=loBVLZDkI0OcUghQrcPKHjzfC1T0fCoziwCV7u8MN/nKmRwP6i709hnkOqMXaa8b2D gN9xFgJRv+xHfpr6Rnqqkd2vcXKfgonTK8n3L3H840VpcR5s6MEYwTCT1DMFCfd6icrF afOnwmf0CmeeU1jOA5MhQ3osFcHXmnxEEY1Qt3zT2nfW/mw+6slbHidEam7L917TJf6/ NZWZSk3pfO9KvaexltYTCUzSuT4KvMcu8YfuhCWeqUGqYKE+SBNWaHoc0e4cDwJTcZa1 9bqIzIt8sFAJi6/OTBUE3QsFVx0yv2kyNQTxebBSbNpdVMQsySe3C6VPj8YpGZ4B5Fke CEGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=o/4m81j8xE9qAXKmMhKAmN4IZZXNyomiS4bHPHlONRs=; b=F+6MG1vKE5OdRScZCIi0I2dUDqmG7CpjFuizvCEfBVnJdT35t70YvTWxvakHRcXPH7 qpQb5Ubampp8Osr+COVNd9dmU1oVAk7tnt22zA+pAP6d6axjFet8UWMJub/EYMeFkm3J AjX6dcNBQ+gb3BGjOV4vf43MzEh5+dx6fy6BeLE/+2rHodUx8T42ybo9t7BpQkAgXjY+ aBw9GbjFNJv9hNLZ91V3+MxWYei4fJ+MvFFYOn9Zrgy4aeBj+wMt8nyeBKrHIoYnDqmo hTGEH08F7O9rmdTPQ3kOjdv9QO2uo+hXBEhgl0GDLhcqMfsNXgWpu5nN0WbfKmWwuvwu AbqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YqFLZxTy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z5-20020a056402274500b0049e04fbb455si5026437edd.40.2023.01.19.13.39.02; Thu, 19 Jan 2023 13:39:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YqFLZxTy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230333AbjASVih (ORCPT + 99 others); Thu, 19 Jan 2023 16:38:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230360AbjASVgk (ORCPT ); Thu, 19 Jan 2023 16:36:40 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4470470BB; Thu, 19 Jan 2023 13:27:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163657; x=1705699657; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=yljn5XVXAdgu3Q2GcXJK+3SbP1Csl5qMglVIozskdYk=; b=YqFLZxTyIsujYz3gN3vQEMR70Y4gSG5cGHBogLBhl17d1BnQkjBddnZ8 h5GrJV4tRpPIZwcXuv/+ZMCNcK9hAv3npLz8E7v6jI2pTv6j0AweU47BN weTM09pUXNaDWzi+sVZkQeMa4Nix2JwqLHgjg4qD+YnHe9rJhgYJe/rJr bKohICN8hImR7tZr994a/6wa9+2nCOlcJXBgpSS7yTgQdowD/2otxTpXz C09Hu2Ulro3gG5d5XKQBMWXlER2N/VJ2ChL9vXzA8tkMEJA0rrAirQuxX NRVWr+ewtf5hmdJstlI/CENHvZZXGpEvlvj6HEBX9LtZ+8hRpntbpyYaa A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119716" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119716" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139111" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139111" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:00 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory Date: Thu, 19 Jan 2023 13:23:01 -0800 Message-Id: <20230119212317.8324-24-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488576161892747?= X-GMAIL-MSGID: =?utf-8?q?1755488576161892747?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs. Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe --- v3: - Add comment in __pte_access_permitted() (Dave) - Remove unneeded shadow stack specific check in __pte_access_permitted() (Jann) arch/x86/include/asm/pgtable.h | 5 +++++ mm/gup.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 87d3068734ec..425ded5dd6ec 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1671,6 +1671,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write) { unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER; + /* + * Write=0,Dirty=1 PTEs are shadow stack, which the kernel + * shouldn't generally allow access to, but since they + * are already Write=0, the below logic covers both cases. + */ if (write) need_pte_bits |= _PAGE_RW; diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..bfd33d9edb89 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -982,7 +982,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) return -EFAULT; if (write) { - if (!(vm_flags & VM_WRITE)) { + if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) { if (!(gup_flags & FOLL_FORCE)) return -EFAULT; /* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */ From patchwork Thu Jan 19 21:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46004 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558126wrn; Thu, 19 Jan 2023 13:42:01 -0800 (PST) X-Google-Smtp-Source: AMrXdXvlgROCl8rLuKERhx6pXUQaonjoEtYDvGOh+lf2dkOBiYlfJTmBSH+Om/P126WmxjYtaEtQ X-Received: by 2002:a17:906:816:b0:871:dd2:4af3 with SMTP id e22-20020a170906081600b008710dd24af3mr12962378ejd.21.1674164521434; Thu, 19 Jan 2023 13:42:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164521; cv=none; d=google.com; s=arc-20160816; b=RJbHF3JAWkFXi65hQ7DTpZgdaSP3erq4gXyHkF8BdUEwt45Hj8I6jjiV8LnwgKbJUu Ax/R/0vVzKI4kf6X2KT9fD8NFeTIAOwNSW9TPTStdo0KqZyOMcVLqQxwro8XW3AbZk5R 9D+z8fNidaXh4/jiAvspuuygonbvoEOONnXng4wame8qpKxVt/68BJeITSpgRsa4sjbv V6mUIhSj7FWJupJrY3lYauByo6Sc7ng9WqU53IH4a0X2LcE50xoWm21W0V1W1Yf4vElk TZ0DnfEwDDfXejE5eU7+xOrvqM+O6f5+iNOc+3NZcDELjpXZ91bVE2s5Zr7WGACCkw2F XQfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=kAPK2G43HTRXIoPjAhCuhjvThw+Gk9Azt2qL3QWNxr4=; b=uCnN+AQjBdJ22e/E5xxmifAPo5cUFDzEo7kw2rA0ZvccNA4hpKvZdCj9ATzv99/xj/ PGbb8PioyXBn8ecfYpscY6/29OfQtpCWgcHcKcvgyYI1AjHRpRWMJGuw0/aGuBjFF4KF J2s3gQR6krqWkXDS1+ltnbp1+ImGgWsOjfbNOBIbr+yvXIZLqibcjaerDDH8D5pHV/qP AqWj08RcvFId4k0J0O7Yqg1JjqWG2C0FL/uI9gRxRkuuHHD1i4lMGKr/u0keffIm3veg AZc0NxwfNCH7SiWJ57BhCdghfbjOyxbfz4O1k8K6r0QDME5xAIpYbgDpD2BVnVZmjaDq VFEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RdZYkr2n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jg10-20020a170907970a00b0085d97b9df6esi26515936ejc.669.2023.01.19.13.41.37; Thu, 19 Jan 2023 13:42:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RdZYkr2n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231196AbjASVjI (ORCPT + 99 others); Thu, 19 Jan 2023 16:39:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230408AbjASVh1 (ORCPT ); Thu, 19 Jan 2023 16:37:27 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BC2DB747; Thu, 19 Jan 2023 13:27:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163657; x=1705699657; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=DRhmUKmYGXqIeL9sLHmMRoFoAPvudeDqkB4y2AZ6uIw=; b=RdZYkr2n5BxVtAFGqpX+mnPbHIq+wZyBdpb81fmD1NWsbN63coKldBOF eqwEuC7Mf6jq79i1Tx5zn1Zp/2mwalAcbIctODvfk52hwlPzvOodmgFlo clKEJsDdqRZCKN98AdMBFhGH/8GdTPgMtb6FroITwpP1xkQ3jXjYFRsmT e3A0o/wtL95B8TnvajQpTwGNp2aZ7XnktsfJOs9u80FZd+jptra+CqHSs fJMGuy+EHcadOotvygoCeASjQCk8fhm7pbrfVR9UpgepLrT6nvzhGtN4s V/rnUYAUvNqbzuzgFbXZ1l2day8LUv/VvBXxgqaEIPdgPe2fVDxurZIrh w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119743" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119743" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:03 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139117" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139117" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:01 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 24/39] x86/mm: Introduce MAP_ABOVE4G Date: Thu, 19 Jan 2023 13:23:02 -0800 Message-Id: <20230119212317.8324-25-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488737056621692?= X-GMAIL-MSGID: =?utf-8?q?1755488737056621692?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly. One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals. On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a general protection fault which will in turn trigger a segfault if it tries to transition to 32 bit mode with shadow stack enabled. This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by. So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though. If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall. Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's own randomization in the bottom up case. For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G both are potentially valid, so both are used. In the bottomup search path, the default behavior is already consistent with MAP_ABOVE4G since mmap base should be above 4GB. Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB. So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode with shadow stack enabled would usually segfault anyway. This is already pretty decent guard rails. But the addition of MAP_ABOVE4G is some small complexity spent to make it make it more complete. Signed-off-by: Rick Edgecombe --- v5: - New patch arch/x86/include/uapi/asm/mman.h | 1 + arch/x86/kernel/sys_x86_64.c | 6 +++++- include/linux/mman.h | 4 ++++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h index 775dbd3aff73..5a0256e73f1e 100644 --- a/arch/x86/include/uapi/asm/mman.h +++ b/arch/x86/include/uapi/asm/mman.h @@ -3,6 +3,7 @@ #define _ASM_X86_MMAN_H #define MAP_32BIT 0x40 /* only give out 32bit addresses */ +#define MAP_ABOVE4G 0x80 /* only map above 4GB */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS #define arch_calc_vm_prot_bits(prot, key) ( \ diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 8cc653ffdccd..06378b5682c1 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.flags = VM_UNMAPPED_AREA_TOPDOWN; info.length = len; - info.low_limit = PAGE_SIZE; + if (!in_32bit_syscall() && (flags & MAP_ABOVE4G)) + info.low_limit = 0x100000000; + else + info.low_limit = PAGE_SIZE; + info.high_limit = get_mmap_base(0); /* diff --git a/include/linux/mman.h b/include/linux/mman.h index 58b3abd457a3..32156daa985a 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -15,6 +15,9 @@ #ifndef MAP_32BIT #define MAP_32BIT 0 #endif +#ifndef MAP_ABOVE4G +#define MAP_ABOVE4G 0 +#endif #ifndef MAP_HUGE_2MB #define MAP_HUGE_2MB 0 #endif @@ -50,6 +53,7 @@ | MAP_STACK \ | MAP_HUGETLB \ | MAP_32BIT \ + | MAP_ABOVE4G \ | MAP_HUGE_2MB \ | MAP_HUGE_1GB) From patchwork Thu Jan 19 21:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46003 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557495wrn; Thu, 19 Jan 2023 13:40:10 -0800 (PST) X-Google-Smtp-Source: AMrXdXuTa+QYJvi7LNBjGQJgCmNhl6oUmZYvIjBYZd9vnwt+k4MAJpPcngo1Ha/8ezv3HkCroRWZ X-Received: by 2002:a17:906:e0c5:b0:86d:67ee:d607 with SMTP id gl5-20020a170906e0c500b0086d67eed607mr22891270ejb.64.1674164410753; Thu, 19 Jan 2023 13:40:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164410; cv=none; d=google.com; s=arc-20160816; b=DaEbpYNhF8veRB3l0x0HlecPxQwJEXAIHBBnHVqCdIL+M8UQERHZWOiep50RPaIyX0 /iy5eD0RTR70XSMNRg5cPp/IM3B03jlgnN/1SG1Ap/pfIa/sJAYXXOuTIbNXeoy81q0B 5Rcgf1YwUNb1sUsCmjlTTIOo9w1M3MEA3+r1PjpFotfSjkPnOuJHnF4eUHDqq0I1shup 6mZ9UiIMWVcTNF7FuDaeGFemSKZATfrfd6UhivjSK5QrDOSdzLJU3DBdeXEVDnvo84aM AU6pW9QGfAijOb44j5LKlwjbiq7FlnXc2HnpFPdIg2I3+yScKsLFneEj2RSvzRPV6eO7 lLYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=XA8Iioxyvl6Xy5Wa5VtmFxF3mszcNtBcOHPIPzhymEA=; b=DWWQsJ0LpdqWrKt60+XnRnKbxEzGs+7SDepuBwdbLdhixvXpeSSKHIKwNnZiIQDg1N R07L3ZShVlIqIiN+WMpTu2Dum7CmjSqUHRT65uW2GVwBOL4gnOMRN40Cc/1FHWZ5TPV7 IBZUrdRCUzUUs9h2ZpSt/MJZ98J3whdnKJph98TK2ebr0D2Dx5PhWyNnoyJZkiBo4ysy ax+Jkcnq2GOJ/5aj9gYcaZirSnA0zp2wTbxJXUvHXTy+Rr4xsEg1qoz+8V8eWmvSYUzx NRQlUZBahCIuZSeomuM6e7EJ9IE6xJ1/pWodToOsFHO3E9dGdPrZZyINVTvlnX0KR4wj Vscw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="B8xsrRM/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n14-20020a05640205ce00b00483b5bcc9c3si21458900edx.616.2023.01.19.13.39.46; Thu, 19 Jan 2023 13:40:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="B8xsrRM/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231191AbjASVjB (ORCPT + 99 others); Thu, 19 Jan 2023 16:39:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231128AbjASVh1 (ORCPT ); Thu, 19 Jan 2023 16:37:27 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2EDF1E1C8; Thu, 19 Jan 2023 13:27:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163657; x=1705699657; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=7r4JBhnxBM6qBcyW6ndYNR0sg/Hfe6OTwhDLlSBw690=; b=B8xsrRM/Vktn8dyxr859PtnNes3HxgBrt5XXeG7q/fI+y+66KX8EQytD cRVfBmldMk2eI0ef50IRzsNAcK1dbdexQjORESHjbLBNaAVDtPDJ7M7n0 hLfQskMfOqkaawFNqm3Y/hh2keQ2K9pNmMGCdHDCC9WWOmVlnmvGVP4nC 1sre+ZQ/LBZl3zEj9D/uYs/Th6GaqID8hdYz65vEV03YnQpPs9NffadiG YzxJxXW2XVwoshgbfxGr5nIcXrNXdqBXFU30DFGA9W+UKxs7gmwo8pID6 GxUZMM8UBuJkvU2LKQks4SgEzfbVDnu8WQvU/PAnXw9XyeBg1HddZ06XB A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119771" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119771" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:04 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139124" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139124" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:03 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 25/39] mm: Warn on shadow stack memory in wrong vma Date: Thu, 19 Jan 2023 13:23:03 -0800 Message-Id: <20230119212317.8324-26-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488621062101131?= X-GMAIL-MSGID: =?utf-8?q?1755488621062101131?= The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly. One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured. In order to check if a pte is shadow stack in core mm code, add default implementations for pte_shstk() and pmd_shstk(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix typo in commit log v3: - New patch arch/x86/include/asm/pgtable.h | 2 ++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 2 ++ mm/memory.c | 2 ++ 4 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 425ded5dd6ec..356f1d43e403 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -129,6 +129,7 @@ static inline bool pte_dirty(pte_t pte) return pte_flags(pte) & _PAGE_DIRTY_BITS; } +#define pte_shstk pte_shstk static inline bool pte_shstk(pte_t pte) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) @@ -147,6 +148,7 @@ static inline bool pmd_dirty(pmd_t pmd) return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } +#define pmd_shstk pmd_shstk static inline bool pmd_shstk(pmd_t pmd) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 49ce1f055242..04d0bc466e43 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -539,6 +539,20 @@ static inline pte_t pte_mkwrite_shstk(pte_t pte) } #endif +#ifndef pte_shstk +static inline bool pte_shstk(pte_t pte) +{ + return false; +} +#endif + +#ifndef pmd_shstk +static inline bool pmd_shstk(pmd_t pte) +{ + return false; +} +#endif + #ifndef pmd_mkwrite_shstk static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index fbb8beb9265e..5bd71da75dec 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1700,6 +1700,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(orig_pmd)); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index 5e5107232a26..c4cc38baffc5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1381,6 +1381,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(ptent)); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); From patchwork Thu Jan 19 21:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46002 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp557463wrn; Thu, 19 Jan 2023 13:40:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXuhD5220NO0OPfePo8xG8x1ac+dL24KriXZJYr9YnI8osaoxPtCP+L7Qxga0tXTFhLAVx5u X-Received: by 2002:a17:907:c307:b0:84c:c121:dc53 with SMTP id tl7-20020a170907c30700b0084cc121dc53mr13931625ejc.34.1674164404392; Thu, 19 Jan 2023 13:40:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164404; cv=none; d=google.com; s=arc-20160816; b=gdY0jJu6grPFd9wmy4RjqZUMV9apKM824BTQtdqY7Ouu138oktWWG/0bLluklqt6LQ wmuemzUZD6FAbC0Zv/y5K45ECIFtc/tD9k5tJm+VDg8xHl6Q4ZtZ9ffHCO9lBBXyZoEh eq30N52XdhEHRnRryimhn3UKP8LpxqDJtOzy9uIgQnT3rC8xN6JatUydxv+TRS1T+Oe6 asHX4voiKU0I+pGRtITmazjEA6EPb59em2hWanmANAxHFnEDO1LJ6cn6K94oxD+y3yLD m+aNWlshjAXLynZkDANl8ytYiUOTWRxACeB7qdTizTgzF1cLgYTn6lDjq1FR6Vh0w+Ev hJKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=oh2+crOjRcnFhQrw+0Ej8TqzGPwj3bIwotS5/bXBRpE=; b=K8q8YHNhs6eYNhi9egeeixTP+iiC1dVf1St7I3NnqDOlZuS233AD6Fy0C/rrxAirAh pz7jF1a/Y5Io3MNVaRmggNTSvi4okoX/7xXXTYNqB3YkpSuw9TtvNqxe9duoV95LmxxF x2yikfITIUmP8eVu29l1cSJ97bFuWQ/D48TMu39yWdztnhoMcWYtERSDQjE8LcICq+Iu yYISFb3EC3+WlWjDM7mrKAK8hTXxyFv5H/WrlUgi29qJgFA2oVdncmyEYTSrLLz8LlDR 7AvEmV78UrZG96DfB0EIbdX/y7+KrP0EWLuayhqDEXD2tHhzVE0b/jpo6S+v7b2qUi1h DVCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bRJal37k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dd3-20020a1709069b8300b008775fc70972si5589559ejc.186.2023.01.19.13.39.40; Thu, 19 Jan 2023 13:40:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bRJal37k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231176AbjASViq (ORCPT + 99 others); Thu, 19 Jan 2023 16:38:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230361AbjASVgk (ORCPT ); Thu, 19 Jan 2023 16:36:40 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E70B35895D; Thu, 19 Jan 2023 13:27:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163657; x=1705699657; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0FS3biIis1HBrs8s6jBReO51I+7M0YdUM2Pqpl2DgjA=; b=bRJal37k0b+4NUDFF0jABbbI6/xTpb4C3pijE9I+xKGEs98CZazfdZGw uoF8yQ0q3rJ9Gv0NqsxNQqmLLDtVyrVPV/LQ9SQdCSBziAORMpvJiuV3v zoctwBgRDgqBpZrQ+FmYRggaoJDugPkVQ1+SV2HU62/QMN5mk2iFHInIB bZ9EoPSKHlTIu2JHhkp9faGOP6+XXeqUGjP/PAhZofwaU8CrjBhYiJJSc 1/R8xFnwmNzO6k6f5g4E31Q4q43jrq93AH1q/Yjop4DvWhZYi6zTlbSQM m3ZtWRSq9rRGib4zeux7vMrmKVxRUG/1RefavN3bS0HFz+dtoc0/Y9eAS w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119805" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119805" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:06 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139130" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139130" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:04 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 26/39] x86: Introduce userspace API for shadow stack Date: Thu, 19 Jan 2023 13:23:04 -0800 Message-Id: <20230119212317.8324-27-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488614400766268?= X-GMAIL-MSGID: =?utf-8?q?1755488614400766268?= From: "Kirill A. Shutemov" Add three new arch_prctl() handles: - ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error. - ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec(). This is preparation patch. It does not implement any features. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Kirill A. Shutemov [tweaked with feedback from tglx] Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v4: - Remove references to CET and replace with shadow stack (Peterz) v3: - Move shstk.c Makefile changes earlier (Kees) - Add #ifdef around features_locked and features (Kees) - Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees) - Fix typo in commit log (Kees) - Switch arch_prctl() numbers to avoid conflict with LAM v2: - Only allow one enable/disable per call (tglx) - Return error code like a normal arch_prctl() (Alexander Potapenko) - Make CET only (tglx) arch/x86/include/asm/processor.h | 6 +++++ arch/x86/include/asm/shstk.h | 21 +++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 6 +++++ arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/process_64.c | 7 ++++- arch/x86/kernel/shstk.c | 44 +++++++++++++++++++++++++++++++ 6 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/shstk.c diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 4e35c66edeb7..e0734f417273 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -28,6 +28,7 @@ struct vm86; #include #include #include +#include #include #include @@ -475,6 +476,11 @@ struct thread_struct { */ u32 pkru; +#ifdef CONFIG_X86_USER_SHADOW_STACK + unsigned long features; + unsigned long features_locked; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h new file mode 100644 index 000000000000..58f9ee675be0 --- /dev/null +++ b/arch/x86/include/asm/shstk.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SHSTK_H +#define _ASM_X86_SHSTK_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +long shstk_prctl(struct task_struct *task, int option, unsigned long features); +void reset_thread_features(void); +#else +static inline long shstk_prctl(struct task_struct *task, int option, + unsigned long features) { return -EINVAL; } +static inline void reset_thread_features(void) {} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_SHSTK_H */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..b2b3b7200b2d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,10 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +/* Don't use 0x3001-0x3004 because of old glibcs */ + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 92446f1dedd7..b366641703e3 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -146,6 +146,8 @@ obj-$(CONFIG_CALL_THUNKS) += callthunks.o obj-$(CONFIG_X86_CET) += cet.o +obj-$(CONFIG_X86_USER_SHADOW_STACK) += shstk.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..71094c8a305f 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip, load_gs_index(__USER_DS); } + reset_thread_features(); + loadsegment(fs, 0); loadsegment(es, _ds); loadsegment(ds, _ds); @@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - + case ARCH_SHSTK_ENABLE: + case ARCH_SHSTK_DISABLE: + case ARCH_SHSTK_LOCK: + return shstk_prctl(task, option, arg2); default: ret = -EINVAL; break; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c new file mode 100644 index 000000000000..41ed6552e0a5 --- /dev/null +++ b/arch/x86/kernel/shstk.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * shstk.c - Intel shadow stack support + * + * Copyright (c) 2021, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include + +void reset_thread_features(void) +{ + current->thread.features = 0; + current->thread.features_locked = 0; +} + +long shstk_prctl(struct task_struct *task, int option, unsigned long features) +{ + if (option == ARCH_SHSTK_LOCK) { + task->thread.features_locked |= features; + return 0; + } + + /* Don't allow via ptrace */ + if (task != current) + return -EINVAL; + + /* Do not allow to change locked features */ + if (features & task->thread.features_locked) + return -EPERM; + + /* Only support enabling/disabling one feature at a time. */ + if (hweight_long(features) > 1) + return -EINVAL; + + if (option == ARCH_SHSTK_DISABLE) { + return -EINVAL; + } + + /* Handle ARCH_SHSTK_ENABLE */ + return -EINVAL; +} From patchwork Thu Jan 19 21:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46005 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558153wrn; Thu, 19 Jan 2023 13:42:06 -0800 (PST) X-Google-Smtp-Source: AMrXdXvHb7lGHoxbP8Xoa9N9IWcpnTbaJu5Ite+ZMh3qK/JHTMBHD58bUajWjLhjL8A4GaJm35no X-Received: by 2002:aa7:cd59:0:b0:494:912d:303b with SMTP id v25-20020aa7cd59000000b00494912d303bmr11605781edw.17.1674164526074; Thu, 19 Jan 2023 13:42:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164526; cv=none; d=google.com; s=arc-20160816; b=NZZlCxfx9Q96wTLEsAw+jRKQKCXzJw38OJLNAQn55UDRUT2DUhHiE+uI05Ex+0jMdZ a3CRw4vPQjAw6ZtR1B7YlbjZ6ZL0XDBUW2Yh2lzjI1qtuG/Hd1QqekWmlMeacuE2VZUV v74uWf/oHlxDYuCj8p4JLjZ3G0Isuj0EMLqhnX6hdmr8O7TRksZpd8BTKfBkc/BgivRz z6BqikAItd/qqmU2gsUYd4MbuHcIXmP67wWVooQoinaT3sKUPSf2TSpNZN1sSjwIyUzN H0DN4w58OZRX6iXsxFOAjPrWiJr/yNgc5EussW618y1RmCdzIuMJdAEOLYDI7tFshvBW 7MJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=bs+9z9aIIF14yXerZa6rGEtRQZc7+0mOjtqR3MupuLg=; b=eD4jmdt6bpDDBPJgXh9+LU4PT8kHYliHIFdeGexxWisJQywRAG0ZW/duXqtQuUubzx elYDjhDrLoV0upG9Igbx0WMMCPPPEA3tWews7EH0LTpwaCdDOdJZSlthuROSa/yAZvJH M8nSOQOpA+pU2MPgb0swf7q/U+MW3OeDYUntT62EEi2VQrYThu7nRRE4xB3t7nUcbZJi J5SWkNh/dLbfKAMXOwSpkEBH2wvI5w+prXd2Pv2v2Am2c5lznhAGwZOLAsexzg5ULNuu AsGA8EVDOaT3Zi6Hmj/UO5PfhqpQwwidbUTGj7BqBin/tFRi9940aOHGycvUYXLvuwok OHbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YHjiJCNM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q20-20020a056402519400b0049e0fd70dcesi6167313edd.293.2023.01.19.13.41.42; Thu, 19 Jan 2023 13:42:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YHjiJCNM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231151AbjASVjk (ORCPT + 99 others); Thu, 19 Jan 2023 16:39:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231143AbjASVhd (ORCPT ); Thu, 19 Jan 2023 16:37:33 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14E7A82D43; Thu, 19 Jan 2023 13:27:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163657; x=1705699657; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Ifxi0e32c84RQATdXOs3rUNHxe57SwrjYewWnLIO6BE=; b=YHjiJCNMUpOYENSQPAKBAYTXBeVtqnA7pNj2ZWxTi7Gke0lOAEXRsZc7 bVZaIvBZ+qmzUZFqqVV15UofDuDMr+JDVogtnHxzsKW8fHuT/rBH21J5W oG1R1EGfTlZZ+J1MinLXMcvQ0qTueMgE0FPrxuHrqIG0j3+wlMt8lRc4J 8AVmeN/HQRM5uza68WS3rlnmDwkoCP8gnFHXKaa3e+mZ9IKe/LgrjKx3A I3ratEDE2TaWNsYeUufcXQS+I/Lbl9wpwo8qspxm04kGfdD3Ql7qTWZ+U yS56DSyDmSK+BhVw12cmiJcR+OWJ+R+YeSiQP9Hy3GT/niYvE9i3SQOlW A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119835" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119835" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:08 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139135" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139135" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:06 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 27/39] x86/shstk: Add user-mode shadow stack support Date: Thu, 19 Jan 2023 13:23:05 -0800 Message-Id: <20230119212317.8324-28-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488741951317158?= X-GMAIL-MSGID: =?utf-8?q?1755488741951317158?= From: Yu-cheng Yu Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB). Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this. Do not support IA32 emulation or x32. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Switch to EOPNOTSUPP - Use MAP_ABOVE4G - Move set_clr_bits_msrl() to patch where it is first used v4: - Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz) v3: - Use define for set_clr_bits_msrl() (Kees) - Make some functions static (Kees) - Change feature_foo() to features_foo() (Kees) - Centralize shadow stack size rlimit checks (Kees) - Disable x32 support v2: - Get rid of unnessary shstk->base checks - Don't support IA32 emulation arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 146 ++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index e0734f417273..3c257a1a0757 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -479,6 +479,8 @@ struct thread_struct { #ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features; unsigned long features_locked; + + struct thread_shstk shstk; #endif /* Floating point and extended processor state */ diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 58f9ee675be0..f40414a982e8 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -8,12 +8,19 @@ struct task_struct; #ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { + u64 base; + u64 size; +}; + long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long features) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index b2b3b7200b2d..7dfd9dc00509 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,4 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 41ed6552e0a5..f39e5d3b9303 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -8,14 +8,160 @@ #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +static bool features_enabled(unsigned long features) +{ + return current->thread.features & features; +} + +static void features_set(unsigned long features) +{ + current->thread.features |= features; +} + +static void features_clr(unsigned long features) +{ + current->thread.features &= ~features; +} + +static unsigned long alloc_shstk(unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; + struct mm_struct *mm = current->mm; + unsigned long addr, unused; + + mmap_write_lock(mm); + addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + + mmap_write_unlock(mm); + + return addr; +} + +static unsigned long adjust_shstk_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + return PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G)); +} + +static void unmap_shadow_stack(u64 base, u64 size) +{ + while (1) { + int r; + + r = vm_munmap(base, size); + + /* + * vm_munmap() returns -EINTR when mmap_lock is held by + * something else, and that lock should not be held for a + * long time. Retry it for the case. + */ + if (r == -EINTR) { + cond_resched(); + continue; + } + + /* + * For all other types of vm_munmap() failure, either the + * system is out of memory or there is bug. + */ + WARN_ON_ONCE(r); + break; + } +} + +static int shstk_setup(void) +{ + struct thread_shstk *shstk = ¤t->thread.shstk; + unsigned long addr, size; + + /* Already enabled */ + if (features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* Also not supported for 32 bit and x32 */ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_32bit_syscall()) + return -EOPNOTSUPP; + + size = adjust_shstk_size(0); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, addr + size); + wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN); + fpregs_unlock(); + + shstk->base = addr; + shstk->size = size; + features_set(ARCH_SHSTK_SHSTK); + + return 0; +} + void reset_thread_features(void) { + memset(¤t->thread.shstk, 0, sizeof(struct thread_shstk)); current->thread.features = 0; current->thread.features_locked = 0; } +void shstk_free(struct task_struct *tsk) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return; + + if (!tsk->mm) + return; + + unmap_shadow_stack(shstk->base, shstk->size); +} + + +static int shstk_disable(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* Already disabled? */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + fpregs_lock_and_load(); + /* Disable WRSS too when disabling shadow stack */ + wrmsrl(MSR_IA32_U_CET, 0); + wrmsrl(MSR_IA32_PL3_SSP, 0); + fpregs_unlock(); + + shstk_free(current); + features_clr(ARCH_SHSTK_SHSTK); + + return 0; +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { From patchwork Thu Jan 19 21:23:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46008 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558294wrn; Thu, 19 Jan 2023 13:42:33 -0800 (PST) X-Google-Smtp-Source: AMrXdXurhNgMw6uBj0XM2ZhYYOkGTvV4Jp9Drc2BBcJLjoYVsiSTN2foBVklBGZaC6QKFjJ0J2Lr X-Received: by 2002:a17:907:1042:b0:7c1:5863:f8c4 with SMTP id oy2-20020a170907104200b007c15863f8c4mr11781604ejb.21.1674164552911; Thu, 19 Jan 2023 13:42:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164552; cv=none; d=google.com; s=arc-20160816; b=wHs0ENqca6UbfJbGIuNaqx2h+MC6qUGJTMHjT+wkfAOueHh3rBizCuYYPoCYQ4YElK 3NfPME5RHLojD3GaNSvJ7BObhE/t+yYAd28m4WH56AGbATSen1MIzVpcAzbQ8DkBd/I/ xCOUB1Z/eKCjNoCmQwlnKsp6LWwyZXcrSL9DAwDPrmVWzq8DQsP3zHP0hjZJjgc8EqF1 LuxE66wVcumNjc9irlDLLCio5Qq8J8+OXuBAtU4I/7xKrQ1Mcc/HaUXcq+L9oxIY+g9F WGnqCBEZwWQzLkFoEz0CsEazi+IjPpLt6pPFz3x7bYtL43B8+I39Zjzu8DnGqxrwluPn ZlLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=LF9eWuGfmBXCMQNhJgKUQvTdsk+a5XWVtnwIrkM2vbU=; b=Gme1yT8ienaXIsD68YXb/XfztNfnra+lV/Yqi4K8lpFtOa+whsUugm9ust8oL+An4P ngdsaYTOuzqgoCAic6KS6EwMw7I1dkNqmN38f9Zwcb/7LNUZmgOlDNQIOHLfArysuecG YeUKykxB7uFcN35nmcUyRbdwq5UX5qqAoCV+m21E2CM4G4ZtQC1HA2k33L4oFFDpvHa/ 10K1CmmtgpEC6uwmlIJSwLV3duZGU8VBm5tNDNqPrK9nH198N2P4r7BNA3Zfx/9qTVbO 67N87I5FnBhdAkEtJ1Tnt/L4zygNOk0gI/fbfdbqyJitpFnxhS6bW1TYtaRKQFFvXOdv K/4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OHCpMTPW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cw15-20020a170906478f00b0086d67a10582si18581824ejc.805.2023.01.19.13.42.08; Thu, 19 Jan 2023 13:42:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OHCpMTPW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230021AbjASVlo (ORCPT + 99 others); Thu, 19 Jan 2023 16:41:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230500AbjASVhz (ORCPT ); Thu, 19 Jan 2023 16:37:55 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 996804616D; Thu, 19 Jan 2023 13:27:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163667; x=1705699667; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sY0pQZGQdI/cNzp/wwjwE3pcURvBjqOaRcvBZzBOV/8=; b=OHCpMTPWUmNj04Fp5VChAxn2AXun00RS8ZHkrzi26EnwqVZ4K22PGQVu TfCB/4DJxatHln6urSoUDoSkK6MobjXOHzuAatNlPPQOMt80wNWWk4u6L SPDvBcZxnuayHEuMQCuSD4tFOU80xndLSW9s7pcD+XL5PCSoorA8zkalc uFu2kslB99HkZjhpc+KPoilaoYZcf8a8OHTe1aAefs9rK/dwXomNux8Xf xaz8cezIh5UL0Ushs80Ki7CpiJYkniiZWdMUyVGB/QgZ58CvxEgkQy7sR /uTLtWF/vkZ+TunLpUzXA2ujWVfmxe1o2iiYfgmsjBmNyO0T4GIp2U/uQ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119861" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119861" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:09 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139140" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139140" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:08 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 28/39] x86/shstk: Handle thread shadow stack Date: Thu, 19 Jan 2023 13:23:06 -0800 Message-Id: <20230119212317.8324-29-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488770293827910?= X-GMAIL-MSGID: =?utf-8?q?1755488770293827910?= From: Yu-cheng Yu When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways. With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks. For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack. For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it. Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB. For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork(). Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function. Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent. During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Suggested-by: Thomas Gleixner Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe --- v3: - Fix update_fpu_shstk() stub (Mike Rapoport) - Fix chunks around alloc_shstk() in wrong patch (Kees) - Fix stack_size/flags swap (Kees) - Use centalized stack size logic (Kees) v2: - Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx) v1: - Expand commit log. - Add more comments. - Switch to xsave helpers. Yu-cheng v30: - Update comments about clone()/clone3(). (Borislav Petkov) arch/x86/include/asm/fpu/sched.h | 3 +- arch/x86/include/asm/mmu_context.h | 2 ++ arch/x86/include/asm/shstk.h | 7 +++++ arch/x86/kernel/fpu/core.c | 41 +++++++++++++++++++++++++++- arch/x86/kernel/process.c | 18 +++++++++++- arch/x86/kernel/shstk.c | 44 ++++++++++++++++++++++++++++-- 6 files changed, 110 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/fpu/sched.h b/arch/x86/include/asm/fpu/sched.h index b2486b2cbc6e..54c9c2fd1907 100644 --- a/arch/x86/include/asm/fpu/sched.h +++ b/arch/x86/include/asm/fpu/sched.h @@ -11,7 +11,8 @@ extern void save_fpregs_to_fpstate(struct fpu *fpu); extern void fpu__drop(struct fpu *fpu); -extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal); +extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal, + unsigned long shstk_addr); extern void fpu_flush_thread(void); /* diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index e01aa74a6de7..9714f08d941b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -147,6 +147,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + shstk_free(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index f40414a982e8..172a69052770 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -15,11 +15,18 @@ struct thread_shstk { long shstk_prctl(struct task_struct *task, int option, unsigned long features); void reset_thread_features(void); +int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, + unsigned long stack_size, + unsigned long *shstk_addr); void shstk_free(struct task_struct *p); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long features) { return -EINVAL; } static inline void reset_thread_features(void) {} +static inline int shstk_alloc_thread_stack(struct task_struct *p, + unsigned long clone_flags, + unsigned long stack_size, + unsigned long *shstk_addr) { return 0; } static inline void shstk_free(struct task_struct *p) {} #endif /* CONFIG_X86_USER_SHADOW_STACK */ diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 7317bfd5ea36..c72262479f03 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -552,8 +552,41 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu) } } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp) +{ + struct cet_user_state *xstate; + + /* If ssp update is not needed. */ + if (!ssp) + return 0; + + xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave, + XFEATURE_CET_USER); + + /* + * If there is a non-zero ssp, then 'dst' must be configured with a shadow + * stack and the fpu state should be up to date since it was just copied + * from the parent in fpu_clone(). So there must be a valid non-init CET + * state location in the buffer. + */ + if (WARN_ON_ONCE(!xstate)) + return 1; + + xstate->user_ssp = (u64)ssp; + + return 0; +} +#else +static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr) +{ + return 0; +} +#endif + /* Clone current's FPU state on fork */ -int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal) +int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal, + unsigned long ssp) { struct fpu *src_fpu = ¤t->thread.fpu; struct fpu *dst_fpu = &dst->thread.fpu; @@ -613,6 +646,12 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal) if (use_xsave()) dst_fpu->fpstate->regs.xsave.header.xfeatures &= ~XFEATURE_MASK_PASID; + /* + * Update shadow stack pointer, in case it changed during clone. + */ + if (update_fpu_shstk(dst, ssp)) + return 1; + trace_x86_fpu_copy_src(src_fpu); trace_x86_fpu_copy_dst(dst_fpu); diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index e57cd31bfec4..13a0a81d70b9 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "process.h" @@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + shstk_free(tsk); fpu__drop(fpu); } @@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) struct inactive_task_frame *frame; struct fork_frame *fork_frame; struct pt_regs *childregs; + unsigned long shstk_addr = 0; int ret = 0; childregs = task_pt_regs(p); @@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) frame->flags = X86_EFLAGS_FIXED; #endif - fpu_clone(p, clone_flags, args->fn); + /* Allocate a new shadow stack for pthread if needed */ + ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size, + &shstk_addr); + if (ret) + return ret; + + fpu_clone(p, clone_flags, args->fn, shstk_addr); /* Kernel thread ? */ if (unlikely(p->flags & PF_KTHREAD)) { @@ -220,6 +229,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) if (!ret && unlikely(test_tsk_thread_flag(current, TIF_IO_BITMAP))) io_bitmap_share(p); + /* + * If copy_thread() if failing, don't leak the shadow stack possibly + * allocated in shstk_alloc_thread_stack() above. + */ + if (ret) + shstk_free(p); + return ret; } diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index f39e5d3b9303..111ea56115d2 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -47,7 +47,7 @@ static unsigned long alloc_shstk(unsigned long size) unsigned long addr, unused; mmap_write_lock(mm); - addr = do_mmap(NULL, addr, size, PROT_READ, flags, + addr = do_mmap(NULL, 0, size, PROT_READ, flags, VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); mmap_write_unlock(mm); @@ -126,6 +126,40 @@ void reset_thread_features(void) current->thread.features_locked = 0; } +int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, + unsigned long stack_size, unsigned long *shstk_addr) +{ + struct thread_shstk *shstk = &tsk->thread.shstk; + unsigned long addr, size; + + /* + * If shadow stack is not enabled on the new thread, skip any + * switch to a new shadow stack. + */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + /* + * For CLONE_VM, except vfork, the child needs a separate shadow + * stack. + */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) != CLONE_VM) + return 0; + + + size = adjust_shstk_size(stack_size); + addr = alloc_shstk(size); + if (IS_ERR_VALUE(addr)) + return PTR_ERR((void *)addr); + + shstk->base = addr; + shstk->size = size; + + *shstk_addr = addr + size; + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; @@ -134,7 +168,13 @@ void shstk_free(struct task_struct *tsk) !features_enabled(ARCH_SHSTK_SHSTK)) return; - if (!tsk->mm) + /* + * When fork() with CLONE_VM fails, the child (tsk) already has a + * shadow stack allocated, and exit_thread() calls this function to + * free it. In this case the parent (current) and the child share + * the same mm struct. + */ + if (!tsk->mm || tsk->mm != current->mm) return; unmap_shadow_stack(shstk->base, shstk->size); From patchwork Thu Jan 19 21:23:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46010 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558497wrn; Thu, 19 Jan 2023 13:43:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXusk95G14AHoFuNIOitiFK9t1JeqITxBqXqlNGK1NBbb9VmUF5NTjRD5r/UHx4QXcI8u3y1 X-Received: by 2002:a50:ed91:0:b0:48e:c073:9453 with SMTP id h17-20020a50ed91000000b0048ec0739453mr27607070edr.15.1674164593082; Thu, 19 Jan 2023 13:43:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164593; cv=none; d=google.com; s=arc-20160816; b=oTiusceJZ2+fNbNKAQjYIHxT1RQsC+V5q+FAGwmPAbrUHf45prMltzAR4HmsD83dGc M+atRhEp5SmeLaP6Rm2/IGG0nmJFZT8c9JmTl1oN1Vud8GfCzx8v/wpdtnSNwuqNbk48 njsRzD0HXHNtynfbIQ4uETe7MjCFq/TOsFrhp9DDy6BeKO1H6XwwYDBvehq+iKxH7qI2 TA7pLSWJBii1hxpOfKpPfmg31T/ptq0/hQ7DCQSI3uQmC0MGdOxGV5jPAex0RlyE7v/C /8jKcg4PFKJT8TT1Cz0soZkghlZePFwRSQNxPmWiWQt6Wrnfd8t4BOfBlqzLw6WfGIR4 QKSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=5Fw8vcaOAUH/Ghj13uHxxsRffcwnoPq1Ec7oLYAhwu4=; b=JeHHL2IIvfoZaumOFdUGYZXugO2KMpXmkBvlgRdL/j7FITirpo5Vefxyoczal8S9D6 Yv4rt8yOZgthlFRppCSL2GLDb0hO87rgKN/OQZ+UK4KvkljYRgsOooyqEXFUTd7PvfX7 qZx4QS8fPdWruw3u/c8XjXe+bMnNGOT/RCMuaaH+OlFmQv78nR8FGTP3uLWpAOPOB7U/ hguYbmw0yBtPC/hS2JDmvcNkQdDnfsL9u60PyWINPq+p51W4R6rYMqKBkJYBmE0oGz5y nhbOQcvMM1urzocplQFNGk4C5U0ncdy+vcJbOND2z4K8T7SLKGU8XrtCldXmc4hJg1AO /Qkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lhcS+wGS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fd1-20020a056402388100b00469bac8510csi36997601edb.583.2023.01.19.13.42.47; Thu, 19 Jan 2023 13:43:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lhcS+wGS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230375AbjASVly (ORCPT + 99 others); Thu, 19 Jan 2023 16:41:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjASViD (ORCPT ); Thu, 19 Jan 2023 16:38:03 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 275AB525E; Thu, 19 Jan 2023 13:27:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163670; x=1705699670; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JqwiNktiNn9E4briGN+4X4+lTVwiARdl/uQ7DJ6w35c=; b=lhcS+wGScBnsFpoFGYg89PyRxKXos8P6FevUMaBJu95B+gSFng+3NF5o RFJqiJ41LDi7VQpiDZS5yFmOf4iuDSeOsy59kt27AYMDMbEZQ2dZYlmQh bZkxKIzDuFl6ofITjc7nM2+d6ZNb8mklMm6jgvnVOnm8GkaHL0QN1jBxO BfgYuHHr3uJ31LcF66HqNnIJVHuJkJ8rMiWVtiZyBfKa9SLc6rK6TrH2u LzmVDqNEUzo+Brvn0GmGWtkhw3vEvmygR2D1pAsYB9WeSc/EEneJRgF+n jLb26jG9cZWnR8tp0+TVmR+D1MXy5Sd9RMRgJOIb1/dtzzmD+1AL4LVTj w==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119884" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119884" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:11 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139145" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139145" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:09 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 29/39] x86/shstk: Introduce routines modifying shstk Date: Thu, 19 Jan 2023 13:23:07 -0800 Message-Id: <20230119212317.8324-30-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488811994635236?= X-GMAIL-MSGID: =?utf-8?q?1755488811994635236?= From: Yu-cheng Yu Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction. A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area. Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack. In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks. To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Kees Cook Reviewed-by: Kees Cook --- v5: - Fix typo in commit log v3: - Drop shstk_check_rstor_token() - Fail put_shstk_data() if bit 63 is set in the data (Kees) - Add comment in create_rstor_token() (Kees) - Pull in create_rstor_token() changes from future patch (Kees) v2: - Add data helpers for writing to shadow stack. v1: - Use xsave helpers. arch/x86/include/asm/special_insns.h | 13 +++++ arch/x86/kernel/shstk.c | 73 ++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index de48d1389936..d6cd9344f6c7 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static inline int write_user_shstk_64(u64 __user *addr, u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); + return 0; +fail: + return -EFAULT; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + #define nop() asm volatile ("nop") static inline void serialize(void) diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 111ea56115d2..3e470917eb0b 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -25,6 +25,8 @@ #include #include +#define SS_FRAME_SIZE 8 + static bool features_enabled(unsigned long features) { return current->thread.features & features; @@ -40,6 +42,35 @@ static void features_clr(unsigned long features) current->thread.features &= ~features; } +/* + * Create a restore token on the shadow stack. A token is always 8-byte + * and aligned to 8. + */ +static int create_rstor_token(unsigned long ssp, unsigned long *token_addr) +{ + unsigned long addr; + + /* Token must be aligned */ + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + addr = ssp - SS_FRAME_SIZE; + + /* + * SSP is aligned, so reserved bits and mode bit are a zero, just mark + * the token 64-bit. + */ + ssp |= BIT(0); + + if (write_user_shstk_64((u64 __user *)addr, (u64)ssp)) + return -EFAULT; + + if (token_addr) + *token_addr = addr; + + return 0; +} + static unsigned long alloc_shstk(unsigned long size) { int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; @@ -160,6 +191,48 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, return 0; } +static unsigned long get_user_shstk_addr(void) +{ + unsigned long long ssp; + + fpregs_lock_and_load(); + + rdmsrl(MSR_IA32_PL3_SSP, ssp); + + fpregs_unlock(); + + return ssp; +} + +static int put_shstk_data(u64 __user *addr, u64 data) +{ + if (WARN_ON_ONCE(data & BIT(63))) + return -EINVAL; + + /* + * Mark the high bit so that the sigframe can't be processed as a + * return address. + */ + if (write_user_shstk_64(addr, data | BIT(63))) + return -EFAULT; + return 0; +} + +static int get_shstk_data(unsigned long *data, unsigned long __user *addr) +{ + unsigned long ldata; + + if (unlikely(get_user(ldata, addr))) + return -EFAULT; + + if (!(ldata & BIT(63))) + return -EINVAL; + + *data = ldata & ~BIT(63); + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; From patchwork Thu Jan 19 21:23:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46012 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558782wrn; Thu, 19 Jan 2023 13:44:01 -0800 (PST) X-Google-Smtp-Source: AMrXdXthaK3G4uY+BRk2r3E0BkhY3rk7A7avkbaNqUDCZDOsa+47QN90UwqTXX82Qg/5s6CIhvos X-Received: by 2002:a17:906:7e14:b0:86e:2c11:9bce with SMTP id e20-20020a1709067e1400b0086e2c119bcemr12180893ejr.38.1674164641556; Thu, 19 Jan 2023 13:44:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164641; cv=none; d=google.com; s=arc-20160816; b=hPY2DYWS+H5f09wqeYC6VY5NI+ZNueAcHVV1MHU2YObVMGqeX/dVOFn8h7Hg2PLVJo u69oYN2Z6mTWEjDud2qFZn4o0hSj0m42V7T+7eUo8EZ24L1jL1SvZUTNBO5Gr4gbZYcF TJrvyjkKv6W7KGNOkFvtyBv4C2H6fr1CX9Xr0sbNLWrtjft07/PALTU68uisvd+bdK9L uBJiN7iY6ubTk8VmHqqwQqlmd4ZW8Kw6x/zsnpaAHZIAhbpWPwej7Y8SjFSRQFcCHtMy zTfRIlabvgBoH4yoHMjGWQn5op1tR7L4vsU/AagRrV5qQS2UvCU6Zf/ldkZY+tgSl/HW B6QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=BfytEEK/+VcJ5OwhEeIY6zyWKiTF9926RCmcxKRzpw0=; b=VXGDga+0rAbglafvv3x+V2ML+JCpkr3+6wn8WiBiPbjgaRuH5q9h9QId7sxBo8szBt hHO8MKMEPCa49RCvaU/cER0BXPNNjcdFFKDVCXCeoYrbXZ8bN8aehKP5Xj51GVh3OBLl jGCEW6TQJwDI9RgKiEt0DqSzICbj3Sb5e7Ovu7iOGzAdsYr5Dlu8PUV0rm1GLqjYi3I6 vKLMUZ7mUnvTDvaffIgwHLK7gz/nfY0Nth5lHC+K2VafYnCS5PXsWP8CfYlUxOb1hBbA mdt4v6O07PYtXua2nnZKLLs7vJvtl6IUHh2Gdbbz98Hng6jlNehjIAoUlcyQR+G4yl9W iLgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dEwq4k1K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hw9-20020a170907a0c900b00877919193a7si359864ejc.189.2023.01.19.13.43.36; Thu, 19 Jan 2023 13:44:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dEwq4k1K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230226AbjASVmT (ORCPT + 99 others); Thu, 19 Jan 2023 16:42:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230318AbjASViL (ORCPT ); Thu, 19 Jan 2023 16:38:11 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA11E22787; Thu, 19 Jan 2023 13:27:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163673; x=1705699673; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9f0c+DcTpRuupUlQlGr5FM1/mXL/cB4F6Skj1zH+SWI=; b=dEwq4k1KLp/Mlmt7C8S7BZHazE0LNGPTCuD0sGOAwzT/DWZ4LF+6qiBm DKlV48LW7lFtq/vAb6c4BtTyTbvv389f8ndZvt2LLX1Fr689/D6OV9byC cAe3aJx3bJkkxZXsy1u8Rum/CUliJk2hY3GfAhv7E5VdWVySzbr0GlgJk gAOdfI/BfU1YRnef06I2zdxaVLd8W40FwqczLDblRWn76PKG+P/HX5a4V ZaPxo0jo2cFZjTZxcrbGcJsjAsxM7oCUqUgCWPKT9tl+KXYLnpaADGfM/ DsCGrIuGAtmPcyqasgzrq2aOzAT+HbysmDY4kvFbAlfEkOPWi2ZDpAvlM Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119905" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119905" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:12 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139150" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139150" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:11 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 30/39] x86/shstk: Handle signals for shadow stack Date: Thu, 19 Jan 2023 13:23:08 -0800 Message-Id: <20230119212317.8324-31-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488863105850939?= X-GMAIL-MSGID: =?utf-8?q?1755488863105850939?= From: Yu-cheng Yu When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks. One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer. The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks. For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack. So, when handling a signal push - the SSP pointing in the shadow stack data format - the restorer address below the restore token. In sigreturn, verify SSP is stored in the data format and pop the shadow stack. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Cc: Andy Lutomirski Cc: Cyrill Gorcunov Cc: Florian Weimer Cc: H. Peter Anvin Cc: Kees Cook --- v3: - Drop shstk_setup_rstor_token() (Kees) - Drop x32 signal support, since x32 support is dropped v2: - Switch to new shstk signal format v1: - Use xsave helpers. - Expand commit log. Yu-cheng v27: - Eliminate saving shadow stack pointer to signal context. arch/x86/include/asm/shstk.h | 5 ++ arch/x86/kernel/shstk.c | 98 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_64.c | 6 +++ 4 files changed, 110 insertions(+) diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 172a69052770..746c040f7cb6 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -6,6 +6,7 @@ #include struct task_struct; +struct ksignal; #ifdef CONFIG_X86_USER_SHADOW_STACK struct thread_shstk { @@ -19,6 +20,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size, unsigned long *shstk_addr); void shstk_free(struct task_struct *p); +int setup_signal_shadow_stack(struct ksignal *ksig); +int restore_signal_shadow_stack(void); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long features) { return -EINVAL; } @@ -28,6 +31,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long stack_size, unsigned long *shstk_addr) { return 0; } static inline void shstk_free(struct task_struct *p) {} +static inline int setup_signal_shadow_stack(struct ksignal *ksig) { return 0; } +static inline int restore_signal_shadow_stack(void) { return 0; } #endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 3e470917eb0b..56e7ca8e42cc 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -233,6 +233,104 @@ static int get_shstk_data(unsigned long *data, unsigned long __user *addr) return 0; } +static int shstk_push_sigframe(unsigned long *ssp) +{ + unsigned long target_ssp = *ssp; + + /* Token must be aligned */ + if (!IS_ALIGNED(*ssp, 8)) + return -EINVAL; + + if (!IS_ALIGNED(target_ssp, 8)) + return -EINVAL; + + *ssp -= SS_FRAME_SIZE; + if (put_shstk_data((void *__user)*ssp, target_ssp)) + return -EFAULT; + + return 0; +} + +static int shstk_pop_sigframe(unsigned long *ssp) +{ + unsigned long token_addr; + int err; + + err = get_shstk_data(&token_addr, (unsigned long __user *)*ssp); + if (unlikely(err)) + return err; + + /* Restore SSP aligned? */ + if (unlikely(!IS_ALIGNED(token_addr, 8))) + return -EINVAL; + + /* SSP in userspace? */ + if (unlikely(token_addr >= TASK_SIZE_MAX)) + return -EINVAL; + + *ssp = token_addr; + + return 0; +} + +int setup_signal_shadow_stack(struct ksignal *ksig) +{ + void __user *restorer = ksig->ka.sa.sa_restorer; + unsigned long ssp; + int err; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + if (!restorer) + return -EINVAL; + + ssp = get_user_shstk_addr(); + if (unlikely(!ssp)) + return -EINVAL; + + err = shstk_push_sigframe(&ssp); + if (unlikely(err)) + return err; + + /* Push restorer address */ + ssp -= SS_FRAME_SIZE; + err = write_user_shstk_64((u64 __user *)ssp, (u64)restorer); + if (unlikely(err)) + return -EFAULT; + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + fpregs_unlock(); + + return 0; +} + +int restore_signal_shadow_stack(void) +{ + unsigned long ssp; + int err; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || + !features_enabled(ARCH_SHSTK_SHSTK)) + return 0; + + ssp = get_user_shstk_addr(); + if (unlikely(!ssp)) + return -EINVAL; + + err = shstk_pop_sigframe(&ssp); + if (unlikely(err)) + return err; + + fpregs_lock_and_load(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + fpregs_unlock(); + + return 0; +} + void shstk_free(struct task_struct *tsk) { struct thread_shstk *shstk = &tsk->thread.shstk; diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 004cb30b7419..356253e85ce9 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -40,6 +40,7 @@ #include #include #include +#include static inline int is_ia32_compat_frame(struct ksignal *ksig) { diff --git a/arch/x86/kernel/signal_64.c b/arch/x86/kernel/signal_64.c index 0e808c72bf7e..cacf2ede6217 100644 --- a/arch/x86/kernel/signal_64.c +++ b/arch/x86/kernel/signal_64.c @@ -175,6 +175,9 @@ int x64_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) frame = get_sigframe(ksig, regs, sizeof(struct rt_sigframe), &fp); uc_flags = frame_uc_flags(regs); + if (setup_signal_shadow_stack(ksig)) + return -EFAULT; + if (!user_access_begin(frame, sizeof(*frame))) return -EFAULT; @@ -260,6 +263,9 @@ SYSCALL_DEFINE0(rt_sigreturn) if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; + if (restore_signal_shadow_stack()) + goto badframe; + if (restore_altstack(&frame->uc.uc_stack)) goto badframe; From patchwork Thu Jan 19 21:23:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558752wrn; Thu, 19 Jan 2023 13:43:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXukKMQlqXBgPwptXgWw6LkD0rccGw4hPFTAA+sF6JVT9tAuDO+zGBHArIrPuy/VbfO10dHk X-Received: by 2002:a17:907:9a86:b0:870:8b4f:8a71 with SMTP id km6-20020a1709079a8600b008708b4f8a71mr12468455ejc.6.1674164636845; Thu, 19 Jan 2023 13:43:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164636; cv=none; d=google.com; s=arc-20160816; b=x3a9DRVUCDQaEfW4xNVTfgTdgHv3xnpdiYCbNgwFnABXooccFY/sy4qzsseGi2MqQH pODPGl82gjcwpiQnqJZsdTliXZYVc8hFIeK2mbYGg7herbfVmDSWC8WK4RT7Zb/k7yAJ TZ90XrkPb+l7tyWv1FEMXE+yXj/D69tGyccFzkUc0eId+zCgxExn8RQO40mLp+7xrXO9 iQkH8lG8NclwjALu+vKtQDp0gSU0ovz8ytIDvHKy8aDbG7A1XuvnHc3dndSjvmtnHkeU UkfANBfTfzw/6qUiT8CVfqbQENoREZckAmcZNbh7Qz1p9Ym+q9j56yMJfnwBzevG+D97 Le7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=eLJHhDzZvRpEUGF1Yt556eEckzPL/rnA9Q5SOcyr52E=; b=cuLFIHWJdX2nZGXr1/AzIY/Rg+kvh1MblNCgqrQNHJGfCqbAwhy3WONcC2sC14rNnI if7tvrsXqeY1CmgcD/ZkWiOpQrLHmDGat6cGNBGzNQBO6H2Pc3tTF9tB8lMErIswhMIO iIhMzLC4BAZZtG1h6YIFUUNfNZpdSNXTThx2uqkAgvTbetc0YPYm/pYtfTwamAHRvlc2 4C/Q9sAKuwByvWB/9KhIIllut+8aNjvmkNbzgBOLp9zsjnFEM/xJc6uPTzr28t3f7gu/ hf/XlKvDuXTgwAQ6spx9Bz69K0VGhbG9SPf3s7DVZmwsXgd6qLeF2tQwXM0pSRQAcyjJ S/+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=GFyZaQRz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bb2-20020a1709070a0200b008626e197ac3si27237799ejc.692.2023.01.19.13.43.33; Thu, 19 Jan 2023 13:43:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=GFyZaQRz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230232AbjASVmv (ORCPT + 99 others); Thu, 19 Jan 2023 16:42:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230316AbjASViL (ORCPT ); Thu, 19 Jan 2023 16:38:11 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03A5A7C840; Thu, 19 Jan 2023 13:27:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163673; x=1705699673; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Lpl3uucf4iqvzsR2gzdDBd0io6AfGsBhVzdSqVLnSOA=; b=GFyZaQRzvQPW9kgzTatkRlzvTGBSNx+S2eIK9mlAx1bLYUsmTl4lWsIg yph9htAVnX6+vLFQ8dxYSWUkcUFksYaxa+l8kGAdWmKMLK7tkwLnH1aRu 7b7mNWf+Ae1CAKJhVw/jYFGWi+vRTdLlOMZ4Gndf/vS2HjMUQlYGKu/KE hrGFkWRca0jASVSFStP4eLjiyyWEI7kIshPPa8hJnwEx/vnQ4Jot2sCDh X/9HD42jVxQb2ZKDGTztOorf+g+GHahyjrS+QjPQWaVucZCIRt/cQtC3+ TwS8SLs0wrGAnqTpwq9IC+PHkdUPlYOTFhFzCYFyPxh6WBNWlGkiGhu1F g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119927" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119927" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:14 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139157" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139157" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:12 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 31/39] x86/shstk: Introduce map_shadow_stack syscall Date: Thu, 19 Jan 2023 13:23:09 -0800 Message-Id: <20230119212317.8324-32-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488858391633118?= X-GMAIL-MSGID: =?utf-8?q?1755488858391633118?= When operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks. Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable. Previously, a new PROT_SHADOW_STACK was attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window. The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token. First, a new madvise() flag was explored, which could operate on the PROT_SHADOW_STACK memory. This had a couple downsides: 1. Extra checks were needed in mprotect() to prevent writable memory from ever becoming PROT_SHADOW_STACK. 2. Extra checks/vma state were needed in the new madvise() to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to. 3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior. So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way. The following example demonstrates how to create a new shadow stack with map_shadow_stack: void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN); Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix addr/mapped_addr (Kees) - Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this) - Return error for addresses below 4G v3: - Change syscall common -> 64 (Kees) - Use bit shift notation instead of 0x1 for uapi header (Kees) - Call do_mmap() with MAP_FIXED_NOREPLACE (Kees) - Block unsupported flags (Kees) - Require size >= 8 to set token (Kees) v2: - Change syscall to take address like mmap() for CRIU's usage v1: - New patch (replaces PROT_SHADOW_STACK). arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/uapi/asm/mman.h | 3 ++ arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++---- include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 2 +- kernel/sys_ni.c | 1 + 6 files changed, 58 insertions(+), 9 deletions(-) diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index c84d12608cd2..f65c671ce3b1 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 64 map_shadow_stack sys_map_shadow_stack # # Due to a historical design error, certain syscalls are numbered differently diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h index 5a0256e73f1e..8148bdddbd2c 100644 --- a/arch/x86/include/uapi/asm/mman.h +++ b/arch/x86/include/uapi/asm/mman.h @@ -13,6 +13,9 @@ ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) #endif +/* Flags for map_shadow_stack(2) */ +#define SHADOW_STACK_SET_TOKEN (1ULL << 0) /* Set up a restore token in the shadow stack */ + #include #endif /* _ASM_X86_MMAN_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 56e7ca8e42cc..e857083b9e14 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -71,19 +72,31 @@ static int create_rstor_token(unsigned long ssp, unsigned long *token_addr) return 0; } -static unsigned long alloc_shstk(unsigned long size) +static unsigned long alloc_shstk(unsigned long addr, unsigned long size, + unsigned long token_offset, bool set_res_tok) { int flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G; struct mm_struct *mm = current->mm; - unsigned long addr, unused; + unsigned long mapped_addr, unused; - mmap_write_lock(mm); - addr = do_mmap(NULL, 0, size, PROT_READ, flags, - VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + if (addr) + flags |= MAP_FIXED_NOREPLACE; + mmap_write_lock(mm); + mapped_addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); mmap_write_unlock(mm); - return addr; + if (!set_res_tok || IS_ERR_VALUE(mapped_addr)) + goto out; + + if (create_rstor_token(mapped_addr + token_offset, NULL)) { + vm_munmap(mapped_addr, size); + return -EINVAL; + } + +out: + return mapped_addr; } static unsigned long adjust_shstk_size(unsigned long size) @@ -134,7 +147,7 @@ static int shstk_setup(void) return -EOPNOTSUPP; size = adjust_shstk_size(0); - addr = alloc_shstk(size); + addr = alloc_shstk(0, size, 0, false); if (IS_ERR_VALUE(addr)) return PTR_ERR((void *)addr); @@ -179,7 +192,7 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, size = adjust_shstk_size(stack_size); - addr = alloc_shstk(size); + addr = alloc_shstk(0, size, 0, false); if (IS_ERR_VALUE(addr)) return PTR_ERR((void *)addr); @@ -373,6 +386,36 @@ static int shstk_disable(void) return 0; } +SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags) +{ + bool set_tok = flags & SHADOW_STACK_SET_TOKEN; + unsigned long aligned_size; + + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + if (flags & ~SHADOW_STACK_SET_TOKEN) + return -EINVAL; + + /* If there isn't space for a token */ + if (set_tok && size < 8) + return -EINVAL; + + if (addr && addr <= 0xFFFFFFFF) + return -EINVAL; + + /* + * An overflow would result in attempting to write the restore token + * to the wrong location. Not catastrophic, but just return the right + * error code and block it. + */ + aligned_size = PAGE_ALIGN(size); + if (aligned_size < size) + return -EOVERFLOW; + + return alloc_shstk(addr, aligned_size, size, set_tok); +} + long shstk_prctl(struct task_struct *task, int option, unsigned long features) { if (option == ARCH_SHSTK_LOCK) { diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index 33a0ee3bcb2e..392dc11e3556 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags); asmlinkage long sys_set_mempolicy_home_node(unsigned long start, unsigned long len, unsigned long home_node, unsigned long flags); +asmlinkage long sys_map_shadow_stack(unsigned long addr, unsigned long size, unsigned int flags); /* * Architecture-specific system calls diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 45fa180cc56a..b12940ec5926 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) #undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 860b2dcf3ac4..cb9aebd34646 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -381,6 +381,7 @@ COND_SYSCALL(vm86old); COND_SYSCALL(modify_ldt); COND_SYSCALL(vm86); COND_SYSCALL(kexec_file_load); +COND_SYSCALL(map_shadow_stack); /* s390 */ COND_SYSCALL(s390_pci_mmio_read); From patchwork Thu Jan 19 21:23:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46009 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558501wrn; Thu, 19 Jan 2023 13:43:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXtvQXemqCTyshXouMNq1FJt21QY7ic2+F8tciLw6ai1fSoN+9P43Ubh7tLz+uowcyUjlMXA X-Received: by 2002:a17:907:d684:b0:7c1:e78:1e2 with SMTP id wf4-20020a170907d68400b007c10e7801e2mr14762066ejc.11.1674164593382; Thu, 19 Jan 2023 13:43:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164593; cv=none; d=google.com; s=arc-20160816; b=uEIfVRSLpiyRR6MLJnaM2/g/W8RdkmwZFBEc1E/JXEfK+6jf0JCXLZs7/XohSH3ww8 aKNQ+yQrAufkTMvb5Vrg72HQMJMKL7pPJgMXPlIRGaKvTuH7Z9Fl1muuLzJUn2OuDT0p /XHyipvwWL04VK7qhEDPUwm7MafYNmWeLnXYOz5J6FJSGrJ5AkXhb9sGkBiuUXP+jmvN YSuSBF1trIUrufGWx0FMX1bu0F5nmfA0VupkjeaxX6fy7/9q7yR+moB6qZjE5N8gxIVQ YYA/8RmYWeYfWyfLZ2hL3sMhTbDIGoIIJ90kcP/W17Ft6V8G/OQo4P+RrBvp0GPq6YLe iyBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=e6Ucp8srnUL/pIV8RzS949UTzc2xSGWpmpvzoBTxJjU=; b=xTzyb9LJpBLZN51jzIuMhid5+Np+6Ko1pJjlqeWb1vBDBn212zGIXnfaLscxlBAxfN SsOcmm+Caop9/I7/+vLbwOBribJxWiC/wNsMEPLL9zMu1SON+uHEaMyF+tz9Rn/iyjPx MWcY3FBzT4OzWZOBc66/eMKAMqTOqY49WddpCi4evAZWDmBLHsZiXPyrs83LpyN0ZB5s KUMqLyQIXDIc49qqvPohq5OPjYm4tjnpP3QjUX491l9c8g08aV5sl4qLM4Scb/0kFCvk J9N85Dxoe4kAdpiyHzNG7+WLkbe4Y8yEfBCb84C2+N2vJSwoDbhN26po9/l1yu0JqRVS rtMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WI55GtVj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wt2-20020a170906ee8200b0084c45d8a688si19558392ejb.891.2023.01.19.13.42.48; Thu, 19 Jan 2023 13:43:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WI55GtVj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231152AbjASVmD (ORCPT + 99 others); Thu, 19 Jan 2023 16:42:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230313AbjASViE (ORCPT ); Thu, 19 Jan 2023 16:38:04 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0304E470B8; Thu, 19 Jan 2023 13:27:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163673; x=1705699673; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=miL5v57uRD6gfvhLMCiJeO3l14zaMekSGW7g+Pa5v08=; b=WI55GtVjnY+2frhnTyMYjZijFRaGE8TCsV4sAMtHrLXauHWIaopFy7r0 1G4NMDnDDcqvavy37MspwhWrufn/8P/XVzZHMWQI6/4x5X0zoCBtD2fwn a7mHTQlqPpH1TRE3ak0x8T21fJ8LaXj0AcxXYw3+gXSYImKf8+Q7C+4Vh sf+sB9gqhoOlncbl+EkWnco72DzONFc6EOoiyupz8Rjfg8x+mMhpyvO6K o6bL9/x6Bj6BbvV1LOPVrhcCPU3ADE32EQET5w4N8aohSNt78jSRmmlhu TbZP/TvgKqSWulr83dKujQ8PvbeAmtwlP35DFF7ZKlu089Gyt7x+QuRie g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119949" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119949" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139164" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139164" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:14 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 32/39] x86/shstk: Support WRSS for userspace Date: Thu, 19 Jan 2023 13:23:10 -0800 Message-Id: <20230119212317.8324-33-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488812952773692?= X-GMAIL-MSGID: =?utf-8?q?1755488812952773692?= For the current shadow stack implementation, shadow stacks contents can't easily be provisioned with arbitrary data. This property helps apps protect themselves better, but also restricts any potential apps that may want to do exotic things at the expense of a little security. The x86 shadow stack feature introduces a new instruction, WRSS, which can be enabled to write directly to shadow stack permissioned memory from userspace. Allow it to get enabled via the prctl interface. Only enable the userspace WRSS instruction, which allows writes to userspace shadow stacks from userspace. Do not allow it to be enabled independently of shadow stack, as HW does not support using WRSS when shadow stack is disabled. From a fault handler perspective, WRSS will behave very similar to WRUSS, which is treated like a user access from a #PF err code perspective. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Switch to EOPNOTSUPP - Move set_clr_bits_msrl() to patch where it is first used - Commit log formatting v3: - Make wrss_control() static - Fix verbiage in commit log (Kees) v2: - Add some commit log verbiage from (Dave Hansen) v1: - New patch. arch/x86/include/asm/msr.h | 11 +++++++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/shstk.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h index 65ec1965cd28..a4b86eb537d6 100644 --- a/arch/x86/include/asm/msr.h +++ b/arch/x86/include/asm/msr.h @@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs); int msr_set_bit(u32 msr, u8 bit); int msr_clear_bit(u32 msr, u8 bit); +/* Helper that can never get accidentally un-inlined. */ +#define set_clr_bits_msrl(msr, set, clear) do { \ + u64 __val, __new_val; \ + \ + rdmsrl(msr, __val); \ + __new_val = (__val & ~(clear)) | (set); \ + \ + if (__new_val != __val) \ + wrmsrl(msr, __new_val); \ +} while (0) + #ifdef CONFIG_SMP int rdmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 *l, u32 *h); int wrmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 l, u32 h); diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 7dfd9dc00509..e31495668056 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -28,5 +28,6 @@ /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1) #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index e857083b9e14..71dbb49b93cd 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -364,6 +364,35 @@ void shstk_free(struct task_struct *tsk) unmap_shadow_stack(shstk->base, shstk->size); } +static int wrss_control(bool enable) +{ + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) + return -EOPNOTSUPP; + + /* + * Only enable wrss if shadow stack is enabled. If shadow stack is not + * enabled, wrss will already be disabled, so don't bother clearing it + * when disabling. + */ + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return -EPERM; + + /* Already enabled/disabled? */ + if (features_enabled(ARCH_SHSTK_WRSS) == enable) + return 0; + + fpregs_lock_and_load(); + if (enable) { + set_clr_bits_msrl(MSR_IA32_U_CET, CET_WRSS_EN, 0); + features_set(ARCH_SHSTK_WRSS); + } else { + set_clr_bits_msrl(MSR_IA32_U_CET, 0, CET_WRSS_EN); + features_clr(ARCH_SHSTK_WRSS); + } + fpregs_unlock(); + + return 0; +} static int shstk_disable(void) { @@ -381,7 +410,7 @@ static int shstk_disable(void) fpregs_unlock(); shstk_free(current); - features_clr(ARCH_SHSTK_SHSTK); + features_clr(ARCH_SHSTK_SHSTK | ARCH_SHSTK_WRSS); return 0; } From patchwork Thu Jan 19 21:23:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46017 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp559408wrn; Thu, 19 Jan 2023 13:45:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXstsYTAJtRQ1xLwuj+PuKSCKAaYRzeS5RRZLFHHh6a9Bw4cIUYcLYDZ+rfQkpkfDRCuhDTT X-Received: by 2002:a05:6402:3224:b0:45c:834b:f287 with SMTP id g36-20020a056402322400b0045c834bf287mr14535651eda.4.1674164740590; Thu, 19 Jan 2023 13:45:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164740; cv=none; d=google.com; s=arc-20160816; b=sHx7aMIbXCF+m8zS+PPVFxhM1Rp1z2oBIW5DduHgXVSk+ljfDBlI8zNQ7LliPH1aoK FyjLWruRcw65QvVCWP8x5rvPwooUR9wiuPzc0a2xfqxpSkcH/IB+UD0c2ryJYautHtBr zBvpAvmd9hIRx1Rbe9Bj+8O7seeqNka2BqwKV3SpduZuOslTzlOr6doXSSeAQKxUZY3b KBv7lC1SEXaOTt5+IHZlSIaEfjusTMM5gw7l9GXnz/NfQGnIlZKiohyK9zJ43MxpBg4e WjVqackGLFHc5Tvsu5XOd9kDqOq/agcbSq5q3sjE+u5LtTwndqgAcO92koZfzgdi2tOM w4EQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=upYGst2M+pM9taclLJcrPM9etxXHvFq2ULPd7LORduM=; b=BToPODwAYig1gvffUfB+CeTUhn/xRQs7bnJt45AxBdwuVrTjgH3+Og0LxRCwZac2WR 6rr2DMHYhqc4Hpai1GeI/kxqf2DO8LzNvmsmjU33zzPD34xbglVkYwLAaVscK1Snkg8m lNydef1NWZmhT7Vdr1ZUGX+L0Z0lOKN5irkhXePDNnlUVN85nJjhLTfeqklUiuy2y9Sk uNC7hMGSa0ff1N+1O0+9GwMkSpYNClSqKbIZ8I/RXmxAL+UxsbzLAhHN0phbc66Pcnxg IMlJvG7fKKA5OLcJd9qDPPMyv380RAqPa053BEZ3/NsCGzTao+zi/z8WrJJOdsYbfGwn m4vg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gQkrwrQ4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h12-20020a056402280c00b0048c0b5672e4si51341048ede.37.2023.01.19.13.45.16; Thu, 19 Jan 2023 13:45:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gQkrwrQ4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231243AbjASVmg (ORCPT + 99 others); Thu, 19 Jan 2023 16:42:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230092AbjASViK (ORCPT ); Thu, 19 Jan 2023 16:38:10 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 036F24ABC2; Thu, 19 Jan 2023 13:27:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163673; x=1705699673; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=vY8qQFdDU3u0a0tvehCZCS7MwIYptOrO+eN8SXLOirs=; b=gQkrwrQ4Dn6lvzoLLIOoZj6nwoj2pw+qVTv02bXHhd7Ix3u3k+bemd5x rU+E9ClBPy78ZtArP0KQsxrnzNb6tTAHA9Y/SEKAlWK9koXLa5HXJbPGi qWxbgBmvMelkQn8oKtx+j8iN0sBZHhD06SowEv5I0JbSIOxjEa+y4Y3sH AQX9ilJQHkJpUpRfLXZnbFTL70uf5vUsxV54O04OjTILRJCU6gH4bafYi MUrsUq68uyoDsLTYQMnRGXhze0S940xweCwcmJs6sfe5DfdDOSO18tyte 9sb7unTi272m62rH+56a7SC4OP9ONC/0X8o88JBaqoNBAb9sQrwVaMrfF A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119970" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119970" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:17 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139169" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139169" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:16 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 33/39] x86: Expose thread features in /proc/$PID/status Date: Thu, 19 Jan 2023 13:23:11 -0800 Message-Id: <20230119212317.8324-34-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488966868615256?= X-GMAIL-MSGID: =?utf-8?q?1755488966868615256?= Applications and loaders can have logic to decide whether to enable shadow stack. They usually don't report whether shadow stack has been enabled or not, so there is no way to verify whether an application actually is protected by shadow stack. Add two lines in /proc/$PID/status to report enabled and locked features. Since, this involves referring to arch specific defines in asm/prctl.h, implement an arch breakout to emit the feature lines. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Kirill A. Shutemov [Switched to CET, added to commit log] Signed-off-by: Rick Edgecombe --- v4: - Remove "CET" references v3: - Move to /proc/pid/status (Kees) v2: - New patch arch/x86/kernel/cpu/proc.c | 23 +++++++++++++++++++++++ fs/proc/array.c | 6 ++++++ include/linux/proc_fs.h | 2 ++ 3 files changed, 31 insertions(+) diff --git a/arch/x86/kernel/cpu/proc.c b/arch/x86/kernel/cpu/proc.c index 099b6f0d96bd..31c0e68f6227 100644 --- a/arch/x86/kernel/cpu/proc.c +++ b/arch/x86/kernel/cpu/proc.c @@ -4,6 +4,8 @@ #include #include #include +#include +#include #include "cpu.h" @@ -175,3 +177,24 @@ const struct seq_operations cpuinfo_op = { .stop = c_stop, .show = show_cpuinfo, }; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +static void dump_x86_features(struct seq_file *m, unsigned long features) +{ + if (features & ARCH_SHSTK_SHSTK) + seq_puts(m, "shstk "); + if (features & ARCH_SHSTK_WRSS) + seq_puts(m, "wrss "); +} + +void arch_proc_pid_thread_features(struct seq_file *m, struct task_struct *task) +{ + seq_puts(m, "x86_Thread_features:\t"); + dump_x86_features(m, task->thread.features); + seq_putc(m, '\n'); + + seq_puts(m, "x86_Thread_features_locked:\t"); + dump_x86_features(m, task->thread.features_locked); + seq_putc(m, '\n'); +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..7ac43ecda1c2 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +__weak void arch_proc_pid_thread_features(struct seq_file *m, + struct task_struct *task) +{ +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -451,6 +456,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_cpus_allowed(m, task); cpuset_task_status_allowed(m, task); task_context_switch_counts(m, task); + arch_proc_pid_thread_features(m, task); return 0; } diff --git a/include/linux/proc_fs.h b/include/linux/proc_fs.h index 0260f5ea98fe..80ff8e533cbd 100644 --- a/include/linux/proc_fs.h +++ b/include/linux/proc_fs.h @@ -158,6 +158,8 @@ int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task); #endif /* CONFIG_PROC_PID_ARCH_STATUS */ +void arch_proc_pid_thread_features(struct seq_file *m, struct task_struct *task); + #else /* CONFIG_PROC_FS */ static inline void proc_root_init(void) From patchwork Thu Jan 19 21:23:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558868wrn; Thu, 19 Jan 2023 13:44:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXtyC2z7uAJXU6uKtFX+p6FlRazo6E7/CwPtkDJ1ilBsxkRlxGNEDxYCBty2FXnY3a5U7wNy X-Received: by 2002:a05:6402:448:b0:492:798:385e with SMTP id p8-20020a056402044800b004920798385emr24680945edw.33.1674164656022; Thu, 19 Jan 2023 13:44:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164656; cv=none; d=google.com; s=arc-20160816; b=uqjJR2y2bNOJ/u3ed8yAXugDSqeUxfjrsaYgj2JUS80eQB/AvJoWjUKBGQMT16rpG0 ZcQUiAzplHKrWJrGzih1dx1sFq+ZVsxueuN2RNqC09ljWnw/Sc2EAVSdO4uwFrDKNs/k Htx0QZKM0s9jeW8+2kVmsQQhql8qY32iYzDySmw935b8fFnWqsIDWH83dLt/RkwwFt3l 3vGQvq8aGmcaNHIZpGiuYD9DOV3d6setPQ+5fmqqxypRmF9SecT2Ibf8WV6YogTa1ps4 fgxfuUDzTt33e3EVP2Wxray0vGtG4s6K+KzI9xBHywpNoG35+2kk9B+9Fi3iSdkZwMWv 3zfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=LARyYXf5hRWem17uRVeQYirgVhi/Ae5Eh0pNi1jng38=; b=KTmncHoUCFeWFX4Iy64gDASinkoAKp84hqk2CeTDE2x/CR45cFGoRMnAnkUXY7un9U QqHgzTaiMd3yDnQA+fDk+eHVsPV4jM7mXpyxbA06knIjuXKSBDbV2RWeFYq1F7186Qtm KzB+Bh6KW6JGYfOpGI0Fr9unUFYPgHCJYNOVBGSIzDGNd/Wex7ZeYePrKAbXo1lQEK2m ZNUL1t3MOMfHIYS5KQwyRQatMgZROJgiQ/JkCmAR0pVikpbFJMsN4vrpqmPybG2qIuAQ Sp5J8scGpp7eGkmgojGsf02w/l7+uX3XHj5dwOo0WXZ//sANh72S45fyA+qgqCCuk+vx IVuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P2OoiS+h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t19-20020a056402525300b0049b63678b4fsi31139994edd.358.2023.01.19.13.43.52; Thu, 19 Jan 2023 13:44:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P2OoiS+h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230481AbjASVnQ (ORCPT + 99 others); Thu, 19 Jan 2023 16:43:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231167AbjASVik (ORCPT ); Thu, 19 Jan 2023 16:38:40 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F40737544; Thu, 19 Jan 2023 13:27:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163678; x=1705699678; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sQY1E+dUSY4CcT1mmXrJaIQvOz4XoIsJlzVbJWGjXhw=; b=P2OoiS+hwDLY6GdBku6zRAS/PkdSptjlD6Vzbo/ybb4J8U3Z2faRDQTQ y7Y58C5/UzgZQdbaJF6nEJb4SXz9RbaDE54LE3IsB/m1/DLsNKnfxvFrk IAhUKoE50LVd6Z07EXbCUDoZ3golLfZ6xyC/vAjUyJhLZeLZFNIL5mFRx WHh/i3YGvQOetdDHHkvbYFQKZdvZKPyvGchG2EzRAuxumMTWUtwLKsGuZ vomDaCwiS7/uj/KlKoeC6LAtP2hPIY3MyMcL9r17WApdbg49TF0jd3O6x 7f8HBYybAU/qSmVH3GKaWiVyaa3F7HJVVsz1d9urOk2bKWUvnd0t/HGSH g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119994" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119994" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:19 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139174" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139174" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:17 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 34/39] x86/shstk: Wire in shadow stack interface Date: Thu, 19 Jan 2023 13:23:12 -0800 Message-Id: <20230119212317.8324-35-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488878222398312?= X-GMAIL-MSGID: =?utf-8?q?1755488878222398312?= The kernel now has the main shadow stack functionality to support applications. Wire in the WRSS and shadow stack enable/disable functions into the existing shadow stack API skeleton. Tested-by: Pengfei Xu Tested-by: John Allen Reviewed-by: Kees Cook Signed-off-by: Rick Edgecombe --- v4: - Remove "CET" references v2: - Split from other patches arch/x86/kernel/shstk.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 71dbb49b93cd..07142e6f05f6 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -465,9 +465,17 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features) return -EINVAL; if (option == ARCH_SHSTK_DISABLE) { + if (features & ARCH_SHSTK_WRSS) + return wrss_control(false); + if (features & ARCH_SHSTK_SHSTK) + return shstk_disable(); return -EINVAL; } /* Handle ARCH_SHSTK_ENABLE */ + if (features & ARCH_SHSTK_SHSTK) + return shstk_setup(); + if (features & ARCH_SHSTK_WRSS) + return wrss_control(true); return -EINVAL; } From patchwork Thu Jan 19 21:23:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp560314wrn; Thu, 19 Jan 2023 13:47:59 -0800 (PST) X-Google-Smtp-Source: AMrXdXsZtIL/WeTtZ4JHEEJHuteMSl5TgYdl8we69YTqk9D4oEqvWFCX9Asi78JT87B5yY/3eS4n X-Received: by 2002:a17:907:8999:b0:877:83ea:2bfc with SMTP id rr25-20020a170907899900b0087783ea2bfcmr2774758ejc.39.1674164879187; Thu, 19 Jan 2023 13:47:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164879; cv=none; d=google.com; s=arc-20160816; b=O/JWvaoEHqpn+wRBuTYzYaiW8iTBzd38cRDfyDLjFWhE5GempFleGtIXp1Ssoju8qD jC4/SCrdJ3oOoICSNtuQw3U0P7IkwCOnT1MhL7MMwH5100tjoGqmmXXk8YyrDYQR32hi 1ozrVZ1MDqHbKKswxYGb+wtJcb/icY43By2aimnUvm8i4fmCRZmZ1WPOY7Oof9JFJ20/ l32LHEkfmo892HW8e0h9DFGYA06vIyv2zCdK3D6QCDPHBi8ugl6F1UxP9J+Lg1XRd5eo WK3JLHqNLCysSieS5RHJg9ZyKLsuWORr+8Q7Me2Wv9oLxlmpfYgnKGi4qenyO+5+hr3t Z3kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=TgTd1mfufOsbbmS0K83XaaAGrMI8Q2ihbQqiYwpYOW0=; b=BSExCQoILoTOqEcM2v4RwR85xFabmSYJUkEmdkrX6erhaVlJAufrtvXUp1EK2pE+Wy 3SJKwJULZUWeNMV5udy8uLaxLsBAHT7cbgB7QqJ9tm2U3aI/aeNqaxY4HAyhfrwqE0Iv +BjMlaZy3JZG6RcbhfcrIpyzu18TMm2ABmHL6LsHzVZE+4VTgAc8TA0N/OD2uuGFAo7i iXXwztxwhwj+zo1cPnlM6LS/axWA6082GO1hAjn3p8IPMkvxK/K76tv1RtTFka1Ku7fp muOK7l9yWsbgV968inwFxqfdpPownEenEDMuf+LBO3nTDNlTdc2XK7W+2yOBJFp4c6rn uJ/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WT9uIDwU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dt17-20020a170907729100b0084d1d7fbd27si44836652ejc.205.2023.01.19.13.47.35; Thu, 19 Jan 2023 13:47:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WT9uIDwU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231338AbjASVoQ (ORCPT + 99 others); Thu, 19 Jan 2023 16:44:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229788AbjASViy (ORCPT ); Thu, 19 Jan 2023 16:38:54 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C85E459560; Thu, 19 Jan 2023 13:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163679; x=1705699679; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=+geAUGtKJ5/tBElMoeiPJqxtmy5X7bBSU640+SHskXE=; b=WT9uIDwUpxSLOucCcjFpBCp265W2lAcSVhnJkcLFhw3Vlofbr6l8ce5Z LXoifkizqVaQQZRmrdXq9EklfRbhFrDv8sAY4KSRg7VbFYLL9dJ52cM5/ 7ERVrK15ElY+HcIUoHd8STxsY+0m8WkPv4EwLERscM38CWBm4mRyk/PO4 QS0pOBuSIhLxHLvCWhyTmLiVN71Q3TK5dunc+W3HPKUVeOa20VIFYar8r hAnTAZ4SlQSyt0c6xOZuOojJAW+/JTp/1yrmhpI1AmSFynf2NwVGtc4Af HnHU/QzxoIWsVg+tLEPD4Sp9KoUei6117HGCLCNo3fViYIjD//ZjIiSMB g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323120020" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323120020" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:20 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139180" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139180" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:19 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 35/39] selftests/x86: Add shadow stack test Date: Thu, 19 Jan 2023 13:23:13 -0800 Message-Id: <20230119212317.8324-36-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755489112478345758?= X-GMAIL-MSGID: =?utf-8?q?1755489112478345758?= Add a simple selftest for exercising some shadow stack behavior: - map_shadow_stack syscall and pivot - Faulting in shadow stack memory - Handling shadow stack violations - GUP of shadow stack memory - mprotect() of shadow stack memory - Userfaultfd on shadow stack memory Since this test exercises a recently added syscall manually, it needs to find the automatically created __NR_foo defines. Per the selftest documentation, KHDR_INCLUDES can be used to help the selftest Makefile's find the headers from the kernel source. This way the new selftest can be built inside the kernel source tree without installing the headers to the system. So also add KHDR_INCLUDES as described in the selftest docs, to facilitate this. Tested-by: Pengfei Xu Tested-by: John Allen Co-developed-by: Yu-cheng Yu Signed-off-by: Yu-cheng Yu Signed-off-by: Rick Edgecombe --- v5: - Update 32 bit signal test with new ABI and better asm v4: - Add test for 32 bit signal ABI blocking v3: - Change "+m" to "=m" in write_shstk() (Andrew Cooper) - Fix userfaultfd test with transparent huge pages by doing a MADV_DONTNEED, since the token write faults in the while stack with huge pages. v2: - Change print statements to more align with other selftests - Add more tests - Add KHDR_INCLUDES to Makefile tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 667 ++++++++++++++++++ 2 files changed, 669 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..cfc8a26ad151 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx test_shadow_stack # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall @@ -34,7 +34,7 @@ BINARIES_64 := $(TARGETS_C_64BIT_ALL:%=%_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) -CFLAGS := -O2 -g -std=gnu99 -pthread -Wall +CFLAGS := -O2 -g -std=gnu99 -pthread -Wall $(KHDR_INCLUDES) # call32_from_64 in thunks.S uses absolute addresses. ifeq ($(CAN_BUILD_WITH_NOPIE),1) diff --git a/tools/testing/selftests/x86/test_shadow_stack.c b/tools/testing/selftests/x86/test_shadow_stack.c new file mode 100644 index 000000000000..5a3b4f6d1a1d --- /dev/null +++ b/tools/testing/selftests/x86/test_shadow_stack.c @@ -0,0 +1,667 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * This program test's basic kernel shadow stack support. It enables shadow + * stack manual via the arch_prctl(), instead of relying on glibc. It's + * Makefile doesn't compile with shadow stack support, so it doesn't rely on + * any particular glibc. As a result it can't do any operations that require + * special glibc shadow stack support (longjmp(), swapcontext(), etc). Just + * stick to the basics and hope the compiler doesn't do anything strange. + */ + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SS_SIZE 0x200000 + +#if (__GNUC__ < 8) || (__GNUC__ == 8 && __GNUC_MINOR__ < 5) +int main(int argc, char *argv[]) +{ + printf("[SKIP]\tCompiler does not support CET.\n"); + return 0; +} +#else +void write_shstk(unsigned long *addr, unsigned long val) +{ + asm volatile("wrssq %[val], (%[addr])\n" + : "=m" (addr) + : [addr] "r" (addr), [val] "r" (val)); +} + +static inline unsigned long __attribute__((always_inline)) get_ssp(void) +{ + unsigned long ret = 0; + + asm volatile("xor %0, %0; rdsspq %0" : "=r" (ret)); + return ret; +} + +/* + * For use in inline enablement of shadow stack. + * + * The program can't return from the point where shadow stack gets enabled + * because there will be no address on the shadow stack. So it can't use + * syscall() for enablement, since it is a function. + * + * Based on code from nolibc.h. Keep a copy here because this can't pull in all + * of nolibc.h. + */ +#define ARCH_PRCTL(arg1, arg2) \ +({ \ + long _ret; \ + register long _num asm("eax") = __NR_arch_prctl; \ + register long _arg1 asm("rdi") = (long)(arg1); \ + register long _arg2 asm("rsi") = (long)(arg2); \ + \ + asm volatile ( \ + "syscall\n" \ + : "=a"(_ret) \ + : "r"(_arg1), "r"(_arg2), \ + "0"(_num) \ + : "rcx", "r11", "memory", "cc" \ + ); \ + _ret; \ +}) + +void *create_shstk(void *addr) +{ + return (void *)syscall(__NR_map_shadow_stack, addr, SS_SIZE, SHADOW_STACK_SET_TOKEN); +} + +void *create_normal_mem(void *addr) +{ + return mmap(addr, SS_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); +} + +void free_shstk(void *shstk) +{ + munmap(shstk, SS_SIZE); +} + +int reset_shstk(void *shstk) +{ + return madvise(shstk, SS_SIZE, MADV_DONTNEED); +} + +void try_shstk(unsigned long new_ssp) +{ + unsigned long ssp; + + printf("[INFO]\tnew_ssp = %lx, *new_ssp = %lx\n", + new_ssp, *((unsigned long *)new_ssp)); + + ssp = get_ssp(); + printf("[INFO]\tchanging ssp from %lx to %lx\n", ssp, new_ssp); + + asm volatile("rstorssp (%0)\n":: "r" (new_ssp)); + asm volatile("saveprevssp"); + printf("[INFO]\tssp is now %lx\n", get_ssp()); + + /* Switch back to original shadow stack */ + ssp -= 8; + asm volatile("rstorssp (%0)\n":: "r" (ssp)); + asm volatile("saveprevssp"); +} + +int test_shstk_pivot(void) +{ + void *shstk = create_shstk(0); + + if (shstk == MAP_FAILED) { + printf("[FAIL]\tError creating shadow stack: %d\n", errno); + return 1; + } + try_shstk((unsigned long)shstk + SS_SIZE - 8); + free_shstk(shstk); + + printf("[OK]\tShadow stack pivot\n"); + return 0; +} + +int test_shstk_faults(void) +{ + unsigned long *shstk = create_shstk(0); + + /* Read shadow stack, test if it's zero to not get read optimized out */ + if (*shstk != 0) + goto err; + + /* Wrss memory that was already read. */ + write_shstk(shstk, 1); + if (*shstk != 1) + goto err; + + /* Page out memory, so we can wrss it again. */ + if (reset_shstk((void *)shstk)) + goto err; + + write_shstk(shstk, 1); + if (*shstk != 1) + goto err; + + printf("[OK]\tShadow stack faults\n"); + return 0; + +err: + return 1; +} + +unsigned long saved_ssp; +unsigned long saved_ssp_val; +volatile bool segv_triggered; + +void __attribute__((noinline)) violate_ss(void) +{ + saved_ssp = get_ssp(); + saved_ssp_val = *(unsigned long *)saved_ssp; + + /* Corrupt shadow stack */ + printf("[INFO]\tCorrupting shadow stack\n"); + write_shstk((void *)saved_ssp, 0); +} + +void segv_handler(int signum, siginfo_t *si, void *uc) +{ + printf("[INFO]\tGenerated shadow stack violation successfully\n"); + + segv_triggered = true; + + /* Fix shadow stack */ + write_shstk((void *)saved_ssp, saved_ssp_val); +} + +int test_shstk_violation(void) +{ + struct sigaction sa; + + sa.sa_sigaction = segv_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* Make sure segv_triggered is set before violate_ss() */ + asm volatile("" : : : "memory"); + + violate_ss(); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tShadow stack violation test\n"); + + return !segv_triggered; +} + +/* Gup test state */ +#define MAGIC_VAL 0x12345678 +bool is_shstk_access; +void *shstk_ptr; +int fd; + +void reset_test_shstk(void *addr) +{ + if (shstk_ptr != NULL) + free_shstk(shstk_ptr); + shstk_ptr = create_shstk(addr); +} + +void test_access_fix_handler(int signum, siginfo_t *si, void *uc) +{ + printf("[INFO]\tViolation from %s\n", is_shstk_access ? "shstk access" : "normal write"); + + segv_triggered = true; + + /* Fix shadow stack */ + if (is_shstk_access) { + reset_test_shstk(shstk_ptr); + return; + } + + free_shstk(shstk_ptr); + create_normal_mem(shstk_ptr); +} + +bool test_shstk_access(void *ptr) +{ + is_shstk_access = true; + segv_triggered = false; + write_shstk(ptr, MAGIC_VAL); + + asm volatile("" : : : "memory"); + + return segv_triggered; +} + +bool test_write_access(void *ptr) +{ + is_shstk_access = false; + segv_triggered = false; + *(unsigned long *)ptr = MAGIC_VAL; + + asm volatile("" : : : "memory"); + + return segv_triggered; +} + +bool gup_write(void *ptr) +{ + unsigned long val; + + lseek(fd, (unsigned long)ptr, SEEK_SET); + if (write(fd, &val, sizeof(val)) < 0) + return 1; + + return 0; +} + +bool gup_read(void *ptr) +{ + unsigned long val; + + lseek(fd, (unsigned long)ptr, SEEK_SET); + if (read(fd, &val, sizeof(val)) < 0) + return 1; + + return 0; +} + +int test_gup(void) +{ + struct sigaction sa; + int status; + pid_t pid; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + fd = open("/proc/self/mem", O_RDWR); + if (fd == -1) + return 1; + + reset_test_shstk(0); + if (gup_read(shstk_ptr)) + return 1; + if (test_shstk_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup read -> shstk access success\n"); + + reset_test_shstk(0); + if (gup_write(shstk_ptr)) + return 1; + if (test_shstk_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup write -> shstk access success\n"); + + reset_test_shstk(0); + if (gup_read(shstk_ptr)) + return 1; + if (!test_write_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup read -> write access success\n"); + + reset_test_shstk(0); + if (gup_write(shstk_ptr)) + return 1; + if (!test_write_access(shstk_ptr)) + return 1; + printf("[INFO]\tGup write -> write access success\n"); + + close(fd); + + /* COW/gup test */ + reset_test_shstk(0); + pid = fork(); + if (!pid) { + fd = open("/proc/self/mem", O_RDWR); + if (fd == -1) + exit(1); + + if (gup_write(shstk_ptr)) { + close(fd); + exit(1); + } + close(fd); + exit(0); + } + waitpid(pid, &status, 0); + if (WEXITSTATUS(status)) { + printf("[FAIL]\tWrite in child failed\n"); + return 1; + } + if (*(unsigned long *)shstk_ptr == MAGIC_VAL) { + printf("[FAIL]\tWrite in child wrote through to shared memory\n"); + return 1; + } + + printf("[INFO]\tCow gup write -> write access success\n"); + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tShadow gup test\n"); + + return 0; +} + +int test_mprotect(void) +{ + struct sigaction sa; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* mprotect a shadow stack as read only */ + reset_test_shstk(0); + if (mprotect(shstk_ptr, SS_SIZE, PROT_READ) < 0) { + printf("[FAIL]\tmprotect(PROT_READ) failed\n"); + return 1; + } + + /* try to wrss it and fail */ + if (!test_shstk_access(shstk_ptr)) { + printf("[FAIL]\tShadow stack access to read-only memory succeeded\n"); + return 1; + } + + /* then back to writable */ + if (mprotect(shstk_ptr, SS_SIZE, PROT_WRITE | PROT_READ) < 0) { + printf("[FAIL]\tmprotect(PROT_WRITE) failed\n"); + return 1; + } + + /* then pivot to it and succeed */ + if (test_shstk_access(shstk_ptr)) { + printf("[FAIL]\tShadow stack access to mprotect() writable memory failed\n"); + return 1; + } + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + printf("[OK]\tmprotect() test\n"); + + return 0; +} + +char zero[4096]; + +static void *uffd_thread(void *arg) +{ + struct uffdio_copy req; + int uffd = *(int *)arg; + struct uffd_msg msg; + + if (read(uffd, &msg, sizeof(msg)) <= 0) + return (void *)1; + + req.dst = msg.arg.pagefault.address; + req.src = (__u64)zero; + req.len = 4096; + req.mode = 0; + + if (ioctl(uffd, UFFDIO_COPY, &req)) + return (void *)1; + + return (void *)0; +} + +int test_userfaultfd(void) +{ + struct uffdio_register uffdio_register; + struct uffdio_api uffdio_api; + struct sigaction sa; + pthread_t thread; + void *res; + int uffd; + + sa.sa_sigaction = test_access_fix_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (uffd < 0) { + printf("[SKIP]\tUserfaultfd unavailable.\n"); + return 0; + } + + reset_test_shstk(0); + + uffdio_api.api = UFFD_API; + uffdio_api.features = 0; + if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + goto err; + + uffdio_register.range.start = (__u64)shstk_ptr; + uffdio_register.range.len = 4096; + uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING; + if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register)) + goto err; + + if (pthread_create(&thread, NULL, &uffd_thread, &uffd)) + goto err; + + reset_shstk(shstk_ptr); + test_shstk_access(shstk_ptr); + + if (pthread_join(thread, &res)) + goto err; + + if (test_shstk_access(shstk_ptr)) + goto err; + + free_shstk(shstk_ptr); + + signal(SIGSEGV, SIG_DFL); + + if (!res) + printf("[OK]\tUserfaultfd test\n"); + return !!res; +err: + free_shstk(shstk_ptr); + close(uffd); + signal(SIGSEGV, SIG_DFL); + return 1; +} + +/* + * Too complicated to pull it out of the 32 bit header, but also get the + * 64 bit one needed above. Just define a copy here. + */ +#define __NR_compat_sigaction 67 + +/* + * Call 32 bit signal handler to get 32 bit signals ABI. Make sure + * to push the registers that will get clobbered. + */ +int sigaction32(int signum, const struct sigaction *restrict act, + struct sigaction *restrict oldact) +{ + register long syscall_reg asm("eax") = __NR_compat_sigaction; + register long signum_reg asm("ebx") = signum; + register long act_reg asm("ecx") = (long)act; + register long oldact_reg asm("edx") = (long)oldact; + int ret = 0; + + asm volatile ("int $0x80;" + : "=a"(ret), "=m"(oldact) + : "r"(syscall_reg), "r"(signum_reg), "r"(act_reg), + "r"(oldact_reg) + : "r8", "r9", "r10", "r11" + ); + + return ret; +} + +sigjmp_buf jmp_buffer; + +void segv_gp_handler(int signum, siginfo_t *si, void *uc) +{ + segv_triggered = true; + + /* + * To work with old glibc, this can't rely on siglongjmp working with + * shadow stack enabled, so disable shadow stack before siglongjmp(). + */ + ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK); + siglongjmp(jmp_buffer, -1); +} + +/* + * Transition to 32 bit mode and check that a #GP triggers a segfault. + */ +int test_32bit(void) +{ + struct sigaction sa; + struct sigaction *sa32; + + /* Create sigaction in 32 bit address range */ + sa32 = mmap(0, 4096, PROT_READ | PROT_WRITE, + MAP_32BIT | MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); + sa32->sa_flags = SA_SIGINFO; + + sa.sa_sigaction = segv_gp_handler; + if (sigaction(SIGSEGV, &sa, NULL)) + return 1; + sa.sa_flags = SA_SIGINFO; + + segv_triggered = false; + + /* Make sure segv_triggered is set before triggering the #GP */ + asm volatile("" : : : "memory"); + + /* + * Set handler to somewhere in 32 bit address space + */ + sa32->sa_handler = (void *)sa32; + if (sigaction32(SIGUSR1, sa32, NULL)) + return 1; + + if (!sigsetjmp(jmp_buffer, 1)) + raise(SIGUSR1); + + if (segv_triggered) + printf("[OK]\t32 bit test\n"); + + return !segv_triggered; +} + +int main(int argc, char *argv[]) +{ + int ret = 0; + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_SHSTK)) { + printf("[SKIP]\tCould not enable Shadow stack\n"); + return 1; + } + + if (ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK)) { + ret = 1; + printf("[FAIL]\tDisabling shadow stack failed\n"); + } + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_SHSTK)) { + printf("[SKIP]\tCould not re-enable Shadow stack\n"); + return 1; + } + + if (ARCH_PRCTL(ARCH_SHSTK_ENABLE, ARCH_SHSTK_WRSS)) { + printf("[SKIP]\tCould not enable WRSS\n"); + ret = 1; + goto out; + } + + /* Should have succeeded if here, but this is a test, so double check. */ + if (!get_ssp()) { + printf("[FAIL]\tShadow stack disabled\n"); + return 1; + } + + if (test_shstk_pivot()) { + ret = 1; + printf("[FAIL]\tShadow stack pivot\n"); + goto out; + } + + if (test_shstk_faults()) { + ret = 1; + printf("[FAIL]\tShadow stack fault test\n"); + goto out; + } + + if (test_shstk_violation()) { + ret = 1; + printf("[FAIL]\tShadow stack violation test\n"); + goto out; + } + + if (test_gup()) { + ret = 1; + printf("[FAIL]\tShadow shadow stack gup\n"); + goto out; + } + + if (test_mprotect()) { + ret = 1; + printf("[FAIL]\tShadow shadow mprotect test\n"); + goto out; + } + + if (test_userfaultfd()) { + ret = 1; + printf("[FAIL]\tUserfaultfd test\n"); + goto out; + } + + if (test_32bit()) { + ret = 1; + printf("[FAIL]\t32 bit test\n"); + } + + return ret; + +out: + /* + * Disable shadow stack before the function returns, or there will be a + * shadow stack violation. + */ + if (ARCH_PRCTL(ARCH_SHSTK_DISABLE, ARCH_SHSTK_SHSTK)) { + ret = 1; + printf("[FAIL]\tDisabling shadow stack failed\n"); + } + + return ret; +} +#endif From patchwork Thu Jan 19 21:23:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46014 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp558968wrn; Thu, 19 Jan 2023 13:44:33 -0800 (PST) X-Google-Smtp-Source: AMrXdXtlW6A3XMPufkyvE88OJ6mwvS1BN2w+JyC41nxdFFoAAmU9CRlJrYI/PTqCi5mmzyk1duha X-Received: by 2002:a17:907:c29c:b0:84d:2171:d79 with SMTP id tk28-20020a170907c29c00b0084d21710d79mr12528315ejc.54.1674164672900; Thu, 19 Jan 2023 13:44:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164672; cv=none; d=google.com; s=arc-20160816; b=V+XcKpznJZczj6WSqwy3TqGx0vS97BLFv4Z920q5lGPoxGWMKDbJBZlg8XVeKqLT// S+HVCfSDmXcBEQ2C+3ZYPHEgWIvuCBRa6o10fd83zBDTmpB+1MScYZeIXky6oen1VJ4+ 5BaWWsyigzefd1fBLG+j82S9NOWIvi5OkZMycbgBXqpjga3s9Dv9lHMMYF9kYJc7G9W/ NU0VwMqy5tpvxBj9QfVtliieC2SRJ03qa/3SWbPCGRCqFzO607/uYZRqUY8QzOqYxGZi UEOAGQ1N0b6jG+aqghk0q126vt8uR2jP3biwfAk/XAx83oakrYUNYrqm6KBrGyDxWjNR Kj2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=UIaubgF5cxlgy9UntqyzMGFahvGgA9BhdHUjWEg2kcA=; b=N+CsvQvtYuZwXK+aigwHMJ8FlAKRy4X9GSbr5+kyvPdDXF90F/s2VHNlqM1Z43s2x7 ZKjpqmO0y2q0tfGjvO5jTxH6vRPdTKlgR6pdX+HKYu1R2VvTCQ0Z05MQLR4flj8uvmSM HHYpa15UEmlGyMf8TGkWKx9nXFivhSO0LTzQP3qh/6exuiwFsrp8nPBozLY0AGfz/yKO dNaGCELH/uGy1dlGhEx1qALBZ2GsmE9dnTy5sDQkVoHB/kIVB3QwcOjqwRhNDb9xZF4D Te77Y2seDXn6O43zHXbTbrUqs+pkpeQM6jElHxcz3+JmChPqPZ72Xsh3bs5Jlm/FN3Bm BVmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PD1B4p16; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xg8-20020a170907320800b00836ecced351si43395621ejb.380.2023.01.19.13.44.08; Thu, 19 Jan 2023 13:44:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PD1B4p16; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231307AbjASVnf (ORCPT + 99 others); Thu, 19 Jan 2023 16:43:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231178AbjASViw (ORCPT ); Thu, 19 Jan 2023 16:38:52 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42C0F10A97; Thu, 19 Jan 2023 13:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163679; x=1705699679; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Y8HP+6SI6A4E79h0YyqSt+wn/k0j51I/AMnEXqlWOFs=; b=PD1B4p16BOBls0XwR8Y2cdpmts1U9YPGrfE8GvFU1S45++EsnSMBuZ8o 2m3MWRM0IYKoVc1s5p/dsaRnxYTqeu+fuO+6Y+Y1DRE1/P+8gqe7+z4Mo pdjXvTQFkMUmT6QREPzqPu+yN1ZFtHiFkB06cts1pYYS+sVILde0h0Fkg +fut8mzK1cURAnspl+OknZENH3sFNIF6eJp2hcg7VACPbkcwg0b00TUV9 p0vREIpGRJgeT0wWPX2NxWeNJPgmjXk1YP1dBqcbBurTr6OMMSYvApw1O MJW0wCsn5Sd59rZ84eFjXBcj1tCra9pgfiBJZq0ZT1kTNNICTDdCIBCom A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323120040" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323120040" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:22 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139185" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139185" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:20 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 36/39] x86/fpu: Add helper for initing features Date: Thu, 19 Jan 2023 13:23:14 -0800 Message-Id: <20230119212317.8324-37-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488896057750504?= X-GMAIL-MSGID: =?utf-8?q?1755488896057750504?= If an xfeature is saved in a buffer, the xfeature's bit will be set in xsave->header.xfeatures. The CPU may opt to not save the xfeature if it is in it's init state. In this case the xfeature buffer address cannot be retrieved with get_xsave_addr(). Future patches will need to handle the case of writing to an xfeature that may not be saved. So provide helpers to init an xfeature in an xsave buffer. This could of course be done directly by reaching into the xsave buffer, however this would not be robust against future changes to optimize the xsave buffer by compacting it. In that case the xsave buffer would need to be re-arranged as well. So the logic properly belongs encapsulated in a helper where the logic can be unified. Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Rick Edgecombe --- v2: - New patch arch/x86/kernel/fpu/xstate.c | 58 +++++++++++++++++++++++++++++------- arch/x86/kernel/fpu/xstate.h | 6 ++++ 2 files changed, 53 insertions(+), 11 deletions(-) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 13a80521dd51..3ff80be0a441 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -934,6 +934,24 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) return (void *)xsave + xfeature_get_offset(xcomp_bv, xfeature_nr); } +static int xsave_buffer_access_checks(int xfeature_nr) +{ + /* + * Do we even *have* xsave state? + */ + if (!boot_cpu_has(X86_FEATURE_XSAVE)) + return 1; + + /* + * We should not ever be requesting features that we + * have not enabled. + */ + if (WARN_ON_ONCE(!xfeature_enabled(xfeature_nr))) + return 1; + + return 0; +} + /* * Given the xsave area and a state inside, this function returns the * address of the state. @@ -954,17 +972,7 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr) */ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) { - /* - * Do we even *have* xsave state? - */ - if (!boot_cpu_has(X86_FEATURE_XSAVE)) - return NULL; - - /* - * We should not ever be requesting features that we - * have not enabled. - */ - if (WARN_ON_ONCE(!xfeature_enabled(xfeature_nr))) + if (xsave_buffer_access_checks(xfeature_nr)) return NULL; /* @@ -984,6 +992,34 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) return __raw_xsave_addr(xsave, xfeature_nr); } +/* + * Given the xsave area and a state inside, this function + * initializes an xfeature in the buffer. + * + * get_xsave_addr() will return NULL if the feature bit is + * not present in the header. This function will make it so + * the xfeature buffer address is ready to be retrieved by + * get_xsave_addr(). + * + * Inputs: + * xstate: the thread's storage area for all FPU data + * xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, + * XFEATURE_SSE, etc...) + * Output: + * 1 if the feature cannot be inited, 0 on success + */ +int init_xfeature(struct xregs_state *xsave, int xfeature_nr) +{ + if (xsave_buffer_access_checks(xfeature_nr)) + return 1; + + /* + * Mark the feature inited. + */ + xsave->header.xfeatures |= BIT_ULL(xfeature_nr); + return 0; +} + #ifdef CONFIG_ARCH_HAS_PKEYS /* diff --git a/arch/x86/kernel/fpu/xstate.h b/arch/x86/kernel/fpu/xstate.h index a4ecb04d8d64..dc06f63063ee 100644 --- a/arch/x86/kernel/fpu/xstate.h +++ b/arch/x86/kernel/fpu/xstate.h @@ -54,6 +54,12 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system_xstate(unsigned int legacy_size); extern void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr); +extern int init_xfeature(struct xregs_state *xsave, int xfeature_nr); + +static inline int xfeature_saved(struct xregs_state *xsave, int xfeature_nr) +{ + return xsave->header.xfeatures & BIT_ULL(xfeature_nr); +} static inline u64 xfeatures_mask_supervisor(void) { From patchwork Thu Jan 19 21:23:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp559150wrn; Thu, 19 Jan 2023 13:45:03 -0800 (PST) X-Google-Smtp-Source: AMrXdXtNIfRsuMCVOZYPctzqT1BU8pdBOGipKTqDt/KHe6yA1cXwEgAZ3MYm5JYCyw+foyidB70a X-Received: by 2002:a17:906:b6d4:b0:877:9632:254 with SMTP id ec20-20020a170906b6d400b0087796320254mr146719ejb.42.1674164703024; Thu, 19 Jan 2023 13:45:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164703; cv=none; d=google.com; s=arc-20160816; b=nU497hlfSCiDpbnK6vBQeGRKHBSqjHWKXiKWYXoDprQ9q8EPQ788ssA34AEdrE1EEz mm4aDHum6w6RU/9DiBQkslRCpX7OhxARlQCkAkcKaOWbXGBEEAb1BizVPsIUj1ImMrq6 XyWEEhepSxOUjNcBx0ySuT4FJArFdqi1i5u5DUD7uciH7tOpK6YP7RX6jP4uFq/2ajQA +7zF2vEUl/s2X+FhG43uAa9By/5rphe1GtlNHp5kbVTnszas7y4J7fFbxwxLdDzH5ysh Se5xPoA5uOb6vhOox2ZJ9kQ2p/i2rFwfJ8fv7DS0x7adZ/Fg/KoNmPjrW/bkb/cYwp/d darw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=dsrFj8bpNT5MvUDiRiCUjp10IycWIyaFRdvlnCTkRv8=; b=C6J6KiS0pqfLfgcDXbNw7RxHdfvLXmMpqXue6VryHMxJjNxA6gOTjVV90zc8imI/hT gd8OS4PwUgH4lQN3fLu7tdKT+lS6jcZZXV6HOAPsCE3+4yqfTl0+SSCKCbrzSxmOPEhi 3Xz2Z956vRwaQqoy0r5H0HQxmjSAD9aIsPgPhohI1Mn0x/Vr7KnRdnjM72Vmm1XOBYjL GYVe8OPXve7om7AgPb7SES29M6BTbEo3GxacD9N5JVD47TZyhzyDFu35Ix+tlc4XMVkf c+HBLg0H4pmAaIoIv9oq4V0WqWv+nEszNuwvE2glmpmGYhlCXM/YUqm1BG4T/OmpYxH/ M4Vg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eRNpu+2Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sc20-20020a1709078a1400b008366ae33e68si19028190ejc.348.2023.01.19.13.44.39; Thu, 19 Jan 2023 13:45:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eRNpu+2Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230455AbjASVn4 (ORCPT + 99 others); Thu, 19 Jan 2023 16:43:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231181AbjASViy (ORCPT ); Thu, 19 Jan 2023 16:38:54 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C876859565; Thu, 19 Jan 2023 13:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163679; x=1705699679; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0itXeYpmwui1qYObTMssZc6TAQWFWu07BF3T37hLMrE=; b=eRNpu+2QSZbYstpqp/lbEyp6mZfeqc9WAf7eCHkhwDAfn2GiRcH1y07a X6tTdY5aaCwz/GhXyvSusqh0NnzOoQjHEvz0+ImZeAFCZDyFhZORLjaFw 0YeaX2wSo1ooLNKgf6y0LOiU9AN4Bw6hXInncZRfxsm2r4VHoQiddoFx5 lP5MrQDY3tvyX1jDA7f4aewEul/LVmAwi/CMHUbSe6up/pqOaAib4sGmj xSzJ4IyF5KXGL0Qu+4S9BBT15zv5tLJ+I2xkB8txsWWW+dg45yIO0v/GV +C4mL6NEembUQOX032khqFof0i5Ht+Pn02N1h7fRQnJcL/HjY5EATzDjV g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323120061" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323120061" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139192" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139192" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:22 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 37/39] x86: Add PTRACE interface for shadow stack Date: Thu, 19 Jan 2023 13:23:15 -0800 Message-Id: <20230119212317.8324-38-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488927561510831?= X-GMAIL-MSGID: =?utf-8?q?1755488927561510831?= From: Yu-cheng Yu Some applications (like GDB) would like to tweak shadow stack state via ptrace. This allows for existing functionality to continue to work for seized shadow stack applications. Provide an regset interface for manipulating the shadow stack pointer (SSP). There is already ptrace functionality for accessing xstate, but this does not include supervisor xfeatures. So there is not a completely clear place for where to put the shadow stack state. Adding it to the user xfeatures regset would complicate that code, as it currently shares logic with signals which should not have supervisor features. Don't add a general supervisor xfeature regset like the user one, because it is better to maintain flexibility for other supervisor xfeatures to define their own interface. For example, an xfeature may decide not to expose all of it's state to userspace, as is actually the case for shadow stack ptrace functionality. A lot of enum values remain to be used, so just put it in dedicated shadow stack regset. The only downside to not having a generic supervisor xfeature regset, is that apps need to be enlightened of any new supervisor xfeature exposed this way (i.e. they can't try to have generic save/restore logic). But maybe that is a good thing, because they have to think through each new xfeature instead of encountering issues when new a new supervisor xfeature was added. By adding a shadow stack regset, it also has the effect of including the shadow stack state in a core dump, which could be useful for debugging. The shadow stack specific xstate includes the SSP, and the shadow stack and WRSS enablement status. Enabling shadow stack or wrss in the kernel involves more than just flipping the bit. The kernel is made aware that it has to do extra things when cloning or handling signals. That logic is triggered off of separate feature enablement state kept in the task struct. So the flipping on HW shadow stack enforcement without notifying the kernel to change its behavior would severely limit what an application could do without crashing, and the results would depend on kernel internal implementation details. There is also no known use for controlling this state via prtace today. So only expose the SSP, which is something that userspace already has indirect control over. Tested-by: Pengfei Xu Tested-by: John Allen Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yu-cheng Yu Reviewed-by: Kees Cook --- v5: - Check shadow stack enablement status for tracee (rppt) - Fix typo in comment v4: - Make shadow stack only. Reduce to only supporting SSP register, and remove CET references (peterz) - Add comment to not use 0x203, because binutils already looks for it in coredumps. (Christina Schimpe) v3: - Drop dependence on thread.shstk.size, and use thread.features bits - Drop 32 bit support v2: - Check alignment on ssp. - Block IBT bits. - Handle init states instead of returning error. - Add verbose commit log justifying the design. arch/x86/include/asm/fpu/regset.h | 7 +-- arch/x86/kernel/fpu/regset.c | 87 +++++++++++++++++++++++++++++++ arch/x86/kernel/ptrace.c | 12 +++++ include/uapi/linux/elf.h | 2 + 4 files changed, 105 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/fpu/regset.h b/arch/x86/include/asm/fpu/regset.h index 4f928d6a367b..697b77e96025 100644 --- a/arch/x86/include/asm/fpu/regset.h +++ b/arch/x86/include/asm/fpu/regset.h @@ -7,11 +7,12 @@ #include -extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active; +extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active, + ssp_active; extern user_regset_get2_fn fpregs_get, xfpregs_get, fpregs_soft_get, - xstateregs_get; + xstateregs_get, ssp_get; extern user_regset_set_fn fpregs_set, xfpregs_set, fpregs_soft_set, - xstateregs_set; + xstateregs_set, ssp_set; /* * xstateregs_active == regset_fpregs_active. Please refer to the comment diff --git a/arch/x86/kernel/fpu/regset.c b/arch/x86/kernel/fpu/regset.c index 6d056b68f4ed..10c092d21809 100644 --- a/arch/x86/kernel/fpu/regset.c +++ b/arch/x86/kernel/fpu/regset.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "context.h" #include "internal.h" @@ -174,6 +175,92 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, return ret; } + +#ifdef CONFIG_X86_USER_SHADOW_STACK +int ssp_active(struct task_struct *target, const struct user_regset *regset) +{ + if (target->thread.features & ARCH_SHSTK_SHSTK) + return regset->n; + + return 0; +} + +int ssp_get(struct task_struct *target, const struct user_regset *regset, + struct membuf to) +{ + struct fpu *fpu = &target->thread.fpu; + struct cet_user_state *cetregs; + + if (!boot_cpu_has(X86_FEATURE_USER_SHSTK)) + return -ENODEV; + + sync_fpstate(fpu); + cetregs = get_xsave_addr(&fpu->fpstate->regs.xsave, XFEATURE_CET_USER); + if (!cetregs) { + /* + * The registers are the in the init state. The init values for + * these regs are zero, so just zero the output buffer. + */ + membuf_zero(&to, sizeof(cetregs->user_ssp)); + return 0; + } + + return membuf_write(&to, (unsigned long *)&cetregs->user_ssp, + sizeof(cetregs->user_ssp)); +} + +int ssp_set(struct task_struct *target, const struct user_regset *regset, + unsigned int pos, unsigned int count, + const void *kbuf, const void __user *ubuf) +{ + struct fpu *fpu = &target->thread.fpu; + struct xregs_state *xsave = &fpu->fpstate->regs.xsave; + struct cet_user_state *cetregs; + unsigned long user_ssp; + int r; + + if (!boot_cpu_has(X86_FEATURE_USER_SHSTK) || + !ssp_active(target, regset)) + return -ENODEV; + + r = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &user_ssp, 0, -1); + if (r) + return r; + + /* + * Some kernel instructions (IRET, etc) can cause exceptions in the case + * of disallowed CET register values. Just prevent invalid values. + */ + if ((user_ssp >= TASK_SIZE_MAX) || !IS_ALIGNED(user_ssp, 8)) + return -EINVAL; + + fpu_force_restore(fpu); + + /* + * Don't want to init the xfeature until the kernel will definitely + * overwrite it, otherwise if it inits and then fails out, it would + * end up initing it to random data. + */ + if (!xfeature_saved(xsave, XFEATURE_CET_USER) && + WARN_ON(init_xfeature(xsave, XFEATURE_CET_USER))) + return -ENODEV; + + cetregs = get_xsave_addr(xsave, XFEATURE_CET_USER); + if (WARN_ON(!cetregs)) { + /* + * This shouldn't ever be NULL because it was successfully + * inited above if needed. The only scenario would be if an + * xfeature was somehow saved in a buffer, but not enabled in + * xsave. + */ + return -ENODEV; + } + + cetregs->user_ssp = user_ssp; + return 0; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + #if defined CONFIG_X86_32 || defined CONFIG_IA32_EMULATION /* diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index dfaa270a7cc9..095f04bdabdc 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -58,6 +58,7 @@ enum x86_regset_64 { REGSET64_FP, REGSET64_IOPERM, REGSET64_XSTATE, + REGSET64_SSP, }; #define REGSET_GENERAL \ @@ -1267,6 +1268,17 @@ static struct user_regset x86_64_regsets[] __ro_after_init = { .active = ioperm_active, .regset_get = ioperm_get }, +#ifdef CONFIG_X86_USER_SHADOW_STACK + [REGSET64_SSP] = { + .core_note_type = NT_X86_SHSTK, + .n = 1, + .size = sizeof(u64), + .align = sizeof(u64), + .active = ssp_active, + .regset_get = ssp_get, + .set = ssp_set + }, +#endif }; static const struct user_regset_view user_x86_64_view = { diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index 4c6a8fa5e7ed..413a15c07121 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -406,6 +406,8 @@ typedef struct elf64_shdr { #define NT_386_TLS 0x200 /* i386 TLS slots (struct user_desc) */ #define NT_386_IOPERM 0x201 /* x86 io permission bitmap (1=deny) */ #define NT_X86_XSTATE 0x202 /* x86 extended state using xsave */ +/* Old binutils treats 0x203 as a CET state */ +#define NT_X86_SHSTK 0x204 /* x86 SHSTK state */ #define NT_S390_HIGH_GPRS 0x300 /* s390 upper register halves */ #define NT_S390_TIMER 0x301 /* s390 timer register */ #define NT_S390_TODCMP 0x302 /* s390 TOD clock comparator register */ From patchwork Thu Jan 19 21:23:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46015 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp559151wrn; Thu, 19 Jan 2023 13:45:03 -0800 (PST) X-Google-Smtp-Source: AMrXdXsECNJgNpGiHOKCBYoAvvRrzAFY0SUhisqQlYNgKW+RL+Nnc5SpMXKmsEEkAOy5yjmKdUvp X-Received: by 2002:a17:906:dd4:b0:877:667b:f1e2 with SMTP id p20-20020a1709060dd400b00877667bf1e2mr4930855eji.11.1674164703020; Thu, 19 Jan 2023 13:45:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164703; cv=none; d=google.com; s=arc-20160816; b=mUD6NTwU3cyLz2y4WQsauTTiI9j4lDcShA2m4ytdcBKhaKt+XiazxNE0/e1sqCfAhS Z3ohVpTfFjvMC9ICwrdQLKU4bps5eHvSXTSAXWuLgd8q5BdNEVj51wjJZTAnblBSv2/I pTPRleShr4KrqWXhOzH2cp4qpdR+Z4MrIKqvjBin5VP4q6eNOdG26vHtNLbWwTWNprvq V7HbwZRGxnMp3bbAEYtJw47E/Q3e7gQpyHy/VDBo5LxQbYqYUv2FeRiIM2ImZ1w9fZlc VEQl34X8HpZqBxEPzCxZUbIKklSDk9Y6/Vqgmf2OzJLqfA15fGXV55TcBhB0xfiMisTV 9qWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=JxHOoebEC0ntNY7urTWVeuLnqJvKXHl/N1WxuofeGTg=; b=UJOLZUVTbgUO/QqN/IgZqxPxbKc+Zpbpj8C7iXzUfOU0G6UJoJ+SykSp0rQjT0/tdq Mf8IciBFHO9jiDSoLnZSWo9ZXHe1oGn6iTvhcVhpFv+VKqbmh6r2+JBcQDPJ81hfAdMl zynafh2BqoZt7/e00dRaEpPdwRllDHBBCpeY+CT7JYeBTlASfnjY5JNOuIBuS5KzkhXS uHaP7ZU73M0mRDwS0TOB2JOlG7ncqLyQKsGYph3kaLKxtv46b0qKzmg0b7atrDFXROpo pA+/3al/5RBcgz/IkMACcliQFCHWO6LK1gOvkZzPV4fqmZMfXgjYG8ShVZ8IMb9F2f4f 3gKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UoeG49WB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ww11-20020a170907084b00b007822665aa36si44780496ejb.430.2023.01.19.13.44.32; Thu, 19 Jan 2023 13:45:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UoeG49WB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231319AbjASVnt (ORCPT + 99 others); Thu, 19 Jan 2023 16:43:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230410AbjASVix (ORCPT ); Thu, 19 Jan 2023 16:38:53 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C88086F885; Thu, 19 Jan 2023 13:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163679; x=1705699679; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=SwMjGEgQ4NIbBGCsK+je+PfgR9ALTBIfAb8eu9YmH38=; b=UoeG49WBTwSDLoZTTRKIZ0iTYsjCPhk5ihw1NIZmpRY3+dGuwk9nYa5g WSsvGDCLLicq2lSnpAU3CAJJtO+VG606JO8v9nhJyIL/Sc/nmeIG0buNT G7xrvMo64wLfZW7k4n1ecjE/84Nh2SsjfiRjaGcZzM3fzearX+tsaD3cD u8js3e2C/BF4q5eikb9Ki3ngSuyPhu9U1t1v2IMaykjl8uYeYno8U3CfS iWJMLE6I+TdqUIBugTkl6STFM+xvR+LkeSQ5r49ij0GyBLqFuncUwwm5Q YTbJ8wU27/93UV2d9mT1CFVyR7G9rLevPxP8ug8F8r9on+XNneXMVovVm Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323120084" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323120084" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139196" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139196" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:23 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Mike Rapoport Subject: [PATCH v5 38/39] x86/shstk: Add ARCH_SHSTK_UNLOCK Date: Thu, 19 Jan 2023 13:23:16 -0800 Message-Id: <20230119212317.8324-39-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755488927787979819?= X-GMAIL-MSGID: =?utf-8?q?1755488927787979819?= From: Mike Rapoport Userspace loaders may lock features before a CRIU restore operation has the chance to set them to whatever state is required by the process being restored. Allow a way for CRIU to unlock features. Add it as an arch_prctl() like the other shadow stack operations, but restrict it being called by the ptrace arch_pctl() interface. Reviewed-by: Kees Cook Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Mike Rapoport [Merged into recent API changes, added commit log and docs] Signed-off-by: Rick Edgecombe --- v4: - Add to docs that it is ptrace only. - Remove "CET" references v3: - Depend on CONFIG_CHECKPOINT_RESTORE (Kees) Documentation/x86/shstk.rst | 4 ++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 9 +++++++-- 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst index f2e6f323cf68..e8ed5fc0f7ae 100644 --- a/Documentation/x86/shstk.rst +++ b/Documentation/x86/shstk.rst @@ -73,6 +73,10 @@ arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) are ignored. The mask is ORed with the existing value. So any feature bits set here cannot be enabled or disabled afterwards. +arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) + Unlock features. 'features' is a mask of all features to unlock. All + bits set are processed, unset bits are ignored. Only works via ptrace. + The return values are as follows. On success, return 0. On error, errno can be:: diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index e31495668056..200efbbe5809 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -25,6 +25,7 @@ #define ARCH_SHSTK_ENABLE 0x5001 #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004 /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 71094c8a305f..d368854fa9c4 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -835,6 +835,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_SHSTK_ENABLE: case ARCH_SHSTK_DISABLE: case ARCH_SHSTK_LOCK: + case ARCH_SHSTK_UNLOCK: return shstk_prctl(task, option, arg2); default: ret = -EINVAL; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index 07142e6f05f6..a639119a21c5 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -452,9 +452,14 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features) return 0; } - /* Don't allow via ptrace */ - if (task != current) + /* Only allow via ptrace */ + if (task != current) { + if (option == ARCH_SHSTK_UNLOCK && IS_ENABLED(CONFIG_CHECKPOINT_RESTORE)) { + task->thread.features_locked &= ~features; + return 0; + } return -EINVAL; + } /* Do not allow to change locked features */ if (features & task->thread.features_locked) From patchwork Thu Jan 19 21:23:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 46018 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp560250wrn; Thu, 19 Jan 2023 13:47:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXvPV2JTh8IfnHxSmXfTElQ0N0+fds8Uum06x1SMG2781IvehCt9pRBjK3oMHLc9A8XdWa0D X-Received: by 2002:a17:906:4557:b0:84d:3a95:cdf5 with SMTP id s23-20020a170906455700b0084d3a95cdf5mr12068456ejq.10.1674164870889; Thu, 19 Jan 2023 13:47:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674164870; cv=none; d=google.com; s=arc-20160816; b=AYlr/u9J++af+tRB5m1FUL5UvxfRXZtUrGDzXf+TLOz1BWa396Xj8cpd0kt9C3EoXT 9p4+dwE5fk6DomIQB3IfRFgUxg+fTxX9WX4TJGsiUKCIBCMz7DtcVWRE2smj2GFgIglO hNbM1qc/nGmSuewE72pRjWVi3T8DBD91Tld6gTHAui/uGV58KTjiSBuDL8H5vKzlGUA9 SzDBadqUmzvhWGH/PAfNJN7ByZ9eKK3HmX6MljUOEelTlwnOj5up2wuQPcjIZw+xSKqM 4PvB2DHgN5VXAv430BkOIwQhitRtyExMfbYD3p6Bwzjx2CWcRQs+xWvBcHFsQPFy/+kn jvkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=cjMBHlgASRiVZwcEv7s62E9H2RxSF9urW8wYPqkrv6A=; b=jsK+98yNe0jF9LLTpXZDSP0WM4NU+hJcc4UaZt9ACpBYZ97JIXSfxIlKrG5S8WlUmu X4ooaJU4yv8jpUjgsNcBQYhSFRmdd2592QEO/B7GfzjjPFcvaM6DhX2WVe1YMaq9KO2u SSJquUcI76YDF6TdFXhh4K6e3Ouj7YGEL+LMntImDLXajwV4yTSvAi9EtYOJd23IxIrA tTI50iEW/o7SXiJa3X/NYfH8NQO349APdnWYzRjk5SE0wmMhP6/87kvGa2YqiqJ+UrIs g6+TtcwYFiIHAK2uJWoMyv91i9jJqlv/CL0DOuH/IVDjL78vbWdIeiVlGI3usYXr+2ze zJ4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ipgO3ufE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f20-20020a056402355400b0049e28a9411bsi14922440edd.259.2023.01.19.13.47.25; Thu, 19 Jan 2023 13:47:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ipgO3ufE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231332AbjASVoF (ORCPT + 99 others); Thu, 19 Jan 2023 16:44:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231180AbjASViy (ORCPT ); Thu, 19 Jan 2023 16:38:54 -0500 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F0FF7C875; Thu, 19 Jan 2023 13:28:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163680; x=1705699680; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Q/roafiLg5FWGNhLb00ov8wc44ATIJKREdVQC5GYlS4=; b=ipgO3ufEFgEAo7SYyiJvYCPkVOXpWCDkMo3S8b3DAyLtfV57O8wkMZuo siVy1P5kxgXiI7i4DcMjfiji+smZpsdlh8QUtoyAiTp0OKuH7ZAoJ3n6F nUPRDX9oBvR6AjuXgphJQSPSWUzYXZamr7mg9rfaS7H4hNvfeCT85ZABu RMC/5FZWgxKrzcjUOLXrhITPQZQhI1VrQjp8SlQTpQpgA/kicoyEELClg q5iYS2D21V7wMttoEvG1uO8RuSLgb9QbJp/RzD8mgFqTbTqYXOYiivSIe P4QQ2rzlsD+9fUzvsNFeKxNq1/G4jaU0SkLuHdEz5Xa1lrDCvt9jekpgA A==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323120102" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323120102" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139201" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139201" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:24:25 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com Subject: [PATCH v5 39/39] x86/shstk: Add ARCH_SHSTK_STATUS Date: Thu, 19 Jan 2023 13:23:17 -0800 Message-Id: <20230119212317.8324-40-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755489103510917520?= X-GMAIL-MSGID: =?utf-8?q?1755489103510917520?= CRIU and GDB need to get the current shadow stack and WRSS enablement status. This information is already available via /proc/pid/status, but this is inconvenient for CRIU because it involves parsing the text output in an area of the code where this is difficult. Provide a status arch_prctl(), ARCH_SHSTK_STATUS for retrieving the status. Have arg2 be a userspace address, and make the new arch_prctl simply copy the features out to userspace. Tested-by: Pengfei Xu Suggested-by: Mike Rapoport Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix typo in commit log v4: - New patch Documentation/x86/shstk.rst | 6 ++++++ arch/x86/include/asm/shstk.h | 4 ++-- arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 8 +++++++- 5 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Documentation/x86/shstk.rst b/Documentation/x86/shstk.rst index e8ed5fc0f7ae..7f4af798794e 100644 --- a/Documentation/x86/shstk.rst +++ b/Documentation/x86/shstk.rst @@ -77,6 +77,11 @@ arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) Unlock features. 'features' is a mask of all features to unlock. All bits set are processed, unset bits are ignored. Only works via ptrace. +arch_prctl(ARCH_SHSTK_STATUS, unsigned long addr) + Copy the currently enabled features to the address passed in addr. The + features are described using the bits passed into the others in + 'features'. + The return values are as follows. On success, return 0. On error, errno can be:: @@ -84,6 +89,7 @@ be:: -ENOTSUPP if the feature is not supported by the hardware or kernel. -EINVAL arguments (non existing feature, etc) + -EFAULT if could not copy information back to userspace The feature's bits supported are:: diff --git a/arch/x86/include/asm/shstk.h b/arch/x86/include/asm/shstk.h index 746c040f7cb6..73de995f55ca 100644 --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -14,7 +14,7 @@ struct thread_shstk { u64 size; }; -long shstk_prctl(struct task_struct *task, int option, unsigned long features); +long shstk_prctl(struct task_struct *task, int option, unsigned long arg2); void reset_thread_features(void); int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size, @@ -24,7 +24,7 @@ int setup_signal_shadow_stack(struct ksignal *ksig); int restore_signal_shadow_stack(void); #else static inline long shstk_prctl(struct task_struct *task, int option, - unsigned long features) { return -EINVAL; } + unsigned long arg2) { return -EINVAL; } static inline void reset_thread_features(void) {} static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 200efbbe5809..1b85bc876c2d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -26,6 +26,7 @@ #define ARCH_SHSTK_DISABLE 0x5002 #define ARCH_SHSTK_LOCK 0x5003 #define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005 /* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index d368854fa9c4..dde43caf196e 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -836,6 +836,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_SHSTK_DISABLE: case ARCH_SHSTK_LOCK: case ARCH_SHSTK_UNLOCK: + case ARCH_SHSTK_STATUS: return shstk_prctl(task, option, arg2); default: ret = -EINVAL; diff --git a/arch/x86/kernel/shstk.c b/arch/x86/kernel/shstk.c index a639119a21c5..3b1433bd63c7 100644 --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -445,8 +445,14 @@ SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsi return alloc_shstk(addr, aligned_size, size, set_tok); } -long shstk_prctl(struct task_struct *task, int option, unsigned long features) +long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { + unsigned long features = arg2; + + if (option == ARCH_SHSTK_STATUS) { + return put_user(task->thread.features, (unsigned long __user *)arg2); + } + if (option == ARCH_SHSTK_LOCK) { task->thread.features_locked |= features; return 0;