From patchwork Mon Oct 9 12:08:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 150013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1824677vqo; Mon, 9 Oct 2023 05:16:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEvIovTZWcO8WeeaJFUmm64HkKdsiKWuqlhxcMHR4hXvn6GJ5b+8xsCam4pWoX+gKMLwnf3 X-Received: by 2002:a05:6808:3028:b0:3a7:1e98:80ad with SMTP id ay40-20020a056808302800b003a71e9880admr17902160oib.9.1696853763970; Mon, 09 Oct 2023 05:16:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696853763; cv=none; d=google.com; s=arc-20160816; b=pmVsRI7Xlgeym7F7Q0Vy915hEVDPKWvZwqPC9NgiATDsbwMLpezBEqtUET/7yutB+Z vBS+qCmw7pGzJSzZoMNSfWkNdYRlwTG6sZ4kXI/O15BZ9KBwcHsTUNxR1c6U+e97j2Ij BcDpeiuC31lL8g5Qi0/HLVsE/TfpQccFvXs69Z8acuACl18N6wJjtb6hHsZgcN5uWYsq 4gAtRpICmJEWbrvysgAgNB/Ow/pRpXh/G5nA+DOoVom94eeD/n0yAU0kDOG06ImHtzkl /F/qII7gFz0FTLpVDJdljITnETe0p9KsQN5V00cWIbww6Uxb44h2MJaYcMAbDLgFrq1W lXrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from :dkim-signature; bh=D1luWdQjCGgcA3yRUm5MGC+7IT3wPpdU5A/8hcfglQA=; fh=YwAsYQRB4CPZjeeoEIUs2LGTp0erkNx6S7AqgPP1nJ4=; b=vdyzHW9nMJhRgEVpB9CuHDK7mrUEiUo+j5mORJCtTiBwr8BsJftKMnbVbqqw3oSlm4 UR0FILtrfjz+7/ITZoZst5likSRnBVNF/U4kqo7ZufvfjYJ0I9Np2GEy9OLKaQ0wULkp Pt0Dl67jVA3yEJ3IyVtqY6N1Mp1j4o8J5BUzpc1q2fh38pY5lgkcL9h6pUceJfG/GgNg 8d69Avm/FIQdEY9UyBvIVo+SM3q8e9bPRUc2epLvYMGb/XEqMIAL34iDNalzEXo/wW7S oPkrTEOkkHRMYIWn70L7cXmQqdalkzGBfEm0ceLN4DZf2T5qiy1PEz98OKPiDyYZnzT2 6aHg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iKhWOUFK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id l71-20020a63884a000000b005891f742953si9293743pgd.29.2023.10.09.05.16.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 05:16:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iKhWOUFK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 9719880A90CD; Mon, 9 Oct 2023 05:16:00 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376294AbjJIMOz (ORCPT + 18 others); Mon, 9 Oct 2023 08:14:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346573AbjJIMOM (ORCPT ); Mon, 9 Oct 2023 08:14:12 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2C7619AA; Mon, 9 Oct 2023 05:13:00 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1240BC433C7; Mon, 9 Oct 2023 12:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696853580; bh=PsDxGrHqBjHeoGwMKffcey2x+5qoqTXr2+aWdiIPZRc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=iKhWOUFKZmqlPTCVntQFX3sftVawRKn9CWFuYhr1HacIW0QavY+74JLMYsIYeJNlh 5zE1wF6oyCriG8ouCwlBUl9VVUfFWlZWCaEXiFiIfbOgs28bt4ZmVgoSAoCS90M+py z6a5f8Z+72aCubjm1ZgHaCIw4uvZQDQMrK6gG8/wENb///QtE2n0X7CIt0zJZ3JAGA 9XJAkg0glYf6c2Z+wQFxF13Pt/h3zRtUtk0R24iXd85uiTglkBjQHu7NEz9uVCrBBp Yum8sKNjcrN3WMuyW/cV/hlv0dT6TDDm11P7zbHMTmqCFv/RO7SZ88ioxX/v9kOYm6 4bIaVcBgqkT9A== From: Mark Brown Date: Mon, 09 Oct 2023 13:08:54 +0100 Subject: [PATCH v6 20/38] arm64/gcs: Context switch GCS state for EL0 MIME-Version: 1.0 Message-Id: <20231009-arm64-gcs-v6-20-78e55deaa4dd@kernel.org> References: <20231009-arm64-gcs-v6-0-78e55deaa4dd@kernel.org> In-Reply-To: <20231009-arm64-gcs-v6-0-78e55deaa4dd@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Kees Cook , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.13-dev-0438c X-Developer-Signature: v=1; a=openpgp-sha256; l=6699; i=broonie@kernel.org; h=from:subject:message-id; bh=PsDxGrHqBjHeoGwMKffcey2x+5qoqTXr2+aWdiIPZRc=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBlI+2hbbP3VnYRsnb17lXU/A+9RkWvZGWWlHoKGK2+ qeseoAGJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZSPtoQAKCRAk1otyXVSH0K4AB/ 9+YBAaqYdbu7vj+lMgN58kY+vS5cul1puBFzMWybte5jONTfDRc1nanSU6A9YBwF32L4/Rwrnd5CvU M5UHfjpl6liN1DtXHDJ45kbOXiJr3LPLqfYBMUrlSt1FWJUUiNwS1zh2KzzgfHBxDipRAwKw+DWQDt tsKt7RIXqtfeJBUiw6tsMzIi0GsyGjma02uGPzTArfkULqyNhm0UhXKlh5VcEWq+aiH/hq+JmG0xwY IPcB7tJCZU/87LiZP134XiiSZXEBNSY6KceFUjtceNJu0EKVXqGpg1+mFwF54krOMNgcTkgZ0mVqeh YS2H2ubIwj+yX4SSSs5PDiyXntB0YF X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Spam-Status: No, score=2.4 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 09 Oct 2023 05:16:00 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779280132327426647 X-GMAIL-MSGID: 1779280132327426647 There are two registers controlling the GCS state of EL0, GCSPR_EL0 which is the current GCS pointer and GCSCRE0_EL1 which has enable bits for the specific GCS functionality enabled for EL0. Manage these on context switch and process lifetime events, GCS is reset on exec(). Also ensure that any changes to the GCS memory are visible to other PEs and that changes from other PEs are visible on this one by issuing a GCSB DSYNC when moving to or from a thread with GCS. Since the current GCS configuration of a thread will be visible to userspace we store the configuration in the format used with userspace and provide a helper which configures the system register as needed. On systems that support GCS we always allow access to GCSPR_EL0, this facilitates reporting of GCS faults if userspace implements disabling of GCS on error - the GCS can still be discovered and examined even if GCS has been disabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/gcs.h | 24 ++++++++++++++++ arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/process.c | 56 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/gcs.c | 39 ++++++++++++++++++++++++++ 5 files changed, 126 insertions(+) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 7c5e95218db6..04594ef59dad 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -48,4 +48,28 @@ static inline u64 gcsss2(void) return Xt; } +#ifdef CONFIG_ARM64_GCS + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return current->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE; +} + +void gcs_set_el0_mode(struct task_struct *task); +void gcs_free(struct task_struct *task); +void gcs_preserve_current_state(void); + +#else + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return false; +} + +static inline void gcs_set_el0_mode(struct task_struct *task) { } +static inline void gcs_free(struct task_struct *task) { } +static inline void gcs_preserve_current_state(void) { } + +#endif + #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index e5bc54522e71..c28681cf9721 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -179,6 +179,12 @@ struct thread_struct { u64 sctlr_user; u64 svcr; u64 tpidr2_el0; +#ifdef CONFIG_ARM64_GCS + unsigned int gcs_el0_mode; + u64 gcspr_el0; + u64 gcs_base; + u64 gcs_size; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 0fcc4eb1a7ab..84bac012f744 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -271,12 +272,32 @@ static void flush_tagged_addr_state(void) clear_thread_flag(TIF_TAGGED_ADDR); } +#ifdef CONFIG_ARM64_GCS + +static void flush_gcs(void) +{ + if (!system_supports_gcs()) + return; + + gcs_free(current); + current->thread.gcs_el0_mode = 0; + write_sysreg_s(0, SYS_GCSCRE0_EL1); + write_sysreg_s(0, SYS_GCSPR_EL0); +} + +#else + +static void flush_gcs(void) { } + +#endif + void flush_thread(void) { fpsimd_flush_thread(); tls_thread_flush(); flush_ptrace_hw_breakpoint(current); flush_tagged_addr_state(); + flush_gcs(); } void arch_release_task_struct(struct task_struct *tsk) @@ -474,6 +495,40 @@ static void entry_task_switch(struct task_struct *next) __this_cpu_write(__entry_task, next); } +#ifdef CONFIG_ARM64_GCS + +void gcs_preserve_current_state(void) +{ + if (task_gcs_el0_enabled(current)) + current->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); +} + +static void gcs_thread_switch(struct task_struct *next) +{ + if (!system_supports_gcs()) + return; + + gcs_preserve_current_state(); + + gcs_set_el0_mode(next); + write_sysreg_s(next->thread.gcspr_el0, SYS_GCSPR_EL0); + + /* + * Ensure that GCS changes are observable by/from other PEs in + * case of migration. + */ + if (task_gcs_el0_enabled(current) || task_gcs_el0_enabled(next)) + gcsb_dsync(); +} + +#else + +static void gcs_thread_switch(struct task_struct *next) +{ +} + +#endif + /* * ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. * Ensure access is disabled when switching to a 32bit task, ensure @@ -533,6 +588,7 @@ struct task_struct *__switch_to(struct task_struct *prev, ssbs_thread_switch(next); erratum_1418040_thread_switch(next); ptrauth_thread_switch_user(next); + gcs_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..4e7cb2f02999 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_GCS) += gcs.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c new file mode 100644 index 000000000000..b0a67efc522b --- /dev/null +++ b/arch/arm64/mm/gcs.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +/* + * Apply the GCS mode configured for the specified task to the + * hardware. + */ +void gcs_set_el0_mode(struct task_struct *task) +{ + u64 gcscre0_el1 = GCSCRE0_EL1_nTR; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE) + gcscre0_el1 |= GCSCRE0_EL1_RVCHKEN | GCSCRE0_EL1_PCRSEL; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_WRITE) + gcscre0_el1 |= GCSCRE0_EL1_STREn; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_PUSH) + gcscre0_el1 |= GCSCRE0_EL1_PUSHMEn; + + write_sysreg_s(gcscre0_el1, SYS_GCSCRE0_EL1); +} + +void gcs_free(struct task_struct *task) +{ + if (task->thread.gcs_base) + vm_munmap(task->thread.gcs_base, task->thread.gcs_size); + + task->thread.gcspr_el0 = 0; + task->thread.gcs_base = 0; + task->thread.gcs_size = 0; +}