From patchwork Mon Jul 24 12:46:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 124944 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp1798824vqg; Mon, 24 Jul 2023 06:26:27 -0700 (PDT) X-Google-Smtp-Source: APBJJlFe3kEwpYXBJ8gx4QiauJf0IeGZjv/EWYLDoRV9rQAi7bXFzhEkN5Ihc/j9upqXvLVLwYUL X-Received: by 2002:a50:ec8d:0:b0:521:aeba:c6c0 with SMTP id e13-20020a50ec8d000000b00521aebac6c0mr8559047edr.36.1690205187243; Mon, 24 Jul 2023 06:26:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690205187; cv=none; d=google.com; s=arc-20160816; b=ljSh1dLmhSHKIYTJ79VbnzJjbpRP98rWms5fjWIcg+InqqZAoUfM/g26wZ72ZgHSXc +GWA3uPcxVB1HVh1wNvqrxhCL/tPak+RtUL2R/ve2ZNUe7FJIKUOFFOmrz6ZYr/YzfVz 7iNLljVfBOnKie8Dk6riLr1HnZIIoItXnSfpcc6E/8hV1RFvnwTxeQolSCctC/60UxQj B1W1zZWPK9vQO8mla7evH/zRtsDsicMWRb6nbeWEb1icw0/+gZ6UJVmhtp7jSw9NnXsQ n9YelluwTWU2XP0tpTfCS2rIaV093Wzx1HBP7p6apU4wWuLTsUWiqicd4j6ZNn4vgKlZ J88A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from :dkim-signature; bh=OBYEPFjzw2uNUlP/XNTvaE+SFBx3CcL5jhsQy8sMKWo=; fh=EdAFSmIgUzZTC6WCy4Jg1wBZ0/m6m2d/OjiRUJ3BsTI=; b=NFZWE29gezmTp9JTgiapvv+wkfHtQc//kGD2xLDmMSUWbMnitbXnozTaGLyFZLy/Wt bErRTi9EAmUeienCCoH+GUY3ioyU1f3PRyVN1kaMEpurCpWFuAFXsKX6su7a3maxfAB+ cg/GLx8R21L5Wnw0gey/CHe/bEjFB3TGJmxV7OQ25IJauZS5+nht6AvRYnGxZXaPBp7G IUhcgCkGBzjRa042dQm/9qg84ONIqxtIMqHmMzN6jwhxl2Q/iz1JIK6BQHoVr3rErsjq 6Pw4hNpRtMdnD7c/BCGvxH7dOEtFHyMpCdtMPQGEpXQog6AUexMrUBIGih8yO17PH69B LT6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JDHRUDtY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j16-20020aa7c0d0000000b00522278e9bc4si2206423edp.651.2023.07.24.06.26.03; Mon, 24 Jul 2023 06:26:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JDHRUDtY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231368AbjGXMvN (ORCPT + 99 others); Mon, 24 Jul 2023 08:51:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjGXMu2 (ORCPT ); Mon, 24 Jul 2023 08:50:28 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7BD03C1E; Mon, 24 Jul 2023 05:48:55 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BDACE61141; Mon, 24 Jul 2023 12:48:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5BBF4C43391; Mon, 24 Jul 2023 12:48:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690202935; bh=6QqMyDxHFpRk7piX0sH9x3sCd3niNs7tTxayGqDSjTA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=JDHRUDtY2Z+qUhq2rLqT/f/quFPTRU4lor6uHd/Pw+Zk0RyYmE9f5yvBpEnpfiRoy B32i/N2yv89N6AUcageILDwN+5rSKah0LaP7ASWAKz8bjieeX5aJ3Kz8iWPj5L5Xgu 8p1R9JJK0LvZctwhJU9jD2dbbvGXxwMaQY8R0K9gN90pSqGRb13167/JwqZlYevRN7 Z3gqWN8AazdprIEEiAkoWvDfFcMI3+7Dr7OonwJbRJJ45a4/YIsbPNm/MjkJz5y6de ZGYxQnKHdH/amakOvh2iH+3Ol8MSHv7v+bmIDP17qX7o8/6lPmO/Vjv4rlIBKFHQ8q Fyb/a6ccl7VGg== From: Mark Brown Date: Mon, 24 Jul 2023 13:46:05 +0100 Subject: [PATCH v2 18/35] arm64/gcs: Context switch GCS registers for EL0 MIME-Version: 1.0 Message-Id: <20230724-arm64-gcs-v2-18-dc2c1d44c2eb@kernel.org> References: <20230724-arm64-gcs-v2-0-dc2c1d44c2eb@kernel.org> In-Reply-To: <20230724-arm64-gcs-v2-0-dc2c1d44c2eb@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Kees Cook , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.13-dev-099c9 X-Developer-Signature: v=1; a=openpgp-sha256; l=6312; i=broonie@kernel.org; h=from:subject:message-id; bh=6QqMyDxHFpRk7piX0sH9x3sCd3niNs7tTxayGqDSjTA=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBkvnKiRP2lywkklCSQI8XEIOl7QuJw7+/qdoFcwTSu 313hkEqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZL5yogAKCRAk1otyXVSH0FjfB/ 0ZdBCtYXBNEahRBCXMqHmxDV1UVf9IjQAv0UBG8u7CCb1xKbh1ormrTI/N0tpk+rKgy3v6RV585jLM +kbDu821D9yAjtsGiVXJOpjSeCLpbEoaZp2OFlpZI7JvR2ngl20dCleUntUa4nnNaf/3lCvdKs1GvB BbPINOXnD8SZj31tX6pw0Ozv2PdypA8L52ugtxVUqvQHPpafIAmtO3aIbVtD5Iu6DuCp3AtppaEOWs jgFn0Xu29cMWgIIbU1vxM5wk3isac8BCdW6PsTS6iMQeqTOPeyU+tcPsXVei0J0M5uIqoCgSzC51CU n98I/Ew9JolbIBBrJ9s3iPyuvlz4S1 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772308594479882068 X-GMAIL-MSGID: 1772308594479882068 There are two registers controlling the GCS state of EL0, GCSPR_EL0 which is the current GCS pointer and GCSCRE0_EL1 which has enable bits for the specific GCS functionality enabled for EL0. Manage these on context switch and process lifetime events, GCS is reset on exec(). Since the current GCS configuration of a thread will be visible to userspace we store the configuration in the format used with userspace and provide a helper which configures the system register as needed. On systems that support GCS we always allow access to GCSPR_EL0, this facilitates reporting of GCS faults if userspace implements disabling of GCS on error - the GCS can still be discovered and examined even if GCS has been disabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/gcs.h | 24 +++++++++++++++++++ arch/arm64/include/asm/processor.h | 6 +++++ arch/arm64/kernel/process.c | 48 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/gcs.c | 39 +++++++++++++++++++++++++++++++ 5 files changed, 118 insertions(+) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 7c5e95218db6..04594ef59dad 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -48,4 +48,28 @@ static inline u64 gcsss2(void) return Xt; } +#ifdef CONFIG_ARM64_GCS + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return current->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE; +} + +void gcs_set_el0_mode(struct task_struct *task); +void gcs_free(struct task_struct *task); +void gcs_preserve_current_state(void); + +#else + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return false; +} + +static inline void gcs_set_el0_mode(struct task_struct *task) { } +static inline void gcs_free(struct task_struct *task) { } +static inline void gcs_preserve_current_state(void) { } + +#endif + #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 3918f2a67970..f1551228a143 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -179,6 +179,12 @@ struct thread_struct { u64 sctlr_user; u64 svcr; u64 tpidr2_el0; +#ifdef CONFIG_ARM64_GCS + unsigned int gcs_el0_mode; + u64 gcspr_el0; + u64 gcs_base; + u64 gcs_size; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 0fcc4eb1a7ab..b78f60d4a1e4 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -271,12 +272,31 @@ static void flush_tagged_addr_state(void) clear_thread_flag(TIF_TAGGED_ADDR); } +#ifdef CONFIG_ARM64_GCS + +static void flush_gcs(void) +{ + if (system_supports_gcs()) { + gcs_free(current); + current->thread.gcs_el0_mode = 0; + write_sysreg_s(0, SYS_GCSCRE0_EL1); + write_sysreg_s(0, SYS_GCSPR_EL0); + } +} + +#else + +static void flush_gcs(void) { } + +#endif + void flush_thread(void) { fpsimd_flush_thread(); tls_thread_flush(); flush_ptrace_hw_breakpoint(current); flush_tagged_addr_state(); + flush_gcs(); } void arch_release_task_struct(struct task_struct *tsk) @@ -474,6 +494,33 @@ static void entry_task_switch(struct task_struct *next) __this_cpu_write(__entry_task, next); } +#ifdef CONFIG_ARM64_GCS + +void gcs_preserve_current_state(void) +{ + if (task_gcs_el0_enabled(current)) + current->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); +} + +static void gcs_thread_switch(struct task_struct *next) +{ + if (!system_supports_gcs()) + return; + + gcs_preserve_current_state(); + + gcs_set_el0_mode(next); + write_sysreg_s(next->thread.gcspr_el0, SYS_GCSPR_EL0); +} + +#else + +static void gcs_thread_switch(struct task_struct *next) +{ +} + +#endif + /* * ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. * Ensure access is disabled when switching to a 32bit task, ensure @@ -533,6 +580,7 @@ struct task_struct *__switch_to(struct task_struct *prev, ssbs_thread_switch(next); erratum_1418040_thread_switch(next); ptrauth_thread_switch_user(next); + gcs_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..4e7cb2f02999 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_GCS) += gcs.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c new file mode 100644 index 000000000000..b0a67efc522b --- /dev/null +++ b/arch/arm64/mm/gcs.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +/* + * Apply the GCS mode configured for the specified task to the + * hardware. + */ +void gcs_set_el0_mode(struct task_struct *task) +{ + u64 gcscre0_el1 = GCSCRE0_EL1_nTR; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE) + gcscre0_el1 |= GCSCRE0_EL1_RVCHKEN | GCSCRE0_EL1_PCRSEL; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_WRITE) + gcscre0_el1 |= GCSCRE0_EL1_STREn; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_PUSH) + gcscre0_el1 |= GCSCRE0_EL1_PUSHMEn; + + write_sysreg_s(gcscre0_el1, SYS_GCSCRE0_EL1); +} + +void gcs_free(struct task_struct *task) +{ + if (task->thread.gcs_base) + vm_munmap(task->thread.gcs_base, task->thread.gcs_size); + + task->thread.gcspr_el0 = 0; + task->thread.gcs_base = 0; + task->thread.gcs_size = 0; +}