From patchwork Fri Aug 11 23:35:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134801 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1479639vqi; Fri, 11 Aug 2023 19:35:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFC9cGb82D5E11aVtqBqDN7GW1Pb0l/Yhcue9EcqWVLwu05Bv7u0H0pz2YtrzZJvYWxZ5RC X-Received: by 2002:a17:906:5306:b0:993:dcca:9607 with SMTP id h6-20020a170906530600b00993dcca9607mr2739930ejo.2.1691807758111; Fri, 11 Aug 2023 19:35:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691807758; cv=none; d=google.com; s=arc-20160816; b=Xb4JCMAs6Wx8nIVphPT4rZdCmo1rg/FkmHQQ1bxdQI9sV6PdB9mdWPLgPSf5x9ygcq rRNMNzPcWGSpryILsYmoiSteWQAM6Mkw7HJfmKuWsmpFWOPoUcOG/lj+UVD9fFGeKYo1 /QY5dToZu4jkdGXZH4aZZPH1jlEzwTFZ+qx9T6Y+fZwZcRH9t6TxEL8LbpC97L/pX3FN LrQEL6NEfBfSMAB52AKLUmh3z5S93+hnxsvUQd7EdXhXbZyAQacy94oTLGmgqZFOnvkq Ca+YbLIuCf5ELbUUU1umdsMqOkllZyXKYVRuC+htU4IXE4tdGhstqh2g4IctXcs7FgKu HhnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=sl+Vx+A1GYD8K7fSxD1TZR0BVDZ/BsJ4aeaPLzyfoKI=; fh=GWMiqReRye56Df+Alo9WkyXkbVgKX4T+jyORwmzTO1M=; b=ziJca9qJX3oUHCQ765YwsTu2ef55iwCGz5MDMIDOAIuzwGgPbUfFmnJRKjY65Hhvmu alUY2YnMUMG0PTOREGB4WpTcqHcEiaYcSD167li4HG9Hk6HN7LaPyjMCLdv2s3bRT5un byO5IaoRg2hHTDv8IX8XO2FK2DR1VBx6LVd/Da8oPMYN8srLd2m4Z2z9F8A57zsonH4k Mejy+YDGz61VDLIHMGXduOGlB6+wwxrZRbXV6NVTwuM0I49IfRh9n8ZsLtVpVmrQv7Pd 0jY83xuasX3sJgS2rddMrhfqVzl2pZau9MQ0w1rovricCbgBheDwoRrXploAmFQcRZmy FkuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=rpTXQIJT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id op8-20020a170906bce800b00997bd26a560si4211188ejb.938.2023.08.11.19.35.34; Fri, 11 Aug 2023 19:35:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=rpTXQIJT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237105AbjHKXgH (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235342AbjHKXgD (ORCPT ); Fri, 11 Aug 2023 19:36:03 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A81A10F5 for ; Fri, 11 Aug 2023 16:36:03 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d4db57d2982so2644222276.3 for ; Fri, 11 Aug 2023 16:36:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796962; x=1692401762; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sl+Vx+A1GYD8K7fSxD1TZR0BVDZ/BsJ4aeaPLzyfoKI=; b=rpTXQIJT/8Urt+8bmPNt6747JatP4h6MwdtEFPz7F80VDbouuOlTtSZowuZeUdzNGg /CkBTfFsXAEKRUSc8LmORZEzoIQZ54nFrUjEt5fLPY+OCU6JtDJyZZ2lt8mXw/iLr2Mu VyjGNPPrDBrheV5zhGgK6G4kKifD1vLUKCoDyNpX3V9JKkM4jktDlHy+Nrj4em3f2iPv 2y/hnRksBcNdhmYreZWjYUBEwuzlau7FTcwhy7feJGXusO6zG+67TG6d2P/t3Sj9PrQQ v155YA/xb03kvefeIAkEeJBCGXdU338VLTd61mIKC7N9qER0vU7nVRe8ZZxyznIxnRk/ OAsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796962; x=1692401762; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sl+Vx+A1GYD8K7fSxD1TZR0BVDZ/BsJ4aeaPLzyfoKI=; b=jBkCCgui4by9JIyvrwXO7WbQXNN+KXRHs7sgDQ0O5lMFD6LWGPCkidAdtTdAuKtHxY mlmbPdKuupLpccrm+U9l/tu28o9+vnvINACG7ZTFdtsiclOfuiLa7+eSzJqfQuMX8bVh SMyDClydr2v0cZJp7s14PKUYXvttSBLkRxvKFNepFZqAzXys0KF/1y47X4g7CkNcs76S Xj7hK6HDM/0g28bPxVtItjexRjrMQg4jSGrjFEJY80U5Sm5yaw0Sf9WH1FrQl39XOvvq BYe5p0vNeqXwUUQ/lhxwHcTBLuSTFCcmCUYGtDWvcOiSJ1kW3EHvcA5o1qgxs6+P8niN OXWg== X-Gm-Message-State: AOJu0YyCwrZOg1ICjHOQ2Ut22nZSR4nN247FOo6X86a9OmG18g0IoI+I zsEhNxJw8l4/PgEATELJrynKqCJaDCaTFTHRIAo= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a05:6902:1816:b0:d08:ea77:52d4 with SMTP id cf22-20020a056902181600b00d08ea7752d4mr55647ybb.12.1691796962289; Fri, 11 Aug 2023 16:36:02 -0700 (PDT) Date: Fri, 11 Aug 2023 23:35:58 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10015; i=samitolvanen@google.com; h=from:subject; bh=Y2Q+0SIOBCkSyV4UDfg2ZXoAjZ/iJOwOOLFDLF3IL4w=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXckQ+uqRSpz9pUeeNm+q+S6fTT449pmndoL hnHHYwHAnCJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7k4SC/91twCv+6Nupvh6PMBIDoibMdPLsOvY3OWtnHulGqXWqgzj9IMZ6w6r+0GcA8pKdJL4+6G Fnx4mG3R7hPXiv/3AbHsAhRkmjgMSrvtGfEb0Co+fRNKuiHm6LxJnL75+FFkix1BTUd8t5ssbBn xgJeb7+NuSkQdtZ6ElCNZFyEuNd1r3bmhAs8bgd5+KiumNIbMddBqTGxKCt9MeTT44mc8IypMDI WDaxzjYp6crsXmuuB1k+m2h2qrjJsF9s3JxlTyakdeoDKGzbylCi2mCAKL5kyRoK/to2QhpBt1E I3qnZCyGZZjbEr5PpLraZ+aF82FNvb0SBr+gd1CQZTg8NeUS2ATEisqQbjUmVL3IuuLxcEZT0+i q6avizXLdZ5zfz5oA6jtzz2pQkZQaxtJ0yZv12sKre3K3LV+3bhyBM97chJfTx+JeXXi4xMoqRr 26gyqUDDpixyjGMDDnZtPyTSs1AeCBv00Dt6l47tgeobC2eabQoclJi0KlEZtB/ngCaFI= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-8-samitolvanen@google.com> Subject: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773989011842768299 X-GMAIL-MSGID: 1773989011842768299 From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.com/ Signed-off-by: Deepak Gupta Acked-by: Guo Ren --- arch/riscv/include/asm/asm.h | 16 +++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++------------------------ arch/riscv/kernel/traps.c | 36 +------------- 5 files changed, 28 insertions(+), 98 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..f403e46e04f2 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,22 @@ .endr .endm +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ #ifndef __ASSEMBLY__ -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index f910dfccbf5d..deb2144d9143 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -397,48 +397,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the kernel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is used so - * we can call into C code to get the per-hart overflow stack. Usage of this - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multiple - * harts concurrently. This isn't a real spinlock because the lock side must - * be taken without a valid stack and only a single register, it's only taken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk = (unsigned long)current->stack; unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); pr_emerg("Insufficient stack space to handle exception!\n"); From patchwork Fri Aug 11 23:35:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134792 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1466266vqi; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWLtVSM7ic1z3HatCoSNDQrecFS5JYTKgMbiWNGH2psrIjY4Yjw6d+UhLo5Tsdn1JYcZkD X-Received: by 2002:aa7:ccd7:0:b0:523:2e23:a0bf with SMTP id y23-20020aa7ccd7000000b005232e23a0bfmr4349060edt.11.1691805212105; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691805212; cv=none; d=google.com; s=arc-20160816; b=pryHekehD/9GVgfpabQ0qGb32KHlyRr38OOqbJp+z7r61EA3ePR6TNs5n/xYp0uxGX 50JO9KerSaQQJLzyIhtWljkFL00Ug2q2S+wb36I8nqCREpAigPJiyC8oq9WC9Jquq+3D tOZ9Kd6iFFmIBA/8N+xSsSVni0Ar3s1OwHsr6SzniAWcF1ALm1XILAV6lW1wW4lAIMV3 vr0VsicV4W5gcXnAzXlSCHtngkDr63Y4UgYbm7Lq8cb5zq4QwE0VPV4vpdAoRR9WckqA jc9pjicwGfvZVcKx6My4XkLfK557LqIVURQrXIty7gTG9DxUHPOdMJujWDWg0P6KsRWd hEyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=rqs3gRDgDiyZZoJyqjneRopVqblYdYfI25KzjLQIWhtE1G7XQjQwek6Of6K7j3QCW/ xUukmK9MtNir4ySvldOTWwNt5TQCO4OiqdnHseTtflzkitHfP2Olv3EKmSAvzBkS/yNY PfyIIhv+biLLPsly0sfcDxbzJAMIJrnNykwVQLFJ9Vvgaqafwaw7FoalWi8Hllq4F2r+ k4rA8xGuu2RQeVuDobhtsR/7Y4RCodZ0r2T3bnR6H+4nxp7udS9ddVfpEXY/0N1dtRm8 I+cTt6YSZpCoxuM/5+8MEJdZRgKQGDSTHS/DkO6mr7+gP5a1IjvaOWNzVS6tijta+GUN Zjsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hNk/V5V9"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a50ec87000000b0052340ed2b5asi4287280edr.174.2023.08.11.18.53.08; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hNk/V5V9"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237148AbjHKXgJ (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236991AbjHKXgF (ORCPT ); Fri, 11 Aug 2023 19:36:05 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA29D10E6 for ; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d1d9814b89fso2526364276.0 for ; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796964; x=1692401764; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; b=hNk/V5V9Cl0RBDK/xApOCjB9hBugkkuVSUe0CaeEJvfxUWOY/dHs5BlzQK0Ht8K4GW wz4FNlEdTQbu49jXP5WDdQc5jl3P8kYStgDJp1EcsRF0QdkshXrIub2xylJHtkojk4ne JShRPYJ8B6jOJlbiGmGCSfFSWOsj0WF/PLyolSwVOkDevRJrdhjhTjMhtDy5vO90G201 BIsKD+9tYPjZm8FB3dSatCzIyzW2fFKtOhxoCYTY/u6aLh+wHQ07/9w4+qNAMfEp2iEg XPjLPfnAf3gGHWk9mdXr635RhY2oJpJnb8Al7Qq+/hRH3xOIMkEcrstW5KQCxtb2FE+i EG2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796964; x=1692401764; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; b=Dpj9taDn5aoUZCkGG6O/DShdyvpuhApur94u5x08Q/8u+WIfUZ58GkM+p6IBvggips X7W/GqYc6Jmn41qsr7IbYKyaJsEV0/TNMKceSmPDUWsVfgBp/wmC+Yx8NGJHlNdK/92r hnOe7IDo2XSegndi6HpHpEcyQmZ8A3EcEwgR8VgENu+Kq4NnoSRdmU+sNTBThcl1p5Bz yqavAwBjETjP9LX8jipOc57P5lIzi9K4fiUv82Y9hRBUhlOX0vyFFdubkfL5XaOxgABC aXXS8n/yWBCt4PISkcJMS7JswoXzqaUszaxlWdZ8PavqfrSjRYSWxFK9rWaZp/+OQJBr L+Ww== X-Gm-Message-State: AOJu0Yw+VSQI11+eLf5uAwHLWpVQWhuZIBr3BOsImRstOMEevB5jvBgz DyrzOh4BB9kK7ow7ypOgNShWC+IXO8JlYLIwXCw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:2d0d:0:b0:d4f:638b:d7fd with SMTP id t13-20020a252d0d000000b00d4f638bd7fdmr52997ybt.9.1691796964113; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) Date: Fri, 11 Aug 2023 23:35:59 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=5591; i=samitolvanen@google.com; h=from:subject; bh=iC4KVySOHJXm7MoqeVidxpUrvxa6iA/di8sMVQphPro=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcl+4pMbxsusEsJNlSct/N38MSVxyzWytYj 3gXIn1dRb6JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7u94C/9i14LHcJc+kPonuHpc/Sr6RicO+kf9zaXFzOL2MKaPq9BaDmMLL6nuG2iHksvhI5boWFV +nA3tePX//NgpDNCfDQUBkwGSEiGRRxy58McWFflyS40n4keNX4+P5qJZekqHzDfH69pmaUE0MQ xrxUzYYyWBD7TxyTGewSmEAqP6bRWJhI6FL0zGPkEvDSw+KIQCWYSIcEOMVhgiw8sGyIZRDS2Pd y6Keo6flOy8SX/b6Si5H2Xoczfh4JZCkLMAQCC/LL8zA24db+TJRrNv+lB+1FW6hCXB+4+t3Zr2 DzINx6H3f+B5qU++tKwqqnCG5qXEbyWLKpLIZqyUNTtp2KtfoW1XbXUXE8BIFc8NWNeNUFEo138 LaBbrdBcscNR9th2G2TBWCjHWmXmWXzb2hGZFTSPg0drLr8BP3MHqva3BhkAKQiLpKttkUqExGH uowMH9lPJ+i7mTffOnRwkH9X1FO6d6TkI0etUkvRZhky2/6KfeNUx8nKV3LGflV/YYs64= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-9-samitolvanen@google.com> Subject: [PATCH 2/5] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773986341897771685 X-GMAIL-MSGID: 1773986341897771685 With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/entry.S | 32 ++++++++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 32 ++++++++---------------------- arch/riscv/kernel/traps.c | 29 ++++----------------------- 5 files changed, 52 insertions(+), 49 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f403e46e04f2..13815a7f907a 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -98,6 +98,11 @@ add \dst, \dst, \tmp .endm +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/irq_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..39875f5e08a6 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,38 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -RISCV_SZPTR + REG_S ra, (sp) + addi sp, sp, -RISCV_SZPTR + REG_S s0, (sp) + addi s0, sp, 2*RISCV_SZPTR + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -2*RISCV_SZPTR + REG_L s0, (sp) + addi sp, sp, RISCV_SZPTR + REG_L ra, (sp) + addi sp, sp, RISCV_SZPTR + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index d0577cc6a081..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,32 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index deb2144d9143..83319b6816da 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,31 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); irqentry_exit(regs, state); From patchwork Fri Aug 11 23:36:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1451723vqi; Fri, 11 Aug 2023 18:07:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE5s4HSJWh147tR6TjM4/03qV/+F0qkPn6QCL/uzm8ih/ZngYbhMRPIpGIKteIioNPaxu/y X-Received: by 2002:ac2:597b:0:b0:4fe:cc2:247a with SMTP id h27-20020ac2597b000000b004fe0cc2247amr2393249lfp.49.1691802445001; Fri, 11 Aug 2023 18:07:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691802444; cv=none; d=google.com; s=arc-20160816; b=dUMFZm2EkxOEoEJPTGk0QFrDkN4YXkjepnUr+cI+rMfzScsr46xpROt7Xtwu35n9dr QYfy6G5USfJLvte9x4WcFR06OvlV/eizEJTW6j6zLDe9fSYEkzS2xeikHxsmE9WTiF0J 7jrf0cuiOVIZem7/vmIGgqp1m6mIKmb5dIyNUHxO4aYoqGcAgjgcmCkzETycuo9jRPa2 4g646okNGBp3OVyqzXoWNvvdk8wNB7gHrIPHQnujJeKeTegvfiwadkR5Q2CNQ5++QqQ3 s5mURYIiiUhpBr6FA+qdg5l8sD4nLo/wBoIq8UPYHDVopNA7+hpLy2zKxe6MSW+r6dWt h4tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=uqSttns2QMMHKhEq1iT/6N0civk3/DgixVYj3d7z750=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=HHHP9ySaoCH98OCLekcogj0eKZMl1KlUNd0+TgrBDk0SnwrLgYnTbNaUp4pNBX8Hk7 TPyzfDUFxqxNoN7IQSJYtzokPWvCLrZxIxKhkCbKdFalv6Jq3oJAgmZMT+owuVp7AXcE R7+u3FRWK1YeInf7rdIus/lHWB1vQ4v3pamS+v/pE/n+oeouhQtmCj/G0RwPN49DO/3h eOFpAcdq5Y6YyUep4nU8I4ne8cfBozCxFk2m8GzKzemUAGnlmc6LYiTCE1mo1FrRel6n RuqRAQoSDto/y3Vh/qWxLXUARLrNi25dJxwy8lcCVpPyeN/oYfvU4Ey3hpAWDZXfADi9 5B6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=X34fuYj3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oz2-20020a170906cd0200b0099bd8c1f67csi4181344ejb.499.2023.08.11.18.07.00; Fri, 11 Aug 2023 18:07:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=X34fuYj3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237181AbjHKXgS (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237089AbjHKXgG (ORCPT ); Fri, 11 Aug 2023 19:36:06 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 578D710E6 for ; Fri, 11 Aug 2023 16:36:06 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-564f73e2c59so3068183a12.2 for ; Fri, 11 Aug 2023 16:36:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796966; x=1692401766; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uqSttns2QMMHKhEq1iT/6N0civk3/DgixVYj3d7z750=; b=X34fuYj30ll5ylhB7Aqg1vD8KbeZ19k7cBalDKC4+jHgFwr/RyYNj6Pr6V61x2DfL+ EJCnT8nIK0diol/XAn3zkp0CBQHYK32A+lAxwsdw8V0WmgFSaGLh6MeDrKTy+O1YtGQL lRW4ToSimJ7KM73g7/WgJ/lZD0oHxf4tn3qUdGyg4O4aRd94vZCmXlgckCK9wtoEnTIx By1wmnSVOyT/6qltlLMq/NHoH3JbmEp+o19oXoVJJzJxY75DUC/F8bAwXqxVHXMraBwG CNQJwBuxXfuA8b8ReBpbrRr4G875agXduf79V73IbjdTK0GMrf/tPtRPCyES9mwJzUQL 9upw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796966; x=1692401766; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uqSttns2QMMHKhEq1iT/6N0civk3/DgixVYj3d7z750=; b=GinBnmCNfhfGJsLQMsbdj8C5iUf8Dk47IEteW+tS8ZXg6f/Xk3I8zpPKa5HTxJDi0h Hcvoz4gdM7ZzdGyvoIK7Eo1H0m/fEMWCcLa4U/5EUSYgSQdviPyFD/LaFtyZk6eZn4Tk MIEVkn6uKF3O6Aw6jVWJhOviDgVjWv4nTWxE3tW8HSfH7YOI+B+d98oEBCBidwl3NmB9 nO2nn31Wupv9F9QIPyOrviGcE86f2e+/p0YLGddxqOhMGkrIKJ1TJyBlqNHyoshbcAo0 ewo6FzvoZgmkAo6Ng7SNGoZm5w50df9lPnQOTFuGHFHjoT5nqXepzUXeaoLPPimVBNzr 4PXw== X-Gm-Message-State: AOJu0Yy0KKq/NMbeHDB2LDzyk8GnXFOWI8tOHUC6eFp290rBFWfh3ftm IxV0rOErr46YwIW18H6mToA0c87wNdl6WDtOVQI= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:902:da90:b0:1bc:8249:2547 with SMTP id j16-20020a170902da9000b001bc82492547mr1100101plx.12.1691796965861; Fri, 11 Aug 2023 16:36:05 -0700 (PDT) Date: Fri, 11 Aug 2023 23:36:00 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3217; i=samitolvanen@google.com; h=from:subject; bh=tt28FmYKneUlEuS934Ul5UfwYg3GE/8Wy9t+zpPCYws=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcZHCAlTnd4EYJhbusgalCGC+w098BuJsGg pjfWx3ysWCJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7vtjC/9aDXQZ1XeqW1KEFEIdmhB56J7RualjvUaGwqw1lObaqRwOTOHqVWWF23giW/63wj+kC2d uHjLduotttcVZkHZsC6bSVritbMUdqcBJPktHpbake9N1DDHjOKuNQV7Nb8/N4fdKLoTbsIhX8J c9wUZRHH+uECCk0s/Vthl8g0IjkG8bxxiTWZPmj9U6cBnZreRQNTMLyS0VAeDbMe1fDLYwCxUOD m7jKpu3cmKHJ0+q07Tw7ImOGqjT+v/47NwVRAfE3wnQ2qG23TXar0JY6Ve/GXc8Cl5iQqIkW8I6 vqNSlOuam4C9fCr1mNVlroRnndKOjNwoqNMGFOpCRRtZboI2i9jEI4sy77bqFyu9Z2AaZODMEky Z8lDTU3yZB+Dfy0y9jrRLlFmdooFQZZqg92id+407SgbATsSpewepWk6+xfbx/2ctL/zCdrzrKE Wqbc1Mw5wnbcu9O9FXMhy5KyHuHKxN0/EpYpF8+0511pfnh6hGOTgck63WxCg0nkRDL/o= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-10-samitolvanen@google.com> Subject: [PATCH 3/5] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773983440809520856 X-GMAIL-MSGID: 1773983440809520856 In Clang 17, -fsanitize=shadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 13815a7f907a..c53388f784a2 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -103,6 +103,14 @@ REG_L \dst, 0(\dst) .endm +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 39875f5e08a6..2b4248c6b0a9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 11c3b94c4534..79b5a863c782 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_entry.S index 12b52afe09a4..556a4b166d8c 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -60,10 +60,7 @@ END(__cpu_suspend_enter) ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer #ifdef CONFIG_MMU /* Save A0 and A1 */ From patchwork Fri Aug 11 23:36:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134781 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1443234vqi; Fri, 11 Aug 2023 17:43:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDu0HFF/sE5ceawsJbJEXRmBjZYHesfdaNwo9AKUqfYSo/AV5FOCyaZvBv5z5L7zMorBxP X-Received: by 2002:a17:906:2ce:b0:99c:bb4d:f5a1 with SMTP id 14-20020a17090602ce00b0099cbb4df5a1mr3165067ejk.4.1691800989732; Fri, 11 Aug 2023 17:43:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691800989; cv=none; d=google.com; s=arc-20160816; b=gfBPudrcF9FiJ8eYqmwvgfJ7Hv0Y1e7/hSD693Qt5rt7K/AixQ4gI07DnKA+i8JzWC zmEyKtsNO92pIa7KBztP4JVwH/3S6EI67xUK/oWJ83ocndWTFV/K9hJ7Pudwpazj1PQ5 p7aLtOnjmd4KQONiN9pm0Yr01MLNnZ62BYq3By1lr2fwC5evH7z0uY7Hijh/xBlG2oZs 4sosVPsKv/gcmzd5UwtlHS/86NU0IEWwQJGfWOBZJfSv81yJOuHPLuhmlWD+3tmw+o/P SQqVxMwlxHOexcYDV5vNLP2kMZPI2hP0MSaiI6P5KCXyp0+YqgJ6rRZYJt4zf+jE8EWL 9Ezg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Bh2SMH2bG9I0xKALEIFQB5QNQoDjerh8Q1sJkce2DIk=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=yYwG/KDkqNMMmqWoYstkS8MVTMqu/3E3oJ8w78SxSuGEwpJDkpqyFCpZvtlCt/22DZ L0vCXI90GHXV4yvHphMQS0aO26SyIfXkGzSpb7AcIc1VJlVYzBcdgfce3z3DiMNGPhA0 +N51HFzFTdGNzwjBDQEW9TC6ox8XSka2QcuXOpHlxK7rVzbKVb6H6mtC63Tr/PFCH8Tl j8nd+8GXbZ0BrIXdhgUDSDdlRyt1KGwBwMu0xoqLD0uczDLI/KfWUuEWKOXWeZ4+wX7q aWSTssd1VLTHS9R3BP6vnlE39ot0cY8XdiHu5h4hqCZYYzW+MeR60NUhV210tJ/abPza Sclg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=gvS+jS69; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m3-20020a1709060d8300b009930294ae72si4227085eji.293.2023.08.11.17.42.44; Fri, 11 Aug 2023 17:43:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=gvS+jS69; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236991AbjHKXgU (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235342AbjHKXgI (ORCPT ); Fri, 11 Aug 2023 19:36:08 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28D1810F5 for ; Fri, 11 Aug 2023 16:36:08 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d62a7776f69so2431465276.3 for ; Fri, 11 Aug 2023 16:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796967; x=1692401767; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bh2SMH2bG9I0xKALEIFQB5QNQoDjerh8Q1sJkce2DIk=; b=gvS+jS69ksGoDJUMrUJII+1hQrHq/NhM3acNv4ScOaKqy2V1pi4I1F4GIydEELmeEl ex0ydE9XPcSDIiWRl8uMIp7KntwyI0W7Fucu+h5hdGVqWMsTQhtrtBJGs43R5K9peeur bwAsxUWoFa5LbStrHSGvJMV434Ny8DCmwwljAUp5yToUppch68nLSkRQIcrsWB1QwAPE 2XokZ2TSSe5p2y0rpj0UCxVZJD164Mq52XK2KIIFzY4VaX1STqLk12sBhGx/k6NlS/mO rbT3HG8YDNrZPkNOda3ITjvz5ofhN0/TfYMwxvI523ZIJSp0bFqCif3S2e8j4DOSXftf sH+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796967; x=1692401767; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bh2SMH2bG9I0xKALEIFQB5QNQoDjerh8Q1sJkce2DIk=; b=a1KNBqDxsNYbJyONDHOwyLcX+xg81JkoVPFauvXZckCHA3/SA4xLbTSS5Y2qm/jxmj adEzEPV5K7qRCB0WjfCi9kfUwLF5OEbqm65gr5BXmb6LSElR9du+3w0c+pjjNpAYDo9n 2ZHbnXbmex0jY1X8rOB8oC+2PNRqtUvJr1hpLwKM/zO4Q/1YH1inifN8OC9AGimne9N3 zWhsmsAdpReBrmhVNfwfE6n6PeFBeZ2Gm5fOnqCrvuQ5kYPyUeDKo+dA5wnAKQaTcl3m LMey96sjDsXGbkdPDknHpXiycoEw8cgzbW2H1CqdmhxPZREGsrRZcGPPUEy26JuAhfGH uQ1w== X-Gm-Message-State: AOJu0Yw5a6NN5ElbDxy8RQSF+jsz1/OhEij/hMzk6dEsdpUg8jCQiTn6 ENhK/sBlwUvc8BOlglAS79qwcaIiq3OKcQ8hXOM= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:dbcc:0:b0:d08:8cf5:c5d8 with SMTP id g195-20020a25dbcc000000b00d088cf5c5d8mr49616ybf.5.1691796967456; Fri, 11 Aug 2023 16:36:07 -0700 (PDT) Date: Fri, 11 Aug 2023 23:36:01 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9645; i=samitolvanen@google.com; h=from:subject; bh=ueeQJcvzAnbsh8FPJDYWxlDi+/d25912Nt0U7lLHt5U=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcDaoAzJtGtIOuaRCf2i9GhcmPPBJhBrA33 Ww51acNiOKJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7krpDACfvorMMZXm+b/+jamk9TK5vm4BfjQxEIFqMXVRqmhV4RcdDgLSgrPDJjn0yruhkgAyD9+ cBzu3xS3s01CSBII8JbF6u6wpu5vWr+GCkj6oAnqZ/zmB5Ak59Q6NxQQ7U8yvb52s6jZJRydN4+ H0fjGV7B2S6vFcj18ECNa4t5V+N2BYSAWetwjMpHTpsQzlw0iAwkxzCUAtwWN1TXsVmRmry8mFM dbcEmR+kwnHad+TTIyzHdENKA1n9+h2qnompvDdb0Rf6yjQIjdKBQ/VcwJNP3R1LVXg++XqAYlT rTSvmvQPoWNEF+Ugv5nnDyr0odobsmhcg/0vZWkiMoadklSqTHDbWWVIYMXjSkGgVRlAmICsLcE E1A7MOQ0YmHJmmSAS4dw5jlIg0hLPx8dkzEMJ0WFCzvjaIrVwaaIxgsbsGS+FqewGurBzUu2mjb ulBb9oWP+4ITnttsy7Tcnc/lLVZzCUM1N7rwWGdx4cx1ZRDXyRO+08zEZOeVVqll+8EHc= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-11-samitolvanen@google.com> Subject: [PATCH 4/5] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773981914947757391 X-GMAIL-MSGID: 1773981914947757391 Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..8fe31ec59da4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU @@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=shadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6ec6d52a4180..e518a74640fb 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS += --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index c53388f784a2..ac77738835ae 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -103,6 +103,11 @@ REG_L \dst, 0(\dst) .endm +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -110,6 +115,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_sp = init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ + INIT_SCS \ } void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 9f535d5de33f..177cef43a2ee 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 2b4248c6b0a9..ad34507d3c96 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -277,6 +284,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -294,6 +303,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 79b5a863c782..c3d0ee77483b 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current .Lsecondary_start_common: @@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makefile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY endif # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) # Disable profiling and instrumentation for VDSO code GCOV_PROFILE := n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index dc20e166983e..d5d60c040560 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE += -fstack-protector-strong endif +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o += $(PURGATORY_CFLAGS) From patchwork Fri Aug 11 23:36:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1447638vqi; Fri, 11 Aug 2023 17:56:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGV6zkDkavnTss4kdgZ0Wo5LiVrOJHCAITbEf3LgQwPadjYsEm5UTuYSClHfNyAQAvZt2TZ X-Received: by 2002:a05:6a20:8f21:b0:133:71e4:c172 with SMTP id b33-20020a056a208f2100b0013371e4c172mr5058474pzk.15.1691801801648; Fri, 11 Aug 2023 17:56:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691801801; cv=none; d=google.com; s=arc-20160816; b=FWQp5Pfe5eawvNhW3vU0/SwD6Ukj0bDG0ape7j83EFMNmgKddQ0JnW9pGcBbkyQQvD 0A9KodXwrWtXzu9cs+XHu1SNfK2+iMIsfb6lZyXQQ62YCyvj5dS6H0GYNRYQ6xLr2uNv vbaQvD3ISxK+H2jju+Ne7at4NJmJo5gQ9sfs8EagCs0btiEw6wpheWSNZjg/HQBQkt19 Xlx7e5i0qh0D0e/XhYz3j6vd5AMI6mXIVLb/kBRtCqrly7xTzXNt4S1RNz75L6IKgSgE J8yd2a/d2zWEgywccpvnXksghA6PbP9Uq/8kpqkJv9s/1+hQbsN4hetrGdUJHdKbAi3X luEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=kBj5RzCnkDfdO/ghtaV4N3R+P7gjt0yxECGcEY+HMK8GYBckXzTJ21mf2r+EZ+bb6G uXvm6XZhi8Jgg6up53N0ryj/koK3b/V0iw7iPpYdwRbJ+EmC7x298jgzRk+4J33NOd53 KBGdtNETviRNXyM9F3qV//lGnUm4QVPmwuGpwdcPsZhmNqC4BqCsR1Tt7ps0oTZ391Cp SnNZjQs9MzfteBNXcR4UkDICvjsludSVrCLZNBxyORNHB+qTiqa2EXAqp8zYBRs6di78 a2zrz3IkEdcfAk9QoAtQsiTsCORvo2udcC1s5PPdaqSIm3DZ+1qwrwQ5FVWGiHlP+qmL P1UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="mtVd1W3/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k190-20020a6384c7000000b00564b8ae6081si4280743pgd.15.2023.08.11.17.56.27; Fri, 11 Aug 2023 17:56:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="mtVd1W3/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237211AbjHKXgX (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237144AbjHKXgJ (ORCPT ); Fri, 11 Aug 2023 19:36:09 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A97910FB for ; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-563ab574cb5so3015763a12.1 for ; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796969; x=1692401769; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; b=mtVd1W3/3q051Qstg/rkwh22KJtXJgwRmJjPFHPrSUwrwcnjBaqya9IaLakBib/tGp MZTlDO3i3B4tO+B0FXZi24NgY0QmiN3TLRaIN/r9BVzhVd6lXdijvcEwht3QqyGFD25d iAg948H3L3HcDIX/PDfVlKzDxQL0FtAOPKhRa0fRjcPK/dRobJigKyFzAfEEvzpV65fs kGDPQqp+LC2W1h1PDbovFc73zR/ks5uXhEXNhFQtNnxEmYALG9EGpnOyEhYzt92Xo9Yq 5nvT9GMKmOWUbwmKtjMTTTSyuaOqrL5J7LhfgrsQFnUpeCuAUc08CGdSsrD74A12oTl/ 4BBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796969; x=1692401769; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; b=kXLKz/Yp8f9mr+bM8F1cTr8ti6PeK7y3tSejHads9DCxNdSk2YbGlKiMy1MWq18Mh0 K7NKOZ5oBdVqkVRQRR0uJuUpkuJpJL40i+aDZF/xyj1PZY/TNaVizAsZVBRDLlTsYZ7W zKYD5G1ECNhSun0TIrHLsaSB4BvELBpcZTDh21ll9b97G31UWGNJSs9c5DUavCneFlA5 E0B5/T1rYvcTZ9BV0Sj5hV4U55ZfdwFYIX0CmH/Ey4jh+h9tJu+vLWwWzGHSB85Ru7sK Pk2m7L5g8Lx7ivfnhLL6f0dwg1kWEHqkWp8L9oWUdFVs8QO7O9v1uTN6sAisNORp7KRq C6QQ== X-Gm-Message-State: AOJu0Ywvh1+HogDXv9Eq2PPAeaWfGy40gm6retrg3kix1iSWtXHIScS4 PmU18QUPP3qdsC5zMAig+VrnKWAsLK0+hoW91a0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:8c57:0:b0:564:aeb6:c383 with SMTP id q23-20020a638c57000000b00564aeb6c383mr638656pgn.1.1691796969048; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) Date: Fri, 11 Aug 2023 23:36:02 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3124; i=samitolvanen@google.com; h=from:subject; bh=wBRXjmNHfhdcdsjrWxKjQVM4d1RVOAig5wuWuTXp+sw=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcweRCoqkszjtNMVnBaMVhIhZi/o4cME2C4 t5Qc51TbeWJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7oToC/0bVrByxxtfXjE7idOKMhQ47Ftkbwzrwd/qVpuGk5sKwMO1ZCCpNUD1WjHmpT9DM/jRjii ID7y7zSN8jMwvkBKVqWFcrA/Eaa0m5fWafqe4cjjWupJSB/ep6zYn/Rg6HLdIw7XP4JK/Vc4y4r TzWyrycXtowJuCYuF+VVNuP27Hv37AH2QTpPxcpmbb7m3TwnCZ3WQKLI40KhXg8gonXaggVqg3q X5caBqH0OE2M8dRrYxeDPBlFrqtmfu0hbgTF/flJQdHf2d68ZJHYv8m5rdu94Bnwwf6XZFg+Q0R ZQlYPtqVBdg6YwftizeZzNr6QjUFpqQ7A4buby+3pAI85S6NE4IF5Nces27nX/oKcSXM9+L4gog 10WOtL3ybC+F9bnk4ud+g4NkqrfaEszHa/UYGP2doaSEY7wnGja38VeGjqq9XMANxgYlDyRpWrs vGsTNxyJYa2V8imEkZFgy9kgnfoyRyITlo1VmA8GQU3rxJACgMLpVtK/P/BITWXbgVKXU= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-12-samitolvanen@google.com> Subject: [PATCH 5/5] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773982765587192623 X-GMAIL-MSGID: 1773982765587192623 When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index ad34507d3c96..c86b76584d2d 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -238,12 +238,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, (sp) addi s0, sp, 2*RISCV_SZPTR + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -2*RISCV_SZPTR REG_L s0, (sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) = + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq)