From patchwork Wed Sep 27 22:47:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 145623 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2946624vqu; Wed, 27 Sep 2023 15:53:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFaeFhlD5w0rprSzmraQYB9G4x9uQx1HLVF1RS3IEXavKG5dV9mTop/ZtqgJnHUYOOI5HE5 X-Received: by 2002:a5d:4587:0:b0:313:f463:9d40 with SMTP id p7-20020a5d4587000000b00313f4639d40mr2644234wrq.65.1695855219520; Wed, 27 Sep 2023 15:53:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695855219; cv=none; d=google.com; s=arc-20160816; b=ej6/FXiT6e0rjU7gatzQxMdNBM5amcT1Veq8AoqeN5mIPq3+DkwLeqc5H0BXLBu3rI qicFkUU5e8BPyvNWRy3CFMFECFC1dEWvPLurgrGbTyvAt2trPNZYPOdO4JFBYHx+jHgW ZCPLK+1qp/s1av+fdj99YMUHQn+BFomF9JA8g2Nq+V+tOAPayJE9KHvCzOQkC4PCNKUA +D4+yEmZYGQ1s+6mi0hYrLdUMPbp7NIqWZzdVjLKghoG3M7ZJvNpkB9TwOM/M/UPXOMM ImptcmRLh/XuMe4PGf59SPbRoeP8rQFhoDM5+ACGXjyBs6SYttV8WQsOdVp8S7We17k8 2Yyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=mNJN6BYrhbygu/rBGeXZ0fXLwVgosVxKqBS6YQtnbPk=; fh=sH4eJD3i5ssW0rM0Ce917c0Vx7BD7zqkNukvJkVUQVA=; b=jGjQrSy7cDE6R8hVcxtpGeB01vnJaiI62n47gTDFpe94qrhEjNS+OEILxtNI6HPMys 2rklIVsTT8S+Kl0Cg3nR2b9ZWp3T0OwhfpFumBzQ3UypW5Y96XDbpCtTjz9H9Cy9n2l3 +/pCq4MS/hOBs5t6EuPIlIW5EHOURoP3n8e12vaJPIenFZATIjTIcb81LitiBXXBlA1G jpoYvzdBY3/MS6LjGZWzs5XZCzc6cc0BtmiJEVpUnaO56x38epcKJAIxSIPl8/AW+ziI rFEiJXV/3AsU1+C3p3KXr1BLTWUca3HD+kqsS5dKCfr6yDIfyzLg5IYWPqGi5UPutRMT uXJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=zOLGv7x4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id p16-20020a1709066a9000b0099d6d4c11d4si13643517ejr.782.2023.09.27.15.53.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 15:53:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=zOLGv7x4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 3C2A8826FA91; Wed, 27 Sep 2023 15:48:44 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230013AbjI0WsM (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229862AbjI0WsG (ORCPT ); Wed, 27 Sep 2023 18:48:06 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 688E9191 for ; Wed, 27 Sep 2023 15:48:04 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5a1d352a86dso125232017b3.0 for ; Wed, 27 Sep 2023 15:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854883; x=1696459683; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mNJN6BYrhbygu/rBGeXZ0fXLwVgosVxKqBS6YQtnbPk=; b=zOLGv7x47hPkF/gRoDwv2yOLXuLqXyKd1jWk2zcZWs+f9LdpJyE8dNWlR1wNn0nnax CfDG7mHiuHGA6+bZZF5G7H2oXcY4mUEy4HqPjOakHjKTMnWsm1gvB20Qpe3VcZmqHZQ6 UyF/1kBP3+zT6D8VhCo88pLnH027iILWuF6MSAG4WlHurWnsLg/mkyCUnnWVQEZRCujE SXMXdQ1gHF4WqjUwWvpiQIwUtaVaD8+FOCdmZdZEz9w31sDLffJPSQtbc+3SnapAy5BC JgWG558fODYa0TXm2iLpbfLpYLpl1Tvo/286aO73z78byGcQsWLX1ea2YBKfvWHOMyZv 6I0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854883; x=1696459683; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mNJN6BYrhbygu/rBGeXZ0fXLwVgosVxKqBS6YQtnbPk=; b=wHEI8kCYshMy46sKqdqcoMU5XPu5fzdVXijgOyMbh1UkBAQkkr2rbj4QZ6himSChUn fHcENoxcEWw48Pg0szs0RWuMM9FwCmdYoSo+ij+cd/mQ8xw7T74cQ3mHpbtGvItKAd6+ +mYcMB6GuTyzbzgknfM0ykQjwLW+Q7E1SXEv6duFab+rlFAciTl1I6s/t2x6SYH2CAEX JvQfrTo1LJbbhdlTU8NFAc5+j75LEVNL+blo2zwOCPZ3290wYFjXUKrg6w+X2iWFtq31 opwXCb/RIwJs+4OXc33a+BtfN/NzaTAwU1k5xqZbL+Yp+UR3gvjfxC4+jtOXbTQeYwI5 r3kg== X-Gm-Message-State: AOJu0YxL84n1Dyw/JhxaBY6KJGhplIPvJ/6VjZx08bgHoXNrAwvrV/Ax vaG0bVU0dEeuQgN8vrUnmKkzRq/osKd+AGOkcdA= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:83d0:0:b0:c78:c530:6345 with SMTP id v16-20020a2583d0000000b00c78c5306345mr48094ybm.7.1695854883653; Wed, 27 Sep 2023 15:48:03 -0700 (PDT) Date: Wed, 27 Sep 2023 22:47:59 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10961; i=samitolvanen@google.com; h=from:subject; bh=GJOR8OCraLgZs/lo8GjwnqPkf/sLrnWYlpF+fdB5yyU=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEdzk65AOG19tvr5oZD3GinKkgZy9vwq2kkc hrVm1bIiYWJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7pQFC/4z1E6e3V9tysijinGpzs4dGPpQGBihGVx5WL+LfHDExfnBU/nAqM2LSWiyjmr/JS0wS6G gEg0ePcwf4I+awbLoT10E/Q0yxgmspPnYVRSoiT+6JrnheoQ1II/OQNk/k3+l1tOk1wTIhKwm4N 9h6FTzLFLSuMQ9drkPQGGcTuBAZkrG6CtEMOWB1W0avFdTW/ulApG51uRktGAHm4V2MqVWgA64U X2vHCCDvtANeiyxUiEyzQM6XXJgVGVh2Di6JyGGm2qao9V+s9ih2CyGK/063dJOrtd/5Th+wR/m N5lcOJv7kYji+8pWte3GlSXQhqH69J07qKCPl3Lru4EjWsi9qVUROJFjL4oTHnlZy3FD11JJWsT 8RRLNTpwdWir2SsnjGG5T+1cN/ai45tRRGpDSU5J+qYGHUmtHcKskXIXZopv/c73ad1EtcaZYSk D1mDFA4bwrb220+CwkYXfqWlMyQxGRObkQVgjLVRG/e3QeymDd+cMAqbp2cEXFpGNEIes= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-9-samitolvanen@google.com> Subject: [PATCH v4 1/6] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang , Sami Tolvanen X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:44 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778233082470436548 X-GMAIL-MSGID: 1778233082470436548 From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.com/ Signed-off-by: Deepak Gupta Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Acked-by: Guo Ren Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm-prototypes.h | 1 - arch/riscv/include/asm/asm.h | 22 ++++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++--------------------- arch/riscv/kernel/traps.c | 36 +------------ 6 files changed, 34 insertions(+), 99 deletions(-) diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 61ba8ed43d8f..36b955c762ba 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -25,7 +25,6 @@ DECLARE_DO_ERROR_INFO(do_trap_ecall_s); DECLARE_DO_ERROR_INFO(do_trap_ecall_m); DECLARE_DO_ERROR_INFO(do_trap_break); -asmlinkage unsigned long get_overflow_stack(void); asmlinkage void handle_bad_stack(struct pt_regs *regs); asmlinkage void do_page_fault(struct pt_regs *regs); asmlinkage void do_irq(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..bfb4c26f113c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,28 @@ .endr .endm +#ifdef CONFIG_SMP +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm +#else /* CONFIG_SMP */ +.macro asm_per_cpu dst sym tmp + la \dst, \sym +.endm +#endif /* CONFIG_SMP */ + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ #ifndef __ASSEMBLY__ -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index 19807c4d3805..0063a195deca 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -402,48 +402,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the kernel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is used so - * we can call into C code to get the per-hart overflow stack. Usage of this - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multiple - * harts concurrently. This isn't a real spinlock because the lock side must - * be taken without a valid stack and only a single register, it's only taken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk = (unsigned long)current->stack; unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); pr_emerg("Insufficient stack space to handle exception!\n"); From patchwork Wed Sep 27 22:48:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 146100 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3290767vqu; Thu, 28 Sep 2023 05:59:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGPKyrjzRvyvV0bevtl4XPQ76QrRmUmADQlnv/ZIdrYtupzl0sHAqrmf70vJW6ahZUDvSwT X-Received: by 2002:a05:6808:1783:b0:3a8:4d1f:9dd0 with SMTP id bg3-20020a056808178300b003a84d1f9dd0mr1255056oib.30.1695905977831; Thu, 28 Sep 2023 05:59:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695905977; cv=none; d=google.com; s=arc-20160816; b=Zm+qzBuSSazjo3M1dicIL+jcbKLwylzbHcgAienymVRH7VsNqC2JlHtjcIyFhmmmwg YZrBAZTqSjPCkfkqgdBRqnjHYj/hmF79Dov0zlyS6wrWQ594biGgqy2YLPMBFgBy2+oA Lsa6fZjayT3BFvilAN+fkQAeiVFtRGpREwRXYvjMcWCVjT8dY+g0tqwCVkp6PsHMvr5o hOnDuCua2W5zHWxpjzl3XZRKlIFxnHthpK8Lbw3qFSFrRDFgD2mVJtYILxEjdyxeHGDU RFbxBr0KlNI3kbhAxRW5MRezgEvJEokRy6X+1bR9TmBH8iK3hO3Yr3kG1psJeBmblyXC 270A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=CSH8j6rxC8KfbRVvtZuqLTVmkQvpSCOmXR9/i25FaVY=; fh=yvN63QVq0nFna1WLidysZ4ad5g6OhQJzYTOcPhqxM3o=; b=Lnv/Ow5XpDqXWK9jvE4tb9zRC0POYjKlxnBxhvOxjzBqDWEKxLU1zHogndSfVOTS6y adl2nFuHtsJ16iueZGudAfjFIbQmFMFHQgy1EChm7UCM4QoC5xnaF8ATK7lkfjg6oo0n Uxk4v4KcM3/IgBEkOYmQoxvg0idJ41wI4vI0+2TClAGl6UJK2k1jgfqprRKtXmAhITC5 d81IGvn+BkD8M9umZ9Ify3LRSfaKmSMYOaYoinkuCyckMV6HJWI+8f2XBfvkWSHY5naI 60f3xwD73QjxAljHdEirku5WZuTkTAlgGTjInoMVYe/qM3eqiwZ5XRSDTCQtqnYeFm6L ce+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=kGQHEavv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id r27-20020aa7845b000000b0068fcb779a3esi4393073pfn.278.2023.09.28.05.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Sep 2023 05:59:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=kGQHEavv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 371D582277AF; Wed, 27 Sep 2023 15:48:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230038AbjI0WsQ (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229987AbjI0WsH (ORCPT ); Wed, 27 Sep 2023 18:48:07 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E70B5126 for ; Wed, 27 Sep 2023 15:48:05 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1c725aa19eeso27301675ad.1 for ; Wed, 27 Sep 2023 15:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854885; x=1696459685; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CSH8j6rxC8KfbRVvtZuqLTVmkQvpSCOmXR9/i25FaVY=; b=kGQHEavv4aCtzuiFmPY/87hdCCTxaURY59k+Gaw6uArkcYBQpEbQjiujnleGbkfhNK g1thMzNmhJ8fpiPReH8/g3XgNRJL8vbJS6nuTs+VoXMc4m6Ly9jFf99NOQH6QLK1b0r5 DHqPeHrvnEqL4nNDI0KVJyAUL06yAvKd95ntUEqKj0tLRJiK0iq0lFvR2hfKeThiGBKu 17rtSCTzk3Ods5QfvSJt0vT091UQPUzWefPp8v7fKH+JI+ADhcjeTvVWPiMXvV86m8nf 24oC+2lhshb2CWb++MMMYkAg/WDnI8zKyERSHqGUo6MRLv8KMC9tEWBGdaMOIGEY65eU l8Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854885; x=1696459685; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CSH8j6rxC8KfbRVvtZuqLTVmkQvpSCOmXR9/i25FaVY=; b=TbefT93s9AvQJ9uSegTS8+9L4xWUKWdruSJQaad+LtQejumu4pdB+h3q8Rg45k4E3e s6KSB8B5i/lX9S66zS8JVEqq8H4iyegAtdgnYOonGtWm4hSE4ZHLuEVErMoNjXPu+yoi ZMK0/SM44uDdf/s9dQ8fcHZNrgwDvH8hMQG254VQiVAMe/lb5YOpb1HSMMOvv4AXOKah b6+WIMUSjdVErLcFi22KrvufmMwoVAdXWrXZdtSGrAX8GgfAcXTRZZTTydbU03G/KOQ6 bMadaQEQJtEepQ8j1h0Hsxo0+gRSYFzT5KC2DY4loeTt2pu3sMEf2lZf8OD6j40JRYlV xn2g== X-Gm-Message-State: AOJu0Yw8uRheMiwETZUnqrJCcHYwCfFHRAh2N6ki+DQxEeTrpDTCuMZ6 uKITlJduSQxSdVcJrVMX5J9G+zYbC/Li8lwbcVo= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:902:e812:b0:1c5:5ca3:a5ff with SMTP id u18-20020a170902e81200b001c55ca3a5ffmr40178plg.7.1695854885335; Wed, 27 Sep 2023 15:48:05 -0700 (PDT) Date: Wed, 27 Sep 2023 22:48:00 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=6693; i=samitolvanen@google.com; h=from:subject; bh=S1G6SXUek1nxAO4+E1Y770ytvcISKUGnRC45RExsZQA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEdLlQn+/m+f+Rv1Ps+71ZLlxyOG0Umtpt0F TAPtQdnM/eJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7g35DACh9uowckTh2GQEknUhXHLoRfrqjfx/Tg5AwtpmVbYHOqemhl12rqv9/7QxBySlJPeVDuz eoIHca7ZU2ZbvolXE9Hsd/hxeC1gykiiTt8nv8DXoBviiysUZnjshjIcRx1aQBWyWmD4rRnVFK0 gI8aJdvizXSFgwTXd2q0DO93bxGfm3qOYjttbcr+33zUD7J/No7kmqjdovARcvPGILmGohBGMac agcXDsK7z7fIhQR45QNGDFctpC967S5kcuomQRELrUXKYqSt5AtMeZpzkz4vlZFwcdJubVg7HTv TNBjBNW6yWDxZ14qSMhdOs3UqXB29Sx7HgsYFd1j+C96Ixq9Ott1cu3M4W2/z/QWPLsvwa0g2DV DA1PumQjHIC5M9Z4wKVUHxnxGVcYcDvZRgD4V2jnk1Zn94kDan3tK4bzd4w3E6qFOSEj/N2qXXX vWXz9AJMC6z9Tsn/BD3mSfBfc63N917W3vV1y+j52PRWhM7gbD3krWtu8IID5CgnoPyow= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-10-samitolvanen@google.com> Subject: [PATCH v4 2/6] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778286306211499983 X-GMAIL-MSGID: 1778286306211499983 With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redundant inline assembly. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor Reviewed-by: Guo Ren --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 30 +++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 35 +++++++----------------------- arch/riscv/kernel/traps.c | 32 ++++----------------------- 6 files changed, 55 insertions(+), 55 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index bfb4c26f113c..8e446be2d57c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -104,6 +104,11 @@ .endm #endif /* CONFIG_SMP */ +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/irq_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 9f535d5de33f..0af8860f9d68 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -14,6 +14,7 @@ #include #include #include +#include #include void asm_offsets(void); @@ -480,4 +481,8 @@ void asm_offsets(void) OFFSET(KERNEL_MAP_VIRT_ADDR, kernel_mapping, virt_addr); OFFSET(SBI_HART_BOOT_TASK_PTR_OFFSET, sbi_hart_boot_data, task_ptr); OFFSET(SBI_HART_BOOT_STACK_PTR_OFFSET, sbi_hart_boot_data, stack_ptr); + + DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_ALIGN)); + OFFSET(STACKFRAME_FP, stackframe, fp); + OFFSET(STACKFRAME_RA, stackframe, ra); } diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..a306562636e4 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,36 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -STACKFRAME_SIZE_ON_STACK + REG_S ra, STACKFRAME_RA(sp) + REG_S s0, STACKFRAME_FP(sp) + addi s0, sp, STACKFRAME_SIZE_ON_STACK + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -STACKFRAME_SIZE_ON_STACK + REG_L ra, STACKFRAME_RA(sp) + REG_L s0, STACKFRAME_FP(sp) + addi sp, sp, STACKFRAME_SIZE_ON_STACK + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index a8efa053c4a5..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,35 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index 0063a195deca..cda6dcdb0376 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -352,34 +352,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); irqentry_exit(regs, state); From patchwork Wed Sep 27 22:48:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 145688 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3021657vqu; Wed, 27 Sep 2023 19:10:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFXFQtAPgvjPOxRd9umBleP6MD9tqCs2kiJR9dIFtfcGyuIlUwEgCDIgmzr3YQExwWS11AJ X-Received: by 2002:a05:6a00:3908:b0:68e:36bc:194a with SMTP id fh8-20020a056a00390800b0068e36bc194amr3930597pfb.15.1695867033388; Wed, 27 Sep 2023 19:10:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695867033; cv=none; d=google.com; s=arc-20160816; b=qfFw8fztBjdp/yQy2wDVVzzJNmWf6wzlNF5s/1sut4W1CtJNptTYpT+mHM7BxuuV6Y VKjnog9mNcq9wrszBt5xFZ2lg1npovcNts7XJoD286WWMjSJJnfvD55XsOXfwN82IcB0 Yw5pmh3qrTjObesHLhW0gYMBAp1RK/C4IcFvB6vTgfqBTuBFMWOqm6lFhSo02EjvQX9f l8IxioBd+hCuMOsef6yUABb6ZSAxIVHxtrh+suwTFTUlweMLKfozVa7x14IN8c1sXt/v 2UEj1MtsiLRqc4/k2r89w4B+yXCODdzfUtFc2hRJNv81SvOwbQdOdggEFppGTs3bF8Cm tDFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=iZRzZZUde58Au/vMHi9s1sr9e50uKHmMzB9hG9Emfi8=; fh=yvN63QVq0nFna1WLidysZ4ad5g6OhQJzYTOcPhqxM3o=; b=Z68iowbbOUPzq3DmHDhYkSKq9Nh8mRAeDiMj+IKnjTw1IfzCwpAPuu4+fBq6fJIW/m lf69m7+It2slhb+q8mtLwD63YOhXZKgubFtKPrl0/h+kPxgDEL/fMjsPAvM7TCh9T40I rVpwTiKaCBiSE0wzFg35kae+MBWAr0BfvCB8iO3TuwEiAdpKKw3pD477GHnuGmr5YrXd Etu5BfIbRzdL3vDdNviIc8Y4IJsX3EHuf/fp1HfEIqtzhdqrEZULD/giYf4Pp9nUW74d 8HQQ3HFbXxlqIIXm7jgYBvBe7jwhgp7Ghoo1jrmX3QvRGqYAD7tJq2faKNToJVuBvxPi Ua5Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=MMvujdwY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id l3-20020a056a00140300b00690208f26d7si18271804pfu.295.2023.09.27.19.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 19:10:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=MMvujdwY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id EDE0482277B1; Wed, 27 Sep 2023 15:48:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbjI0WsU (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229975AbjI0WsK (ORCPT ); Wed, 27 Sep 2023 18:48:10 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1971193 for ; Wed, 27 Sep 2023 15:48:07 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d817775453dso18004640276.2 for ; Wed, 27 Sep 2023 15:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854887; x=1696459687; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iZRzZZUde58Au/vMHi9s1sr9e50uKHmMzB9hG9Emfi8=; b=MMvujdwYKXKRJprzFVBYU59g7lhPDJ0NYYc+URTCje5ajJID/En3NEuZG+lz8RD9tV WWxKmKcftzQipUbpFKnOu9WlGo4pxV59clWT1V/Ej/cp0yXW9ghY7OFnLLOYHn3OEOIC sOd7uukOAwhh5eFkOVR29lkdC3XtuLwgmfT+E1U2hbw06uScFMsnKwxnupGieEJgfmP0 ir+/r/gstasy2rg2eu7aHe2RwlOsZzgJoCj6NSm+2svhonHu+n4TpvEAdDnZDo9WMJ+j d63j4WT8a7+4GiaEVdC0nna5m0l4c+D3I+k+MlQwV+qY9uDE5KF04DH/TqZnbLa/7YB6 WPuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854887; x=1696459687; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iZRzZZUde58Au/vMHi9s1sr9e50uKHmMzB9hG9Emfi8=; b=bVCS5tMsXcpOnSse4LlCvxF5bmPz2O8vPzUNRJkq6x+CrddGG0dcrwDJEBQkBf+K09 5yAFByDL1Z/JhCRIjmVpGXArsgIiMa1II6n9lFVUDjCz1c327IAkwxeI0h+crQ99YrrX 49ylMlW/JQC4wdtOf5QRaBNU1tf5x99gEo+A6U0f5+ByIJ+RhsF+Yl4x45UmkP1gVszm wbwNs4SPlA6EPAjakdNpjMMMhNTKVTdN1HfMjDcB4vL5DQvAivkXl8UeMOlYzMJZwHPK CpTsOBiwM3F7ZDnKHTQW2dWAW6qHNLSn27DKU1QQGPdvCYQsxFT/8mm2n3PJhhVnldcr Cmyw== X-Gm-Message-State: AOJu0Yx6R9ewwRyN3l3qYiIuFNRcXd7ruRnPGu7wJTbS/ef0JJ7lmKRp KCD/9Tk1hmu5TBCp75UIqrb8Lclr8ZGANoa/w9w= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:77c8:0:b0:d86:56f4:e4a3 with SMTP id s191-20020a2577c8000000b00d8656f4e4a3mr47419ybc.13.1695854886912; Wed, 27 Sep 2023 15:48:06 -0700 (PDT) Date: Wed, 27 Sep 2023 22:48:01 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3282; i=samitolvanen@google.com; h=from:subject; bh=R9zKwYTLpHe6QQ7XyQD72MWWY4El/ER6AIXVUR/GJUM=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEdZRzIvfGvPyQVjyzKduUT/iUIvXAAcWfD+ dGjTCtj8a6JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7vxkC/4lyaxKokcSbtqaj17rhTY1rhzJjWKwRCnuvZWzRstnDuir45jKziPslpAWi1iVs4RHF5x qreszbKjdycXuhJYRxcRe6hhHwB+MraLf+Jmo9RRUrC5vJrRwF4G4jMYJlYoEvCtKXrin9W7kYA lSX1MRkRZB7fwcX+42TsHFgVDTdkCgHi2VXX9HyMdy1BEjocetNYmgpwSy3jb/2sYwU9rOeEP+d xPuWVYmhdclejB1IRxi1FlxFV9dtGt/8drHYZaS0+mhI+6oC98fA8xqDLchezP4eqDL2BbTrAop m4GCYz2lwbxsAvQuj/G8ryvpfdC9INZUpo49ZnuqH9bAEQFLNBDL8W24eDJfiIggFROFKvicuWf bmwVX9GWrCQzHkG6Z9sT735briNXedUmD5tg1TizousIT92Wq54P/gQRA4WXCwWgeZMTTTQ7m2M 4Akymo49EdgXk6dPjkQb4//KenjqGWTqBerQSlGLryhBvRQrAAVQW0OWp3/+xHxCLiuOs= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-11-samitolvanen@google.com> Subject: [PATCH v4 3/6] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:31 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778245470246154231 X-GMAIL-MSGID: 1778245470246154231 In Clang 17, -fsanitize=shadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 8e446be2d57c..f34dd1a526a1 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,14 @@ REG_L \dst, 0(\dst) .endm +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index a306562636e4..6215dcf2e83b 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 3710ea5d160f..a0484d95d8fb 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_entry.S index f7960c7c5f9e..d5cf8b575777 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -61,10 +61,7 @@ END(__cpu_suspend_enter) SYM_TYPED_FUNC_START(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer #ifdef CONFIG_MMU /* Save A0 and A1 */ From patchwork Wed Sep 27 22:48:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 145633 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2961310vqu; Wed, 27 Sep 2023 16:26:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IElilPhakNYE7bGD4g7h8NsSkv4NQEdXKsqGXarZ4JZFTKOW9qur8t77j+AcEPsDO5CqMFU X-Received: by 2002:aca:d19:0:b0:3ae:5f15:8bd6 with SMTP id 25-20020aca0d19000000b003ae5f158bd6mr3160993oin.35.1695857163322; Wed, 27 Sep 2023 16:26:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695857163; cv=none; d=google.com; s=arc-20160816; b=x58jqJwFzuLUN4lNDUT+/NpTUSDmf1yeMrmHPJ9aQoGLQvcnUhDb+pVm/wZuDez4Y1 95CRXr2/GuOXKJpeJKlNa1MA64JyQ6zQZSsR3XekLqmmXLYmk5i6Dzm5Ns42JY3Z/kYz ewaRlZr0hZlNHj7Tx3VGmlGUToGCedt5HVc+ZX/YWGhNsJLezbtx/K6su49P2qq9tv4L ElY2/RIlGleRMc43hxVRasytc+tMRQy04ujHtykX0UbIu4438dxeBKzG6sgI9gwH8zuI 8jmtQT63Ff+nk+EwVcDWcYjUyOvHq5o4dSqIK6+e1i5qeJvGXO7bV8k/lhBQ9FetdNND U/ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=UjogArc4Ol2Tu2wVu5HYhWlgdLzR2HyaaCEJTIGL1ME=; fh=yvN63QVq0nFna1WLidysZ4ad5g6OhQJzYTOcPhqxM3o=; b=IHzbqvO8E5I7PS38w1m2NjbgLy5Fmv5YkHPmTDaVLgjmdMGWXyald3CMqT+r81UOlW 1waYFWhcnrl8oj9CjPynTDuMTQ1dgcxFwEJIeKPGKuEpUOmaThiHUedPSSguOECr98Nb jJTcPVOHk4MNuvuZnjwD0569bhnL1Fc23MPHWVlkrvAqhTIoKkkZBwzd0ol3eszmPGgg QZ5m62lnLf+oJmBiWoqzTPTNFd/nDbXLOXeodPpyno/bbkTXjL+n1sWBGD850leY7mvV EY4ilcj+YoIujHmop2X8mHyJnlts4U4hAPcov9ASSmKzoJFC/zINtY0bd58lmo2xNO7L +TrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="h7/RsBSU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id g20-20020a056a001a1400b0068fcf6fe22esi17505090pfv.306.2023.09.27.16.26.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 16:26:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="h7/RsBSU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id BF35B82D755F; Wed, 27 Sep 2023 15:48:36 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230071AbjI0WsX (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229960AbjI0WsL (ORCPT ); Wed, 27 Sep 2023 18:48:11 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B09D512A for ; Wed, 27 Sep 2023 15:48:09 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59f4c72c66fso200565567b3.2 for ; Wed, 27 Sep 2023 15:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854889; x=1696459689; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UjogArc4Ol2Tu2wVu5HYhWlgdLzR2HyaaCEJTIGL1ME=; b=h7/RsBSUmj32JdZ+FogGrCwfCTHK7mwnykUORGQAmK687LXWOXzDTdBjmf8MQ5qql4 rwI7ee2EdEmvgPaq6DmsADOdzrdizV+K5Tq3Fso4CDavjq/q/HwTWbfZuHnXKtLEEsQf iEoTBp0kh7tOPUDnJmKbviBvwc38d7OuDhwplRxOPmI/J/wD4posYBTsuF2IdqlIv4oJ 9cKzSDGmgMixnPPLlULHOn/bE2c4Sg4Gcg+DCrVA2X0ywm1UwmB2np1TacbkvlRWfq/g 1Hux3+s6HWQosuRFzCepqtOPSUi57Wu7+TJAFqPaGGvI9JotYBFdpwzILPMrEIVxvNvi Gp+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854889; x=1696459689; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UjogArc4Ol2Tu2wVu5HYhWlgdLzR2HyaaCEJTIGL1ME=; b=mi2Ew7kpAvHem7mtkwoolt4OO7L9WH5VAg18gb/acmW4pxeNHj2+rNJawRO8H6WIZp WGzWCz3StpWzA/vKuOSPfI4n2PaoP2ZYytFvgdMwDM5rJt933vEB3wbpgd+fMovVlB/Z LSybeSTshyL5ApgaQwFidsIgcCxKsbGhMeC/2/e6WtM/rU6hhzXkePt3cFiXrPcwMt/N LGMzKkNjaPTji9E3UCPffAfFibeaxDQ1mK39U9mNgD5SkxSZmbntQKBKy9yqlrc+136u pVr9oMlcerSampDTwwoTV7+HMJcCpbfNZyMFpC7l3rRCdUNOZ6YTvzcVc7Y5LXzJM3zM znDQ== X-Gm-Message-State: AOJu0YzoJWvnzQSXcaxej6y03LmcLHfNwEayes6OaMl9WxCQLVnc/Cf8 NKxamWDlgx+nPbJ5IfHJ7WCI1/OZfDOidJyOiTc= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:4dc6:0:b0:d81:90b6:cedc with SMTP id a189-20020a254dc6000000b00d8190b6cedcmr45254ybb.3.1695854888882; Wed, 27 Sep 2023 15:48:08 -0700 (PDT) Date: Wed, 27 Sep 2023 22:48:02 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9659; i=samitolvanen@google.com; h=from:subject; bh=yw/kz6qVoLV9/zzAHdEXeH4UXswKqFcfKUE7jb2zKUk=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEds/VsrP50aF0lwBOaJwOiBeaGJNlMIHWXo wP23QvFjeiJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7sGeC/sFu6YRgLELFq8ildZNtq5ri2wh4h9UBxspPil3xG3H36TktsRnR9jN1ccehLwGW+wBlWq nEKL3mAp8N18Y3rQoaCpUi2cfT9TfKf6DbmtAxsvvfeeg9UhSXlolr3XwtwX2+roATGNZeYhoNE 59CEcqS8CxiObRZZqPmtFyGqMvWYxfK/Zbxmq56jjGdAKy9SWmzDJRGEakAlg9AQSf4s0AIkvAN 2TLqF2dHl2q/DW0omjsRQdE/peSPZwhwitfGh4VTcSsH5EudR+R50jDNp3cGdyfmwMIAzTdlK0C dTI3uGWf9zkVd9xPvT6AGpAynfAdv2Txrha2bEG8WY191seBtnjWuQDgn0ggno6kdGe0P3PGGp1 wucGs64jnPo+i3opdERML+XpfWhjwSNgCVtbEStQJkrA9O7o2hbuC/cfGEBryOpOjYhfv9FLRNc Bj8DSGKv+cCvmY+ULbeW3oo1Mv1nSvvJM0B2V/fUWIGMhzWmylMvB0Md8n1zzQoGzfuvQ= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-12-samitolvanen@google.com> Subject: [PATCH v4 4/6] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:36 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778235120897920930 X-GMAIL-MSGID: 1778235120897920930 Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d607ab0f7c6d..2f14ccb5fb01 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -48,6 +48,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_USES_CFI_TRAPS if CFI_CLANG @@ -174,6 +175,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=shadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 1329e060c548..304b94446507 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS += --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f34dd1a526a1..b0487b39e674 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,11 @@ REG_L \dst, 0(\dst) .endm +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -116,6 +121,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_sp = init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ + INIT_SCS \ } void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 0af8860f9d68..a03129f40c46 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 6215dcf2e83b..52793193a763 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -275,6 +282,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -292,6 +301,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index a0484d95d8fb..18f97ec0f7ed 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current .Lsecondary_start_common: @@ -289,6 +291,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -307,6 +310,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_current #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makefile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY endif # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) # Disable profiling and instrumentation for VDSO code GCOV_PROFILE := n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index 9e6476719abb..6a3c16bd5ca3 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -81,6 +81,10 @@ ifdef CONFIG_CFI_CLANG PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_CFI) endif +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o += $(PURGATORY_CFLAGS) From patchwork Wed Sep 27 22:48:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 145811 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3129058vqu; Thu, 28 Sep 2023 00:20:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFIkwsE75OwXIAYrEpQ23rmuU69q5vVchPkCRBINHoNE+4bUbiKhyzds2Yco2ht67FEkl35 X-Received: by 2002:a17:90a:74cc:b0:277:4be4:7a84 with SMTP id p12-20020a17090a74cc00b002774be47a84mr744298pjl.19.1695885611168; Thu, 28 Sep 2023 00:20:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695885611; cv=none; d=google.com; s=arc-20160816; b=WqvueTnQXPsDaqDcxdc2xl3p0igcapuTA4lIY4PCjjwbR9jnijOQyW4mSC407GqeAu x/WFY6flsFh0eZ9ruIUpbWSw7r2eOOX/5ebKvh3mAohTOM8Td7UbXxiYYrN2lDI/1RcV UZXkq4rM5WS9ShdvAbu7yPTnGNgS2TvYj9uNFoRJwpigfycdfFKj4H43SkzxkMOKpLmr wXaAFuzlUbNcHkgxlVV5BcsOwXZOEOOk6YWrFwHYoaEZGYy6XaQ7vjMBaaoAU1YP7NUQ WB3HjBJDHiAgRbrIkopX53lJh3FCzAl4MI/KrdVd6tkn6RijJHwWKWlIun8NugXyQSjn wUbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=3UA3PzXciXcLUc7CxJsx04dBK4in+Q1qERJTsVZ4p+I=; fh=yvN63QVq0nFna1WLidysZ4ad5g6OhQJzYTOcPhqxM3o=; b=U+p9CE63mOhN63mUFINeRIKgCOD3c2wtSZjPbV6UKuITBy38IOK0eZkMgyUiBG/O4i Fc+RitSDkwwbFddyslyxysu3+HnzKw8WQUlg7lBkEuqYpABaSS4f1aRWabVAg9p4ENk+ 8+Lg/Dro0zQoByWVNIXd/Ii/IrQbzE8e5kjX7KZwb6BJoU75A3Nhz36JlkiGbOmSm9bA So+7QMT0/OkhHxjd57TiTkUvXXPN8TedVpT83G6njDzk+zQDaKwcdg62rW7D7/ULiLYU A1o2f1b7qFgvjtbZDnCeNIWs8jOlbIZL3J+Wo7O6ZcSLu9ucKZiUxx/utB2u6FZokqmY KDKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mQFtQvNx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id z14-20020a1709027e8e00b001c5fb45613dsi12816036pla.318.2023.09.28.00.20.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Sep 2023 00:20:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mQFtQvNx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 0121182D7559; Wed, 27 Sep 2023 15:48:35 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230075AbjI0WsZ (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230017AbjI0WsN (ORCPT ); Wed, 27 Sep 2023 18:48:13 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A1E1FB for ; Wed, 27 Sep 2023 15:48:11 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1c725aa19eeso27302875ad.1 for ; Wed, 27 Sep 2023 15:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854891; x=1696459691; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3UA3PzXciXcLUc7CxJsx04dBK4in+Q1qERJTsVZ4p+I=; b=mQFtQvNx3OjJt8dEyvzofTGIoYMOtISxmii6C6W3QMTiJw5tKlhyCnlpx/0eDPLE4O t2R3zWXilyeMcKbqZKT2fOmb0hTC4LzUA7Hj+MaUaAcB9GIS7JcJ3V84QqR9j+szZleY UHswGqEebOHy8UxgzlUVUaGXg0Ga57f5bDrgyls9O9xJ2iTx459bauGxgBg8fE3DEiMe xm5q+5LaWJ9J37vatZ+N16hbQQULiUZPU1gRyVaDnnDYFDJ3YzA71OE/ov3QWSqALd08 iKBjvgjeRCrAvJByItud/r/iaIsjpiGSm1oxRCdE0bVzs4uuVMxBy+HIhK1epuHxGOZc 6UFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854891; x=1696459691; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3UA3PzXciXcLUc7CxJsx04dBK4in+Q1qERJTsVZ4p+I=; b=iWup7YGr5VmCewcQXvNM7mbVhC7IWWvph3icSvx4GF6HMrQ4BRVHUfPNFyHiynNloi V6qYWW1IFFK5AY8LddKscBTc9OuSpZcBCT/9U4SCs5aYQmiIxhU1ezsB7fRW0qU9Rlys FHpBq8iRYMLKlap+3U672g7Z2UEURjUwDwfe4tgT5dzeMxhheABOrycYO9+crxckVtNq 1xbxiNpSZscZ6OYIat5mhXNcLoNkzpvMEvJjHHyHHI6M9CeUnMyL6lkK89ZlcnRO1CEZ S9jMXw12iLf2y6EuyB7ElX7+Apb3qqIi60wB+VgWLjwHmDQGtadfcHrh3H/NGl/HvCKs x2Fg== X-Gm-Message-State: AOJu0Yx7KQC6z3GHOtvRVGhWixk352dAda3BfYKYOJ3pspz350KpbLo+ J3BYqg8bgidd1EBGKDgnQQl3Ffo0dGMtBU6fMFU= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:902:d4ce:b0:1c0:ac09:4032 with SMTP id o14-20020a170902d4ce00b001c0ac094032mr40690plg.9.1695854890733; Wed, 27 Sep 2023 15:48:10 -0700 (PDT) Date: Wed, 27 Sep 2023 22:48:03 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3222; i=samitolvanen@google.com; h=from:subject; bh=4iP9jD07sRLKZy9z4wycr0gGeMqN4nW8GVhSWT+pt0c=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEdc0ndW+W8xAwxA/Cqwr/zGM8KrW/4U7Qvn D6/24dFgZmJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7j/TC/9QYIFVRwBHilpMvJsGF5ZuLLmHXC+6PyGWcKJr1nfV4YChcnH2AyhrR3Jtsn0GDJrOc8N 35cE87DH6HMvHXoAdjwYFdSZfEOQUulLMiyIZ37pg2TWFbhI4B0s95x/8KXGpIgMnApDq8FRKkr dkX4ZyFmhTYLNIpmNAleOednmNBtzsKapufJKcTpmhCq8SwzCTpkINVE7MsKmugeis556J8Sq9n py+Q6LsRmlhB7THYIcqHSiye89YKOv7oCvgahLR8miSX2x9iG7s8Mla1J5GqT274t3WrXdNPy6l B1nug8SBfJtw7s7N8My+43WNYyLLeJEzZwbPjTn9WGY1jqgcPbMigApDdUKe8Fdxb/oCiX/DVXC Tky38rffJrp5N9fn0HexYQXlNxa1ATOdk1/RRB9kKz6Er1hZRieQWrM4xBxWZMV5cd3WF1d0HmP PtTL6WVk+2I7pBm+006Qa1nQBDboCBjiHxppqh3VuXLq2JGFPc5C2ycM/tKhII8IFb28E= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-13-samitolvanen@google.com> Subject: [PATCH v4 5/6] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:36 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778264950972281163 X-GMAIL-MSGID: 1778264950972281163 When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 52793193a763..3a0db310325a 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -237,12 +237,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, STACKFRAME_FP(sp) addi s0, sp, STACKFRAME_SIZE_ON_STACK + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -STACKFRAME_SIZE_ON_STACK REG_L ra, STACKFRAME_RA(sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) = + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) From patchwork Wed Sep 27 22:48:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 145631 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2957105vqu; Wed, 27 Sep 2023 16:15:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEdtbe369+2lsb2jnkPZkf7U65kqVU9miHUIn2J/el8Kih7OxuRP8c1D1yQGnWgpt3GArrS X-Received: by 2002:a17:90b:3ec4:b0:274:dd15:87bf with SMTP id rm4-20020a17090b3ec400b00274dd1587bfmr10876612pjb.20.1695856514196; Wed, 27 Sep 2023 16:15:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695856514; cv=none; d=google.com; s=arc-20160816; b=FbrFxr1FO+3s8D58cP4KrbJOOJe3q2K4nrLLrbuZ3iyEcw0yjlqScPNwJ037DxxgX3 exNxglpucWn9Cw/sbUw4lu9jhscl9FODTIvLLC3zod9zpQ9F5vPkVd3JCpO35DZqkyUi 0P3gj+kqUGQ8tGgBNz0U2Xh0uHbUWUr5M9C7nQx12Uj9CirbaLzD/t0eFS8fXtmRkbN9 gdoflFm7wYxxZACzGQF3kRf2MdD/d1tcXxj0jg0xebpfi/oDsZ72HSM9Na7plpr0qI4e 966waK58fZ1DfFOSFo0oeAa616LCNxPdNSO/2Wr4U2wTQjwwTeYsQbpqhUvr+Nlx7Uaa XhnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=6h9NlhRD8hF3JGpC0yj7VRjIJqQSgsUALXY/NiPb6h0=; fh=yvN63QVq0nFna1WLidysZ4ad5g6OhQJzYTOcPhqxM3o=; b=v/bREM5YTZ3uyJ1nWJw4gXWNRoT5Xd7Rd5Z58DMgNLxvdxFZKkxwxjl9l0l+uYJ8IO e4RH3tH3Udz7yCGEQq2mL4u5Dm8lGKE6o0Orub5KdMp0nw6bjqRChk7FMvOj4i8cW6Zq ffLZ/xJkiKy7cZK+am2tVSpGa03rgaDJRp4vQnykFlKPZxOcCgzjP+veFqAoQJrSYOO3 GSasP81CESRhAgNO8GkMByWbr1zDpEnfMtBED4zyrgYD8qu8cOYKnS6idK860ZBBKQD5 dXz7BeLvLoIKSzBYHvXL0MFJMMOxlq7GijmROrb36kU4FzOiVoLOHrkgGyhHPybawFJB uX4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="W9P9xxx/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id i11-20020a17090a974b00b00271af83a25asi5148622pjw.59.2023.09.27.16.15.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 16:15:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="W9P9xxx/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 8997282D7572; Wed, 27 Sep 2023 15:48:43 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230049AbjI0Wsd (ORCPT + 20 others); Wed, 27 Sep 2023 18:48:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230036AbjI0WsO (ORCPT ); Wed, 27 Sep 2023 18:48:14 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9342102 for ; Wed, 27 Sep 2023 15:48:12 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1c5fddcf2f1so135790325ad.0 for ; Wed, 27 Sep 2023 15:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695854892; x=1696459692; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6h9NlhRD8hF3JGpC0yj7VRjIJqQSgsUALXY/NiPb6h0=; b=W9P9xxx/gnYx5osYpeI76vzlOzUEZfBvOLYTAO10FfRylT1mRq2bCMYcS1+hZft3AY eGZxePpApMxNN6c3WlWJBYdNELeMLib6wf1nyJ5oA4p5JDg7FCt1iW+URV2HCAkBPm2b eqQeNetpEjf0hkEkQ0YyuyF1TCs6Vs9UTvLLXUReoJtc6/JIdz05h1AAlp1beXi2B0lQ isFXBAT7jFPTgJQ8P+kuxLj6ro5eRYIb2ZFthvt5nDPRxsDGJoktPaerzsb8uVoQZzDu n62biOqvhU0Oo26flZg4+asLR8fOy1TKIRJYK5XRG5g8V4oNQYf9ODdXlY15hK8yzQ+i 2DEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695854892; x=1696459692; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6h9NlhRD8hF3JGpC0yj7VRjIJqQSgsUALXY/NiPb6h0=; b=oWKkRB7qu8zr+7JcouQvxbeuXqo44qFvVnYE5l3LGh3qHItGKZQPzuHtLfPKib4Z7r i+kAY4TGNeQMY+tgrEDYR9HZNP9mdfVKgLbFueL4HQbHEv/fHwsJFcFYtQbtckJOBNfd t5f2OWadFWaqvUy5qPluC6F9CjAg0/vAZKnpihor4NgXSTkldVJpqwlTJzcNnU1JWKFL mW3wrkTwHCR8SN8e6bz+aENjPI70aWUTEA4Sk78LXMrQ1tBUUyU3PfF2VCRLA44rSq6x N2emBL1gUzbYKm5yiPMIvr+cwFRTveCG2n7HNEpwIpjE3v1C526rh7wFAecR0X9aSY65 fFmw== X-Gm-Message-State: AOJu0Yz71/nrM7dM5v3awK3SlMAcAJPP+laIxxttpcyL8rpIQfq6OPIJ zQPVVIQ1P2mW0+EDwZ96/DBtniKs+XD9RA3ZdHE= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:903:2350:b0:1c6:d25:8750 with SMTP id c16-20020a170903235000b001c60d258750mr47383plh.10.1695854892186; Wed, 27 Sep 2023 15:48:12 -0700 (PDT) Date: Wed, 27 Sep 2023 22:48:04 +0000 In-Reply-To: <20230927224757.1154247-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230927224757.1154247-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=2082; i=samitolvanen@google.com; h=from:subject; bh=4N5TA1RYsqEG6l9M0xplln+cBa1Qt0iz0MCbO5csiTU=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBlFLEdbuqhUGRkvHlpnRFoLVngxWUbGDzCKHZjB gXmrezGpZuJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZRSxHQAKCRBMtfaEi7xW 7uL3C/9V/COsf71wMsf8LAD8PbXFSUssu/Ulme80Bc+IvZbku+VH0yCAi+0as+ACJEpRsTnNbIQ pScn4vacTqNH+5w4IriZtrxed/+JlJlJ3oFsv2kgI5IxJP44McljBfgT1GRyGCHurDabqPsJYX5 TaGUz9a7+gFF6iVgMLHb9uNWunfwjDU0mlECp6rp+occD5rm4k0XsmhUdYYDNB2N7vwL4ar7t7Y bRUrWcDXIDTzByGd1qZFeUPJ4XRuZmExIhm3XrLYrmbZvkYIH6oKzVv5knn9MHzzjsjfdiVDoME e2Bibk2Mj6F9ABYdpfXOcVNTMcGMZtsjvwmAv9Qx3j7PwTa5naJ0np9PriPDIM2jsshK4hOOa65 Z+fMEcQII52/5PTX5HX9sXBqn/ly0o9tU3LTzbR4GGVs6W1Rzk4Yoden9lVz+2Ti9QdMMMmqjSM hvdcZcYO8OTXRXZtW4dNPx1A4wMY6pfJadi3BaYatIPrJyHzova74Yk95MQNhaBqyh0DU= X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230927224757.1154247-14-samitolvanen@google.com> Subject: [PATCH v4 6/6] lkdtm: Fix CFI_BACKWARD on RISC-V From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 27 Sep 2023 15:48:43 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778234440428645340 X-GMAIL-MSGID: 1778234440428645340 On RISC-V, the return address is before the current frame pointer, unlike on most other architectures. Use the correct offset on RISC-V to fix the CFI_BACKWARD test. Signed-off-by: Sami Tolvanen Acked-by: Kees Cook Tested-by: Nathan Chancellor --- drivers/misc/lkdtm/cfi.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/misc/lkdtm/cfi.c b/drivers/misc/lkdtm/cfi.c index fc28714ae3a6..6a33889d0902 100644 --- a/drivers/misc/lkdtm/cfi.c +++ b/drivers/misc/lkdtm/cfi.c @@ -68,12 +68,20 @@ static void lkdtm_CFI_FORWARD_PROTO(void) #define no_pac_addr(addr) \ ((__force __typeof__(addr))((uintptr_t)(addr) | PAGE_OFFSET)) +#ifdef CONFIG_RISCV +/* https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-cc.adoc#frame-pointer-convention */ +#define FRAME_RA_OFFSET (-1) +#else +#define FRAME_RA_OFFSET 1 +#endif + /* The ultimate ROP gadget. */ static noinline __no_ret_protection void set_return_addr_unchecked(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected) @@ -88,7 +96,8 @@ static noinline void set_return_addr(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected)