From patchwork Fri Aug 11 23:35:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134792 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1466266vqi; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWLtVSM7ic1z3HatCoSNDQrecFS5JYTKgMbiWNGH2psrIjY4Yjw6d+UhLo5Tsdn1JYcZkD X-Received: by 2002:aa7:ccd7:0:b0:523:2e23:a0bf with SMTP id y23-20020aa7ccd7000000b005232e23a0bfmr4349060edt.11.1691805212105; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691805212; cv=none; d=google.com; s=arc-20160816; b=pryHekehD/9GVgfpabQ0qGb32KHlyRr38OOqbJp+z7r61EA3ePR6TNs5n/xYp0uxGX 50JO9KerSaQQJLzyIhtWljkFL00Ug2q2S+wb36I8nqCREpAigPJiyC8oq9WC9Jquq+3D tOZ9Kd6iFFmIBA/8N+xSsSVni0Ar3s1OwHsr6SzniAWcF1ALm1XILAV6lW1wW4lAIMV3 vr0VsicV4W5gcXnAzXlSCHtngkDr63Y4UgYbm7Lq8cb5zq4QwE0VPV4vpdAoRR9WckqA jc9pjicwGfvZVcKx6My4XkLfK557LqIVURQrXIty7gTG9DxUHPOdMJujWDWg0P6KsRWd hEyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=rqs3gRDgDiyZZoJyqjneRopVqblYdYfI25KzjLQIWhtE1G7XQjQwek6Of6K7j3QCW/ xUukmK9MtNir4ySvldOTWwNt5TQCO4OiqdnHseTtflzkitHfP2Olv3EKmSAvzBkS/yNY PfyIIhv+biLLPsly0sfcDxbzJAMIJrnNykwVQLFJ9Vvgaqafwaw7FoalWi8Hllq4F2r+ k4rA8xGuu2RQeVuDobhtsR/7Y4RCodZ0r2T3bnR6H+4nxp7udS9ddVfpEXY/0N1dtRm8 I+cTt6YSZpCoxuM/5+8MEJdZRgKQGDSTHS/DkO6mr7+gP5a1IjvaOWNzVS6tijta+GUN Zjsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hNk/V5V9"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a50ec87000000b0052340ed2b5asi4287280edr.174.2023.08.11.18.53.08; Fri, 11 Aug 2023 18:53:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hNk/V5V9"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237148AbjHKXgJ (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236991AbjHKXgF (ORCPT ); Fri, 11 Aug 2023 19:36:05 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA29D10E6 for ; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d1d9814b89fso2526364276.0 for ; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796964; x=1692401764; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; b=hNk/V5V9Cl0RBDK/xApOCjB9hBugkkuVSUe0CaeEJvfxUWOY/dHs5BlzQK0Ht8K4GW wz4FNlEdTQbu49jXP5WDdQc5jl3P8kYStgDJp1EcsRF0QdkshXrIub2xylJHtkojk4ne JShRPYJ8B6jOJlbiGmGCSfFSWOsj0WF/PLyolSwVOkDevRJrdhjhTjMhtDy5vO90G201 BIsKD+9tYPjZm8FB3dSatCzIyzW2fFKtOhxoCYTY/u6aLh+wHQ07/9w4+qNAMfEp2iEg XPjLPfnAf3gGHWk9mdXr635RhY2oJpJnb8Al7Qq+/hRH3xOIMkEcrstW5KQCxtb2FE+i EG2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796964; x=1692401764; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=00MDTkG4twmkTIj7hahtsfZCe3BOd3kJWcjxWEDZHsc=; b=Dpj9taDn5aoUZCkGG6O/DShdyvpuhApur94u5x08Q/8u+WIfUZ58GkM+p6IBvggips X7W/GqYc6Jmn41qsr7IbYKyaJsEV0/TNMKceSmPDUWsVfgBp/wmC+Yx8NGJHlNdK/92r hnOe7IDo2XSegndi6HpHpEcyQmZ8A3EcEwgR8VgENu+Kq4NnoSRdmU+sNTBThcl1p5Bz yqavAwBjETjP9LX8jipOc57P5lIzi9K4fiUv82Y9hRBUhlOX0vyFFdubkfL5XaOxgABC aXXS8n/yWBCt4PISkcJMS7JswoXzqaUszaxlWdZ8PavqfrSjRYSWxFK9rWaZp/+OQJBr L+Ww== X-Gm-Message-State: AOJu0Yw+VSQI11+eLf5uAwHLWpVQWhuZIBr3BOsImRstOMEevB5jvBgz DyrzOh4BB9kK7ow7ypOgNShWC+IXO8JlYLIwXCw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:2d0d:0:b0:d4f:638b:d7fd with SMTP id t13-20020a252d0d000000b00d4f638bd7fdmr52997ybt.9.1691796964113; Fri, 11 Aug 2023 16:36:04 -0700 (PDT) Date: Fri, 11 Aug 2023 23:35:59 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=5591; i=samitolvanen@google.com; h=from:subject; bh=iC4KVySOHJXm7MoqeVidxpUrvxa6iA/di8sMVQphPro=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcl+4pMbxsusEsJNlSct/N38MSVxyzWytYj 3gXIn1dRb6JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7u94C/9i14LHcJc+kPonuHpc/Sr6RicO+kf9zaXFzOL2MKaPq9BaDmMLL6nuG2iHksvhI5boWFV +nA3tePX//NgpDNCfDQUBkwGSEiGRRxy58McWFflyS40n4keNX4+P5qJZekqHzDfH69pmaUE0MQ xrxUzYYyWBD7TxyTGewSmEAqP6bRWJhI6FL0zGPkEvDSw+KIQCWYSIcEOMVhgiw8sGyIZRDS2Pd y6Keo6flOy8SX/b6Si5H2Xoczfh4JZCkLMAQCC/LL8zA24db+TJRrNv+lB+1FW6hCXB+4+t3Zr2 DzINx6H3f+B5qU++tKwqqnCG5qXEbyWLKpLIZqyUNTtp2KtfoW1XbXUXE8BIFc8NWNeNUFEo138 LaBbrdBcscNR9th2G2TBWCjHWmXmWXzb2hGZFTSPg0drLr8BP3MHqva3BhkAKQiLpKttkUqExGH uowMH9lPJ+i7mTffOnRwkH9X1FO6d6TkI0etUkvRZhky2/6KfeNUx8nKV3LGflV/YYs64= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-9-samitolvanen@google.com> Subject: [PATCH 2/5] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773986341897771685 X-GMAIL-MSGID: 1773986341897771685 With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/entry.S | 32 ++++++++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 32 ++++++++---------------------- arch/riscv/kernel/traps.c | 29 ++++----------------------- 5 files changed, 52 insertions(+), 49 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f403e46e04f2..13815a7f907a 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -98,6 +98,11 @@ add \dst, \dst, \tmp .endm +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/irq_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..39875f5e08a6 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,38 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -RISCV_SZPTR + REG_S ra, (sp) + addi sp, sp, -RISCV_SZPTR + REG_S s0, (sp) + addi s0, sp, 2*RISCV_SZPTR + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -2*RISCV_SZPTR + REG_L s0, (sp) + addi sp, sp, RISCV_SZPTR + REG_L ra, (sp) + addi sp, sp, RISCV_SZPTR + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index d0577cc6a081..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,32 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index deb2144d9143..83319b6816da 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,31 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); irqentry_exit(regs, state);