From patchwork Fri Aug 11 23:36:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 134784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1447638vqi; Fri, 11 Aug 2023 17:56:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGV6zkDkavnTss4kdgZ0Wo5LiVrOJHCAITbEf3LgQwPadjYsEm5UTuYSClHfNyAQAvZt2TZ X-Received: by 2002:a05:6a20:8f21:b0:133:71e4:c172 with SMTP id b33-20020a056a208f2100b0013371e4c172mr5058474pzk.15.1691801801648; Fri, 11 Aug 2023 17:56:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691801801; cv=none; d=google.com; s=arc-20160816; b=FWQp5Pfe5eawvNhW3vU0/SwD6Ukj0bDG0ape7j83EFMNmgKddQ0JnW9pGcBbkyQQvD 0A9KodXwrWtXzu9cs+XHu1SNfK2+iMIsfb6lZyXQQ62YCyvj5dS6H0GYNRYQ6xLr2uNv vbaQvD3ISxK+H2jju+Ne7at4NJmJo5gQ9sfs8EagCs0btiEw6wpheWSNZjg/HQBQkt19 Xlx7e5i0qh0D0e/XhYz3j6vd5AMI6mXIVLb/kBRtCqrly7xTzXNt4S1RNz75L6IKgSgE J8yd2a/d2zWEgywccpvnXksghA6PbP9Uq/8kpqkJv9s/1+hQbsN4hetrGdUJHdKbAi3X luEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; fh=EU0TIckDJHUB8oxmYASKGRuQqUEMHHFj6AqpMEFrOc4=; b=kBj5RzCnkDfdO/ghtaV4N3R+P7gjt0yxECGcEY+HMK8GYBckXzTJ21mf2r+EZ+bb6G uXvm6XZhi8Jgg6up53N0ryj/koK3b/V0iw7iPpYdwRbJ+EmC7x298jgzRk+4J33NOd53 KBGdtNETviRNXyM9F3qV//lGnUm4QVPmwuGpwdcPsZhmNqC4BqCsR1Tt7ps0oTZ391Cp SnNZjQs9MzfteBNXcR4UkDICvjsludSVrCLZNBxyORNHB+qTiqa2EXAqp8zYBRs6di78 a2zrz3IkEdcfAk9QoAtQsiTsCORvo2udcC1s5PPdaqSIm3DZ+1qwrwQ5FVWGiHlP+qmL P1UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="mtVd1W3/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k190-20020a6384c7000000b00564b8ae6081si4280743pgd.15.2023.08.11.17.56.27; Fri, 11 Aug 2023 17:56:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="mtVd1W3/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237211AbjHKXgX (ORCPT + 99 others); Fri, 11 Aug 2023 19:36:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237144AbjHKXgJ (ORCPT ); Fri, 11 Aug 2023 19:36:09 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A97910FB for ; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-563ab574cb5so3015763a12.1 for ; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691796969; x=1692401769; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; b=mtVd1W3/3q051Qstg/rkwh22KJtXJgwRmJjPFHPrSUwrwcnjBaqya9IaLakBib/tGp MZTlDO3i3B4tO+B0FXZi24NgY0QmiN3TLRaIN/r9BVzhVd6lXdijvcEwht3QqyGFD25d iAg948H3L3HcDIX/PDfVlKzDxQL0FtAOPKhRa0fRjcPK/dRobJigKyFzAfEEvzpV65fs kGDPQqp+LC2W1h1PDbovFc73zR/ks5uXhEXNhFQtNnxEmYALG9EGpnOyEhYzt92Xo9Yq 5nvT9GMKmOWUbwmKtjMTTTSyuaOqrL5J7LhfgrsQFnUpeCuAUc08CGdSsrD74A12oTl/ 4BBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691796969; x=1692401769; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VZF+IDNpZAY/LDD7db/2FGsLr/FEvCnem9lcmfmloZ0=; b=kXLKz/Yp8f9mr+bM8F1cTr8ti6PeK7y3tSejHads9DCxNdSk2YbGlKiMy1MWq18Mh0 K7NKOZ5oBdVqkVRQRR0uJuUpkuJpJL40i+aDZF/xyj1PZY/TNaVizAsZVBRDLlTsYZ7W zKYD5G1ECNhSun0TIrHLsaSB4BvELBpcZTDh21ll9b97G31UWGNJSs9c5DUavCneFlA5 E0B5/T1rYvcTZ9BV0Sj5hV4U55ZfdwFYIX0CmH/Ey4jh+h9tJu+vLWwWzGHSB85Ru7sK Pk2m7L5g8Lx7ivfnhLL6f0dwg1kWEHqkWp8L9oWUdFVs8QO7O9v1uTN6sAisNORp7KRq C6QQ== X-Gm-Message-State: AOJu0Ywvh1+HogDXv9Eq2PPAeaWfGy40gm6retrg3kix1iSWtXHIScS4 PmU18QUPP3qdsC5zMAig+VrnKWAsLK0+hoW91a0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:8c57:0:b0:564:aeb6:c383 with SMTP id q23-20020a638c57000000b00564aeb6c383mr638656pgn.1.1691796969048; Fri, 11 Aug 2023 16:36:09 -0700 (PDT) Date: Fri, 11 Aug 2023 23:36:02 +0000 In-Reply-To: <20230811233556.97161-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20230811233556.97161-7-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3124; i=samitolvanen@google.com; h=from:subject; bh=wBRXjmNHfhdcdsjrWxKjQVM4d1RVOAig5wuWuTXp+sw=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk1sXcweRCoqkszjtNMVnBaMVhIhZi/o4cME2C4 t5Qc51TbeWJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNbF3AAKCRBMtfaEi7xW 7oToC/0bVrByxxtfXjE7idOKMhQ47Ftkbwzrwd/qVpuGk5sKwMO1ZCCpNUD1WjHmpT9DM/jRjii ID7y7zSN8jMwvkBKVqWFcrA/Eaa0m5fWafqe4cjjWupJSB/ep6zYn/Rg6HLdIw7XP4JK/Vc4y4r TzWyrycXtowJuCYuF+VVNuP27Hv37AH2QTpPxcpmbb7m3TwnCZ3WQKLI40KhXg8gonXaggVqg3q X5caBqH0OE2M8dRrYxeDPBlFrqtmfu0hbgTF/flJQdHf2d68ZJHYv8m5rdu94Bnwwf6XZFg+Q0R ZQlYPtqVBdg6YwftizeZzNr6QjUFpqQ7A4buby+3pAI85S6NE4IF5Nces27nX/oKcSXM9+L4gog 10WOtL3ybC+F9bnk4ud+g4NkqrfaEszHa/UYGP2doaSEY7wnGja38VeGjqq9XMANxgYlDyRpWrs vGsTNxyJYa2V8imEkZFgy9kgnfoyRyITlo1VmA8GQU3rxJACgMLpVtK/P/BITWXbgVKXU= X-Mailer: git-send-email 2.41.0.640.ga95def55d0-goog Message-ID: <20230811233556.97161-12-samitolvanen@google.com> Subject: [PATCH 5/5] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773982765587192623 X-GMAIL-MSGID: 1773982765587192623 When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index ad34507d3c96..c86b76584d2d 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -238,12 +238,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, (sp) addi s0, sp, 2*RISCV_SZPTR + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -2*RISCV_SZPTR REG_L s0, (sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) = + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq)