From patchwork Mon Apr 24 18:13:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 87130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2927630vqo; Mon, 24 Apr 2023 11:18:04 -0700 (PDT) X-Google-Smtp-Source: AKy350apukTZ1H/ztHv/KzBSsi3aZ4aQFJU9i7+AH0qVmFRQt4CSibd+PRZxFTfCduIX2U8foOwP X-Received: by 2002:a17:903:40cb:b0:1a1:b65c:dea7 with SMTP id t11-20020a17090340cb00b001a1b65cdea7mr13908572pld.47.1682360283729; Mon, 24 Apr 2023 11:18:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682360283; cv=none; d=google.com; s=arc-20160816; b=GI2L9xkc11vsynvKUOgreN1C53XfW7D2cZSTemAx5j+9zwcOqldkNsYF+fZWbw/T6a FZODcYD0ZB8cFbgHsl501EIv+IcidMW3YmwDl9EoD2aFOoXvKbnEWLew5XZUnnuQ2tMT LKcxf8rhDH1svWQN84En9AVLGXP7YtxD3xukEp8KyePSRgeq3OhWPGJOXn9wu1Mo9Ibn eAmeEN+aZpN/FZXysm7pww5xGLe374qsQQ3on5IgX4CsmbaedaYG7Cf5p3s16Ts+XCWu R8M6uO/DmmY1wRyTa99VAx6VsELM/OXnlb6xUAhKLyt0RX+ezL7nTDpoTjc4CP8IWqhq P8VA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:date:mime-version:content-transfer-encoding :message-id:references:subject:cc:to:dkim-signature:dkim-signature :from; bh=5sjVCOTOKGYRbG6sN/eB48N/lr9EVJZJBYkiv7M5Xtk=; b=l626XRutl4npb3mX27hIJoH9wVaR2kwGpamVY8yiGbLaCc/sowqAaudNU9fNtZGxUK YacDTW/X5i4aga2dmplSbiHx2BdD21H1Tnw8nlgNf/v6+bfEnmK1avnKJPX0UtAvoPaU lNh0Vk/yfMhsGj8bfthhNSevV9HjX7L3pAuRoJYY1vfUN5Q5394onQ6db5cyEFhRp7oU BroIfepgFPBJCHOcbIN7AuzwS7kcnqVPwTR329YjEQRlXUqlP5U/kzW5Q4IgYBllUwn8 Bz+WgwvKTNLP1FdOyNHQosFFTLxQb74QDVIFwn/zxJFUdcIs7AUnsFoseB4MJX905oO7 ui1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=crCBb33r; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=6MQhgk3m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h7-20020a170902f54700b001a6dc4ab8f3si12580365plf.574.2023.04.24.11.17.51; Mon, 24 Apr 2023 11:18:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=crCBb33r; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=6MQhgk3m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232315AbjDXSNm (ORCPT + 99 others); Mon, 24 Apr 2023 14:13:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231332AbjDXSNf (ORCPT ); Mon, 24 Apr 2023 14:13:35 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABA004C03 for ; Mon, 24 Apr 2023 11:13:28 -0700 (PDT) From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1682360007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=5sjVCOTOKGYRbG6sN/eB48N/lr9EVJZJBYkiv7M5Xtk=; b=crCBb33rNTQwC2r/kIJQ//fkPQbfblrSkp+F+69SMbhp39firjtqFIkJdlhE0KTfabwQd4 0N84vilNU1P3scJrmtccbHAlR5O3ik/GRZLcclX0Ol9O+ac1DTOjfqBCBD/yqdkk50dIly +SisVGXk0MiHS2sLOc/K98Y6+dqs5ecc7P0jBY8bZhWG5sJMor9FX+4DglujcaDLb5Hkx6 ETRGBtHwGB3N1V14j7xU2OVNzDCSj+/oeHp9gQ+oifr1vq1MSXjtyVL8jT/85FK3JoRuS2 ExwIszlj08ni7auFv0SMU7sNEHaFq9/bS+ALBjKI3/DDDKUlaVZ0p+4IRYuf0A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1682360007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=5sjVCOTOKGYRbG6sN/eB48N/lr9EVJZJBYkiv7M5Xtk=; b=6MQhgk3mRc2sr+JJRXGfLNksi2WYIs0nUwH9yDndM520W+O+CUWvkza3g2XkK/s5i2+elO 6tATPS+L+glmhKBQ== To: Linus Torvalds Cc: linux-kernel@vger.kernel.org, x86@kernel.org Subject: [GIT pull] irq/core for 6.4-rc1 References: <168235968801.840202.17752066425816055574.tglx@xen13> Message-ID: <168235969096.840202.4914424488387974308.tglx@xen13> MIME-Version: 1.0 Date: Mon, 24 Apr 2023 20:13:26 +0200 (CEST) X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764082616898392194?= X-GMAIL-MSGID: =?utf-8?q?1764082616898392194?= Linus, please pull the latest irq/core branch from: git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git irq-core-2023-04-24 up to: f37202aa6e5d: Merge tag 'irqchip-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms into irq/core Interrupt core and drivers updates: - Core: - Add tracepoints for tasklet callbacks which makes it possible to analyze individual tasklet functions instead of guess working from the overall duration of tasklet processing - Ensure that secondary interrupt threads have their affinity adjusted correctly. - Drivers: - A large rework of the RISC-V IPI management to prepare for a new RISC-V interrupt architecture - Small fixes and enhancements all over the place - Removal of support for various obsolete hardware platforms and the related code Thanks, tglx ------------------> Alain Volmat (1): irqchip/st: Remove stih415/stih416 and stid127 platforms support Anup Patel (7): RISC-V: Clear SIP bit only when using SBI IPI operations irqchip/riscv-intc: Allow drivers to directly discover INTC hwnode RISC-V: Treat IPIs as normal Linux IRQs RISC-V: Allow marking IPIs as suitable for remote FENCEs RISC-V: Use IPIs for remote TLB flush when possible RISC-V: Use IPIs for remote icache flush when possible irqchip/riscv-intc: Add empty irq_eoi() for chained irq handlers Jianmin Lv (5): irqchip/loongson-eiointc: Fix returned value on parsing MADT irqchip/loongson-eiointc: Fix incorrect use of acpi_get_vec_parent irqchip/loongson-eiointc: Fix registration of syscore_ops irqchip/loongson-pch-pic: Fix registration of syscore_ops irqchip/loongson-pch-pic: Fix pch_pic_acpi_init calling John Keeping (1): genirq: Update affinity of secondary threads Lingutla Chandrasekhar (1): softirq: Add trace points for tasklet entry/exit Marc Zyngier (1): irqchip/gic: Drop support for board files Mason Huo (1): irqchip/irq-sifive-plic: Add syscore callbacks for hibernation Rob Herring (1): irqchip: Use of_property_read_bool() for boolean properties Sebastian Reichel (1): irqchip/gic-v3: Add Rockchip 3588001 erratum workaround Shanker Donthineni (1): irqchip/gicv3: Workaround for NVIDIA erratum T241-FABRIC-4 Álvaro Fernández Rojas (1): irqchip/bcm-6345-l1: Request memory region Documentation/arm64/silicon-errata.rst | 5 + arch/arm64/Kconfig | 10 ++ arch/riscv/Kconfig | 2 + arch/riscv/include/asm/irq.h | 4 + arch/riscv/include/asm/sbi.h | 9 +- arch/riscv/include/asm/smp.h | 49 +++++++--- arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/cpu-hotplug.c | 3 +- arch/riscv/kernel/irq.c | 21 +++- arch/riscv/kernel/sbi-ipi.c | 77 +++++++++++++++ arch/riscv/kernel/sbi.c | 100 +++---------------- arch/riscv/kernel/smp.c | 171 +++++++++++++++++---------------- arch/riscv/kernel/smpboot.c | 5 +- arch/riscv/mm/cacheflush.c | 5 +- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++--- drivers/clocksource/timer-clint.c | 65 ++++++++++--- drivers/firmware/smccc/smccc.c | 26 +++++ drivers/firmware/smccc/soc_id.c | 28 +----- drivers/irqchip/Kconfig | 3 + drivers/irqchip/irq-bcm6345-l1.c | 6 +- drivers/irqchip/irq-csky-apb-intc.c | 2 +- drivers/irqchip/irq-gic-v2m.c | 2 +- drivers/irqchip/irq-gic-v3-its.c | 35 +++++++ drivers/irqchip/irq-gic-v3.c | 115 +++++++++++++++++++--- drivers/irqchip/irq-gic.c | 60 +----------- drivers/irqchip/irq-loongson-eiointc.c | 32 ++++-- drivers/irqchip/irq-loongson-pch-pic.c | 6 +- drivers/irqchip/irq-riscv-intc.c | 71 ++++++++------ drivers/irqchip/irq-sifive-plic.c | 93 +++++++++++++++++- drivers/irqchip/irq-st.c | 15 --- include/linux/arm-smccc.h | 18 ++++ include/linux/irqchip/arm-gic.h | 6 -- include/trace/events/irq.h | 47 +++++++++ kernel/irq/manage.c | 5 +- kernel/softirq.c | 9 +- 35 files changed, 819 insertions(+), 380 deletions(-) create mode 100644 arch/riscv/kernel/sbi-ipi.c diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst index ec5f889d7681..9e311bc43e05 100644 --- a/Documentation/arm64/silicon-errata.rst +++ b/Documentation/arm64/silicon-errata.rst @@ -172,6 +172,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+ @@ -205,6 +207,9 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | Qualcomm Tech. | Kryo4xx Gold | N/A | ARM64_ERRATUM_1286807 | +----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ +| Rockchip | RK3588 | #3588001 | ROCKCHIP_ERRATUM_3588001 | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Fujitsu | A64FX | E#010001 | FUJITSU_ERRATUM_010001 | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1023e896d46b..02789215d251 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1150,6 +1150,16 @@ config NVIDIA_CARMEL_CNP_ERRATUM If unsure, say Y. +config ROCKCHIP_ERRATUM_3588001 + bool "Rockchip 3588001: GIC600 can not support shareability attributes" + default y + help + The Rockchip RK3588 GIC600 SoC integration does not support ACE/ACE-lite. + This means, that its sharability feature may not be used, even though it + is supported by the IP itself. + + If unsure, say Y. + config SOCIONEXT_SYNQUACER_PREITS bool "Socionext Synquacer: Workaround for GICv3 pre-ITS" default y diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index eb7f29a412f8..139055bcfcae 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -63,6 +63,8 @@ config RISCV select GENERIC_GETTIMEOFDAY if HAVE_GENERIC_VDSO select GENERIC_IDLE_POLL_SETUP select GENERIC_IOREMAP if MMU + select GENERIC_IRQ_IPI if SMP + select GENERIC_IRQ_IPI_MUX if SMP select GENERIC_IRQ_MULTI_HANDLER select GENERIC_IRQ_SHOW select GENERIC_IRQ_SHOW_LEVEL diff --git a/arch/riscv/include/asm/irq.h b/arch/riscv/include/asm/irq.h index e4c435509983..43b9ebfbd943 100644 --- a/arch/riscv/include/asm/irq.h +++ b/arch/riscv/include/asm/irq.h @@ -12,6 +12,10 @@ #include +void riscv_set_intc_hwnode_fn(struct fwnode_handle *(*fn)(void)); + +struct fwnode_handle *riscv_get_intc_hwnode(void); + extern void __init init_IRQ(void); #endif /* _ASM_RISCV_IRQ_H */ diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 945b7be249c1..acab4410ef2a 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -271,8 +271,7 @@ long sbi_get_marchid(void); long sbi_get_mimpid(void); void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); -void sbi_clear_ipi(void); -int sbi_send_ipi(const struct cpumask *cpu_mask); +void sbi_send_ipi(unsigned int cpu); int sbi_remote_fence_i(const struct cpumask *cpu_mask); int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, unsigned long start, @@ -335,4 +334,10 @@ unsigned long riscv_cached_mvendorid(unsigned int cpu_id); unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); +#if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +void sbi_ipi_init(void); +#else +static inline void sbi_ipi_init(void) { } +#endif + #endif /* _ASM_RISCV_SBI_H */ diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 3831b638ecab..c4b77017ec58 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -15,12 +15,10 @@ struct seq_file; extern unsigned long boot_cpu_hartid; -struct riscv_ipi_ops { - void (*ipi_inject)(const struct cpumask *target); - void (*ipi_clear)(void); -}; - #ifdef CONFIG_SMP + +#include + /* * Mapping between linux logical cpu index and hartid. */ @@ -33,9 +31,6 @@ void show_ipi_stats(struct seq_file *p, int prec); /* SMP initialization hook for setup_arch */ void __init setup_smp(void); -/* Called from C code, this handles an IPI. */ -void handle_IPI(struct pt_regs *regs); - /* Hook for the generic smp_call_function_many() routine. */ void arch_send_call_function_ipi_mask(struct cpumask *mask); @@ -44,11 +39,22 @@ void arch_send_call_function_single_ipi(int cpu); int riscv_hartid_to_cpuid(unsigned long hartid); -/* Set custom IPI operations */ -void riscv_set_ipi_ops(const struct riscv_ipi_ops *ops); +/* Enable IPI for CPU hotplug */ +void riscv_ipi_enable(void); + +/* Disable IPI for CPU hotplug */ +void riscv_ipi_disable(void); -/* Clear IPI for current CPU */ -void riscv_clear_ipi(void); +/* Check if IPI interrupt numbers are available */ +bool riscv_ipi_have_virq_range(void); + +/* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ +void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); + +/* Check if we can use IPIs for remote FENCEs */ +DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); +#define riscv_use_ipi_for_rfence() \ + static_branch_unlikely(&riscv_ipi_for_rfence) /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -85,14 +91,29 @@ static inline unsigned long cpuid_to_hartid_map(int cpu) return boot_cpu_hartid; } -static inline void riscv_set_ipi_ops(const struct riscv_ipi_ops *ops) +static inline void riscv_ipi_enable(void) +{ +} + +static inline void riscv_ipi_disable(void) { } -static inline void riscv_clear_ipi(void) +static inline bool riscv_ipi_have_virq_range(void) +{ + return false; +} + +static inline void riscv_ipi_set_virq_range(int virq, int nr, + bool use_for_rfence) { } +static inline bool riscv_use_ipi_for_rfence(void) +{ + return false; +} + #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 4cf303a779ab..67f542be1bea 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -74,6 +74,7 @@ obj-$(CONFIG_PERF_EVENTS) += perf_callchain.o obj-$(CONFIG_HAVE_PERF_REGS) += perf_regs.o obj-$(CONFIG_RISCV_SBI) += sbi.o ifeq ($(CONFIG_RISCV_SBI), y) +obj-$(CONFIG_SMP) += sbi-ipi.o obj-$(CONFIG_SMP) += cpu_ops_sbi.o endif obj-$(CONFIG_HOTPLUG_CPU) += cpu-hotplug.o diff --git a/arch/riscv/kernel/cpu-hotplug.c b/arch/riscv/kernel/cpu-hotplug.c index f7a832e3a1d1..39235cf50652 100644 --- a/arch/riscv/kernel/cpu-hotplug.c +++ b/arch/riscv/kernel/cpu-hotplug.c @@ -13,7 +13,7 @@ #include #include #include -#include +#include bool cpu_has_hotplug(unsigned int cpu) { @@ -43,6 +43,7 @@ int __cpu_disable(void) remove_cpu_topology(cpu); numa_remove_cpu(cpu); set_cpu_online(cpu, false); + riscv_ipi_disable(); irq_migrate_all_off_this_cpu(); return ret; diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 7207fa08d78f..eb9a68a539e6 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -7,8 +7,26 @@ #include #include +#include +#include #include -#include +#include + +static struct fwnode_handle *(*__get_intc_node)(void); + +void riscv_set_intc_hwnode_fn(struct fwnode_handle *(*fn)(void)) +{ + __get_intc_node = fn; +} + +struct fwnode_handle *riscv_get_intc_hwnode(void) +{ + if (__get_intc_node) + return __get_intc_node(); + + return NULL; +} +EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); int arch_show_interrupts(struct seq_file *p, int prec) { @@ -21,4 +39,5 @@ void __init init_IRQ(void) irqchip_init(); if (!handle_arch_irq) panic("No interrupt controller found."); + sbi_ipi_init(); } diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c new file mode 100644 index 000000000000..a4559695ce62 --- /dev/null +++ b/arch/riscv/kernel/sbi-ipi.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Multiplex several IPIs over a single HW IPI. + * + * Copyright (c) 2022 Ventana Micro Systems Inc. + */ + +#define pr_fmt(fmt) "riscv: " fmt +#include +#include +#include +#include +#include +#include + +static int sbi_ipi_virq; + +static void sbi_ipi_handle(struct irq_desc *desc) +{ + struct irq_chip *chip = irq_desc_get_chip(desc); + + chained_irq_enter(chip, desc); + + csr_clear(CSR_IP, IE_SIE); + ipi_mux_process(); + + chained_irq_exit(chip, desc); +} + +static int sbi_ipi_starting_cpu(unsigned int cpu) +{ + enable_percpu_irq(sbi_ipi_virq, irq_get_trigger_type(sbi_ipi_virq)); + return 0; +} + +void __init sbi_ipi_init(void) +{ + int virq; + struct irq_domain *domain; + + if (riscv_ipi_have_virq_range()) + return; + + domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(), + DOMAIN_BUS_ANY); + if (!domain) { + pr_err("unable to find INTC IRQ domain\n"); + return; + } + + sbi_ipi_virq = irq_create_mapping(domain, RV_IRQ_SOFT); + if (!sbi_ipi_virq) { + pr_err("unable to create INTC IRQ mapping\n"); + return; + } + + virq = ipi_mux_create(BITS_PER_BYTE, sbi_send_ipi); + if (virq <= 0) { + pr_err("unable to create muxed IPIs\n"); + irq_dispose_mapping(sbi_ipi_virq); + return; + } + + irq_set_chained_handler(sbi_ipi_virq, sbi_ipi_handle); + + /* + * Don't disable IPI when CPU goes offline because + * the masking/unmasking of virtual IPIs is done + * via generic IPI-Mux + */ + cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, + "irqchip/sbi-ipi:starting", + sbi_ipi_starting_cpu, NULL); + + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + pr_info("providing IPIs using SBI IPI extension\n"); +} diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index 5c87db8fdff2..92b9b759ab3d 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -17,7 +17,7 @@ unsigned long sbi_spec_version __ro_after_init = SBI_SPEC_VERSION_DEFAULT; EXPORT_SYMBOL(sbi_spec_version); static void (*__sbi_set_timer)(uint64_t stime) __ro_after_init; -static int (*__sbi_send_ipi)(const struct cpumask *cpu_mask) __ro_after_init; +static void (*__sbi_send_ipi)(unsigned int cpu) __ro_after_init; static int (*__sbi_rfence)(int fid, const struct cpumask *cpu_mask, unsigned long start, unsigned long size, unsigned long arg4, unsigned long arg5) __ro_after_init; @@ -130,17 +130,6 @@ void sbi_shutdown(void) } EXPORT_SYMBOL(sbi_shutdown); -/** - * sbi_clear_ipi() - Clear any pending IPIs for the calling hart. - * - * Return: None - */ -void sbi_clear_ipi(void) -{ - sbi_ecall(SBI_EXT_0_1_CLEAR_IPI, 0, 0, 0, 0, 0, 0, 0); -} -EXPORT_SYMBOL(sbi_clear_ipi); - /** * __sbi_set_timer_v01() - Program the timer for next timer event. * @stime_value: The value after which next timer event should fire. @@ -157,17 +146,12 @@ static void __sbi_set_timer_v01(uint64_t stime_value) #endif } -static int __sbi_send_ipi_v01(const struct cpumask *cpu_mask) +static void __sbi_send_ipi_v01(unsigned int cpu) { - unsigned long hart_mask; - - if (!cpu_mask || cpumask_empty(cpu_mask)) - cpu_mask = cpu_online_mask; - hart_mask = __sbi_v01_cpumask_to_hartmask(cpu_mask); - + unsigned long hart_mask = + __sbi_v01_cpumask_to_hartmask(cpumask_of(cpu)); sbi_ecall(SBI_EXT_0_1_SEND_IPI, 0, (unsigned long)(&hart_mask), 0, 0, 0, 0, 0); - return 0; } static int __sbi_rfence_v01(int fid, const struct cpumask *cpu_mask, @@ -216,12 +200,10 @@ static void __sbi_set_timer_v01(uint64_t stime_value) sbi_major_version(), sbi_minor_version()); } -static int __sbi_send_ipi_v01(const struct cpumask *cpu_mask) +static void __sbi_send_ipi_v01(unsigned int cpu) { pr_warn("IPI extension is not available in SBI v%lu.%lu\n", sbi_major_version(), sbi_minor_version()); - - return 0; } static int __sbi_rfence_v01(int fid, const struct cpumask *cpu_mask, @@ -248,55 +230,18 @@ static void __sbi_set_timer_v02(uint64_t stime_value) #endif } -static int __sbi_send_ipi_v02(const struct cpumask *cpu_mask) +static void __sbi_send_ipi_v02(unsigned int cpu) { - unsigned long hartid, cpuid, hmask = 0, hbase = 0, htop = 0; - struct sbiret ret = {0}; int result; + struct sbiret ret = {0}; - if (!cpu_mask || cpumask_empty(cpu_mask)) - cpu_mask = cpu_online_mask; - - for_each_cpu(cpuid, cpu_mask) { - hartid = cpuid_to_hartid_map(cpuid); - if (hmask) { - if (hartid + BITS_PER_LONG <= htop || - hbase + BITS_PER_LONG <= hartid) { - ret = sbi_ecall(SBI_EXT_IPI, - SBI_EXT_IPI_SEND_IPI, hmask, - hbase, 0, 0, 0, 0); - if (ret.error) - goto ecall_failed; - hmask = 0; - } else if (hartid < hbase) { - /* shift the mask to fit lower hartid */ - hmask <<= hbase - hartid; - hbase = hartid; - } - } - if (!hmask) { - hbase = hartid; - htop = hartid; - } else if (hartid > htop) { - htop = hartid; - } - hmask |= BIT(hartid - hbase); - } - - if (hmask) { - ret = sbi_ecall(SBI_EXT_IPI, SBI_EXT_IPI_SEND_IPI, - hmask, hbase, 0, 0, 0, 0); - if (ret.error) - goto ecall_failed; + ret = sbi_ecall(SBI_EXT_IPI, SBI_EXT_IPI_SEND_IPI, + 1UL, cpuid_to_hartid_map(cpu), 0, 0, 0, 0); + if (ret.error) { + result = sbi_err_map_linux_errno(ret.error); + pr_err("%s: hbase = [%lu] failed (error [%d])\n", + __func__, cpuid_to_hartid_map(cpu), result); } - - return 0; - -ecall_failed: - result = sbi_err_map_linux_errno(ret.error); - pr_err("%s: hbase = [%lu] hmask = [0x%lx] failed (error [%d])\n", - __func__, hbase, hmask, result); - return result; } static int __sbi_rfence_v02_call(unsigned long fid, unsigned long hmask, @@ -410,13 +355,11 @@ void sbi_set_timer(uint64_t stime_value) /** * sbi_send_ipi() - Send an IPI to any hart. - * @cpu_mask: A cpu mask containing all the target harts. - * - * Return: 0 on success, appropriate linux error code otherwise. + * @cpu: Logical id of the target CPU. */ -int sbi_send_ipi(const struct cpumask *cpu_mask) +void sbi_send_ipi(unsigned int cpu) { - return __sbi_send_ipi(cpu_mask); + __sbi_send_ipi(cpu); } EXPORT_SYMBOL(sbi_send_ipi); @@ -641,15 +584,6 @@ long sbi_get_mimpid(void) } EXPORT_SYMBOL_GPL(sbi_get_mimpid); -static void sbi_send_cpumask_ipi(const struct cpumask *target) -{ - sbi_send_ipi(target); -} - -static const struct riscv_ipi_ops sbi_ipi_ops = { - .ipi_inject = sbi_send_cpumask_ipi -}; - void __init sbi_init(void) { int ret; @@ -696,6 +630,4 @@ void __init sbi_init(void) __sbi_send_ipi = __sbi_send_ipi_v01; __sbi_rfence = __sbi_rfence_v01; } - - riscv_set_ipi_ops(&sbi_ipi_ops); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 8c3b59f1f9b8..5f985a197eff 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -13,14 +13,15 @@ #include #include #include +#include #include #include #include #include #include +#include #include -#include #include #include #include @@ -44,11 +45,10 @@ void __init smp_setup_processor_id(void) cpuid_to_hartid_map(0) = boot_cpu_hartid; } -/* A collection of single bit ipi messages. */ -static struct { - unsigned long stats[IPI_MAX] ____cacheline_aligned; - unsigned long bits ____cacheline_aligned; -} ipi_data[NR_CPUS] __cacheline_aligned; +static DEFINE_PER_CPU_READ_MOSTLY(int, ipi_dummy_dev); +static int ipi_virq_base __ro_after_init; +static int nr_ipi __ro_after_init = IPI_MAX; +static struct irq_desc *ipi_desc[IPI_MAX] __read_mostly; int riscv_hartid_to_cpuid(unsigned long hartid) { @@ -100,48 +100,14 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs) } #endif -static const struct riscv_ipi_ops *ipi_ops __ro_after_init; - -void riscv_set_ipi_ops(const struct riscv_ipi_ops *ops) -{ - ipi_ops = ops; -} -EXPORT_SYMBOL_GPL(riscv_set_ipi_ops); - -void riscv_clear_ipi(void) -{ - if (ipi_ops && ipi_ops->ipi_clear) - ipi_ops->ipi_clear(); - - csr_clear(CSR_IP, IE_SIE); -} -EXPORT_SYMBOL_GPL(riscv_clear_ipi); - static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op) { - int cpu; - - smp_mb__before_atomic(); - for_each_cpu(cpu, mask) - set_bit(op, &ipi_data[cpu].bits); - smp_mb__after_atomic(); - - if (ipi_ops && ipi_ops->ipi_inject) - ipi_ops->ipi_inject(mask); - else - pr_warn("SMP: IPI inject method not available\n"); + __ipi_send_mask(ipi_desc[op], mask); } static void send_ipi_single(int cpu, enum ipi_message_type op) { - smp_mb__before_atomic(); - set_bit(op, &ipi_data[cpu].bits); - smp_mb__after_atomic(); - - if (ipi_ops && ipi_ops->ipi_inject) - ipi_ops->ipi_inject(cpumask_of(cpu)); - else - pr_warn("SMP: IPI inject method not available\n"); + __ipi_send_mask(ipi_desc[op], cpumask_of(cpu)); } #ifdef CONFIG_IRQ_WORK @@ -151,59 +117,98 @@ void arch_irq_work_raise(void) } #endif -void handle_IPI(struct pt_regs *regs) +static irqreturn_t handle_IPI(int irq, void *data) { - unsigned int cpu = smp_processor_id(); - unsigned long *pending_ipis = &ipi_data[cpu].bits; - unsigned long *stats = ipi_data[cpu].stats; + int ipi = irq - ipi_virq_base; + + switch (ipi) { + case IPI_RESCHEDULE: + scheduler_ipi(); + break; + case IPI_CALL_FUNC: + generic_smp_call_function_interrupt(); + break; + case IPI_CPU_STOP: + ipi_stop(); + break; + case IPI_CPU_CRASH_STOP: + ipi_cpu_crash_stop(smp_processor_id(), get_irq_regs()); + break; + case IPI_IRQ_WORK: + irq_work_run(); + break; +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST + case IPI_TIMER: + tick_receive_broadcast(); + break; +#endif + default: + pr_warn("CPU%d: unhandled IPI%d\n", smp_processor_id(), ipi); + break; + } - riscv_clear_ipi(); + return IRQ_HANDLED; +} - while (true) { - unsigned long ops; +void riscv_ipi_enable(void) +{ + int i; - /* Order bit clearing and data access. */ - mb(); + if (WARN_ON_ONCE(!ipi_virq_base)) + return; - ops = xchg(pending_ipis, 0); - if (ops == 0) - return; + for (i = 0; i < nr_ipi; i++) + enable_percpu_irq(ipi_virq_base + i, 0); +} - if (ops & (1 << IPI_RESCHEDULE)) { - stats[IPI_RESCHEDULE]++; - scheduler_ipi(); - } +void riscv_ipi_disable(void) +{ + int i; - if (ops & (1 << IPI_CALL_FUNC)) { - stats[IPI_CALL_FUNC]++; - generic_smp_call_function_interrupt(); - } + if (WARN_ON_ONCE(!ipi_virq_base)) + return; - if (ops & (1 << IPI_CPU_STOP)) { - stats[IPI_CPU_STOP]++; - ipi_stop(); - } + for (i = 0; i < nr_ipi; i++) + disable_percpu_irq(ipi_virq_base + i); +} - if (ops & (1 << IPI_CPU_CRASH_STOP)) { - ipi_cpu_crash_stop(cpu, get_irq_regs()); - } +bool riscv_ipi_have_virq_range(void) +{ + return (ipi_virq_base) ? true : false; +} - if (ops & (1 << IPI_IRQ_WORK)) { - stats[IPI_IRQ_WORK]++; - irq_work_run(); - } +DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); -#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST - if (ops & (1 << IPI_TIMER)) { - stats[IPI_TIMER]++; - tick_receive_broadcast(); - } -#endif - BUG_ON((ops >> IPI_MAX) != 0); +void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +{ + int i, err; + + if (WARN_ON(ipi_virq_base)) + return; + + WARN_ON(nr < IPI_MAX); + nr_ipi = min(nr, IPI_MAX); + ipi_virq_base = virq; - /* Order data access and bit testing. */ - mb(); + /* Request IPIs */ + for (i = 0; i < nr_ipi; i++) { + err = request_percpu_irq(ipi_virq_base + i, handle_IPI, + "IPI", &ipi_dummy_dev); + WARN_ON(err); + + ipi_desc[i] = irq_to_desc(ipi_virq_base + i); + irq_set_status_flags(ipi_virq_base + i, IRQ_HIDDEN); } + + /* Enabled IPIs for boot CPU immediately */ + riscv_ipi_enable(); + + /* Update RFENCE static key */ + if (use_for_rfence) + static_branch_enable(&riscv_ipi_for_rfence); + else + static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { @@ -223,7 +228,7 @@ void show_ipi_stats(struct seq_file *p, int prec) seq_printf(p, "%*s%u:%s", prec - 1, "IPI", i, prec >= 4 ? " " : ""); for_each_online_cpu(cpu) - seq_printf(p, "%10lu ", ipi_data[cpu].stats[i]); + seq_printf(p, "%10u ", irq_desc_kstat_cpu(ipi_desc[i], cpu)); seq_printf(p, " %s\n", ipi_names[i]); } } diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index ddb2afba6d25..00b53913d4c6 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -30,7 +30,6 @@ #include #include #include -#include #include #include "head.h" @@ -158,12 +157,12 @@ asmlinkage __visible void smp_callin(void) struct mm_struct *mm = &init_mm; unsigned int curr_cpuid = smp_processor_id(); - riscv_clear_ipi(); - /* All kernel threads share the same mm context. */ mmgrab(mm); current->active_mm = mm; + riscv_ipi_enable(); + store_cpu_topology(curr_cpuid); notify_cpu_starting(curr_cpuid); numa_add_cpu(curr_cpuid); diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index fcd6145fbead..20cec5e7cdbf 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -19,7 +19,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI)) + if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -67,7 +67,8 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI)) { + } else if (IS_ENABLED(CONFIG_RISCV_SBI) && + !riscv_use_ipi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index ef701fa83f36..77be59aadc73 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id) & asid_mask; if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 6cfe2ab73eb0..9a55e733ae99 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -17,6 +17,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -31,6 +34,7 @@ /* CLINT manages IPI and Timer for RISC-V M-mode */ static u32 __iomem *clint_ipi_base; +static unsigned int clint_ipi_irq; static u64 __iomem *clint_timer_cmp; static u64 __iomem *clint_timer_val; static unsigned long clint_timer_freq; @@ -41,12 +45,10 @@ u64 __iomem *clint_time_val; EXPORT_SYMBOL(clint_time_val); #endif -static void clint_send_ipi(const struct cpumask *target) +#ifdef CONFIG_SMP +static void clint_send_ipi(unsigned int cpu) { - unsigned int cpu; - - for_each_cpu(cpu, target) - writel(1, clint_ipi_base + cpuid_to_hartid_map(cpu)); + writel(1, clint_ipi_base + cpuid_to_hartid_map(cpu)); } static void clint_clear_ipi(void) @@ -54,10 +56,18 @@ static void clint_clear_ipi(void) writel(0, clint_ipi_base + cpuid_to_hartid_map(smp_processor_id())); } -static struct riscv_ipi_ops clint_ipi_ops = { - .ipi_inject = clint_send_ipi, - .ipi_clear = clint_clear_ipi, -}; +static void clint_ipi_interrupt(struct irq_desc *desc) +{ + struct irq_chip *chip = irq_desc_get_chip(desc); + + chained_irq_enter(chip, desc); + + clint_clear_ipi(); + ipi_mux_process(); + + chained_irq_exit(chip, desc); +} +#endif #ifdef CONFIG_64BIT #define clint_get_cycles() readq_relaxed(clint_timer_val) @@ -125,12 +135,19 @@ static int clint_timer_starting_cpu(unsigned int cpu) enable_percpu_irq(clint_timer_irq, irq_get_trigger_type(clint_timer_irq)); + enable_percpu_irq(clint_ipi_irq, + irq_get_trigger_type(clint_ipi_irq)); return 0; } static int clint_timer_dying_cpu(unsigned int cpu) { disable_percpu_irq(clint_timer_irq); + /* + * Don't disable IPI when CPU goes offline because + * the masking/unmasking of virtual IPIs is done + * via generic IPI-Mux + */ return 0; } @@ -170,6 +187,12 @@ static int __init clint_timer_init_dt(struct device_node *np) return -ENODEV; } + /* Find parent irq domain and map ipi irq */ + if (!clint_ipi_irq && + oirq.args[0] == RV_IRQ_SOFT && + irq_find_host(oirq.np)) + clint_ipi_irq = irq_of_parse_and_map(np, i); + /* Find parent irq domain and map timer irq */ if (!clint_timer_irq && oirq.args[0] == RV_IRQ_TIMER && @@ -177,9 +200,9 @@ static int __init clint_timer_init_dt(struct device_node *np) clint_timer_irq = irq_of_parse_and_map(np, i); } - /* If CLINT timer irq not found then fail */ - if (!clint_timer_irq) { - pr_err("%pOFP: timer irq not found\n", np); + /* If CLINT ipi or timer irq not found then fail */ + if (!clint_ipi_irq || !clint_timer_irq) { + pr_err("%pOFP: ipi/timer irq not found\n", np); return -ENODEV; } @@ -219,6 +242,19 @@ static int __init clint_timer_init_dt(struct device_node *np) goto fail_iounmap; } +#ifdef CONFIG_SMP + rc = ipi_mux_create(BITS_PER_BYTE, clint_send_ipi); + if (rc <= 0) { + pr_err("unable to create muxed IPIs\n"); + rc = (rc < 0) ? rc : -ENODEV; + goto fail_free_irq; + } + + irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + clint_clear_ipi(); +#endif + rc = cpuhp_setup_state(CPUHP_AP_CLINT_TIMER_STARTING, "clockevents/clint/timer:starting", clint_timer_starting_cpu, @@ -228,13 +264,10 @@ static int __init clint_timer_init_dt(struct device_node *np) goto fail_free_irq; } - riscv_set_ipi_ops(&clint_ipi_ops); - clint_clear_ipi(); - return 0; fail_free_irq: - free_irq(clint_timer_irq, &clint_clock_event); + free_percpu_irq(clint_timer_irq, &clint_clock_event); fail_iounmap: iounmap(base); return rc; diff --git a/drivers/firmware/smccc/smccc.c b/drivers/firmware/smccc/smccc.c index 60ccf3e90d7d..db818f9dcb8e 100644 --- a/drivers/firmware/smccc/smccc.c +++ b/drivers/firmware/smccc/smccc.c @@ -17,9 +17,13 @@ static enum arm_smccc_conduit smccc_conduit = SMCCC_CONDUIT_NONE; bool __ro_after_init smccc_trng_available = false; u64 __ro_after_init smccc_has_sve_hint = false; +s32 __ro_after_init smccc_soc_id_version = SMCCC_RET_NOT_SUPPORTED; +s32 __ro_after_init smccc_soc_id_revision = SMCCC_RET_NOT_SUPPORTED; void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit) { + struct arm_smccc_res res; + smccc_version = version; smccc_conduit = conduit; @@ -27,6 +31,18 @@ void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit) if (IS_ENABLED(CONFIG_ARM64_SVE) && smccc_version >= ARM_SMCCC_VERSION_1_3) smccc_has_sve_hint = true; + + if ((smccc_version >= ARM_SMCCC_VERSION_1_2) && + (smccc_conduit != SMCCC_CONDUIT_NONE)) { + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_ARCH_SOC_ID, &res); + if ((s32)res.a0 >= 0) { + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_SOC_ID, 0, &res); + smccc_soc_id_version = (s32)res.a0; + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_SOC_ID, 1, &res); + smccc_soc_id_revision = (s32)res.a0; + } + } } enum arm_smccc_conduit arm_smccc_1_1_get_conduit(void) @@ -44,6 +60,16 @@ u32 arm_smccc_get_version(void) } EXPORT_SYMBOL_GPL(arm_smccc_get_version); +s32 arm_smccc_get_soc_id_version(void) +{ + return smccc_soc_id_version; +} + +s32 arm_smccc_get_soc_id_revision(void) +{ + return smccc_soc_id_revision; +} + static int __init smccc_devices_init(void) { struct platform_device *pdev; diff --git a/drivers/firmware/smccc/soc_id.c b/drivers/firmware/smccc/soc_id.c index dd7c3d5e8b0b..890eb454599a 100644 --- a/drivers/firmware/smccc/soc_id.c +++ b/drivers/firmware/smccc/soc_id.c @@ -42,41 +42,23 @@ static int __init smccc_soc_init(void) if (arm_smccc_get_version() < ARM_SMCCC_VERSION_1_2) return 0; - if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) { - pr_err("%s: invalid SMCCC conduit\n", __func__); - return -EOPNOTSUPP; - } - - arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, - ARM_SMCCC_ARCH_SOC_ID, &res); - - if ((int)res.a0 == SMCCC_RET_NOT_SUPPORTED) { + soc_id_version = arm_smccc_get_soc_id_version(); + if (soc_id_version == SMCCC_RET_NOT_SUPPORTED) { pr_info("ARCH_SOC_ID not implemented, skipping ....\n"); return 0; } - if ((int)res.a0 < 0) { - pr_info("ARCH_FEATURES(ARCH_SOC_ID) returned error: %lx\n", - res.a0); - return -EINVAL; - } - - arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_SOC_ID, 0, &res); - if ((int)res.a0 < 0) { + if (soc_id_version < 0) { pr_err("ARCH_SOC_ID(0) returned error: %lx\n", res.a0); return -EINVAL; } - soc_id_version = res.a0; - - arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_SOC_ID, 1, &res); - if ((int)res.a0 < 0) { + soc_id_rev = arm_smccc_get_soc_id_revision(); + if (soc_id_rev < 0) { pr_err("ARCH_SOC_ID(1) returned error: %lx\n", res.a0); return -EINVAL; } - soc_id_rev = res.a0; - soc_dev_attr = kzalloc(sizeof(*soc_dev_attr), GFP_KERNEL); if (!soc_dev_attr) return -ENOMEM; diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig index 7dc990eb2c9b..09e422da482f 100644 --- a/drivers/irqchip/Kconfig +++ b/drivers/irqchip/Kconfig @@ -7,6 +7,7 @@ config IRQCHIP config ARM_GIC bool + depends on OF select IRQ_DOMAIN_HIERARCHY select GENERIC_IRQ_EFFECTIVE_AFF_MASK if SMP @@ -35,6 +36,7 @@ config ARM_GIC_V3 select IRQ_DOMAIN_HIERARCHY select PARTITION_PERCPU select GENERIC_IRQ_EFFECTIVE_AFF_MASK if SMP + select HAVE_ARM_SMCCC_DISCOVERY config ARM_GIC_V3_ITS bool @@ -535,6 +537,7 @@ config TI_PRUSS_INTC config RISCV_INTC bool depends on RISCV + select IRQ_DOMAIN_HIERARCHY config SIFIVE_PLIC bool diff --git a/drivers/irqchip/irq-bcm6345-l1.c b/drivers/irqchip/irq-bcm6345-l1.c index 6899e37810a8..fa113cb2529a 100644 --- a/drivers/irqchip/irq-bcm6345-l1.c +++ b/drivers/irqchip/irq-bcm6345-l1.c @@ -257,6 +257,9 @@ static int __init bcm6345_l1_init_one(struct device_node *dn, if (!cpu->map_base) return -ENOMEM; + if (!request_mem_region(res.start, sz, res.name)) + pr_err("failed to request intc memory"); + for (i = 0; i < n_words; i++) { cpu->enable_cache[i] = 0; __raw_writel(0, cpu->map_base + reg_enable(intc, i)); @@ -335,8 +338,7 @@ static int __init bcm6345_l1_of_init(struct device_node *dn, for_each_cpu(idx, &intc->cpumask) { struct bcm6345_l1_cpu *cpu = intc->cpus[idx]; - pr_info(" CPU%u at MMIO 0x%p (irq = %d)\n", idx, - cpu->map_base, cpu->parent_irq); + pr_info(" CPU%u (irq = %d)\n", idx, cpu->parent_irq); } return 0; diff --git a/drivers/irqchip/irq-csky-apb-intc.c b/drivers/irqchip/irq-csky-apb-intc.c index 42d8a2438ebc..6710691e4c25 100644 --- a/drivers/irqchip/irq-csky-apb-intc.c +++ b/drivers/irqchip/irq-csky-apb-intc.c @@ -68,7 +68,7 @@ static void __init ck_set_gc(struct device_node *node, void __iomem *reg_base, gc->chip_types[0].chip.irq_mask = irq_gc_mask_clr_bit; gc->chip_types[0].chip.irq_unmask = irq_gc_mask_set_bit; - if (of_find_property(node, "csky,support-pulse-signal", NULL)) + if (of_property_read_bool(node, "csky,support-pulse-signal")) gc->chip_types[0].chip.irq_unmask = irq_ck_mask_set_bit; } diff --git a/drivers/irqchip/irq-gic-v2m.c b/drivers/irqchip/irq-gic-v2m.c index f1e75b35a52a..f2ff4387870d 100644 --- a/drivers/irqchip/irq-gic-v2m.c +++ b/drivers/irqchip/irq-gic-v2m.c @@ -421,7 +421,7 @@ static int __init gicv2m_of_init(struct fwnode_handle *parent_handle, u32 spi_start = 0, nr_spis = 0; struct resource res; - if (!of_find_property(child, "msi-controller", NULL)) + if (!of_property_read_bool(child, "msi-controller")) continue; ret = of_address_to_resource(child, 0, &res); diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 586271b8aa39..fa4641a5dfd8 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -42,9 +42,11 @@ #define ITS_FLAGS_CMDQ_NEEDS_FLUSHING (1ULL << 0) #define ITS_FLAGS_WORKAROUND_CAVIUM_22375 (1ULL << 1) #define ITS_FLAGS_WORKAROUND_CAVIUM_23144 (1ULL << 2) +#define ITS_FLAGS_FORCE_NON_SHAREABLE (1ULL << 3) #define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING (1 << 0) #define RDIST_FLAGS_RD_TABLES_PREALLOCATED (1 << 1) +#define RDIST_FLAGS_FORCE_NON_SHAREABLE (1 << 2) #define RD_LOCAL_LPI_ENABLED BIT(0) #define RD_LOCAL_PENDTABLE_PREALLOCATED BIT(1) @@ -2359,6 +2361,9 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser, its_write_baser(its, baser, val); tmp = baser->val; + if (its->flags & ITS_FLAGS_FORCE_NON_SHAREABLE) + tmp &= ~GITS_BASER_SHAREABILITY_MASK; + if ((val ^ tmp) & GITS_BASER_SHAREABILITY_MASK) { /* * Shareability didn't stick. Just use @@ -3096,6 +3101,9 @@ static void its_cpu_init_lpis(void) gicr_write_propbaser(val, rbase + GICR_PROPBASER); tmp = gicr_read_propbaser(rbase + GICR_PROPBASER); + if (gic_rdists->flags & RDIST_FLAGS_FORCE_NON_SHAREABLE) + tmp &= ~GICR_PROPBASER_SHAREABILITY_MASK; + if ((tmp ^ val) & GICR_PROPBASER_SHAREABILITY_MASK) { if (!(tmp & GICR_PROPBASER_SHAREABILITY_MASK)) { /* @@ -3120,6 +3128,9 @@ static void its_cpu_init_lpis(void) gicr_write_pendbaser(val, rbase + GICR_PENDBASER); tmp = gicr_read_pendbaser(rbase + GICR_PENDBASER); + if (gic_rdists->flags & RDIST_FLAGS_FORCE_NON_SHAREABLE) + tmp &= ~GICR_PENDBASER_SHAREABILITY_MASK; + if (!(tmp & GICR_PENDBASER_SHAREABILITY_MASK)) { /* * The HW reports non-shareable, we must remove the @@ -4710,6 +4721,19 @@ static bool __maybe_unused its_enable_quirk_hip07_161600802(void *data) return true; } +static bool __maybe_unused its_enable_rk3588001(void *data) +{ + struct its_node *its = data; + + if (!of_machine_is_compatible("rockchip,rk3588")) + return false; + + its->flags |= ITS_FLAGS_FORCE_NON_SHAREABLE; + gic_rdists->flags |= RDIST_FLAGS_FORCE_NON_SHAREABLE; + + return true; +} + static const struct gic_quirk its_quirks[] = { #ifdef CONFIG_CAVIUM_ERRATUM_22375 { @@ -4755,6 +4779,14 @@ static const struct gic_quirk its_quirks[] = { .mask = 0xffffffff, .init = its_enable_quirk_hip07_161600802, }, +#endif +#ifdef CONFIG_ROCKCHIP_ERRATUM_3588001 + { + .desc = "ITS: Rockchip erratum RK3588001", + .iidr = 0x0201743b, + .mask = 0xffffffff, + .init = its_enable_rk3588001, + }, #endif { } @@ -5096,6 +5128,9 @@ static int __init its_probe_one(struct resource *res, gits_write_cbaser(baser, its->base + GITS_CBASER); tmp = gits_read_cbaser(its->base + GITS_CBASER); + if (its->flags & ITS_FLAGS_FORCE_NON_SHAREABLE) + tmp &= ~GITS_CBASER_SHAREABILITY_MASK; + if ((tmp ^ baser) & GITS_CBASER_SHAREABILITY_MASK) { if (!(tmp & GITS_CBASER_SHAREABILITY_MASK)) { /* diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index fd134e1f481a..6fcee221f201 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -24,6 +24,9 @@ #include #include #include +#include +#include +#include #include #include @@ -47,6 +50,7 @@ struct redist_region { struct gic_chip_data { struct fwnode_handle *fwnode; + phys_addr_t dist_phys_base; void __iomem *dist_base; struct redist_region *redist_regions; struct rdists rdists; @@ -59,6 +63,10 @@ struct gic_chip_data { struct partition_desc **ppi_descs; }; +#define T241_CHIPS_MAX 4 +static void __iomem *t241_dist_base_alias[T241_CHIPS_MAX] __read_mostly; +static DEFINE_STATIC_KEY_FALSE(gic_nvidia_t241_erratum); + static struct gic_chip_data gic_data __read_mostly; static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key); @@ -179,6 +187,39 @@ static inline bool gic_irq_in_rdist(struct irq_data *d) } } +static inline void __iomem *gic_dist_base_alias(struct irq_data *d) +{ + if (static_branch_unlikely(&gic_nvidia_t241_erratum)) { + irq_hw_number_t hwirq = irqd_to_hwirq(d); + u32 chip; + + /* + * For the erratum T241-FABRIC-4, read accesses to GICD_In{E} + * registers are directed to the chip that owns the SPI. The + * the alias region can also be used for writes to the + * GICD_In{E} except GICD_ICENABLERn. Each chip has support + * for 320 {E}SPIs. Mappings for all 4 chips: + * Chip0 = 32-351 + * Chip1 = 352-671 + * Chip2 = 672-991 + * Chip3 = 4096-4415 + */ + switch (__get_intid_range(hwirq)) { + case SPI_RANGE: + chip = (hwirq - 32) / 320; + break; + case ESPI_RANGE: + chip = 3; + break; + default: + unreachable(); + } + return t241_dist_base_alias[chip]; + } + + return gic_data.dist_base; +} + static inline void __iomem *gic_dist_base(struct irq_data *d) { switch (get_intid_range(d)) { @@ -337,7 +378,7 @@ static int gic_peek_irq(struct irq_data *d, u32 offset) if (gic_irq_in_rdist(d)) base = gic_data_rdist_sgi_base(); else - base = gic_data.dist_base; + base = gic_dist_base_alias(d); return !!(readl_relaxed(base + offset + (index / 32) * 4) & mask); } @@ -588,7 +629,7 @@ static int gic_set_type(struct irq_data *d, unsigned int type) if (gic_irq_in_rdist(d)) base = gic_data_rdist_sgi_base(); else - base = gic_data.dist_base; + base = gic_dist_base_alias(d); offset = convert_offset_index(d, GICD_ICFGR, &index); @@ -1708,6 +1749,43 @@ static bool gic_enable_quirk_hip06_07(void *data) return false; } +#define T241_CHIPN_MASK GENMASK_ULL(45, 44) +#define T241_CHIP_GICDA_OFFSET 0x1580000 +#define SMCCC_SOC_ID_T241 0x036b0241 + +static bool gic_enable_quirk_nvidia_t241(void *data) +{ + s32 soc_id = arm_smccc_get_soc_id_version(); + unsigned long chip_bmask = 0; + phys_addr_t phys; + u32 i; + + /* Check JEP106 code for NVIDIA T241 chip (036b:0241) */ + if ((soc_id < 0) || (soc_id != SMCCC_SOC_ID_T241)) + return false; + + /* Find the chips based on GICR regions PHYS addr */ + for (i = 0; i < gic_data.nr_redist_regions; i++) { + chip_bmask |= BIT(FIELD_GET(T241_CHIPN_MASK, + (u64)gic_data.redist_regions[i].phys_base)); + } + + if (hweight32(chip_bmask) < 3) + return false; + + /* Setup GICD alias regions */ + for (i = 0; i < ARRAY_SIZE(t241_dist_base_alias); i++) { + if (chip_bmask & BIT(i)) { + phys = gic_data.dist_phys_base + T241_CHIP_GICDA_OFFSET; + phys |= FIELD_PREP(T241_CHIPN_MASK, i); + t241_dist_base_alias[i] = ioremap(phys, SZ_64K); + WARN_ON_ONCE(!t241_dist_base_alias[i]); + } + } + static_branch_enable(&gic_nvidia_t241_erratum); + return true; +} + static const struct gic_quirk gic_quirks[] = { { .desc = "GICv3: Qualcomm MSM8996 broken firmware", @@ -1739,6 +1817,12 @@ static const struct gic_quirk gic_quirks[] = { .mask = 0xe8f00fff, .init = gic_enable_quirk_cavium_38539, }, + { + .desc = "GICv3: NVIDIA erratum T241-FABRIC-4", + .iidr = 0x0402043b, + .mask = 0xffffffff, + .init = gic_enable_quirk_nvidia_t241, + }, { } }; @@ -1798,7 +1882,8 @@ static void gic_enable_nmi_support(void) gic_chip.flags |= IRQCHIP_SUPPORTS_NMI; } -static int __init gic_init_bases(void __iomem *dist_base, +static int __init gic_init_bases(phys_addr_t dist_phys_base, + void __iomem *dist_base, struct redist_region *rdist_regs, u32 nr_redist_regions, u64 redist_stride, @@ -1814,6 +1899,7 @@ static int __init gic_init_bases(void __iomem *dist_base, pr_info("GIC: Using split EOI/Deactivate mode\n"); gic_data.fwnode = handle; + gic_data.dist_phys_base = dist_phys_base; gic_data.dist_base = dist_base; gic_data.redist_regions = rdist_regs; gic_data.nr_redist_regions = nr_redist_regions; @@ -1841,10 +1927,13 @@ static int __init gic_init_bases(void __iomem *dist_base, gic_data.domain = irq_domain_create_tree(handle, &gic_irq_domain_ops, &gic_data); gic_data.rdists.rdist = alloc_percpu(typeof(*gic_data.rdists.rdist)); - gic_data.rdists.has_rvpeid = true; - gic_data.rdists.has_vlpis = true; - gic_data.rdists.has_direct_lpi = true; - gic_data.rdists.has_vpend_valid_dirty = true; + if (!static_branch_unlikely(&gic_nvidia_t241_erratum)) { + /* Disable GICv4.x features for the erratum T241-FABRIC-4 */ + gic_data.rdists.has_rvpeid = true; + gic_data.rdists.has_vlpis = true; + gic_data.rdists.has_direct_lpi = true; + gic_data.rdists.has_vpend_valid_dirty = true; + } if (WARN_ON(!gic_data.domain) || WARN_ON(!gic_data.rdists.rdist)) { err = -ENOMEM; @@ -2050,6 +2139,7 @@ static void __iomem *gic_of_iomap(struct device_node *node, int idx, static int __init gic_of_init(struct device_node *node, struct device_node *parent) { + phys_addr_t dist_phys_base; void __iomem *dist_base; struct redist_region *rdist_regs; struct resource res; @@ -2063,6 +2153,8 @@ static int __init gic_of_init(struct device_node *node, struct device_node *pare return PTR_ERR(dist_base); } + dist_phys_base = res.start; + err = gic_validate_dist_version(dist_base); if (err) { pr_err("%pOF: no distributor detected, giving up\n", node); @@ -2094,8 +2186,8 @@ static int __init gic_of_init(struct device_node *node, struct device_node *pare gic_enable_of_quirks(node, gic_quirks, &gic_data); - err = gic_init_bases(dist_base, rdist_regs, nr_redist_regions, - redist_stride, &node->fwnode); + err = gic_init_bases(dist_phys_base, dist_base, rdist_regs, + nr_redist_regions, redist_stride, &node->fwnode); if (err) goto out_unmap_rdist; @@ -2411,8 +2503,9 @@ gic_acpi_init(union acpi_subtable_headers *header, const unsigned long end) goto out_redist_unmap; } - err = gic_init_bases(acpi_data.dist_base, acpi_data.redist_regs, - acpi_data.nr_redist_regions, 0, gsi_domain_handle); + err = gic_init_bases(dist->base_address, acpi_data.dist_base, + acpi_data.redist_regs, acpi_data.nr_redist_regions, + 0, gsi_domain_handle); if (err) goto out_fwhandle_free; diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c index 95e3d2a71db6..412196a7dad5 100644 --- a/drivers/irqchip/irq-gic.c +++ b/drivers/irqchip/irq-gic.c @@ -1081,10 +1081,6 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq, return 0; } -static void gic_irq_domain_unmap(struct irq_domain *d, unsigned int irq) -{ -} - static int gic_irq_domain_translate(struct irq_domain *d, struct irq_fwspec *fwspec, unsigned long *hwirq, @@ -1167,11 +1163,6 @@ static const struct irq_domain_ops gic_irq_domain_hierarchy_ops = { .free = irq_domain_free_irqs_top, }; -static const struct irq_domain_ops gic_irq_domain_ops = { - .map = gic_irq_domain_map, - .unmap = gic_irq_domain_unmap, -}; - static int gic_init_bases(struct gic_chip_data *gic, struct fwnode_handle *handle) { @@ -1219,30 +1210,9 @@ static int gic_init_bases(struct gic_chip_data *gic, gic_irqs = 1020; gic->gic_irqs = gic_irqs; - if (handle) { /* DT/ACPI */ - gic->domain = irq_domain_create_linear(handle, gic_irqs, - &gic_irq_domain_hierarchy_ops, - gic); - } else { /* Legacy support */ - /* - * For primary GICs, skip over SGIs. - * No secondary GIC support whatsoever. - */ - int irq_base; - - gic_irqs -= 16; /* calculate # of irqs to allocate */ - - irq_base = irq_alloc_descs(16, 16, gic_irqs, - numa_node_id()); - if (irq_base < 0) { - WARN(1, "Cannot allocate irq_descs @ IRQ16, assuming pre-allocated\n"); - irq_base = 16; - } - - gic->domain = irq_domain_add_legacy(NULL, gic_irqs, irq_base, - 16, &gic_irq_domain_ops, gic); - } - + gic->domain = irq_domain_create_linear(handle, gic_irqs, + &gic_irq_domain_hierarchy_ops, + gic); if (WARN_ON(!gic->domain)) { ret = -ENODEV; goto error; @@ -1297,23 +1267,6 @@ static int __init __gic_init_bases(struct gic_chip_data *gic, return ret; } -void __init gic_init(void __iomem *dist_base, void __iomem *cpu_base) -{ - struct gic_chip_data *gic; - - /* - * Non-DT/ACPI systems won't run a hypervisor, so let's not - * bother with these... - */ - static_branch_disable(&supports_deactivate_key); - - gic = &gic_data[0]; - gic->raw_dist_base = dist_base; - gic->raw_cpu_base = cpu_base; - - __gic_init_bases(gic, NULL); -} - static void gic_teardown(struct gic_chip_data *gic) { if (WARN_ON(!gic)) @@ -1325,7 +1278,6 @@ static void gic_teardown(struct gic_chip_data *gic) iounmap(gic->raw_cpu_base); } -#ifdef CONFIG_OF static int gic_cnt __initdata; static bool gicv2_force_probe; @@ -1570,12 +1522,6 @@ IRQCHIP_DECLARE(cortex_a7_gic, "arm,cortex-a7-gic", gic_of_init); IRQCHIP_DECLARE(msm_8660_qgic, "qcom,msm-8660-qgic", gic_of_init); IRQCHIP_DECLARE(msm_qgic2, "qcom,msm-qgic2", gic_of_init); IRQCHIP_DECLARE(pl390, "arm,pl390", gic_of_init); -#else -int gic_of_init_child(struct device *dev, struct gic_chip_data **gic, int irq) -{ - return -ENOTSUPP; -} -#endif #ifdef CONFIG_ACPI static struct diff --git a/drivers/irqchip/irq-loongson-eiointc.c b/drivers/irqchip/irq-loongson-eiointc.c index d15fd38c1756..90181c42840b 100644 --- a/drivers/irqchip/irq-loongson-eiointc.c +++ b/drivers/irqchip/irq-loongson-eiointc.c @@ -280,9 +280,6 @@ static void acpi_set_vec_parent(int node, struct irq_domain *parent, struct acpi { int i; - if (cpu_has_flatmode) - node = cpu_to_node(node * CORES_PER_EIO_NODE); - for (i = 0; i < MAX_IO_PICS; i++) { if (node == vec_group[i].node) { vec_group[i].parent = parent; @@ -343,19 +340,27 @@ static int __init pch_pic_parse_madt(union acpi_subtable_headers *header, if (parent) return pch_pic_acpi_init(parent, pchpic_entry); - return -EINVAL; + return 0; } static int __init pch_msi_parse_madt(union acpi_subtable_headers *header, const unsigned long end) { + struct irq_domain *parent; struct acpi_madt_msi_pic *pchmsi_entry = (struct acpi_madt_msi_pic *)header; - struct irq_domain *parent = acpi_get_vec_parent(eiointc_priv[nr_pics - 1]->node, msi_group); + int node; + + if (cpu_has_flatmode) + node = cpu_to_node(eiointc_priv[nr_pics - 1]->node * CORES_PER_EIO_NODE); + else + node = eiointc_priv[nr_pics - 1]->node; + + parent = acpi_get_vec_parent(node, msi_group); if (parent) return pch_msi_acpi_init(parent, pchmsi_entry); - return -EINVAL; + return 0; } static int __init acpi_cascade_irqdomain_init(void) @@ -379,6 +384,7 @@ int __init eiointc_acpi_init(struct irq_domain *parent, int i, ret, parent_irq; unsigned long node_map; struct eiointc_priv *priv; + int node; priv = kzalloc(sizeof(*priv), GFP_KERNEL); if (!priv) @@ -416,13 +422,19 @@ int __init eiointc_acpi_init(struct irq_domain *parent, parent_irq = irq_create_mapping(parent, acpi_eiointc->cascade); irq_set_chained_handler_and_data(parent_irq, eiointc_irq_dispatch, priv); - register_syscore_ops(&eiointc_syscore_ops); - cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_LOONGARCH_STARTING, + if (nr_pics == 1) { + register_syscore_ops(&eiointc_syscore_ops); + cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_LOONGARCH_STARTING, "irqchip/loongarch/intc:starting", eiointc_router_init, NULL); + } - acpi_set_vec_parent(acpi_eiointc->node, priv->eiointc_domain, pch_group); - acpi_set_vec_parent(acpi_eiointc->node, priv->eiointc_domain, msi_group); + if (cpu_has_flatmode) + node = cpu_to_node(acpi_eiointc->node * CORES_PER_EIO_NODE); + else + node = acpi_eiointc->node; + acpi_set_vec_parent(node, priv->eiointc_domain, pch_group); + acpi_set_vec_parent(node, priv->eiointc_domain, msi_group); ret = acpi_cascade_irqdomain_init(); return ret; diff --git a/drivers/irqchip/irq-loongson-pch-pic.c b/drivers/irqchip/irq-loongson-pch-pic.c index 437f1af693d0..e5fe4d50be05 100644 --- a/drivers/irqchip/irq-loongson-pch-pic.c +++ b/drivers/irqchip/irq-loongson-pch-pic.c @@ -311,7 +311,8 @@ static int pch_pic_init(phys_addr_t addr, unsigned long size, int vec_base, pch_pic_handle[nr_pics] = domain_handle; pch_pic_priv[nr_pics++] = priv; - register_syscore_ops(&pch_pic_syscore_ops); + if (nr_pics == 1) + register_syscore_ops(&pch_pic_syscore_ops); return 0; @@ -403,6 +404,9 @@ int __init pch_pic_acpi_init(struct irq_domain *parent, int ret, vec_base; struct fwnode_handle *domain_handle; + if (find_pch_pic(acpi_pchpic->gsi_base) >= 0) + return 0; + vec_base = acpi_pchpic->gsi_base - GSI_MIN_PCH_IRQ; domain_handle = irq_domain_alloc_fwnode(&acpi_pchpic->address); diff --git a/drivers/irqchip/irq-riscv-intc.c b/drivers/irqchip/irq-riscv-intc.c index 499e5f81b3fe..f229e3e66387 100644 --- a/drivers/irqchip/irq-riscv-intc.c +++ b/drivers/irqchip/irq-riscv-intc.c @@ -26,20 +26,7 @@ static asmlinkage void riscv_intc_irq(struct pt_regs *regs) if (unlikely(cause >= BITS_PER_LONG)) panic("unexpected interrupt cause"); - switch (cause) { -#ifdef CONFIG_SMP - case RV_IRQ_SOFT: - /* - * We only use software interrupts to pass IPIs, so if a - * non-SMP system gets one, then we don't know what to do. - */ - handle_IPI(regs); - break; -#endif - default: - generic_handle_domain_irq(intc_domain, cause); - break; - } + generic_handle_domain_irq(intc_domain, cause); } /* @@ -59,22 +46,27 @@ static void riscv_intc_irq_unmask(struct irq_data *d) csr_set(CSR_IE, BIT(d->hwirq)); } -static int riscv_intc_cpu_starting(unsigned int cpu) -{ - csr_set(CSR_IE, BIT(RV_IRQ_SOFT)); - return 0; -} - -static int riscv_intc_cpu_dying(unsigned int cpu) +static void riscv_intc_irq_eoi(struct irq_data *d) { - csr_clear(CSR_IE, BIT(RV_IRQ_SOFT)); - return 0; + /* + * The RISC-V INTC driver uses handle_percpu_devid_irq() flow + * for the per-HART local interrupts and child irqchip drivers + * (such as PLIC, SBI IPI, CLINT, APLIC, IMSIC, etc) implement + * chained handlers for the per-HART local interrupts. + * + * In the absence of irq_eoi(), the chained_irq_enter() and + * chained_irq_exit() functions (used by child irqchip drivers) + * will do unnecessary mask/unmask of per-HART local interrupts + * at the time of handling interrupts. To avoid this, we provide + * an empty irq_eoi() callback for RISC-V INTC irqchip. + */ } static struct irq_chip riscv_intc_chip = { .name = "RISC-V INTC", .irq_mask = riscv_intc_irq_mask, .irq_unmask = riscv_intc_irq_unmask, + .irq_eoi = riscv_intc_irq_eoi, }; static int riscv_intc_domain_map(struct irq_domain *d, unsigned int irq, @@ -87,11 +79,39 @@ static int riscv_intc_domain_map(struct irq_domain *d, unsigned int irq, return 0; } +static int riscv_intc_domain_alloc(struct irq_domain *domain, + unsigned int virq, unsigned int nr_irqs, + void *arg) +{ + int i, ret; + irq_hw_number_t hwirq; + unsigned int type = IRQ_TYPE_NONE; + struct irq_fwspec *fwspec = arg; + + ret = irq_domain_translate_onecell(domain, fwspec, &hwirq, &type); + if (ret) + return ret; + + for (i = 0; i < nr_irqs; i++) { + ret = riscv_intc_domain_map(domain, virq + i, hwirq + i); + if (ret) + return ret; + } + + return 0; +} + static const struct irq_domain_ops riscv_intc_domain_ops = { .map = riscv_intc_domain_map, .xlate = irq_domain_xlate_onecell, + .alloc = riscv_intc_domain_alloc }; +static struct fwnode_handle *riscv_intc_hwnode(void) +{ + return intc_domain->fwnode; +} + static int __init riscv_intc_init(struct device_node *node, struct device_node *parent) { @@ -126,10 +146,7 @@ static int __init riscv_intc_init(struct device_node *node, return rc; } - cpuhp_setup_state(CPUHP_AP_IRQ_RISCV_STARTING, - "irqchip/riscv/intc:starting", - riscv_intc_cpu_starting, - riscv_intc_cpu_dying); + riscv_set_intc_hwnode_fn(riscv_intc_hwnode); pr_info("%d local interrupts mapped\n", BITS_PER_LONG); diff --git a/drivers/irqchip/irq-sifive-plic.c b/drivers/irqchip/irq-sifive-plic.c index ff47bd0dec45..e1484905b7bd 100644 --- a/drivers/irqchip/irq-sifive-plic.c +++ b/drivers/irqchip/irq-sifive-plic.c @@ -17,6 +17,7 @@ #include #include #include +#include #include /* @@ -67,6 +68,8 @@ struct plic_priv { struct irq_domain *irqdomain; void __iomem *regs; unsigned long plic_quirks; + unsigned int nr_irqs; + unsigned long *prio_save; }; struct plic_handler { @@ -78,6 +81,7 @@ struct plic_handler { */ raw_spinlock_t enable_lock; void __iomem *enable_base; + u32 *enable_save; struct plic_priv *priv; }; static int plic_parent_irq __ro_after_init; @@ -229,6 +233,71 @@ static int plic_irq_set_type(struct irq_data *d, unsigned int type) return IRQ_SET_MASK_OK; } +static int plic_irq_suspend(void) +{ + unsigned int i, cpu; + u32 __iomem *reg; + struct plic_priv *priv; + + priv = per_cpu_ptr(&plic_handlers, smp_processor_id())->priv; + + for (i = 0; i < priv->nr_irqs; i++) + if (readl(priv->regs + PRIORITY_BASE + i * PRIORITY_PER_ID)) + __set_bit(i, priv->prio_save); + else + __clear_bit(i, priv->prio_save); + + for_each_cpu(cpu, cpu_present_mask) { + struct plic_handler *handler = per_cpu_ptr(&plic_handlers, cpu); + + if (!handler->present) + continue; + + raw_spin_lock(&handler->enable_lock); + for (i = 0; i < DIV_ROUND_UP(priv->nr_irqs, 32); i++) { + reg = handler->enable_base + i * sizeof(u32); + handler->enable_save[i] = readl(reg); + } + raw_spin_unlock(&handler->enable_lock); + } + + return 0; +} + +static void plic_irq_resume(void) +{ + unsigned int i, index, cpu; + u32 __iomem *reg; + struct plic_priv *priv; + + priv = per_cpu_ptr(&plic_handlers, smp_processor_id())->priv; + + for (i = 0; i < priv->nr_irqs; i++) { + index = BIT_WORD(i); + writel((priv->prio_save[index] & BIT_MASK(i)) ? 1 : 0, + priv->regs + PRIORITY_BASE + i * PRIORITY_PER_ID); + } + + for_each_cpu(cpu, cpu_present_mask) { + struct plic_handler *handler = per_cpu_ptr(&plic_handlers, cpu); + + if (!handler->present) + continue; + + raw_spin_lock(&handler->enable_lock); + for (i = 0; i < DIV_ROUND_UP(priv->nr_irqs, 32); i++) { + reg = handler->enable_base + i * sizeof(u32); + writel(handler->enable_save[i], reg); + } + raw_spin_unlock(&handler->enable_lock); + } +} + +static struct syscore_ops plic_irq_syscore_ops = { + .suspend = plic_irq_suspend, + .resume = plic_irq_resume, +}; + static int plic_irqdomain_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hwirq) { @@ -345,6 +414,7 @@ static int __init __plic_init(struct device_node *node, u32 nr_irqs; struct plic_priv *priv; struct plic_handler *handler; + unsigned int cpu; priv = kzalloc(sizeof(*priv), GFP_KERNEL); if (!priv) @@ -363,15 +433,21 @@ static int __init __plic_init(struct device_node *node, if (WARN_ON(!nr_irqs)) goto out_iounmap; + priv->nr_irqs = nr_irqs; + + priv->prio_save = bitmap_alloc(nr_irqs, GFP_KERNEL); + if (!priv->prio_save) + goto out_free_priority_reg; + nr_contexts = of_irq_count(node); if (WARN_ON(!nr_contexts)) - goto out_iounmap; + goto out_free_priority_reg; error = -ENOMEM; priv->irqdomain = irq_domain_add_linear(node, nr_irqs + 1, &plic_irqdomain_ops, priv); if (WARN_ON(!priv->irqdomain)) - goto out_iounmap; + goto out_free_priority_reg; for (i = 0; i < nr_contexts; i++) { struct of_phandle_args parent; @@ -441,6 +517,11 @@ static int __init __plic_init(struct device_node *node, handler->enable_base = priv->regs + CONTEXT_ENABLE_BASE + i * CONTEXT_ENABLE_SIZE; handler->priv = priv; + + handler->enable_save = kcalloc(DIV_ROUND_UP(nr_irqs, 32), + sizeof(*handler->enable_save), GFP_KERNEL); + if (!handler->enable_save) + goto out_free_enable_reg; done: for (hwirq = 1; hwirq <= nr_irqs; hwirq++) { plic_toggle(handler, hwirq, 0); @@ -461,11 +542,19 @@ static int __init __plic_init(struct device_node *node, plic_starting_cpu, plic_dying_cpu); plic_cpuhp_setup_done = true; } + register_syscore_ops(&plic_irq_syscore_ops); pr_info("%pOFP: mapped %d interrupts with %d handlers for" " %d contexts.\n", node, nr_irqs, nr_handlers, nr_contexts); return 0; +out_free_enable_reg: + for_each_cpu(cpu, cpu_present_mask) { + handler = per_cpu_ptr(&plic_handlers, cpu); + kfree(handler->enable_save); + } +out_free_priority_reg: + kfree(priv->prio_save); out_iounmap: iounmap(priv->regs); out_free_priv: diff --git a/drivers/irqchip/irq-st.c b/drivers/irqchip/irq-st.c index 1b83512b29c6..819a12297b58 100644 --- a/drivers/irqchip/irq-st.c +++ b/drivers/irqchip/irq-st.c @@ -15,10 +15,7 @@ #include #include -#define STIH415_SYSCFG_642 0x0a8 -#define STIH416_SYSCFG_7543 0x87c #define STIH407_SYSCFG_5102 0x198 -#define STID127_SYSCFG_734 0x088 #define ST_A9_IRQ_MASK 0x001FFFFF #define ST_A9_IRQ_MAX_CHANS 2 @@ -44,22 +41,10 @@ struct st_irq_syscfg { }; static const struct of_device_id st_irq_syscfg_match[] = { - { - .compatible = "st,stih415-irq-syscfg", - .data = (void *)STIH415_SYSCFG_642, - }, - { - .compatible = "st,stih416-irq-syscfg", - .data = (void *)STIH416_SYSCFG_7543, - }, { .compatible = "st,stih407-irq-syscfg", .data = (void *)STIH407_SYSCFG_5102, }, - { - .compatible = "st,stid127-irq-syscfg", - .data = (void *)STID127_SYSCFG_734, - }, {} }; diff --git a/include/linux/arm-smccc.h b/include/linux/arm-smccc.h index 220c8c60e021..f196c19f8e55 100644 --- a/include/linux/arm-smccc.h +++ b/include/linux/arm-smccc.h @@ -226,6 +226,24 @@ void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit); extern u64 smccc_has_sve_hint; +/** + * arm_smccc_get_soc_id_version() + * + * Returns the SOC ID version. + * + * When ARM_SMCCC_ARCH_SOC_ID is not present, returns SMCCC_RET_NOT_SUPPORTED. + */ +s32 arm_smccc_get_soc_id_version(void); + +/** + * arm_smccc_get_soc_id_revision() + * + * Returns the SOC ID revision. + * + * When ARM_SMCCC_ARCH_SOC_ID is not present, returns SMCCC_RET_NOT_SUPPORTED. + */ +s32 arm_smccc_get_soc_id_revision(void); + /** * struct arm_smccc_res - Result from SMC/HVC call * @a0-a3 result values from registers 0 to 3 diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h index 5686711b0f40..2223f95079ce 100644 --- a/include/linux/irqchip/arm-gic.h +++ b/include/linux/irqchip/arm-gic.h @@ -151,12 +151,6 @@ int gic_of_init(struct device_node *node, struct device_node *parent); */ int gic_of_init_child(struct device *dev, struct gic_chip_data **gic, int irq); -/* - * Legacy platforms not converted to DT yet must use this to init - * their GIC - */ -void gic_init(void __iomem *dist , void __iomem *cpu); - void gic_send_sgi(unsigned int cpu_id, unsigned int irq); int gic_get_cpu_id(unsigned int cpu); void gic_migrate_target(unsigned int new_cpu_id); diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h index eeceafaaea4c..a07b4607b663 100644 --- a/include/trace/events/irq.h +++ b/include/trace/events/irq.h @@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise, TP_ARGS(vec_nr) ); +DECLARE_EVENT_CLASS(tasklet, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func), + + TP_STRUCT__entry( + __field( void *, tasklet) + __field( void *, func) + ), + + TP_fast_assign( + __entry->tasklet = t; + __entry->func = func; + ), + + TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func) +); + +/** + * tasklet_entry - called immediately before the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_entry, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + +/** + * tasklet_exit - called immediately after the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_exit, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + #endif /* _TRACE_IRQ_H */ /* This part must be outside protection */ diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 8ce75495e04f..d2742af0f0fd 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -189,9 +189,12 @@ void irq_set_thread_affinity(struct irq_desc *desc) { struct irqaction *action; - for_each_action_of_desc(desc, action) + for_each_action_of_desc(desc, action) { if (action->thread) set_bit(IRQTF_AFFINITY, &action->thread_flags); + if (action->secondary && action->secondary->thread) + set_bit(IRQTF_AFFINITY, &action->secondary->thread_flags); + } } #ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..1b725510dd0f 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -793,10 +793,15 @@ static void tasklet_action_common(struct softirq_action *a, if (tasklet_trylock(t)) { if (!atomic_read(&t->count)) { if (tasklet_clear_sched(t)) { - if (t->use_callback) + if (t->use_callback) { + trace_tasklet_entry(t, t->callback); t->callback(t); - else + trace_tasklet_exit(t, t->callback); + } else { + trace_tasklet_entry(t, t->func); t->func(t->data); + trace_tasklet_exit(t, t->func); + } } tasklet_unlock(t); continue;