From patchwork Mon Nov 14 09:39:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anup Patel X-Patchwork-Id: 19641 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2057836wru; Mon, 14 Nov 2022 01:51:29 -0800 (PST) X-Google-Smtp-Source: AA0mqf7mtpLlY+LK+TvA6lvLPUs8vEjsFQJkeqbD6ES1Gy2ufuU9bQmXvwCJ576YCzjflLi/9xOO X-Received: by 2002:a05:6a00:1893:b0:563:4e9:5342 with SMTP id x19-20020a056a00189300b0056304e95342mr12997178pfh.56.1668419489167; Mon, 14 Nov 2022 01:51:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668419489; cv=none; d=google.com; s=arc-20160816; b=GE/k9/H1ExWUxdhas7mpfNw/9aRP71COgJexGliPZrneW12qqxovd7ZHlru90XfJBz pFQoA15/6oCDRenR4yfWUrjdQzxkV6M0PXZQi7J023HiVV4PDY4yckY2WGajK2SWpwTh KAt1XeReXyQPkc39ywihR38KxzkxLgXLZ21gCaXaH9SM9IPQ/EnLp/jpY5OXEtPURz7P CwyXWyerU+WkF5zGFbXmV4QulXeVUre4eUAYoytXNzezwPomp8ASV3UiZ0VygR5FnZyU 6QQl4d1kccwKvANFGS7DpD2ZlurMg4b7jaFSmpwPLSX4Sh74qfM14ilUxk6VLmHUA+zU eR7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fyR5SzCa2XgzFku7z7smGvot1vjVKCAAMkMCMMFre8k=; b=Uu6vBQaCbxuv4rcjn0ouwMAgSFfDPql1iCH6NjsAYmA+SfPD9oie9OwzH5n3oPHeQc +0F6JjUOMWwY+UyemP4X4xIJnmQGv8d260JrdzgWEILqDV5WD6UIqmCqthOqFdLMdbN2 El3AvT9QNLci0ZZ3SmWLjh+0aEJshMa0DzB/KsMXpW0bgpX2FWrRr6GtTq3IP6XB+ioV ARzcRzTpAJcbjwq52EUJ+HdJB9CbVC3U7rHkR/UuP+By+o11DU3/BiAFSZcvcSasvAWk RbRrBrzd7bCWgMH1uk1M36iDdhw2ieN6Fa42N0jf28kkmOTL/bMJ2KIvQAOfov6u8f9D Kl4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=PusBjvOE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j13-20020a170903024d00b0018863062f04si10421765plh.432.2022.11.14.01.51.15; Mon, 14 Nov 2022 01:51:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=PusBjvOE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236755AbiKNJkA (ORCPT + 99 others); Mon, 14 Nov 2022 04:40:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236901AbiKNJjz (ORCPT ); Mon, 14 Nov 2022 04:39:55 -0500 Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96ED61AF1E for ; Mon, 14 Nov 2022 01:39:48 -0800 (PST) Received: by mail-oi1-x22a.google.com with SMTP id n205so10861909oib.1 for ; Mon, 14 Nov 2022 01:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fyR5SzCa2XgzFku7z7smGvot1vjVKCAAMkMCMMFre8k=; b=PusBjvOEpWHKd4s8oYTlNcEM+nRxt/ImIJfjge5KkCm+phNlxlM9HYew4kauLT/fqL qW9k6iEs0dpocQQmUq7694ilC/yMUAAp4K9gJqHEKUdUvidksp+xfTlQqAoV7n/fZvsh uleiUwCStwu1lm0TJvXNSyOvBKeVoMP60/Ua/YEzxnVZdOK12/jqNyk/EEon+sFKBlR8 zpcZvqCv0sU0IstXFYZ2hJew7lv9inw1nZM/fXvjQ/uP1bBQuNXAG1Uvk/523Uxm/zbB ATdwcutK7svu+0S2kz9DEojurx2WJQ2qvsV/BVNv9H9KtNTC0xETu5wAJ2sdjNiix+it V9Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fyR5SzCa2XgzFku7z7smGvot1vjVKCAAMkMCMMFre8k=; b=3f/XgxheIgsmletuKj6UIZL5g3ohX3dLl2ApLazxADUA9s77UuNStsMAOsGw29M2Nb Mx5xTHeXm3V0iGkmUhs6G01qtBrfJbASp1C86cjB2gkRUAl3DZS1Q2wWVBO7J9tmnUbg /k0/Ayhyy2rbey41pG01f8cDbhYPiE9fR/q7PYvBJfMtjbdW1TjUuBCBQLtv2acZf0v5 ZIxipt9jeWhEcemg+BBHVr8fqa6LSJ7hE07nr5LSIu98z9cLmfWBNrMvMQjtZvw0YIeI aPrxng2RNi/SLMn/imkLeBQ/FjkRndy6mMZphNRj4U4OT7B3P86dM6Yjoc1zRKFKfCpa X22g== X-Gm-Message-State: ANoB5pmQZ6lNUHAP8pkDOrITv/oa5KoVMmifLlGkvenFVyYzjUaJdXUa 0GesCi9x1/wtFofSBnHuKABC91wjBe8zGg== X-Received: by 2002:aca:2305:0:b0:35a:7760:43f3 with SMTP id e5-20020aca2305000000b0035a776043f3mr5358936oie.19.1668418787823; Mon, 14 Nov 2022 01:39:47 -0800 (PST) Received: from anup-ubuntu64-vm.. ([103.97.165.210]) by smtp.gmail.com with ESMTPSA id l12-20020a4ae38c000000b0049f3bdd791esm3222677oov.26.2022.11.14.01.39.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Nov 2022 01:39:47 -0800 (PST) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano Cc: Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH v11 3/7] genirq: Add mechanism to multiplex a single HW IPI Date: Mon, 14 Nov 2022 15:09:00 +0530 Message-Id: <20221114093904.1669461-4-apatel@ventanamicro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221114093904.1669461-1-apatel@ventanamicro.com> References: <20221114093904.1669461-1-apatel@ventanamicro.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749464634346064462?= X-GMAIL-MSGID: =?utf-8?q?1749464634346064462?= All RISC-V platforms have a single HW IPI provided by the INTC local interrupt controller. The HW method to trigger INTC IPI can be through external irqchip (e.g. RISC-V AIA), through platform specific device (e.g. SiFive CLINT timer), or through firmware (e.g. SBI IPI call). To support multiple IPIs on RISC-V, we add a generic IPI multiplexing mechanism which help us create multiple virtual IPIs using a single HW IPI. This generic IPI multiplexing is inspired from the Apple AIC irqchip driver and it is shared by various RISC-V irqchip drivers. Signed-off-by: Anup Patel --- include/linux/irq.h | 18 +++ kernel/irq/Kconfig | 5 + kernel/irq/Makefile | 1 + kernel/irq/ipi-mux.c | 268 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 kernel/irq/ipi-mux.c diff --git a/include/linux/irq.h b/include/linux/irq.h index c3eb89606c2b..5ab702cb0a5b 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -1266,6 +1266,24 @@ int __ipi_send_mask(struct irq_desc *desc, const struct cpumask *dest); int ipi_send_single(unsigned int virq, unsigned int cpu); int ipi_send_mask(unsigned int virq, const struct cpumask *dest); +/** + * struct ipi_mux_ops - IPI multiplex operations + * + * @ipi_mux_pre_handle: Optional function called before handling parent IPI + * @ipi_mux_post_handle:Optional function called after handling parent IPI + * @ipi_mux_send: Trigger parent IPI on target CPUs + */ +struct ipi_mux_ops { + void (*ipi_mux_pre_handle)(unsigned int parent_virq, void *data); + void (*ipi_mux_post_handle)(unsigned int parent_virq, void *data); + void (*ipi_mux_send)(unsigned int parent_virq, void *data, + const struct cpumask *mask); +}; + +void ipi_mux_process(void); +int ipi_mux_create(unsigned int parent_virq, unsigned int nr_ipi, + const struct ipi_mux_ops *ops, void *data); + #ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER /* * Registers a generic IRQ handling function as the top-level IRQ handler in diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig index db3d174c53d4..df17dbc54b02 100644 --- a/kernel/irq/Kconfig +++ b/kernel/irq/Kconfig @@ -86,6 +86,11 @@ config GENERIC_IRQ_IPI depends on SMP select IRQ_DOMAIN_HIERARCHY +# Generic IRQ IPI Mux support +config GENERIC_IRQ_IPI_MUX + bool + depends on SMP + # Generic MSI interrupt support config GENERIC_MSI_IRQ bool diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile index b4f53717d143..f19d3080bf11 100644 --- a/kernel/irq/Makefile +++ b/kernel/irq/Makefile @@ -15,6 +15,7 @@ obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += cpuhotplug.o obj-$(CONFIG_PM_SLEEP) += pm.o obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o obj-$(CONFIG_GENERIC_IRQ_IPI) += ipi.o +obj-$(CONFIG_GENERIC_IRQ_IPI_MUX) += ipi-mux.o obj-$(CONFIG_SMP) += affinity.o obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o diff --git a/kernel/irq/ipi-mux.c b/kernel/irq/ipi-mux.c new file mode 100644 index 000000000000..259e00366dd7 --- /dev/null +++ b/kernel/irq/ipi-mux.c @@ -0,0 +1,268 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Multiplex several virtual IPIs over a single HW IPI. + * + * Copyright The Asahi Linux Contributors + * Copyright (c) 2022 Ventana Micro Systems Inc. + */ + +#define pr_fmt(fmt) "ipi-mux: " fmt +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct ipi_mux_cpu { + atomic_t enable; + atomic_t bits; + struct cpumask send_mask; +}; + +struct ipi_mux_control { + void *data; + unsigned int nr; + unsigned int parent_virq; + struct irq_domain *domain; + const struct ipi_mux_ops *ops; + struct ipi_mux_cpu __percpu *cpu; +}; + +static struct ipi_mux_control *imux; +static DEFINE_STATIC_KEY_FALSE(imux_pre_handle); +static DEFINE_STATIC_KEY_FALSE(imux_post_handle); + +static void ipi_mux_mask(struct irq_data *d) +{ + struct ipi_mux_cpu *icpu = this_cpu_ptr(imux->cpu); + + atomic_andnot(BIT(irqd_to_hwirq(d)), &icpu->enable); +} + +static void ipi_mux_unmask(struct irq_data *d) +{ + u32 ibit = BIT(irqd_to_hwirq(d)); + struct ipi_mux_cpu *icpu = this_cpu_ptr(imux->cpu); + + atomic_or(ibit, &icpu->enable); + + /* + * The atomic_or() above must complete before the atomic_read() + * below to avoid racing ipi_mux_send_mask(). + */ + smp_mb__after_atomic(); + + /* If a pending IPI was unmasked, raise a parent IPI immediately. */ + if (atomic_read(&icpu->bits) & ibit) + imux->ops->ipi_mux_send(imux->parent_virq, imux->data, + cpumask_of(smp_processor_id())); +} + +static void ipi_mux_send_mask(struct irq_data *d, const struct cpumask *mask) +{ + u32 ibit = BIT(irqd_to_hwirq(d)); + struct ipi_mux_cpu *icpu = this_cpu_ptr(imux->cpu); + struct cpumask *send_mask = &icpu->send_mask; + unsigned long flags; + int cpu; + + /* + * We use send_mask as a per-CPU variable so disable local + * interrupts to avoid being preempted. + */ + local_irq_save(flags); + + cpumask_clear(send_mask); + + for_each_cpu(cpu, mask) { + icpu = per_cpu_ptr(imux->cpu, cpu); + atomic_or(ibit, &icpu->bits); + + /* + * The atomic_or() above must complete before + * the atomic_read() below to avoid racing with + * ipi_mux_unmask(). + */ + smp_mb__after_atomic(); + + if (atomic_read(&icpu->enable) & ibit) + cpumask_set_cpu(cpu, send_mask); + } + + /* Trigger the parent IPI */ + imux->ops->ipi_mux_send(imux->parent_virq, imux->data, send_mask); + + local_irq_restore(flags); +} + +static const struct irq_chip ipi_mux_chip = { + .name = "IPI Mux", + .irq_mask = ipi_mux_mask, + .irq_unmask = ipi_mux_unmask, + .ipi_send_mask = ipi_mux_send_mask, +}; + +static int ipi_mux_domain_alloc(struct irq_domain *d, unsigned int virq, + unsigned int nr_irqs, void *arg) +{ + int i; + + for (i = 0; i < nr_irqs; i++) { + irq_set_percpu_devid(virq + i); + irq_domain_set_info(d, virq + i, i, + &ipi_mux_chip, d->host_data, + handle_percpu_devid_irq, NULL, NULL); + } + + return 0; +} + +static const struct irq_domain_ops ipi_mux_domain_ops = { + .alloc = ipi_mux_domain_alloc, + .free = irq_domain_free_irqs_top, +}; + +/** + * ipi_mux_process - Process multiplexed virtual IPIs + */ +void ipi_mux_process(void) +{ + struct ipi_mux_cpu *icpu = this_cpu_ptr(imux->cpu); + irq_hw_number_t hwirq; + unsigned long ipis; + int en; + + if (static_branch_unlikely(&imux_pre_handle)) + imux->ops->ipi_mux_pre_handle(imux->parent_virq, imux->data); + + /* + * Reading enable mask does not need to be ordered as long as + * this function called from interrupt handler because only + * the CPU itself can change it's own enable mask. + */ + en = atomic_read(&icpu->enable); + + /* + * Clear the IPIs we are about to handle. This pairs with the + * atomic_fetch_or_release() in ipi_mux_send_mask(). + */ + ipis = atomic_fetch_andnot(en, &icpu->bits) & en; + + for_each_set_bit(hwirq, &ipis, imux->nr) + generic_handle_domain_irq(imux->domain, hwirq); + + if (static_branch_unlikely(&imux_post_handle)) + imux->ops->ipi_mux_post_handle(imux->parent_virq, imux->data); +} + +static void ipi_mux_handler(struct irq_desc *desc) +{ + struct irq_chip *chip = irq_desc_get_chip(desc); + + chained_irq_enter(chip, desc); + ipi_mux_process(); + chained_irq_exit(chip, desc); +} + +/** + * ipi_mux_create - Create virtual IPIs multiplexed on top of a single + * parent IPI. + * @parent_virq: virq of the parent per-CPU IRQ + * @nr_ipi: number of virtual IPIs to create. This should + * be <= BITS_PER_TYPE(int) + * @ops: multiplexing operations for the parent IPI + * @data: opaque data used by the multiplexing operations + * + * If the parent IPI > 0 then ipi_mux_process() will be automatically + * called via chained handler. + * + * If the parent IPI <= 0 then it is responsibility of irqchip drivers + * to explicitly call ipi_mux_process() for processing muxed IPIs. + * + * Returns first virq of the newly created virtual IPIs upon success + * or <=0 upon failure + */ +int ipi_mux_create(unsigned int parent_virq, unsigned int nr_ipi, + const struct ipi_mux_ops *ops, void *data) +{ + struct fwnode_handle *fwnode; + struct irq_domain *domain; + int rc; + + if (imux) + return -EEXIST; + + if (BITS_PER_TYPE(int) < nr_ipi || !ops || !ops->ipi_mux_send) + return -EINVAL; + + if (parent_virq && + !irqd_is_per_cpu(irq_desc_get_irq_data(irq_to_desc(parent_virq)))) + return -EINVAL; + + imux = kzalloc(sizeof(*imux), GFP_KERNEL); + if (!imux) + return -ENOMEM; + + imux->cpu = alloc_percpu(typeof(*imux->cpu)); + if (!imux->cpu) { + rc = -ENOMEM; + goto fail_free_mux; + } + + fwnode = irq_domain_alloc_named_fwnode("IPI-Mux"); + if (!fwnode) { + pr_err("unable to create IPI Mux fwnode\n"); + rc = -ENOMEM; + goto fail_free_cpu; + } + + domain = irq_domain_create_simple(fwnode, nr_ipi, 0, + &ipi_mux_domain_ops, NULL); + if (!domain) { + pr_err("unable to add IPI Mux domain\n"); + rc = -ENOMEM; + goto fail_free_fwnode; + } + + domain->flags |= IRQ_DOMAIN_FLAG_IPI_SINGLE; + irq_domain_update_bus_token(domain, DOMAIN_BUS_IPI); + + rc = __irq_domain_alloc_irqs(domain, -1, nr_ipi, + NUMA_NO_NODE, NULL, false, NULL); + if (rc <= 0) { + pr_err("unable to alloc IRQs from IPI Mux domain\n"); + goto fail_free_domain; + } + + imux->domain = domain; + imux->data = data; + imux->nr = nr_ipi; + imux->parent_virq = parent_virq; + imux->ops = ops; + + if (imux->ops->ipi_mux_pre_handle) + static_branch_enable(&imux_pre_handle); + + if (imux->ops->ipi_mux_post_handle) + static_branch_enable(&imux_post_handle); + + if (parent_virq > 0) + irq_set_chained_handler(parent_virq, ipi_mux_handler); + + return rc; + +fail_free_domain: + irq_domain_remove(domain); +fail_free_fwnode: + irq_domain_free_fwnode(fwnode); +fail_free_cpu: + free_percpu(imux->cpu); +fail_free_mux: + kfree(imux); + imux = NULL; + return rc; +}