From patchwork Thu Oct 19 23:35:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 155744 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp716997vqb; Thu, 19 Oct 2023 16:36:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHK+FiNpBvNNiomgTfjpFcAcm7jRzpR0qHYlPc687v6Kb0dcBxfNFoa/FNgkjrlX0JyRQIi X-Received: by 2002:a05:6a20:258f:b0:15f:faab:1be1 with SMTP id k15-20020a056a20258f00b0015ffaab1be1mr337451pzd.20.1697758609079; Thu, 19 Oct 2023 16:36:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697758609; cv=none; d=google.com; s=arc-20160816; b=u9SwvfC228jvFwOjwuhMd3aCD5VtXSEo08lC3bgMufkGnNZcH2buN0eEwnq+y/KT5Q EDC1mi34rsMPcAk2E2e1sQZYYT1Svso9oaPis8lnJdiUzcUriWzgvUNRpYoZbefe30iS 7TvhZS771JOBfQsC+LCRr+v0sWsU7fF3kaK2NVcKCUmYbRuZLBtkA6fQX3lbZwo8a33n nc05pHNQRTfgoOa2R4AbLyrfXKxxEKUrDWrrtY/Dl3PmoVMHIrhnA2cs+K5wOt3O3jY7 T6CslBvEyhOevhWLuCm8EDPiUJLkeWZMPmEIW0bEK5nSfgByKj+Uhu7yFQZJKg1eAP2L T3Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ir1e5LLRoJTaVuB1t5DCWNiqiwXQ7yz29Zw3UmGiPYs=; fh=Xtw5ULhofgKZ29v5WdXlT+QfFq0h1vKmlRJPSOHLxgo=; b=pxpnK8KpMw6REZ8LYpsEQHZnK8YpqfW7O5AEb+WdjcjzfWxqeHVVW1PLLV3haxfX1V MHGF1AIk1jj3F/xfN925w+gDQ/CULEaMUC45SNg2Ddm+CZkDAdD3SksyiPqYTEDp3NK7 JVqlohbyQ9kpC+0HJFs66pBUmgFcXcQWVK7Aq7nQ4YK9ZCiOvFfOJRb486ytLSDI6zVg UAwSb7/z+lYm/TBimVgjo3Eg8ShDT36K3QFFAVAUK1o82CZWyUUfbkcR/3E/b7Rjong3 5I34FnX/Vq0uZO8NG55YbBkoxKFmQaOD0W9I88Puozyn/dWcFTNWRSvbgyrnsxcHzZMq 9aBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DcvzMQNC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id ba6-20020a170902720600b001c9daca280esi510618plb.235.2023.10.19.16.36.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Oct 2023 16:36:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DcvzMQNC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 2539E804755C; Thu, 19 Oct 2023 16:36:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346721AbjJSXgG (ORCPT + 26 others); Thu, 19 Oct 2023 19:36:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346745AbjJSXgB (ORCPT ); Thu, 19 Oct 2023 19:36:01 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A443918C; Thu, 19 Oct 2023 16:35:59 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2015C433C9; Thu, 19 Oct 2023 23:35:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697758559; bh=G8Ft7qYrBUqFVecE25yG9J1QI44MxA99SLR4I8DhQCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DcvzMQNCuu5kN4SZyJr2LBpbsCdYKAvYqtLErvqiyMnKxHzBqlLY/lla5VggsyeoJ 71LkIljQUmMR3q35JnmCs2jcbodm8Kb8+XuduevxCq/YNT8V1tiOlQz+9Y0VspoWWs STK7kNWfgOHMpUVLtsuDOwaSxR+HakakeDWK2p2fHKo+FpHS3hnt9P0sy8ocjFo2WY cCHyoW163m5TIkfRf/mENYHiQj0n9nMZOlvJVnDjJV9lyND0QFl1xlZAAY5lr6PG4v /36B3/7RGxeS2PIWOXfdtmZmxQbC+lN2DDPOkThp3TYvM7/fGvjjaqZwxHNNLm5b1v 4ASc41AUt2mOQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E . McKenney" , Steven Rostedt , Uladzislau Rezki , rcu , Zqiang , Lai Jiangshan , "Liam R . Howlett" , Peter Zijlstra , Sebastian Siewior , Thomas Gleixner Subject: [PATCH 2/4] softirq: Introduce raise_ksoftirqd_irqoff() Date: Fri, 20 Oct 2023 01:35:41 +0200 Message-Id: <20231019233543.1243121-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231019233543.1243121-1-frederic@kernel.org> References: <20231019233543.1243121-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 19 Oct 2023 16:36:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780228931118930263 X-GMAIL-MSGID: 1780228931118930263 Provide a function to raise a softirq vector and force the wakeup of ksoftirqd along the way, irrespective of the current interrupt context. This is going to be used by rcutiny to fix and optimize the triggering of quiescent states from idle. Fixes: cff9b2332ab7 ("kernel/sched: Modify initial boot task idle setup") Cc: Liam R. Howlett Cc: Peter Zijlstra (Intel) Cc: Sebastian Siewior Cc: Thomas Gleixner Signed-off-by: Frederic Weisbecker --- include/linux/interrupt.h | 1 + kernel/softirq.c | 71 +++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 29 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 558a1a329da9..301d2956e746 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -608,6 +608,7 @@ extern void raise_softirq_no_wake(unsigned int nr); extern void raise_softirq_irqoff(unsigned int nr); extern void raise_softirq(unsigned int nr); +extern void raise_ksoftirqd_irqoff(unsigned int nr); DECLARE_PER_CPU(struct task_struct *, ksoftirqd); diff --git a/kernel/softirq.c b/kernel/softirq.c index acfed6f3701d..9c29a8ced1c3 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -659,35 +659,6 @@ void irq_exit(void) lockdep_hardirq_exit(); } -/* - * This function must run with irqs disabled! - */ -inline void raise_softirq_irqoff(unsigned int nr) -{ - raise_softirq_no_wake(nr); - - /* - * If we're in an interrupt or softirq, we're done - * (this also catches softirq-disabled code). We will - * actually run the softirq once we return from - * the irq or softirq. - * - * Otherwise we wake up ksoftirqd to make sure we - * schedule the softirq soon. - */ - if (!in_interrupt() && should_wake_ksoftirqd()) - wakeup_softirqd(); -} - -void raise_softirq(unsigned int nr) -{ - unsigned long flags; - - local_irq_save(flags); - raise_softirq_irqoff(nr); - local_irq_restore(flags); -} - void raise_softirq_no_wake(unsigned int nr) { lockdep_assert_irqs_disabled(); @@ -695,6 +666,48 @@ void raise_softirq_no_wake(unsigned int nr) or_softirq_pending(1UL << nr); } +/* + * This function must run with irqs disabled! + */ +static inline void __raise_softirq_irqoff(unsigned int nr, bool threaded) +{ + raise_softirq_no_wake(nr); + + if (threaded && should_wake_ksoftirqd()) + wakeup_softirqd(); +} + +/* + * This function must run with irqs disabled! + */ +inline void raise_softirq_irqoff(unsigned int nr) +{ + bool threaded; + /* + * If in an interrupt or softirq (servicing or disabled + * section), the vector will be handled at the end of + * the interrupt or softirq servicing/disabled section. + * Otherwise the vector must rely on ksoftirqd. + */ + threaded = !in_interrupt(); + + __raise_softirq_irqoff(nr, threaded); +} + +void raise_softirq(unsigned int nr) +{ + unsigned long flags; + + local_irq_save(flags); + raise_softirq_irqoff(nr); + local_irq_restore(flags); +} + +void raise_ksoftirqd_irqoff(unsigned int nr) +{ + __raise_softirq_irqoff(nr, true); +} + void open_softirq(int nr, void (*action)(struct softirq_action *)) { softirq_vec[nr].action = action;