From patchwork Thu Oct 19 23:35:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 155743 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp717001vqb; Thu, 19 Oct 2023 16:36:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHdLhWJb4KvhQrQ2t+mStZw1TdYULC6FoU2heefbtLmC2mP2RQLpnNj/iDBrdW/Tq+aPE+N X-Received: by 2002:a05:6a00:9a6:b0:69b:5395:59a8 with SMTP id u38-20020a056a0009a600b0069b539559a8mr203520pfg.34.1697758609941; Thu, 19 Oct 2023 16:36:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697758609; cv=none; d=google.com; s=arc-20160816; b=CA/eEm/qPG2dlIy3dKk15eqoJt3EW2otDsjPuZTTV6agThufMzhYgykDhX7ssEhtat oia7VTd8bT7llCViBHnt2mmJGvcFLrboO/R7jRoT86lTmNvHMXCEaVaA0b9xC4ZU+aa/ AtQivtaok/z+Dd0d3kUcVCugPQriJUGIcqR/AFUVpNn3atOoRW2lRR0Wka9lHXGzGxJo nqpwZ72n2jILL/5p9ODpGp1hjBKfputV0V1dbHA9JL42/02LeHR9SZfLTy6lFnFsgre5 +1k9AtmPk+2PWBK3IzFLgiwGOh/DvHsboz20uX60tlKniE+F7wdR4rQ88BIfLeQQZ4ql 2h6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P6F6Gc6mBGVhK3J2RHAaz/Y/yPXritc0ejaqsdY6/iU=; fh=Xtw5ULhofgKZ29v5WdXlT+QfFq0h1vKmlRJPSOHLxgo=; b=hNRD1eOZz0fle4I6N+BOKggA5KPYd/WKKqOaasXNW4k7LTB0At22xVV9VpaERxD8iJ wkGcevnK6Gpd1dVnkAK880LR1fFz86X7Vwqu73RQBYFNI3lP/JlAedHraWfH5xXdSeKJ rV1CWMh0wgE88NWRjOKdW1N7PPHtXRGKI2qDvIESvik+iEXE/EKO2m0MsiGTe4dDQaen 0P33VzWJxZnZ/rMNO0yf20+yDOM4qrfYa2hIMshEUrlH85hRq2zCdZ2lzc05XCUGCI1a 66EnNV2AXHIO/bKA4ue0o57WlFYdoWcPQNIZ9e4ETZL+LpFoA0ubIALsPnF87q4OoTP+ N+nA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=k0MThQ7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id k70-20020a633d49000000b00584b293d17fsi595304pga.550.2023.10.19.16.36.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Oct 2023 16:36:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=k0MThQ7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id CA4D3822D0B5; Thu, 19 Oct 2023 16:36:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346743AbjJSXgK (ORCPT + 26 others); Thu, 19 Oct 2023 19:36:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346736AbjJSXgF (ORCPT ); Thu, 19 Oct 2023 19:36:05 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D19D11D; Thu, 19 Oct 2023 16:36:03 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B71A5C433C7; Thu, 19 Oct 2023 23:35:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697758563; bh=dFLlbmcwk20GgIBfNZC+zS71NZd8v6vjwcVszfRzl3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=k0MThQ7u+sfp4jgen87T3Ojc9/LZ4+Ib4ysdidYB5VkASFBaj3DtTqJ4JDqj0jP58 Y0N1CZtiW0bCl7tGqFJSYgzC3ADPVwhDkki7uOT+D9L6hMpuQFVgciUxSwDYNYZTGA YE0iWsUuznM1F9VB2uF7HSES9tKxTYqdpqcOCkqiFbzpA3BxG3aAQEHG12R6ap46XK 4KS/wzBS2ta3bSfCwanFWn9f3kNb83k5erF3BNrCp2RVJEUOxwzC6W05BruDdawb+d 0IgcI1Qr+xNz1RhJagW26kcB6yNa54vJupF6rG8DFxG9qSaOBzyaIuXp869QQRx1d9 eVD/wSSpuPe0g== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E . McKenney" , Steven Rostedt , Uladzislau Rezki , rcu , Zqiang , Lai Jiangshan , "Liam R . Howlett" , Peter Zijlstra , Sebastian Siewior , Thomas Gleixner Subject: [PATCH 3/4] rcu: Make tiny RCU use ksoftirqd to trigger a QS from idle Date: Fri, 20 Oct 2023 01:35:42 +0200 Message-Id: <20231019233543.1243121-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231019233543.1243121-1-frederic@kernel.org> References: <20231019233543.1243121-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 19 Oct 2023 16:36:20 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780228932632223933 X-GMAIL-MSGID: 1780228932632223933 The commit: cff9b2332ab7 ("kernel/sched: Modify initial boot task idle setup") fixed an issue where rcutiny would request a quiescent state with setting TIF_NEED_RESCHED in early boot when init/0 has the PF_IDLE flag set but interrupts aren't enabled yet. A subsequent call to cond_resched() would then enable IRQs too early. When callbacks are enqueued in idle, RCU currently performs the following: 1) Call resched_cpu() to trigger exit from idle and go through the scheduler to call rcu_note_context_switch() -> rcu_qs() 2) rcu_qs() notes the quiescent state and raises RCU_SOFTIRQ if there is a callback, waking up ksoftirqd since it isn't called from an interrupt. However the call to resched_cpu() can opportunistically be replaced and optimized with raising RCU_SOFTIRQ and forcing ksoftirqd wakeup instead. It's worth noting that RCU grace period polling while idle is then suboptimized but such a usecase can be considered very rare or even non-existent. The advantage of this optimization is that it also works if PF_IDLE is set early because ksoftirqd is created way after IRQs are enabled on boot and it can't be awaken before its creation. If raise_ksoftirqd_irqoff() is called after the first scheduling point but before kostfirqd is created, nearby voluntary schedule calls are expected to provide the desired quiescent state and in the worst case the first launch of ksoftirqd is close enough on the first initcalls. Fixes: cff9b2332ab7 ("kernel/sched: Modify initial boot task idle setup") Cc: Liam R. Howlett Cc: Peter Zijlstra (Intel) Cc: Sebastian Siewior Cc: Thomas Gleixner Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney --- kernel/rcu/tiny.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index fec804b79080..9460e4e9d84c 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -190,12 +190,15 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_save(flags); *rcu_ctrlblk.curtail = head; rcu_ctrlblk.curtail = &head->next; - local_irq_restore(flags); if (unlikely(is_idle_task(current))) { - /* force scheduling for rcu_qs() */ - resched_cpu(0); + /* + * Force resched to trigger a QS and handle callbacks right after. + * This also takes care of avoiding too early rescheduling on boot. + */ + raise_ksoftirqd_irqoff(RCU_SOFTIRQ); } + local_irq_restore(flags); } EXPORT_SYMBOL_GPL(call_rcu); @@ -228,8 +231,16 @@ unsigned long start_poll_synchronize_rcu(void) unsigned long gp_seq = get_state_synchronize_rcu(); if (unlikely(is_idle_task(current))) { - /* force scheduling for rcu_qs() */ - resched_cpu(0); + unsigned long flags; + + /* + * Force resched to trigger a QS. This also takes care of avoiding + * too early rescheduling on boot. It's suboptimized but GP + * polling on idle isn't expected much as a usecase. + */ + local_irq_save(flags); + raise_ksoftirqd_irqoff(RCU_SOFTIRQ); + local_irq_restore(flags); } return gp_seq; }