From patchwork Sun Oct 16 16:22:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3057 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077873wrs; Sun, 16 Oct 2022 09:24:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5mm9KhUyRxHt9TX86lsbWNy3UiG1dBKTeVbPkud07ga8/42bc5sgVh+BOhW14Ue47bwzs5 X-Received: by 2002:a63:8648:0:b0:461:722b:ffc8 with SMTP id x69-20020a638648000000b00461722bffc8mr7276862pgd.118.1665937487608; Sun, 16 Oct 2022 09:24:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937487; cv=none; d=google.com; s=arc-20160816; b=tsgh/A4jItJx2r7gsoERfpe0MhsvYRcCSouC5xddepGPPpHoMtJG1SurEPVT+NJWdj 9woL6/Fc5Afj+Yu4rjWZ00CNX94UbkzPJZmgyHMoOeEBlcUZptoXJGSnlmDZAtKoIM/g wj1YfdRg5x7xsOTfXLo9elO9O5zL+CzuBu8cdHQ3Oo1u+uEYqTQhYwX3eTmi2xh+GytJ EBPwQKg8DY9vLe9nUUB28/62aXgTimtC3jGA+nmSt2iTCVZn6b7/gkBx6J3nI7068e1a sRgTllWauGx5ssIwCS9NH5mjM1+4WNKIoNvINaGVp9zYbTekyV/glxsn9ROe7GUjl9wX vYqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=HRvoXIVAlOUU5XZzdOJLEIP/jUAcr1FMIUdQl216/cOOQGeRDRD/SZmXWho6qmqnEj NhWoLj64CYsndMtSkmoPt02mZs5IPaSaJByGBuSMS4f4A7DkPaYlKjhqWiYJzG3g4T3L Vcrralzv26vSO7KSKIWoHvdYwuY3V53bM81TnDcqYtaeP7OGRuynfcAp5oGusNhWQude uELf+EggZEh+HwK4/pwKg/K5mzfMBMDjoy4dERahFx5LW0VPOdtsoUuYmugxJP49d4Ap NEbwuB1RgTLlKvp17mKqefrPX5OpDRomZFlbzF9BrVYSa6ocWTQLbF8U+TLfb63nTJNf tk/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="yj/YGf+8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p4-20020a170902a40400b0017f9636902fsi8852505plq.391.2022.10.16.09.24.35; Sun, 16 Oct 2022 09:24:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="yj/YGf+8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229904AbiJPQXr (ORCPT + 99 others); Sun, 16 Oct 2022 12:23:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229934AbiJPQXj (ORCPT ); Sun, 16 Oct 2022 12:23:39 -0400 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D026D3CBCB for ; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) Received: by mail-qk1-x72a.google.com with SMTP id j21so5395221qkk.9 for ; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=yj/YGf+8EA2ZpPX+5IFaseWn+7sFZFipCIU5JHL04HFdaMTYUJQQ4hCZrnCkhzCiXI WKJ5akQdzdCq4T8fLwF/E0+LjQ0eHnJ/viRtqu1ppvmLMWELcLSrbwMal8aPxNIfjDwO zk+vuG4c0Kv+ddJgdg6JRrPhXXitTZs9uaV9A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=4/0cYqhEzWTqgrQ47yDZ5DB4e97JCQ/lL6QzVZeaYqqvkJNfR+B4hTF9Eibg2j4Lq1 xW2jKLrRYXRnZeLkpaIpqsD4ZIPMQzXGrpu+qTcG/T7ODJgfLIjR57timrRgRsvYlyoa qNP4YAn7DNVljmE82hzmQ2iVmcG9hIJXpgrBUSKxt7UzDzh7LcTJtK8duLQlpJWUoeKl MsTDXZAnz0JrvQlrP5DCaF3mk9i6hv1Fr8Ot4Gdm86n3Txx+tFJuFknoPfV7WzqB5663 v+crWrDuM+t+Q2IHKr3L/QY89dq0G/v7wFkSnOb/cOi4kd+cgabUTVczOPA0lpSpNjzk vDyQ== X-Gm-Message-State: ACrzQf0N4xscoteFX4v9TmuG2d7jk1U82w5WmdotlkVmxOktmns7/T3S 3tsvw3tOjuM8hzxAGScl0NckaA== X-Received: by 2002:a05:620a:1729:b0:6ee:cf01:6810 with SMTP id az41-20020a05620a172900b006eecf016810mr4905001qkb.555.1665937415383; Sun, 16 Oct 2022 09:23:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, Joel Fernandes Subject: [PATCH v9 01/13] rcu: Fix missing nocb gp wake on rcu_barrier() Date: Sun, 16 Oct 2022 16:22:53 +0000 Message-Id: <20221016162305.2489629-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862066821145414?= X-GMAIL-MSGID: =?utf-8?q?1746862066821145414?= From: Frederic Weisbecker In preparation of RCU lazy changes, wake up the RCU nocb gp thread if needed after an entrain. Otherwise, the RCU barrier callback can wait in the queue for several seconds before the lazy callbacks in front of it are serviced. Reported-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Change-Id: I830269cd41b18862a1a58b26ce3292c6c4457bc7 --- kernel/rcu/tree.c | 11 +++++++++++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 5 +++++ 3 files changed, 17 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..67a1ae5151f5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3894,6 +3894,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) { unsigned long gseq = READ_ONCE(rcu_state.barrier_sequence); unsigned long lseq = READ_ONCE(rdp->barrier_seq_snap); + bool wake_nocb = false; + bool was_alldone = false; lockdep_assert_held(&rcu_state.barrier_lock); if (rcu_seq_state(lseq) || !rcu_seq_state(gseq) || rcu_seq_ctr(lseq) != rcu_seq_ctr(gseq)) @@ -3902,7 +3904,14 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + /* + * Flush bypass and wakeup rcuog if we add callbacks to an empty regular + * queue. This way we don't wait for bypass timer that can reach seconds + * if it's fully lazy. + */ + was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -3910,6 +3919,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rcu_barrier_trace(TPS("IRQNQ"), -1, rcu_state.barrier_sequence); } rcu_nocb_unlock(rdp); + if (wake_nocb) + wake_nocb_gp(rdp, false); smp_store_release(&rdp->barrier_seq_snap, gseq); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c..925dd98f8b23 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -439,6 +439,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f77a6d7e1356..094fd454b6c3 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1558,6 +1558,11 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) { } +static bool wake_nocb_gp(struct rcu_data *rdp, bool force) +{ + return false; +} + static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j) { From patchwork Sun Oct 16 16:22:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078212wrs; Sun, 16 Oct 2022 09:26:05 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6TLi9Z3N0mjeCHg8kYDEHQxqjPxmcM1BdEO2bEv4ZhLIj9oQFxLjxSGLsy4t9ZVe+eq4NS X-Received: by 2002:a17:902:cecf:b0:185:475a:4073 with SMTP id d15-20020a170902cecf00b00185475a4073mr7962232plg.60.1665937565460; Sun, 16 Oct 2022 09:26:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937565; cv=none; d=google.com; s=arc-20160816; b=nekJFiJpuGDJ00YKiFnXdBsoDc80c3Fc4g7ukusykMxLvRKOz6fa/UDPyQDe3E/w0e nUpCTsiBlBmHSk/NDp/7jfXwiLsdavc0dxUs/rk5853IICh5YSZZi/vEC4tCs9SMVvVL zEuq4QNYztiKV4sbN622zW5ClhamMIuiMJZrmeH7kDjVGl6XkP7Pllmx7uLk53LN1YCr qRF3+70I/0WGWOzO6YEbXONfbquauI/IAYXr6nhU6EejxF0+FK2NxNO5sgnEDB42g1bI DKL4xKQDL4TzgP/f4u0rSQv7fSWQyxb1gpnbcLOb26OwlsGIxRfdNm+1XtRNq9iIyPEJ LKzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jxHkwXeU8t3o10TBzHgB8dUBc1AVBDaAMYcAAkthrB8=; b=PfFHLztB4SO4Mmh5NfsqwY+rhtpvsYaa2vZHW44THrjaiKv6nC/1e2Zf/02L6G/wmO Iy2qz7fzdM/NMikB3iSdE/kKs8qqxZOaNewVdMim0hVeaAHPVENo+FbtODwm5g8XNnZF 7wvKA6jALoO9Ghq15Pynqh6XYGRQPdF/nBXrahnZF678vW77qErtyz+yxcrqkZBdDjlg R2Z1NZPB3JEP71YUZRdEwJ59/4S5UsdXy4DtyDaEdZyacKfMlcC3eRoAwhKeFY3jbuWg IQPpEMEaiFt4Wg/5k/mQgeSB+gCgchMR6U+5Paoniq39duy8vyawvRdANLBBS9Mfk2l8 JKZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=CoqGhVOb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c13-20020a655a8d000000b0043c0b4f2b68si9430896pgt.318.2022.10.16.09.25.52; Sun, 16 Oct 2022 09:26:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=CoqGhVOb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229925AbiJPQYE (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbiJPQXo (ORCPT ); Sun, 16 Oct 2022 12:23:44 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35F423C8F8 for ; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id f8so5415335qkg.3 for ; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jxHkwXeU8t3o10TBzHgB8dUBc1AVBDaAMYcAAkthrB8=; b=CoqGhVObkaNtmq/+Sqs5yV1TfOCd82B5FyaiS8YVRCX1at6di+8CymgBiI5KX5Em7W sZpAoVfZEYel6/AKDBznC0NIUq9ZyyXukfe2er3/J+sZyp2e2k7chFgn1jDpzGvLOGd0 xG/sOcwo5UChnrvmH+8aY1DekCP5WTDIpwzWQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jxHkwXeU8t3o10TBzHgB8dUBc1AVBDaAMYcAAkthrB8=; b=cIn03oIDP9CuWW/PJr8P9JRrf41j+M06ixFobz8i7Sk3a69HGh4Gft15XuIbFCNcQc KPp5MxlJbPLmMDptYGX5UckJHyWOLlkh+HmUZk8/iqJotPVYZM5jznmf0OH2Kb2tciut bkczB7UvBTn6flzNE3i3Z07VzQvTSN25MnMz7helWgTO2WzGHnytKJu7kEN9ZS0rpvkI ZLZkNK1RY0QmGCvm48pI9q7PFNt4N5f7h242MzEyCrGfDhwYWkjc+QwWeFKimu+PNwY2 0SgUk8TiNS32lwXLIcsohBRu528QF3XWwVsPSNpbUR78XxpGJSCOwwNaqiEH9UDY3OZo r/+w== X-Gm-Message-State: ACrzQf1HLQ1afacNz+OjnujvzevUwEHSjA1DVpWskphZw0n71dR+VN7M nT3kP6lpH7/T+s5lANdaR44VLg== X-Received: by 2002:a05:620a:2a0c:b0:6cf:9085:683b with SMTP id o12-20020a05620a2a0c00b006cf9085683bmr5025164qkp.159.1665937416019; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 02/13] rcu: Make call_rcu() lazy to save power Date: Sun, 16 Oct 2022 16:22:54 +0000 Message-Id: <20221016162305.2489629-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862148599889020?= X-GMAIL-MSGID: =?utf-8?q?1746862148599889020?= Implement timer-based RCU callback batching (also known as lazy callbacks). With this we save about 5-10% of power consumed due to RCU requests that happen when system is lightly loaded or idle. By default, all async callbacks (queued via call_rcu) are marked lazy. An alternate API call_rcu_flush() is provided for the few users, for example synchronize_rcu(), that need the old behavior. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists which were originally introduced to address lock contention, to handle lazy CBs as well. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. Suggested-by: Paul McKenney Acked-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Change-Id: I7dc21f6143d79f6893dade07a5cd448de8b83457 --- include/linux/rcupdate.h | 7 ++ kernel/rcu/Kconfig | 8 ++ kernel/rcu/rcu.h | 8 ++ kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 129 ++++++++++++++++++++----------- kernel/rcu/tree.h | 11 ++- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_nocb.h | 159 +++++++++++++++++++++++++++++++-------- 8 files changed, 244 insertions(+), 82 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 08605ce7379d..40ae36904825 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -108,6 +108,13 @@ static inline int rcu_preempt_depth(void) #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_RCU_LAZY +void call_rcu_flush(struct rcu_head *head, rcu_callback_t func); +#else +static inline void call_rcu_flush(struct rcu_head *head, + rcu_callback_t func) { call_rcu(head, func); } +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index f53ad63b2bc6..edd632e68497 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -314,4 +314,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index be5979da07f5..65704cbc9df7 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -474,6 +474,14 @@ enum rcutorture_type { INVALID_RCU_FLAVOR }; +#if defined(CONFIG_RCU_LAZY) +unsigned long rcu_lazy_get_jiffies_till_flush(void); +void rcu_lazy_set_jiffies_till_flush(unsigned long j); +#else +static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return 0; } +static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +#endif + #if defined(CONFIG_TREE_RCU) void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq); diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index a33a8d4942c3..810479cf17ba 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -44,7 +44,7 @@ static struct rcu_ctrlblk rcu_ctrlblk = { void rcu_barrier(void) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); } EXPORT_SYMBOL(rcu_barrier); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 67a1ae5151f5..f4b390f86865 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,47 +2728,8 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -2809,7 +2770,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -2831,8 +2792,84 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); +#ifdef CONFIG_RCU_LAZY +/** + * call_rcu_flush() - Queue RCU callback for invocation after grace period, and + * flush all lazy callbacks (including the new one) to the main ->cblist while + * doing so. + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. + * + * Use this API instead of call_rcu() if you don't want the callback to be + * invoked after very long periods of time, which can happen on systems without + * memory pressure and on systems which are lightly loaded or mostly idle. + * This function will cause callbacks to be invoked sooner than later at the + * expense of extra power. Other than that, this function is identical to, and + * reuses call_rcu()'s logic. Refer to call_rcu() for more details about memory + * ordering and other functionality. + */ +void call_rcu_flush(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, false); +} +EXPORT_SYMBOL_GPL(call_rcu_flush); +#endif + +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * By default the callbacks are 'lazy' and are kept hidden from the main + * ->cblist to prevent starting of grace periods too soon. + * If you desire grace periods to start very soon, use call_rcu_flush(). + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, true); +} +EXPORT_SYMBOL_GPL(call_rcu); /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (5 * HZ) @@ -3507,7 +3544,7 @@ void synchronize_rcu(void) if (rcu_gp_is_expedited()) synchronize_rcu_expedited(); else - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); return; } @@ -3910,7 +3947,7 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) * if it's fully lazy. */ was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); @@ -4334,7 +4371,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp = this_cpu_ptr(&rcu_data); my_rnp = my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake = rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 925dd98f8b23..fcb5d696eb17 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -263,14 +263,16 @@ struct rcu_data { unsigned long last_fqs_resched; /* Time of last rcu_resched(). */ unsigned long last_sched_clock; /* Jiffies of last rcu_sched_clock_irq(). */ + long lazy_len; /* Length of buffered lazy callbacks. */ int cpu; }; /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -441,9 +443,10 @@ static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j); + unsigned long j, bool lazy); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 18e9b4cd78ef..5cac05600798 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -937,7 +937,7 @@ void synchronize_rcu_expedited(void) /* If expedited grace periods are prohibited, fall back to normal. */ if (rcu_gp_is_normal()) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); return; } diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 094fd454b6c3..ab9ce0ebec23 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,31 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } +/* + * LAZY_FLUSH_JIFFIES decides the maximum amount of time that + * can elapse before lazy callbacks are flushed. Lazy callbacks + * could be flushed much earlier for a number of other reasons + * however, LAZY_FLUSH_JIFFIES will ensure no lazy callbacks are + * left unsubmitted to RCU after those many jiffies. + */ +#define LAZY_FLUSH_JIFFIES (10 * HZ) +static unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_LAZY +// To be called only from test code. +void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush = jif; +} +EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); + +unsigned long rcu_lazy_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +#endif + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -269,10 +294,14 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); /* - * Bypass wakeup overrides previous deferments. In case - * of callback storm, no need to wake up too early. + * Bypass wakeup overrides previous deferments. In case of + * callback storm, no need to wake up too early. */ - if (waketype == RCU_NOCB_WAKE_BYPASS) { + if (waketype == RCU_NOCB_WAKE_LAZY && + rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) { + mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + } else if (waketype == RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else { @@ -293,10 +322,13 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * proves to be initially empty, just return false because the no-CB GP * kthread may need to be awakened in this case. * + * Return true if there was something to be flushed and it succeeded, otherwise + * false. + * * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; @@ -310,7 +342,20 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + /* + * If the new CB requested was a lazy one, queue it onto the main + * ->cblist so we can take advantage of a sooner grade period. + */ + if (lazy && rhp) { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); + rcu_cblist_enqueue(&rcl, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } else { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } + rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +371,13 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + return rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); } /* @@ -345,7 +390,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } /* @@ -367,12 +412,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j = jiffies; long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + bool bypass_is_lazy = (ncbs == READ_ONCE(rdp->lazy_len)); lockdep_assert_irqs_disabled(); @@ -417,25 +464,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // If there hasn't yet been all that many ->cblist enqueues // this jiffy, tell the caller to enqueue onto ->cblist. But flush // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // Lazy CBs throttle this back and do immediate bypass queuing. + if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || + if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || + (ncbs && bypass_is_lazy && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush))) || ncbs >= qhimark) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy)) { if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -463,13 +514,24 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + + if (lazy) + WRITE_ONCE(rdp->lazy_len, rdp->lazy_len + 1); + if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ - if (ncbs) { + // A wake up of the grace period kthread or timer adjustment + // needs to be done only if: + // 1. Bypass list was fully empty before (this is the first + // bypass list entry), or: + // 2. Both of these conditions are met: + // a. The bypass list previously had only lazy CBs, and: + // b. The new CB is non-lazy. + if (ncbs && (!bypass_is_lazy || lazy)) { local_irq_restore(flags); } else { // No-CBs GP kthread might be indefinitely asleep, if so, wake. @@ -497,8 +559,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, unsigned long flags) __releases(rdp->nocb_lock) { + long bypass_len; unsigned long cur_gp_seq; unsigned long j; + long lazy_len; long len; struct task_struct *t; @@ -512,9 +576,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } // Need to actually to a wakeup. len = rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len = READ_ONCE(rdp->lazy_len); if (was_alldone) { rdp->qlen_last_fqs_check = len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len == lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -605,12 +676,12 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) static void nocb_gp_wait(struct rcu_data *my_rdp) { bool bypass = false; - long bypass_ncbs; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; unsigned long j = jiffies; + bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; @@ -640,24 +711,43 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { + long bypass_ncbs; + bool flush_bypass = false; + long lazy_ncbs; + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); lockdep_assert_held(&rdp->nocb_lock); bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + flush_bypass = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + flush_bypass = true; } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); continue; /* No callbacks here, try next. */ } + + if (flush_bypass) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass = true; + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs == lazy_ncbs) + lazy = true; + else + bypass = true; } rnp = rdp->mynode; @@ -705,12 +795,20 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -1036,7 +1134,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1278,6 +1376,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + WRITE_ONCE(rdp->lazy_len, 0); mutex_init(&rdp->nocb_gp_kthread_mutex); } @@ -1564,13 +1663,13 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) } static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { return true; } static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, bool lazy) { return false; } From patchwork Sun Oct 16 16:22:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3059 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077896wrs; Sun, 16 Oct 2022 09:24:53 -0700 (PDT) X-Google-Smtp-Source: AMsMyM71eoT3BeQYUad6ZngPmL1NuWLo83rz8W43gtogUdOK8vIuA2fuPdNwsHkNz0vUUZ/8MbIi X-Received: by 2002:a05:6a00:27a0:b0:566:9cd9:3843 with SMTP id bd32-20020a056a0027a000b005669cd93843mr8096120pfb.17.1665937493125; Sun, 16 Oct 2022 09:24:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937493; cv=none; d=google.com; s=arc-20160816; b=PhXqAzaBqzuiSlvzjjhvYLmlgnelz2Nr98KWeIOlMDEWK2fYhpQSE06vIUwOAu6mab SJzaIKc+nx370Wn/qhnC7/U2jncyewN5ipFX/FRD7cvQ++apGXqyqLPJGkhJQv5vTvdm NApZ43Nwwcdq4lt54EzJgH8nl+3WFj/BANGC2kUbOY1jOgblaneFtf/rafuUVN8KBxnp +6zBmnmlG7AH5y7kOUZccfiGc8idCXaDgD9UmzQQGdb5bdXSK8p9OiSrWy4IZbba4mDU spciyWskJVKe/TEfroQtZkYC1kr+Axrjgb85FK4KQCPLhcm9f+OaFv5opWp9IUoGgA62 mStA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZziBLLlKHVzOSjc34J63Rv49GuOQiJ4SzWGeYcNVKSE=; b=FpIs26ZtlCrfChgxUEcw7gZirPjJC7pocUNenhAmRek48cm/yD4Ht0C+W7D4/FWY6d 4IFRG5z4Qphd1zPQwBn7dkJiQiSBK8bhR/alMOgAOo248rPCklq+beQ/xYPaH4hGlAWi nhLN5ggEt2HlFRjBp3m7+CBk4T8TZQT5I16lqdUcvaK+YX2ouNcBqMi1H5wqt2Af38ez HklevNWK4YFl87QpHz/jlLBoNvShp3pwQ4dYWVK0mrm1wLbsR8n6VDDIV9fZbUZDms4M C2Glc1JP9qa8S0mCyRXXZVdhy/NX+8mn4+lu8oo2OztFgQurE6MOnSRGbVsZsY2ZhemD PHIw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=o6NYymtW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l1-20020a170902f68100b00179d2e2e690si10997129plg.332.2022.10.16.09.24.39; Sun, 16 Oct 2022 09:24:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=o6NYymtW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229910AbiJPQX4 (ORCPT + 99 others); Sun, 16 Oct 2022 12:23:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229902AbiJPQXm (ORCPT ); Sun, 16 Oct 2022 12:23:42 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 362F83CBFC for ; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id f22so6495996qto.3 for ; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZziBLLlKHVzOSjc34J63Rv49GuOQiJ4SzWGeYcNVKSE=; b=o6NYymtWFFRcLKV9JGD5gsps1y6NUcfW3Qzq/Gg3Yy9WqLVAYmQJiUhHhH5CPhF1zT jD1drN+HTN1kjbs9R/euhIkLJH4lJwAj2TE2k+jfgEO5Ur65OpUgLvUANNWx1opLesWd Wld0TrKwgvAVFfLEx4dsPDGhvzbHH2nbeQwaE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZziBLLlKHVzOSjc34J63Rv49GuOQiJ4SzWGeYcNVKSE=; b=UP6WTTD7SDqoHo9RYoOFG++jMXMCVlgy0yplu1hnFLzDCMLf4q4q6kpo+Uqzk9ugWb HXAIZYx1OZfTRe9jvQ7+/Y3lxAO7qStUXx+qIQfizcyG4UYwYCuJwracQ6nxKzAeFd74 nu1uBlxS8UKL6QI2bgib+XBH7jUYJ8nQWrUVTSF/cZh20saxgh9Zm+k15SkZtFLXhzz0 2mPwpk0PoOT8flzAjmY7moCYjqOrbZZjgipBPwrM6PKbYaMqxxhn+FJMDxSwKFBVISW7 pCAkOKfvVKZfas+pnsMSzdecBp6hzuwR+Y0GGNeNdi65wSCT5v6a/PqnLEh0JT2wVc+m z2pA== X-Gm-Message-State: ACrzQf3Z7NF2qA+o3Iqi5pSV3T/Rw42jPvkXAksQs7wEuOIwpqbiGtvX 8K/GvUkXKN0tNbznNFIdnmP7xHwNrAdlaA== X-Received: by 2002:a05:622a:283:b0:39c:d772:7e02 with SMTP id z3-20020a05622a028300b0039cd7727e02mr5663660qtw.369.1665937416460; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:36 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 03/13] rcu: Refactor code a bit in rcu_nocb_do_flush_bypass() Date: Sun, 16 Oct 2022 16:22:55 +0000 Message-Id: <20221016162305.2489629-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862072782597355?= X-GMAIL-MSGID: =?utf-8?q?1746862072782597355?= This consolidates the code a bit and makes it cleaner. Functionally it is the same. Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ab9ce0ebec23..717c0591c037 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -327,10 +327,11 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * * Note that this function always returns true if rhp is NULL. */ -static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, +static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp_in, unsigned long j, bool lazy) { struct rcu_cblist rcl; + struct rcu_head *rhp = rhp_in; WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp)); rcu_lockdep_assert_cblist_protected(rdp); @@ -345,16 +346,16 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* * If the new CB requested was a lazy one, queue it onto the main - * ->cblist so we can take advantage of a sooner grade period. + * ->cblist so that we can take advantage of the grace-period that will + * happen regardless. But queue it onto the bypass list first so that + * the lazy CB is ordered with the existing CBs in the bypass list. */ if (lazy && rhp) { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); - rcu_cblist_enqueue(&rcl, rhp); - WRITE_ONCE(rdp->lazy_len, 0); - } else { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); - WRITE_ONCE(rdp->lazy_len, 0); + rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + rhp = NULL; } + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); From patchwork Sun Oct 16 16:22:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077875wrs; Sun, 16 Oct 2022 09:24:49 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4GJIoOd0f07JYs1sSWq3TqMfgj2LToEM/ygl8+OE1VaXkiFmuReVuD6KJMQNQuhrGVWaAM X-Received: by 2002:a17:903:40cf:b0:17f:7d7e:95b0 with SMTP id t15-20020a17090340cf00b0017f7d7e95b0mr7729908pld.78.1665937488718; Sun, 16 Oct 2022 09:24:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937488; cv=none; d=google.com; s=arc-20160816; b=w90mXaxAv3lW3Ilu40fgxYE7oIfxPXG8WGbV5l9IU4/hSOmUpJaSbhM6zFFpwhlwgG 85TNBA8N2KmAkzftMD5LAf6I4OFavNIDT25dsGJuRgvJW8qTxFuO+dSTXEw34FjCn6G8 IDcauMHm17XO+1SJhLW+OlQJstq3rGqP1A/VnfOWVDITux0jzpNtTjn0JO8FkuIQcP60 klqb+qbzRMNaTCZPixziUBljGeTgbcFOgRp0LvQIy7mWWsgLOl4klIf0DvMKucleYDff zZyGyegRq7n2XzZf5sg6den8zLAvGBv6wGHxQBRBOyBPv/gUfSBsIh3OXWpSdbUEZwPy 48Xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=O99Qgvv8EMn7R2VX8zNwhW7077dknEew8817iqFh+Z0=; b=IYKuHr963Xs8SAT+vwNMtvnISGcjKstlqvy2d7oWc2d3fGpLh5LFBGb8kvepQMg5I3 Lyu8Rkw5L62kpxTY91kon002pxGaELoCkMOC3Mo89zd8V31r4lpcr7Ddd1kev0AYQZhL gyXdT4OiUiSmidN2Py/M5AdbyHs3TQAUAoDvFfcWqimEZ6BS2n6mLTf2PIQcGP9ndt// JdOvINNLoZ2+Ku6p1vyMfyT+vJSVVfNcl22uj/orY4o9y6N3XjLRltBP9HwPJJGQtO4T wIEk9e1ip2bbhiVE/wQgH2bo7QjX3RMFHnSUFDVZx6JSGRI15mVKaY/ahn6TcmbHCC4i tXvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=A0iw44OI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r12-20020a17090a1bcc00b0020d895425a4si10957014pjr.3.2022.10.16.09.24.36; Sun, 16 Oct 2022 09:24:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=A0iw44OI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229950AbiJPQXv (ORCPT + 99 others); Sun, 16 Oct 2022 12:23:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229895AbiJPQXl (ORCPT ); Sun, 16 Oct 2022 12:23:41 -0400 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A981C3D5A4 for ; Sun, 16 Oct 2022 09:23:38 -0700 (PDT) Received: by mail-qk1-x729.google.com with SMTP id f8so5415345qkg.3 for ; Sun, 16 Oct 2022 09:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O99Qgvv8EMn7R2VX8zNwhW7077dknEew8817iqFh+Z0=; b=A0iw44OIXFnSt6q7ohniyWUHk+cUylfjjbIh1F4lTWAyPpei5ZY5NInt+6TUjLZyqT GQbzNAYDP5YQDIpkg/1CC+ISYld8+pJRCj4CzKCHI2txJCFmFKE+xXPXb70gbpfpxqM8 D7wxzReb589Z93lS7ZQ4saIztNiFghiEs2suw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O99Qgvv8EMn7R2VX8zNwhW7077dknEew8817iqFh+Z0=; b=F4Y93wUf0KCAwQXnJeUJ2vzkrIp9VcS9aOik/GriATLZ0XkKb34CRM+DnFLKAKOQee MgWOxH3CY+Us4uXmwdqDgfatLrPyqtmugFh+6Ac4JNaV1vv//jDE8LRka8NAw72W3Xek 2c4hTDLP/YAI0yUtnNNyD0Dts76R3pE87jpKnDbH9BiM22Gwabfy6S70qMVWZ66tertf 7urIP5M9XHBPbf1RJJeR5+aCuUvpEZJKvPHW+V5muIGSR84Swq5jcXoqoVpU5VBuPoAL 4BH0tn93UY5LwPqC9JGzSlMbF5RsyB2kh4Otyl8g5rlcN/sLUnosTA94x/W7fPj3ZUzf FL0Q== X-Gm-Message-State: ACrzQf3z0gcmWQF+YO6Oo/MpWmJVNiEdpMe7PWXjobOtUfXeKMQeTDs5 iJtwwZtieler0Kw+LAoQ/VaLIMOd5FpfHA== X-Received: by 2002:a05:620a:191d:b0:6ee:e3cf:b2db with SMTP id bj29-20020a05620a191d00b006eee3cfb2dbmr1245456qkb.633.1665937417334; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:36 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, Vineeth Pillai , Joel Fernandes Subject: [PATCH v9 04/13] rcu: shrinker for lazy rcu Date: Sun, 16 Oct 2022 16:22:56 +0000 Message-Id: <20221016162305.2489629-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862068264515207?= X-GMAIL-MSGID: =?utf-8?q?1746862068264515207?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 717c0591c037..dc014a0b97b7 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask, From patchwork Sun Oct 16 16:22:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077931wrs; Sun, 16 Oct 2022 09:25:03 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5w+auqAtxvsfLLQizU24JTdA/Cxz33Nzs7ZuPHQBUBgzPso6C5YRQg4+OIZy4CDmi3b3pH X-Received: by 2002:a17:90b:17c9:b0:20d:489b:55c0 with SMTP id me9-20020a17090b17c900b0020d489b55c0mr9288762pjb.232.1665937503353; Sun, 16 Oct 2022 09:25:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937503; cv=none; d=google.com; s=arc-20160816; b=N+KRlSt6kr1AIyMiMNOsBFYElO9UKiGVehkPsqmMs7F8gjKvywzwSGMxN0q+mwl8FW 1t6AcZ/9jqbQWu6jP/IEK7KMNUzsBImjqimTeRpPXgKQypPQt+XYyXtahbqFSuKMWbfU 9062c1tgvIHPk5++qhRTlDAeihHt3HZyqDQzgsmNYUYx6RWfB4vFGmsvtvJrDcYwEWBM DFLEAm26WHx2983R9Sgzss3p/nXZiSyMfw141BZqBJygP47PXNQN1yi73r8KMSsKewCE hJwoySZ/ONkzkZ1r8A/0BguvSyetzLQ81n8PPOTM+VD/jxKFSg9bCIXQnB7nOdKqynMA vXeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MFv2k9n8x/vbjzME1MIB+yYKdyjDx7vFk8XwTZ/52oM=; b=LDs7pAbv+MfpP2W4EK3W9O19tjD8PJtM7kFhsxfYOuOUmVYMwBGdcO09ES6Z7kz6ab Pb8Wo04vrSE8m/8+PDJZ96vyPkOO/Rj8xNgyBnLQsRriAQ9t5lbLFcnk0av2WOCua6QY pozPj+TEy8sdte4K1RfFekwRLpMymO0YXwWBHG4Wfu+AGpI399DOaTDZcuHSVnWIoPUM UhQMaDo6gJUS1tT+8MXPaY/436gn/YYLS5lrvfDN8JtJ+DFAcrsPyoytpf22yypkDAB8 e7VxkGNP9R8Lnmh2N1apMEytEXSA4NGpsG23fItpCVjOXZGU0Q4qVo018To2lxXg/dKf HI1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=pFoAWi3k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 184-20020a6301c1000000b00430b25c0134si9955306pgb.670.2022.10.16.09.24.50; Sun, 16 Oct 2022 09:25:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=pFoAWi3k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229829AbiJPQYJ (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229919AbiJPQXo (ORCPT ); Sun, 16 Oct 2022 12:23:44 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE8D736DEA for ; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id a24so6477734qto.10 for ; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MFv2k9n8x/vbjzME1MIB+yYKdyjDx7vFk8XwTZ/52oM=; b=pFoAWi3kBEAGDc2aHFGMmCJvf5Zfz5LxHgotdiEB9LFKrsTbW15ZnDMU1HUBK5zm8W crxIjDZHN1eKJsY0Bq+u2RNV3MIm5nOZdhzA2SuantNmQf2+oFOg6DzxFxue8tSYymUg j3DuWTFxWngANZ4oR094/l7N7vAUXimfSN/7c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MFv2k9n8x/vbjzME1MIB+yYKdyjDx7vFk8XwTZ/52oM=; b=v5i0jvtZl8rz6DpmWhU6k1u9gZd1rAhQ96A0WbdqlFoK+IGYRida6XKSDujlcL6nB3 kphbXv+r/lRyXCXvSXQQS1Yer/iEneVxUi+Lh+qD9K+klsPTO8CZ6YD2viARHWJiO2Uy npKw3AWMOruJamk/VR/D2gFYzCWlVWwKISLxKVaY3pyBJq36mcOCEWI0mHxHc/oP/jai 0ek6hFmGdruCfzjOsFcMJ8BIXiG3JlGOu9DrcifnZbks0pwWTGWxU4wvspzoXzzVjzHK nv0Smd4uFwKaXX7i+vothLLMLcxds1thX2cdfTTPGsklO0F+sqLBmMYESU6i3wnVP+yq 5EAg== X-Gm-Message-State: ACrzQf1ZDeFAphOg23MJngEVKOAVKhIHJRoF3jsQnoF9RxvctdSnzgww an5Pi+77C0wXSuaQCzZfGsUqMA== X-Received: by 2002:ac8:5995:0:b0:39a:acd0:ecd with SMTP id e21-20020ac85995000000b0039aacd00ecdmr5638044qte.206.1665937417931; Sun, 16 Oct 2022 09:23:37 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:37 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 05/13] rcuscale: Add laziness and kfree tests Date: Sun, 16 Oct 2022 16:22:57 +0000 Message-Id: <20221016162305.2489629-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862083861226894?= X-GMAIL-MSGID: =?utf-8?q?1746862083861226894?= We add 2 tests to rcuscale, first one is a startup test to check whether we are not too lazy or too hard working. Two, emulate kfree_rcu() itself to use call_rcu() and check memory pressure. In my testing, the new call_rcu() does well to keep memory pressure under control, similar to kfree_rcu(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcuscale.c | 68 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3ef02d4a8108..bbdcac1804ec 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?"); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate."); +torture_param(int, kfree_by_call_rcu, 0, "Use call_rcu() to emulate kfree_rcu()?"); static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); @@ -659,6 +660,14 @@ struct kfree_obj { struct rcu_head rh; }; +/* Used if doing RCU-kfree'ing via call_rcu(). */ +static void kfree_call_rcu(struct rcu_head *rh) +{ + struct kfree_obj *obj = container_of(rh, struct kfree_obj, rh); + + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -696,6 +705,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; + if (kfree_by_call_rcu) { + call_rcu(&(alloc_ptr->rh), kfree_call_rcu); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -767,11 +781,59 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } +// Used if doing RCU-kfree'ing via call_rcu(). +static unsigned long jiffies_at_lazy_cb; +static struct rcu_head lazy_test1_rh; +static int rcu_lazy_test1_cb_called; +static void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb = jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { - long i; int firsterr = 0; + long i; + unsigned long jif_start; + unsigned long orig_jif; + + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_by_call_rcu && !IS_ENABLED(CONFIG_RCU_LAZY)) { + pr_alert("CONFIG_RCU_LAZY is disabled, falling back to kfree_rcu() " + "for delayed RCU kfree'ing\n"); + kfree_by_call_rcu = 0; + } + + if (kfree_by_call_rcu) { + /* do a test to check the timeout. */ + orig_jif = rcu_lazy_get_jiffies_till_flush(); + + rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start = jiffies; + jiffies_at_lazy_cb = 0; + call_rcu(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); + + rcu_lazy_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are not being lazy as expected!\n"); + WARN_ON_ONCE(1); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are being too lazy!\n"); + WARN_ON_ONCE(1); + return -1; + } + } kfree_nrealthreads = compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -784,7 +846,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } - pr_alert("kfree object size=%zu\n", kfree_mult * sizeof(struct kfree_obj)); + pr_alert("kfree object size=%zu, kfree_by_call_rcu=%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_by_call_rcu); kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]), GFP_KERNEL); From patchwork Sun Oct 16 16:22:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3064 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078091wrs; Sun, 16 Oct 2022 09:25:36 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4eg6eC+eUsSJYLeQFtfEWhufUC9AZMllo8ibFSs/6GzPVwFhCVMi5zFuS/++4ClSyn5+08 X-Received: by 2002:a17:902:da8b:b0:178:391d:d0b1 with SMTP id j11-20020a170902da8b00b00178391dd0b1mr7849525plx.32.1665937535931; Sun, 16 Oct 2022 09:25:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937535; cv=none; d=google.com; s=arc-20160816; b=RTo7BUwxLQUIUiaB2sqwBDltY+XA8XiaWCBxDOWYFHkKjahrAUfhPNL4rJAB0waU2P qiU3ZNt8xpPjp4YgyM94GYU7DcM/FZYoRiXZ61FcBQJ6vpD9YlmPTzV0xplONxIYwfYt DtW2SNiEkhRxwrCOdYW+eVA3qV/tm9EU0mUdi4JEPNRLyOs2xKzM+uubnSr9KZQhc3w4 IHrw1EjaEdoVwZtq51tYjKUod8uBDKduiLluM3BBxv7WVHFSvB8JQ/2Aori2yI9E0nil bn41Kw1R+1MSb6WCZPOpx0YtrhSTPIinUm0OfA8fgR+4y9eeCy5RHdzx1DX5G4vXwPH0 CEtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YCISppMzdHX/DQsNZveHbYvxNRQkJmaJ4AIlPWlWRgs=; b=fSHO+w51dzjGvJOMjfex2njmHe4e37+lK8+WgQMLzWKTNuWFBhjITIXmPJE6BkkErh LlFGyn1Cp3wLAhBW8XP0nxtkMbP4FXGsqFBvT7/H/GgDcATJf8m59BtfTzE5ENk0zvXJ uxnqM2HiNvJ2duC/tzdVWloGwWxq1aoxSGZ5zjv9nXwQyq0JwQiZ4RqaUhETf6aDPSUY JoB3cfZbAcUJ1pMQ1CMizPjWDbT1Ou8ErVqdbof1qb1gI31DCmKHNbmHA9+hklFPbYWp YScdHfs9GpTFCawM5Ic0oUNnDnshVNS0Ck7TlDLBF4V3Xq/MY+fIuFJuycZdE2/r1i3C Y+/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=T0aH16H0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a24-20020a1709027d9800b0017835995e40si8924687plm.414.2022.10.16.09.25.23; Sun, 16 Oct 2022 09:25:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=T0aH16H0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229930AbiJPQYM (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229889AbiJPQXo (ORCPT ); Sun, 16 Oct 2022 12:23:44 -0400 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9FE619030 for ; Sun, 16 Oct 2022 09:23:40 -0700 (PDT) Received: by mail-qk1-x72b.google.com with SMTP id o22so5398618qkl.8 for ; Sun, 16 Oct 2022 09:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YCISppMzdHX/DQsNZveHbYvxNRQkJmaJ4AIlPWlWRgs=; b=T0aH16H0gjhc829ZwPg95man8+E5dBVOoqyeuStiVen/WktGOsf2hdhMwjW3MNDJyp FDFX2078EezAeccQNhKIglhXgb0z9Oz1j0CTQoHB9Lzas0JzQjGMNm9KsRm4ye1VWVmo mkr8Qpug60rSgHIIQDOO73wfKOStUXrQwmYos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YCISppMzdHX/DQsNZveHbYvxNRQkJmaJ4AIlPWlWRgs=; b=K+7aqNAahp023WOnXgR38nfxwCCTV8cgEKybAUrgWYqgInkjQuk+smX7dAMrbFmAdE AZ9+m8AYSegRz1ulSIrFc7lkU7cMxKrpUureFEoVEjBKSgVOHQITNbczE/VwvvMwNd1i sHRB7Pc30533ES85xD4Jq24+dZpfGqPm1FjjFmQeYjle9SZzNjFeaYsrHauwo2U5W/38 4o1icuMtVs885MGNbxyPFxhbpavnVbPGv61+UQS0m4SgdTNzpovZ0VVYCeawHJP7XigP JfbIJd+7RtUfDbR7tRgCVv8eC9H1Rg/dRXNSSW4Utq8lH4O6T+rJXv/iTCYGh1CfH41x xD3A== X-Gm-Message-State: ACrzQf1eAYl+PDm3lDHW+K0O2xThkJ8LKICjTHj2GgFztwOn928aQ0j1 Fnw8eqAEEn4b+TJInTXXI1WFbA== X-Received: by 2002:a05:620a:4252:b0:6d8:9682:c490 with SMTP id w18-20020a05620a425200b006d89682c490mr5058744qko.4.1665937418550; Sun, 16 Oct 2022 09:23:38 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:38 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 06/13] percpu-refcount: Use call_rcu_flush() for atomic switch Date: Sun, 16 Oct 2022 16:22:58 +0000 Message-Id: <20221016162305.2489629-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862117311047565?= X-GMAIL-MSGID: =?utf-8?q?1746862117311047565?= call_rcu() changes to save power will slow down the percpu refcounter's "per-CPU to atomic switch" path. The primitive uses RCU when switching to atomic mode. The enqueued async callback wakes up waiters waiting in the percpu_ref_switch_waitq. Due to this, per-CPU refcount users will slow down, such as blk_pre_runtime_suspend(). Use the call_rcu_flush() API instead which reverts to the old behavior. Signed-off-by: Joel Fernandes (Google) --- lib/percpu-refcount.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index e5c5315da274..65c58a029297 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -230,7 +230,8 @@ static void __percpu_ref_switch_to_atomic(struct percpu_ref *ref, percpu_ref_noop_confirm_switch; percpu_ref_get(ref); /* put after confirmation */ - call_rcu(&ref->data->rcu, percpu_ref_switch_to_atomic_rcu); + call_rcu_flush(&ref->data->rcu, + percpu_ref_switch_to_atomic_rcu); } static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref) From patchwork Sun Oct 16 16:22:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3061 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077956wrs; Sun, 16 Oct 2022 09:25:08 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6Bf5odmhRTSZQVxT5LSoClgFSA/1jaFH2sCdjiBHCAyAEfYa/QpSgeCyjtqOajfMv+EG8v X-Received: by 2002:a17:90b:3502:b0:20c:e1ce:46d6 with SMTP id ls2-20020a17090b350200b0020ce1ce46d6mr28521490pjb.94.1665937507811; Sun, 16 Oct 2022 09:25:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937507; cv=none; d=google.com; s=arc-20160816; b=SBAcd4PRZoJUcW0jlxWw45EBlxEP+R5skpWCE95dw8KAB18VjJvFbHb2PYPjRmIVCJ pTQaGbqRt8QOgFRR+RWiDjOlVOXpBIq8vD8vt6p9wb/nY6/oMsbOeaCtGoz37rkS02+v BLwQsFU3SgXFrb6mUyrfTKHXxHpvz+2aXJdutMgOYzL3pDWgWtye67vrBUhCP1ZGV1T/ cxF/kOoTCes3YyYfy7EGgJf10lfnf/0DWPgrpcdZFVTSCwxyejVNL+HJFl23eKVo5d6U Wbrd4mo87JQIuCePRZrJpiTvBRNSHXuR1HXfJPyGjzWKHrOu3V+oMGrXlPzXJPHbZmJy YHjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pLBYpsFEJKnpduUs5aM0LOOoZhLo8S1nCnZi0UB2KAw=; b=wGJdd5Gj8HM7V8bz92sQhc4247WTq3N0VVQYZ/ck06U4H0UjpgcCWme3xmIjcjGeye lrs5CU5yP4OpuT2Zgty5NmDtrTEumKys8H9ZajSI6cqmpzJoB/nllMzJwwFx9+u9G5Ku ypdNE3mL9ogXOTNafnFY6Zf/54eMFI98ypQGoSRL5+sFjxsMLhYTiEC4pyTlhloS4WMO sCMzzQ+QCEtDoskIAliRBoQiUapmeYr3u0Q4x+8dh5PTrXkMebQJ9kiTfrG/9CfkHJBg g7T/0B4X8KTTXtuav1ntH1Xxbdn8vHjV1/zpDuIPdZ5G9T5VEN46S917jw8X2AIDvnfr ID4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=RCMpa87j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id np18-20020a17090b4c5200b0020a689fcb8asi9961748pjb.28.2022.10.16.09.24.54; Sun, 16 Oct 2022 09:25:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=RCMpa87j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229871AbiJPQYB (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229911AbiJPQXn (ORCPT ); Sun, 16 Oct 2022 12:23:43 -0400 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE70E3BC5B for ; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) Received: by mail-qk1-x729.google.com with SMTP id t25so5412050qkm.2 for ; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pLBYpsFEJKnpduUs5aM0LOOoZhLo8S1nCnZi0UB2KAw=; b=RCMpa87j4Yk0l/K9GgHa918zJweD8ZC5bp6tXAzXf9jjm4FbawEWZfd0h0FYrpS2i7 yTr2WMUUyIdKn8yUCbTs9Ah2xlZvlZGv18SQK4/OmJP03NvxlLLMMPZ8ceelFIdf6GXh A8tr2UU67BYyh2oPlk70FXcxeCsfLoCpdnitU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pLBYpsFEJKnpduUs5aM0LOOoZhLo8S1nCnZi0UB2KAw=; b=vrDUyhQMUjv8Rgv1b/GrDkJIu7nmZ58I0KmH7kCpAIHmkCexiF2NHSKF+85a22EsWc zLJShEpKlyf4c8dtd+DYY6YVLuB1nubbbbVSySYavIdYFhupDkTGVc/7/3G/6mUcjtl6 B2KPa1wlAfRD9rNE2PcPjxcnmRoJq2+ij7/zMAz45aAbIwHNJb0yPoXdejQp0KlxFDAL zrr10i9euGHmtTSz/jk37fxNFOPJ9wGQ50z3d5VTRNoorKRoecX0Thv2ZRMcJQnrzsUP NW3ivvbP0+ax36uV0zGNtDUTfS85opOcyR1I5J2FUsQLPPbn50P9Po5239VU76WjUYFc OzUg== X-Gm-Message-State: ACrzQf12npA6VdiSsJxsbmrSXlH3rpb/iEkrDvfb1dc735+SYwrUJPSq GcF4fV363CEw2fZtwrK5XuWHMQ== X-Received: by 2002:a05:620a:2618:b0:6ea:908:120e with SMTP id z24-20020a05620a261800b006ea0908120emr4939543qko.645.1665937419224; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:38 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 07/13] rcu/sync: Use call_rcu_flush() instead of call_rcu Date: Sun, 16 Oct 2022 16:22:59 +0000 Message-Id: <20221016162305.2489629-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862087842279527?= X-GMAIL-MSGID: =?utf-8?q?1746862087842279527?= call_rcu() changes to save power will slow down rcu sync. Use the call_rcu_flush() API instead which reverts to the old behavior. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/sync.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/sync.c b/kernel/rcu/sync.c index 5cefc702158f..bdce3b5d7f71 100644 --- a/kernel/rcu/sync.c +++ b/kernel/rcu/sync.c @@ -44,7 +44,7 @@ static void rcu_sync_func(struct rcu_head *rhp); static void rcu_sync_call(struct rcu_sync *rsp) { - call_rcu(&rsp->cb_head, rcu_sync_func); + call_rcu_flush(&rsp->cb_head, rcu_sync_func); } /** From patchwork Sun Oct 16 16:23:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3063 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078055wrs; Sun, 16 Oct 2022 09:25:28 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4v61UKfBDFlYwAy34JJD2th2aREy4iJ3iRjrLiAqjcUdoB/x9O7whZD8nrtPhWhjKDfRor X-Received: by 2002:a17:903:41d2:b0:183:6545:39c4 with SMTP id u18-20020a17090341d200b00183654539c4mr8137677ple.124.1665937528123; Sun, 16 Oct 2022 09:25:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937528; cv=none; d=google.com; s=arc-20160816; b=TO/KirvikkN6XOKzmBedxN0ykb9EFcgVSpZmHq1f4otMmcWT6aE1uQGrzhyIdJSf/T okOyy/K5GUJE5vNfsGeI/bBdsbPpqP9VPpfobuMZzHwSpTZeYncmGKzwosbfzuNYsweZ 2xXUDv5UMMIKtJxst360fAoGigE2NVD/+4NPIX24mdbNSvfKfwy6xYHpZn9m9yqoJB+7 ipZnKRiaI5x0zxTQvbBvkQH0UJxr0i3NxTvSN/IbdmdsV3VwMNL974aE/Kv7AMKNRbLA hLGDabYNVjSVj4D2po0X1xRpNEkJqvSEsDtmUAIxYSAxwLhdh9zgZM2rlnGss53XIoa1 y+zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IEECeHAL+4jB0ct1Miev7aIBczUJQZG4eKLvCmwiPRs=; b=NNuHgmxDc4NFliLvxqHl+rmpU4L1Q9jT8Z2q5xErd4CMItMGHkFK0qSd5r0zYjr/PK Eg9wF5dZQ2emXqed/kMlPoZQ5jpxqGjFx+TEEYuyEFOU+6iDYS0PoS7yIBBV0QEN0n6N 52KEql38H+fmkbqs6VasXpHK5Hk+neaJmbF7+d0A52/xilIZDm5LReDijx4hc9wMzF4k 8Lq6+utYrLM1Kt6Dl3YWulQAdtXcMCQvbO5j+Xu8AXzNelAofDSofTZpT+HSUODAsUEG l/ZEGKduJVTtIvEG3iYMUxATalKBUBaBxOLHkhkO+DG22pdoZs07vP+wKeZb64q/SmfD Dmpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=vycGQD6t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h7-20020a635307000000b00450200a1078si9810074pgb.853.2022.10.16.09.25.14; Sun, 16 Oct 2022 09:25:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=vycGQD6t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229873AbiJPQYV (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbiJPQXo (ORCPT ); Sun, 16 Oct 2022 12:23:44 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 131153D58F for ; Sun, 16 Oct 2022 09:23:42 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id m6so5404748qkm.4 for ; Sun, 16 Oct 2022 09:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IEECeHAL+4jB0ct1Miev7aIBczUJQZG4eKLvCmwiPRs=; b=vycGQD6tA80I0QDKMFJTpotcypYOCSzuyXbA3dDGZ+CQ3qQXHGtK09Uyu/ySS6wEgz R1JZoCZzhHxNamY1N5+s+ejOngl986X9jX9JUINf4V7BiHU5I2RKenhW+E7rJP5jWT0i Uq4tY4W2tzoj5+A+DEuaoGo1/Ikue+EBnX/nE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IEECeHAL+4jB0ct1Miev7aIBczUJQZG4eKLvCmwiPRs=; b=H0/PhJFWjWTTXq++noav93GpTuduGkSFMYlBLjXBXWAJ9KVUSwgb0YFd6+h9yP4alg +3+jO8YksdPdX6FTgSe6QczyvPwmpFLbRix59GNICtVq7c4B2VPKJuVzf9hQnsVsgbEX N6TwWpqsIHXi6jeK2sFMKDzISnuY9ZGDlcl32NXIZ0q53wEdFBjQXusVeC43A28MHmD6 f45efxqPby5mt7rJQgFLWy+4q56dRsAtYg7yhFvVD2k8XFYXdSVlfquL7/goyf/yAeeY becNFsDEVElg4HPLXs7eRaKtD5ephPlTX5Ke4XS6WRrn8ULeJaovJsctslXwVp9E6zx/ 2kCg== X-Gm-Message-State: ACrzQf2ZeYtMy0WgdvRJnWplOxgoYcsbGONqYRGI5eE1so4RwiQ2a7Tj 4vGUm1lT3JEMggE50LP5oc9J8Q== X-Received: by 2002:a05:620a:e:b0:6ee:86e5:66f8 with SMTP id j14-20020a05620a000e00b006ee86e566f8mr5013762qki.163.1665937419752; Sun, 16 Oct 2022 09:23:39 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:39 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 08/13] rcu/rcuscale: Use call_rcu_flush() for async reader test Date: Sun, 16 Oct 2022 16:23:00 +0000 Message-Id: <20221016162305.2489629-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862109576523767?= X-GMAIL-MSGID: =?utf-8?q?1746862109576523767?= rcuscale uses call_rcu() to queue async readers. With recent changes to save power, the test will have fewer async readers in flight. Use the call_rcu_flush() API instead to revert to the old behavior. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcuscale.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index bbdcac1804ec..0385e9b12399 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -176,7 +176,7 @@ static struct rcu_scale_ops rcu_ops = { .get_gp_seq = rcu_get_gp_seq, .gp_diff = rcu_seq_diff, .exp_completed = rcu_exp_batches_completed, - .async = call_rcu, + .async = call_rcu_flush, .gp_barrier = rcu_barrier, .sync = synchronize_rcu, .exp_sync = synchronize_rcu_expedited, From patchwork Sun Oct 16 16:23:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078364wrs; Sun, 16 Oct 2022 09:26:36 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4zRhGajkRGYVozgLWgF7ydj0+DTz+jOUCPmrUAAeyAjs3JyWY8kLzanOGcpLFmmD27MD+E X-Received: by 2002:aa7:800a:0:b0:565:af23:f5a4 with SMTP id j10-20020aa7800a000000b00565af23f5a4mr8310286pfi.42.1665937596523; Sun, 16 Oct 2022 09:26:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937596; cv=none; d=google.com; s=arc-20160816; b=FDUQ/LTnQ+O/eK3jIGwrzC2G0x10RZGIMwlQED+/aw9p5qaXFZ/YwEexKvnUtuDwL5 kO31nMxqWRX45EAqxewP5lhHefWm+ztvNMne9bLklDRBo46eH1Z/MgcVLuDrBH07HoK0 WiIYRkfHc4T8iWC3nLsjuRswKoPpB+FrJ6px0kRRwuGPtYZC/anPXwNySSlOm8KvE1Dz 9+v0jOLaFAJ6laRc7uqcIDGhHdTcNzLfL+Irny0ul8m1PtBKyo54CRv7yy6qbIsF+nL4 2VBGG3uAy3fAE8OywO7QHGLcSgEzYXlnX6gujR/hzJBx4R5TXvnAu8lSsQoELzGbdE9G Plkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U8qbMoDbUFjsjnsgxbcBmmoHnL4WHNlLEOswqF5VihY=; b=BtrAImPidYyuR5n+mNsGRIlhvhZnBrz24z+pv836Mot5OjcVMDA20woS6b1w0vaNXp o0Y3svxGRJXke8Ex5DVOlvXUNvBYINALLkSBfq6RZdKPRZf4TlxbuEWIclpIxmii6px7 01knEkggupy5lZ2NkYo2u5pkJG+rBZyS/2RZwqACDNbplo4WC0HrRPXL2Y7wxEh/Ns8K KKLgCV/+T6duquXu7JWLW0BQDhMP94nrMIN5Jcx2vyt1PG6CVTwGwK4Snyo3AHFpQuOH QWRc3GBJapH8B+7e58Gm4ufL3eQEovf4wT98YkYXpONQWfGNTZlc3FepEt6ainN1yaMZ WHBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=cPVn8FK4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s13-20020a170903214d00b0017d22c23b3esi7955606ple.309.2022.10.16.09.26.22; Sun, 16 Oct 2022 09:26:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=cPVn8FK4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229933AbiJPQYh (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229947AbiJPQXt (ORCPT ); Sun, 16 Oct 2022 12:23:49 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 611A53ECC6 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id z8so6500213qtv.5 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U8qbMoDbUFjsjnsgxbcBmmoHnL4WHNlLEOswqF5VihY=; b=cPVn8FK4CZBrAZ54M7KyfKKXkgSC09g74rPm8n5OxaX0too8i+UKW9oibRperEQbdB caGBzcF4Y/av5TTpiExqn2n7TqsKqj+ELYhFHnNn+EX5s3NPocr8W/EUabv236NHx3at EM0iYxYDDNzjjFxe4XpBuqZJ9v6kXZ2zkQ9gg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U8qbMoDbUFjsjnsgxbcBmmoHnL4WHNlLEOswqF5VihY=; b=jkmamF0slqXl+sDNUaEJb2VuDSokeo0lPVfx3ywiCPpqxnkdWSYJRCESby+kpIAZxl q6lI+9lv2vf/T7JfXckuUHZ4UKWhjT6X0NYNpbut+TlB7ZaC2pLuw8gRzezHYPNti55e p+jMR5gJmO+1xnkPeQ2yrDkkciMNzs1Ma/8LITw7h8p4tCB6dBndMOAtKHJdrdAo14gP IvIXorvdr1wppTWaUn3A0e+ASNN+KL8PjJHT/jPlaJeglok3F+EUN0Ps3WkDud5otCbH rs+RhVSLeI6R7nPPVDUNGyf0DvHsk3K+pCJxyU5iG2+xLyspAjSWHy30Ahuzteq3Wxmu W3YQ== X-Gm-Message-State: ACrzQf0iHrt0nql85MDaR+ehdLPhqocYtLpt7MoKvO6puzREiyNlgdIv x2uJNDu0OMueO/9hIkkeS4+0VA== X-Received: by 2002:ac8:5c45:0:b0:39c:dcea:c552 with SMTP id j5-20020ac85c45000000b0039cdceac552mr5547567qtj.128.1665937420317; Sun, 16 Oct 2022 09:23:40 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:39 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 09/13] rcu/rcutorture: Use call_rcu_flush() where needed Date: Sun, 16 Oct 2022 16:23:01 +0000 Message-Id: <20221016162305.2489629-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862181473122241?= X-GMAIL-MSGID: =?utf-8?q?1746862181473122241?= call_rcu() changes to save power will change the behavior of rcutorture tests. Use the call_rcu_flush() API instead which reverts to the old behavior. Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcutorture.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 684e24f12a79..fd56202ae4f4 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -514,7 +514,7 @@ static unsigned long rcu_no_completed(void) static void rcu_torture_deferred_free(struct rcu_torture *p) { - call_rcu(&p->rtort_rcu, rcu_torture_cb); + call_rcu_flush(&p->rtort_rcu, rcu_torture_cb); } static void rcu_sync_torture_init(void) @@ -559,7 +559,7 @@ static struct rcu_torture_ops rcu_ops = { .start_gp_poll_exp_full = start_poll_synchronize_rcu_expedited_full, .poll_gp_state_exp = poll_state_synchronize_rcu, .cond_sync_exp = cond_synchronize_rcu_expedited, - .call = call_rcu, + .call = call_rcu_flush, .cb_barrier = rcu_barrier, .fqs = rcu_force_quiescent_state, .stats = NULL, @@ -863,7 +863,7 @@ static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) static void synchronize_rcu_mult_test(void) { - synchronize_rcu_mult(call_rcu_tasks, call_rcu); + synchronize_rcu_mult(call_rcu_tasks, call_rcu_flush); } static struct rcu_torture_ops tasks_ops = { @@ -3432,13 +3432,13 @@ static void rcu_test_debug_objects(void) /* Try to queue the rh2 pair of callbacks for the same grace period. */ preempt_disable(); /* Prevent preemption from interrupting test. */ rcu_read_lock(); /* Make it impossible to finish a grace period. */ - call_rcu(&rh1, rcu_torture_leak_cb); /* Start grace period. */ + call_rcu_flush(&rh1, rcu_torture_leak_cb); /* Start grace period. */ local_irq_disable(); /* Make it harder to start a new grace period. */ - call_rcu(&rh2, rcu_torture_leak_cb); - call_rcu(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ + call_rcu_flush(&rh2, rcu_torture_leak_cb); + call_rcu_flush(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ if (rhp) { - call_rcu(rhp, rcu_torture_leak_cb); - call_rcu(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ + call_rcu_flush(rhp, rcu_torture_leak_cb); + call_rcu_flush(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ } local_irq_enable(); rcu_read_unlock(); From patchwork Sun Oct 16 16:23:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3065 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078190wrs; Sun, 16 Oct 2022 09:26:01 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4znvkkZQnMLrHEQU3KtW2oXI6MU9mpX74kFBpyTVUiyyPgVgkaSrKMPLbsRWgN4bs0eoDR X-Received: by 2002:a17:90b:2395:b0:20a:ddd6:d4f2 with SMTP id mr21-20020a17090b239500b0020addd6d4f2mr9105073pjb.158.1665937561709; Sun, 16 Oct 2022 09:26:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937561; cv=none; d=google.com; s=arc-20160816; b=JcZy01ZTSnHpdrzxhgLeYMWrSCv1AgH02M14NoWQUBRDIr2nkUdHVzC8MpUaKrKl4h 5QzzgPNkkCWoeAOkTPnZOMeZtGIkNrRO0FLwgStiQCMD/4y83xufsn6klwMelCfbSf4g nTJT0q/aRqZSkbnDbWKXJf0Y6bAjH9bIfcHEiHGE8C6zXLe13snAU8yKBc9sMLfyYCgT Scdt59chXVacgPvbjmSS+JQSM80DEQ53F0qK8fhzghNuL/e/frGtMyrMdzbW+wt1aM2h gyu3ZTpb8YQygrJOunSPiu6R0MffcaBGSRAIReBbaknKHaUJZnXUj6QjiIXlIzZd2xmW wEKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jd4X4oYun9/nS5S7eG1Y6MJBlHQXAJ22sEc4uBevOrk=; b=w2YXWtoEd0Oi4LtViAozSTDyaIxpwzgbw+hK4eu1vy4GEpXOdYxaBZojcE5otv4kbp EeecU6bL5xeeOixk+nbBhRfb+JUbdF3djBIV7d1ohBszSSYcDpk3qEUlV4ZpB6Cqevs7 GRCOawpQ1pKNMVlUL0zlGU1LEKgnl+pO8HQxMWHAEHZ3o51qKGOKclTdouNXrfas0Fwt 6fAx/kyXr1V0dfWQOFNE4vqOtexU0ZASIkDDhoZp69OlIOR6d2Qocpc5IfKFgptxtrYe 6xqfy5eHtFgzFdNWRDfLicraar/sCk7iHkVvlACQbgQ2qw0qHhGbv5HBbkKXh9biJJQr 9jxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=mrEHqR75; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t190-20020a6381c7000000b004606d9040b9si9645707pgd.649.2022.10.16.09.25.49; Sun, 16 Oct 2022 09:26:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=mrEHqR75; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229961AbiJPQYn (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229897AbiJPQXy (ORCPT ); Sun, 16 Oct 2022 12:23:54 -0400 Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 972983B720 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) Received: by mail-qk1-x730.google.com with SMTP id a5so5401155qkl.6 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jd4X4oYun9/nS5S7eG1Y6MJBlHQXAJ22sEc4uBevOrk=; b=mrEHqR75/ASzUJYL2Vr1FpBcCW1ogrf9oXsAim5CnXNXTSdkazxj+/9mKkDSVa01Rp A0I7JfTL5YB/qYLf3G92MvHWjYyk9MuBZB+Y288CC7uABNd34F84+s0jyORWJ80iWfzM mmPmsBik655Fuf0CGK0nq+Rn+D1GJYLHvEr0Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jd4X4oYun9/nS5S7eG1Y6MJBlHQXAJ22sEc4uBevOrk=; b=UeFMULegKz67i4IHeOVMs05fVSJahvXaBKMKKi1W7+3ZZDahgr+Y1ffS2cptYkbwWT yDJrGDSXZEOhuN3O51+VGKzgAaltEYdQ8cgbp1ano8m6QEk39ucTXb5c63HZnQSPr1ow QqZodSaWaE4LAMn4s+UiLmvhMO3JwsnqQDC0K4dKj1jaeFF6+lTlitahktbfu3aAlkIE 2jm19W2H7BA4mAG4PWmerlEq4cvlDtd8AVVDKo4vmPRGHMVqDMi6QdvlnvdoIeo7YQBL Y1bxmVzpg1HLsC0ELmEpRSGPMyMxFbLZOn3Q9AUPe42lXPfJ9FDQXfekrrY+ccltE6qp LE5g== X-Gm-Message-State: ACrzQf2aCe8G7MT1XbNerQkooNV8MFFTRGgjpBpkYQEcgc4LXG9BAsZG OoSFlu8YPDu1NdTAM4+VhgO70w== X-Received: by 2002:a05:620a:2913:b0:6ee:d427:1c5c with SMTP id m19-20020a05620a291300b006eed4271c5cmr5142629qkp.484.1665937420846; Sun, 16 Oct 2022 09:23:40 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:40 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, Uladzislau Rezki , Joel Fernandes Subject: [PATCH v9 10/13] scsi/scsi_error: Use call_rcu_flush() instead of call_rcu() Date: Sun, 16 Oct 2022 16:23:02 +0000 Message-Id: <20221016162305.2489629-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862144705795791?= X-GMAIL-MSGID: =?utf-8?q?1746862144705795791?= From: Uladzislau Rezki Slow boot time is seen on KVM running typical Linux distributions due to SCSI layer calling call_rcu(). Recent changes to save power may be causing this slowness. Using call_rcu_flush() fixes the issue and brings the boot time back to what it originally was. Convert it. Tested-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) --- drivers/scsi/scsi_error.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index 448748e3fba5..a56cfd612e3a 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -312,7 +312,7 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd) * Ensure that all tasks observe the host state change before the * host_failed change. */ - call_rcu(&scmd->rcu, scsi_eh_inc_host_failed); + call_rcu_flush(&scmd->rcu, scsi_eh_inc_host_failed); } /** From patchwork Sun Oct 16 16:23:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3062 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078046wrs; Sun, 16 Oct 2022 09:25:26 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5Twomwixk3jE+9YyTm5akXC6SAZf9TABkWw7w+0odQMX8/r6MuhM2fePS9lZoGPpUFibE4 X-Received: by 2002:a17:902:f60b:b0:178:6a49:d4e3 with SMTP id n11-20020a170902f60b00b001786a49d4e3mr8068007plg.75.1665937525769; Sun, 16 Oct 2022 09:25:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937525; cv=none; d=google.com; s=arc-20160816; b=jm/r+V2r1HVyhsnQTIQXlGZq2BjnSp+EI1pD5N2XlpW3wX9UEJ4btxMMxyYqDmyQub Bpqp2lfjaLxGkvqoKEu35qo+F940hgqp+vDtZvWNZcOv/LuN1AbYZXgwu4BxGJFXOH8d IsBAWMx8rW1kREE35WMaFlcStvD7j22gVFM91wq0ve5J8vU5zvxKD1j96yAFC1LgZHiu nzuMePg9HvhZfLGVOhyE+9gQneN0WIc1t37SGMVZbKsdTUlVJVx7m2Xk1kygCJ29UgME llcwMRx/Rk7kTylUUHG/tXDFC8TAISywQagF+kj89JTm96aXHlmi8ZP3R1dcyRlBARO4 1ucQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8IMqQmd/XghF0vc5rGTchLEnwifIWdplO8ZK5ZR9Yzk=; b=A4KosX8+No8XX5PJl9IuIQyIK8hX5kSYIuiOkT7Mc2H396C8gIQolad4u7uiFZZDIM 1mcT5dcHsXLJT1SHyViIpFv17V6seysMHSO5XV56P6BBYLPXKSkYfsA0OGxLI6yYt8aW p3otdaVu4cMeiMf2I/jb8daM0507nWNgkerREy0YrTohXC3fDHbmt5hgnxqgudw6NlCt tlN+0Tkag4t1Qsqn/p1h7zsU/F/ZFzjW/PhahfL1usbujoFIcPDEBpHqBwW9QAYejiyW RpXOBhcpzgH8Rud4WpgLtmVZTYck/sdy+mSt+zmzlKlvVunZLULeTZReHRGIm32gKg6Y u2ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=py7m4915; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 132-20020a63008a000000b0043adff6d43dsi9324227pga.583.2022.10.16.09.25.12; Sun, 16 Oct 2022 09:25:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=py7m4915; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229772AbiJPQYd (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229764AbiJPQXy (ORCPT ); Sun, 16 Oct 2022 12:23:54 -0400 Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF9A233E22 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) Received: by mail-qk1-x732.google.com with SMTP id a5so5401159qkl.6 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8IMqQmd/XghF0vc5rGTchLEnwifIWdplO8ZK5ZR9Yzk=; b=py7m4915l5o5WsLd4bMO+78MLiz5ALzpVVpmzpKDyUl2p5IK12I+DK7uMK1ctklRZc 0ctsYyMCESSdCSlRIci8QFRlMwzZFyL/wDKPbOgLpI9GsDGWmO+UREym0nzKAi3x0z06 1U5rjt6VxS5AmaRVMN/lURrp/Fv3eAx599HvA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8IMqQmd/XghF0vc5rGTchLEnwifIWdplO8ZK5ZR9Yzk=; b=Y2vm1rSHed6KB0wRwRi891ImRZFu1IwpSkFqsqViz9qABsNjKGZYnaqRGOrqnCDQ/z MQIrCxNNmJA8wVoH4v6Lk+mMkFg4PpycP45/arPZUwDwR6An1IXNbMAVwiqfq18E1Sxd e8ToBd1piLn0xB97uckva0/EpUb0dWf97/jq0MmFy7fTCM4pokgjGD050L0rehSsfCXs StXWeF53ZQ9BEC1lUB1CSp4JZgah9hb/OBKGyDv44An23ZNTa60fIBUC4M3yh9tPgTw6 n3iWaXjJzLj3xP9XVmgddOG46KfQVQDjszAQQ2t6u8rmWoElEUUYrcDTpoT+FHC3X/n6 wJ9A== X-Gm-Message-State: ACrzQf2kSXBAIfVoiCuPziHEm23hnTWjsVrEowU/Y57JXQpIBHrhJAuX AXAD8GJlFyoW/yfbNP+AeT3o6A== X-Received: by 2002:a05:620a:2045:b0:6ec:bd25:ea67 with SMTP id d5-20020a05620a204500b006ecbd25ea67mr5004470qka.85.1665937421414; Sun, 16 Oct 2022 09:23:41 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:41 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, Uladzislau Rezki , Joel Fernandes Subject: [PATCH v9 11/13] workqueue: Make queue_rcu_work() use call_rcu_flush() Date: Sun, 16 Oct 2022 16:23:03 +0000 Message-Id: <20221016162305.2489629-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862106544840755?= X-GMAIL-MSGID: =?utf-8?q?1746862106544840755?= From: Uladzislau Rezki call_rcu() changes to save power will slow down RCU workqueue items queued via queue_rcu_work(). This may not be an issue, however we cannot assume that workqueue users are OK with long delays. Use call_rcu_flush() API instead which reverts to the old behavior. Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index aeea9731ef80..fe1146d97f1a 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1771,7 +1771,7 @@ bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork) if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) { rwork->wq = wq; - call_rcu(&rwork->rcu, rcu_work_rcufn); + call_rcu_flush(&rwork->rcu, rcu_work_rcufn); return true; } From patchwork Sun Oct 16 16:23:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078287wrs; Sun, 16 Oct 2022 09:26:23 -0700 (PDT) X-Google-Smtp-Source: AMsMyM72JHxRp/qumoI+kr2LCDJNVccsEHmXHrwLjQwiAwpuwLnqsJYsZVDzn1mL1hhVb/YhsDnH X-Received: by 2002:a17:902:e212:b0:178:5c:8248 with SMTP id u18-20020a170902e21200b00178005c8248mr7941127plb.102.1665937582905; Sun, 16 Oct 2022 09:26:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937582; cv=none; d=google.com; s=arc-20160816; b=X2UYMBWPbDlelAF86TkQ/Smsgtk+/xGNygYxIZmy3PvXJ8S5cK/OLB4Wi0zpiUj1sw ExFa9FT1Mif7PjDqw/MRa1LgTCZoxshzpH92L31E8iD7bC8zPo19/m55YymQMY0ecMHW nsYSHtBIKuKokqeH7pdg73L0N0uuyuXdbUC3als2v1TZLs1jvVtVshitbI+Lc/ICDgVy DWhdGm9Uz6XvDUWEBGE3qC8yu9Et6tDnza18uQriCmSBEcJRGiZbMIrRLdBGqVeU+8DW xHjoJI7ItZsqOHvx6krjQYoOIWvyasHIHdVf90/OCR9/CvYZGwDWQYJ0phgJo7ekZYDl oTYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bEtsx5s7XOiwD9ITg4bkmVjdZ73X1TK7P0DFf+XigW8=; b=lkC08NcXVlRMIgECmD3yJGAWWxHac9HOjzt7gbxFmltaB4syFifatOLUtziiQNnjEW k63ZPEG/DhkfP7DcO+b4pMmXQGW8+muqpqKACDIUIHTcwXUhPI+MGj75bLx8u9mwCyVo EUgynUv4kwG9pL7Fi3BYsBNL6pFJ570oQvW/N7vrKP9weXEo7fJWuydyXHzZt4cOTI6j Z5OV5nyEJjRVUulJT70HfodCjdBUNQ4dYUUQ+WkqLC5iuFQXixC/L55g8M5gbO+OO9ze 7v0QCKd4sbflPxEl349/3i1vnXK+UJc4B8CACkIlDhM//KneZZKVF/XJM0lzzAGL+8O5 Vqzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=pOMzhiN5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cp8-20020a056a00348800b0054096da12b1si8112362pfb.39.2022.10.16.09.26.10; Sun, 16 Oct 2022 09:26:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=pOMzhiN5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229622AbiJPQYZ (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229830AbiJPQXr (ORCPT ); Sun, 16 Oct 2022 12:23:47 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A98A3CBC7 for ; Sun, 16 Oct 2022 09:23:42 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id z30so5385093qkz.13 for ; Sun, 16 Oct 2022 09:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bEtsx5s7XOiwD9ITg4bkmVjdZ73X1TK7P0DFf+XigW8=; b=pOMzhiN56m8msun4IsqBgxhAdONJhpbswx/IqkVNkSzTeoZyLosoi3Vb7h3daV57N2 tbMeo2SLUtbhULfjpnA/VGwusi+VwdrN9fM3xRm7qFybgF62fYRKEGDPgdni8VnOptlp 3VpNZBINyx4R1NGkCDqif21jM8JelQbOO5AKQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bEtsx5s7XOiwD9ITg4bkmVjdZ73X1TK7P0DFf+XigW8=; b=MpW0/rUP0t/PDAcqVep/s2un8cQoHngxV3jzV+i8UMWVVjqP1J/FgKdcP86B8Ay0H7 hc8BhPMCH6AFbuPfvQj6uO3+63p2QFDFzHh6wg2LhStkosxVs+CBWqjkfLuxbA1Z453+ EIj8TR0tsejDNnTjw9dfzuENuKJUtX5+ouIGnAbhB08YWBP8wpEjdVwPIdo/421kRnpT vYWrbkbol1XRRxVO3Y8yaUfBoJTxH8dCpfo5sSeMOaqkXllm7U5bwi3Jlh3vihB2Pdzz CBQjzOf3wsSC7RRfsElrn60K3VZraVL2Hf5cIa7YWCOZhCDfa9yg04OmOoQPhzyjHaqQ eKpQ== X-Gm-Message-State: ACrzQf0dnca4AthGc9pNlW+qRaoNvH5W3WuSnjsnc4JZur0QbReCQwhl +DoNJIr+r7kD4DppE9bVDJWUTg== X-Received: by 2002:a05:620a:28c1:b0:6ee:d4f1:21df with SMTP id l1-20020a05620a28c100b006eed4f121dfmr4953922qkp.724.1665937421930; Sun, 16 Oct 2022 09:23:41 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:41 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 12/13] rxrpc: Use call_rcu_flush() instead of call_rcu() Date: Sun, 16 Oct 2022 16:23:04 +0000 Message-Id: <20221016162305.2489629-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862166733798277?= X-GMAIL-MSGID: =?utf-8?q?1746862166733798277?= call_rcu() changes to save power may cause slowness. Use the call_rcu_flush() API instead which reverts to the old behavior. We find this via inspection that the RCU callback does a wakeup of a thread. This usually indicates that something is waiting on it. To be safe, let us use call_rcu_flush() here instead. Signed-off-by: Joel Fernandes (Google) --- net/rxrpc/conn_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/rxrpc/conn_object.c b/net/rxrpc/conn_object.c index 22089e37e97f..fdcfb509cc44 100644 --- a/net/rxrpc/conn_object.c +++ b/net/rxrpc/conn_object.c @@ -253,7 +253,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn) * must carry a ref on the connection to prevent us getting here whilst * it is queued or running. */ - call_rcu(&conn->rcu, rxrpc_destroy_connection); + call_rcu_flush(&conn->rcu, rxrpc_destroy_connection); } /* From patchwork Sun Oct 16 16:23:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1078261wrs; Sun, 16 Oct 2022 09:26:18 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4zUWwWJST4j78sJlZfvCcGg8Xx0s/kZ81ZFammBO1pUkANFRn4xozi+skWxP1aWwDROWIT X-Received: by 2002:a17:902:e885:b0:183:dcb7:c4f8 with SMTP id w5-20020a170902e88500b00183dcb7c4f8mr7707886plg.160.1665937578127; Sun, 16 Oct 2022 09:26:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937578; cv=none; d=google.com; s=arc-20160816; b=yvy/9VXxPpB0larh6qmYE1hvT9t/URRj+5O9kKYBqwuFQj8spk1TovZw4A7MymzSpl S/MUY4jrnlxm9hWuSYBQuS+LnogLG6mMu06LQwtRbHawfsLZIO2td0TAkrDQP0QD2POO IGsvY9m8pvn70GMY1cz5FERCsL9+AMu/A+b1pRCagrJryFqROenMc3/v5P9xwjkBY6cK C6Ofo1l5qwVEeB3wv5ZAxaJiP+lIrpLtQHjS+TIejYaeV/YoCwBACSWTFdYyQOVw0oGU 9C9f2k1QofuUrSlGWAk2+RJ3ThkQDipM7HJUMq78JnKBap6vOfQv9p24Z+xCsJflrGRp FrKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hZ9x4BtDFYUfED6r9s6odmQcYTViF4DPSlxHT+KcbOI=; b=sr+nkbORN04oIQAPiQT15NW3j3LmRfoAGxKB2/wXRzoNNUXmMtvANKy9al9fgbjCzd fx8wlWVZDewTEz+9ifKSC9Rl6OpLsX3zGVLzFp8hcBg0trfTXNaDXC5w4dzsxsqCU0Zy GB3tY8l6XKgxRZDqjxdDj5cZ1rzOJBuiYjokk0MbAJyvjrIYFv3yBjWsxrIzfLKVjKVc WmYJtdzmvIdS+FLSqHrYo8tbACxIdkdUfq8TqR9RzPF2jwU4Hht/joOW7nbv+VmzStWn qb7drhtsPLqBAbvNE8LS+aGuuh575znenjTqy8zIAsESDuDeVgSyfyMn/yp6ozGSORwR mESQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="G8RD/qSK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w4-20020a656944000000b0044b5e15db73si9516088pgq.249.2022.10.16.09.26.05; Sun, 16 Oct 2022 09:26:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="G8RD/qSK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229835AbiJPQYt (ORCPT + 99 others); Sun, 16 Oct 2022 12:24:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbiJPQXz (ORCPT ); Sun, 16 Oct 2022 12:23:55 -0400 Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1119A3ECD0 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) Received: by mail-qk1-x733.google.com with SMTP id m6so5404796qkm.4 for ; Sun, 16 Oct 2022 09:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hZ9x4BtDFYUfED6r9s6odmQcYTViF4DPSlxHT+KcbOI=; b=G8RD/qSKPtxPvy/bEl8swbr0PqqVAiFSkvohygiXh0Ahx4dGPQGb7L77Bo2KfBq4dK 9M63FYIMMRoR+tVGQPiXiWRSo7L6Idn4ui2khvMugG4yRA/G+URk0zwsMdofH/sHdTyw 2RcCmANQL7maNSO6o1Cqos0t0MOqYgAg3lIGE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hZ9x4BtDFYUfED6r9s6odmQcYTViF4DPSlxHT+KcbOI=; b=54BcrWlPWJUpskyx+BgT+nki+PY+gIc4/WOSbtxusp0EZHwD70pYS7c8B8VF6yF02a 9DvxEVJ7udH43E2v3649RknzrllAM4V8+H+XV2wTHfYAZZ3DRcrnYRaXFU2SuydBUWVy qpwol9S16QFpoDtyN/nDhFb/cq9szL5S/a1KPlZ1wJNZnmlYZ/o1JbZ/N8etEoBHhGPX gyx4e9Eq9vS5VrcheIsUR/BHoiqPte+NctPY/Q5j/4MrBG0n5HKS+nU2oxgMkZhTLvUA EkeFmgFLj6/jHwoejO4jfoj4q2wUgnU6GCqLPaxUbqfZ0b1qLEPaVBUIxPOqNBhATg48 P3vw== X-Gm-Message-State: ACrzQf376agGS7YlDFFmPkhBfXa0XYCXav2iRNo2RZSNUWz6LExN6IPj bmtEhjUp3Ee7eLpa+VqlyyMdDvj6TsuGtQ== X-Received: by 2002:a05:620a:d8c:b0:6a7:91a2:c827 with SMTP id q12-20020a05620a0d8c00b006a791a2c827mr4887021qkl.407.1665937422501; Sun, 16 Oct 2022 09:23:42 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:42 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" Subject: [PATCH v9 13/13] rcu/debug: Add wake-up debugging for lazy callbacks Date: Sun, 16 Oct 2022 16:23:05 +0000 Message-Id: <20221016162305.2489629-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862162254535878?= X-GMAIL-MSGID: =?utf-8?q?1746862162254535878?= This patch adds initial debugging for lazy callback: whether the callback does a wake up or not. We see that callbacks doing wake ups are usually associated with synchronous use cases (SCSI, rcu_sync, synchronize_rcu() etc). The code is not very intrusive as almost all the logic is in 'lazy-debug.h' with just a few calls from tree.c In the future, we will add more functionalities such as ensuring callbacks execute in bounded time. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/Kconfig | 7 ++ kernel/rcu/lazy-debug.h | 154 ++++++++++++++++++++++++++++++++++++++++ kernel/rcu/tree.c | 9 +++ 3 files changed, 170 insertions(+) create mode 100644 kernel/rcu/lazy-debug.h diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index edd632e68497..08c06f739187 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -322,4 +322,11 @@ config RCU_LAZY To save power, batch RCU callbacks and flush after delay, memory pressure or callback list growing too big. +config RCU_LAZY_DEBUG + bool "RCU callback lazy invocation debugging" + depends on RCU_LAZY + default n + help + Debugging to catch issues caused by delayed RCU callbacks. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/lazy-debug.h b/kernel/rcu/lazy-debug.h new file mode 100644 index 000000000000..b8399b51d06a --- /dev/null +++ b/kernel/rcu/lazy-debug.h @@ -0,0 +1,154 @@ +#include +#include + +#ifdef CONFIG_RCU_LAZY_DEBUG +#include +#include + +static DEFINE_PER_CPU(bool, rcu_lazy_cb_exec) = false; +static DEFINE_PER_CPU(void *, rcu_lazy_ip) = NULL; + +static DEFINE_RAW_SPINLOCK(lazy_funcs_lock); + +#define FUNC_SIZE 1024 +static unsigned long lazy_funcs[FUNC_SIZE]; +static int nr_funcs; + +static void __find_func(unsigned long ip, int *B, int *E, int *N) +{ + unsigned long *p; + int b, e, n; + + b = n = 0; + e = nr_funcs - 1; + + while (b <= e) { + n = (b + e) / 2; + p = &lazy_funcs[n]; + if (ip > *p) { + b = n + 1; + } else if (ip < *p) { + e = n - 1; + } else + break; + } + + *B = b; + *E = e; + *N = n; + + return; +} + +static bool lazy_func_exists(void* ip_ptr) +{ + int b, e, n; + unsigned long flags; + unsigned long ip = (unsigned long)ip_ptr; + + raw_spin_lock_irqsave(&lazy_funcs_lock, flags); + __find_func(ip, &b, &e, &n); + raw_spin_unlock_irqrestore(&lazy_funcs_lock, flags); + + return b <= e; +} + +static int lazy_func_add(void* ip_ptr) +{ + int b, e, n; + unsigned long flags; + unsigned long ip = (unsigned long)ip_ptr; + + raw_spin_lock_irqsave(&lazy_funcs_lock, flags); + if (nr_funcs >= FUNC_SIZE) { + raw_spin_unlock_irqrestore(&lazy_funcs_lock, flags); + return -1; + } + + __find_func(ip, &b, &e, &n); + + if (b > e) { + if (n != nr_funcs) + memmove(&lazy_funcs[n+1], &lazy_funcs[n], + (sizeof(*lazy_funcs) * (nr_funcs - n))); + + lazy_funcs[n] = ip; + nr_funcs++; + } + + raw_spin_unlock_irqrestore(&lazy_funcs_lock, flags); + return 0; +} + +static void rcu_set_lazy_context(void *ip_ptr) +{ + bool *flag = this_cpu_ptr(&rcu_lazy_cb_exec); + *flag = lazy_func_exists(ip_ptr); + + if (*flag) { + *this_cpu_ptr(&rcu_lazy_ip) = ip_ptr; + } else { + *this_cpu_ptr(&rcu_lazy_ip) = NULL; + } +} + +static void rcu_reset_lazy_context(void) +{ + bool *flag = this_cpu_ptr(&rcu_lazy_cb_exec); + *flag = false; +} + +static bool rcu_is_lazy_context(void) +{ + return *(this_cpu_ptr(&rcu_lazy_cb_exec)); +} + +static void +probe_waking(void *ignore, struct task_struct *p) +{ + // kworker wake ups don't appear to cause performance issues. + // Ignore for now. + if (!strncmp(p->comm, "kworker", 7)) + return; + + if (WARN_ON(!in_nmi() && !in_hardirq() && rcu_is_lazy_context())) { + pr_err("*****************************************************\n"); + pr_err("RCU: A wake up has been detected from a lazy callback!\n"); + pr_err("The callback name is: %ps\n", *this_cpu_ptr(&rcu_lazy_ip)); + pr_err("The task it woke up is: %s (%d)\n", p->comm, p->pid); + pr_err("This could cause performance issues! Check the stack.\n"); + pr_err("*****************************************************\n"); + } +} + +static void rcu_lazy_debug_init(void) +{ + int ret; + pr_info("RCU Lazy CB debugging is turned on, system may be slow.\n"); + + ret = register_trace_sched_waking(probe_waking, NULL); + if (ret) + pr_info("RCU: Lazy debug ched_waking probe could not be registered."); +} + +#else + +static int lazy_func_add(void* ip_ptr) +{ + return -1; +} + + +static void rcu_set_lazy_context(void *ip_ptr) +{ +} + +static void rcu_reset_lazy_context(void) +{ +} + +static void rcu_lazy_debug_init(void) +{ +} + +#endif diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f4b390f86865..2b2a8d84896d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -67,6 +67,7 @@ #include "tree.h" #include "rcu.h" +#include "lazy-debug.h" #ifdef MODULE_PARAM_PREFIX #undef MODULE_PARAM_PREFIX @@ -2245,7 +2246,10 @@ static void rcu_do_batch(struct rcu_data *rdp) f = rhp->func; WRITE_ONCE(rhp->func, (rcu_callback_t)0L); + + rcu_set_lazy_context(f); f(rhp); + rcu_reset_lazy_context(); rcu_lock_release(&rcu_callback_map); @@ -2770,6 +2774,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) } check_cb_ovld(rdp); + + if (lazy) + lazy_func_add(func); + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. @@ -4805,6 +4813,7 @@ void __init rcu_init(void) rcu_early_boot_tests(); kfree_rcu_batch_init(); + rcu_lazy_debug_init(); rcu_bootup_announce(); sanitize_kthread_prio(); rcu_init_geometry();