From patchwork Sat Jul 29 14:27:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 128154 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1077198vqg; Sat, 29 Jul 2023 08:05:05 -0700 (PDT) X-Google-Smtp-Source: APBJJlEqqCjQj6CXwPNPM/p3L+hBXUdRZMwmIfGBKOY3b5hALE+yyyALC5gNQNaqGuR3ma43z9e4 X-Received: by 2002:a19:ca16:0:b0:4fb:89cd:9616 with SMTP id a22-20020a19ca16000000b004fb89cd9616mr3345341lfg.0.1690643105393; Sat, 29 Jul 2023 08:05:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690643105; cv=none; d=google.com; s=arc-20160816; b=ZRTyJc6xjfj8JpC8hKN2/ZfUn7pEg82NQx3FV5oJaJQYCoZxNHrlzxEbZSweWpi72V x5CYirFb1h9B4tkXLTB5cO0kjK5B8S4gu03GbA9iuFs/C/XiFMsuJ5lht0gJCjLag0AA f06eZMHcpycyELppXkRikzq5L46pFwKGFFA7uxL14PB5BCH67U05xzbOk1plhg6p6F/f 1zXUQ59tSyb6HzjSopZWo8Ct0uGz9fkNFDTZcb+IHrW+W2juA+lwbiVqAKkOMH5aYJ5v FSpRFHxySkvt+gMF78DGSjX3QqdiIlyMfcOxv+SO0RFr6memR1bgD0Aoyw06JTZId4yg NQuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iwxEGA9bB7b0DElBaIYcQs9XoGIY+NHTZRuaPpVwP9M=; fh=XpN01c8C0O5LT/1Tybmn18TqAUWs71ylP4XyhYXexmU=; b=cadcddMFxURuGunD1RPUz1hu08mLTrhkSqCevf87OlZp1ogY3CkxOk9jxHYu6B41uL cvBlZPqB8BKNmrRnQ9FPMZ/+HF8dPjq51bCWLn4bJjbAZ7+MBjaFwZj0IUyjrNgNxC8I 1U61j+1GsuetBT8GCwzUEIsFRrm/+w5c7e7qRiN4sfFCIvblTAx/9aHSKIy5YZx7eilN hcaBo/gmAJGzXJuZUEDzG6yZkJaUbg0omxKhHEsRGyincbHqD0aJIKrZjnYH1H0hIAHW D1oQ5bYUOVVKNL47CiQOvRrgMJhlOu2igZrjy8O9gMVArEQtKqF4iPJawhd7nfOWL9aE vvyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=bgVc8RgM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d19-20020aa7d693000000b00522b10235f3si1120878edr.163.2023.07.29.08.04.41; Sat, 29 Jul 2023 08:05:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=bgVc8RgM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231365AbjG2O2y (ORCPT + 99 others); Sat, 29 Jul 2023 10:28:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231687AbjG2O2x (ORCPT ); Sat, 29 Jul 2023 10:28:53 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8DDE3C38 for ; Sat, 29 Jul 2023 07:28:26 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id af79cd13be357-76c9e9642b1so38959185a.3 for ; Sat, 29 Jul 2023 07:28:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640896; x=1691245696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iwxEGA9bB7b0DElBaIYcQs9XoGIY+NHTZRuaPpVwP9M=; b=bgVc8RgM8MUE3/FR/qxJAAVnZPDSL8B/L8Y55nDBH4dQN+gKdet8DkRYtYGE5rhJKB 5PFNZ2s+lGod3ijPAm+7YYjP7qeW1FBV3BCBrlG7+C52Vjzf4HQApIQy+8ztJrApuvFA jh0kfZEj3pdfu2H7JHKa7COI15JG/bknJpcRI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640896; x=1691245696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iwxEGA9bB7b0DElBaIYcQs9XoGIY+NHTZRuaPpVwP9M=; b=GL+nNzu6yFALb0OEZnhSzJ5S796vLIKy5H6qmY/t1gk32XzhyX4+Nrt2poDo5IGifc W4hgfT2LWvlZ5/2LHDKb0hDKO+WDaCA4e5cyHZd8+iIyNWIRpj4MTV8vOzfJB3ftDSig hN1IqP4ZG3jycxAgMoO+gOdY+9JLFrOUxx+JzhK+Pe568wijVi+gGHXKydZYY/nn5oyg Q2bAX8vynayVAGAC4gxrGh7rwnkoeAXfAKXpbyjz6kGhYYNf1Tw/X5giaMcF2hux5RVL mHIq8C/oQiAL3HLfdFEAUe6q4BXRiK9uaB8jc2IIfiZ+Eld5Ceh5IM9WuNZkBPHw5FKP larg== X-Gm-Message-State: ABy/qLZKMOSotV6gF4aPhIEqepJVcri6NPMH0cqDX5aXNo7BOYNJ7Guw o8qPO4BXOGp4J95oig5vzJ1ucizxmCs7CbwlEGE= X-Received: by 2002:a92:cdab:0:b0:348:8050:d7a with SMTP id g11-20020a92cdab000000b0034880500d7amr3082832ild.30.1690640875892; Sat, 29 Jul 2023 07:27:55 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:55 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , stable@vger.kernel.org, Davidlohr Bueso , Josh Triplett Subject: [PATCH v2 1/5] rcutorture: Fix stuttering races and other issues Date: Sat, 29 Jul 2023 14:27:31 +0000 Message-ID: <20230729142738.222208-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772767784982712764 X-GMAIL-MSGID: 1772767784982712764 The stuttering code isn't functioning as expected. Ideally, it should pause the torture threads for a designated period before resuming. Yet, it fails to halt the test for the correct duration. Additionally, a race condition exists, potentially causing the stuttering code to pause for an extended period if the 'spt' variable is non-zero due to the stutter orchestration thread's inadequate CPU time. Moreover, over-stuttering can hinder RCU's progress on TREE07 kernels. This happens as the stuttering code may run within a softirq due to RCU callbacks. Consequently, ksoftirqd keeps a CPU busy for several seconds, thus obstructing RCU's progress. This situation triggers a warning message in the logs: [ 2169.481783] rcu_torture_writer: rtort_pipe_count: 9 This warning suggests that an RCU torture object, although invisible to RCU readers, couldn't make it past the pipe array and be freed -- a strong indication that there weren't enough grace periods during the stutter interval. To address these issues, this patch sets the "stutter end" time to an absolute point in the future set by the main stutter thread. This is then used for waiting in stutter_wait(). While the stutter thread still defines this absolute time, the waiters' waiting logic doesn't rely on the stutter thread receiving sufficient CPU time to halt the stuttering as the halting is now self-controlled. Cc: stable@vger.kernel.org Signed-off-by: Joel Fernandes (Google) --- kernel/torture.c | 45 ++++++++++++--------------------------------- 1 file changed, 12 insertions(+), 33 deletions(-) diff --git a/kernel/torture.c b/kernel/torture.c index 6ba62e5993e7..fd353f98162f 100644 --- a/kernel/torture.c +++ b/kernel/torture.c @@ -720,7 +720,7 @@ static void torture_shutdown_cleanup(void) * suddenly applied to or removed from the system. */ static struct task_struct *stutter_task; -static int stutter_pause_test; +static ktime_t stutter_till_abs_time; static int stutter; static int stutter_gap; @@ -730,30 +730,16 @@ static int stutter_gap; */ bool stutter_wait(const char *title) { - unsigned int i = 0; bool ret = false; - int spt; + ktime_t till_ns; cond_resched_tasks_rcu_qs(); - spt = READ_ONCE(stutter_pause_test); - for (; spt; spt = READ_ONCE(stutter_pause_test)) { - if (!ret && !rt_task(current)) { - sched_set_normal(current, MAX_NICE); - ret = true; - } - if (spt == 1) { - torture_hrtimeout_jiffies(1, NULL); - } else if (spt == 2) { - while (READ_ONCE(stutter_pause_test)) { - if (!(i++ & 0xffff)) - torture_hrtimeout_us(10, 0, NULL); - cond_resched(); - } - } else { - torture_hrtimeout_jiffies(round_jiffies_relative(HZ), NULL); - } - torture_shutdown_absorb(title); + till_ns = READ_ONCE(stutter_till_abs_time); + if (till_ns && ktime_before(ktime_get(), till_ns)) { + torture_hrtimeout_ns(till_ns, 0, HRTIMER_MODE_ABS, NULL); + ret = true; } + torture_shutdown_absorb(title); return ret; } EXPORT_SYMBOL_GPL(stutter_wait); @@ -764,23 +750,16 @@ EXPORT_SYMBOL_GPL(stutter_wait); */ static int torture_stutter(void *arg) { - DEFINE_TORTURE_RANDOM(rand); - int wtime; + ktime_t till_ns; VERBOSE_TOROUT_STRING("torture_stutter task started"); do { if (!torture_must_stop() && stutter > 1) { - wtime = stutter; - if (stutter > 2) { - WRITE_ONCE(stutter_pause_test, 1); - wtime = stutter - 3; - torture_hrtimeout_jiffies(wtime, &rand); - wtime = 2; - } - WRITE_ONCE(stutter_pause_test, 2); - torture_hrtimeout_jiffies(wtime, NULL); + till_ns = ktime_add_ns(ktime_get(), + jiffies_to_nsecs(stutter)); + WRITE_ONCE(stutter_till_abs_time, till_ns); + torture_hrtimeout_jiffies(stutter - 1, NULL); } - WRITE_ONCE(stutter_pause_test, 0); if (!torture_must_stop()) torture_hrtimeout_jiffies(stutter_gap, NULL); torture_shutdown_absorb("torture_stutter"); From patchwork Sat Jul 29 14:27:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 128157 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1102227vqg; Sat, 29 Jul 2023 09:02:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlFwnqvsez47w7wTqCPKAuq/TClawG0hTmAmG6Xs6BnYujuOOHZc4EdBWSdfsP0dtMamqRCA X-Received: by 2002:a05:6a00:2ea6:b0:686:2539:9ff2 with SMTP id fd38-20020a056a002ea600b0068625399ff2mr5590385pfb.9.1690646536886; Sat, 29 Jul 2023 09:02:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690646536; cv=none; d=google.com; s=arc-20160816; b=eybL+vIqilQQ4+hjU078cpMnwT3YLzV5yCGe6K9OEfki7h0RCUXcpqjLAIm99fgLmQ y/UH7WsHYMCrhe/s5UmdmyMTV8dzNZNMwWHc/VtvpG8OKv8smsE495YeeM26hTWBoBjL yETfwF6v2+WEGzAwuRUYK1ttqZcs5hlaQ5Dj6UKBEwrk/b82Qh+sQKFkIgc15skOp1pg dma+gZ4zSXdgD1DSmgOeWXV7/euHva2dVzP82xfZhwVESiLZ8gXBLKUyVWoR5xKi3ble qnn9Jg8KlAog6Ngy9P2K9dWPsEdxD4iWKnmE8ulGC3wlacVEcrp9qtauL3g0ROWVweiO RBDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=K/AP9GOcXVQHz8AOI7UjuaU+sYzEqAzG6F3W2fxwbBg=; fh=vo+3QWVzV6cDj4PyFvXSjNhvhmJuhaqIIZk8NUVTGGo=; b=ii1J5vaCgU20zFCJ/121y7iZiyvhVjes5LNi4PfyTPJpaTWGViFK9jJVBqErVY5TlR 2rnB0k6cCvvywXN7NlPoZpzXQUIRibeOQP9yW1NngyHgKx84h5tI9XJaxWPZtbvgssCg MKY+xNvWD4Yn7Pc9cXlX7GP5ZNbhw4Q1kuLJfHMG4yV4agfJpv1dfQKVVQKFJ3N5C3Ud b0j8/+89S9gxtOriSWE/unnN4itDakQFz7vfi3O2U0M0RADvCKpfSOIXPAnu5cdDgy+J STA9kJ3zAXJrNHtICXBKGEbqgAzQ6m7/Ntv4Pybg6tOy8MBlR7KigY9fNYwBM5o90K00 DwTg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=J4kyjvgZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cm10-20020a056a00338a00b00653b5ab16c3si3083906pfb.265.2023.07.29.09.02.03; Sat, 29 Jul 2023 09:02:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=J4kyjvgZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231944AbjG2O2C (ORCPT + 99 others); Sat, 29 Jul 2023 10:28:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231918AbjG2O16 (ORCPT ); Sat, 29 Jul 2023 10:27:58 -0400 Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE3162686 for ; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) Received: by mail-il1-x132.google.com with SMTP id e9e14a558f8ab-348de515667so13590085ab.1 for ; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640877; x=1691245677; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K/AP9GOcXVQHz8AOI7UjuaU+sYzEqAzG6F3W2fxwbBg=; b=J4kyjvgZ82W4wgXCeEeMZsVCguRFpikKkcE2HcgJj1qeY7vt+QnMIYGWOOT+WB4w/d QxBIq/qctXIL+Pq+QR5781uOJJWRU23mvLFOkz/5oH2U307qLuDCMVxppr9LWan2wYgB x4GjVCbkk7A28bEYLtEBKdloUvZLupH9Z4E6M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640877; x=1691245677; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K/AP9GOcXVQHz8AOI7UjuaU+sYzEqAzG6F3W2fxwbBg=; b=MmbCjhmnhhzS+cV4pIno0bermGYFpi59srOVhM0NWii3lzPecxpKkPEaGBy/7RwRVl UK+6r8rJk8niLD1yqz9UN9jtDByXARzXI6kxS7WQI6Mq2OlYAGVEZuqsjMDeiZci4A5D SHUAZcdZkejHdiiu+KYq8JRi9UD39xy2botbHI28KX7b88RMhboc69WVKacaOZY0lR6m zxmGPI3ZyL1c8m8y0oKto2S+vPCZCrf5sISflUC+rP7WGGmtLC4u6BrZIOtnDmrrKGeK tBfV5b+jAy1B03izf/cO7nSJmsVW8TeUn+JuqlLtJLYzUBhoaZH/5sBG+uprfSIsZk6P CtiQ== X-Gm-Message-State: ABy/qLZTWYXoW924/o/fA03kK1INuyktAARTGlttc/NWM62t8weZ6Wmf +lJ3wWqNwi2aO3N18jLq7F1wemR1BvUf2BhPS4A= X-Received: by 2002:a92:c7c1:0:b0:345:ad81:ecaf with SMTP id g1-20020a92c7c1000000b00345ad81ecafmr2084447ilk.3.1690640876885; Sat, 29 Jul 2023 07:27:56 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:56 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Lai Jiangshan , Josh Triplett , Steven Rostedt , Mathieu Desnoyers Subject: [PATCH v2 2/5] srcu: Fix error handling in init_srcu_struct_fields() Date: Sat, 29 Jul 2023 14:27:32 +0000 Message-ID: <20230729142738.222208-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772771382946077143 X-GMAIL-MSGID: 1772771382946077143 The current error handling in init_srcu_struct_fields() is a bit inconsistent. If init_srcu_struct_nodes() fails, the function either returns -ENOMEM or 0 depending on whether ssp->sda_is_static is true or false. This can make init_srcu_struct_fields() return 0 even if memory allocation failed! Simplify the error handling by always returning -ENOMEM if either init_srcu_struct_nodes() or the per-CPU allocation fails. This makes the control flow easier to follow and avoids the inconsistent return values. Add goto labels to avoid duplicating the error cleanup code. Link: https://lore.kernel.org/r/20230404003508.GA254019@google.com Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/srcutree.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 20d7a238d675..f1a905200fc2 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -255,29 +255,31 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); - if (!ssp->sda) { - if (!is_static) - kfree(ssp->srcu_sup); - return -ENOMEM; - } + if (!ssp->sda) + goto err_free_sup; init_srcu_struct_data(ssp); ssp->srcu_sup->srcu_gp_seq_needed_exp = 0; ssp->srcu_sup->srcu_last_gp_end = ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { - if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!ssp->srcu_sup->sda_is_static) { - free_percpu(ssp->sda); - ssp->sda = NULL; - kfree(ssp->srcu_sup); - return -ENOMEM; - } - } else { - WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); - } + if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) + goto err_free_sda; + WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); } ssp->srcu_sup->srcu_ssp = ssp; smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, 0); /* Init done. */ return 0; + +err_free_sda: + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda = NULL; + } +err_free_sup: + if (!is_static) { + kfree(ssp->srcu_sup); + ssp->srcu_sup = NULL; + } + return -ENOMEM; } #ifdef CONFIG_DEBUG_LOCK_ALLOC From patchwork Sat Jul 29 14:27:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 128156 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1101266vqg; Sat, 29 Jul 2023 09:00:55 -0700 (PDT) X-Google-Smtp-Source: APBJJlEXHOnxO9XsjaNGvNd9EsUiJX7JoULgmu1SEpcH40DE+305QMLfcx4zGB/WZOXui+pw+wq4 X-Received: by 2002:a05:6a21:7782:b0:12e:ae87:45d2 with SMTP id bd2-20020a056a21778200b0012eae8745d2mr4599719pzc.51.1690646455097; Sat, 29 Jul 2023 09:00:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690646455; cv=none; d=google.com; s=arc-20160816; b=07fMBhcP/qJitSDjNsZw3gSDaGrTWgcDBIKFCCQwRLF0fPLEK7HBGHLnNceprI92YR l65rzr3GIziBwNK7W6jjXa/aOgWbidiaUzNqTTNSSbnsiwbDLnRVLND9hHnMFzU/l1Wi HDQmS/B9zc/7jQB4cZHhoqmpTjivS1fKWEBqc8Wma7XRtuyqB7RYHiW4vMDp7tLliOR0 NzNL7z+f/h9Ew3XEUv0KO996zi4uyaWgPKvAzoH0NdphsrJnztmU8Z6q+lcbqaVRvc33 8WDXzVFsJwyH0zpaBg9dGql1AS2AYi693hUNyiMmXZ+JjSESezJUbkchXpz8voeyG7Ia U6JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; fh=ensvRh18ObyfbTOVA1HyWvWinOtWNSxxC/ejDkozH34=; b=Y7hEuFzRbg/blTlJvjnkSDzPACfNjhIikHkbWLOlkDDx/FUnYVVv4Fy15uCu1GU0TY 2PXJUm5I4kN5Ai9+qSWD4ayqEOfU/iZ0cBWll8FBdZjaNdRk182VEt8QCgyTB+/jisHb peUj/lYHtn+4NCueMqX57Xd8ibR7cwuTFY31ZlNwnqGKXbhznnLnf9XNOHYpwoCjbYxm /tKj1jlM5YrcYDepAMlpM5Nf49apevvMH3gvGJmLJ6htoFnbT/GeWAwzXPco8/K7xEsZ wMvUaiNlyh8gjlPoTISYvoUI8QfqZTuQxRD29U1u80t4dHTgjG5aX6l4GeIaw0p+H6MJ PUkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=plXTHquW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cm10-20020a056a00338a00b00653b5ab16c3si3083906pfb.265.2023.07.29.09.00.39; Sat, 29 Jul 2023 09:00:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=plXTHquW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231965AbjG2O2F (ORCPT + 99 others); Sat, 29 Jul 2023 10:28:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229619AbjG2O17 (ORCPT ); Sat, 29 Jul 2023 10:27:59 -0400 Received: from mail-il1-x12c.google.com (mail-il1-x12c.google.com [IPv6:2607:f8b0:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBA4AE7A for ; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) Received: by mail-il1-x12c.google.com with SMTP id e9e14a558f8ab-34914684b62so353225ab.3 for ; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640878; x=1691245678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=plXTHquWdh4bDvLFn2iXXCHJmvv/+MoXo8lRqYopE64G7FgOHv+cnT2c05bBwokNCo f2+s+ItwB5Hhqd7VCNCKAVOINhdBw91B9XtHD1TQBm0bxbtcj/QaCeXNKMY+P2vShgB/ AyVO2DIkMfhumM/yGpIR5gkL7/1244kpfN2Zs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640878; x=1691245678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=Qya9OLad6LWWqBl07H2mxVxYhEwybDTGI+KqTfMcxOhbCdMC3ygr4spgmQbAsmAtiA 2jrq6ZTQGy2Q1J2cxo7PzjR8J2lK3Uqf8UAAbK1DT5ou+YxEDaIaQiFpHCFAeZwe/TLN jPPufDPnBl6U2Lk2LPlkfnqx1RSrjLkfu96NZzB/BJuNNOhhhuuIbH22dEjQCkUxHC5T e87m/WYDzwZSHOYfOqD8BKs2o3V915L9df5Np87m69QYNuwnuafJb/FmMDarG6w45/nV Lkz7MMdVMMZEBiP7n1vXLX9lFO5XzRFATb1rm8hbwIs5FxkFo9ygSjZN6pwAv4sYnTfO 8dgw== X-Gm-Message-State: ABy/qLYYumvClh7ASHJUR/cAnsWEdQH6J7SLfTsVbZWukv30j0YHoFvC JMTq/eeRnKu/zCy8vc691TqPOeODkh8ccmlQGJo= X-Received: by 2002:a05:6e02:1d94:b0:348:f6cb:e585 with SMTP id h20-20020a056e021d9400b00348f6cbe585mr2922758ila.30.1690640877759; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:57 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 3/5] tree/nocb: Adjust RCU_NOCB_WAKE_* macros from weaker to stronger Date: Sat, 29 Jul 2023 14:27:33 +0000 Message-ID: <20230729142738.222208-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772771297102304860 X-GMAIL-MSGID: 1772771297102304860 This is needed to make the next patch work correctly as we rely on the strength of the wakeup when comparing deferred-wakeup types across different CPUs. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 192536916f9a..0f40a9c2b78d 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -288,8 +288,8 @@ struct rcu_data { /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 -#define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE_LAZY 1 +#define RCU_NOCB_WAKE_BYPASS 2 #define RCU_NOCB_WAKE 3 #define RCU_NOCB_WAKE_FORCE 4 From patchwork Sat Jul 29 14:27:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 128162 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1130391vqg; Sat, 29 Jul 2023 10:05:25 -0700 (PDT) X-Google-Smtp-Source: APBJJlGZycBr4z+IN4yG2tioJ/tABuFHoKkrM2F1+2TYCU4IgSc6zZa8a5sR9z5g1sUicERao5Mq X-Received: by 2002:a17:902:e88c:b0:1b8:9195:1dd8 with SMTP id w12-20020a170902e88c00b001b891951dd8mr5615763plg.51.1690650324815; Sat, 29 Jul 2023 10:05:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690650324; cv=none; d=google.com; s=arc-20160816; b=bz23qeikgNSWRTMSzMu5WsgG9vzbcdRmt64bC7R4jeVyDqTEImV5pQ5hVIycUN1+Sk +5Uun4RaqTLyyPVTYVbm0298uC2VAo5PNcEeW9O9hfkzBTqhW38YvFawTSOxONGR2We7 5zPRRtJbLrXhJNenmNCxqdRLsM3XP+r7mZogAyo6ARE5l2ka3Y1nd3jwr2Q4KJdKhkzk 8/tvjtrPnZV8SJcsvhQFhOm/krYzPK1A2teHtUVFRqY2DlDNzx5rUudp+8wvD3MZdZtn lx6PG8uH4B9TSccyQpB1KAgyqVC801JA42LT5rVVjDAobCcFJ18RG7UqDJzK7KipGwbF 80YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; fh=ensvRh18ObyfbTOVA1HyWvWinOtWNSxxC/ejDkozH34=; b=zayGsqv5u3m/75tlfU7KI0Nw2TCsdQXCQV/emEOIsZPg39yYwvfgaiL1E4YfaXW+ks T28/NFl0yDJMmNO8lK79fos+CybvRpK9lOI1cuWrzqehPphRYt77ClLQPnXm0gCMAW7U XuoTOeEHix+bABDRWkT6br3FmcGBDHOCiZECKxhqBrandOZeuaz1QDiisCUFsu+yfhLg dJFoTfkVZ0f/b6QuoJduuKFrWkzAVHlFboTc7JzD8x62DLE2OY3xHwAGqqblUcEAn2ZH SbyuuI5XD5ZCkSFmgFrI39L3JhuUd14Grp5w4llVG4IIbu+p1hrD+H3mp+ZuR2MeuhMF d3wg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=HsW5SYF4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z191-20020a6333c8000000b00530b3b98fc5si4855412pgz.417.2023.07.29.10.04.58; Sat, 29 Jul 2023 10:05:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=HsW5SYF4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232007AbjG2O2I (ORCPT + 99 others); Sat, 29 Jul 2023 10:28:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231919AbjG2O2B (ORCPT ); Sat, 29 Jul 2023 10:28:01 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF95A30F4 for ; Sat, 29 Jul 2023 07:27:59 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-76c64da0e46so142001639f.0 for ; Sat, 29 Jul 2023 07:27:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=HsW5SYF4O4C1xrNtb8sm4pRTfnrrViTLGL7v+Gn/Ah7UCbBgZ2opbMTyrHiGdhoaEn gcNOhQcOz0QL2+Q2P4xe34TxFbuIhXlf9EExkeLpiag6Pz3CQW2C2PScS3kCNl4lYs5+ 6y9zYLXR2axTa6ZjwfeIofwo3CIgGps6icYMg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=ft8y3Iq4Nybu4vm7/hYZgp1feZWmww1GxCj818YvKVSlUW2jxXu5XVL6IEdICMBnkS S6FCuUNlegiFgDsGwtJQDrALIDR0zK+sl2zKt8tpl4BbR748br0/B9yhu2XoMk9ERDlB 1qyUX/MvhClR6LrZVAyc8noyuEdfB1p/fRxLgdyRup2dTUgiQbOzN2Xlu48QexXTs9kb nmWatmuLUNaHjxpoOIzqt6GHVr+RoHt5UxvYiVGzeHIHRYS+l6bi/jA2lCY/1p2mzw1F 65drC8hTQbQJvx49DatGA11JP0toDmJYJhJ2BUTKDv8PRcw/xxNtdj9qk+raWosg04Ou pW9g== X-Gm-Message-State: ABy/qLYsRLFSYm9oGi3tkz9D9r5NmMmTy3kr2Chw33ArQWZD3I+5Bwre JxHEUVLy0+DCoB4t7feHnVa2lc8XuV3yVGmgFkI= X-Received: by 2002:a5e:a90c:0:b0:787:34d:f223 with SMTP id c12-20020a5ea90c000000b00787034df223mr2373322iod.11.1690640878695; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:58 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 4/5] tree/nocb: Improve readability of nocb_gp_wait() Date: Sat, 29 Jul 2023 14:27:34 +0000 Message-ID: <20230729142738.222208-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772775355127254778 X-GMAIL-MSGID: 1772775355127254778 The nocb_gp_wait() function contains logic to check each rdp's bypass list, flush if needed, and decide on wakeups. This makes the function hard to follow. Split out the bypass checking and flushing into a separate helper nocb_gp_flush_wake(). The new function encapsulates the logic to: - Check if the bypass needs to be flushed - Flush if needed - Return info on wakeups (lazy, bypass or none) nocb_gp_wait() now becomes simpler by calling the helper to handle the bypass flushing and waking logic. This splitting improves the readability and maintainability of the code by encapsulating related logic into a function with a clear purpose. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 113 ++++++++++++++++++++++++----------------- 1 file changed, 66 insertions(+), 47 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 5598212d1f27..c805825c3f00 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -670,22 +670,68 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("EndSleep")); } +/* + * Given an rdp, flush its bypass list if needed and return information about + * if a deferred-wakeup needs to be organized depending on whether things are + * still in the bypass list. Also tell caller if the list was flushed and if it + * is still empty after any flushing. + */ +static int nocb_gp_flush_wake(struct rcu_data *rdp, bool *empty, bool *flush) +{ + long bypass_ncbs; + long lazy_ncbs; + unsigned long j = jiffies; + + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); + lockdep_assert_held(&rdp->nocb_lock); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + *flush = false; + *empty = false; + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { + *empty = true; + return RCU_NOCB_WAKE_NOT; + } + + if (*flush) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + + if (bypass_ncbs) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + return (bypass_ncbs == lazy_ncbs ? RCU_NOCB_WAKE_LAZY : + RCU_NOCB_WAKE_BYPASS); + } + return RCU_NOCB_WAKE_NOT; +} + /* * No-CBs GP kthreads come here to wait for additional callbacks to show up * or for grace periods to end. */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass = false; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; - unsigned long j = jiffies; - bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; + int defer_wake_type = RCU_NOCB_WAKE_NOT; struct rcu_data *rdp, *rdp_toggling = NULL; struct rcu_node *rnp; unsigned long wait_gp_seq = 0; // Suppress "use uninitialized" warning. @@ -712,44 +758,24 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { - long bypass_ncbs; - bool flush_bypass = false; - long lazy_ncbs; + int defer_wake_type_one = RCU_NOCB_WAKE_NOT; + bool flushed; + bool empty; - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); - lockdep_assert_held(&rdp->nocb_lock); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + defer_wake_type_one = nocb_gp_flush_wake(rdp, &empty, &flushed); - if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { - rcu_nocb_unlock_irqrestore(rdp, flags); - continue; /* No callbacks here, try next. */ - } + // We may need to do a deferred wakeup later for bypass/lazy + // So note down what we learnt from the rdp. + defer_wake_type = max(defer_wake_type_one, defer_wake_type); - if (flush_bypass) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + // Did we make any updates to main cblist? If not, no + // non-deferred wake up to do for this rdp. + if (!flushed && empty) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; } - if (bypass_ncbs) { - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); - if (bypass_ncbs == lazy_ncbs) - lazy = true; - else - bypass = true; - } rnp = rdp->mynode; // Advance callbacks if helpful and low contention. @@ -792,23 +818,16 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) rcu_gp_kthread_wake(); } - my_rdp->nocb_gp_bypass = bypass; + my_rdp->nocb_gp_bypass = (defer_wake_type == RCU_NOCB_WAKE_BYPASS); my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; // At least one child with non-empty ->nocb_bypass, so set // timer in order to avoid stranding its callbacks. - if (!rcu_nocb_poll) { - // If bypass list only has lazy CBs. Add a deferred lazy wake up. - if (lazy && !bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, - TPS("WakeLazyIsDeferred")); - // Otherwise add a deferred bypass wake up. - } else if (bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); - } - } + if (!rcu_nocb_poll && defer_wake_type != RCU_NOCB_WAKE_NOT) + wake_nocb_gp_defer(my_rdp, defer_wake_type, + defer_wake_type == RCU_NOCB_WAKE_LAZY ? + TPS("WakeLazyIsDeferred") : TPS("WakeBypassIsDeferred")); if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ From patchwork Sat Jul 29 14:27:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 128148 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1071701vqg; Sat, 29 Jul 2023 07:55:07 -0700 (PDT) X-Google-Smtp-Source: APBJJlG61p0K8xJ91Y2bepLCPWUoCcT3hx/l4fZAFCZEvPV8dHNGv0D9XrQEN1ogBc72pFaFAZeL X-Received: by 2002:a05:6402:510a:b0:51e:5898:a23d with SMTP id m10-20020a056402510a00b0051e5898a23dmr9915158edd.5.1690642507238; Sat, 29 Jul 2023 07:55:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690642507; cv=none; d=google.com; s=arc-20160816; b=W/QjWNBO0FyLZRF+Ms6cruIl4hkhAJYot7R4p/ICV7EB2xb7vCNcexJXgXCjEtUIob UqP2OBirDEAiSxY6ZOwz5S0+MMI6SOu+I/qlzSC5tZIC7GP0+KJ+lOkTN5n3dcqgFrLa I3qyCAAj6axd/bTcqh1TNcKC8yO4JroSlVExsAZwW0rjoyK3tv0SXqHWt+kaZMfvE439 QBJ1TIJoDOFmNWPRbxgifE4WmpoM3UsVi0bYvVBYDK6tqoVgj8ENgCtu3rfKXqQbFmo8 5QC/oOVw6ud22w9ZZmuRAgjOmnj8p+3VhQYc1jiM29hGn7Di+6bI9Mhv7JTe42Qk1TGd CWEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qV1mKGTo1HAZ4CYm++JYZHEnfRnVP5qO40eCYy1XceI=; fh=ensvRh18ObyfbTOVA1HyWvWinOtWNSxxC/ejDkozH34=; b=ty4GOTTBqcOv7uUdBQxgXXY9qnmTL7i9TZMLyfWHDcRnYbnBBXEARYVG5ukYkmtE6F o9vzRn7tp5n5lO9h/Pqj3NRnLbh50nfYKw1XSPE+gcwKzTp4JgdfO3L0y8VVT6CguvEn +EguVgSN9cGflbfT8Xob4vwX0Fn4He3RZ3EgEcZV8Wa4iXa8P+ldtKruEcPTeR8uHIv5 T6ci77hT2H5+I/97MgoHgte41F+AxHjaW4mlw9utj6UvEMhPq/kbxO8q2Rd/Hemch9gf 5NBXWhhtRfhd+VUFfeR0m991b4uak63mN5iZbFe0n5Ox1HeBknest5oCzmshcKTR7zv9 c5Gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=S7Lx25DB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ay16-20020a056402203000b005226e632ac2si4266444edb.177.2023.07.29.07.54.43; Sat, 29 Jul 2023 07:55:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=S7Lx25DB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232065AbjG2O2O (ORCPT + 99 others); Sat, 29 Jul 2023 10:28:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231948AbjG2O2D (ORCPT ); Sat, 29 Jul 2023 10:28:03 -0400 Received: from mail-il1-x134.google.com (mail-il1-x134.google.com [IPv6:2607:f8b0:4864:20::134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDEB530F4 for ; Sat, 29 Jul 2023 07:28:01 -0700 (PDT) Received: by mail-il1-x134.google.com with SMTP id e9e14a558f8ab-348dfefd1a4so14299185ab.1 for ; Sat, 29 Jul 2023 07:28:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640880; x=1691245680; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qV1mKGTo1HAZ4CYm++JYZHEnfRnVP5qO40eCYy1XceI=; b=S7Lx25DBRYt0tjnRLA8AhoSntDcL9y6o1ekAFGNTXrA+Mf8j2i9gpElBv/rLiVGE+U zWI6Fe+2ZdWxeJeiB7VSx4TdmVubfqmVlZqJL446+4d/C1CxnIvrOkDkUtLpNqsM04VS C7GzcZjEatmp3kLZRPfZhDK8JUcMEnuxVNDUk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640880; x=1691245680; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qV1mKGTo1HAZ4CYm++JYZHEnfRnVP5qO40eCYy1XceI=; b=LFnbXBdCvag47iTkFe7t877Qj44qalQvkHujPf1rvGFj1Y3Y3iNjXFXziX0cVi5TuV 8F2g/WfDIuipKUCi2rd5a1AHH8P0PxaXr/SLPZg6WTTvT9k3MpFHbIzP5M/b+LrheUYz Je9UiwXS1OMbeuLGcUxW+5V4//KhcgHAymQEQsTQT6KVSzs8E2H8+4w169K9bB9AjFyY QTYDmbIgeefCLIbzw2w4jpNe7AZhlxF1XV6dhQpdgX1Sfa4z/KfftlbB8OfQycGVzT4Q bVjlu7Xnll2myvDezvCB9Lj0H41Qz/3xDdvLiyxs4mLYRZ+li4pKeUMLPOX5YlCRrucQ 3idw== X-Gm-Message-State: ABy/qLaOWi+g8ffkuMKePKw64gCFKMOFBHeLDwQnLxOIe8pDvGK7QxHm +/i9+w3sQpokm3254QzlMyazYsTPrXnZrvPFZEk= X-Received: by 2002:a92:c566:0:b0:345:df7f:efc4 with SMTP id b6-20020a92c566000000b00345df7fefc4mr3289147ilj.27.1690640880705; Sat, 29 Jul 2023 07:28:00 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:28:00 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 5/5] rcu/tree: Remove superfluous return from void call_rcu* functions Date: Sat, 29 Jul 2023 14:27:36 +0000 Message-ID: <20230729142738.222208-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772767157557161295 X-GMAIL-MSGID: 1772767157557161295 The return keyword is not needed here. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cb1caefa8bd0..7c79480bfaa0 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2713,7 +2713,7 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) */ void call_rcu_hurry(struct rcu_head *head, rcu_callback_t func) { - return __call_rcu_common(head, func, false); + __call_rcu_common(head, func, false); } EXPORT_SYMBOL_GPL(call_rcu_hurry); #endif @@ -2764,7 +2764,7 @@ EXPORT_SYMBOL_GPL(call_rcu_hurry); */ void call_rcu(struct rcu_head *head, rcu_callback_t func) { - return __call_rcu_common(head, func, IS_ENABLED(CONFIG_RCU_LAZY)); + __call_rcu_common(head, func, IS_ENABLED(CONFIG_RCU_LAZY)); } EXPORT_SYMBOL_GPL(call_rcu);