From patchwork Sat Oct 29 13:28:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12710 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1351988wru; Sat, 29 Oct 2022 06:34:02 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5BkKY8rWxOsOafTsKL/DBgLqTbyEb1U8bCOuLCzuFLV0X8tS0k3uCQ6GvyGTv5NPzLCMJZ X-Received: by 2002:a17:906:fe44:b0:77c:e313:a8e8 with SMTP id wz4-20020a170906fe4400b0077ce313a8e8mr4016777ejb.700.1667050442220; Sat, 29 Oct 2022 06:34:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667050442; cv=none; d=google.com; s=arc-20160816; b=sGK+PqljhRh1owQSAaR0pLcf455tRe8vJLcz2lvG5ZdLlL+vz+noDwm0B8Bqkf6RBp EBwiM1Liq8rAL1RJfAeGelG1eUk3w3rO2SHtdwyytkcMTUeFX83dFftpjwmEpKgu3evE 0CPhLMs+nmVdH/pTNJWZ2s2OZD+xhJrXToExvsLyS5jTGieqfwh+YRuY3+5fkWaCSMAo V9S/SoKlROE9HqrQVsh1MHTGVyaDpY7+I6NWJpAksjXB9Hh7+JxDba+FxqHP12WjyN57 EgKunQqbBGY61nd/Zv8UIBEUUqnBM+B7/A/jkcVD0/RrRAaXRaArVGnuM1UVdQeDvq6y s7jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=QCeOrmF4sULE5fXDGGQt+e1Jnohk08bl1WSeFsgJlgA=; b=mI3gOg5vs8okia7xPXZS6Yw08HvFzmTuGybb42MM1AAAmTuxZebHr1HQBkK366tULt K/dILNFIijZ14BXo9ndrZJvYxElv2pSZ1MPWvxw4YOjEIgkoYj7xdVAniPFP/X5SJMOV KTmwzwCBe4MVNoAtH9OrtzDYVWOYotpItP8Picx/n+ReZN+dk38xzrh+61Rtl3ch6Rn6 SnGcZ4UALWPfpvswAPlBykr5SAOwzNz3NM0lvaU50VXQ40LJ2AJkUHJehwNpKn01RfFN PXfOUruTOXJ/nIY28u/F29zpbGAKSOA58WdVuaKMEzNHrJBM1u6FDqfagZsRkwqia0pI yMsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=xMvWDbz0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qw18-20020a1709066a1200b0078d2197d670si1843201ejc.661.2022.10.29.06.33.36; Sat, 29 Oct 2022 06:34:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=xMvWDbz0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229544AbiJ2N3P (ORCPT + 99 others); Sat, 29 Oct 2022 09:29:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229456AbiJ2N3O (ORCPT ); Sat, 29 Oct 2022 09:29:14 -0400 Received: from mail-qv1-xf29.google.com (mail-qv1-xf29.google.com [IPv6:2607:f8b0:4864:20::f29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A36452BE1C for ; Sat, 29 Oct 2022 06:29:13 -0700 (PDT) Received: by mail-qv1-xf29.google.com with SMTP id w10so5787689qvr.3 for ; Sat, 29 Oct 2022 06:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=QCeOrmF4sULE5fXDGGQt+e1Jnohk08bl1WSeFsgJlgA=; b=xMvWDbz0oeAQAIrRFyo+vUr+dMPxopQnuakIKzTkv+acMn9+1gxGF0ASPykFKV99ek 8Z5q3vjtUY2jrsiRp1+jE80bVJrfMz8D6Yn228v7hFV0EQ/KmdTZqIzO5kqUDNyY1dFN h9IjdWjbLI2WPORr7k2EjnRc+RXKIeLDc+FmQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QCeOrmF4sULE5fXDGGQt+e1Jnohk08bl1WSeFsgJlgA=; b=ut0R42fohP7MSs1h1TSkNHPCgk8sIH1/OBlm9mPcaqwc5/W0sOlyo0UYWBR04+9FXT mS/fVRz/iUvNO1BTKl32rNiAgLtOmM9pmlSNMpyM7Fv+uZwJVveGI1J9t3Ea7pHWjJY1 SNSv2fbb4tetSJ0qbYhN1T0pFyq5QmDZIkxioJJhkbK4w3c+XwZWL2JA+hzU8Umh+82N d6nBkECajRIoNocRlP87x8RRR94OynjNSlmVWIebju7XeTYm4Rj6z2pFDZwKm0zEok5Q Mv7qhtEr3X0ya+aRiFIvHiwaRZkcxXQMIQLTUZDhKzwoEB3P9MabDeSdAT7xkeANY115 jHiA== X-Gm-Message-State: ACrzQf0E+fpITnJK6BeEus+qjIBR/4xfMWn+QnVgZ/ZTeyd8fSPMn8P3 C5QYAKO4ukRvmjNFhZV1IzHDVdoIcE7AKQ== X-Received: by 2002:a05:6214:29e7:b0:4bb:db99:c393 with SMTP id jv7-20020a05621429e700b004bbdb99c393mr360121qvb.12.1667050152789; Sat, 29 Oct 2022 06:29:12 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (48.230.85.34.bc.googleusercontent.com. [34.85.230.48]) by smtp.gmail.com with ESMTPSA id m22-20020a05620a291600b006f926a0572asm1085843qkp.27.2022.10.29.06.29.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Oct 2022 06:29:11 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, paulmck@kernel.org, urezki@gmail.com, Joel Fernandes Subject: [PATCH RFC] rcu/kfree: Do not request RCU when not needed Date: Sat, 29 Oct 2022 13:28:56 +0000 Message-Id: <20221029132856.3752018-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748029084460704691?= X-GMAIL-MSGID: =?utf-8?q?1748029084460704691?= On ChromeOS, I am (almost) always seeing the optimization trigger. Tested boot up and trace_printk'ing how often it triggers. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 591187b6352e..3e4c50b9fd33 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2935,6 +2935,7 @@ struct kfree_rcu_cpu_work { /** * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period + * @rdp: The rdp of the CPU that this kfree_rcu corresponds to. * @head: List of kfree_rcu() objects not yet waiting for a grace period * @bkvhead: Bulk-List of kvfree_rcu() objects not yet waiting for a grace period * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period @@ -2964,6 +2965,8 @@ struct kfree_rcu_cpu { struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; + struct rcu_data *rdp; + unsigned long last_gp_seq; bool initialized; int count; @@ -3167,6 +3170,7 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp) mod_delayed_work(system_wq, &krcp->monitor_work, delay); return; } + krcp->last_gp_seq = krcp->rdp->gp_seq; queue_delayed_work(system_wq, &krcp->monitor_work, delay); } @@ -3217,7 +3221,17 @@ static void kfree_rcu_monitor(struct work_struct *work) // be that the work is in the pending state when // channels have been detached following by each // other. - queue_rcu_work(system_wq, &krwp->rcu_work); + // + // NOTE about gp_seq wrap: In case of gp_seq overflow, + // it is possible for rdp->gp_seq to be less than + // krcp->last_gp_seq even though a GP might be over. In + // this rare case, we would just have one extra GP. + if (krcp->last_gp_seq && + rcu_seq_completed_gp(krcp->last_gp_seq, krcp->rdp->gp_seq)) { + queue_work(system_wq, &krwp->rcu_work.work); + } else { + queue_rcu_work(system_wq, &krwp->rcu_work); + } } } @@ -4802,6 +4816,8 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + krcp->rdp = per_cpu_ptr(&rcu_data, cpu); + krcp->last_gp_seq = 0; for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp;