From patchwork Wed Oct 25 14:09:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 158119 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp2628828vqx; Wed, 25 Oct 2023 07:10:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEJV/DzjWBVT/ETRPXUWu5OFWfFA7vvI9Muaw7cuChgfNw5Utz0CIAwp8V8tGzKmquE70NQ X-Received: by 2002:a25:ab03:0:b0:d80:1604:f6e9 with SMTP id u3-20020a25ab03000000b00d801604f6e9mr15225458ybi.44.1698243006875; Wed, 25 Oct 2023 07:10:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698243006; cv=none; d=google.com; s=arc-20160816; b=Rj+00jk6/pTIvJsce/E8di7croEWCctztG746xYk1oj9MmY0iJbG3XfpnGvjzlfzVK wHhwgCra+fC1LYtW4mHjXa4wh0nGOyV1JbNUUwxihmb9JwOWxYCaTIbLt6iO2VlzDuEK eO2itb/hGrCU8EYBdg3LHENmGUIXSgJso1Nc/z/GT4adv2MAZqP9UPwHGOaXzDjji5C+ 9FknC944DohpQeK56MxixUdXbzR60TJRiPOSpTMWJs6pbOKiEScPML5Inyn3/XYdB52b d63Y0nIjlCw61XVG/BxmopleelGri4TcqKCkfrFeF6QY/gFzBWMaG3h1mZC5BlNoHJuY HT8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7SVLg2JlHugSU3IaNjtj7eUv+h55sIJkPWfpBUSnL5c=; fh=4iZcRCmzpaCZKgp7nvxh/m3AX3kfm//yPbGO/jIXleI=; b=z6nYzREkFvi9ilvDVmaUN4lBK5Q4iu9uPrA2qAYwwQyPScFC1G0aw51pjifU813W8q xZ1EO37MaoY5J8bvBxelAZnOihbslOaw2w4NJhysz5+UlPYEJgIdqRAf1OuroaRaY9aR xUw03OL91l36j+T0IhqzyqcHqreS84wthekFewANy3/4EZWn/pzEUQyN4hVpxzlNIHrW tlxegjRDxDs1reA7djL6oUoU/YE19RLhLObpcxbWyl49mQ/V3mxg/sidi51eATFSv5cR 4NXyoAbfRgRiEpkM4EB21v42IUt0SMJvpCTf/vRx0+g+vuHStakFpsi1wPm+IrlcOEgd fBPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=UoPOWTto; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id 206-20020a2500d7000000b00d9ac920f65bsi10177941yba.321.2023.10.25.07.10.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:10:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=UoPOWTto; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1FA598026DA9; Wed, 25 Oct 2023 07:09:52 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344824AbjJYOJb (ORCPT + 26 others); Wed, 25 Oct 2023 10:09:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344631AbjJYOJZ (ORCPT ); Wed, 25 Oct 2023 10:09:25 -0400 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DA6A186; Wed, 25 Oct 2023 07:09:21 -0700 (PDT) Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-507962561adso8329398e87.0; Wed, 25 Oct 2023 07:09:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698242959; x=1698847759; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7SVLg2JlHugSU3IaNjtj7eUv+h55sIJkPWfpBUSnL5c=; b=UoPOWTtoUpzQvDdpbt8fx5db59sSPx/GOUajXdSfYdniZ1T7lm6aV3UlsY2XfUptcK uOc2xzEzseq76WRAAg45VscGBoMTQnIOJwuWs8XjB01MjwijRNzDBiyAV4Om30fGXudN 74vBG9XgmQw75Wfulet5WOmS1c0GR8zOpnzmliWWgAvhCVCZXhmQA9N+8Zq+S/BwFEyR wDU6ZZbb7LcOq8J/R/zBF5ykdJ/kP6Jx8J59UrqLBi7BZQMd3OxqvaXStB51yOKhYar7 pIt5hbshyHSV2sLyc1pg5aecNhO/8/Ysn32lAT18EHXP2dd1kUP3k2ovpxCkIodMievz 1c1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698242959; x=1698847759; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7SVLg2JlHugSU3IaNjtj7eUv+h55sIJkPWfpBUSnL5c=; b=M/gObXZlqKKwUpobGJYKmqFqPvlgmQPOoTOci8toUpAVDsflKrZhDZZdI2pOWSDA10 RKndy7rUAc2ntdM1uMmhW7S1quSqVDNG51c2vQXX014qYv5H1ydaxs+SGzEL84Q/KAGH M7tpsRlH5V3Pil/fBcUWw6PPkhn8KZGWV6g6/WWrcsxyaApIY1EaMTjBI6s39uQTYcsd UEwe+cjkjCYIerHTL/flIAliMreNaaKxUVvqIpVgOcTr9KNuRS0tC0OzFFwVU/PjiTa0 +HtehmFdMJwciRIPPSsE1DuYZhtObRcGD8TXRVn7Q99masDU4C6ZRZgY4rR6gSPSPWtm 7YHg== X-Gm-Message-State: AOJu0YzHql4q3ZQNU3QIyFd2b+XbCC5qsTM5AJvt1UHy4iiwc+yLQmPq DcEOaZuR+5OUXjA1AiWO4Nk= X-Received: by 2002:ac2:53a5:0:b0:4fd:c844:6a43 with SMTP id j5-20020ac253a5000000b004fdc8446a43mr10537490lfh.43.1698242959231; Wed, 25 Oct 2023 07:09:19 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id o8-20020ac24e88000000b00507977e9a38sm2555680lfr.35.2023.10.25.07.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:09:18 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH 1/3] rcu: Reduce synchronize_rcu() waiting time Date: Wed, 25 Oct 2023 16:09:13 +0200 Message-Id: <20231025140915.590390-2-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231025140915.590390-1-urezki@gmail.com> References: <20231025140915.590390-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:09:52 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780736859126172145 X-GMAIL-MSGID: 1780736859126172145 A call to a synchronize_rcu() can be optimized from time point of view. Different workloads can be affected by this especially the ones which use this API in its time critical sections. For example if CONFIG_RCU_NOCB_CPU is set, the wakeme_after_rcu() callback can be delayed and such delay depends on: - where in a nocb list it is located; - how fast previous callbacks completed. 1. On our Android devices i can easily trigger the scenario when it is a last in the list out of ~3600 callbacks: <...>-29 [001] d..1. 21950.145313: rcu_batch_start: rcu_preempt CBs=3613 bl=28 ... <...>-29 [001] ..... 21950.152578: rcu_invoke_callback: rcu_preempt rhp=00000000b2d6dee8 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152579: rcu_invoke_callback: rcu_preempt rhp=00000000a446f607 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152580: rcu_invoke_callback: rcu_preempt rhp=00000000a5cab03b func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152581: rcu_invoke_callback: rcu_preempt rhp=0000000013b7e5ee func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152582: rcu_invoke_callback: rcu_preempt rhp=000000000a8ca6f9 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152583: rcu_invoke_callback: rcu_preempt rhp=000000008f162ca8 func=wakeme_after_rcu.cfi_jt <...>-29 [001] d..1. 21950.152625: rcu_batch_end: rcu_preempt CBs-invoked=3612 idle=.... 2. We use cpuset/cgroup to classify tasks and assign them into different cgroups. For example "backgrond" group which binds tasks only to little CPUs or "foreground" which makes use of all CPUs. Tasks can be migrated between groups by a request if an acceleration is needed. See below an example how "surfaceflinger" task gets migrated. Initially it is located in the "system-background" cgroup which allows to run only on little cores. In order to speed it up it can be temporary moved into "foreground" cgroup which allows to use big/all CPUs: cgroup_attach_task(): -> cgroup_migrate_execute() -> cpuset_can_attach() -> percpu_down_write() -> rcu_sync_enter() -> synchronize_rcu() -> now move tasks to the new cgroup. -> cgroup_migrate_finish() rcuop/1-29 [000] ..... 7030.528570: rcu_invoke_callback: rcu_preempt rhp=00000000461605e0 func=wakeme_after_rcu.cfi_jt PERFD-SERVER-1855 [000] d..1. 7030.530293: cgroup_attach_task: dst_root=3 dst_id=22 dst_level=1 dst_path=/foreground pid=1900 comm=surfaceflinger PERFD-SERVER-1855 [000] d..1. 7030.530383: cgroup_attach_task: dst_root=3 dst_id=22 dst_level=1 dst_path=/foreground pid=1900 comm=surfaceflinger TimerDispatch-2768 [002] d..5. 7030.537542: sched_migrate_task: comm=surfaceflinger pid=1900 prio=98 orig_cpu=0 dest_cpu=4 "A moving time" depends on how fast synchronize_rcu() completes. See the first trace line. The migration has not occurred until the sync was done first. Please note, number of different callbacks to be invoked can be thousands. 3. To address this drawback, maintain a separate track that consists of synchronize_rcu() callers only. The GP-kthread, that drivers a GP either wake-ups a worker to drain all list or directly wakes-up end user if it is one in the drain list. 4. This patch improves the performance of synchronize_rcu() approximately by ~30% on synthetic tests. The real test case, camera launch time, shows below figures(time is in milliseconds): 542 vs 489 diff: 9% 540 vs 466 diff: 13% 518 vs 468 diff: 9% 531 vs 457 diff: 13% 548 vs 475 diff: 13% 509 vs 484 diff: 4% Synthetic test: Hardware: x86_64 64 CPUs, 64GB of memory - 60.000 tasks(simultaneous); - each task does(1000 loops) synchronize_rcu(); kfree(p); default: CONFIG_RCU_NOCB_CPU: takes 323 seconds to complete all users; patch: CONFIG_RCU_NOCB_CPU: takes 240 seconds to complete all users. Please note, by default this functionality is OFF and the old way is still used instead, In order to activate it, please do: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 150 +++++++++++++++++++++++++++++++++++++++++- kernel/rcu/tree_exp.h | 2 +- 2 files changed, 150 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 78554e7181dd..63e46a8f3e60 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1384,6 +1384,121 @@ static void rcu_poll_gp_seq_end_unlocked(unsigned long *snap) raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } +/* + * There are three lists for handling synchronize_rcu() users. + * A first list corresponds to new coming users, second for users + * which wait for a grace period and third is for which a grace + * period is passed. + */ +static struct sr_normal_state { + struct llist_head curr; /* request a GP users. */ + struct llist_head wait; /* wait for GP users. */ + struct llist_head done; /* ready for GP users. */ + + /* + * In order to quickly add a batch of nodes to already + * existing list, which is done, a tail is maintained. + */ + struct llist_node *wait_tail; +} sr; + +/* Disabled by default. */ +static int rcu_normal_wake_from_gp; +module_param(rcu_normal_wake_from_gp, int, 0644); + +static void rcu_sr_normal_complete(struct llist_node *node) +{ + struct rcu_synchronize *rs = container_of( + (struct rcu_head *) node, struct rcu_synchronize, head); + unsigned long oldstate = (unsigned long) rs->head.func; + + WARN_ONCE(!poll_state_synchronize_rcu(oldstate), + "A full grace period is not passed yet: %lu", + rcu_seq_diff(get_state_synchronize_rcu(), oldstate)); + + /* Finally. */ + complete(&rs->completion); +} + +static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) +{ + struct llist_node *done, *rcu, *next; + + done = llist_del_all(&sr.done); + if (!done) + return; + + llist_for_each_safe(rcu, next, done) + rcu_sr_normal_complete(rcu); +} +static DECLARE_WORK(sr_normal_gp_cleanup, rcu_sr_normal_gp_cleanup_work); + +/* + * This is hard-coded and it is a maximum number of + * synchronize_rcu() users(might be +1 extra), which + * are awaken directly by the rcu_gp_kthread(). The + * reset is deferred to a dedicated worker. + */ +#define MAX_SR_WAKE_FROM_GP 5 + +/* + * Helper function for rcu_gp_cleanup(). + */ +static void rcu_sr_normal_gp_cleanup(void) +{ + struct llist_node *head, *tail, *pos; + int i = 0; + + tail = READ_ONCE(sr.wait_tail); + head = llist_del_all(&sr.wait); + + llist_for_each_safe(pos, head, head) { + rcu_sr_normal_complete(pos); + + if (++i == MAX_SR_WAKE_FROM_GP) { + /* If last, process it also. */ + if (head && !head->next) + continue; + break; + } + } + + if (head) { + /* Can be not empty. */ + llist_add_batch(head, tail, &sr.done); + queue_work(system_highpri_wq, &sr_normal_gp_cleanup); + } +} + +/* + * Helper function for rcu_gp_init(). + */ +static void rcu_sr_normal_gp_init(void) +{ + struct llist_node *head, *tail; + + tail = llist_del_all(&sr.curr); + head = llist_reverse_order(tail); + + if (!head) + return; + + /* + * A waiting list of GP should be empty on this step, + * since a GP-kthread, rcu_gp_init() -> gp_cleanup(), + * rolls it over. If not, it is a BUG, warn a user. + */ + WARN_ON_ONCE(!llist_empty(&sr.wait)); + + WRITE_ONCE(sr.wait_tail, tail); + llist_add_batch(head, tail, &sr.wait); +} + +static void rcu_sr_normal_add_req(struct rcu_synchronize *rs) +{ + llist_add((struct llist_node *) &rs->head, &sr.curr); +} + /* * Initialize a new grace period. Return false if no grace period required. */ @@ -1418,6 +1533,7 @@ static noinline_for_stack bool rcu_gp_init(void) /* Record GP times before starting GP, hence rcu_seq_start(). */ rcu_seq_start(&rcu_state.gp_seq); ASSERT_EXCLUSIVE_WRITER(rcu_state.gp_seq); + rcu_sr_normal_gp_init(); trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq, TPS("start")); rcu_poll_gp_seq_start(&rcu_state.gp_seq_polled_snap); raw_spin_unlock_irq_rcu_node(rnp); @@ -1787,6 +1903,9 @@ static noinline void rcu_gp_cleanup(void) } raw_spin_unlock_irq_rcu_node(rnp); + // Make synchronize_rcu() users aware of the end of old grace period. + rcu_sr_normal_gp_cleanup(); + // If strict, make all CPUs aware of the end of the old grace period. if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) on_each_cpu(rcu_strict_gp_boundary, NULL, 0); @@ -3500,6 +3619,35 @@ static int rcu_blocking_is_gp(void) return true; } +/* + * Helper function for the synchronize_rcu() API. + */ +static void synchronize_rcu_normal(void) +{ + struct rcu_synchronize rs; + + if (READ_ONCE(rcu_normal_wake_from_gp)) { + init_rcu_head_on_stack(&rs.head); + init_completion(&rs.completion); + + /* + * This code might be preempted, therefore take a GP + * snapshot before adding a request. + */ + rs.head.func = (void *) get_state_synchronize_rcu(); + rcu_sr_normal_add_req(&rs); + + /* Kick a GP and start waiting. */ + (void) start_poll_synchronize_rcu(); + + /* Now we can wait. */ + wait_for_completion(&rs.completion); + destroy_rcu_head_on_stack(&rs.head); + } else { + wait_rcu_gp(call_rcu_hurry); + } +} + /** * synchronize_rcu - wait until a grace period has elapsed. * @@ -3551,7 +3699,7 @@ void synchronize_rcu(void) if (rcu_gp_is_expedited()) synchronize_rcu_expedited(); else - wait_rcu_gp(call_rcu_hurry); + synchronize_rcu_normal(); return; } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 6d7cea5d591f..279a37beb05a 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -987,7 +987,7 @@ void synchronize_rcu_expedited(void) /* If expedited grace periods are prohibited, fall back to normal. */ if (rcu_gp_is_normal()) { - wait_rcu_gp(call_rcu_hurry); + synchronize_rcu_normal(); return; } From patchwork Wed Oct 25 14:09:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 158120 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp2629890vqx; Wed, 25 Oct 2023 07:11:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IExOHq3FoV6CFhAaFWwo3mflGgsCcKlmpDqycDwCRNzUiVuJLiW/yav/O+S3Evb4kgUmCVR X-Received: by 2002:a25:8c83:0:b0:d9b:9aeb:8c26 with SMTP id m3-20020a258c83000000b00d9b9aeb8c26mr14442863ybl.40.1698243083815; Wed, 25 Oct 2023 07:11:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698243083; cv=none; d=google.com; s=arc-20160816; b=U9TGJiXrihSjMH0iHbVNGEKHv79K4VP6h1D2ilPlQoQ+32nxoJ8mgV06gMjxTw+n4j Nn/b/e0P4DN4VjKo/MXyCOOM9QQ4nFG1uNzi+GBCmxQ5neHznFkxJvDCvYAPRWrnuDdJ tqncP23mnR76M8Z9kRT3h8CGeSAbrqOKyp7n2No4BWx3YUN7vVrpgvDo9P5Fz4XT7TkZ mSlNWZSC95L7Kx1iMKjN+siJzBONzEVUlpy5C/M+q6kx5wbzWWF1I1FXuhxJy0Efs02X /r/LmU8toAP+DYC4JhRR7ZkFcAe4VrL/DUF5RVYNu74MEAyUI8EX4LjsmHzivFPtJsZE MTrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bHDEJ3lmtmiX3laFHnI04RPu1+R3BEM2SvSjdFI6IHw=; fh=4iZcRCmzpaCZKgp7nvxh/m3AX3kfm//yPbGO/jIXleI=; b=IO4F6lEJLAUqQqolLKs6P1kozAn4SWWOiHWth49wIYay7UzfGZg52qzPwaYEqrbQwh QBwPzjvkN2NXXVoGuhmzru6rREEOJqsAl3MNyLBO8Poh9LPksGCHr9g4ns0iguvFXG31 mV0nkGiN1qh/n1TLScjBPG82o34UgvP28tleVzLrzUDL3r/mVi51KMa7xAkvP+Twlmoz f5rTLRI2NSMImPucC7NOQ/x839znVJf8o5tf1yBaDUu3XaLZEUGrx0WaQVps0Aj0PXOy 7dqxBKIjbjTxzPxYBlPHe8vC93yiPT3lmZT9s2MhIBMp98DAcBWOvS4hVs7rfR85YYbr E9xA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=bV+G90lw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id v123-20020a252f81000000b00d4dea91ca51si9700956ybv.208.2023.10.25.07.11.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:11:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=bV+G90lw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 79B99806E173; Wed, 25 Oct 2023 07:10:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344639AbjJYOJe (ORCPT + 26 others); Wed, 25 Oct 2023 10:09:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344802AbjJYOJ1 (ORCPT ); Wed, 25 Oct 2023 10:09:27 -0400 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 354DD99; Wed, 25 Oct 2023 07:09:22 -0700 (PDT) Received: by mail-lf1-x12c.google.com with SMTP id 2adb3069b0e04-50802148be9so3291544e87.2; Wed, 25 Oct 2023 07:09:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698242960; x=1698847760; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bHDEJ3lmtmiX3laFHnI04RPu1+R3BEM2SvSjdFI6IHw=; b=bV+G90lwvN9BcfEZj224zGCwJFYBE24UNYELzszmLEq8mkLEjp4zimKfkDI9yTeKV9 tHvowpymvKUbzs823PhvQUIl3TmN9qD9h+ztXhz9fWWtU6NXMt6YaAjxnMW7CJlB4z0H hl+KEfxUgkkav6B1ZH+8I0oxwAmGOeGw6AM1pwBzh85kqK+h7o6K9ZSM+XDZQQEXL2Tj 9+f1nvk5q6NqczyiNqxhc85n6a+xVWtJv3SOxHUuaoZiIWUCaXPVsNnLeQCtbmcmSO45 H/EQx7XtOYK7SZGulI0qq/67QqhYp83hptZsVk7z4TnWBB1zOUbHQ3Sx1jGGQR7TXzp6 en+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698242960; x=1698847760; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bHDEJ3lmtmiX3laFHnI04RPu1+R3BEM2SvSjdFI6IHw=; b=EbTKbBkCf122pqXBmq5wkr+QbSWGeKnjLrP0zRRglMEnqAmUo5o2Isvx+np45rgbPO jEreXTQqGj5uq+EIoMIOSDzzhHt3wPFzV0GivUXwrbAmfNLYVPTX8jCSs8VIX+1+trrr 0wxvNGvGBYcaKPd+N6Scs/wovw5ATQSwJUqYsBon+aQu04HAf25hvzI4NvPt3c89WWFe +s9Z7sHKpRVIu4EAbcie+0ML1elSExjPgTrId1bClxFFnBZf8wLeQwOSLPGsrctCc9+G c3ZgvpIlL9Thvl6WKYyPihVIf+vV8KHGwVXicD+l4zZ/B2PxZSDSlHcUSdoDX58+hnR/ L7DA== X-Gm-Message-State: AOJu0Yx6gbZS2odTjBR+RV5WT0C48QL+RBctIewMdcQC1dSs8sdHdoFQ f+NPw7TKyI1webz9bEoaMgw= X-Received: by 2002:a19:ee16:0:b0:507:a5e7:724 with SMTP id g22-20020a19ee16000000b00507a5e70724mr10723334lfb.38.1698242960027; Wed, 25 Oct 2023 07:09:20 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id o8-20020ac24e88000000b00507977e9a38sm2555680lfr.35.2023.10.25.07.09.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:09:19 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH 2/3] rcu: Add a trace event for synchronize_rcu_normal() Date: Wed, 25 Oct 2023 16:09:14 +0200 Message-Id: <20231025140915.590390-3-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231025140915.590390-1-urezki@gmail.com> References: <20231025140915.590390-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:10:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780736940138351992 X-GMAIL-MSGID: 1780736940138351992 Add an rcu_sr_normal() trace event. It takes three arguments first one is the name of RCU flavour, second one is a user id which triggeres synchronize_rcu_normal() and last one is an event. There are two traces in the synchronize_rcu_normal(). On entry, when a new request is registered and on exit point when request is completed. Please note, CONFIG_RCU_TRACE=y is required to activate traces. Signed-off-by: Uladzislau Rezki (Sony) --- include/trace/events/rcu.h | 27 +++++++++++++++++++++++++++ kernel/rcu/tree.c | 2 ++ 2 files changed, 29 insertions(+) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 2ef9c719772a..31b3e0d3e65f 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -707,6 +707,33 @@ TRACE_EVENT_RCU(rcu_invoke_kfree_bulk_callback, __entry->rcuname, __entry->p, __entry->nr_records) ); +/* + * Tracepoint for a normal synchronize_rcu() states. The first argument + * is the RCU flavor, the second argument is a pointer to rcu_head the + * last one is an event. + */ +TRACE_EVENT_RCU(rcu_sr_normal, + + TP_PROTO(const char *rcuname, struct rcu_head *rhp, const char *srevent), + + TP_ARGS(rcuname, rhp, srevent), + + TP_STRUCT__entry( + __field(const char *, rcuname) + __field(void *, rhp) + __field(const char *, srevent) + ), + + TP_fast_assign( + __entry->rcuname = rcuname; + __entry->rhp = rhp; + __entry->srevent = srevent; + ), + + TP_printk("%s rhp=0x%p event=%s", + __entry->rcuname, __entry->rhp, __entry->srevent) +); + /* * Tracepoint for exiting rcu_do_batch after RCU callbacks have been * invoked. The first argument is the name of the RCU flavor, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 63e46a8f3e60..25c7618b4533 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3626,6 +3626,7 @@ static void synchronize_rcu_normal(void) { struct rcu_synchronize rs; + trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request")); if (READ_ONCE(rcu_normal_wake_from_gp)) { init_rcu_head_on_stack(&rs.head); init_completion(&rs.completion); @@ -3646,6 +3647,7 @@ static void synchronize_rcu_normal(void) } else { wait_rcu_gp(call_rcu_hurry); } + trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("complete")); } /** From patchwork Wed Oct 25 14:09:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 158118 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp2628815vqx; Wed, 25 Oct 2023 07:10:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFIVbA2ApQ/LC0Jq3j5coTaRlwGOtydEIVLIEjt2dX/95S++4YhHjiVL2eVAKski9TWSm91 X-Received: by 2002:a1f:2413:0:b0:496:b3b7:5d4c with SMTP id k19-20020a1f2413000000b00496b3b75d4cmr9181768vkk.16.1698243006059; Wed, 25 Oct 2023 07:10:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698243006; cv=none; d=google.com; s=arc-20160816; b=QmBOPU/Fif2ItnbhvCGkaLk5YvqD7JS3h3f9KPNKel42KpFXo+mEapfr6fnyskXFby xqmmCJXXy0nKSCTgku3Oo/5ntyybW33o7An0kuzPcScXL6lmB8LVmJrxVXyGUrVGq5zB H28dXA+9b1QCKq7yxEHVu+D8JjQUyKzxSUjHDVTI0RPNKRuDozTqeiCDlaoGiTkztn0+ BA99QNTVnm7txx/lzpWDQBSXvFc+TbSw0JHEtKWFSCSbfrLCljUhFcsPDJsRtf8cN6cv TzCWBrtnjJNadodsQwW5RcNloou63hiyTbAhDfix5XJCcvb6l7huecY/1XWwXYb6bEXW teAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kH2SzlXV0J1R/rkJlHdBb1iPR68EpEWsCPbBxpCRKIE=; fh=4iZcRCmzpaCZKgp7nvxh/m3AX3kfm//yPbGO/jIXleI=; b=CGpRsWZm6t6zzHhlywjwpXpk+fX03dKXjAjBv8NVaXv96B/tbX03rPH0i15iCSbrTa E25rQ0LNc6g0X63QLpj4xj49bdD+oIlDpb5yKpjGH6InUeFpulX9QVod7vqtLeVoeNIE BpbtzQHYxBS+D6UXkB3bnepjOedgOg5lcw+ccYF6bzBDP/bUtmUbSNOoLE5LQzbt7k3l /KdI/DjO/6cxTF/maC9rNH3IvOiS+QOh/eHvM4xLeAQx6tcOB1lopNfVsNFsLtZp++ku 7A6ScKOr/6c1hxXIfqcyBRPJ1xFSAmhViZrGwmrdrua3hQNDt9UiA1c6/oWxirtBakNZ nU2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=UGT24DF9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id w188-20020a1f94c5000000b0049d1a631709si1460582vkd.35.2023.10.25.07.10.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:10:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=UGT24DF9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 4CEEF80F61A6; Wed, 25 Oct 2023 07:09:56 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344815AbjJYOJ2 (ORCPT + 26 others); Wed, 25 Oct 2023 10:09:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344639AbjJYOJZ (ORCPT ); Wed, 25 Oct 2023 10:09:25 -0400 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD73018A; Wed, 25 Oct 2023 07:09:22 -0700 (PDT) Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-508126afb9bso1752334e87.0; Wed, 25 Oct 2023 07:09:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698242961; x=1698847761; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kH2SzlXV0J1R/rkJlHdBb1iPR68EpEWsCPbBxpCRKIE=; b=UGT24DF9t1jtKOXD0m5KmLlMjjZUlmZips6OnNlV92AHhVo1AOSVOkM0hqNfhQydA7 ELwTZDFuk6rqzn7hPFrRoc5QZxkinpqGwk8l+Gm9kTlV+1E3WTEZFFGNmXJiAMdeRhB1 O6ABhSjbf/1uHSw3FCr4y8Ag2BzvZNxlbVEY0h/H4GJATkxz1NNq1hiBUu4zKxIK06EM OYmc33Kv1ydUwEYuURaVVHAPPKhFYjyO9hF8koLWcDACqK4wHHmkzQdbTPjgYlRPvzuI dx6flrz3wnPS6owNsuY3Tio+LQ3Bj8slxKc+JkAulpSUAiUKYFsqRMS045LFiI+hju8j PaEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698242961; x=1698847761; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kH2SzlXV0J1R/rkJlHdBb1iPR68EpEWsCPbBxpCRKIE=; b=BlBYS1cPagxjdBG/lMQH2hHrjEN9K5vwDyY7OMVm0ySk1F/wMXBD0L8Vo1OpuX0bHd 1FrthcormjGgVA38a83jS7MJoj2EWoXYo3lKWS6MOswOpbViWswbrNdt0GFQFHt645w6 +IBNYW/gWW+nVN9JGYtl+PX+ZnXMEsRmpRpBxybq01533iv3eMtxdCl2drDW1vNMUw6g UVvKVkl40/DX5QuxlLe8udxzfKTB4N/a8QCmDW2d0fsaKWV/hRfLPAa4LhuBksef/ljZ tnKA0jaCDxxRI41QSqwO6GTWODQtPa4P8Mrp8lBUjQgIKRilb+YP8OCoPRaim+7NDdkL +G2A== X-Gm-Message-State: AOJu0YzaZADzrekP5DR8b2Dwj+v/jvDlb7HezlnBO5PFUwlnFd6vEKgB 6NA46G53ABrKgg7req6BYo8= X-Received: by 2002:a05:6512:314e:b0:507:9702:c11d with SMTP id s14-20020a056512314e00b005079702c11dmr9580249lfi.64.1698242960848; Wed, 25 Oct 2023 07:09:20 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id o8-20020ac24e88000000b00507977e9a38sm2555680lfr.35.2023.10.25.07.09.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:09:20 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH 3/3] doc: Add rcutree.rcu_normal_wake_from_gp to kernel-parameters.txt Date: Wed, 25 Oct 2023 16:09:15 +0200 Message-Id: <20231025140915.590390-4-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231025140915.590390-1-urezki@gmail.com> References: <20231025140915.590390-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:09:56 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780736858690129907 X-GMAIL-MSGID: 1780736858690129907 This commit adds rcutree.rcu_normal_wake_from_gp description to the kernel-parameters.txt file. Signed-off-by: Uladzislau Rezki (Sony) --- Documentation/admin-guide/kernel-parameters.txt | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 03a4bdb7512b..f41bcf17ff58 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5009,6 +5009,18 @@ this kernel boot parameter, forcibly setting it to zero. + rcutree.rcu_normal_wake_from_gp= [KNL] + Reduces a latency of synchronize_rcu() call. This approach + maintains its own track of synchronize_rcu() callers, so it + does not interact with regular callbacks because it does not + use a call_rcu[_hurry]() path. + + Please note, the rcu_normal_wake_from_gp parameter takes effect + if the rcupdate.rcu_normal is active also. + + Faster version is preferred set to 1. + Default is 0. + rcuscale.gp_async= [KNL] Measure performance of asynchronous grace-period primitives such as call_rcu().