From patchwork Wed Oct 4 12:34:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148254 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98398vqb; Wed, 4 Oct 2023 05:36:27 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFpBFvIuMgnBCYhGBOctr8+wXehk9C8ZdPs1JdcgGJyrJIhjE0XDsJUh8tDjPMuJD4vFLjv X-Received: by 2002:a05:6e02:b4c:b0:34c:b992:58b1 with SMTP id f12-20020a056e020b4c00b0034cb99258b1mr2767719ilu.2.1696422987408; Wed, 04 Oct 2023 05:36:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422987; cv=none; d=google.com; s=arc-20160816; b=pbSO+uhWADYDmTbGQkruz8w5f1GTVNSZaIGTbyy3VXgmdoyWSkGK/hJ8i0SJthLwYd 2KTPgou+tWquQh/IZdYU7mSZMYjPtIIpSJWfAqRZR7nliIY7br95UFM55wwnPW2/ujvk 1BPUO6Dgj83m8b2UlN32x9UE5/1KJZeshqrkMsijJsY5sbLRsU7eaxxKEIl1o2p3XSTD adcMHRcbP8Nli9Qa78Rgq3bIKLco81kllYS6n3xxkhAAmjSvEsdeMRA/b5RYCMh7t+8u EtgDaIJbxxfxFKz/J0XoyvIT7RVZVc9VsWr6iTGY6b388Kke3Mxm8l2enEk2MnkXaJ/T 1rmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=2ZHSfoxNVfe828MrfNDKWiKa0rU0JZbDLb7SUO4brts=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=0l24kQw0+/GugYlvlbHk+yUVe8hn4Y2QVPuBJ/m9lb+Y/3K8C5PuzMK8NZ3gHlolaq 8wssI2yrN0uJjkSrh6E3qYkt33mCzcOPKXDuMFVBb7/qU0tXULS2GAPZ0qSkCBj/auOA KDcpCpczha/ZNS+OLA6aKYB5KMZ/CEqI4IJSi8NUFitfb5vLxbWUs96u6OB2QQ6r6VCt ne8nBriMISrqeWkY+2h/PoJtWFYeEgzYToL+VcRoMu1eYlJ+3x9U9DAAk7tGLHmU7Uq6 HZNbQt6fPP0GfqBtf19CbHYPhUvPWQ95O7T51jWkutC4FxVMV3NTaQmRtgywlGiFazEo KsSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="XKS1DMu/"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id g25-20020a633759000000b005859de837c9si3760662pgn.105.2023.10.04.05.36.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="XKS1DMu/"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 65563807214D; Wed, 4 Oct 2023 05:35:35 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233247AbjJDMfc (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233099AbjJDMfa (ORCPT ); Wed, 4 Oct 2023 08:35:30 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3C1ABF for ; Wed, 4 Oct 2023 05:35:27 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZHSfoxNVfe828MrfNDKWiKa0rU0JZbDLb7SUO4brts=; b=XKS1DMu/waoOHui1J6PmDLbV3W5XSSy+sqIC/BixILtS8JJQAtNGc7wFqsXwk4os7DeJBI gIvdpDDtQYAy5CDTO1uuvPwtJYkyGGxkDj/B7B+/M3yIxdCiDvoouIEa/d2BFQELmjo+8C E9MNouWmy42JTVdrwvvoSZEFHvU0iMj7+H8DOwrz8/BO6WboqMR/el3Qr9jk0Esn9D5RcR SlKq93yYc2Ib6Uq/O2yw9RUWMGN3wxS8Yx3HCU05bGXsnG7rk0aipwsxT266Y4xl/sjiqA aNtkeuCN6lLuKQmB91ySFoLNfjzI9W3cCNkV5plz/KnFFmFsHKhL6pMEHZjzAg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ZHSfoxNVfe828MrfNDKWiKa0rU0JZbDLb7SUO4brts=; b=tbs8E8IzWgWCFdOYwM8uWCeZ8Q6ekfq3O5YC6QDuLtnXEf0o2yjyED6kp87FJBA1RoAHpm Y8aaEpV1MmJ6OvBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 01/25] tick/sched: Cleanup confusing variables Date: Wed, 4 Oct 2023 14:34:30 +0200 Message-Id: <20231004123454.15691-2-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:35 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828430414174409 X-GMAIL-MSGID: 1778828430414174409 tick_nohz_stop_tick() contains the expires (u64 variable) and tick (ktime_t) variable. In the beginning the value of expires is written to tick. Afterwards non of the variables is changed. They are only used for checks. Drop the not required variable tick and use always expires instead. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 8e9a9dcf60d5..bdad2abdc0c7 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -886,7 +886,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); u64 basemono = ts->timer_expires_base; u64 expires = ts->timer_expires; - ktime_t tick = expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; @@ -909,7 +908,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) /* Skip reprogram of event if its not changed */ if (ts->tick_stopped && (expires == ts->next_tick)) { /* Sanity check: make sure clockevent is actually programmed */ - if (tick == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) + if (expires == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) return; WARN_ON_ONCE(1); @@ -934,7 +933,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) trace_tick_stop(1, TICK_DEP_MASK_NONE); } - ts->next_tick = tick; + ts->next_tick = expires; /* * If the expiration time == KTIME_MAX, then we simply stop @@ -949,11 +948,11 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) } if (ts->nohz_mode == NOHZ_MODE_HIGHRES) { - hrtimer_start(&ts->sched_timer, tick, + hrtimer_start(&ts->sched_timer, expires, HRTIMER_MODE_ABS_PINNED_HARD); } else { - hrtimer_set_expires(&ts->sched_timer, tick); - tick_program_event(tick, 1); + hrtimer_set_expires(&ts->sched_timer, expires); + tick_program_event(expires, 1); } } From patchwork Wed Oct 4 12:34:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148256 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98455vqb; Wed, 4 Oct 2023 05:36:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHPofLG74TbVqSliXpshxEo5uJhlcvSJT9xx3VBxQsiaV0s0CMExrQrnGeNL++Vi4J0Q9Vk X-Received: by 2002:a17:90a:f414:b0:271:8195:8 with SMTP id ch20-20020a17090af41400b0027181950008mr1905651pjb.36.1696422991431; Wed, 04 Oct 2023 05:36:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422991; cv=none; d=google.com; s=arc-20160816; b=DnZbBIZ6hovMMr2zgTBS4nBPfRE8BNwUI50eSKphPUj9rgFvdj1/RpsWZuVy8PE1IC q2tDtjLG9t95wgvPyigNmjXiVmMh29BDgf7m+ci2cDM4OAMy/1bjINUUiu5ZBTFOHUe+ oDRwy0K04IW6MVfML6eZoB80xBhNyxGn0n4GtBVrwVBM75/aC5ukfDu6+A5QfgX+7Pit 6IFwcLJg4H6y5WAWiZxiijzbY6QrhhdlHK4EfoNRLJ6qJqPLIcsCo+GVbcNm9Bfwj1ul TxSJWsFLPDIK058Uy0TGH10HFWKPwHUhwoqEYWfyfCtfDKvKmTG359eUccOrnlTSekIx Fx4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=Qz4nJBaG6XRtjj3mRkMTDjA4wkxAWKGzl0cHJtp9Oe0=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=kIrI+iyzHL/R4NUkcv7d5TMYVtPgdMQDYCrKY5x+qpk8/Q5sxITTJpEFRxNQwTEeYt LPJzWvPJx22fprScw4I07W7EyL9Q0h0aoa6Ltmi47xwHRCrvI4k2DtMRQmKtktBRRn7t ZOA3aKxo7/eSXhl9MPT3dxmeOtMieTJV0a57dZbRjjOsSxtqf8hFIw7ogam0/aWu1ARH ldJQthA7H3NosBT6g7x8IHWTmUlnincZ3djk6PhktewUsbDeopSSoHCXb5JHhQGGN0I2 ++4Omqr0lxs3E1gbQfdkBzS2WynN6SUdwf6DS1P/57us2Ep9t1+ZxkKL8oiR35yOPjfH 3dhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BIozC7Py; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id lr17-20020a17090b4b9100b002747da1ef66si1523346pjb.53.2023.10.04.05.36.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BIozC7Py; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 4ADE18045959; Wed, 4 Oct 2023 05:35:46 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242375AbjJDMff (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233336AbjJDMfc (ORCPT ); Wed, 4 Oct 2023 08:35:32 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A09CB0 for ; Wed, 4 Oct 2023 05:35:28 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qz4nJBaG6XRtjj3mRkMTDjA4wkxAWKGzl0cHJtp9Oe0=; b=BIozC7PyZ4VhQOCf4rAfBpbcQ7TgnF/JfDtlw7mlsRDKvWNbWOyEamfwSre+uzcIh8bmNQ MNg+c1mk0p6wtXauMY8pQrNRb5+qGggoexuscNVjelnE8Sv6zezgM6Ym7tiaEQTBHbKlEo t9qGlU7wzEpmqeQ2yWwqvCUsvocrgXbc/Zg2Dn4AEpdo2TNzL3j7PGIzH2cpdBvwDtFYEI 6HlUqNBd9Evc7k30v5rm1GNYhBITNjeTggg3PwIjZUKVEx/vEkrRdLvouCEyODoFyL2c3q /WGvqyUNT2/uvcFL45xHRBWxSP5Qm4DAB1CTgpc2yLbS/xQag2tChfePA+o7Ew== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qz4nJBaG6XRtjj3mRkMTDjA4wkxAWKGzl0cHJtp9Oe0=; b=CHFn5XMk74YVpQGH5n+8y0lqHnXKS2zTHVWFKCa+56NDFEzcfcQW4uiJDc6bqoVS8IKl4w majPpeh3Rwnn8WBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 02/25] tick-sched: Warn when next tick seems to be in the past Date: Wed, 4 Oct 2023 14:34:31 +0200 Message-Id: <20231004123454.15691-3-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:46 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828434734728123 X-GMAIL-MSGID: 1778828434734728123 When the next tick is in the past, the delta between basemono and the next tick gets negativ. But the next tick should never be in the past. The negative effect of a wrong next tick might be a stop of the tick and timers might expire late. To prevent expensive debugging when changing underlying code, add a WARN_ON_ONCE into this code path. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index bdad2abdc0c7..b3cf535881a4 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -842,6 +842,8 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * If the tick is due in the next period, keep it ticking or * force prod the timer. */ + WARN_ON_ONCE(basemono > next_tick); + delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { /* From patchwork Wed Oct 4 12:34:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148255 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98439vqb; Wed, 4 Oct 2023 05:36:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHuFXOb5afkaSNXx/BU1MDSAm/McHURHXvKEh4gRoxUNjfmQGc48B/5d5Kwb3x7Widv4Wnl X-Received: by 2002:a17:90b:33cc:b0:277:4497:f0c2 with SMTP id lk12-20020a17090b33cc00b002774497f0c2mr1998606pjb.39.1696422990449; Wed, 04 Oct 2023 05:36:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422990; cv=none; d=google.com; s=arc-20160816; b=0pNd0xtrcOlVx+b1oWD5hpRlNkWknrrFYE5G7I2hwDRXTR4d2HBb+LMxoUFXNfjQwr 43g3vqLsYbZZHi0nKANtFKsFDAsL4IYtfZ5PvzrgaDrhDQhDlSzB2gtLpFO3msR8DQAF aWRzK7JWx6JSEKlxOnacEpnwz0mJuREbvGUyI7vco21uZ9Aq98JV5+M4uD1+FKU/bfbu XiNBg+cOLdVjYnFXnggrLD6ZKU7xU+ciGGDJ1QWB8QyjZ1UTlyYTLKpDZY1jeSR+OiCa rIb4wn6JOhukM93p/P9HzFD+ctu3eNFRRumIYe5Tw1uVbi++LXez48aFnPK4R6kykidR ktdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=AXGP/cV38TkuwT0IRhJ9YYY7aFG2G6iSF6EqZ7aAS+o=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=U1Gbn9e2bh0PEJoi3mlJy3NpbbYf02Nd5oIOq1uaW5pofYou+rL+CM2kN9CxIUP9fo kH1UFbgkbdKaLdR0hsulTfhMywajU/7Bgj+whQQ1UCjbjp95TbPQXZAdywxzKcmzUSwi nSF2kzHjiXMSpTD5khw5yHjLSU5/pGxikbR5dvF47+YPxcxHv0k3kAACaW0x90UW9w64 GqY4pEuRzIbTUZwxVUBWaZqmTBQw5WXxaZJxphOwm30bLC2vTwsEkvemQ0igk+GVpfTp JTfD5RXJPL7iBxm8z+JlSJ2zrbq9UjiRG7sYMLTze8BBhJIeIUPX1Vsn10RH6W7Wz8pn opoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=mCLdP9Sz; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GWXVBCrm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id pi4-20020a17090b1e4400b00277636a9e5csi1484563pjb.96.2023.10.04.05.36.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=mCLdP9Sz; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GWXVBCrm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 05E0380B1BDA; Wed, 4 Oct 2023 05:35:46 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242383AbjJDMfi (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242313AbjJDMfc (ORCPT ); Wed, 4 Oct 2023 08:35:32 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19EF8BF for ; Wed, 4 Oct 2023 05:35:29 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AXGP/cV38TkuwT0IRhJ9YYY7aFG2G6iSF6EqZ7aAS+o=; b=mCLdP9Sz8EnBzXJX1cpWsLaMbxR7yBHBthK1dXImUp5/1a4RJY6/2zTW9g9VG6gxLbEvqB LB2zeTRuEBPLCCKi+VE7dWf6Wv9JBXfrDO9D74AyVgcMYJ1gHqXZ3ts5N4npijeFIx3DVS kTJg98s0HfyGgpqdC6SeijQ4uy3dwi9efOfXk7q9gxXvoafo61HdiFs9pEt3Nbusy4hbT1 hH9wZJshZI6kvoop02euzfKTrCm+kFU8NvNWVmxqx7us6gPr+eWw1YzCfCTS4vZTqB5HCj dFFL4hQ39Gyt2IiUpM+3QqV5+wBEcqceFz5bL0JMvqgypKiG4NA8en71090VcQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AXGP/cV38TkuwT0IRhJ9YYY7aFG2G6iSF6EqZ7aAS+o=; b=GWXVBCrmoEVjjFNM0cucO6c5WGmhbmj1qsAX4zXMVXYQfZsTYVYDDA0RuF72hw13/S3+U5 UmBtuN00urfORyBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 03/25] timer: Do not IPI for deferrable timers Date: Wed, 4 Oct 2023 14:34:32 +0200 Message-Id: <20231004123454.15691-4-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:46 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828433938674293 X-GMAIL-MSGID: 1778828433938674293 Deferrable timers do not prevent CPU from going idle and are not taken into account on idle path. Sending an IPI to a remote CPU when a new first deferrable timer was enqueued will wake up the remote CPU and but nothing will be done regarding the deferrable timers. Drop IPI completely when a new first deferrable timer was enqueued. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v8: Update comment v6: new patch --- kernel/time/timer.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 63a8ce7177dd..7d06351aff54 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -571,18 +571,15 @@ static int calc_wheel_index(unsigned long expires, unsigned long clk, static void trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) { - if (!is_timers_nohz_active()) - return; - /* - * TODO: This wants some optimizing similar to the code below, but we - * will do that when we switch from push to pull for deferrable timers. + * Deferrable timers do not prevent CPU from entering dynticks and are + * not taken into account on idle/nohz_full path. An IPI when a new + * deferrable timer is enqueued will wake up the remote CPU but nothing + * will be done with the deferrable timer base. Therefore skip remote + * IPI for deferrable timers completely. */ - if (timer->flags & TIMER_DEFERRABLE) { - if (tick_nohz_full_cpu(base->cpu)) - wake_up_nohz_cpu(base->cpu); + if (!is_timers_nohz_active() || timer->flags & TIMER_DEFERRABLE) return; - } /* * We might have to IPI the remote CPU if the base is idle and the From patchwork Wed Oct 4 12:34:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148257 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98457vqb; Wed, 4 Oct 2023 05:36:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH1THCt29XKwvF6X8lYwaQGZ+mvFfTG3uLdXLWLTnPQyvl7zt7qvlXPbBcT8a1dFYYCpEOd X-Received: by 2002:a05:6a20:7d85:b0:15c:b7bb:2bb5 with SMTP id v5-20020a056a207d8500b0015cb7bb2bb5mr2658212pzj.7.1696422991540; Wed, 04 Oct 2023 05:36:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422991; cv=none; d=google.com; s=arc-20160816; b=FtQNVMF15HrFu4EUVjPOz4ZAZwHKzsZvg9ePDwttWgdWt1SpGTIWaDoC53Nq15HLzl +54Zq32T7q9Kurh3eJBMkR8szzVW39tRH9K0z2DrIYi77DB2jvjcoTHos4Sm5HZX8UgM BmRwZh3ya6bemtAh/J52jS1j7IW/b7wUyb7FfhfyrZov+07XS3E0jSWPL18gowr6U3wl AQTkgXuG5SuGq7jqTpZedCJu4Ee/qSGNtQ5iKP4iwivRTQVRwY8asSbtSv4UePW/eFMq eXQokr67ZKDuLrFsrjoqDnQVLnUbh3U4yUNEfLNaQ+7BwUtxvmNxxX/4ljkoYnXI/f3+ Smaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=MAxMuF84OJMFwj1mVQcHjPmQ7zTcHiYKK0tzuJi+A1g=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=FjB3AuNMr6wr7fOpFxnC170uQvoMeeJopQJSfbakyWrmumh+jc8sYxmwASNX8wyrwI bAy0QrVvgGgTzJqCYqxSIVLrS337tg3CAkRyLMP/VGr746iYrLvvF0ZgACCcqVDBou5W T1BixiXfNHSH5g5+zLkQ0AMYA0gThUd5NVVd2LNDOtw+50DE/WSXDGat1a3xcu0SKPci Y1nHmpvwLGamb/p06+H7XATBydU1Qj9c9QhdDcwIBl79I58LEvgZv57MSJaR0HyeBvE1 wNgMQs6kwhbCcc33Ab8x5/RL0ZtOwCNQZGHAjztagT7kBfFj1LLSGe1aGb0bLPrbvoe6 HDtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=NGiVHVMK; dkim=neutral (no key) header.i=@linutronix.de header.b=gKxbWVn7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id k8-20020a056a00134800b0068fb4842092si3817911pfu.183.2023.10.04.05.36.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=NGiVHVMK; dkim=neutral (no key) header.i=@linutronix.de header.b=gKxbWVn7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 5061B802C8ED; Wed, 4 Oct 2023 05:35:49 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242389AbjJDMfl (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242353AbjJDMfc (ORCPT ); Wed, 4 Oct 2023 08:35:32 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E5EEC1 for ; Wed, 4 Oct 2023 05:35:29 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MAxMuF84OJMFwj1mVQcHjPmQ7zTcHiYKK0tzuJi+A1g=; b=NGiVHVMKQ89c2HqOfoJKJwMmDbG3+ohTltyLW1E7IzR22caosCulch2P490f0kWBtaRGbd KP5bDXJydfUfFPuGaFqTJoEaiU4X8XdW6VuMYFTjcxawK5VKBkZp+P2ru5yzT2IChIeBl2 h9hafLvRBdvbqm0ZSRTQGkIVtuQFJzvxqKInsDdDPIe81yEn0wy1tWSjjQPQSDbV/6nhi7 2FxeM2e2LTe97EvaNLypD/daOhdNiUaZmnv+Q8j7qp8W8Z9S7NKJCWlmEyeQhv7w42HX9O rLCLIWFywCAaqkpPlmTd+TFh4v4M0AjIXALAy3o2Gjit+veJaQMQ0+7mS/TAfg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MAxMuF84OJMFwj1mVQcHjPmQ7zTcHiYKK0tzuJi+A1g=; b=gKxbWVn7euG7rhu4l9t6pbj4lH9l8lg3FF9JrO40qXbMHGCaS/OLkU6TC5pL3M6jikIBzB /JJFsQ9FYtojaeBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 04/25] timer: Move store of next event into __next_timer_interrupt() Date: Wed, 4 Oct 2023 14:34:33 +0200 Message-Id: <20231004123454.15691-5-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:49 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828435077614258 X-GMAIL-MSGID: 1778828435077614258 Both call sites of __next_timer_interrupt() store return value directly in base->next_expiry. Move the store into __next_timer_interrupt() and to make its purpose more clear, rename function to next_expiry_recalc(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- v6: Fix typos in commit message and drop not required return as suggested by Peter Zijlstra v4: rename function as suggested by Frederic Weisbecker --- kernel/time/timer.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7d06351aff54..5e17244a9465 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1800,8 +1800,10 @@ static int next_pending_bucket(struct timer_base *base, unsigned offset, /* * Search the first expiring timer in the various clock levels. Caller must * hold base->lock. + * + * Store next expiry time in base->next_expiry. */ -static unsigned long __next_timer_interrupt(struct timer_base *base) +static void next_expiry_recalc(struct timer_base *base) { unsigned long clk, next, adj; unsigned lvl, offset = 0; @@ -1867,10 +1869,9 @@ static unsigned long __next_timer_interrupt(struct timer_base *base) clk += adj; } + base->next_expiry = next; base->next_expiry_recalc = false; base->timers_pending = !(next == base->clk + NEXT_TIMER_MAX_DELTA); - - return next; } #ifdef CONFIG_NO_HZ_COMMON @@ -1930,7 +1931,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base->lock); if (base->next_expiry_recalc) - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); nextevt = base->next_expiry; /* @@ -2013,7 +2014,7 @@ static inline void __run_timers(struct timer_base *base) WARN_ON_ONCE(!levels && !base->next_expiry_recalc && base->timers_pending); base->clk++; - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); while (levels--) expire_timers(base, heads + levels); From patchwork Wed Oct 4 12:34:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148258 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98459vqb; Wed, 4 Oct 2023 05:36:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE/Z6qJF89M5Po4Ja/amhdKy4wxKKbXhtIpwswO2VA7XIVc4zAjuAo76WjQB9a+AlKCcDpi X-Received: by 2002:a17:902:c449:b0:1c5:a09e:170a with SMTP id m9-20020a170902c44900b001c5a09e170amr1785488plm.54.1696422991531; Wed, 04 Oct 2023 05:36:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422991; cv=none; d=google.com; s=arc-20160816; b=o4ZVMKyWvXrMtjX1qoz6kt8eCtzhQedpS9y6bBw7QIC9QW09WRa/oB0O1Nki2s55ce uVRt+VfScnkh00iUOUK3G2Q63f8kppOAoPHcFKqbubLE12chVUsOnA3wnwLewVrHz0vx qtgjT6AGNp7NcFsVGUkH0JU6f6NxxN5AkjWolLQgGjZJjegb4Js25A2E36GVahKvGVtW q03G/o0Pjcbt3pEIvPanoZp3MJ2Qa6AQdCUDehxTtIW1iiGIDRD0CtQGErC7krlPCnnK fOXHxFCo5jrLJMiMK2CvuIfxV5mJQAD4GdsFpGuREYA/q2/e7h13LJdDTXeaSe/MulRo 2AmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=fTra8o64I3zbTAJb74gQJVLr6rBx5QpDSUDk/VGWLOU=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=k3RwC/IVgJkkhaaWtSpsxlNq+kjL8l7IIIWjZ5acYxWITCH8rpabdyhur9kFN+WZRa 4Z4pjPHnIzv3TUbU2W5MDIDJQNL3OVx/laxyACqc/30h0Rw6M3T1mKirqj6RehFFY5GV ajpNYg2eLca9o8p1gTBelCld1ZrQLmpJi2vK638pyT1h8VUEgGeyyfbWso76obrHBmsc 8wIcxEAL7K1OKQOi1m4FlBDlDurM7pgmxJlaD8U0qLHNxsclVzsPvrGgVrhEFpNnOfQT npxYnexVw8BmWCtf9vzPDkzwSBvJhx/+4cQ2x+HyhsVyHroG62Iad6NukUPUlsR42+73 RBTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="sA0UW/pv"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=rPMWo+lO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id c7-20020a170902d48700b001c741de8529si3839465plg.380.2023.10.04.05.36.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="sA0UW/pv"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=rPMWo+lO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id E8C728031C47; Wed, 4 Oct 2023 05:35:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242427AbjJDMfp (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233099AbjJDMfd (ORCPT ); Wed, 4 Oct 2023 08:35:33 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AF98A9 for ; Wed, 4 Oct 2023 05:35:30 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fTra8o64I3zbTAJb74gQJVLr6rBx5QpDSUDk/VGWLOU=; b=sA0UW/pvV/0I55AunJMU0m83CMjHJk5r5kdLxlyfE5RHs6n2hfsQX33oRxvi9/LC/hLyyf acAxxQ3LRzUlbNQApBnpa9BDd6PU0ElLSc88hLv0grsakqDGORuuYTfl5ytsnTF/bMcT+g LsWtmKJB8Z932SK3BDWZQXgpBIvlwm3vnS6oVRM18V+RDQzbyOV5I8+v9wg8dSCRrX6sZb CX3TGzAQNFADyBimYEDHtOmCjkjKt1NZMvyELWh8/IlYe9CSvymheg5VuMVYjyh/h3+nzS qwmncbE0c2RxFcLJ4fwM4jVwBwhPUiKpz4odW2lj6e1xqfYajr6buP+Ip0gpMw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fTra8o64I3zbTAJb74gQJVLr6rBx5QpDSUDk/VGWLOU=; b=rPMWo+lOy4PE80VFeDYCp2Y6PFaJPY1u8tmVAKOXxSKx1uhqwSb5Gl87jgPVYAz7ibOnqC WpV+cYoGPUx5t9Bw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 05/25] timers: Clarify check in forward_timer_base() Date: Wed, 4 Oct 2023 14:34:34 +0200 Message-Id: <20231004123454.15691-6-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:51 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828434784424749 X-GMAIL-MSGID: 1778828434784424749 The current check whether a forward of the timer base is required can be simplified by using an already existing comparison function which is easier to read. The related comment is outdated and was not updated when the check changed in commit 36cd28a4cdd0 ("timers: Lower base clock forwarding threshold"). Use time_before_eq() for the check and replace the comment by copying the comment from the same check inside get_next_timer_interrupt(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 5e17244a9465..31aed8353db1 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -944,11 +944,10 @@ static inline void forward_timer_base(struct timer_base *base) unsigned long jnow = READ_ONCE(jiffies); /* - * No need to forward if we are close enough below jiffies. - * Also while executing timers, base->clk is 1 offset ahead - * of jiffies to avoid endless requeuing to current jiffies. + * Check whether we can forward the base. We can only do that when + * @basej is past base->clk otherwise we might rewind base->clk. */ - if ((long)(jnow - base->clk) < 1) + if (time_before_eq(jnow, base->clk)) return; /* From patchwork Wed Oct 4 12:34:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp99271vqb; Wed, 4 Oct 2023 05:37:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEI1NtGK8pCoYWBZnKkB6ZPzCU4EnFvHnVhoTKp1chrB844PEJI9Kkn+XjpQ1DgJpl2Vr1K X-Received: by 2002:a05:6a21:a5aa:b0:15d:fc71:1b9e with SMTP id gd42-20020a056a21a5aa00b0015dfc711b9emr2850333pzc.49.1696423050343; Wed, 04 Oct 2023 05:37:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423050; cv=none; d=google.com; s=arc-20160816; b=fvxJE+xHuMN6wKEGi4tSYpO3l3rLLcu5I2F/g086tsEnCiSPllfcuQXRdaBLH18hEf whcZXO4VfnPHqAOFnxRNZbu3cRohZwifIMTeYgkOTJDSXf6EAg3VrKEpuaV8v8CqQ3gl e3J2MVTsvu2jJYgpkL/zLKGkKWZkvhSuzNqyi5HqRPMF+xY+oROQw3asp03Ilyc+iFVc kAIi/jIe6XyQ2GnsTtPh8VmAV4FGQxIg7exIZUrIbncOUCCh4me25i1Mi9d5yqQwH2Y/ iLzVLvO77K2OotY88mnEWzLhanT6lkaMNje0oFNpg5qdYf5tQNDK+lChKC/lQ9OZXBoh IsaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=W54m7rvqf/KqCMpTV79oOtmZr9LItvdWHEN45lcUeaw=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=vieOwP1SFbs7kwijKG18mxnBygxpR7/DWpqbnWVPbnvqPCRW1IFs2YVRPv+GQJtYQY ycwziJfAA+nBkikaxKIhJZsQszPVpOIOm0SZlwQUiZ97gukc1ctaueSL0863FH6mieBB HLBOKxCxlJuwIzhGBBaE+O7OKNthXqR41K3zT9k8ElvDOy0GMLrirTSSiZLBr4eyhYIm cHku82QXbDtxxqThbHrMAcLoaiAkLDjvUnrvwVp1zh7VcXMgWJaX8lZLHFpZlhDLr8QL 6QjKZn/G4oH5q9U98TVEyiNUvrN9WYEo27wDF8BfTaMy0AG7zhneMUOG1AHjYRtOu2dC mhfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Efrc1xoo; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id j2-20020a056a00174200b0068fe0125a33si3711236pfc.37.2023.10.04.05.37.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:37:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Efrc1xoo; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id A312A81ADCEB; Wed, 4 Oct 2023 05:37:16 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242381AbjJDMfs (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242362AbjJDMfd (ORCPT ); Wed, 4 Oct 2023 08:35:33 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A438193 for ; Wed, 4 Oct 2023 05:35:30 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W54m7rvqf/KqCMpTV79oOtmZr9LItvdWHEN45lcUeaw=; b=Efrc1xoorJOzy8tsALP1Bhr1M7Nxl+bJb/TLuAn1JJNB+EXwQE2BEjWHaW9r482myE0wWL uyLPSjVJw/V3m5a2xAAfe0w7iX/ytMsCbLv2tlTECGXATITJthK/1y6An9HyHv+7ky8UNx fnHJSOY2T7+agSXCrPYFIOj9ItFLcUo0G7iR/mN98qfQSUB2Ws3EteNzW/outR2GYMFdcx ZFt0ZjSVe6aKa5SGx5o1vMRv+CpHEF2kFZa1OL6kuHQgkcL0dIRC0Sc9kL61V9WnqSZppx f6yrhTvO2oWkhTA1l9pdm8ybiwUDZcfyp9uv9BD0L4FC9nMTEMZCKyY5jcB3Pw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W54m7rvqf/KqCMpTV79oOtmZr9LItvdWHEN45lcUeaw=; b=HnIgZMU7Y2g9X5WBxvEHh86H4zJMqVCS4RU8TGz0WgX52KTUJ1COlkvacxUBh/tihoDYYG 6WDL1TBNzELczOCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 06/25] timers: Split out forward timer base functionality Date: Wed, 4 Oct 2023 14:34:35 +0200 Message-Id: <20231004123454.15691-7-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:37:16 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828496788253464 X-GMAIL-MSGID: 1778828496788253464 Forwarding timer base is done when the next expiry value is calculated and when a new timer is enqueued. When the next expiry value is calculated the jiffies value is already available and does not need to be reread a second time. Splitting out the forward timer base functionality to make it executable via both contextes - those where jiffies are already known and those, where jiffies need to be read. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 31aed8353db1..2f6afd1da891 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -939,30 +939,34 @@ get_target_base(struct timer_base *base, unsigned tflags) return get_timer_this_cpu_base(tflags); } -static inline void forward_timer_base(struct timer_base *base) +static inline void __forward_timer_base(struct timer_base *base, + unsigned long basej) { - unsigned long jnow = READ_ONCE(jiffies); - /* * Check whether we can forward the base. We can only do that when * @basej is past base->clk otherwise we might rewind base->clk. */ - if (time_before_eq(jnow, base->clk)) + if (time_before_eq(basej, base->clk)) return; /* * If the next expiry value is > jiffies, then we fast forward to * jiffies otherwise we forward to the next expiry value. */ - if (time_after(base->next_expiry, jnow)) { - base->clk = jnow; + if (time_after(base->next_expiry, basej)) { + base->clk = basej; } else { if (WARN_ON_ONCE(time_before(base->next_expiry, base->clk))) return; base->clk = base->next_expiry; } + } +static inline void forward_timer_base(struct timer_base *base) +{ + __forward_timer_base(base, READ_ONCE(jiffies)); +} /* * We are using hashed locking: Holding per_cpu(timer_bases[x]).lock means From patchwork Wed Oct 4 12:34:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148259 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98464vqb; Wed, 4 Oct 2023 05:36:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGaJDcMbs9YnzFDP41yM6eEm1HwvAB9EtGpeY9ymfaPU0T1+VYxVw+SXqnH5Au/N4QuvRWb X-Received: by 2002:a17:90a:e144:b0:26b:3751:652a with SMTP id ez4-20020a17090ae14400b0026b3751652amr2085766pjb.38.1696422991979; Wed, 04 Oct 2023 05:36:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422991; cv=none; d=google.com; s=arc-20160816; b=DVQnabYlYVNtJSw/L5f4I4JzsXRJyabxCmqAfyPTrwIRbrQZe4XSQTwjYHBGtL3eVX enht7GU6YGOp3/DgY/XRu4tSLIvWNrj6gRdUpesf/EUE9hsPEakUXi9LZocfmQPd6Fej kvdQP/LD2kB+Y9Gd6umBJSgFFvfdvb42AO+G6yhlZiRSX2OabtJ15349grXkbWZYRPYn bWatLVK2oMJNao/GXqycdmK8DllpEzAYoa6SZGwVbuRGTE2j5UMBL4jOhs87gME9TABp 5SIfMZj7aDeYYuCt8zsZX/OUavqcWz75ri4IV84+Y778KIOJzH0BRlOlessheLviMtBs spbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=TACaL1lFRKPWpkhfzVfD3M8bG5zQsJmcuSPp/gOAMUM=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=aU6hnUAea/P/bDMx5IOdLGqNdzudYbh4J+nf6iRIQOWhqTk8yelEZnceb1d16S8BSy b5tQXOCwj+CvoC5jvvVS2s4N/y55UVSPXq7DeWQBiKjfWamSNIQJ1Ch2ReuEsYiWCfly sj6u7m/XUhBO0Kjedt8MNilXHyxfYbc4Qkh7tms6+khM0YpU/NNs2dJi0q4flQeDEqm1 FauM1GGtQpGkhavvoXYNJtPdR3/n5dKXCCESHrjtNHc17waNgiwFhhoeD37BaraV9A7w RYNuVvlyY93h7hY6ZovILSFLbRIhJmt5z3p2eGfVykEVJ0AFih3Il0XPQrMKGY/TY5VH eKhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=LW5O10L3; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="f/cvwydX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id ms21-20020a17090b235500b0025bb4a1c12esi1377361pjb.148.2023.10.04.05.36.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=LW5O10L3; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="f/cvwydX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 346EC802D42B; Wed, 4 Oct 2023 05:35:59 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242409AbjJDMfu (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242369AbjJDMfe (ORCPT ); Wed, 4 Oct 2023 08:35:34 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49125BF for ; Wed, 4 Oct 2023 05:35:31 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TACaL1lFRKPWpkhfzVfD3M8bG5zQsJmcuSPp/gOAMUM=; b=LW5O10L3K/13uU8Y/WizuaxwUjNbbEFTne4OjEEzLrAOTlX0f5E1sU9zYiEDesd9RStzju rcdIC9V+wKtNSFXHSiKXzApMmy4s0mz3BSHlJCEF7yyClrGObOgR5wRIDVVRu29cA9OGER 4bZaDCZgHutQ6vKti5lxrmQJPtJVHx/sA1V5jaPVPU+ruJsQrsjZrkobBWNVIRHbkaP+ba vo3F2FnwjvkuFSxEkxrSg7lhbUkKBGNh0rjeyRE3E9+EvEnXoqMmLIHQA7CoM4scSouyM7 57E4l5kJU7wTMQdxf6lZwGRmH2tQ57seDXgvk/DIueRaDeJBCZD3N0TfKoEhDg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TACaL1lFRKPWpkhfzVfD3M8bG5zQsJmcuSPp/gOAMUM=; b=f/cvwydXPVYbloV/MLHuClxhZ4pur5aHlYO74vCB/JxgoWTJfTrH14sdSnUTBJu/CdxMxy AqNgfFlMXVer0rAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 07/25] timers: Use already existing function for forwarding timer base Date: Wed, 4 Oct 2023 14:34:36 +0200 Message-Id: <20231004123454.15691-8-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:35:59 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828435359186021 X-GMAIL-MSGID: 1778828435359186021 There is an already existing function for forwarding the timer base. Forwarding the timer base is implemented directly in get_next_timer_interrupt() as well. To avoid code duplication, replace implementation for forwarding timer base with the direct call to the already existing function. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2f6afd1da891..dc58c479d35a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1939,15 +1939,9 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* * We have a fresh next event. Check whether we can forward the - * base. We can only do that when @basej is past base->clk - * otherwise we might rewind base->clk. + * base. */ - if (time_after(basej, base->clk)) { - if (time_after(nextevt, basej)) - base->clk = basej; - else if (time_after(nextevt, base->clk)) - base->clk = nextevt; - } + __forward_timer_base(base, basej); if (time_before_eq(nextevt, basej)) { expires = basem; From patchwork Wed Oct 4 12:34:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148266 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98694vqb; Wed, 4 Oct 2023 05:36:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEI2kJ3MHJ684C6RimPhk+kmUvG/6jmakpdnfD5IIQuhzxh+q7OOHjimwcwxjmVCWbSN0q2 X-Received: by 2002:a05:6a20:431c:b0:136:ea0e:d23 with SMTP id h28-20020a056a20431c00b00136ea0e0d23mr2514176pzk.11.1696423005756; Wed, 04 Oct 2023 05:36:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423005; cv=none; d=google.com; s=arc-20160816; b=ZWVe/8uh7KnskpVpbJZH+12gZIk3oS4r/YZY1s/XMgKOAhW1c7P79E7OJy5VNpzQKq M/Q8SvIbJjZz1DPxLeX1CewEIQOYJx/Dynovhd7+GelylVVrjc1NmHZWcIHfQHpT7t/5 +DMyZhWrIj4HQ8T69MVKosVAmyA89l7K2kvB3MjJa3CCmDk/xCy/ehvD6t3urIRrMXL8 zngGl2oGbZf9qvWK8NCfWmdPvgFwb1IKQ3rHCZfEhifvuSdkSCA2YD+03QSm2NgsjQOR fLwcu6rU1YXyp940O8t4/KSRvnnv4c6K7oS7dw54lMnm8vnyIcacqU/3U9qJ3H8VgXmS LGXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=dxpi78IDwxZZgGLEvaVUb//vKcdGLGnLWfwEyUx5Nmw=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=nbcXty1q5z5wn/oIMAXmpzei5OofrDlMlHtrBNJ49wkT6W+0EwykCDgTo7Bf0DMKhd Tw26kw/oRyBTuD/kEbZaVJ+wW3DoxdkCIG/ZAt1iEy9Q5PhBTRoiPuOCs/lUFAXcEDQ4 P4TMTzeYxkin3iB1HGUMj0bIP/7ZcLqwpn+Meij04xp8ZoyHrKMt0Oa2YfTjmkws5R9n KaNdoEv4Hkb7bToHDvXwhFBoN6INKITXnVj0d4qzlWnQtvZObdT8IEQks4avwg4AbnIo ObB84p9x2vohFRGZDtyQMW6I4govLS4jUFy9/7iDp4O59PP7FG/HByALNYaYx2yE3jw/ i34g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BDLOxBrp; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id t12-20020a056a0021cc00b0068fbd15c036si3838860pfj.286.2023.10.04.05.36.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BDLOxBrp; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 3DCA580BD51F; Wed, 4 Oct 2023 05:36:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242412AbjJDMfy (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242371AbjJDMfe (ORCPT ); Wed, 4 Oct 2023 08:35:34 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3BC4C1 for ; Wed, 4 Oct 2023 05:35:31 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dxpi78IDwxZZgGLEvaVUb//vKcdGLGnLWfwEyUx5Nmw=; b=BDLOxBrpk3OptCr9bdEYLu1NT+9K7T/RiwoEvkZ/7gmRSnp35L1Ky0L5I76iA0nT6WpyF/ /UAflRMfwUegkPMshVRxDOX46Ng8CXg+0RjKSp/JD2ZfmUJWF9HQP4JG0d0gHiPQO3XDKU CRAtp0MTRbGrsi04whb4Q2ET2AKv4V/35kS1QR0LIvvXkwump5FR6wUEgQV7bpRHOjPkhe W3ZaAJsdUFWBJ4WjU4cNDcyE5jk21IpJU1lECmoFHmKS0Shy1EWc5eGhkr6LeAyqL1kAh2 7qHcP0n+Y6206fWDV9dDpDFhzqn/y8ODkl0V/hSIapte4S78tQBHOzW+/jS05g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dxpi78IDwxZZgGLEvaVUb//vKcdGLGnLWfwEyUx5Nmw=; b=ZjvV/Gf61Hjp7Rgy1OmNsLZBFCK3wKahqbT70dfSzdcmtdWYmwpBuE+3dGglFB3SyiNNCP Nks9H9Zliz8MaGCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 08/25] timer: Rework idle logic Date: Wed, 4 Oct 2023 14:34:37 +0200 Message-Id: <20231004123454.15691-9-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828449887880360 X-GMAIL-MSGID: 1778828449887880360 From: Thomas Gleixner To improve readability of the code, split base->idle calculation and expires calculation into separate parts. Thereby the following subtle change happens if the next event is just one jiffy ahead and the tick was already stopped: Originally base->is_idle remains true in this situation. Now base->is_idle turns to false. This may spare an IPI if a timer is enqueued remotely to an idle CPU that is going to tick on the next jiffy. Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v4: Change condition to force 0 delta and update commit message (Frederic) --- kernel/time/timer.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index dc58c479d35a..18f8aac9b19a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1943,21 +1943,20 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ __forward_timer_base(base, basej); - if (time_before_eq(nextevt, basej)) { - expires = basem; - base->is_idle = false; - } else { - if (base->timers_pending) - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - /* - * If we expect to sleep more than a tick, mark the base idle. - * Also the tick is stopped so any added timer must forward - * the base clk itself to keep granularity small. This idle - * logic is only maintained for the BASE_STD base, deferrable - * timers may still see large granularity skew (by design). - */ - if ((expires - basem) > TICK_NSEC) - base->is_idle = true; + /* + * Base is idle if the next event is more than a tick away. Also + * the tick is stopped so any added timer must forward the base clk + * itself to keep granularity small. This idle logic is only + * maintained for the BASE_STD base, deferrable timers may still + * see large granularity skew (by design). + */ + base->is_idle = time_after(nextevt, basej + 1); + + if (base->timers_pending) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } raw_spin_unlock(&base->lock); From patchwork Wed Oct 4 12:34:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148260 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98583vqb; Wed, 4 Oct 2023 05:36:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFDrTeahpouM0B8JKH5DHJxTUJZA/4bV+uyW2J1hn5TSt84h9SOuODZKcFYj2ALEWCapRMj X-Received: by 2002:a05:6358:88f:b0:150:8ba9:56 with SMTP id m15-20020a056358088f00b001508ba90056mr2082742rwj.2.1696422996851; Wed, 04 Oct 2023 05:36:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422996; cv=none; d=google.com; s=arc-20160816; b=B0ALNtXMf+fuV8mXWYK5FdgVPe93yeTAZVsmijBb9kCnn3jDi3E4GRruxjOn1D+1BW LjL5fCDOeh4cEbZQBkOA4zuMmtvBwqZBvREfjTOtOoctBZ+c+tb+rW7V6sRAv2lTgrDD uCR8v8Fku6Hgh+jlS6D+WwUApQhdUbF3dMrxWn75V+r7ovNbsHCutpCajHcmt2GY3k+b UQo5LAd13iTlxQoQdpstEGdRPBQrwukHWsheH6ZmS4cnq44CwY4/Y71KRWgflqLeAr7L ++OxreWHBWg9uLnatCIJ6VOdAO8wo/dCeFSqWspXKzwQqkhbi1DgWpftpfAa6ZT6GruI tOiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=JvV/GTzIlX0Cx+6nbNdm3uB0dglqhwMGFjF0Qs3Csts=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=ulBMwf3ZtucW+VNqgoVsNB6pLZ3WWQOUcyDws7FYml3kgKTjyZhloCasuom2f4V3Fm 57FnfzeWa3emuyy2W27R2lAHCHLO7SYcEnS/l7qecotbDsL/8k/gFis13qmr1WP1WEsD /u+NOSpq7LrdTykvjhZkDvYrQ8RTGjP76XWEVxtnkNFO+Oc9ozbMcvL356cXm+wDvFQX x7bCqQCCEh5y+T48IJI3q+DZMMvC8BhW2IEHImEVUcxnb+rrYbpLk+NMhYIGweGtToGM NyCMjm/rb9zz9RzU561KhG4korerppsWWws7nT88+YwRwMTE+TUf0pZUU/1RQYzw0Esq v1jA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=anbpNzaF; dkim=neutral (no key) header.i=@linutronix.de header.b=N6cePmux; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id r20-20020a6560d4000000b0056c2f508886si3488744pgv.423.2023.10.04.05.36.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=anbpNzaF; dkim=neutral (no key) header.i=@linutronix.de header.b=N6cePmux; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 35FB0801B421; Wed, 4 Oct 2023 05:36:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242403AbjJDMf6 (ORCPT + 18 others); Wed, 4 Oct 2023 08:35:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242376AbjJDMff (ORCPT ); Wed, 4 Oct 2023 08:35:35 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 695AFC6 for ; Wed, 4 Oct 2023 05:35:32 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JvV/GTzIlX0Cx+6nbNdm3uB0dglqhwMGFjF0Qs3Csts=; b=anbpNzaF4gSVJYCje1CvoII4DNAP+K00Xko6363w/r9C54Ss9u48ITN+CsVTms4TpahdsJ tqXY+v2mOpyUmHytzorE+MN5OXdzM1pMN55anRLPCyT8+XviZTOU2ABl6PossHk6ybWaRi 18beY7ITHPVkMfQ5dXhnFBQClLLxWmuu4vWZF29jMC1vajgVmiwOBZwVEFwgJX2iazzGJa Z5plhNpKRuTYIZv4vceHtVZQRxrYEb2Ee8FwqOLhcfeA4r6IqBuMtqK4n7upOmqPG+plgJ +q9c86bL7B+EFiGxEkzNi0Jkn1b4K1/aCdDFIm3rhIk6f6ByVGj0fi193VtoTw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JvV/GTzIlX0Cx+6nbNdm3uB0dglqhwMGFjF0Qs3Csts=; b=N6cePmuxnggtkUWqTVBUfH7Yjf0o6ukWOKrXHR0Pj/o5cOdJIlIgadV9XxYYQdHtONDg3i +QqgcHp36b0UjsAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 09/25] timer: Split out get next timer functionality Date: Wed, 4 Oct 2023 14:34:38 +0200 Message-Id: <20231004123454.15691-10-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828440407578471 X-GMAIL-MSGID: 1778828440407578471 Split out get next timer functionality to make it reusable in other places. Thereby the order of getting the next expiry, forwarding the base clock and mark timer bases as idle, is changed. This change of order shouldn't have any impact, as nothing inside the function relies on the idle value or the updated timer base clock. Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 18f8aac9b19a..f443aa807fbc 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,6 +1911,24 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } +static inline unsigned long __get_next_timer_interrupt(unsigned long basej, + struct timer_base *base) +{ + unsigned long nextevt; + + if (base->next_expiry_recalc) + next_expiry_recalc(base); + nextevt = base->next_expiry; + + if (base->timers_pending) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + } + + return nextevt; +} + /** * get_next_timer_interrupt - return the time (clock mono) of the next timer * @basej: base time jiffies @@ -1933,9 +1951,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return expires; raw_spin_lock(&base->lock); - if (base->next_expiry_recalc) - next_expiry_recalc(base); - nextevt = base->next_expiry; + nextevt = __get_next_timer_interrupt(basej, base); /* * We have a fresh next event. Check whether we can forward the @@ -1952,14 +1968,10 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ base->is_idle = time_after(nextevt, basej + 1); - if (base->timers_pending) { - /* If we missed a tick already, force 0 delta */ - if (time_before(nextevt, basej)) - nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - } raw_spin_unlock(&base->lock); + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + return cmp_next_hrtimer_event(basem, expires); } From patchwork Wed Oct 4 12:34:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148264 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98666vqb; Wed, 4 Oct 2023 05:36:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEL8wuyVxNFBxwB1Mn+w2mb+0E5Epst19en06vBjFYF4bTtTlQ2hUmm78LVWZFw833NNROV X-Received: by 2002:a05:6a00:1956:b0:68c:1004:1feb with SMTP id s22-20020a056a00195600b0068c10041febmr2495987pfk.32.1696423003103; Wed, 04 Oct 2023 05:36:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423003; cv=none; d=google.com; s=arc-20160816; b=A/mwva04TIlPR6jR2im9LBqTcj5SqE9chAjTKcrMgevIiOHqKqwywt3fjsxUTVCFJX o+t2tXrmSb46HIEJeko3vpB86A/27aV371nNR25vtHmNmU5gW4+4uO8OoxSzwd8MdI5p 5R8+M4VTffZeCkvUEf4HsGX/TT+8z76EG+GIcYaj6MBZVKWa3Eq/ezgTFLXY/GmR2qRD s4TT1SlbkGvGFl2eMrkDA53aF9b3GrYXsQfiO7ufSkzqahluSGjk0A/eNzfqpv5OF8YJ ueLFahJRNsQvU73SsF0gUQQxxuqKwxve03B5ogpBiwe8VIpj7ZXVwOGeEkBHavnKrL9i dR+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=f/q/IZGOBCByN7ctkTK1opSNX4mw41Rp8Yoe6wCW+ls=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=uSwCxUdmW2Qyy9WCjf3BfRT5PPpVDMpLjjntwwfpCqCk7ZOERhuyA8XaI/my85FQ1u gKZ63jHsFnEPdBcga1OhqYdxIhMRESPKgxN9vJkQz/FlJ5ebWjTQfqa+OjnSJ2iQET4c luOQl+nK0ksDn/ldxePstG9mUELucGxrfGT9GqGMbm3nE73IFqvywD6K6PDRqIEyW3PT yl3AkMJ6kvmXfYL03dl6s5CysgpC5GR7QPYFWMLS0u4SmlulfMH111MgRXvLIEJO0eyN xzfqM9PmLDPAQYd5dcjsFBhFNkSoEuLeMPQ4iR5dkfw5F4j0Pb6b4X7XhCLE7geYivtw tIXg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FITpH5ek; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id m22-20020a056a00081600b00689f1186cfdsi3855001pfk.29.2023.10.04.05.36.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FITpH5ek; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id B1CB28348D85; Wed, 4 Oct 2023 05:36:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242667AbjJDMgd (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242394AbjJDMfn (ORCPT ); Wed, 4 Oct 2023 08:35:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3C92D3 for ; Wed, 4 Oct 2023 05:35:32 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f/q/IZGOBCByN7ctkTK1opSNX4mw41Rp8Yoe6wCW+ls=; b=FITpH5ekOEGXTKWb7Cg4KxAfSZZE2dXVixEzf60S0AYTH3nxohwlUavh4JWd4m9+vAZc7t 1rjGezjbE4F4PN3F1ri+Z/0bjUNeEFDICmwDUi79CIwWEiuTWZ9QCMPgilP+4a2dGvWF07 xtbTKVF/vPyrsUrhcQnQzP1HAn7xigyeHFJfji/RRvV0prno/ZOHlcubVZsa0QzDR5OA2Y 1mHICa1eHk4zQaTXbNyjTCa8VHddZJwo4+bireF9uJYjUeJAiOe1HOY7Sj/TymdSpA13Ff 85dprfJgr1s0fodSnH5Y2v/NyFem+NGNigkRK85XgIFaCrP19to50jmvs7QnoA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f/q/IZGOBCByN7ctkTK1opSNX4mw41Rp8Yoe6wCW+ls=; b=XeBthWl1I+T1u+Bn4yFhNaK/Qe/v2Rp/zUka5bVHiTR+30a692II7ZDIy1CVk0ttgnqct6 g0W9KupvEWd9E7AQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 10/25] timers: Move marking timer bases idle into tick_nohz_stop_tick() Date: Wed, 4 Oct 2023 14:34:39 +0200 Message-Id: <20231004123454.15691-11-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:38 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828446454621587 X-GMAIL-MSGID: 1778828446454621587 The timer base is marked idle when get_next_timer_interrupt() is executed. But the decision whether the tick will be stopped and whether the system is able to go idle is done later. When the timer bases is marked idle and a new first timer is enqueued remote an IPI is raised. Even if it is not required because the tick is not stopped and the timer base is evaluated again at the next tick. To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and get_next_timer_interrupt() is streamlined by only looking for the next timer interrupt. All other work is postponed to timer_set_idle() which is called by tick_nohz_stop_tick(). While at it a whitespace damage is fixed as well. Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 38 ++++++++++++++++++++++++----------- kernel/time/timer.c | 40 +++++++++++++++++++++++++++++++++---- 3 files changed, 63 insertions(+), 16 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 649f2b48e8f0..b035606a6f5e 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -164,6 +164,7 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem); +u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b3cf535881a4..7e1fdbc6d5f0 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -846,11 +846,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { - /* - * Tell the timer code that the base is not idle, i.e. undo - * the effect of get_next_timer_interrupt(): - */ - timer_clear_idle(); /* * We've not stopped the tick yet, and there's a timer in the * next period, so no point in stopping it either, bail. @@ -886,12 +881,35 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) { struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); + unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - u64 expires = ts->timer_expires; + bool timer_idle = ts->tick_stopped; + u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; + /* + * Now the tick should be stopped definitely - so timer base needs to be + * marked idle as well to not miss a newly queued timer. + */ + expires = timer_set_idle(basejiff, basemono, &timer_idle); + if (!timer_idle) { + /* + * Do not clear tick_stopped here when it was already set - it will + * be retained on next idle iteration when tick expired earlier + * than expected. + */ + expires = basemono + TICK_NSEC; + + /* Undo the effect of timer_set_idle() */ + timer_clear_idle(); + } else if (expires < ts->timer_expires) { + ts->timer_expires = expires; + } else { + expires = ts->timer_expires; + } + /* * If this CPU is the one which updates jiffies, then give up * the assignment and let it be taken by the CPU which runs @@ -926,7 +944,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) * first call we save the current tick time, so we can restart * the scheduler tick in nohz_restart_sched_tick. */ - if (!ts->tick_stopped) { + if (!ts->tick_stopped && timer_idle) { calc_load_nohz_start(); quiet_vmstat(); @@ -989,7 +1007,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now) /* * Cancel the scheduled timer and restore the tick */ - ts->tick_stopped = 0; + ts->tick_stopped = 0; tick_nohz_restart(ts, now); } @@ -1145,10 +1163,6 @@ void tick_nohz_idle_stop_tick(void) void tick_nohz_idle_retain_tick(void) { tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); - /* - * Undo the effect of get_next_timer_interrupt() called from - * tick_nohz_next_event(). - */ timer_clear_idle(); } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index f443aa807fbc..8518f7aa7319 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1950,6 +1950,40 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (cpu_is_offline(smp_processor_id())) return expires; + raw_spin_lock(&base->lock); + nextevt = __get_next_timer_interrupt(basej, base); + raw_spin_unlock(&base->lock); + + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + + return cmp_next_hrtimer_event(basem, expires); +} + +/** + * timer_set_idle - Set the idle state of the timer bases (if possible) + * @basej: base time jiffies + * @basem: base time clock monotonic + * @idle: pointer to store the value of timer_base->in_idle + * + * Returns the next timer expiry. + * + * hrtimers are not taken into account once more, as they already have been + * taken into account when asking for the next timer expiry. + */ +u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt; + + /* + * Pretend that there is no timer pending if the cpu is offline. + * Possible pending timers will be migrated later to an active cpu. + */ + if (cpu_is_offline(smp_processor_id())) { + *idle = true; + return KTIME_MAX; + } + raw_spin_lock(&base->lock); nextevt = __get_next_timer_interrupt(basej, base); @@ -1966,13 +2000,11 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) * maintained for the BASE_STD base, deferrable timers may still * see large granularity skew (by design). */ - base->is_idle = time_after(nextevt, basej + 1); + base->is_idle = *idle = time_after(nextevt, basej + 1); raw_spin_unlock(&base->lock); - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - - return cmp_next_hrtimer_event(basem, expires); + return basem + (u64)(nextevt - basej) * TICK_NSEC; } /** From patchwork Wed Oct 4 12:34:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148267 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98706vqb; Wed, 4 Oct 2023 05:36:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE2oRfmz10NXB3Julqt4baKOqQlqIdijD/Mxda0V1DoznarPTBfrAw0Z0fbPTD20l1C2cIT X-Received: by 2002:a05:6a21:a101:b0:164:e94b:d3ad with SMTP id aq1-20020a056a21a10100b00164e94bd3admr1830100pzc.28.1696423006781; Wed, 04 Oct 2023 05:36:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423006; cv=none; d=google.com; s=arc-20160816; b=F+C0Jb/8nRtCIukkovfZR6q1xr3bSnAsa1exWqDG9BpEPV7okKNjLPk64yh67KgXlL 6kgQiprZI+TJJ+jZBNa/akcejkhTbsDofes7w+f0BnPEOSDSKohO/wtpx2OupzAAGqDF bGXHsifta9vuOfTQ+HgvcZRfFBuhC9nNUOw/1RE3ndY28wbP6Of0U/syZWQ2Qcm2xtHc DyzfiiH//+O29IdxbmkRcYAEEOQ49yMw0YaztPWesA9Sxd0SfSA8zB4dm2S7oscq5943 btgL/BubvVnWgqhov2ydE0JiP7xUhVzRdM4567KCoEnAxn77UfCPFD/3Q1teLjGRvg0n VLeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=R6p91zzmcTgFtBegVAgz86f9rPOZl2ESGLLtaCGNKmE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=bMtJnRabRXSOPwfZ9BTDHMizbBKQy2H4XekIntnFMskzfP3vV/6dyHStNg2prWFojI wYFzwTW6DrBTsFRHnQ4ko+xWCLww1Bw4YeVrOxlciCiBxREuyYngeRVkbtJCGzkq7o72 up2/tz8zxe5TEakNK2WJ0Olh+W1SCrSuWyn6anhCUiD+ZUvtXNbY/F8LgVmYp7Niq3NO 0vTgCiKnnp9FUFChZbhL9COq0M2og3GaQFwfd5zz7gyTA/0xV7gZEBFoOJtLE7+86lqf eczuAY0kNefu+jjVzgvLTouTwNdyL1sW4Q+0pL0amW0j0CJoJtoFaXHxJ9ascpuoq0tJ hYOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tD5J+oia; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=p4HI0Tmi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id bv130-20020a632e88000000b00577723d24b0si3475339pgb.46.2023.10.04.05.36.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tD5J+oia; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=p4HI0Tmi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 701A2834822A; Wed, 4 Oct 2023 05:36:45 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242628AbjJDMgb (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242393AbjJDMfn (ORCPT ); Wed, 4 Oct 2023 08:35:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91FB6D9 for ; Wed, 4 Oct 2023 05:35:33 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R6p91zzmcTgFtBegVAgz86f9rPOZl2ESGLLtaCGNKmE=; b=tD5J+oiaWrqrU1ELhMzwRcDVY01AqSqRWzluw0BE0LA9bqRLkpt/mTPfu/6X5qYIQm7/6r azGNPmxrTGX7M2KTsSaEQRGMORPVKnsrDvvFjl+6VtmnFZkZxfhv0EHg11C43a9GDqIK8v hObFc6IZs5xltBwsQNQqQdhb+2tQyhVnb2M8nZ9EOzfOPwZT9WKv38d4wGpGPnyYvdfBa4 haJBSp9CEa4z2BOe6Gn/QavcG2GTsHfKTbAklvgqN2PhubiqwOBTL4eEuIe42UwiNbHReY QIK0UzjalspQk7MYaH2vRt2OIs8Hy+joCll0qaReQlsAc9J9mxb6NkTJmI3QmA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R6p91zzmcTgFtBegVAgz86f9rPOZl2ESGLLtaCGNKmE=; b=p4HI0TmibWqsmsZsVt86GITKCM8Cm5JkQzQKKPtx1yqL/g4yCL49TSeheoAImYhl51+X6o DFLUUenUA6uxCwAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 11/25] timers: Introduce add_timer() variants which modify timer flags Date: Wed, 4 Oct 2023 14:34:40 +0200 Message-Id: <20231004123454.15691-12-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:45 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828450978212787 X-GMAIL-MSGID: 1778828450978212787 Timer might be used as pinned timer (using add_timer_on()) and later on as non pinned timers using add_timer(). When the NOHZ timer pull at expiry model is in place, TIMER_PINNED flag is required to be used whenever a timer needs to expire on a dedicated CPU. Flag must no be set, if expiration on a dedicated CPU is not required. add_timer_on()'s behavior will be changed during the preparation patches for the NOHZ timer pull at expiry model to unconditionally set TIMER_PINNED flag. To be able to reset/set the flag when queueing a timer, two variants of add_timer() are introduced. This is a preparatory patch and has no functional change. Signed-off-by: Anna-Maria Behnsen --- New in v6 --- include/linux/timer.h | 2 ++ kernel/time/timer.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/timer.h b/include/linux/timer.h index 9162f275819a..6f96661480dd 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -180,6 +180,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires); #define NEXT_TIMER_MAX_DELTA ((1UL << 30) - 1) extern void add_timer(struct timer_list *timer); +extern void add_timer_local(struct timer_list *timer); +extern void add_timer_global(struct timer_list *timer); extern int try_to_del_timer_sync(struct timer_list *timer); extern int timer_delete_sync(struct timer_list *timer); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 8518f7aa7319..3172b8094097 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1245,6 +1245,40 @@ void add_timer(struct timer_list *timer) } EXPORT_SYMBOL(add_timer); +/** + * add_timer_local - Start a timer on the local CPU + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is set. + * + * See add_timer() for further details. + */ +void add_timer_local(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags |= TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_local); + +/** + * add_timer_global - Start a timer without TIMER_PINNED flag set + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is unset. + * + * See add_timer() for further details. + */ +void add_timer_global(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags &= ~TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_global); + /** * add_timer_on - Start a timer on a particular CPU * @timer: The timer to be started From patchwork Wed Oct 4 12:34:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148263 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98657vqb; Wed, 4 Oct 2023 05:36:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE3PEbU3kZ/7d0u6wDiSqh0eb8pIREb43xNyntsZykJ3gdWdglpzyKxY2ux79ANs1iVm7tk X-Received: by 2002:a17:902:da90:b0:1c6:de7:addd with SMTP id j16-20020a170902da9000b001c60de7adddmr2817685plx.48.1696423002314; Wed, 04 Oct 2023 05:36:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423002; cv=none; d=google.com; s=arc-20160816; b=U6P5YrV04jEdM4dufX+5fdfZE16E6chQdC03t3rfwWDD7yr5Ol/e9lv9TrUMKWeeoc C2D/66MtWFq9eWvRFZXaAIOEHJNYp9Pq0gkQZKTjlfPxs5a+Ldx1LYuNcwopZ/Iif9UK FskUg1f+8mIY5yM4k9zRkn0O7l51cMDfvnu2GV7AhD/+eWNTLUvI7g+FH4Ero4Idc/6L A9oGIu0pi1PmmN4QVuPVYk9wLgHD3Hwr+mBdiECi4ofkWiJo/8GpClnTdbHgoLXB4rQe /09NEj7vl4EpXakwbSvy5KQ4jUMalyN1MlI8ODUqvuTuEL+B6jU6ETSIqMQ+FyEhLqSL rvFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=6DPHaSkgekB2I+0lkcOsz7/7iYLTa5GbGumx2pBBGqU=; fh=6m6KAu2D1n2eK6jNJ9BIoRiulvR238oCr7uvY5/A720=; b=cyV6VxhpB9Pa2M7JXkNHs60040d6ZOpwdDR36HhsetQ7bRnl2vOs0DpForXpy8xBgX DoYmNYAs2gTXqJ0WmOPU1L2svWDXZj6n6JWTHM2t7LRIIYq8tmSEl+wcxoSND7na7RIq tLum4CKTuSpCkmwAC0+C5dYpRW0QGA6mwIl67PbNNE32YtRg/fxCpdeYyywVWtmK7orB SAZ5N/NjfMQJHBq8/bAzu07jT1nFNyYAH/CKeFr/DtWYBrCwYz0Amkpd+JH3PQ2fJWfj 8JMf9yvMSXoBJRTywNgDpTytC6iEBJcmG8ceBxL437Fb2xJfttp891e9+F/Uln+dmGPt gSYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Qd8FZSDA; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GF+aQ626; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id j10-20020a170902da8a00b001bbc0ca8650si4068604plx.370.2023.10.04.05.36.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Qd8FZSDA; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GF+aQ626; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id EB8A982159CF; Wed, 4 Oct 2023 05:36:09 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242493AbjJDMgE (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242358AbjJDMfn (ORCPT ); Wed, 4 Oct 2023 08:35:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37565DC for ; Wed, 4 Oct 2023 05:35:34 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6DPHaSkgekB2I+0lkcOsz7/7iYLTa5GbGumx2pBBGqU=; b=Qd8FZSDAYvAL7lxkOI5jQrs5CgGcU4uAc7nnagLndWZobaOuf/Dvk0zyWZDa/uAw/ItCrc 3VwoOTu/u1QZWjLt46Th72F4+6gSSj4+p732/V7jTJcFvocbTp/DxqXirqwwjaj8H4t99Z 7oU5lVYUGnqhJQklI8UJpl+1tBs1yXXhwqGQYnfEIL4DakSLcyrstgXhvslUuIurs3hjRc WWeKwnm79rFQvRIh6sG6QbxeNM3quvtE+G0VEe3vpW879Hrhctvp4y/o86iFrBs187XYjJ JjxHRZhOZ8xWs2uEoyj1tsMYZEDFAjUKbJl3NiVrMF3yQCAiPLWpxcnXpV2Uxw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6DPHaSkgekB2I+0lkcOsz7/7iYLTa5GbGumx2pBBGqU=; b=GF+aQ6268WK32U46sW7y8gJaofxzjHKXNrx1llzCNt/AaQqBRSn1wulaQgmV1OESGvG5gD 01BmrJM8D6fQPzBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Tejun Heo , Lai Jiangshan Subject: [PATCH v8 12/25] workqueue: Use global variant for add_timer() Date: Wed, 4 Oct 2023 14:34:41 +0200 Message-Id: <20231004123454.15691-13-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:09 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828445946576786 X-GMAIL-MSGID: 1778828445946576786 The implementation of the NOHZ pull at expiry model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be dropped. This is required for call sites which use the timer alternately as pinned and not pinned timer like workqueues do. Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker Acked-by: Tejun Heo Cc: Tejun Heo Cc: Lai Jiangshan --- v6: - New patch: As v6 provides unconditially setting TIMER_PINNED flag in add_timer_on() workqueue requires new add_timer_global() variant. --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index c85825e17df8..36474a2c4b6f 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1958,7 +1958,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, if (unlikely(cpu != WORK_CPU_UNBOUND)) add_timer_on(timer, cpu); else - add_timer(timer); + add_timer_global(timer); } /** From patchwork Wed Oct 4 12:34:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148261 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98622vqb; Wed, 4 Oct 2023 05:36:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGAgxGg7WnLtcVx7Uvhb5zVHZL9AYHhnnWNi0HPTcDZ/mmHjNBmRJldHmk4Ir9rE2JJ5AbY X-Received: by 2002:a17:902:e74e:b0:1bc:3908:14d with SMTP id p14-20020a170902e74e00b001bc3908014dmr9566772plf.2.1696422999772; Wed, 04 Oct 2023 05:36:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696422999; cv=none; d=google.com; s=arc-20160816; b=lDzh2azn2gWrt+6u1brbh285MfSIxTb1KzWheYgekoGXe/s4qxBJqVc2sUd28RVA3n NDFVrIgZd5x1ivpGWpnO51SMWQdIZK97h0UpXuEuu+Yb75Jhh9WcNXAHraufSk67HExa EDi1iVUptJN8sMj4hEHKw/YIQBUzFIYSVX2rSHVcKlI3ifiv4zpodD7DVt30JuDNePZH rUDYQVY/GSSMsVyM8fBe81s42kCnadGov9mHZ0i7pLpaiRN2Gr3K0sHfT9P+6TsmgYYy qmaThsuem3Uw+fo4+dFcPKU6M2wFj3cSko+p1lOhzGmGKDobBnvfsGsxVeeyhvi8xCgS Z9dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=9vhsM+uCD+bRTV7auZs4wdMq75JFKQoeaDSLzvQWIuw=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=pp8m4VM4670kX2ZqbHpl1Zr1nTe4B//hoQe7FUo41VHuIsVjABnSopaqqhGnZZUb5k +Y565cCp+uWe0bbwK+GrHo3ZUo0C+yPanJAH4EljFnR4jbQa7qzNuqR9qiLDtLoSRCia +S9aa/kfWA+je94WiZassod32zwMsAP9RYwGDtFrfgGB3B4pHHpNxW0gXYFvZ04eHQCl /h7tOK3mCQ/jn2Wac2h/RDPEf0JGIKeRK5s8w/5p9scNIPEdBmvEN7UNQl0pBdZIWlXh RsIe2PR4NV3HXaaLj3UhqmCKI1PzoxxPu1MLDZJ3Dy3XRiV7Tfowd4hRrOWOd+/Jk0jz J2bw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AZrpuIUB; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id d14-20020a170902cece00b001c75866c987si3713358plg.584.2023.10.04.05.36.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AZrpuIUB; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 4A6638213F20; Wed, 4 Oct 2023 05:36:07 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242353AbjJDMgC (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242398AbjJDMfn (ORCPT ); Wed, 4 Oct 2023 08:35:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC939E3 for ; Wed, 4 Oct 2023 05:35:34 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422933; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9vhsM+uCD+bRTV7auZs4wdMq75JFKQoeaDSLzvQWIuw=; b=AZrpuIUBupEwhI/vllltdbYxH4NadgULzlfSmoFHxwCreO6WBp4lvLieme4ivLCZ4s9eKz jMv/s3vINkZmsWe6iGANmXnwh8+jqwrcyvbFNtkXRgrmI+U4OluF54NPcQhX+IM0/C4guD WZpGDczq3Xn1JLyxcAzHWlFKL9Rg3NQAISXN0Kr3/Lsb+rFkMMdZVCq8uTHoqPMcswjf26 ZR0ntNmKWnt3MH3ym/xDiSCSuHSvSa0Ghm8I5kiEn2f/HlRK0cVNGhO7kfHSLHuqDHdvo9 jZKrHU+wETRoLbfZ+8e/YnDiYNsk/pfQzV24S3G/OHp/oTNeaEHS+QyDJPtezQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422933; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9vhsM+uCD+bRTV7auZs4wdMq75JFKQoeaDSLzvQWIuw=; b=aTFajETp4frynwLO5rtXBAmWDbop9zYbQLoj4X5p5R3VeHN0d6zgHdqNtmphS9NNxh08SO c1dWsIdJFU315qDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 13/25] timer: add_timer_on(): Make sure TIMER_PINNED flag is set Date: Wed, 4 Oct 2023 14:34:42 +0200 Message-Id: <20231004123454.15691-14-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:07 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828443628535798 X-GMAIL-MSGID: 1778828443628535798 When adding a timer to the timer wheel using add_timer_on(), it is an implicitly pinned timer. With the timer pull at expiry time model in place, TIMER_PINNED flag is required to make sure timers end up in proper base. Add TIMER_PINNED flag unconditionally when add_timer_on() is executed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3172b8094097..853a8ee48ffc 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1284,7 +1284,10 @@ EXPORT_SYMBOL(add_timer_global); * @timer: The timer to be started * @cpu: The CPU to start it on * - * Same as add_timer() except that it starts the timer on the given CPU. + * Same as add_timer() except that it starts the timer on the given CPU and + * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in + * the next round, add_timer_global() should be used instead as it unsets + * the TIMER_PINNED flag. * * See add_timer() for further details. */ @@ -1298,6 +1301,9 @@ void add_timer_on(struct timer_list *timer, int cpu) if (WARN_ON_ONCE(timer_pending(timer))) return; + /* Make sure timer flags have TIMER_PINNED flag set */ + timer->flags |= TIMER_PINNED; + new_base = get_timer_cpu_base(timer->flags, cpu); /* From patchwork Wed Oct 4 12:34:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148265 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98689vqb; Wed, 4 Oct 2023 05:36:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHB0u/pKXxMmb5YtlcAS/KhDnoHyIRTyp0qYJncIMCPBU77+PSr/InmMKnrtK0dMql9wJVt X-Received: by 2002:a05:6830:6b44:b0:6b7:4af2:492d with SMTP id dc4-20020a0568306b4400b006b74af2492dmr1802379otb.38.1696423005439; Wed, 04 Oct 2023 05:36:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423005; cv=none; d=google.com; s=arc-20160816; b=BK568HXwCGa+henIoMH/SOEku7apVLGb1w4PEMxImxZj/j8dgdbQl0W4yby5ZroQ+X 5tXqGF7JeWwz8T2rekbCdtB0Bq/zYXHjyEwWEc0QSNt/D9F5ugMDCMuRG5YRCUZc760X 65LC91wEZglAiWCHywKrsowvHFGZN+gAV81X8PAxRPfN6/rCeLUPT15XyPNkiMMiPc+W 2WI4ml/2Gvt5dDiHkF1HaNoF/829xJHa39GD1Kd4yD9pwq+lmwQ/aiz8Z84xXZp7qXg2 rP1LXLV7PuKWm/o/WIUFhhDl3eTpdEAak8CVZ2iTg033I46cHHrfqKrQhHH/FCRvWD2o SV2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=mz5QdBnNgn288WAkuPYs1KdwbgiRL6T25mdaPooSrW8=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=tlRNBauRUs010cDiqII+kaVlVexxyj/JO9I9+ov7PzQp5kraU9zq97fBR2sCTqulsy a+gdG4hoK81RiR+NYy1IUP+4f0utZg8Hjn3l+ln+LKvyOYjxMvWNIxBr2UEMoRaf5V3s C2/iLLC/gzRXhLdY3vFuTSWS6v4ESd6upX5rxzzqENVHqMspdLOkuli3Ot+bhTRdBLgY ha2ScctZw1BkTqy717NBJmBVI5WHzFfmgavIsRmoLNHboDOziBn/j78Bf25IvsvjoCHa dllA2lQmRhKgt542jxWjMVsSR1mPZaiI4DznUa7w7Wlnn4u45ahBpR/gST4DHS85FI/D DG/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="dcV/kah2"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id t23-20020a656097000000b00585a6bdce46si3645282pgu.308.2023.10.04.05.36.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="dcV/kah2"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 323FE82159F2; Wed, 4 Oct 2023 05:36:14 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242507AbjJDMgG (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242399AbjJDMfn (ORCPT ); Wed, 4 Oct 2023 08:35:43 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 670A8E8 for ; Wed, 4 Oct 2023 05:35:35 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mz5QdBnNgn288WAkuPYs1KdwbgiRL6T25mdaPooSrW8=; b=dcV/kah2sZANpyHtK1hKbSQiHNfz+pTcJF2EgsNt7jDpGibYjJsseIJ7OFgnJMkHFiv3p2 /pbWKF0G4x6E3qkx2rEANukPZvRYE1LHnMXvnEc0VSk7n1BnDGThcVuwfs7+fWy4PGfXiF m6Ktz2/fpAJR4AEpNbwPzA176y6uo/OHjupwRhUnC6boE1L07Gbka/dayr5kIc4PQt9GMu EUk86yjNl4sTOh8cfbdkIOKy6o99zD2Dx/pZ+BSZgQIMQY6U9PMg94SFos12YntSw1yhmY j3nz4JCISPYPboOLSG0QxW5N7DpZKPC0PrBxcGPqnCNwGh0taOVkx5dZTfFZpw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mz5QdBnNgn288WAkuPYs1KdwbgiRL6T25mdaPooSrW8=; b=6yDPT0oLpbTKkm3ZNkk1/ZTUhSmoN0AelRNlyHvFGnRA2sh9eda4y8KSlCP0/klza4pdeO o+ABVbI11HfuZcAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 14/25] timers: Ease code in run_local_timers() Date: Wed, 4 Oct 2023 14:34:43 +0200 Message-Id: <20231004123454.15691-15-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:14 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828449281985280 X-GMAIL-MSGID: 1778828449281985280 The logic for raising a softirq the way it is implemented right now, is readable for two timer bases. When increasing numbers of timer bases, code gets harder to read. With the introduction of the timer migration hierarchy, there will be three timer bases. Therefore ease the code. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v5: New patch to decrease patch size of follow up patches --- kernel/time/timer.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 853a8ee48ffc..1aafa0edc74c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2123,16 +2123,14 @@ static void run_local_timers(void) struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Wed Oct 4 12:34:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148262 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98637vqb; Wed, 4 Oct 2023 05:36:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFo/dlUUqbIngAVvrTcAObXxgHp6MW3ja+ipTkfFWk1VhiYl8HjBDVjQzSWF/bIdAwRONZE X-Received: by 2002:a05:6a00:39a2:b0:691:2d4:23b2 with SMTP id fi34-20020a056a0039a200b0069102d423b2mr2538125pfb.15.1696423000651; Wed, 04 Oct 2023 05:36:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423000; cv=none; d=google.com; s=arc-20160816; b=IxBj7bKhnhxFi4zNus+OWwqD6labkaWfj6SUeSRUhy6RIJH06D3nL3wM50nUDbssxF Xg0nfCPaYzzivNfx5FTVLVUCpFmjVrDYIWUa+8hBhdilS0KuCkNCsSkN6sz0KGlCzCsr OrSIhWP7UDRdw3S9jr5OnISfS/TX/4CRtZJ8knEqPANme582woknKkGAtMsaep5/JIp1 iOioM+SnShGhjWlhaqVwFI0MrYY+4hDwpZqvilGhWNbbDONETQkoF1BwQouw8x0pGqft Z84BMdpztK3NHUCDV1KwaLNRFIfmTfGM9bYU0vJr5FGHNmc+agb41+WJfEt8MVbnRoj2 V56g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=pz8RP5Igf6Tpz8GzWwXPtm9q6OrinBuBcc/7z4I5pxE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=SYB2XDJFne/B6SRLjOGbtU/v7SWpd+1QwH+NqffXxoac4FIIn7Ezzvyo+Td/XX4CGr LyQ5elC2w5smKrroCFvk6ZndNEf+HZeYcmoIayFXdn1Qv/cAuMOMfHMyughXN2NR8lAn pqwHfSDv7JGNydEkc5UwZ+m1IBO+nWGpCkCshaBTSK9HAtzs+OpF/meuhtniZWJcpSfX 9rC+mdb+ABYThctLgaTQCduuxEVp5IGWxhaHhtQdCqvtsSnMO0gByI4xSs0Dniwikh5N laVFwzcm3s8jIsNXe3HOzzlcvIS1r1jVgrb/nX8ti5nXcfLVMdaH4LbSXWlc0+O/UaPT xNbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BOmtu4iv; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=V2dSsPDW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id fj22-20020a056a003a1600b0069026a3de0asi3800189pfb.138.2023.10.04.05.36.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BOmtu4iv; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=V2dSsPDW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id D4E2C8348D84; Wed, 4 Oct 2023 05:36:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242445AbjJDMgK (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242404AbjJDMfo (ORCPT ); Wed, 4 Oct 2023 08:35:44 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC677F0 for ; Wed, 4 Oct 2023 05:35:35 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pz8RP5Igf6Tpz8GzWwXPtm9q6OrinBuBcc/7z4I5pxE=; b=BOmtu4ivdYU3oYnKkDb5HmQBpDOSvMbX/dkOE0+254ai79drRP/yGbyo9UEPydHO/zVnNu 53pEjywBAMuqf8OQtAnBwuVHO2JCD1u62IQikBtYiR+ALUoPYJxG0X2/LQt2gG0Mv07XwD A8bZHp5I9/zuLlGZt+P/XHY/qcZdq7UGxqp4mGouASlDhToFw2cTAPjDRIs6YMY4+Czodv 1Srrd72499fupEzwkagwz6M8N5hgGh+4XikHbtJwE6ptxPX+g707DMIkjGb/T+e6h+WiE4 j2izWFog0+uPaxcCy0L75B5Qx+c1KSwNwiId4uUHos46AIdl0thPu0qD5F+fxw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pz8RP5Igf6Tpz8GzWwXPtm9q6OrinBuBcc/7z4I5pxE=; b=V2dSsPDW7CR9I5JAq3+WZfAkcwN/At9CYrDwXDO/EusJ8FoXnZAClGdw4eUR1ozqH1XMvV +/AHhP4uQ+bf3zCQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 15/25] timer: Split next timer interrupt logic Date: Wed, 4 Oct 2023 14:34:44 +0200 Message-Id: <20231004123454.15691-16-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828444582710248 X-GMAIL-MSGID: 1778828444582710248 Logic for getting next timer interrupt (no matter of recalculated or already stored in base->next_expiry) is split into a separate function "next_timer_interrupt()" to make it available for new call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1aafa0edc74c..7468a805d302 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1951,14 +1951,20 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } +static unsigned long next_timer_interrupt(struct timer_base *base) +{ + if (base->next_expiry_recalc) + next_expiry_recalc(base); + + return base->next_expiry; +} + static inline unsigned long __get_next_timer_interrupt(unsigned long basej, struct timer_base *base) { unsigned long nextevt; - if (base->next_expiry_recalc) - next_expiry_recalc(base); - nextevt = base->next_expiry; + nextevt = next_timer_interrupt(base); if (base->timers_pending) { /* If we missed a tick already, force 0 delta */ From patchwork Wed Oct 4 12:34:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148273 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98827vqb; Wed, 4 Oct 2023 05:36:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHuVe/geSbhTqBABsUYvIcNJiM9rbrPTg61KIxq1R+HUCiNwD2yy8y4N0j/FhgH86MpryFz X-Received: by 2002:a92:ca0a:0:b0:351:e70:2cec with SMTP id j10-20020a92ca0a000000b003510e702cecmr2818628ils.20.1696423015724; Wed, 04 Oct 2023 05:36:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423015; cv=none; d=google.com; s=arc-20160816; b=IXGLtBiO05qG9JMvdmyrBT3UNOV3rtlun+YjFA4FyNN7M2mLiZFv9M+ekmqzIIHSGY w57oRmkm8AmaIXCFOKnkbV7OGZku8+T5lWXS7+dtDvrPxSmgwu0Da+pI/JiYtwkWZN9P gOX2TtvvGClrEcjj4bUDa4jycekg0nCpK219/hlW6ybhZgrZpZcMh2tjg0iBlKd8qAbT 66FMaPgd1Gnsu7FQODagvPBTMM5kxopZBtzL31osKSd8+gGIBjhhoYcTElQjkWw5U8Qi dvS3eU75dMd3T6zdEHP3pYnirrEUeMgZxy+7kwIyNJcj9khegc87RNRJpMhse8TcC/oT SyPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=CZyQ6bszTJODJTbCmkBo3GgjO020qPxfYuFgw3jxJIk=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=QM+Ph5OEX8+0VqGxk3r18bmVcwSkGh8qEXZXLE+nGpslAXpHYZjUNa4hlqeBg2Qkwe E2Yoa6XpX+Xo/RFstobOsOc2BSZfSlRZwEOWSu4cjreyg91P/mH1KJbBgFsk4CySWotP meoKlycfA8BCOqTe5jfn6ORJC/6B1ujbeW2hltjzWcVZDYGkPkuQR3ImOXS2jlPN/AUq 1P6dZLdiixSveLFoogY1C58ZNuhX5xMg54uxtQtHxRIGxZFr4cWikWO3MO+sDJoFwOTb knUK6kRf2qlhkGcniwpiOvrsjkodTGXPG1X8SHExh7i+jHjSK13R2MQz9nH6xBOJaPpY NNgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dsFfOUBF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z16-20020a656650000000b00578aa7b0e03si3583890pgv.695.2023.10.04.05.36.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dsFfOUBF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 57E2E82164E1; Wed, 4 Oct 2023 05:36:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242563AbjJDMg1 (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242378AbjJDMfo (ORCPT ); Wed, 4 Oct 2023 08:35:44 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01901FD for ; Wed, 4 Oct 2023 05:35:36 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422935; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CZyQ6bszTJODJTbCmkBo3GgjO020qPxfYuFgw3jxJIk=; b=dsFfOUBFeDkM79mwvDmunu1Wjh7gWf2JUpgBt4sDqfOlXm5Rd0oD/Mv7Qu6Vi2xTsNk9P3 +8sTfbgd09HlItVXrkkce7H7b31GlTIIxbGk+QehRiRGEBxy+xExMWGINXsxoxV6Jlwo/P 0FtHivGQkCV5s6sBuhwwR5leUOXtOBQXGrG/+Y2Fkjd4JV+vABd8lU6NQak9ul5fN8Iokd X77Q4cpcL4W8rjxLCWlnAayompNYIw0wQx+soqBKIs1WAMExmvTLfK7s2RxcbG5ISDnPYb QjKpx8aC8DnTYS4j8FkPgB7Z2JJu510H0+zv9Cj86rkSDwe2/m4ioU/hBf3Dsg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422935; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CZyQ6bszTJODJTbCmkBo3GgjO020qPxfYuFgw3jxJIk=; b=oCsy9OKVzdIIId/upGfmXYb9pCEAe/HIiDj/PlKSv+mmwyso+Pt31PvxG7tDD2QwllMBox t8rZcuZcDnsv8fCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v8 16/25] timer: Keep the pinned timers separate from the others Date: Wed, 4 Oct 2023 14:34:45 +0200 Message-Id: <20231004123454.15691-17-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:37 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828460064623074 X-GMAIL-MSGID: 1778828460064623074 Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: - Drop set TIMER_PINNED flag in add_timer_on() and drop related warning. add_timer_on() fix is splitted into a separate patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker v5: - Add WARN_ONCE() in add_timer_on() - Decrease patch size by splitting into three patches (this patch and the two before) v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 106 ++++++++++++++++++++++++++++++++------------ 1 file changed, 77 insertions(+), 29 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7468a805d302..1cd214c2f3f1 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1960,13 +1972,31 @@ static unsigned long next_timer_interrupt(struct timer_base *base) } static inline unsigned long __get_next_timer_interrupt(unsigned long basej, - struct timer_base *base) + struct timer_base *base_local, + struct timer_base *base_global) { - unsigned long nextevt; + unsigned long nextevt, nextevt_local, nextevt_global; + bool local_first; - nextevt = next_timer_interrupt(base); + nextevt_local = next_timer_interrupt(base_local); + nextevt_global = next_timer_interrupt(base_global); - if (base->timers_pending) { + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + nextevt = local_first ? nextevt_local : nextevt_global; + + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; @@ -1985,7 +2015,7 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base_local, *base_global; u64 expires = KTIME_MAX; unsigned long nextevt; @@ -1996,9 +2026,16 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (cpu_is_offline(smp_processor_id())) return expires; - raw_spin_lock(&base->lock); - nextevt = __get_next_timer_interrupt(basej, base); - raw_spin_unlock(&base->lock); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt = __get_next_timer_interrupt(basej, base_local, base_global); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); expires = basem + (u64)(nextevt - basej) * TICK_NSEC; @@ -2018,7 +2055,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base_local, *base_global; unsigned long nextevt; /* @@ -2030,25 +2067,35 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) return KTIME_MAX; } - raw_spin_lock(&base->lock); - nextevt = __get_next_timer_interrupt(basej, base); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt = __get_next_timer_interrupt(basej, base_local, base_global); /* * We have a fresh next event. Check whether we can forward the * base. */ - __forward_timer_base(base, basej); + __forward_timer_base(base_local, basej); + __forward_timer_base(base_global, basej); /* - * Base is idle if the next event is more than a tick away. Also + * Bases are idle if the next event is more than a tick away. Also * the tick is stopped so any added timer must forward the base clk * itself to keep granularity small. This idle logic is only - * maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * maintained for the BASE_LOCAL and BASE_GLOBAL base, deferrable + * timers may still see large granularity skew (by design). */ - base->is_idle = *idle = time_after(nextevt, basej + 1); + *idle = time_after(nextevt, basej + 1); + + /* We need to mark both bases in sync */ + base_local->is_idle = base_global->is_idle = *idle; - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return basem + (u64)(nextevt - basej) * TICK_NSEC; } @@ -2060,15 +2107,14 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); } #endif @@ -2114,11 +2160,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -2126,7 +2174,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); From patchwork Wed Oct 4 12:34:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148269 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98757vqb; Wed, 4 Oct 2023 05:36:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFUU20/l5FYutSCNGS/5urCjTgD+bpof1A+EmU/QG/combHDG0cc+ypt4vK2nkHtTEE7tsn X-Received: by 2002:a05:6808:1b0e:b0:3af:6cb9:ffc7 with SMTP id bx14-20020a0568081b0e00b003af6cb9ffc7mr2744706oib.16.1696423010178; Wed, 04 Oct 2023 05:36:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423010; cv=none; d=google.com; s=arc-20160816; b=sE2OAImqmYPJH7WTZ9qI/OJRVtnAlfEYutnZz98P+Jyrqs28Ohc/0zGxLrgUN9+UfG wk+Ca+BU05h7/o0uljLop160BHmhowxdmF+uhk29mh9ag3VdCVJuFINjFHpFDCcgWTDJ bvm8UYX1sdVRrkBCWd2t/aIsM+vUCR+2IHlZBZYGQT2GZwmEPwZBjq+7JCMQviI6HdRF bFug6vMX7hZVNnUNjs9FQOvDapjSMFFKtOlW4XHhi5S7Ap04EmOEr3P5NbWCeQr2R1RL MhmCNQ0cyJPhprkQoWNMc2g6rdZYihXUTWnih30wV25ilbCQhGICRBkgsB9N0fXh7VQf +k9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=V2oLs1aq9wqZoLE3N9pO7i2Jg8pkH2UAvQ/YD0cSS3w=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=IH0+KXuSNpj4mvY5iJ/Qx2+5EMY+/rjLfDKP6EVL9QqrPBFxFwPK1paLLOGWA20Me+ aqR0JdUeIfFi13NEJCUVtMwb1MDtgUaNvWI0J9w2lEUwvJGxMbQTqHu7JRPa92o4H0XS 6JVn8/Z4nyo8AgcXoHMadliqQkfvtxFJjc81Uj9lL9trQRauHcaTKEcbX59AokmDNFWi NV+yh5i9jDnocu222uSjPJ11ZFu+KXMir4NqBuJzweODAEVzvRbslDyBPvpr+5/xecEx yzgo7Te9/N3PpPWUl3OSirJvHoAGzV5njH37z6HH58rIbq0hfHzsC+Go4oJHyEJtDqxK ruIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ErrEdK5o; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id g25-20020a633759000000b00585a16c102dsi3672682pgn.39.2023.10.04.05.36.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ErrEdK5o; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id C876980C5FB8; Wed, 4 Oct 2023 05:36:31 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242467AbjJDMgL (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242388AbjJDMfo (ORCPT ); Wed, 4 Oct 2023 08:35:44 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 405F0FE for ; Wed, 4 Oct 2023 05:35:37 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422935; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2oLs1aq9wqZoLE3N9pO7i2Jg8pkH2UAvQ/YD0cSS3w=; b=ErrEdK5o4fLjNOKgR1xLbb+lqKbZXJX4q+elFfDJDA//0AWfEsqGosv6wzPLQwjZPQ7bKu zWclerP4eKvrN4mAIYaUqpyzZZHqL3ULIsjADaDkEZQTPZatbMvYgO6gVOk0DWtulh+hiU GZ9xGY/kz+d3oLfEO5rFjzToqXZxzcPYLsnkP0ogcZjQnM1MlWEjtFC/87/5gyYD+3H4HV 9cCtiBBH+mmNMHPfMPeKzVBiCB5JYYTJB1hJeHlKFIj/pFdZ9R0FwbTEaYWS0nVEh6+71u DoMpZ07XeR7/EWdlN1lUtXaQCqan+KMw8KWn/tGPKaixi0Gtonmqjo9qEYikPA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422935; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2oLs1aq9wqZoLE3N9pO7i2Jg8pkH2UAvQ/YD0cSS3w=; b=GAjl7C54dXXudkkC69QtWxiFiXq0RXPiPnqm0zbhDHbxYRfSZfRqBLIwnbsd53fRUtFV5f a2BbuSzQvpntn3BA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v8 17/25] timer: Retrieve next expiry of pinned/non-pinned timers separately Date: Wed, 4 Oct 2023 14:34:46 +0200 Message-Id: <20231004123454.15691-18-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:31 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828454013111903 X-GMAIL-MSGID: 1778828454013111903 For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have separate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 48 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 37 insertions(+), 11 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1cd214c2f3f1..4230fc6fa1ed 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,6 +221,11 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -1971,9 +1976,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } -static inline unsigned long __get_next_timer_interrupt(unsigned long basej, +static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 basem, struct timer_base *base_local, - struct timer_base *base_global) + struct timer_base *base_global, + struct timer_events *tevt) { unsigned long nextevt, nextevt_local, nextevt_global; bool local_first; @@ -1996,12 +2002,32 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, nextevt = local_first ? nextevt_local : nextevt_global; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the @nextevt is at max. one tick away, use @nextevt and store + * it in the local expiry value. The next global event is irrelevant in + * this case and can be left as KTIME_MAX. + */ + if (time_before_eq(nextevt, basej + 1)) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + return tevt->local; } + /* + * Update tevt->* values: + * + * If the local queue expires first, then the global event can + * be ignored. If the global queue is empty, nothing to do + * either. + */ + if (!local_first && base_global->timers_pending) + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + return nextevt; } @@ -2015,16 +2041,15 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; struct timer_base *base_local, *base_global; - u64 expires = KTIME_MAX; - unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ if (cpu_is_offline(smp_processor_id())) - return expires; + return KTIME_MAX; base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); @@ -2032,14 +2057,14 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - nextevt = __get_next_timer_interrupt(basej, base_local, base_global); + __get_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt.local = min_t(u64, tevt.local, tevt.global); - return cmp_next_hrtimer_event(basem, expires); + return cmp_next_hrtimer_event(basem, tevt.local); } /** @@ -2055,6 +2080,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; struct timer_base *base_local, *base_global; unsigned long nextevt; @@ -2073,7 +2099,7 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - nextevt = __get_next_timer_interrupt(basej, base_local, base_global); + nextevt = __get_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); /* * We have a fresh next event. Check whether we can forward the @@ -2097,7 +2123,7 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - return basem + (u64)(nextevt - basej) * TICK_NSEC; + return min_t(u64, tevt.local, tevt.global); } /** From patchwork Wed Oct 4 12:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148270 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98775vqb; Wed, 4 Oct 2023 05:36:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEX+4KXDgvvzQjqopHk7lYuEo9TSULAaSf7SeW3SEJBZVhNHlz8w2/ZhtNUK7XcnxUC/VVo X-Received: by 2002:a05:6a00:1993:b0:68e:2c3a:8775 with SMTP id d19-20020a056a00199300b0068e2c3a8775mr2433057pfl.33.1696423011469; Wed, 04 Oct 2023 05:36:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423011; cv=none; d=google.com; s=arc-20160816; b=FR92V1AVWSLZZbqedpqCwSznFgmsw4FEjjdHNQd6I54ZzkJOcqbrctUGOt4mXZOTWJ SpfucnVpOO5CULawQJMNme2kLfV/TeBuMJYzoQo9DDECn4B2G/TQVvlUJquHxJK8rjBo woXWJAywSKb99iju+pbxWSR6J5sx192gqBrT0LPxmd02VwG8sjW7YXVmmjcsQhsmCEdp aDEYJe19oYZL7p4xx/E48569HGL2/28ZL1wKK99XnE3//GBIBqg0AdEHeyKq/1dFKQnV Zn5AlyLlEAZYqwzvypGV2hvYUS2AGlPYlQyDG6sxzECyW3X+UwbOb5UytnZVQR9Bg15M OCfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=PYE4qjcpMvpvRPLtFGjkbKuz071Vlys4ffOshPkJzPM=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=A77cbnTuVD2SPhv9sXj3EU/H3XAiFnTpmlAfR7FrohPMDZCLNLAPVNePWObO9ZesGn B0By63YvJ/HE1rTlr0Ou3NnqbIKJ/h0eeOFJj/+PALt3h93JbBhMSdeNJPVZbQam87CN ASRqa5EXr1P7ccpnPY9vLbN1xZPF+zGG4Ieaqu2aFdswPWyC4BxV+P304gkAQyy0USuu Gs4lqJq2QJQTDqGI+tPPEPxk5CKnpcNhR8k9DJ3M83oiH4gxitK1WoBgKRGsohH8WSf5 I4iL5CpvYiEI1epzCOV+nB6LZy2ioRS01JeBBcm2S+lCSwwqEWEgHQVKW0hOiELIFMyp ayaQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=zB4VvhF4; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id t23-20020a62d157000000b00690de92ffe3si3383530pfl.309.2023.10.04.05.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=zB4VvhF4; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 5581882516FB; Wed, 4 Oct 2023 05:36:44 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242465AbjJDMgY (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242419AbjJDMfp (ORCPT ); Wed, 4 Oct 2023 08:35:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A73EE106 for ; Wed, 4 Oct 2023 05:35:37 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422936; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PYE4qjcpMvpvRPLtFGjkbKuz071Vlys4ffOshPkJzPM=; b=zB4VvhF4vv/KNyy86ETm1uDqt4dNpl+dbiQYCpi/lMo7g23qLXbl+5zA2C1beMPiSDCANe 2D7HoBlrCPy0eMmmdBOVpyOiL2ZeB7DTpwGfE21dRqWZ4mtVuYXFlDzfl97wIDSsxTq5pX nUuX7qltUVYUlnelarpkoRqdX4r28jCSXB31y9GZPIV4nDXa8UAhl6yqbUhcL8UqX7lt3r Wnhj7QLc1fPVzZy/gWASE1JaclowXo6p0LJVMKeYWsvIqMjAjqHt8RUz0f4fyHAdnqkllW 6CMPygGZH8/cgCN3AnmoDC6mChROdKVMkFoSmb2NZGy5lmipXlts3+mw0yHbIg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422936; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PYE4qjcpMvpvRPLtFGjkbKuz071Vlys4ffOshPkJzPM=; b=gc7NqSHIVqrsBUySm1W+IZ8NZ7pgDH+rnNSRCK+1NYOTkHPp0LqWHr7uGCFD1W1iynhq1+ zgk4tZDZXYsdnTDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 18/25] timer: Split out "get next timer interrupt" functionality Date: Wed, 4 Oct 2023 14:34:47 +0200 Message-Id: <20231004123454.15691-19-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:44 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828455806137398 X-GMAIL-MSGID: 1778828455806137398 The functionality for getting the next timer interrupt in get_next_timer_interrupt() is split into a separate function fetch_next_timer_interrupt() to be usable by other call sites. This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: s/splitted/split v5: Update commit message v4: Fix typo in comment --- kernel/time/timer.c | 51 +++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 20 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4230fc6fa1ed..c3061b28214e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1976,12 +1976,12 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } -static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 basem, - struct timer_base *base_local, - struct timer_base *base_global, - struct timer_events *tevt) +static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_base *base_local, + struct timer_base *base_global, + struct timer_events *tevt) { - unsigned long nextevt, nextevt_local, nextevt_global; + unsigned long nextevt_local, nextevt_global; bool local_first; nextevt_local = next_timer_interrupt(base_local); @@ -2000,21 +2000,6 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 else local_first = base_local->timers_pending; - nextevt = local_first ? nextevt_local : nextevt_global; - - /* - * If the @nextevt is at max. one tick away, use @nextevt and store - * it in the local expiry value. The next global event is irrelevant in - * this case and can be left as KTIME_MAX. - */ - if (time_before_eq(nextevt, basej + 1)) { - /* If we missed a tick already, force 0 delta */ - if (time_before(nextevt, basej)) - nextevt = basej; - tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; - return tevt->local; - } - /* * Update tevt->* values: * @@ -2028,6 +2013,32 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 if (base_local->timers_pending) tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + return local_first ? nextevt_local : nextevt_global; +} + +static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_base *base_local, + struct timer_base *base_global, + struct timer_events *tevt) +{ + unsigned long nextevt; + + nextevt = fetch_next_timer_interrupt(basej, basem, + base_local, base_global, tevt); + + /* + * If the nextevt is at max. one tick away, use the @nextevt and store + * it in the local expiry value. Reset of next global event to KTIME_MAX + * to make sure it is ignored. + */ + if (time_before_eq(nextevt, basej + 1)) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt->global = KTIME_MAX; + } + return nextevt; } From patchwork Wed Oct 4 12:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98839vqb; Wed, 4 Oct 2023 05:36:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IECqgDSFhXlSTRfc2ZR0Y/Cw2K1LawYNIZolujw27V2UMzUyytlLpcoCuqWu9pqn0MntcIb X-Received: by 2002:a17:902:8e87:b0:1b2:4852:9a5f with SMTP id bg7-20020a1709028e8700b001b248529a5fmr2028529plb.54.1696423016475; Wed, 04 Oct 2023 05:36:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423016; cv=none; d=google.com; s=arc-20160816; b=r+fVwFZ/mQ1f1efHb1e9NoBf6nQy7Y5VFEWj/DUFbBoaY7z5ex+YadM0qowWiuNpJT 6SjZekZKcOsOB6QMAivRX/85NlbWPKasd0V1ULww8HY9ZlTdRvUG+8TmnsHgQn75b5mM FKfexpWa1o7AZRLqTZZXcuEkLzkGhjoe2ItpmjQjPLVeo8Dk1ktEnD2VBzV0pgleYdh4 s2MgO204MnoM3YxTFn2BiygTozpGwvbIIJwYdCZ3v3W/M7nqno7n55+qOAM2b004nktN eviNElqE+WrF7qvUki+uJ9znNFwS8o2s0194w5uXdPvxZ2j7rmsFoukHKfnMpbe6EwTO L1tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=QACr81bz714MeCGL3gllUl2Nlnw9ik8CMB4MxkBQEQc=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=SXphRL4x12k6LofPSm47W1JXNdpELel1mmRyAQM1vyqO/rTHY1Hjh5IrvhRe7ud+ME P409QbdmjXa7GgR9GX0nrdvkbErZu+4x3bXe1S83jj+CoXEvbhZ4yBM2uE1Ro6/YO0SM pROeo9slzGWGxmKTP4BOoso9jaOKDdK3K0If1A78KLV7XV/h1PlNLqEbhFUyJevu/9qW X8JwGRVHPb7FqfZQXbG0FWziEb2r9wJboY9lL7mXa05KDmUtqlF5EpILIwll3j5i58hv dRDNtP3P7Ju9djLJDcy9yVt29V2DD2Yq1qEITaImsPWi9uLuNHsqsAwT1GNN/w8JNw6l sk2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=K8cDLMkh; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id q2-20020a170902e30200b001c724cd1125si3341538plc.203.2023.10.04.05.36.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=K8cDLMkh; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D169982164EC; Wed, 4 Oct 2023 05:36:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242491AbjJDMgR (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242421AbjJDMfp (ORCPT ); Wed, 4 Oct 2023 08:35:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55F1E109 for ; Wed, 4 Oct 2023 05:35:38 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422936; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QACr81bz714MeCGL3gllUl2Nlnw9ik8CMB4MxkBQEQc=; b=K8cDLMkhIl9jSNDDBcAcfHcoA/acgRfw443uKotsXCB7uc3qnguuRXlXa5Kqc3BmdkjVuO nnCkb02YomwFh5lpEv33jfrAX4zsDb/UlGY2OdPS7TVV2KcQDNIVArnsfnwQcJshzjdrgf z5KjhSlTMj29E69cROikykyYCAVPz915PKq/J4A6PqHEO6+5+gy6Z6fRq6vTmUn1DpFDf3 yzHYsGwhdu1eCoMorK0lLGPECTRnvluV8lk71vKtW6/Y0KjltbFQjiOHAnARTHwarKxHgC /vo6NqgtHhgx8aAjCO/T3Sey5zJxVvejS389q64glxIudCkMboAC7cYZ0/5Jkw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422936; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QACr81bz714MeCGL3gllUl2Nlnw9ik8CMB4MxkBQEQc=; b=RXiLI0RQZzrhT3E/isMxxk+c+AZeO2+fDmy/adcP9ISHbDsdDhqoMABYMgPukXHa7ndD2W DgstDnFJ8iuZL0CQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 19/25] timer: Add get next timer interrupt functionality for remote CPUs Date: Wed, 4 Oct 2023 14:34:48 +0200 Message-Id: <20231004123454.15691-20-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:37 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828461048284917 X-GMAIL-MSGID: 1778828461048284917 To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Locking of the timer bases and getting the information for the next timer interrupt functionality is split into separate functions. This is required to be compliant with lock ordering when the new model is in place. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v8: - Update comment v7: - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section - change lock, fetch functions to be unconditional - split out unlock function into a separate function v6: - introduce timer_lock_remote_bases() to fix race --- kernel/time/tick-internal.h | 10 +++++ kernel/time/timer.c | 76 ++++++++++++++++++++++++++++++++++--- 2 files changed, 81 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index b035606a6f5e..206010ae2a53 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -155,6 +160,11 @@ extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; +extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); +extern void timer_lock_remote_bases(unsigned int cpu); +extern void timer_unlock_remote_bases(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index c3061b28214e..ae4b6f62b082 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,11 +221,6 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); -struct timer_events { - u64 local; - u64 global; -}; - #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -2016,6 +2011,77 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, return local_first ? nextevt_local : nextevt_global; } +# ifdef CONFIG_SMP +/** + * fetch_next_timer_interrupt_remote + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + * + * Caller needs to make sure timer base locks are held (use + * timer_lock_remote_bases() for this purpose). + */ +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_held(&base_local->lock); + lockdep_assert_held(&base_global->lock); + + fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt); +} + +/** + * timer_unlock_remote_bases - unlock timer bases of cpu + * @cpu: Remote CPU + * + * Unlocks the remote timer bases. + */ +void timer_unlock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); +} + +/** + * timer_lock_remote_bases - lock timer bases of cpu + * @cpu: Remote CPU + * + * Locks the remote timer bases. + */ +void timer_lock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_irqs_disabled(); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); +} +# endif /* CONFIG_SMP */ + static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 basem, struct timer_base *base_local, struct timer_base *base_global, From patchwork Wed Oct 4 12:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148272 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98797vqb; Wed, 4 Oct 2023 05:36:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEjkHSrPTn2cGMpyuhB9XeubHOWe7YM4dhyE0VUp4g0edMeg/IDIqgUK+3jla/gRu+D88Ya X-Received: by 2002:a17:902:d491:b0:1c7:2e8f:c3b5 with SMTP id c17-20020a170902d49100b001c72e8fc3b5mr3347677plg.20.1696423013505; Wed, 04 Oct 2023 05:36:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423013; cv=none; d=google.com; s=arc-20160816; b=W0jz85us85rCujzqIlb0NVk5QA/cMXQyVLXADguv+RkfJBx6eXbaqASlp6kqmBmne5 NeOm9mQXzlhhQr4exugEfpZthsRSgNenwZFTJC8xj+0qHrJSl2mGIwnpAQ3QBNXJJDpt s2UY4DioL5U9QPq51HdHJ/XPH/TFXkqdJOPTfbWvyoPIkvDR61uPhMueALw64x1mBJc7 wTKO77BHSApwGPcTOZRpsNizxoahd4kdAaekRWOnukERjDY7rGkijC8WPMt+77wqgyUA i6CynfSOwxHk3nrXHthgqsNUwSHdls/BZM/qjJd9bEpzNyEShqQB41GgrjnvMzeJnnIj SWDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=UbU5bSuCcQVQPyYupy8izkqKsaMYBNty7tW6QmE8iKw=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=SlwsX3jTeBRRN9/pxVesP2UFGM1nAzigMZf+IYPBARNZJR2nIzs+cIvlqY3Vw7aSAv Yzq/rGqroarp4URLqae8eVRf9/Q5kFNHSCq1UDoBdk3z07TVDOZyrWQwSP0po4bAtJz0 8VZcoR0vtqTp91iOUl81uzvjCU2OMGvoUKa/Ta09wzoUnISkIkRHCUjzjDkuipn9QJsh /PiDCHixUhTNsTwaSAYo7u90WQm2iFznyMwxiyewn/TZbV+kJadqwhMFRH29rgPWhcx2 0/ryHV0CSJYm/GBOdmpPMOt/PEZ6lDPgWLq6fgWw8H58BJF8cAqb5QRmqtRfVZ4p9qP9 HYUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Vm18mK0h; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=RIpnPbrF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id r2-20020a170902c60200b001c72476ca30si3359865plr.281.2023.10.04.05.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Vm18mK0h; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=RIpnPbrF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 351528031C50; Wed, 4 Oct 2023 05:36:34 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242556AbjJDMg0 (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242423AbjJDMfp (ORCPT ); Wed, 4 Oct 2023 08:35:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7BC010E for ; Wed, 4 Oct 2023 05:35:38 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UbU5bSuCcQVQPyYupy8izkqKsaMYBNty7tW6QmE8iKw=; b=Vm18mK0h624IkxPC+RuwP1V5JUc3VQxNiMyNVnyWmF7zg92JWgzSsrKwhS+/JmbfHIDAdg F+nuWN+TKYm3T/e4dQpp7sKaBFJ6RXyHIaFHf32rP0IBdXKrt3sdEDjT1t5rOzC/Qmaz2t MQbjt5MxG12Y7wyWvbkktmndl0iI1Eg8y6TsQPrd3TrkMXt5B4E+D5DfkwA2NfB2rJJV1R WGIQM8cSMqDOwtvIWo10dmDlNUm4ZQBkSDAP9JRzkbr4OMFpAz8FY0INfmde/y/xemHCVJ C0WNgXJUn5g+qksiopfRhaN4tKbaP7ThMFoqJv/mozSR75kq0Jyyda42ErQQuQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UbU5bSuCcQVQPyYupy8izkqKsaMYBNty7tW6QmE8iKw=; b=RIpnPbrFOlsf3zuhbSOlZCumUoG2m+tm3EaT2AMfmQpLFX+dltm9Ely/xBzr20jO9yYwmM BLOavtfRUQCHILCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v8 20/25] timer: Restructure internal locking Date: Wed, 4 Oct 2023 14:34:49 +0200 Message-Id: <20231004123454.15691-21-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:35 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828458051728650 X-GMAIL-MSGID: 1778828458051728650 From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index ae4b6f62b082..8893f5dd1d66 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2230,11 +2230,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -2254,21 +2250,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Wed Oct 4 12:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp99397vqb; Wed, 4 Oct 2023 05:37:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH5Ljls4fu4LDXQoP127qPYR0qa5UoSwdHSjJhYpQx2FvQ+BMmtAqhz4KwQ0QZkeGVubC6t X-Received: by 2002:a05:6808:23c4:b0:3a7:5611:a53b with SMTP id bq4-20020a05680823c400b003a75611a53bmr2934921oib.0.1696423064660; Wed, 04 Oct 2023 05:37:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423064; cv=none; d=google.com; s=arc-20160816; b=fImN7aswpxtkaGNhHChDqBm3GDH1G0LFjnWK/TTRiMjBeirTUbiTIBlgRgX2Ev3ZKF pqIkTeDMa4AkVYgQvAPVlKYPYoUemwOqlqp/8yjeH+3CN+B48qnL+dRcjbfi7oiGH1LV Z43S7RoFVLi5hPuUAbLDqEFArg4Qb9gcnsNQKAvCGD6TaR2J1loiTrwUISXaz+ttv8g8 J3s8KO+u0GJHOW0kT+/3qkRgTWYHUetmo5n1unRL+wrlal/jL6ZKmqbVOjj+/+oCPUFp coObMYGIF/6EMQ0jjNpwNC/Rq9hDRKIAmQlWmon1IfgY78Du1RH726Yh76XCrgp1av8O 9jug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=KnlNVqR9WmkhQaFL3egg7gM5XjHK7g3gYDo7hW6w0sA=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=NSkOxvLq1MK1UAN3tulX64G7+VwEKM9OoUxJlzLezjaTmaGgXsgx02h9RvJguGpQ5v aPI7Iv7//Hh7BFoLEWvXufSLVNScvcp/0FwYoGhlQ07kiSY39EmZU5L+U0njjFJAcnCm /QEDPZW07oUfTsqAwvH+T93eGQb54Rx4A1yGLY8hiIbb/KNTN4iapRO+cGfYfWIOAPPH PGPdENiusZfNdMgEv2sF9lH+bj2h8cH7MWUWM9NMZ5fGkt3+I9FdyFdCUhkrUp+beAHB rTrxCdztFdD+klh7Wy9Zv6UcjAGRA1ROvj9O74rUXaAPQqi0QUcax8S2onKGKhU6PuZz 1kuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2TZ9kJex; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id u186-20020a6385c3000000b00578d3f8d4d4si3450066pgd.448.2023.10.04.05.37.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:37:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2TZ9kJex; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id BC49181A9B73; Wed, 4 Oct 2023 05:37:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242540AbjJDMgU (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242425AbjJDMfp (ORCPT ); Wed, 4 Oct 2023 08:35:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76ED111D for ; Wed, 4 Oct 2023 05:35:39 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KnlNVqR9WmkhQaFL3egg7gM5XjHK7g3gYDo7hW6w0sA=; b=2TZ9kJex3ROSXsaXZ7E6YlGyRhNQcoeofEqkGjEFrQtNA9JI7/+5BOxaf2P5Qtw7d4zpO+ G9MR1zjHfkvsM/a4vDvRz7ReaEAwvgoe3n9HMc9v4xD1vanpbtMRDsIj9wGujTkHP3gOUG gruyK2jWfbvnZEOMbkyXr1voAKd0tCNCb9FxxoTmzb9N1M5GXMzEPBwEQ+3Wd8unB0QEfn y6GK7wMm4vbwVbFIhiAutGYsQHZhE74BiYMb/GkfGbfrHSUfykP3IWSdlVVSIln5ixxECx 1YperYHSE3JgjrLQu+PGPrV3K/aXXfkCq7RCaTYE9KDf0CjzCDpIe7cxgzv5Tw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KnlNVqR9WmkhQaFL3egg7gM5XjHK7g3gYDo7hW6w0sA=; b=ls2cXdt0RpOukfXjfgEx4zuqZfQoPuAoG1FP29n6w5srR0nfJ/iEL78XXgngJh8lvaP0T8 GMX4WEeUul/0XzDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 21/25] timer: Check if timers base is handled already Date: Wed, 4 Oct 2023 14:34:50 +0200 Message-Id: <20231004123454.15691-22-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:37:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828511832157857 X-GMAIL-MSGID: 1778828511832157857 Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local cpu, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen --- v6: Drop double negation --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 8893f5dd1d66..7f06553e44de 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2232,6 +2232,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Wed Oct 4 12:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148268 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98725vqb; Wed, 4 Oct 2023 05:36:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHk8As0BxB0Z76iqfAGdio3anKa8VCJBlxXEQw5ytaJttcpGLDD5+7xOhQj7YsTr44zvs2N X-Received: by 2002:a17:902:d2c2:b0:1c7:3526:dfcd with SMTP id n2-20020a170902d2c200b001c73526dfcdmr2056205plc.52.1696423008059; Wed, 04 Oct 2023 05:36:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423008; cv=none; d=google.com; s=arc-20160816; b=EWF6X+7ftU2qbaYBQ0Zk2EJWnP0X2Ha08r1icr9MgUeN1ykRScPMtzSiYvbce4n5dG Xw9ShME8FpqIOr/NhF+n5qGh1KQ/QNgcCEJ9u0M4imd7sMkuWUe7RjkMK4qot7kqO0sM PYAU/q5pl3hpIlNttBRRIliomy3YC8/gxyF8wjCCk8Padjpm5J3K64RI3D5FMn3dmVql nSx0oOXKnq5SRHc8K1ijj5l9PmVXKSHBc55wHi9AgMRNbfTmTDd0GDQ9hHhDhWhgso1d D1N9Q7fwVHLIr8aR9z1kwyudRpU4LHHpBSYibDr1piO47GJ9tq/X0WeSmSi8Hml6jTk9 qWbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=ch0dJixaaY+pPoIMtZ/2D1P4Dl6dWsbaFTlXDeDmHfo=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=ZG2eFNHABdYTVcMOaUv5PMxAJhz4oLABzfFDsB7DkuJBqqQho1fLcliH+6IaP0m4bP yZUbCZ7aQvbF/nbQJ/I/P1ghMkz81yMcypcn9+gCwt2rODHZKv3qNpgnSqjQWQqh7DKD Pe98oEG5TQ+nYbhzkoRX1l7rfETqPpqgjtQikl/LJb/a5duVzsw3ICnnxB/egonYoIEh 2Z6DJBIouKI0Zif3F1bZGp2PGrhdUl51oF4oUmayIoyw3jW4ZtCedNQRBqBNX51C+am2 WvzIFeu6s22wEfTP6fiFwcsXNrPjAcaEEuIJl1zwel40jnqEv6NoDh4QD6xwZQNiRZH/ xEyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=EWEnazDZ; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id p14-20020a170902e74e00b001c62d7e4831si4027145plf.588.2023.10.04.05.36.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=EWEnazDZ; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id A676482159FF; Wed, 4 Oct 2023 05:36:17 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242492AbjJDMgO (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242432AbjJDMfp (ORCPT ); Wed, 4 Oct 2023 08:35:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F03B7121 for ; Wed, 4 Oct 2023 05:35:39 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ch0dJixaaY+pPoIMtZ/2D1P4Dl6dWsbaFTlXDeDmHfo=; b=EWEnazDZolfpBGPeMVQr4P6bwSvWc5rXvr3cujc035imvKbovtJuoA3eSiXKwGIwg502Is aKE/bAslzFSsygvIc/mbRqupzsiPyuW5lngr2Up5h9AszVVMSqO9Af8IFSMyTUfgFfXOqu clPbDFGEGxtZlXOKIJEhNpGbQqwSioYrpTr0aFrbgeDPK6ioy/SExPalLFHCLn3nqqOgif jC1JNHIQjBb+uF9hrjRm4DwAXGWxwT/kc1ybLhv+MN2icWryR7HBApEU8Uchs8t2a0/r7A tuZ/tjtrHP7aUq2Nv8ueCgIZWacA6OeCxNarvL4BJynBNhg9y4Mm4WnaakhOtA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ch0dJixaaY+pPoIMtZ/2D1P4Dl6dWsbaFTlXDeDmHfo=; b=jrRtLk7S+jsStbrVpfrcFWHXpIo5Z8IQZdT0KGNlw/PKwrqR97BmQk3LLty8vm43LyZJnp j/aQNVezBwbpqkBQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v8 22/25] tick/sched: Split out jiffies update helper function Date: Wed, 4 Oct 2023 14:34:51 +0200 Message-Id: <20231004123454.15691-23-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:17 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828451814198887 X-GMAIL-MSGID: 1778828451814198887 From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global function in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 206010ae2a53..7dbf5ab43d06 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -158,6 +158,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 7e1fdbc6d5f0..169886a87cc1 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -798,18 +798,30 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - u64 basemono, next_tick, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + u64 basemono, next_tick, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Wed Oct 4 12:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148271 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98770vqb; Wed, 4 Oct 2023 05:36:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHvCElZkRy73Jm/LPqH9W5M8l/KQSh7LOIF69VTJOGqGAEWanVeItPLAcza7q6y0hPoSdoj X-Received: by 2002:a05:6e02:170c:b0:350:f510:3990 with SMTP id u12-20020a056e02170c00b00350f5103990mr2972787ill.2.1696423011186; Wed, 04 Oct 2023 05:36:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423011; cv=none; d=google.com; s=arc-20160816; b=ZJIEeh7o3rKpAvlFROLVL01uh7iExPefItCJMy9fTRns0V4VWo3NWhuS8xIZEvHeap peTmTBXGVg11vSfZpvVbG9E1wk5Ljn4ZvKf+s3u7HqlRMtCK6Ne9PU+uUO0EHeMDGXuP 2fDl09UPBt2nBCs4loV3mTzWdroErfGqFecIanxe/AKBWJfs2ARhoZ+6Ju3gFD6Jt5es bWU1qjb9hLWeNm+tStAuugAnFnFHc78F85cf3/1/SEhWaB9HN1QxUG1xXONgNDCv/jI7 c/zmS0DVBRs477jZAyK8DFf/svnXwCui5XKH/7bk/1bx5xK2HR9iX9kuVU9XTvwK2o05 yUMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=uD4GtjbozKXmXyIEErTZhqTB6alursQL+ikTUHHcdZg=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=HhXvMDxD0AplwTPnNEan4AutfroJAQQZpVGmgGuTOf08MRtYNxVQ3Mfw6nRzHZKDN2 a02ergWMQ38vZHcrlTQTu6lLXroI3uTqjJOTQV+yAFfHAcIr7GjslVjzXsXm4ETKFD5x f7ORb7vuvSwgcVc+rtGd0iq7ilImzIBgQaJhLW3EcNbJRuLvIHiEvwltGjPbshMwy0Nk I3WW8vGCJmTEmLNlaDJUsVQ/0UDQMmnrSEGvS3/JlGH4R7z7fwhE89PwqO2l8RO9hSDt vY1dCzhtTqPy/Ei7us8GdM0XadfnSfjjxYHE5OPf149PY2RQ0wFmkKeBxt7xRBC6r6m0 sF8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PeYQkFn7; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id b128-20020a633486000000b005855424fc29si3791844pga.53.2023.10.04.05.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PeYQkFn7; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id C824C8348D9E; Wed, 4 Oct 2023 05:36:49 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242419AbjJDMgm (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242439AbjJDMft (ORCPT ); Wed, 4 Oct 2023 08:35:49 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92F1C136 for ; Wed, 4 Oct 2023 05:35:40 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uD4GtjbozKXmXyIEErTZhqTB6alursQL+ikTUHHcdZg=; b=PeYQkFn7Z403qIcbjrzSJozWVP8ierwq21b/U+M5X2EI0KeiYAhjeSPtjrcMpHdiupbK9B 8/fTAdZVsI0zp9lVEwIQhuq+sBPnZ67K6cemNjMxJx0tOwqbCpfHP845vlkIAvz0uPQf80 OmQJ3lnCTiCq0uDADE2dpjgNNGbEIEoPgoqGC0fTnrAT9HAcmPtmOmHyxh8hRIcqN5yzFY X3dHBqopDYmdin6CfzieDu6H/Syzu0Tjq3Ajc5LEDK5yk98VcjPB7kgAArNC5yx8bDVzzB bOGylyeOlaqkTAwdw0TqNbdNELeIVzGw9ZLFeB7XKJjZ7KpuFeaE4BM+mRZ3ag== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uD4GtjbozKXmXyIEErTZhqTB6alursQL+ikTUHHcdZg=; b=Dtp/ryapi2aGB2/A3lYCskKZYMshaco6bfIZZoRN3QnazC/WlChRYuCyHYgyeDi9wxBuCX ztGdvFLk3TldsZAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 23/25] timer: Implement the hierarchical pull model Date: Wed, 4 Oct 2023 14:34:52 +0200 Message-Id: <20231004123454.15691-24-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:49 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828455487807309 X-GMAIL-MSGID: 1778828455487807309 Placing timers at enqueue time on a target CPU based on dubious heuristics does not make any sense: 1) Most timer wheel timers are canceled or rearmed before they expire. 2) The heuristics to predict which CPU will be busy when the timer expires are wrong by definition. So placing the timers at enqueue wastes precious cycles. The proper solution to this problem is to always queue the timers on the local CPU and allow the non pinned timers to be pulled onto a busy CPU at expiry time. Therefore split the timer storage into local pinned and global timers: Local pinned timers are always expired on the CPU on which they have been queued. Global timers can be expired on any CPU. As long as a CPU is busy it expires both local and global timers. When a CPU goes idle it arms for the first expiring local timer. If the first expiring pinned (local) timer is before the first expiring movable timer, then no action is required because the CPU will wake up before the first movable timer expires. If the first expiring movable timer is before the first expiring pinned (local) timer, then this timer is queued into a idle timerqueue and eventually expired by some other active CPU. To avoid global locking the timerqueues are implemented as a hierarchy. The lowest level of the hierarchy holds the CPUs. The CPUs are associated to groups of 8, which are separated per node. If more than one CPU group exist, then a second level in the hierarchy collects the groups. Depending on the size of the system more than 2 levels are required. Each group has a "migrator" which checks the timerqueue during the tick for remote expirable timers. If the last CPU in a group goes idle it reports the first expiring event in the group up to the next group(s) in the hierarchy. If the last CPU goes idle it arms its timer for the first system wide expiring timer to ensure that no timer event is missed. Signed-off-by: Anna-Maria Behnsen --- v8: - Review of Frederic: - Fix hotplug race (introduction of wakeup_recalc) - Make wakeup and wakeup_recalc logic consistent all over the place - Fix child/group state race and read it with locks held - Add more clarifying comments - Fix grammar all over the place - change integers which act as boolean value into bool - rewrite condition in tmigr_check_migrator() without negation - Improve update events logic with a check of the first event - Implement a quick forecast which is called when get_next_timer_interrupt() is executed. v7: - Review remarks of Frederic and bigeasy: - change logic in tmigr_handle_remote_cpu() - s/kzalloc/kcalloc - move timer_expire_remote() into NO_HZ_COMMON && SMP config section - drop DBG_BUG_ON() makro and use only WARN_ON_ONCE() - remove leftovers from sibling logic during setup - Move timer_expire_remote() into tick-internal.h - Add documentation section about "Required event and timerqueue update after remote expiry" - Fix fallout of kernel test robot v6: - Fix typos - Review remarks of Peter Zijlstra (locking, struct member cleanup, use atomic_try_cmpxchg(), update struct member descriptions) - Fix race in tmigr_handle_remote_cpu() (Frederic Weisbecker) v5: - Review remarks of Frederic - Return nextevt when CPU is marked offline in timer migration hierarchy instead of KTIME_MAX - Fix update of group events issue, after remote expiring v4: - Fold typo fix in comment into proper patch "timer: Split out "get next timer interrupt" functionality" - Update wrong comment for tmigr_state union definition - Fix fallout of kernel test robot --- include/linux/cpuhotplug.h | 1 + kernel/time/Makefile | 3 + kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 81 +- kernel/time/timer_migration.c | 1612 +++++++++++++++++++++++++++++++++ kernel/time/timer_migration.h | 144 +++ 6 files changed, 1836 insertions(+), 6 deletions(-) create mode 100644 kernel/time/timer_migration.c create mode 100644 kernel/time/timer_migration.h diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 06dda85f0424..ed992268f206 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -244,6 +244,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, + CPUHP_AP_TMIGR_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, CPUHP_AP_RANDOM_ONLINE, diff --git a/kernel/time/Makefile b/kernel/time/Makefile index 7e875e63ff3b..4af2a264a160 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -17,6 +17,9 @@ endif obj-$(CONFIG_GENERIC_SCHED_CLOCK) += sched_clock.o obj-$(CONFIG_TICK_ONESHOT) += tick-oneshot.o tick-sched.o obj-$(CONFIG_LEGACY_TIMER_TICK) += tick-legacy.o +ifeq ($(CONFIG_SMP),y) + obj-$(CONFIG_NO_HZ_COMMON) += timer_migration.o +endif obj-$(CONFIG_HAVE_GENERIC_VDSO) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 7dbf5ab43d06..fb1568259c21 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -166,6 +166,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, unsigned int cpu); extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); +extern void timer_expire_remote(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7f06553e44de..331c59c0789a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -53,6 +53,7 @@ #include #include "tick-internal.h" +#include "timer_migration.h" #define CREATE_TRACE_POINTS #include @@ -2080,6 +2081,21 @@ void timer_lock_remote_bases(unsigned int cpu) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); } + +static void __run_timer_base(struct timer_base *base); + +/** + * timer_expire_remote - expire global timers of cpu + * @cpu: Remote CPU + * + * Expire timers of global base of remote CPU. + */ +void timer_expire_remote(unsigned int cpu) +{ + struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + __run_timer_base(base); +} # endif /* CONFIG_SMP */ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 basem, @@ -2113,13 +2129,18 @@ static inline unsigned long __get_next_timer_interrupt(unsigned long basej, u64 * @basej: base time jiffies * @basem: base time clock monotonic * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer + * or KTIME_MAX if no timer is pending. If timer of global base was queued + * into timer migration hierarchy, first global timer is not taken into + * account. If it was the last CPU of timer migration hierarchy going idle, + * first global event is taken into account. */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; struct timer_base *base_local, *base_global; + u64 tmigr_expires = KTIME_MAX; + unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -2134,12 +2155,19 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - __get_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); + nextevt = __get_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); + + if (time_after(nextevt, basej + 1)) { + tmigr_expires = tmigr_quick_check(); + + if (tmigr_expires < basem) + tmigr_expires = basem; + } raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - tevt.local = min_t(u64, tevt.local, tevt.global); + tevt.local = min_t(u64, tevt.local, tmigr_expires); return cmp_next_hrtimer_event(basem, tevt.local); } @@ -2194,13 +2222,47 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) */ *idle = time_after(nextevt, basej + 1); + if (*idle) { + u64 next_tmigr; + + /* + * Enqueue first global timer into timer migration + * hierarchy, afterwards tevt.global is no longer used. + */ + next_tmigr = tmigr_cpu_deactivate(tevt.global); + + /* + * If CPU is the last going idle in timer migration + * hierarchy, make sure CPU will wake up in time to handle + * remote timers. next_tmigr == KTIME_MAX if other CPUs are + * still active. + */ + if (next_tmigr < tevt.local) { + u64 tmp; + + /* If we missed a tick already, force 0 delta */ + if (next_tmigr < basem) + next_tmigr = basem; + + tmp = div_u64(next_tmigr - basem, TICK_NSEC); + + nextevt = basej + (unsigned long)tmp; + tevt.local = next_tmigr; + *idle = time_after(nextevt, basej + 1); + } + /* + * Update of nextevt is not required in an else path, as it + * is revisited in !is_idle path only. + */ + } + /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = *idle; raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - return min_t(u64, tevt.local, tevt.global); + return tevt.local; } /** @@ -2218,6 +2280,9 @@ void timer_clear_idle(void) */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); + + /* Activate without holding the timer_base->lock */ + tmigr_cpu_activate(); } #endif @@ -2283,6 +2348,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { run_timer_base(BASE_GLOBAL); run_timer_base(BASE_DEF); + + if (is_timers_nohz_active()) + tmigr_handle_remote(); } } @@ -2297,7 +2365,8 @@ static void run_local_timers(void) for (int i = 0; i < NR_BASES; i++, base++) { /* Raise the softirq only if required. */ - if (time_after_eq(jiffies, base->next_expiry)) { + if (time_after_eq(jiffies, base->next_expiry) || + (i == BASE_DEF && tmigr_requires_handle_remote())) { raise_softirq(TIMER_SOFTIRQ); return; } diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c new file mode 100644 index 000000000000..929b3b94b893 --- /dev/null +++ b/kernel/time/timer_migration.c @@ -0,0 +1,1612 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Infrastructure for migratable timers + * + * Copyright(C) 2022 linutronix GmbH + */ +#include +#include +#include +#include +#include +#include + +#include "timer_migration.h" +#include "tick-internal.h" + +/* + * The timer migration mechanism is built on a hierarchy of groups. The + * lowest level group contains CPUs, the next level groups of CPU groups + * and so forth. The CPU groups are kept per node so for the normal case + * lock contention won't happen across nodes. Depending on the number of + * CPUs per node even the next level might be kept as groups of CPU groups + * per node and only the levels above cross the node topology. + * + * Example topology for a two node system with 24 CPUs each. + * + * LVL 2 [GRP2:0] + * GRP1:0 = GRP1:M + * + * LVL 1 [GRP1:0] [GRP1:1] + * GRP0:0 - GRP0:2 GRP0:3 - GRP0:5 + * + * LVL 0 [GRP0:0] [GRP0:1] [GRP0:2] [GRP0:3] [GRP0:4] [GRP0:5] + * CPUS 0-7 8-15 16-23 24-31 32-39 40-47 + * + * The groups hold a timer queue of events sorted by expiry time. These + * queues are updated when CPUs go in idle. When they come out of idle + * ignore flag of events is set. + * + * Each group has a designated migrator CPU/group as long as a CPU/group is + * active in the group. This designated role is necessary to avoid that all + * active CPUs in a group try to migrate expired timers from other CPUs, + * which would result in massive lock bouncing. + * + * When a CPU is awake, it checks in it's own timer tick the group + * hierarchy up to the point where it is assigned the migrator role or if + * no CPU is active, it also checks the groups where no migrator is set + * (TMIGR_NONE). + * + * If it finds expired timers in one of the group queues it pulls them over + * from the idle CPU and runs the timer function. After that it updates the + * group and the parent groups if required. + * + * CPUs which go idle arm their CPU local timer hardware for the next local + * (pinned) timer event. If the next migratable timer expires after the + * next local timer or the CPU has no migratable timer pending then the + * CPU does not queue an event in the LVL0 group. If the next migratable + * timer expires before the next local timer then the CPU queues that timer + * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0 + * group. + * + * When CPU comes out of idle and when a group has at least a single active + * child, the ignore flag of the tmigr_event is set. This indicates, that + * the event is ignored even if it is still enqueued in the parent groups + * timer queue. It will be removed when touching the timer queue the next + * time. This spares locking in active path as the lock protects (after + * setup) only event information. For more information about locking, + * please read the section "Locking rules". + * + * If the CPU is the migrator of the group then it delegates that role to + * the next active CPU in the group or sets migrator to TMIGR_NONE when + * there is no active CPU in the group. This delegation needs to be + * propagated up the hierarchy so hand over from other leaves can happen at + * all hierarchy levels w/o doing a search. + * + * When the last CPU in the system goes idle, then it drops all migrator + * duties up to the top level of the hierarchy (LVL2 in the example). It + * then has to make sure, that it arms it's own local hardware timer for + * the earliest event in the system. + * + * + * Lifetime rules: + * --------------- + * + * The groups are built up at init time or when CPUs come online. They are + * not destroyed when a group becomes empty due to offlining. The group + * just won't participate in the hierarchy management anymore. Destroying + * groups would result in interesting race conditions which would just make + * the whole mechanism slow and complex. + * + * + * Locking rules: + * -------------- + * + * For setting up new groups and handling events it's required to lock both + * child and parent group. The lock ordering is always bottom up. This also + * includes the per CPU locks in struct tmigr_cpu. For updating the migrator and + * active CPU/group information atomic_try_cmpxchg() is used instead and only + * the per CPU tmigr_cpu->lock is held. + * + * During the setup of groups tmigr_level_list is required. It is protected by + * @tmigr_mutex. + * + * When @timer_base->lock as well as tmigr related locks are required, the lock + * ordering is: first @timer_base->lock, afterwards tmigr related locks. + * + * + * Protection of the tmigr group state information: + * ------------------------------------------------ + * + * The state information with the list of active children and migrator needs to + * be protected by a sequence counter. It prevents a race when updates in a + * child groups are propagated in changed order. The following scenario + * describes what happens without updating the sequence counter: + * + * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well + * as GRP0:1 will not change during the scenario): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU0 migrator = CPU2 + * active = CPU0 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * active idle active idle + * + * + * 1. CPU0 goes idle (changes are updated in GRP0:0; afterwards the current + * states of GRP0:0 and GRP1:0 are stored in the data for walking the + * hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = TMIGR_NONE migrator = CPU2 + * --> active = active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * --> idle idle active idle + * + * 2. CPU1 comes out of idle (changes are update in GRP0:0; afterwards the + * current states of GRP0:0 and GRP1:0 are stored in the data for walking the + * hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = CPU1 migrator = CPU2 + * --> active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle --> active active idle + * + * 3. Here comes the change of the order: Propagating the changes of step 2 + * through the hierarchy to GRP1:0 - nothing to be done, because GRP0:0 + * is already up to date. + * + * 4. Propagating the changes of step 1 through the hierarchy to GRP1:0 + * + * LVL 1 [GRP1:0] + * --> migrator = GRP0:1 + * --> active = GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU1 migrator = CPU2 + * active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle active active idle + * + * Now there is a inconsistent overall state because GRP0:0 is active, but + * it is marked as idle in the GRP1:0. This is prevented by incrementing + * sequence counter whenever changing the state. + * + * + * Required event and timerqueue update after a remote expiry: + * ----------------------------------------------------------- + * + * After a remote expiry of a CPU, a walk through the hierarchy updating the + * events and timerqueues has to be done when there is a 'new' global timer of + * the remote CPU (which is obvious) but also if there is no new global timer, + * but the remote CPU is still idle: + * + * 1. CPU2 is the migrator and does the remote expiry in GRP1:0; expiry of + * evt-CPU0 and evt-CPU1 are equal: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 2. Remove the first event of the timerqueue in GRP1:0 and expire the timers + * of CPU0 (see evt-GRP0:0->cpu value): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 3. After the remote expiry CPU0 has no global timer that needs to be + * enqueued. When skipping the walk, the global timer of CPU1 is not handled, + * as the group event of GRP0:0 is not updated and not enqueued into GRP1:0. The + * walk has to be done to update the group events and timerqueues: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU1 groupevt.cpu = + * --> timerqueue = evt-CPU1 timerqueue = + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * Now CPU2 (migrator) is able to handle the timer of CPU1 as CPU2 only scans + * the timerqueues of GRP0:1 and GRP1:0. + * + * The update of step 3 is valid to be skipped, when the remote CPU went offline + * in the meantime because an update was already done during inactive path. When + * CPU became active in the meantime, update isn't required as well, because + * GRP0:0 is now longer idle. + */ + +static DEFINE_MUTEX(tmigr_mutex); +static struct list_head *tmigr_level_list __read_mostly; + +static unsigned int tmigr_hierarchy_levels __read_mostly; +static unsigned int tmigr_crossnode_level __read_mostly; + +static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu); + +#define TMIGR_NONE 0xFF +#define BIT_CNT 8 + +static DEFINE_STATIC_KEY_FALSE(tmigr_enabled); + +static inline bool is_tmigr_enabled(void) +{ + return static_branch_unlikely(&tmigr_enabled); +} + +/* + * Returns true, when @childmask corresponds to the group migrator or when the + * group is not active - so no migrator is set. + */ +static bool tmigr_check_migrator(struct tmigr_group *group, u8 childmask) +{ + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + return true; + + return false; +} + +static bool tmigr_check_migrator_and_lonely(struct tmigr_group *group, u8 childmask) +{ + bool lonely, migrator = false; + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + migrator = true; + + active = s.active; + lonely = bitmap_weight(&active, BIT_CNT) <= 1; + + return (migrator && lonely); +} + +static bool tmigr_check_lonely(struct tmigr_group *group) +{ + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + active = s.active; + + return bitmap_weight(&active, BIT_CNT) <= 1; +} + +typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *); + +static void __walk_groups(up_f up, void *data, + struct tmigr_cpu *tmc) +{ + struct tmigr_group *child = NULL, *group = tmc->tmgroup; + + do { + WARN_ON_ONCE(group->level >= tmigr_hierarchy_levels); + + if (up(group, child, data)) + break; + + child = group; + group = group->parent; + } while (group); +} + +static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc) +{ + lockdep_assert_held(&tmc->lock); + + __walk_groups(up, data, tmc); +} + +/** + * struct tmigr_walk - data required for walking the hierarchy + * @evt: Pointer to tmigr_event which needs to be queued (of idle + * child group) + * @childmask: childmask of child group + * @nextexp: Next CPU event expiry information which is handed into + * the timer migration code by the timer code + * (get_next_timer_interrupt()); it is furthermore used for + * the first event which is queued, if the timer migration + * hierarchy is completely idle + * @childstate: tmigr_group->migr_state of the child - will be only + * reread when cmpxchg in the group fails (is required for + * the deactivate path and the new timer path) + * @groupstate: tmigr_group->migr_state of the group - will be only + * reread when cmpxchg in the group fails (is required for + * the active, the deactivate and the new timer path) + * @remote: Is set, when the new timer path is executed in + * tmigr_handle_remote_cpu() + */ +struct tmigr_walk { + struct tmigr_event *evt; + u8 childmask; + u64 nextexp; + union tmigr_state childstate; + union tmigr_state groupstate; + bool remote; +}; + +/** + * struct tmigr_remote_data - data required for (check) remote expiry + * hierarchy walk + * @basej: timer base in jiffies + * @now: timer base monotonic + * @nextexp: returns expiry of the first timer in the idle timer + * migration hierarchy to make sure the timer is handled in + * time; it is stored in the per CPU tmigr_cpu struct of + * CPU which expires remote timers + * @childmask: childmask of child group + * @check: is set if there is the need to handle remote timers; + * required in tmigr_check_handle_remote() only + * @tmc_active: this flag indicates, whether the CPU which triggers + * the hierarchy walk is !idle in the timer migration + * hierarchy. When the CPU is idle and the whole hierarchy is + * idle, only the first event of the top level has to be + * considered. + */ +struct tmigr_remote_data { + unsigned long basej; + u64 now; + u64 nextexp; + u8 childmask; + bool check; + bool tmc_active; +}; + +/* + * Returns the next event of the timerqueue @group->events + * + * Removes timers with ignore flag and update next_expiry of the group. Values + * of the group event are updated in tmigr_update_events() only. + */ +static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group) +{ + struct timerqueue_node *node = NULL; + struct tmigr_event *evt = NULL; + + lockdep_assert_held(&group->lock); + + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + while ((node = timerqueue_getnext(&group->events))) { + evt = container_of(node, struct tmigr_event, nextevt); + + if (!evt->ignore) { + WRITE_ONCE(group->next_expiry, evt->nextevt.expires); + return evt; + } + + /* + * Remove next timers with ignore flag, because the group lock + * is held anyway + */ + if (!timerqueue_del(&group->events, node)) + break; + } + + return NULL; +} + +/* + * Return the next event which is already expired of the group timerqueue + * + * Event, which is returned, is also removed from the queue. + */ +static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group, + u64 now) +{ + struct tmigr_event *evt = tmigr_next_groupevt(group); + + if (!evt || now < evt->nextevt.expires) + return NULL; + + /* + * The event is already expired. Remove it. If it's not the last event, + * then update all group event related information. + */ + if (timerqueue_del(&group->events, &evt->nextevt)) + tmigr_next_groupevt(group); + else + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + return evt; +} + +static u64 tmigr_next_groupevt_expires(struct tmigr_group *group) +{ + struct tmigr_event *evt; + + evt = tmigr_next_groupevt(group); + + if (!evt) + return KTIME_MAX; + else + return evt->nextevt.expires; +} + +static bool tmigr_active_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + if (newstate.migrator == TMIGR_NONE) { + newstate.migrator = childmask; + + /* Changes need to be propagated */ + walk_done = false; + } + + newstate.active |= childmask; + + newstate.seq++; + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + goto retry; + } + + if (group->parent && (walk_done == false)) { + data->groupstate.state = atomic_read(&group->parent->migr_state); + data->childmask = group->childmask; + } + + /* + * The group is active and the event will be ignored - the ignore flag is + * updated without holding the lock. In case the bit is set while + * another CPU already handles remote events, nothing happens, because + * it is clear that the CPU became active just in this moment, or in + * worst case the event is handled remote. Nothing to worry about. + */ + group->groupevt.ignore = true; + + return walk_done; +} + +static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) +{ + struct tmigr_walk data; + + data.childmask = tmc->childmask; + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + tmc->cpuevt.ignore = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + tmc->wakeup_recalc = false; + + walk_groups(&tmigr_active_up, &data, tmc); +} + +/** + * tmigr_cpu_activate - set CPU active in timer migration hierarchy + * + * Call site timer_clear_idle() is called with interrupts disabled + */ +void tmigr_cpu_activate(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online || !tmc->idle) + return; + + raw_spin_lock(&tmc->lock); + tmc->idle = false; + __tmigr_cpu_activate(tmc); + raw_spin_unlock(&tmc->lock); +} + +/* + * Returns true, if there is nothing to be propagated to the next level + * + * @data->nextexp is reset to KTIME_MAX; it is reused for the first global event + * which needs to be handled by the migrator (in the top level group). + * + * This is the only place where the group event expiry value is set. + */ +static bool tmigr_update_events(struct tmigr_group *group, + struct tmigr_group *child, + struct tmigr_walk *data) +{ + struct tmigr_event *evt, *first_childevt; + bool walk_done, remote = data->remote; + bool leftmost_change = false; + u64 nextexp; + + if (child) { + raw_spin_lock(&child->lock); + raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING); + + data->childstate.state = atomic_read(&child->migr_state); + data->groupstate.state = atomic_read(&group->migr_state); + + if (data->childstate.active) { + walk_done = true; + goto unlock; + } + + first_childevt = tmigr_next_groupevt(child); + nextexp = child->next_expiry; + evt = &child->groupevt; + } else { + nextexp = data->nextexp; + + /* + * Set @data->nextexp to KTIME_MAX; it is reused for the first + * global event which needs to be handled by the migrator (in + * the top level group). + */ + data->nextexp = KTIME_MAX; + + first_childevt = evt = data->evt; + + /* + * Walking the hierarchy is required in any case when a + * remote expiry was done before. This ensures to not lose + * already queued events in non active groups (see section + * "Required event and timerqueue update after remote + * expiry" in the documentation at the top). + * + * The two call sites which are executed without a remote expiry + * before, are not prevented from propagating changes through + * the hierarchy by the return: + * - When entering this path by tmigr_new_timer(), @evt->ignore + * is never set. + * - tmigr_inactive_up() takes care of the propagation by + * itself and ignores the return value. But an immediate + * return is required because nothing has to be done in this + * level as the event could be ignored. + */ + if (evt->ignore && !remote) + return true; + + raw_spin_lock(&group->lock); + data->groupstate.state = atomic_read(&group->migr_state); + } + + if (nextexp == KTIME_MAX) { + evt->ignore = true; + + /* + * When the next child event could be ignored (nextexp is + * KTIME_MAX) and there was no remote timer handling before or + * the group is already active, there is no need to walk the + * hierarchy even if there is a parent group. + * + * The other way round: even if the event could be ignored, but + * if a remote timer handling was executed before and the group + * is not active, walking the hierarchy is required to not miss + * an enqueued timer in the non active group. The enqueued timer + * of the group needs to be propagated to a higher level to + * ensure it is handled. + */ + if (!remote || data->groupstate.active) { + walk_done = true; + goto unlock; + } + } else { + /* + * An update of @evt->cpu and @evt->ignore flag is required only + * when @child is set (the child is equal or higher than lvl0), + * but it doesn't matter if it is written once more to the per + * CPU event; make the update unconditional. + */ + evt->cpu = first_childevt->cpu; + evt->ignore = false; + } + + walk_done = !group->parent; + + /* + * If the child event is already queued in the group, remove it from the + * queue when the expiry time changed only. + */ + if (timerqueue_node_queued(&evt->nextevt)) { + if (evt->nextevt.expires == nextexp) + goto check_toplvl; + + leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt; + if (!timerqueue_del(&group->events, &evt->nextevt)) + WRITE_ONCE(group->next_expiry, KTIME_MAX); + } + + evt->nextevt.expires = nextexp; + + if (timerqueue_add(&group->events, &evt->nextevt)) { + leftmost_change = true; + WRITE_ONCE(group->next_expiry, nextexp); + } + +check_toplvl: + if (walk_done && (data->groupstate.migrator == TMIGR_NONE)) { + /* + * Nothing to do when first event didn't changed and update was + * done during remote timer handling. + */ + if (remote && !leftmost_change) + goto unlock; + /* + * The top level group is idle and it has to be ensured the + * global timers are handled in time. (This could be optimized + * by keeping track of the last global scheduled event and only + * arming it on the CPU if the new event is earlier. Not sure if + * its worth the complexity.) + */ + data->nextexp = tmigr_next_groupevt_expires(group); + } + +unlock: + raw_spin_unlock(&group->lock); + + if (child) + raw_spin_unlock(&child->lock); + + return walk_done; +} + +static bool tmigr_new_timer_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_walk *data = ptr; + + return tmigr_update_events(group, child, data); +} + +/* + * Returns the expiry of the next timer that needs to be handled. KTIME_MAX is + * returned, when an active CPU will handle all the timer migration hierarchy + * timers. + */ +static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .evt = &tmc->cpuevt, + .nextexp = nextexp }; + + lockdep_assert_held(&tmc->lock); + + if (tmc->remote) + return KTIME_MAX; + + tmc->cpuevt.ignore = false; + data.remote = false; + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + /* If there is a new first global event, make sure it is handled */ + return data.nextexp; +} + +static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, + unsigned long jif) +{ + struct timer_events tevt; + struct tmigr_walk data; + struct tmigr_cpu *tmc; + u64 next = KTIME_MAX; + + tmc = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_lock_irq(&tmc->lock); + + /* + * The remote CPU is offline or the CPU event does not has to be handled + * (the CPU is active or there is no longer an event to expire) or + * another CPU handles the CPU timers already or the next event was + * already expired - return! + */ + if (!tmc->online || tmc->remote || tmc->cpuevt.ignore || + now < tmc->cpuevt.nextevt.expires) { + raw_spin_unlock_irq(&tmc->lock); + return next; + } + + tmc->remote = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* Drop the lock to allow the remote CPU to exit idle */ + raw_spin_unlock_irq(&tmc->lock); + + if (cpu != smp_processor_id()) + timer_expire_remote(cpu); + + /* + * Lock ordering needs to be preserved - timer_base locks before tmigr + * related locks (see section "Locking rules" in the documentation at + * the top). During fetching the next timer interrupt, also tmc->lock + * needs to be held. Otherwise there is a possible race window against + * the CPU itself when it comes out of idle, updates the first timer in + * the hierarchy and goes back to idle. + * + * timer base locks are dropped as fast as possible: After checking + * whether the remote CPU went offline in the meantime and after + * fetching the next remote timer interrupt. Dropping the locks as fast + * as possible keeps the locking region small and prevents holding + * several (unnecessary) locks during walking the hierarchy for updating + * the timerqueue and group events. + */ + local_irq_disable(); + timer_lock_remote_bases(cpu); + raw_spin_lock(&tmc->lock); + + /* + * When the CPU went offline in the meantime, no hierarchy walk has to + * be done for updating the queued events, because the walk was + * already done during marking the CPU offline in the hierarchy. + * + * When the CPU is no longer idle, the CPU takes care of the timers and + * also of the timers in the path to the top. + * + * (See also section "Required event and timerqueue update after + * remote expiry" in the documentation at the top) + */ + if (!tmc->online || !tmc->idle) { + timer_unlock_remote_bases(cpu); + goto unlock; + } else { + /* next event of CPU */ + fetch_next_timer_interrupt_remote(jif, now, &tevt, cpu); + } + + timer_unlock_remote_bases(cpu); + + data.evt = &tmc->cpuevt; + data.nextexp = tevt.global; + data.remote = true; + + /* + * The update is done even when there is no 'new' global timer pending + * on the remote CPU (see section "Required event and timerqueue update + * after remote expiry" in the documentation at the top) + */ + walk_groups(&tmigr_new_timer_up, &data, tmc); + + next = data.nextexp; + +unlock: + tmc->remote = false; + raw_spin_unlock_irq(&tmc->lock); + + return next; +} + +static bool tmigr_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u64 now, next = KTIME_MAX; + struct tmigr_event *evt; + unsigned long jif; + u8 childmask; + + jif = data->basej; + now = data->now; + + childmask = data->childmask; + +again: + /* + * Handle the group only if @childmask is the migrator or if the + * group has no migrator. Otherwise the group is active and is + * handled by its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + raw_spin_lock_irq(&group->lock); + + evt = tmigr_next_expired_groupevt(group, now); + + if (evt) { + unsigned int remote_cpu = evt->cpu; + + raw_spin_unlock_irq(&group->lock); + + next = tmigr_handle_remote_cpu(remote_cpu, now, jif); + + /* check if there is another event, that needs to be handled */ + goto again; + } else { + raw_spin_unlock_irq(&group->lock); + } + + /* Update of childmask for the next level */ + data->childmask = group->childmask; + data->nextexp = next; + + return false; +} + +/** + * tmigr_handle_remote - Handle migratable timers on remote idle CPUs + * + * Called from the timer soft interrupt with interrupts enabled. + */ +void tmigr_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return; + + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + + /* + * NOTE: This is a doubled check because the migrator test will be done + * in tmigr_handle_remote_up() anyway. Keep this check to fasten the + * return when nothing has to be done. + */ + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return; + + data.now = get_jiffies_update(&data.basej); + + /* + * Update @tmc->wakeup only at the end and do not reset @tmc->wakeup to + * KTIME_MAX. Even if tmc->lock is not held during the whole remote + * handling, tmc->wakeup is fine to be stale as it is called in + * interrupt context and tick_nohz_next_event() is executed in interrupt + * exit path only after processing the last pending interrupt. + */ + + __walk_groups(&tmigr_handle_remote_up, &data, tmc); + + raw_spin_lock_irq(&tmc->lock); + WRITE_ONCE(tmc->wakeup, data.nextexp); + raw_spin_unlock_irq(&tmc->lock); +} + +static bool tmigr_requires_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u8 childmask; + + childmask = data->childmask; + + /* + * Handle the group only if the child is the migrator or if the group + * has no migrator. Otherwise the group is active and is handled by its + * own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + /* + * When there is a parent group and the CPU which triggered the + * hierarchy walk is not active, proceed the walk to reach the top level + * group before reading the next_expiry value. + */ + if (group->parent && !data->tmc_active) + goto out; + + /* + * On 32 bit systems the racy lockless check for next_expiry will + * turn into a random number generator. Therefore do the lockless + * check only on 64 bit systems. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + data->nextexp = READ_ONCE(group->next_expiry); + if (data->now >= data->nextexp) { + data->check = true; + return true; + } + } else { + raw_spin_lock(&group->lock); + data->nextexp = group->next_expiry; + if (data->now >= group->next_expiry) { + data->check = true; + raw_spin_unlock(&group->lock); + return true; + } + raw_spin_unlock(&group->lock); + } + +out: + /* Update of childmask for the next level */ + data->childmask = group->childmask; + return false; +} + +/** + * tmigr_requires_handle_remote - Check whether remote timer handling is required + * + * Must be called with interrupts disabled. + */ +int tmigr_requires_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + unsigned int ret = 0; + unsigned long jif; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return ret; + + data.now = get_jiffies_update(&jif); + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + data.tmc_active = !tmc->idle; + data.check = false; + + /* + * When the CPU is active, walking the hierarchy to check whether a + * remote expiry is required. + * + * Check is done lockless as interrupts are disabled and @tmc->idle is + * set only by the local CPU. + */ + if (!tmc->idle) { + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + if (data.nextexp != KTIME_MAX) + ret = 1; + + return ret; + } + + /* + * When the CPU is idle, check whether the recalculation of @tmc->wakeup + * is required. @tmc->wakeup_recalc is set by a remote CPU which is + * about to go offline, was the last active CPU in the whole timer + * migration hierarchy and now delegates handling of the hierarchy to + * this CPU. + * + * Racy lockless check is valid: + * - @tmc->wakeup_recalc is set by the remote CPU before it issues + * reschedule IPI. + * - As interrupts are disabled here this CPU will either observe + * @tmc->wakeup_recalc set before the reschedule IPI can be handled or + * it will observe it when this function is called again on return + * from handling the reschedule IPI. + */ + if (tmc->wakeup_recalc) { + raw_spin_lock(&tmc->lock); + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + if (data.nextexp != KTIME_MAX) + ret = 1; + + WRITE_ONCE(tmc->wakeup, data.nextexp); + tmc->wakeup_recalc = false; + raw_spin_unlock(&tmc->lock); + + return ret; + } + + /* + * When the CPU is idle and @tmc->wakeup is reliable, compare it with + * @data.now. On 64 bit it is valid to do this lockless. On 32 bit + * systems, holding the lock is required to get valid data on concurrent + * writers. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + if (data.now >= READ_ONCE(tmc->wakeup)) + ret = 1; + } else { + raw_spin_lock(&tmc->lock); + if (data.now >= tmc->wakeup) + ret = 1; + raw_spin_unlock(&tmc->lock); + } + + return ret; +} + +static bool tmigr_inactive_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + /* Reset active bit when the child is no longer active */ + if (!data->childstate.active) + newstate.active &= ~childmask; + + if (newstate.migrator == childmask) { + /* + * Find a new migrator for the group, because the child group is + * idle! + */ + if (!data->childstate.active) { + unsigned long new_migr_bit, active = newstate.active; + + new_migr_bit = find_first_bit(&active, BIT_CNT); + + if (new_migr_bit != BIT_CNT) { + newstate.migrator = BIT(new_migr_bit); + } else { + newstate.migrator = TMIGR_NONE; + + /* Changes need to be propagated */ + walk_done = false; + } + } + } + + newstate.seq++; + + WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active)); + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + + /* + * Something changed in the child/parent group in the meantime, + * reread the state of the child and parent; Update of + * data->childstate is required for event handling; + */ + if (child) + data->childstate.state = atomic_read(&child->migr_state); + + goto retry; + } + + data->groupstate = newstate; + data->remote = false; + + /* Event Handling */ + tmigr_update_events(group, child, data); + + if (group->parent && (walk_done == false)) { + data->childmask = group->childmask; + data->childstate = newstate; + data->groupstate.state = atomic_read(&group->parent->migr_state); + } + + /* + * data->nextexp was set by tmigr_update_events() and contains the + * expiry of the first global event which needs to be handled + */ + if (data->nextexp != KTIME_MAX) { + WARN_ON_ONCE(group->parent); + /* + * Top level path - If this CPU is about going offline, wake + * up some random other CPU so it will take over the + * migrator duty and program its timer properly. Ideally + * wake the CPU with the closest expiry time, but that's + * overkill to figure out. + * + * Set wakeup_recalc of remote CPU, to make sure the complete + * idle hierarchy with enqueued timers is reevaluated. + */ + if (!(this_cpu_ptr(&tmigr_cpu)->online)) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned int cpu = smp_processor_id(); + struct tmigr_cpu *tmc_resched; + + cpu = cpumask_any_but(cpu_online_mask, cpu); + tmc_resched = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_unlock(&tmc->lock); + + raw_spin_lock(&tmc_resched->lock); + tmc_resched->wakeup_recalc = true; + raw_spin_unlock(&tmc_resched->lock); + + raw_spin_lock(&tmc->lock); + smp_send_reschedule(cpu); + } + } + + return walk_done; +} + +static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .childmask = tmc->childmask, + .evt = &tmc->cpuevt, + .nextexp = nextexp, + .childstate.state = 0 }; + + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + /* + * If nextexp is KTIME_MAX, the CPU event will be ignored because the + * local timer expires before the global timer, no global timer is set + * or CPU goes offline. + */ + if (nextexp != KTIME_MAX) + tmc->cpuevt.ignore = false; + + walk_groups(&tmigr_inactive_up, &data, tmc); + return data.nextexp; +} + +/** + * tmigr_cpu_deactivate - Put current CPU into inactive state + * @nextexp: The next timer event expiry set in the current CPU + * + * Must be called with interrupts disabled. + * + * Return: the next event expiry of the current CPU or the next event expiry + * from the hierarchy if this CPU is the top level migrator or the hierarchy is + * completely idle. + */ +u64 tmigr_cpu_deactivate(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return nextexp; + + raw_spin_lock(&tmc->lock); + + /* + * The CPU is already deactivated in the timer migration + * hierarchy. tick_nohz_get_sleep_length() calls tick_nohz_next_event() + * and thereby the timer idle path is executed once more. @tmc->wakeup + * holds the first timer, when the timer migration hierarchy is + * completely idle. If there is no new next expiry value handed in which + * should be inserted into the timer migration hierarchy, the wakeup + * time is returned. + */ + if (tmc->idle) { + ret = READ_ONCE(tmc->wakeup); + if (nextexp != KTIME_MAX) { + if (nextexp != tmc->cpuevt.nextevt.expires || + tmc->cpuevt.ignore) { + ret = tmigr_new_timer(tmc, nextexp); + } + } else if (tmc->wakeup_recalc) { + struct tmigr_remote_data data; + + data.now = KTIME_MAX; + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + data.tmc_active = false; + data.check = false; + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + ret = data.nextexp; + } + tmc->wakeup_recalc = false; + goto unlock; + } + + ret = __tmigr_cpu_deactivate(tmc, nextexp); + + tmc->idle = true; + +unlock: + /* + * Make sure the reevaluation of timers in idle path will not miss an + * event. + */ + WRITE_ONCE(tmc->wakeup, ret); + + raw_spin_unlock(&tmc->lock); + return ret; +} + +/** + * tmigr_quick_check - Quick check which is called by tick_nohz_next_event + * + * Returns KTIME_MAX, when it is probable that nothing has to be done (not the + * only one in the level 0 group; and if it is the only one in level 0 group, + * but there are more than a single group active in top level) + * + * Returns first expiry of the top level group, when it is the only one in level + * 0 and top level also only has a single active child. + */ +u64 tmigr_quick_check(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_group *topgroup; + struct list_head lvllist; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online || tmc->idle) + return KTIME_MAX; + + if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask)) + return KTIME_MAX; + + for (int i = tmigr_hierarchy_levels; i > 0 ; i--) { + lvllist = tmigr_level_list[i - 1]; + if (list_is_singular(&lvllist)) { + topgroup = list_first_entry(&lvllist, struct tmigr_group, list); + + if (tmigr_check_lonely(topgroup)) + return READ_ONCE(topgroup->next_expiry); + } else { + continue; + } + } + + return KTIME_MAX; +} + +static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl, + int node) +{ + union tmigr_state s; + + raw_spin_lock_init(&group->lock); + + group->level = lvl; + group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE; + + group->num_children = 0; + + s.migrator = TMIGR_NONE; + s.active = 0; + s.seq = 0; + atomic_set(&group->migr_state, s.state); + + timerqueue_init_head(&group->events); + timerqueue_init(&group->groupevt.nextevt); + group->groupevt.nextevt.expires = KTIME_MAX; + WRITE_ONCE(group->next_expiry, KTIME_MAX); + group->groupevt.ignore = true; +} + +static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, + unsigned int lvl) +{ + struct tmigr_group *tmp, *group = NULL; + + lockdep_assert_held(&tmigr_mutex); + + /* Try to attach to an existing group first */ + list_for_each_entry(tmp, &tmigr_level_list[lvl], list) { + /* + * If @lvl is below the cross numa node level, check whether + * this group belongs to the same numa node. + */ + if (lvl < tmigr_crossnode_level && tmp->numa_node != node) + continue; + + /* Capacity left? */ + if (tmp->num_children >= TMIGR_CHILDREN_PER_GROUP) + continue; + + /* + * TODO: A possible further improvement: Make sure that all CPU + * siblings end up in the same group of the lowest level of the + * hierarchy. Rely on the topology sibling mask would be a + * reasonable solution. + */ + + group = tmp; + break; + } + + if (group) + return group; + + /* Allocate and set up a new group */ + group = kzalloc_node(sizeof(*group), GFP_KERNEL, node); + if (!group) + return ERR_PTR(-ENOMEM); + + tmigr_init_group(group, lvl, node); + + /* Setup successful. Add it to the hierarchy */ + list_add(&group->list, &tmigr_level_list[lvl]); + return group; +} + +static void tmigr_connect_child_parent(struct tmigr_group *child, + struct tmigr_group *parent) +{ + union tmigr_state childstate; + + raw_spin_lock_irq(&child->lock); + raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING); + + child->parent = parent; + child->childmask = BIT(parent->num_children++); + + raw_spin_unlock(&parent->lock); + raw_spin_unlock_irq(&child->lock); + + /* + * To prevent inconsistent states, active children need to be active in + * the new parent as well. Inactive children are already marked inactive + * in the parent group. + */ + childstate.state = atomic_read(&child->migr_state); + if (childstate.migrator != TMIGR_NONE) { + struct tmigr_walk data; + + data.childmask = child->childmask; + data.groupstate.state = atomic_read(&parent->migr_state); + + /* + * There is only one new level per time. When connecting the + * child and the parent and set the child active when the parent + * is inactive, the parent needs to be the uppermost + * level. Otherwise there went something wrong! + */ + WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent); + } +} + +static int tmigr_setup_groups(unsigned int cpu, unsigned int node) +{ + struct tmigr_group *group, *child, **stack; + int top = 0, err = 0, i = 0; + struct list_head *lvllist; + + stack = kcalloc(tmigr_hierarchy_levels, sizeof(*stack), GFP_KERNEL); + if (!stack) + return -ENOMEM; + + do { + group = tmigr_get_group(cpu, node, i); + if (IS_ERR(group)) { + err = PTR_ERR(group); + break; + } + + top = i; + stack[i++] = group; + + /* + * When booting only less CPUs of a system than CPUs are + * available, not all calculated hierarchy levels are required. + * + * The loop is aborted as soon as the highest level, which might + * be different from tmigr_hierarchy_levels, contains only a + * single group. + */ + if (group->parent || i == tmigr_hierarchy_levels || + (list_empty(&tmigr_level_list[i]) && + list_is_singular(&tmigr_level_list[i - 1]))) + break; + + } while (i < tmigr_hierarchy_levels); + + do { + group = stack[--i]; + + if (err < 0) { + list_del(&group->list); + kfree(group); + continue; + } + + WARN_ON_ONCE(i != group->level); + + /* + * Update tmc -> group / child -> group connection + */ + if (i == 0) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&group->lock); + + tmc->tmgroup = group; + tmc->childmask = BIT(group->num_children++); + + raw_spin_unlock_irq(&group->lock); + + /* There are no children that need to be connected */ + continue; + } else { + child = stack[i - 1]; + tmigr_connect_child_parent(child, group); + } + + /* check if uppermost level was newly created */ + if (top != i) + continue; + + WARN_ON_ONCE(top == 0); + + lvllist = &tmigr_level_list[top]; + if (group->num_children == 1 && list_is_singular(lvllist)) { + lvllist = &tmigr_level_list[top - 1]; + list_for_each_entry(child, lvllist, list) { + if (child->parent) + continue; + + tmigr_connect_child_parent(child, group); + } + } + } while (i > 0); + + kfree(stack); + + return err; +} + +static int tmigr_add_cpu(unsigned int cpu) +{ + int node = cpu_to_node(cpu); + int ret; + + mutex_lock(&tmigr_mutex); + ret = tmigr_setup_groups(cpu, node); + mutex_unlock(&tmigr_mutex); + + return ret; +} + +static int tmigr_cpu_online(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned int ret; + + /* First online attempt? Initialize CPU data */ + if (!tmc->tmgroup) { + raw_spin_lock_init(&tmc->lock); + + ret = tmigr_add_cpu(cpu); + if (ret < 0) + return ret; + + if (tmc->childmask == 0) + return -EINVAL; + + timerqueue_init(&tmc->cpuevt.nextevt); + tmc->cpuevt.nextevt.expires = KTIME_MAX; + tmc->cpuevt.ignore = true; + tmc->cpuevt.cpu = cpu; + + tmc->remote = false; + tmc->idle = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + } + raw_spin_lock_irq(&tmc->lock); + __tmigr_cpu_activate(tmc); + tmc->online = true; + raw_spin_unlock_irq(&tmc->lock); + return 0; +} + +static int tmigr_cpu_offline(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&tmc->lock); + tmc->online = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* + * CPU has to handle the local events on his own, when on the way to + * offline; Therefore nextevt value is set to KTIME_MAX + */ + __tmigr_cpu_deactivate(tmc, KTIME_MAX); + raw_spin_unlock_irq(&tmc->lock); + + return 0; +} + +static int __init tmigr_init(void) +{ + unsigned int cpulvl, nodelvl, cpus_per_node, i; + unsigned int nnodes = num_possible_nodes(); + unsigned int ncpus = num_possible_cpus(); + int ret = -ENOMEM; + + /* Nothing to do if running on UP */ + if (ncpus == 1) + return 0; + + /* + * Calculate the required hierarchy levels. Unfortunately there is no + * reliable information available, unless all possible CPUs have been + * brought up and all numa nodes are populated. + * + * Estimate the number of levels with the number of possible nodes and + * the number of possible CPUs. Assume CPUs are spread evenly across + * nodes. We cannot rely on cpumask_of_node() because there only already + * online CPUs are considered. + */ + cpus_per_node = DIV_ROUND_UP(ncpus, nnodes); + + /* Calc the hierarchy levels required to hold the CPUs of a node */ + cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + /* Calculate the extra levels to connect all nodes */ + nodelvl = DIV_ROUND_UP(order_base_2(nnodes), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + tmigr_hierarchy_levels = cpulvl + nodelvl; + + /* + * If a numa node spawns more than one CPU level group then the next + * level(s) of the hierarchy contains groups which handle all CPU groups + * of the same numa node. The level above goes across numa nodes. Store + * this information for the setup code to decide when node matching is + * not longer required. + */ + tmigr_crossnode_level = cpulvl; + + tmigr_level_list = kcalloc(tmigr_hierarchy_levels, sizeof(struct list_head), GFP_KERNEL); + if (!tmigr_level_list) + goto err; + + for (i = 0; i < tmigr_hierarchy_levels; i++) + INIT_LIST_HEAD(&tmigr_level_list[i]); + + pr_info("Timer migration: %d hierarchy levels; %d children per group;" + " %d crossnode level\n", + tmigr_hierarchy_levels, TMIGR_CHILDREN_PER_GROUP, + tmigr_crossnode_level); + + ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online", + tmigr_cpu_online, tmigr_cpu_offline); + if (ret) + goto err; + + static_branch_enable(&tmigr_enabled); + + return 0; + +err: + pr_err("Timer migration setup failed\n"); + return ret; +} +late_initcall(tmigr_init); diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h new file mode 100644 index 000000000000..01c08a5dbb49 --- /dev/null +++ b/kernel/time/timer_migration.h @@ -0,0 +1,144 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _KERNEL_TIME_MIGRATION_H +#define _KERNEL_TIME_MIGRATION_H + +/* Per group capacity. Must be a power of 2! */ +#define TMIGR_CHILDREN_PER_GROUP 8 + +/** + * struct tmigr_event - a timer event associated to a CPU + * @nextevt: The node to enqueue an event in the parent group queue + * @cpu: The CPU to which this event belongs + * @ignore: Hint whether the event could be ignored; it is set when + * CPU or group is active; + */ +struct tmigr_event { + struct timerqueue_node nextevt; + unsigned int cpu; + bool ignore; +}; + +/** + * struct tmigr_group - timer migration hierarchy group + * @lock: Lock protecting the event information and group hierarchy + * information during setup + * @migr_state: State of group (see union tmigr_state) + * @parent: Pointer to parent group + * @groupevt: Next event of group which is only used when group is + * !active. The group event is then queued into parent + * timer queue. + * Ignore bit of @groupevt is set when group is active. + * @next_expiry: Base monotonic expiry time of next event of group; + * Used for racy lockless check whether remote expiry is + * required; it is always reliable + * @events: Timer queue for child events queued in the group + * @childmask: childmask of group in parent group; is set during + * setup and will never change; could be read lockless + * @level: Hierarchy level of group; Required during setup and + * when DEBUG is defined + * @list: List head that is added to per level + * tmigr_level_list; is required during setup when a + * new group needs to be connected to the existing + * hierarchy groups + * @numa_node: Is set to numa node when level < tmigr_crossnode_level; + * otherwise it is set to NUMA_NO_NODE; Required for + * setup only to make sure CPUs and groups are per + * numa node as long as level < tmigr_crossnode_level + * @num_children: Counter of group children to make sure group is only filled + * with TMIGR_CHILDREN_PER_GROUP; Required for setup only + */ +struct tmigr_group { + raw_spinlock_t lock; + atomic_t migr_state; + struct tmigr_group *parent; + struct tmigr_event groupevt; + u64 next_expiry; + struct timerqueue_head events; + u8 childmask; + unsigned int level; + struct list_head list; + int numa_node; + unsigned int num_children; +}; + +/** + * struct tmigr_cpu - timer migration per CPU group + * @lock: Lock protecting tmigr_cpu group information + * @online: Indicates whether CPU is online; In deactivate path it + * is required to know whether the migrator in top level + * group is on the way to go offline when a timer is + * pending. Then another online CPU needs to be rescheduled + * to make sure timers are handled properly; Furthermore + * the information is required in CPU hotplug path as CPU is + * able to go idle before timer migration hierarchy hotplug + * AP is reached. During this phase CPU has to handle + * global timers by its own and does not act as migrator. + * @idle: Indicates whether CPU is idle in timer migration + * hierarchy + * @remote: Is set when timers of CPU are expired remote + * @wakeup_recalc: Indicates, whether a recalculation of @wakeup value is + * required. It is only used when CPU is marked idle in + * timer migration hierarchy. + * @tmgroup: Pointer to parent group + * @childmask: childmask of tmigr_cpu in parent group + * @wakeup: Stores the first timer when the timer migration + * hierarchy is completely idle and remote expiry was done; + * is returned to timer code when tmigr_cpu_deactive() is + * called and group is idle; it is only valid, when + * @wakeup_recalc is not set; + * @cpuevt: CPU event which could be queued into parent group + */ +struct tmigr_cpu { + raw_spinlock_t lock; + bool online; + bool idle; + bool remote; + bool wakeup_recalc; + struct tmigr_group *tmgroup; + u8 childmask; + u64 wakeup; + struct tmigr_event cpuevt; +}; + +/** + * union tmigr_state - state of tmigr_group + * @state: Combined version of the state - only used for atomic + * read/cmpxchg function + * @struct: Split version of the state - only use the struct members to + * update information to stay independent of endianness + */ +union tmigr_state { + u32 state; + /** + * struct - split state of tmigr_group + * @active: Contains each childmask bit of active children + * @migrator: Contains childmask of child which is migrator + * @seq: Sequence counter needs to be increased when update + * to the tmigr_state is done. It prevents a race when + * updates in child groups are propagated in changed + * order. Detailed information about the scenario is + * given in documentation at the begin of + * timer_migration.c. + */ + struct { + u8 active; + u8 migrator; + u16 seq; + } __packed; +}; + +#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) +extern void tmigr_handle_remote(void); +extern int tmigr_requires_handle_remote(void); +extern void tmigr_cpu_activate(void); +extern u64 tmigr_cpu_deactivate(u64 nextevt); +extern u64 tmigr_quick_check(void); +#else +static inline void tmigr_handle_remote(void) { } +static inline int tmigr_requires_handle_remote(void) { return 0; } +static inline void tmigr_cpu_activate(void) { } +static inline u64 tmigr_cpu_deactivate(u64 nextevt) { return KTIME_MAX; } +static inline u64 tmigr_quick_check(void) { return KTIME_MAX; }; +#endif + +#endif From patchwork Wed Oct 4 12:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148276 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98871vqb; Wed, 4 Oct 2023 05:36:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGheoPLB66SZvWynkYvWm7qSUFxqsOFIT3y/X5HxR44wU33eXVj6Lg3MBZCs2EomAE1BSAA X-Received: by 2002:a17:902:868e:b0:1c3:3c91:61cc with SMTP id g14-20020a170902868e00b001c33c9161ccmr2155855plo.13.1696423019316; Wed, 04 Oct 2023 05:36:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423019; cv=none; d=google.com; s=arc-20160816; b=qJIWZ1/YsZuaRo30O1eqiqN6Q2aCVBI39DkxokTEUAFzbOtvVp1B5ol7a/z9KAPC1r ihu3abNXabcARpSY0C5dJiRmwhoCLgfIOzrO+3BKQ6d2JbsFo0NoJP+gxrYeD+6X+uZ1 sLQ8Uh5lj7468GQm8ZrjW8MJN5L0/uwKsJJ9l3d7t4wqIu1du7cRp9FYU+1lqbGKuXi3 3xxNWsrU630B36mCzyaGEFptMpLGvsbr6OK1csyT20sNM2J6/3UFPp2j23ZK0aez6r1h OZY0uMOb95da+l2MdhmluRH0uoSQhyUCLo1JlpVgIIw6ceFYVx8Cihhksl5iOQMa4Ky+ RXSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=M2kC6VaC2c5TI/lPvw0ZZGnu6gr4GCJ5Rai4QVFVtKk=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=RjLoENznSTxvJs+qpwv3jWdoYuMQpeUkheoQ3DSU56fnF3xNWfoVHmMkJy3dWaDo7v 7+F2UnD0lg6kyWNiSKfaraVzYddWJ3nfZi3dV37kdsV/ozknvzT/IPXVEIWVYEb7HxG6 ygH5ykHn+r2GQZTT1lqWqyQoCee0ARI8C8x3UL+3yMLXwGE8lywd9rTdYWfLKLUqcxh2 yi3be0Q9AyXUFl8fkcVhPKtMUEUCuHmgO5paBclFofbxj0vOdbOjZQIB5qUn+OgR+RdI 593wTswmUdBupMzd6TJI0XZGjF5A1xKomIIQUkYece/eQp0tuSJE5e/0VVwcApmCNHSz BqPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="dDW/d964"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id s18-20020a170903321200b001c6182d9fdasi3308020plh.326.2023.10.04.05.36.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="dDW/d964"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9A52A8213F20; Wed, 4 Oct 2023 05:36:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242372AbjJDMgg (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242444AbjJDMfr (ORCPT ); Wed, 4 Oct 2023 08:35:47 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F4E918F for ; Wed, 4 Oct 2023 05:35:41 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M2kC6VaC2c5TI/lPvw0ZZGnu6gr4GCJ5Rai4QVFVtKk=; b=dDW/d964Oe4UG5Cqr6WeC9y735iYKNj6g7g7uIrmhJGiG0tQ95AFO2jqj1Uh/MLsghKLQD YmtbUS5r/w2UGJdFr4B2dK0efzZBLwrQEXVBimSk8jDQjRpD1vZflp6u3RY+kocK9lYPf7 pP9mwAinOOsxfzNIp1DLLiHDJ0n3kU3cOzx5re+PJjBClvRAjGChvpGEF2gX4TwrE06JjE /ARC7hO2qGz1p4FfSZsW1rkiUwtE8UTQIRRF6qfaqxr2/3iEwpppUIQJgBpiCIqVeT7nBk kTMPxHJDalHwZaiNThqetXtWS3M8i5TCOe2i3a7gE3c7xF5lUxf/OjQC5U8Mvw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M2kC6VaC2c5TI/lPvw0ZZGnu6gr4GCJ5Rai4QVFVtKk=; b=GYF2UbPa0oGRUxp/sdBjS1o5Z7Jw503Kod1F+2GOGIh0Fcx4Ko7zHezXHd4Y1HbUdFIb5i YgvML3jlXAq+K4Dg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v8 24/25] timer_migration: Add tracepoints Date: Wed, 4 Oct 2023 14:34:53 +0200 Message-Id: <20231004123454.15691-25-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828463597496136 X-GMAIL-MSGID: 1778828463597496136 The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen --- v8: Add wakeup value to tracepoints --- include/trace/events/timer_migration.h | 283 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 24 +++ 2 files changed, 307 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..b8232c1a1ebf --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,283 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->numa_node = child->parent->num_children; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->numa_node = tmc->tmgroup->num_children; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p wakeup=%llu", __entry->cpu, __entry->parent, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +TRACE_EVENT(tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu wakeup=%llu", + __entry->cpu, __entry->parent, __entry->nextevt, __entry->wakeup) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index 929b3b94b893..5ac6b4f04c55 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -14,6 +14,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -514,6 +517,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ group->groupevt.ignore = true; + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -545,6 +550,7 @@ void tmigr_cpu_activate(void) raw_spin_lock(&tmc->lock); tmc->idle = false; + trace_tmigr_cpu_active(tmc); __tmigr_cpu_activate(tmc); raw_spin_unlock(&tmc->lock); } @@ -688,6 +694,9 @@ static bool tmigr_update_events(struct tmigr_group *group, data->nextexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, data->childstate, + data->groupstate, nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -721,6 +730,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + tmc->cpuevt.ignore = false; data.remote = false; @@ -754,6 +765,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return next; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); @@ -838,6 +851,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -1156,6 +1170,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, } } + trace_tmigr_group_set_cpu_inactive(group, newstate, childmask); + return walk_done; } @@ -1244,6 +1260,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1352,6 +1369,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -1369,6 +1387,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irq(&child->lock); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active children need to be active in * the new parent as well. Inactive children are already marked inactive @@ -1450,6 +1470,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irq(&group->lock); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no children that need to be connected */ continue; } else { @@ -1518,6 +1540,7 @@ static int tmigr_cpu_online(unsigned int cpu) WRITE_ONCE(tmc->wakeup, KTIME_MAX); } raw_spin_lock_irq(&tmc->lock); + trace_tmigr_cpu_online(tmc); __tmigr_cpu_activate(tmc); tmc->online = true; raw_spin_unlock_irq(&tmc->lock); @@ -1537,6 +1560,7 @@ static int tmigr_cpu_offline(unsigned int cpu) * offline; Therefore nextevt value is set to KTIME_MAX */ __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); return 0; From patchwork Wed Oct 4 12:34:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 148275 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:254a:b0:403:3b70:6f57 with SMTP id hf10csp98867vqb; Wed, 4 Oct 2023 05:36:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6IBPJBZUl6zs9P2Ih10N+c3+yX2Cb9QkU/4Fdq97ouse0sQEpx+5Km7dDh/UY/+V+BgMf X-Received: by 2002:a05:6870:829e:b0:1d6:6430:f743 with SMTP id q30-20020a056870829e00b001d66430f743mr2482851oae.19.1696423019075; Wed, 04 Oct 2023 05:36:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696423019; cv=none; d=google.com; s=arc-20160816; b=xOPVuyUaXrao5jhdSswTFvP3nlyW8wZKfBvTH9OzOIhM7pH5R5LON2LSTLDpLQOQwy FyJgfEyQ8L4wGXrH0Cd/eZAfHlmpPiFqEpjudyAsXLFI6tPjCoSGvLamQaVCZJOpmVr/ 1tBmbm9BuBjcmolwQ42JjX3yyRvKlguub/qGTkuPEO4YNXt4b7TIVbywUpJMJGStk1xi EAnBJBR8LuB0GbNoar4pej/MCAb6dKDUEN1xWOQqLssMmK+ZY1a5mWx5UZPwnfJDKCib JjfyFaDmalPX+6YQ7vupr4j6VJjbUjPvLEQyWR9dFs1rdj3JsHunssY5kSyG+17QrdiA gruA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=IZwoSRxYo/DnJ3NmzmXhlWk3yyY83AKWWYsGpaNej88=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=bYp8WCnVJ2KdHsqhEgJS7wbBwEB47DDLKBqb+kFpM/cNwqu0XiC0xLhfYM0IsFDlev +2xdlnfqhNPqDxYUZrF7/ySfd7djMWC+05FltleVnh6efbeyYzT44NEuKzfqlLrlhgTc ObCML9F+cZJf+JOnaeld+1YfZXFIFDN8qQHOHGX8+RcZLBXxW1jLiz5URCQZZcFypf8s KckBEcRhgZnnaj6au7W4ZCwVI7owekcoib7sMHC7K8bntWd3vgOnvDYyY+DR5MK/DF1R IAMf+w1kGUw+2dQ8Xkp5WPR0JGUgJ6a+ffoXETQuSf5cfzXEqa0NQ82/3vl4NwSmyjua l3tA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BXhT85Wl; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id t18-20020a634452000000b00564a5b8937asi3539157pgk.341.2023.10.04.05.36.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Oct 2023 05:36:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=BXhT85Wl; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9D4CB82164FA; Wed, 4 Oct 2023 05:36:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242405AbjJDMgj (ORCPT + 18 others); Wed, 4 Oct 2023 08:36:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242410AbjJDMfr (ORCPT ); Wed, 4 Oct 2023 08:35:47 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C073993 for ; Wed, 4 Oct 2023 05:35:41 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1696422940; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IZwoSRxYo/DnJ3NmzmXhlWk3yyY83AKWWYsGpaNej88=; b=BXhT85WlE/18L9u56BGfjgAAOkwHX5JCaEMQx4lC9j0puuoPuDR04tgsOiDlXunrZUw+9+ Qr+vL69wkKnPSIcnd2oxv5gsk2IAWjX4SbS8SNcxFE71zi5b+UVUNE2vntnIeszVL/UpaK nWVu41HvSkjM7Z/6RHtas41GZT6HjQ/D6kAGyZL4+whCQrOn4qRr1/m/ZxfBoPEINfxTKL e9JFXK3sOZ/vAA8UqhwCk+T0aw9JX0nf2Fq9jjo5cPnTeL3BkrByXx9fUXNiYbfhr3fgJJ WXqtr/sg76qRl3cUVb4QHB6F5+QVjI0EZ3hYj4Tcy8b/PqvPDTI1Iw+lV/BHjA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1696422940; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IZwoSRxYo/DnJ3NmzmXhlWk3yyY83AKWWYsGpaNej88=; b=qklfcoV9vnas8DmKEWPa4DSqI/PoyccP6myaCnLFsW16I4hDXUJlsSrZjYxnD5ODCuNqi3 6b4BdL/2H4jaBsBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v8 25/25] timer: Always queue timers on the local CPU Date: Wed, 4 Oct 2023 14:34:54 +0200 Message-Id: <20231004123454.15691-26-anna-maria@linutronix.de> In-Reply-To: <20231004123454.15691-1-anna-maria@linutronix.de> References: <20231004123454.15691-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 04 Oct 2023 05:36:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778828464057167627 X-GMAIL-MSGID: 1778828464057167627 The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the separate storage and eventually pulled at expiry time to a remote CPU. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- v6: - Update TIMER_PINNED flag description. v5: - Move WARN_ONCE() in add_timer_on() into a previous patch - Fold crystallball magic related hunks into this patch v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 14 ++++---------- kernel/time/timer.c | 42 ++++++++++++++++++++---------------------- 2 files changed, 24 insertions(+), 32 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 6f96661480dd..c39ae2ad0035 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -50,16 +50,10 @@ struct timer_list { * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. - * - * Note: Because enqueuing of timers can migrate the timer from one - * CPU to another, pinned timers are not guaranteed to stay on the - * initialy selected CPU. They move to the CPU on which the enqueue - * function is invoked via mod_timer() or add_timer(). If the timer - * should be placed on a particular CPU, then add_timer_on() has to be - * used. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the + * timer was enqueued. When a particular CPU is required, add_timer_on() + * has to be used. Enqueue via mod_timer() and add_timer() is always done + * on the local CPU. */ #define TIMER_CPUMASK 0x0003FFFF #define TIMER_MIGRATING 0x00040000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 331c59c0789a..feb915642457 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle && timer->flags & TIMER_PINNED) wake_up_nohz_cpu(base->cpu); } @@ -941,17 +944,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void __forward_timer_base(struct timer_base *base, unsigned long basej) { @@ -1106,7 +1098,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -2256,8 +2248,14 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) */ } - /* We need to mark both bases in sync */ - base_local->is_idle = base_global->is_idle = *idle; + /* + * base->is_idle information is required to wakeup a idle CPU when + * a new timer was enqueued. Only pinned timers could be enqueued + * remotely into a idle base. Therefore do maintain only + * base_local->is_idle information and ignore base_global->is_idle + * information. + */ + base_local->is_idle = *idle; raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); @@ -2273,13 +2271,13 @@ u64 timer_set_idle(unsigned long basej, u64 basem, bool *idle) void timer_clear_idle(void) { /* - * We do this unlocked. The worst outcome is a remote enqueue sending - * a pointless IPI, but taking the lock would just make the window for - * sending the IPI a few instructions smaller for the cost of taking - * the lock in the exit from idle path. + * We do this unlocked. The worst outcome is a remote pinned timer + * enqueue sending a pointless IPI, but taking the lock would just + * make the window for sending the IPI a few instructions smaller + * for the cost of taking the lock in the exit from idle + * path. Required for BASE_LOCAL only. */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); - __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); /* Activate without holding the timer_base->lock */ tmigr_cpu_activate();