From patchwork Fri Dec 1 09:26:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172339 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991424vqy; Fri, 1 Dec 2023 01:27:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IElGRuwq7Ga3L0flINzmiDUun9RqLI2Oey9jQ+yjyJxu8g6tQrYtWl9RiOuspYBFFSCCNm2 X-Received: by 2002:a05:6830:344a:b0:6d8:16e3:c50b with SMTP id b10-20020a056830344a00b006d816e3c50bmr2651667otu.33.1701422834824; Fri, 01 Dec 2023 01:27:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422834; cv=none; d=google.com; s=arc-20160816; b=yLSz2sNVnI+1uWrkMLQPtHTzGDkvBQZzCbDCWcTKYEjWOjf4B162RJivSF6pG2nF7+ iabEFPxOkmUE4JikJep/0J/Ecfkx3ADFLr1I42ocDqGI7yTwq7SYy3rAlaaMLH9mbNHT iZN0eiqLCaT2mgVCXjfhVXBS2K4+MqAMX98xg4vDRHjyv0cu8vUfpklmKkJAWtd58YCG e3SwrIRWWOw7hn0iNILnOZ4nhRy7+HqWZ4Xu36ntmTGcFl2/Jz43QL/Mw44rUeUF3MCu K7Js9iCudi2XfeQQGPXwuvOksI6+uq/vKq8I7hSE49B71FTtDcGpz7ZEJIfHWAAcBNl3 T9Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=Q6FUfFnmqt4IfzZcGlQkVi9fkygYWanokVt8epCh18g=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=P4v8h8LYJA/GQeJNeewoqa3fZQQ+yT64QARaxlZYAEffTz3eT+gJ37lt7CNLbV7tOB 4LvhUJHq1GkklZScre38Vubu1KjdwFbAeCRa3vmM/a7KdFSpO2MI/FLIQUBO0b1vmH9C 73mt4IhRguoJAdYa9J08gwFsPwUGbi5ePcn9m4r2GzmCvMVp+Bn+MF/0Ix1dISBu4ai5 v5Mgaux6fT6IIeXoEXlGArLRtMH7I9E+VY8sbQZ8oHwd7ZJLohLYAc0SiWzdvlnadtTN CJiX2erV7uMcWlRIeQYWQHJ48xypKce9cBEf1imB4BNlQSvUbea8lHPL7qDbK3yvPi0Q +CCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lgvXqUjF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=swIBVCbe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id ca6-20020a056a02068600b005c63a36938fsi1932194pgb.809.2023.12.01.01.27.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lgvXqUjF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=swIBVCbe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 0712D81552C5; Fri, 1 Dec 2023 01:27:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378013AbjLAJ1D (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229455AbjLAJ1B (ORCPT ); Fri, 1 Dec 2023 04:27:01 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75009194 for ; Fri, 1 Dec 2023 01:27:08 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q6FUfFnmqt4IfzZcGlQkVi9fkygYWanokVt8epCh18g=; b=lgvXqUjFJoLwU9+ZZLvtquxTwT6ofi+2y79zdhzkCm+arupJRiVKmMIVcB3lduxwWCFofb MFOL7JDx6+6MFnMmEPZRzTDdRIidDu055pueLPcpiGoOrPv1g40SAikE1AQ96Zg1aCUWsu /RfyywAtAOETUaaOeWDUdTjDg1K5i8v5TCoy1b8UAilHhdNHc0/3AnPNkftGpd67MIalU9 PXMyPG5DMKfCOgRGIYO2Vtiq31A/P7fOfozAca61lrAgMbutkFYtbzGQ38QYnXCUZcDprj jhoN1vY2o7ZgxvN6IUOgYw/d4bOtBFlACxjARMnb88bChF6B1h2PGlbSvNVcnQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q6FUfFnmqt4IfzZcGlQkVi9fkygYWanokVt8epCh18g=; b=swIBVCbe9PNsC1cAlRdY84Lq5DKzwDPoPs3oPMIaRGhQn1B3L/aGAyxpp9foQnhZvbP1Cq ixM6qqPcxQvkLZBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 01/32] tick-sched: Fix function names in comments Date: Fri, 1 Dec 2023 10:26:23 +0100 Message-Id: <20231201092654.34614-2-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071150561242093 X-GMAIL-MSGID: 1784071150561242093 When referencing functions in comments, it might be helpful to use full function names (including the prefix) to be able to find it when grepping. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: New cleanup patch --- kernel/time/tick-sched.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index be77b021e5d6..5c28cc80fd25 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -920,11 +920,11 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) } /* - * nohz_stop_sched_tick() can be called several times before - * nohz_restart_sched_tick() is called. This happens when - * interrupts arrive which do not cause a reschedule. In the - * first call we save the current tick time, so we can restart - * the scheduler tick in nohz_restart_sched_tick(). + * tick_nohz_stop_sched_tick() can be called several times before + * tick_nohz_restart_sched_tick() is called. This happens when + * interrupts arrive which do not cause a reschedule. In the first call + * we save the current tick time, so we can restart the scheduler tick + * in tick_nohz_restart_sched_tick(). */ if (!ts->tick_stopped) { calc_load_nohz_start(); From patchwork Fri Dec 1 09:26:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172340 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991473vqy; Fri, 1 Dec 2023 01:27:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IEIhphlR213AxuEsMRu7xcH+3g71DTOxB8Goz/3zbvy1zfxMIOJDPk6xppslk112gNP5+ur X-Received: by 2002:a17:903:2444:b0:1cf:da41:729c with SMTP id l4-20020a170903244400b001cfda41729cmr17749851pls.56.1701422842632; Fri, 01 Dec 2023 01:27:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422842; cv=none; d=google.com; s=arc-20160816; b=O9y5/unDrZSGaT8w2iXqajgVykQaRFFLzWZSdUpRI/JhBRq4XbQq8646keiZNdMWSK J/N03ziMKva3Ph4Gx6c+gp+L0igBlRcq2Pzw7HZfSEzSUEde8v20Cn1trNQ9YcKwKaB3 79pyIXFSep12PkRYnWjV38ltXX0hZzZ6pDxovQg+Sm4nu6jZz42NXO7m8psakcw+6FQ5 jLVO41xFCCHfYMUSaUbnMMcc3YcSsRcG28sCWtuaBZPgyua3Yoaod811NQSVPPvJrxGE UeDbEenLDMr4ed0ISS2uRBRzNVWxY4tGG11G7NXXmiJ0zdvc9/i/R+9GVdobWCo3//vx 9+NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=jl8/xzU+GiuSpXdMfzNJncPixT2cuPm0ChHCjye17qI=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=G4eBSHSAMEofGcf6G0y0cronMUAWbKljOmFRJ/TaGjJS4DgwP/7KauO9K/bedh2fbd R5FOD3wtGByLA8fEBpEWyCPwRCvNpshSm4cy8VHa5IU2ZNXflCV84qW/kPb6yo17XSOO jvaJMQpE5dlxlPdfmUCSZWUkEGlm8hxsIMCTUzAtVRB07CameYcUvVnhHyaX11BYNUmz HVejz7ZBYy1ODKtpw54KW9FIhixl+OkwDaprP1HocndFgjHoAx42k4cJRvQOgGW1Pao3 g0LWqnpuhEj5piKGE+ELa/Qq2p+vyJeDG5O1ShlmG/xHa0jn9FKP55hfft9gU940Bq0t 1HFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PRcshUei; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id jk17-20020a170903331100b001d055315494si761018plb.540.2023.12.01.01.27.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PRcshUei; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 1C5BE879439D; Fri, 1 Dec 2023 01:27:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378027AbjLAJ1F (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378004AbjLAJ1C (ORCPT ); Fri, 1 Dec 2023 04:27:02 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E703D10F3 for ; Fri, 1 Dec 2023 01:27:08 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jl8/xzU+GiuSpXdMfzNJncPixT2cuPm0ChHCjye17qI=; b=PRcshUeiqQBP6Ypl/4igU9gnAZax/WZ9o6XJmRkijSQlV8N4QjPU98SYISHBREqA75uWZe aXLpTqOnLIMMSJtBYXQmc9vf6PUTFIdnVOz1ssiOQTEDoFR/CgOrgEXe6urDrJO3mAyMOZ CnXf0fRTJBCt/jCRZ09/255zi7frJtW4JTzBXU4m3SuzthrUHV2wDxvgqhxrl/ebelKOPs 8+OOkyFutkDdkDCpN3W0Cap8wApvWQSwQCdI0nIAKfgUSjYLhsPGYaaWgOTaY6tPN0rP28 fpvOInjCw9cXNbhxFLhb7yxte3vgMP/miUKuEACWllOZLXZawL5HAxaL+esNsA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jl8/xzU+GiuSpXdMfzNJncPixT2cuPm0ChHCjye17qI=; b=Sb+rjhnebJJIO1g9BgSS4qyOLxH+tt5Barqeak47ZCMSmExPUqtDHBEroBlyuhU1DJydwt x7ySNlMzzhm9QdCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 02/32] tick/sched: Cleanup confusing variables Date: Fri, 1 Dec 2023 10:26:24 +0100 Message-Id: <20231201092654.34614-3-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:18 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071158628802024 X-GMAIL-MSGID: 1784071158628802024 tick_nohz_stop_tick() contains the expires (u64 variable) and tick (ktime_t) variable. In the beginning the value of expires is written to tick. Afterwards none of the variables is changed. They are only used for checks. Drop the not required variable tick and use always expires instead. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Fix typo in commit message --- kernel/time/tick-sched.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 5c28cc80fd25..89517cfb6510 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -887,7 +887,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); u64 basemono = ts->timer_expires_base; u64 expires = ts->timer_expires; - ktime_t tick = expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; @@ -910,7 +909,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) /* Skip reprogram of event if it's not changed */ if (ts->tick_stopped && (expires == ts->next_tick)) { /* Sanity check: make sure clockevent is actually programmed */ - if (tick == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) + if (expires == KTIME_MAX || ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) return; WARN_ON_ONCE(1); @@ -935,7 +934,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) trace_tick_stop(1, TICK_DEP_MASK_NONE); } - ts->next_tick = tick; + ts->next_tick = expires; /* * If the expiration time == KTIME_MAX, then we simply stop @@ -950,11 +949,11 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) } if (ts->nohz_mode == NOHZ_MODE_HIGHRES) { - hrtimer_start(&ts->sched_timer, tick, + hrtimer_start(&ts->sched_timer, expires, HRTIMER_MODE_ABS_PINNED_HARD); } else { - hrtimer_set_expires(&ts->sched_timer, tick); - tick_program_event(tick, 1); + hrtimer_set_expires(&ts->sched_timer, expires); + tick_program_event(expires, 1); } } From patchwork Fri Dec 1 09:26:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172341 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991486vqy; Fri, 1 Dec 2023 01:27:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IFPNo0yajOiJNbffPZGtLueaGUn1lMZrf7EOQXDkQ8/YlrHihA72moFH1VzvfnwPySzjL/2 X-Received: by 2002:a9d:67cd:0:b0:6bd:603:797f with SMTP id c13-20020a9d67cd000000b006bd0603797fmr2643080otn.37.1701422845114; Fri, 01 Dec 2023 01:27:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422845; cv=none; d=google.com; s=arc-20160816; b=MjLwj56eSagzaQOAFoi5oMa3B/ADj0wnwJ4EOeKU3vkQFwLhUD5mDpJ+NyZZlgYAV9 yCScJiYCWYb5ZucO8ws/gQxR8FNggekcou4LqUjNMVDhmQq1vD8aGdewvigRGsVMBbby vmEdWfhv6UzazmR+MNQCwuC+HNfEFvwRBD+nFseIg2dBp8UngaZxQfeQ1A4xveIUyxuC dqkAursMd9eNRG/GoTn2Q0cm2oWTxLWcbChiiYec6qLfZ3m8fJDXsnCdHc0wolMlmnuB k1jcos0BnhiXthyDeLuBPx9BNFyBGq405GFMee9AwpKyoVMSk9KvfvkSt8aEfJiY+0Ra +c+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=TpL4QHNJmRbyjNaiO3+z9ClPjQHijZRgI0ccvQpegrw=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=b5pp/ZaiqpXaHqgVA+E2j86r8y96veyaFnhTsW2Sbwywd/xqZofG+gdwJvOKZALDvC UnC5/+10oUP0qFMOISEpvgCUZG1Z2MmfST1W3XfKaSYqhCiQ73CJxSR4YeL8txHfIH/b nrCtfS61NBzA4TH0W5MeA8MFJ/PbZo5K+hlREmYYcuvmfYA/OwOrIj2uSTytQj/SfE/h Nz/HqW2q3pGd24WGya859Fd6SKBdAs/NWHqG1PgFK/dn6MJ0JHeH495B8M1jzaZCxYFb ELQvINg0Re9b+/rsij2nH0rnfxXi6D/L3JVR+fhDLuGh28dxz6JMlc7UYHq41/Q2rPgU aq/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UJJq6noc; dkim=neutral (no key) header.i=@linutronix.de header.b="iUF3pD/b"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id q78-20020a632a51000000b005bd3da15474si3034582pgq.839.2023.12.01.01.27.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UJJq6noc; dkim=neutral (no key) header.i=@linutronix.de header.b="iUF3pD/b"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 222DE8339399; Fri, 1 Dec 2023 01:27:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378038AbjLAJ1I (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378020AbjLAJ1D (ORCPT ); Fri, 1 Dec 2023 04:27:03 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A980E194 for ; Fri, 1 Dec 2023 01:27:09 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TpL4QHNJmRbyjNaiO3+z9ClPjQHijZRgI0ccvQpegrw=; b=UJJq6nocAW/HiAnWK8GyTPEfN1oOPO+9lXJFr9C7HGZTAE0APcdz0FcSis+Nc63mNgmw7f DQPc5vLiU1FzKvIj7a/2povI1jdnoqtwEcyQ4eoRRnp+/slR/Hg9n99bO/DdzE97DLkd42 hzRcSz+mTyBFBysF7BFmGGc2ExkPoKocraYbLNw8ngI0NZCg+akvnglvPjX9CmYUNTqa0i Z+SVu3HynyVTQZWf/Bm6/K2wD5JvHi04S7bTy8ANUEXKTMjAfXPp71O+NVOGHyTBwtWZ62 Jt2ZkJr4otEraNlGF8lGS4zufaLNY/yoA57SyGoui/zyn0ZZgM4oNgQ8y6fvlQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TpL4QHNJmRbyjNaiO3+z9ClPjQHijZRgI0ccvQpegrw=; b=iUF3pD/b/ZHxN9ryp4qCxj+pHqQ1FCpLeP6M2yWc0rOxGEeo31fEwzQLExz/7zDZSZEI2g JFxHUEpcXyBZvzCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 03/32] tick-sched: Warn when next tick seems to be in the past Date: Fri, 1 Dec 2023 10:26:25 +0100 Message-Id: <20231201092654.34614-4-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071161290723538 X-GMAIL-MSGID: 1784071161290723538 When the next tick is in the past, the delta between basemono and the next tick gets negativ. But the next tick should never be in the past. The negative effect of a wrong next tick might be a stop of the tick and timers might expire late. To prevent expensive debugging when changing underlying code, add a WARN_ON_ONCE into this code path. To prevent complete misbehaviour, also reset next_tick to basemono in this case. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Add reset of next_tick to basemono --- kernel/time/tick-sched.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 89517cfb6510..b1b591de781e 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -839,6 +839,10 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) ts->next_timer = next_tick; } + /* Make sure next_tick is never before basemono! */ + if (WARN_ON_ONCE(basemono > next_tick)) + next_tick = basemono; + /* * If the tick is due in the next period, keep it ticking or * force prod the timer. From patchwork Fri Dec 1 09:26:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172342 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991531vqy; Fri, 1 Dec 2023 01:27:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IHpmG6FV1YlxrK5B00CHkISUuQDhlsvK0a/HkJcYRyDK8DIZgpPgeTmiVSLZe4us8Kai+0Q X-Received: by 2002:a05:6a00:8d93:b0:6cd:e5be:83a6 with SMTP id im19-20020a056a008d9300b006cde5be83a6mr5859191pfb.32.1701422851254; Fri, 01 Dec 2023 01:27:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422851; cv=none; d=google.com; s=arc-20160816; b=tuZHld4/miAYPvO+TrecM5dinsjblW1jsUMk8bUMhkKwGV9AwalTQe2Yom3WJZY8k+ 2fspywXB9Ztx0HNmfqO8RCj10HfnlLobImDIytJu3q2ucPeD+ZtS0cWDrX9RpwSyaBWH yBNvBxvyCDjl96eJnV5ztO3K1dgsLBm1bsjhPrrLPRc+PFpbiCvsPTwsulMA0b27yEQU SpVZ9YCu6MBqedYnSGBYca77vplDFIiZ0kS3v0PLPgNR7fXWTk776nI+S8hAE33zf8wW bklIqNsUc1NkBZ/R7P1CnU+zFRhCk7jdkH+g2mQUjyY7bEDJfrd7zeKMk0edQn3UtcVw rtqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=EuZUVPY6DwMtX6TE4Diotf1s/Z9ALR3JOBYxJlQlrWo=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=IGIMoBAZakdwZU9/WkPCKyaWTtm5YumeyEwNZqhlyawv8Ioe/o2HKEHogpkTpLpN+t fIF/EXOU7nwgu9oP8tYPnsMozpocdk5FdGXsQDAP7ia77g8iW0aWLWw6M1Xt8V6bNrUO p0bbaGVwSYc76F9Dd5tDn4qnXXJN7y/4TIyhbqRNMoz27csOo4bvkT1qAheBJMWSL4YB 96UPGxjj1rdx6WrTOZNE7U+BMG0GtnHpSH2zy+BRT8M3WQ6T5XegkJjKVsDqdwFhxc3H axU0mKKMcJoQ5Z5Qi0s2AysXSjPwpxwtqOrTEud4iMG/oKAmw3+FIsZz3yOZ8T9BovD2 oZ6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=OrC8zJ88; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id p11-20020a056a000b4b00b006cb8abd39d3si843316pfo.180.2023.12.01.01.27.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=OrC8zJ88; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 11D1F80EA8E3; Fri, 1 Dec 2023 01:27:30 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378051AbjLAJ1N (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378022AbjLAJ1E (ORCPT ); Fri, 1 Dec 2023 04:27:04 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E61A10F9 for ; Fri, 1 Dec 2023 01:27:10 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuZUVPY6DwMtX6TE4Diotf1s/Z9ALR3JOBYxJlQlrWo=; b=OrC8zJ88T958APlVyh4kv8Y7X5oRZogAMT9iZFq2jQjoc72FPD8+2eqTlL+ycyJp5uoy5l 3odpYGSoVRz5gOZc3hXdLsfql3bkwEFa/2zEDSpsyfIFvvvxxqEZFmjBeWF+Jr7L72PLnS FvVXfQ26Gi46CvgV/kOtMvpQR57EHIzfxazh71tMy3Bg5YX38DelZ+j5ORZOvg8PbycxVF j3dWdVvRgCkkDrhD/IRT959lpQGbfkPIeSZo9bczNh3QxvTdSAHLnyYPrVF/8GO85T6psE On2U2rnGHPTRScX51/upA2x9kRrpgW2hF4aN7+VFa3SOfqaRuuYbztZrmfHTlA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EuZUVPY6DwMtX6TE4Diotf1s/Z9ALR3JOBYxJlQlrWo=; b=iu2NM4FKvnMPiISGNsxUiZzqNgZKsu9vKdvxqy/lJSsO8i3PpLvuA/jj5DgC+UXyMKkZjP y9PPX1zlI6jeCzDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 04/32] tracing/timers: Enhance timer_start tracepoint Date: Fri, 1 Dec 2023 10:26:26 +0100 Message-Id: <20231201092654.34614-5-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:30 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071167821689879 X-GMAIL-MSGID: 1784071167821689879 For starting a timer, the timer is enqueued into a bucket of the timer wheel. The bucket expiry is the defacto expiry of the timer but it is not equal the timer expiry because of increasing granularity when bucket is in a higher level of the wheel. To be able to figure out in a trace whether a timer expired in time or not, the bucket expiry time is required as well. Add bucket expiry time to the timer_start tracepoint and thereby simplify the arguments. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: New in v9 --- include/trace/events/timer.h | 20 ++++++++++---------- kernel/time/timer.c | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h index b4bc2828fa09..99ada928d445 100644 --- a/include/trace/events/timer.h +++ b/include/trace/events/timer.h @@ -46,22 +46,21 @@ DEFINE_EVENT(timer_class, timer_init, /** * timer_start - called when the timer is started - * @timer: pointer to struct timer_list - * @expires: the timers expiry time - * @flags: the timers flags + * @timer: pointer to struct timer_list + * @bucket_expiry: the bucket expiry time */ TRACE_EVENT(timer_start, TP_PROTO(struct timer_list *timer, - unsigned long expires, - unsigned int flags), + unsigned long bucket_expiry), - TP_ARGS(timer, expires, flags), + TP_ARGS(timer, bucket_expiry), TP_STRUCT__entry( __field( void *, timer ) __field( void *, function ) __field( unsigned long, expires ) + __field( unsigned long, bucket_expiry ) __field( unsigned long, now ) __field( unsigned int, flags ) ), @@ -69,15 +68,16 @@ TRACE_EVENT(timer_start, TP_fast_assign( __entry->timer = timer; __entry->function = timer->function; - __entry->expires = expires; + __entry->expires = timer->expires; + __entry->bucket_expiry = bucket_expiry; __entry->now = jiffies; - __entry->flags = flags; + __entry->flags = timer->flags; ), - TP_printk("timer=%p function=%ps expires=%lu [timeout=%ld] cpu=%u idx=%u flags=%s", + TP_printk("timer=%p function=%ps expires=%lu [timeout=%ld] bucket_expiry=%lu cpu=%u idx=%u flags=%s", __entry->timer, __entry->function, __entry->expires, (long)__entry->expires - __entry->now, - __entry->flags & TIMER_CPUMASK, + __entry->bucket_expiry, __entry->flags & TIMER_CPUMASK, __entry->flags >> TIMER_ARRAYSHIFT, decode_timer_flags(__entry->flags & TIMER_TRACE_FLAGMASK)) ); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 63a8ce7177dd..a81d793a43d0 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -606,7 +606,7 @@ static void enqueue_timer(struct timer_base *base, struct timer_list *timer, __set_bit(idx, base->pending_map); timer_set_idx(timer, idx); - trace_timer_start(timer, timer->expires, timer->flags); + trace_timer_start(timer, bucket_expiry); /* * Check whether this is the new first expiring timer. The From patchwork Fri Dec 1 09:26:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172347 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991668vqy; Fri, 1 Dec 2023 01:27:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IFdW+QsXpGgeYLq/iO1zHdyTRrfZhP5TYXo9B7VSAgnDBjVmNsnDT8IB/p0xB4AjSeOi6kx X-Received: by 2002:a05:6358:9141:b0:16d:b577:e9f1 with SMTP id r1-20020a056358914100b0016db577e9f1mr23706496rwr.9.1701422870235; Fri, 01 Dec 2023 01:27:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422870; cv=none; d=google.com; s=arc-20160816; b=Y+FHgu01qzTYijdU300572wvPpoqUgc1tTR2Hsn0snxQ/8ubQkbIAe96X3qnJcJIOv YSX8q7NFqNzSvDMpGi01N6MtL6HgmWpYgKEnfmGF/byHofXCUZSlXd1b7Ga+qk2g4HZn yEmAz3rlVyqqEVzu8zCuDgHzdIbFyOHpPur0LdslrmsihQiKtN8iV0SibKFk8fqB5Jqa 3/CMjfIKGZTBVgU6igRfhqeRkh9vnCsMVvwuphx9YRG712Uc3t0h0kbenjMEJTjRZzTx 3ExQheYAOnLg6tRBSUZHMMngoeqLATkBvWB/TrGGA1QlAr/+zcXB5xJphR3Qp+Ukrj74 pnhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=x5NV8oLmppm4cChtNeoTUNwitHPz66rl4TtkJG1MsVE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=Dv+y6TiLaQpLYfH29/XbOqIto/z+g2E+aynRjdvRmH/Bgc9i9RWOtfsm1bVYrOKnZR opTkmyYsOqSOaSo36HArzbCP8FCi6MqCVuCR1yqxE1Q4DJGb4mYQ3UWQGJWSuUxc7igQ otS7ESoIbR0b0jpyiionYvdy5sO3f1DzjNaYkKprXLX/FC7+YxW39M4rJS/KVj9v4znq FvMBq2ZzgQQU2EJT3/ySQE7qzvFRNigNlqoq6ewsKhZQx5XMrK+yjUt67t72ltSj8wv3 cgZukVRkB2Y24YQOtov4zPQ7mnhNp5g+FFbld7IsmFWbKg3ov/KtO0O3uh4IwLiwycRV wyEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ACJG9BLn; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="lV7jqkG/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id e20-20020a656794000000b005ab7b51ab6csi3000969pgr.515.2023.12.01.01.27.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ACJG9BLn; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b="lV7jqkG/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 233E180BD4C6; Fri, 1 Dec 2023 01:27:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378056AbjLAJ1S (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378024AbjLAJ1E (ORCPT ); Fri, 1 Dec 2023 04:27:04 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E533510F3 for ; Fri, 1 Dec 2023 01:27:10 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x5NV8oLmppm4cChtNeoTUNwitHPz66rl4TtkJG1MsVE=; b=ACJG9BLnPweNxaHJJ9w1pbi+SRth6MXk+ue7mImxGjl0rf06m9//BXNESvnefcAi865eVp HYPHH88orIdDbxTxm6maC9Z7zjntcMuoGw+mucvviLDTWAizOZTFZb3cR0lB9l9VR/YNZ2 XThDs7w8kH85z4y9wuHUBDLuV04+NHpj5jZFQwuLoD1AFW0uGzvPXdi0y5cYapX6LTZk+6 sRTA+PR1Tq1vVNVKY5wlJ92DCMEjLBV6bANERRrmFxSDeOraYXvp1AI2n+5WJao2xPnuyI wNBYPxJiugPji/XfpCYlL5YpmVuIMRE2jvgKEs6sgx+JgsjcFf/iMB+WUJNczg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x5NV8oLmppm4cChtNeoTUNwitHPz66rl4TtkJG1MsVE=; b=lV7jqkG/73jefeFGIVa0cvbImUlcDQjpHBuRgRZzzHojWnwJUlYoDZ+Hr85NjGiyrNZEyE 1qQTNsPXqXlEXZCQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 05/32] tracing/timers: Add tracepoint for tracking timer base is_idle flag Date: Fri, 1 Dec 2023 10:26:27 +0100 Message-Id: <20231201092654.34614-6-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:43 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071187560015896 X-GMAIL-MSGID: 1784071187560015896 When debugging timer code the timer tracepoints are very important. There is no tracepoint when the is_idle flag of the timer base changes. Instead of always adding manually trace_printk(), add tracepoints which can be easily enabled whenever required. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: New in v9 --- include/trace/events/timer.h | 20 ++++++++++++++++++++ kernel/time/timer.c | 2 ++ 2 files changed, 22 insertions(+) diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h index 99ada928d445..1ef58a04fc57 100644 --- a/include/trace/events/timer.h +++ b/include/trace/events/timer.h @@ -142,6 +142,26 @@ DEFINE_EVENT(timer_class, timer_cancel, TP_ARGS(timer) ); +TRACE_EVENT(timer_base_idle, + + TP_PROTO(bool is_idle, unsigned int cpu), + + TP_ARGS(is_idle, cpu), + + TP_STRUCT__entry( + __field( bool, is_idle ) + __field( unsigned int, cpu ) + ), + + TP_fast_assign( + __entry->is_idle = is_idle; + __entry->cpu = cpu; + ), + + TP_printk("is_idle=%d cpu=%d", + __entry->is_idle, __entry->cpu) +); + #define decode_clockid(type) \ __print_symbolic(type, \ { CLOCK_REALTIME, "CLOCK_REALTIME" }, \ diff --git a/kernel/time/timer.c b/kernel/time/timer.c index a81d793a43d0..46a9b96a3976 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1964,6 +1964,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if ((expires - basem) > TICK_NSEC) base->is_idle = true; } + trace_timer_base_idle(base->is_idle, base->cpu); raw_spin_unlock(&base->lock); return cmp_next_hrtimer_event(basem, expires); @@ -1985,6 +1986,7 @@ void timer_clear_idle(void) * the lock in the exit from idle path. */ base->is_idle = false; + trace_timer_base_idle(0, smp_processor_id()); } #endif From patchwork Fri Dec 1 09:26:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172343 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991591vqy; Fri, 1 Dec 2023 01:27:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IGlmNnuOuO1SC/IZ9PYfEItZnVD75LqXgaRY94Hg0vFDwg1clYh5ApFOORevouYhlBj7CL4 X-Received: by 2002:a17:903:647:b0:1d0:5247:e505 with SMTP id kh7-20020a170903064700b001d05247e505mr1786318plb.14.1701422859800; Fri, 01 Dec 2023 01:27:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422859; cv=none; d=google.com; s=arc-20160816; b=OrC7u7e1Hf+I2KqbRRu0L/LkwuZrfvU6OECEa03ZV0+2k7KxMQT9Tj/+hnnZKFpuEM JAMgL16F6ogHSpmB3orrGWInkY8RVpKDR0qZjflc9qnVfTU9azv50PIVj9X6CW6PDTJj 1GXvNtSCENofeSaAZ6PGvstoWOaatujEbnp1FyMf2s/Ig1sQVXMT9jNn7IPtf+SKlZcI Bz6EAoJlaTWG0j0JTkQmVaKkuVmCaXEgM7hKkvcYaUFhdEFuusJxAOwaY9MoafSL0o5d JL1g5Mtw/yi68Y41bIUovFRciMB/l20vadkN14cuW0sNHiVzxmi05//piEU2Hu/P58Vy r/cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=l1TO/DAtk+CCicSFDRPTxsBfrzcdsd2AUHicFvATynM=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=b89AvmbMKG1Y+Gt+gDI9fP16ZY/DNVV2Bcb6uluUiirF6b8XOUiCNYl9PbQL4PhpUM pjMN5b89f47c15Xo0zaJmf7QWZntyurURAqyrouBtjpHsDB2bLGnYI64rCfO9O1oshcg rgmSiaJVPz8hDCLmAZZGYQYHHo0UHDhythpb5S9Q+eHoXyOBIsqHa+9QRTLaVaDbZ78o mPphqCkHhyvxJHCb+/BOMeitCAcS1dQCpj/jCSPtqzbZSIEkL+TFgkle1Ec67/WIFQmV tMmyplUZBeOdMNs98lbpk8fOcIGRwjZdI8W/hTd3B8aeUxdigv5F2+Hpfa8X52IFY8Is Di8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lum7zhuf; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=H7J7pHY4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id b7-20020a655cc7000000b005c6034ba3f4si3087905pgt.419.2023.12.01.01.27.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lum7zhuf; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=H7J7pHY4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 81E8180ECF32; Fri, 1 Dec 2023 01:27:38 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378044AbjLAJ1W (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378026AbjLAJ1F (ORCPT ); Fri, 1 Dec 2023 04:27:05 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96970A0 for ; Fri, 1 Dec 2023 01:27:11 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l1TO/DAtk+CCicSFDRPTxsBfrzcdsd2AUHicFvATynM=; b=lum7zhufPRoUPgr02L0+nr6ewMLlA4ZRnPpDBOK7AYfwxMTiURnHboUx9eXPXfeoKFWlWj 6K+fqA2/enXeTY+rxTb1bTvUDQYkIxGrDrSeYCwkD//wVE4rieb8koOh1FJ9VKWKxJwN8c q5GIv3tz1v6iJiiz2G8tevShutyRoQfx3C46SmF+O6uXo3GD6xEn8IhesLlwiSlwRNdOqM FawdCuytlVmtP3sAwki1uQ19SKXShRHkMkfB05uFcoe2elWHOYpMt7SLi2qpM36+ZHZwOy k7zjR6RvzhBQMC/lMuSWkxYfSLELuEZ774PPMwGX9rXn+QzoqxVdTm8Aw805Kw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l1TO/DAtk+CCicSFDRPTxsBfrzcdsd2AUHicFvATynM=; b=H7J7pHY41+OQTTMvDQDwvpdtt26MF/g+9en13VW2NJaVstqWvRjxzwwVa7UwAS0FehJC09 PPxAiRB4LLN99jBQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 06/32] timers: Do not IPI for deferrable timers Date: Fri, 1 Dec 2023 10:26:28 +0100 Message-Id: <20231201092654.34614-7-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:38 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071176682938173 X-GMAIL-MSGID: 1784071176682938173 Deferrable timers do not prevent CPU from going idle and are not taken into account on idle path. Sending an IPI to a remote CPU when a new first deferrable timer was enqueued will wake up the remote CPU but nothing will be done regarding the deferrable timers. Drop IPI completely when a new first deferrable timer was enqueued. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Only a typo fix v8: Update comment v6: new patch --- kernel/time/timer.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 46a9b96a3976..a6e31b09637c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -571,18 +571,15 @@ static int calc_wheel_index(unsigned long expires, unsigned long clk, static void trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) { - if (!is_timers_nohz_active()) - return; - /* - * TODO: This wants some optimizing similar to the code below, but we - * will do that when we switch from push to pull for deferrable timers. + * Deferrable timers do not prevent the CPU from entering dynticks and + * are not taken into account on the idle/nohz_full path. An IPI when a + * new deferrable timer is enqueued will wake up the remote CPU but + * nothing will be done with the deferrable timer base. Therefore skip + * the remote IPI for deferrable timers completely. */ - if (timer->flags & TIMER_DEFERRABLE) { - if (tick_nohz_full_cpu(base->cpu)) - wake_up_nohz_cpu(base->cpu); + if (!is_timers_nohz_active() || timer->flags & TIMER_DEFERRABLE) return; - } /* * We might have to IPI the remote CPU if the base is idle and the From patchwork Fri Dec 1 09:26:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172346 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991664vqy; Fri, 1 Dec 2023 01:27:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IE4Mz8Qd9DGuEDe68olRDFYXImQFXUqBDQSU8pjyEQSg2YdGXN1bu7fMWE1rmtjr1tlEjB6 X-Received: by 2002:a17:90b:a14:b0:286:424b:5be2 with SMTP id gg20-20020a17090b0a1400b00286424b5be2mr5087719pjb.21.1701422869422; Fri, 01 Dec 2023 01:27:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422869; cv=none; d=google.com; s=arc-20160816; b=A8dqZ5t8KFHrw6B5TFWlxvr/k+eXyc7vl2s3qAKTQk/PP2WUUEQeCHHsItuRfTTf65 bmiuIkap/N2eTSuENUmmgTxLGedg1uC8hPDHAstxdhp3ODcbmDDsTZIhl6E/BCoTcy1K ZNI9qsylDt0itfBtxCKs+6wPHTr/5EtfmD0zIsoJ9G6byQL7ngZwFUUk9oLRbCaVmq8U DLyebvBQpQLcUY616ia5uVWtr3OXUWkfpoQMUnSDyCG/gwe/hQ58/0zC9M7EWe0R+wfX 577zutE7I8nRSs4IVHgLHp0LRO5nvB5dyUG/UD9UZlHgV5N3+fuirilkXLK8iOoOgp67 itVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=abaveLygv3WKLQ70azOX5Nhmi35F03D2H7Exz6wvRjs=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=nBPLqTu/ukkb+0D+PbQjO22LTCSwH0Ny79nlU7nNxMxVy3vR4susRdApiViley6rS6 /22nFwvgq/6mcQHAZwD97l54ycN7Z+iKBQ1s9e4n6JEa73Gh2p/UtaQhP43FXlYVzRK3 vAZVUaO1wFpAqO/EdEY+w6Fx9ARIxPgNQknQEgwqXpR1OgY0oiXS+jIYHq5kwa/dzupW 0J2pacMeHupbix+DJdazLU6aPZUiEBiLX1LU0KVrC5x5t7ipVPCe70lVPf1iUFATe5YS jUi2EFErRNwZHk7Y1njCXtc/Vi3TKuj1nCYA/HOSllJLQc/u5DO5yog2fqVmkypzVuVw +hrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=a1nd9r7l; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id e3-20020a17090a4a0300b00285259acbcesi3097742pjh.142.2023.12.01.01.27.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=a1nd9r7l; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 04E4580BD4C4; Fri, 1 Dec 2023 01:27:42 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378087AbjLAJ13 (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378033AbjLAJ1G (ORCPT ); Fri, 1 Dec 2023 04:27:06 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 582C510F3 for ; Fri, 1 Dec 2023 01:27:12 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=abaveLygv3WKLQ70azOX5Nhmi35F03D2H7Exz6wvRjs=; b=a1nd9r7lEBC9ci6wUVUgi1hN4JCBZlGitekHspDsibrFdu+pl6/3sTyy+324ahw5Rz8T2d uZTYVbFq3MMn/Aork8Mx5jwoQm179AXqZBvu+n7AGwyQykOZik6lmJ1xXTsu7yTmC/HkdF 52j3JYYAPDW4Jg3KME89B0f6l7d6zprctDbQgkcff1pjV5mC/jwRLOLQoZ3LolluWMH1mH YXyx1z+xAybkeuZWQL3+Nc/UPifl2f29OmDqW1J2HjXXELtCOe0OrwmJDsD96TOeH86fTW Sx8OSCuD5mmbxt7QitqcraPf+7fQ+bdykkWeOR1RtVUiv6anLw2D4wUnaXBKgw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=abaveLygv3WKLQ70azOX5Nhmi35F03D2H7Exz6wvRjs=; b=e0qcp8TwrijRK7VdVHlEYhM6yUkMF56vK2jMfSzEgzTaI8OYcOICJciV0hJSwlLduf3n9/ MRWLn7kdMu364XBA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 07/32] timers: Move store of next event into __next_timer_interrupt() Date: Fri, 1 Dec 2023 10:26:29 +0100 Message-Id: <20231201092654.34614-8-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:42 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071186656646156 X-GMAIL-MSGID: 1784071186656646156 Both call sites of __next_timer_interrupt() store the return value directly in base->next_expiry. Move the store into __next_timer_interrupt() and to make its purpose more clear, rename the function to next_expiry_recalc(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- v9: Typo fix only v6: Fix typos in commit message and drop not required return as suggested by Peter Zijlstra v4: rename function as suggested by Frederic Weisbecker --- kernel/time/timer.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index a6e31b09637c..3ca706db1d20 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1800,8 +1800,10 @@ static int next_pending_bucket(struct timer_base *base, unsigned offset, /* * Search the first expiring timer in the various clock levels. Caller must * hold base->lock. + * + * Store next expiry time in base->next_expiry. */ -static unsigned long __next_timer_interrupt(struct timer_base *base) +static void next_expiry_recalc(struct timer_base *base) { unsigned long clk, next, adj; unsigned lvl, offset = 0; @@ -1867,10 +1869,9 @@ static unsigned long __next_timer_interrupt(struct timer_base *base) clk += adj; } + base->next_expiry = next; base->next_expiry_recalc = false; base->timers_pending = !(next == base->clk + NEXT_TIMER_MAX_DELTA); - - return next; } #ifdef CONFIG_NO_HZ_COMMON @@ -1930,7 +1931,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base->lock); if (base->next_expiry_recalc) - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); nextevt = base->next_expiry; /* @@ -2015,7 +2016,7 @@ static inline void __run_timers(struct timer_base *base) WARN_ON_ONCE(!levels && !base->next_expiry_recalc && base->timers_pending); base->clk++; - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); while (levels--) expire_timers(base, heads + levels); From patchwork Fri Dec 1 09:26:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172344 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991623vqy; Fri, 1 Dec 2023 01:27:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IE0UnfNITTHBPE5qZztkrCpz2MNHDF1OZiz/wkMYZXFc3ttXegi5kLt0cWouJfT0mmRxrdN X-Received: by 2002:a05:6a20:548e:b0:18c:8d0f:a794 with SMTP id i14-20020a056a20548e00b0018c8d0fa794mr18555840pzk.19.1701422864663; Fri, 01 Dec 2023 01:27:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422864; cv=none; d=google.com; s=arc-20160816; b=rM75cD7MgZIo6U0pyhCFoJNrgZm0nz0yC9xk1qJTgIwDNeeOyPrFJ5rN4ImQNtmmf3 sprS2Ed8bt10R+KwKSfYiwnMnMa7C+lekicO3pQMQ4L/OF3CVb2Es7i67SkHAvxuAZBy isPNfDNLckKGBtgi4jSJk81RtxGbh2CH5MYsaXX5dti8ZLd7weVF3mf6x+MemuRrJnhz bfXQdPjI1XVZzw94zDHkMF+6MhCXRo3MqmaFS7Yo2bkrhcCQXM+1ypBYtd/oAnYRwRai 2nHt1zkngYclaHWq2bnHPzgTtKuUZuJhLelmV51IzW+8JvF02ziHwGhE5gs1PMkyJ88V COOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=gO/udsWQ+bQJ+e/QIsLEl2DMKpANRb9evAUW1EC/BoI=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=md5vDEGGztCUMpcv+zMch6v9AgWVTxaMaKKniXJ8HtjYMoeWffniCEOmMJJJr763uN cnwyF1yWkm0wtn2i4zRCwmEtuALUQpdut/8E2HBlSkv3kE5nMrTQ27DSMm1ZwUKGDBJT BDpF+e9B3p2mJbCWC36cEPDkC+8KCiWwawqQ3TVxxYXannilO0s2RSy4i/AONOHIIoC+ zE2SqBWGXnojmkkB2pxxwaTNJ86qv8GZgyb6qi648v170yEdI3P7NOukzRnTia7qUJUh wDCFmWEzrjbUDL6c62JZK+vONerrrJTZFwjPfWsr1PfoVeTMBuB4Ck5AjV9MGaEsvjCD KrAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="eIQqB/WQ"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=DLkvoQ1h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id ca7-20020a17090af30700b0028596b1c1afsi5433763pjb.159.2023.12.01.01.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="eIQqB/WQ"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=DLkvoQ1h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 3B5F1810F6C3; Fri, 1 Dec 2023 01:27:40 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378066AbjLAJ10 (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378034AbjLAJ1G (ORCPT ); Fri, 1 Dec 2023 04:27:06 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 092D610F9 for ; Fri, 1 Dec 2023 01:27:13 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gO/udsWQ+bQJ+e/QIsLEl2DMKpANRb9evAUW1EC/BoI=; b=eIQqB/WQldigQLY0Gair11m9tMLsVtE1Ib3/oPytcBEd0/qDyfDCkRvTegLi/ZOrzT7FAp m6oEWHBnwrgAMkh9+bzPfsxav1MuLLAhzI9oyKAjTVdcqH2yPJXlWIy6lJF+SCwvcwKr1X 7Pp+MFb8jhvlCfqGjEwPSRKuAt9kEkCnabNBehpLjPbkXIZ8K5IoNUljS6/u50/eSfQgUf VSThVmxWrJu8mpM5Qi5/FgIUXg+bYlY8yit7NI2v3s5jB8u5yF0EAH4bk1q8dEM325dil4 5dNv4T2hfHZdbg62aF0lh9ESjOV1r6Dbv991Z5BpJBgmgLhuWM8DC8/sKkvU4g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gO/udsWQ+bQJ+e/QIsLEl2DMKpANRb9evAUW1EC/BoI=; b=DLkvoQ1hQ176JTziAZ/kfHhh7nRRJW7GCXqvS2BDCpVLtS0WNM3KRr3h6ZYLXNylsajT5P e48W33DVK1JYzuDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 08/32] timers: Clarify check in forward_timer_base() Date: Fri, 1 Dec 2023 10:26:30 +0100 Message-Id: <20231201092654.34614-9-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:40 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071181867783523 X-GMAIL-MSGID: 1784071181867783523 The current check whether a forward of the timer base is required can be simplified by using an already existing comparison function which is easier to read. The related comment is outdated and was not updated when the check changed in commit 36cd28a4cdd0 ("timers: Lower base clock forwarding threshold"). Use time_before_eq() for the check and replace the comment by copying the comment from the same check inside get_next_timer_interrupt(). Move the precious information of the outdated comment to the proper place in __run_timers(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Move precious information of outdated comment to proper place (as suggested by Frederic) --- kernel/time/timer.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3ca706db1d20..66bac56909ba 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -944,11 +944,10 @@ static inline void forward_timer_base(struct timer_base *base) unsigned long jnow = READ_ONCE(jiffies); /* - * No need to forward if we are close enough below jiffies. - * Also while executing timers, base->clk is 1 offset ahead - * of jiffies to avoid endless requeuing to current jiffies. + * Check whether we can forward the base. We can only do that when + * @basej is past base->clk otherwise we might rewind base->clk. */ - if ((long)(jnow - base->clk) < 1) + if (time_before_eq(jnow, base->clk)) return; /* @@ -2015,6 +2014,10 @@ static inline void __run_timers(struct timer_base *base) */ WARN_ON_ONCE(!levels && !base->next_expiry_recalc && base->timers_pending); + /* + * While executing timers, base->clk is set 1 offset ahead of + * jiffies to avoid endless requeuing to current jiffies. + */ base->clk++; next_expiry_recalc(base); From patchwork Fri Dec 1 09:26:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172345 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991627vqy; Fri, 1 Dec 2023 01:27:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IHazwP2bz5ds84o2sQOarfNyP7SncZK2rtM9xN8yrYyWQE77vGJRHWCjLWQivU4xNYRd1iw X-Received: by 2002:aa7:888f:0:b0:6ce:866:7bef with SMTP id z15-20020aa7888f000000b006ce08667befmr496795pfe.5.1701422864986; Fri, 01 Dec 2023 01:27:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422864; cv=none; d=google.com; s=arc-20160816; b=OYN3dMza8Loatfy/U8bg/E6UNG5MRU95vejxdk/hQtJ+IyRiPcrByXA0lucgygejLy ng8Yvaa12gJ+UJU3OhTWl4orgDLHZZF2a7I/cJzJAk/2vt1PCaKKB0Y9m1LrZHAAd5ih N9C7QXxDWNxzQTGMUMzeZlHUMFwScj019x2dbE4L21BbaVJUEsp2/5AB2viGp5KbjZzE dv8dYuoKj2yKBUjhzpMWZqePjRjqAvO9lKvGyrc6hDxUyBvXdtTOxGeSVEA9g2GLop8P hvGuPhZ05dOom9GnWJyC77nljiQf5p54gmYsVxV6DhrQy6hb85bFtkmF+0JH84htOaM/ uBrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=NX2OtYZi8wYLdH4/EBEFqndbEahIgfLRJkHvTArpnf8=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=qlWYumZfGv/ejheOIATJVXcbmVHt3h/1/hUUeE9C54YrEbn7ezzpNU32t4SFqV4T3n 9+DoB8ntaKPADtNDhEklGBLqZXBZeOxHAd6OtiYBBbGjKEwUdkIs9Kco7yMr5Fjr+tT2 RxZJiv36BW6AkDpXTcQNpUIpb66sou5r3u94nFu8muM9UuTCoChj8+gLrig5WP7NTRWE Q83leN+9W09AWFM3uCEu25p13RA8i8DYAt9hS0uURw/u86UPOjCdx90ggfI7Qx+KSaTA voXwGX1t8l6IP9mvXxDgiXwuvSCaCxlm/u7ajjPv8HSh4cZ3oKWAOIFMKIl3NfdJ+fK0 TB/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1ggAH7wj; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id fd3-20020a056a002e8300b0068beee4922csi3135941pfb.23.2023.12.01.01.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1ggAH7wj; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 0FDC780ECF3F; Fri, 1 Dec 2023 01:27:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378099AbjLAJ1d (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378035AbjLAJ1H (ORCPT ); Fri, 1 Dec 2023 04:27:07 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB89710FC for ; Fri, 1 Dec 2023 01:27:13 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NX2OtYZi8wYLdH4/EBEFqndbEahIgfLRJkHvTArpnf8=; b=1ggAH7wj+axl9TgEGJFZ0/uAb3cs+EW+X/QNAiGkGxhubO1WQ+ye67f57geO0uUgLmkZkL HvIPqcMu+1kgF6qnb72qlUj9+XGiJQS0qO4JNx9UYCX5iEGrK9FyAavNTYTSNwNpE9ghjY AfDEUuun8UgRzUrq7qEydbw/pXjJWUdOCUkmVDsiEWDsuPTYlj3u1B6Lr6HDIQzL2LG76+ kQcf4y/R6aiDYIlAQdaPPdghBaLiASdU2NCpEJEfA9zsdj9Hf9ZpuCbdej0w3aUOfxFDWQ ZNCAwSvoyAy9aMCi13puWX28GeFSL9rfPfL9btfjxsrP8FzpUmD8JWwSRMXaOQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NX2OtYZi8wYLdH4/EBEFqndbEahIgfLRJkHvTArpnf8=; b=FWFW6/pnQLncFa4pPnDKR90jyaAVnwrvB+ARbjuC6CzkQWAsi8kK9Gc2XmjQxcA0arZv6p oCcqOjNaJyyrPcDg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 09/32] timers: Split out forward timer base functionality Date: Fri, 1 Dec 2023 10:26:31 +0100 Message-Id: <20231201092654.34614-10-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071182068885736 X-GMAIL-MSGID: 1784071182068885736 Forwarding timer base is done when the next expiry value is calculated and when a new timer is enqueued. When the next expiry value is calculated the jiffies value is already available and does not need to be reread a second time. Splitting out the forward timer base functionality to make it executable via both contextes - those where jiffies are already known and those, where jiffies need to be read. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 66bac56909ba..e289cbd84e8c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -939,30 +939,34 @@ get_target_base(struct timer_base *base, unsigned tflags) return get_timer_this_cpu_base(tflags); } -static inline void forward_timer_base(struct timer_base *base) +static inline void __forward_timer_base(struct timer_base *base, + unsigned long basej) { - unsigned long jnow = READ_ONCE(jiffies); - /* * Check whether we can forward the base. We can only do that when * @basej is past base->clk otherwise we might rewind base->clk. */ - if (time_before_eq(jnow, base->clk)) + if (time_before_eq(basej, base->clk)) return; /* * If the next expiry value is > jiffies, then we fast forward to * jiffies otherwise we forward to the next expiry value. */ - if (time_after(base->next_expiry, jnow)) { - base->clk = jnow; + if (time_after(base->next_expiry, basej)) { + base->clk = basej; } else { if (WARN_ON_ONCE(time_before(base->next_expiry, base->clk))) return; base->clk = base->next_expiry; } + } +static inline void forward_timer_base(struct timer_base *base) +{ + __forward_timer_base(base, READ_ONCE(jiffies)); +} /* * We are using hashed locking: Holding per_cpu(timer_bases[x]).lock means From patchwork Fri Dec 1 09:26:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172353 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991889vqy; Fri, 1 Dec 2023 01:28:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IGWlwpdMZzUDR6OT1Zl4EgDx6VyM7QnN6DY6+//Cmt9ZEnwZAKBzhRZA/5a/ZhVaGYG2pdK X-Received: by 2002:a05:6830:1301:b0:6d8:551d:5b4e with SMTP id p1-20020a056830130100b006d8551d5b4emr2352908otq.20.1701422900481; Fri, 01 Dec 2023 01:28:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422900; cv=none; d=google.com; s=arc-20160816; b=ZPocAQpky4XNiCqJqNQaa1K343LwfK6IfLzocpYRkMaXwd7GRAVVeAPb/yHw4Gw7lU X/wWTVfBrc0naKl3M/XJ1tZ5HRLAUcEIuDdpfkIC25pGjk8coeZVkTuRL6Po8T53XYHy y5OlqENBBo4QRfro5eykk+vavME99XjgxIQmuutG4iElOR08FlLxnTw4MgAf0F89rfGj gBkg324TWsr6m+rW24utzbkfRDyvKA6jRl5c9BDU/ePWTK3TMrOvg7yg0jQ8hG3oyqu0 kq6zzA19bUWRlCLcOjING/tJbpgm263xkD4CyOSLH/NV97emS7GMcfLbpXgfi9rHs/q1 2wgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=+S43URzdkm4GE5deeh35HedlVNn9pKmfGb0vCPSVNJU=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=illFZeIdX4SBbPItG7ViOf/xAwCEogbZTIMN6grhVLed4ItWAxR5kYoKAXvSXQ9rt6 TzAKZRpeWPzcG7reMQ+u6jN61iThpRZBY7z+wnk/yB4b8K5UAlli1znpFeBMP3rIWDcb SsTR4oA24Ya6+k0eP50ayGP8581CqDNI0S4ubBexr/jQhvx2caKcgPxw8uXSXwjMBZcE kqejzwYgll9gfaDMJvHKsxUGfRZgIU6zumtFLEQlehjiePRk0CiOuRO5U+MPjnNaDTCj yP0/5pSlMrAZg2Dmk6y0xzmzV9Hpi+f2jHFKsp4pAGTQeaoVjLZ3FRm86Ee29boJxspa VOCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=el5AdGLC; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=HxtFSzHB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id g10-20020a656cca000000b005c5e2967758si3230998pgw.671.2023.12.01.01.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=el5AdGLC; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=HxtFSzHB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 052AD83D9B66; Fri, 1 Dec 2023 01:28:14 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378104AbjLAJ1f (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378065AbjLAJ1U (ORCPT ); Fri, 1 Dec 2023 04:27:20 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44A8B171A for ; Fri, 1 Dec 2023 01:27:14 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+S43URzdkm4GE5deeh35HedlVNn9pKmfGb0vCPSVNJU=; b=el5AdGLCIQZcbBus9nb/xgs+vY+Tv76/155OcrpNkNOjUWHu2Iay04ijEup/CZKYAfF+qh sLqmjFN9GHKVRUzf9cRZklprE52pJ9L8kUfKjnGdDNVQK2XqUFEQ+6+hzipfaedbaRthZm 6Kl2Obge+HPzfqeWxSHmaScooCQbrXk8SOQ3Ik/mN0vTv4iSaOEZbdKEONwbdxIV5BJU/E WNrcDanyKol7UClRMoZbNgCpdqMNa08bdC3xYLNna5+c2HQf/8kKGMT+GtVdqx2WIXo3RJ RgKdDCKh3NQUBx0fiAdJKLos3M2mXq790/QjjGt3MOHAEXZREpaJo/p+qtLbPw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+S43URzdkm4GE5deeh35HedlVNn9pKmfGb0vCPSVNJU=; b=HxtFSzHBtGc0DacUydRzFJpAQaTMJ4mc5tul79/rcY9HP9L9pECbfeBHsmnvjQJFudbZJQ DO5tluaLRIL1/YCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 10/32] timers: Use already existing function for forwarding timer base Date: Fri, 1 Dec 2023 10:26:32 +0100 Message-Id: <20231201092654.34614-11-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071219155282403 X-GMAIL-MSGID: 1784071219155282403 There is an already existing function for forwarding the timer base. Forwarding the timer base is implemented directly in get_next_timer_interrupt() as well. Remove the code duplication and invoke __forward_timer_base() instead. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index e289cbd84e8c..fee42dda8237 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1939,15 +1939,9 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* * We have a fresh next event. Check whether we can forward the - * base. We can only do that when @basej is past base->clk - * otherwise we might rewind base->clk. + * base. */ - if (time_after(basej, base->clk)) { - if (time_after(nextevt, basej)) - base->clk = basej; - else if (time_after(nextevt, base->clk)) - base->clk = nextevt; - } + __forward_timer_base(base, basej); if (time_before_eq(nextevt, basej)) { expires = basem; From patchwork Fri Dec 1 09:26:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172350 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991780vqy; Fri, 1 Dec 2023 01:28:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IH2on7B6pt1/HH3OYiS/UgiqM+8peI1lqkMNOog3152icdCzEOrZKITOLgTzItEqd7setJM X-Received: by 2002:a05:6358:70b:b0:16d:fe33:4c63 with SMTP id e11-20020a056358070b00b0016dfe334c63mr27182966rwj.24.1701422885071; Fri, 01 Dec 2023 01:28:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422885; cv=none; d=google.com; s=arc-20160816; b=IIT9wb2V8RwUGa7CAiUgt2RSwhoFcWdqNbyjIRkKBKK/VpicejZABA+FvfpLPKFYMe r8GURFYUmLGvkKcDoc6/ywJCgfKUizlvZLggHU3BQbzl2trSB1vhW2+OBHekJm65aDIn WVzZHOGt71ybAPqYkpJ0XlImU3H+lFIx/zJgIwHzMfOZ+bryyAF4XGGGrppRv4TAnIZz 3zul0ZmsYumCmiDprX2r1B5c3a4QxNJQmY7zQce90b5pLiQQqn1ntQrHSANUP1BRVEQ+ vAF9noVBXIcOa346OD9I+eHjMCYhYnkZEve0ftytAviYnzRAXf4mHu/78dL5e0RpxqZu lBtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=OjHpvhdurB+xA/Q5Qjg90zYVhHrCM9ixI3FXScsHyt8=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=CahPC9GDSpvdRMkmBOSD/ld0Mt0f5Gy7CixOcBiRpWxu4OyIYBDq7l5S8z/NoNNPS3 L4OFpx3GVN3WDFSl6H0k4hkU0zBWaYTbWIUMI8wQ358J5z+/pBThlUxfVva8AaKqoPLU fZ/X2e+ebfNj60HqxsfUlpfQNw5u6FzJVo6CiJlMfoqNRIgqfb13yoS7FUJDt3x4tB6H Pa767ypAnmj9mlzGfXSgIEzXRjQUiwVs8aYcsPD36+REVFAwF5/U8dnajB3OpbflE8D7 q7A8m18ic+oMWoaVajJNUS5RYCU6u1ff6EpRyjvQTNX59h6ECfnQuxSD7bpfHc9QX+to HQCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KKGSI58S; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Aml3ir39; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id h8-20020a056a00218800b006cba60de063si2990987pfi.22.2023.12.01.01.28.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KKGSI58S; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Aml3ir39; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 276F2825F14B; Fri, 1 Dec 2023 01:27:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378122AbjLAJ1p (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378050AbjLAJ1W (ORCPT ); Fri, 1 Dec 2023 04:27:22 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03BF8171C for ; Fri, 1 Dec 2023 01:27:14 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OjHpvhdurB+xA/Q5Qjg90zYVhHrCM9ixI3FXScsHyt8=; b=KKGSI58SUmBD5Umq/6Q4hIqdbuQjSnzca6zjvAccM2ulw1J6bMv4sXp813ZrMIM1hVtQVH MspT53qWCRzuqvVxUVkoZ04eH80zx/S7MKiw82z+OJKUhRBCi4XQI0/1WlXc2sqf13LW5W IliQXqnyGckto5IWaqLX7GyU0+RzB8zepKOF/AnVpcJ+6MNpR5HCo26arlDmQf94v6jG3J wUax+CBx48cBjE5Axx9p7sO19r8k8gYwRvLBrc5EXDJDpWBcX66pOL4Ys/OiTDUdjW/xc+ t8UrhivzaOHFUQCi/N8COFa/nSqBBhaGOO7CG3xoQe4tqPVGSUhVWd0rxerE7Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OjHpvhdurB+xA/Q5Qjg90zYVhHrCM9ixI3FXScsHyt8=; b=Aml3ir39NvV+AcjCiHTJCqqN6AvMpP5DwCTpdnC7/AHu45ZPNUabml56+ThbNpQsMZFpb2 eyfv64zM1EqeFwBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 11/32] timers: Rework idle logic Date: Fri, 1 Dec 2023 10:26:33 +0100 Message-Id: <20231201092654.34614-12-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071203084274753 X-GMAIL-MSGID: 1784071203084274753 From: Thomas Gleixner To improve readability of the code, split base->idle calculation and expires calculation into separate parts. While at it, update the comment about timer base idle marking. Thereby the following subtle change happens if the next event is just one jiffy ahead and the tick was already stopped: Originally base->is_idle remains true in this situation. Now base->is_idle turns to false. This may spare an IPI if a timer is enqueued remotely to an idle CPU that is going to tick on the next jiffy. Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Re-ordering to not hurt the eyes and update comment v4: Change condition to force 0 delta and update commit message (Frederic) --- kernel/time/timer.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index fee42dda8237..0826018d9873 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1943,22 +1943,23 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ __forward_timer_base(base, basej); - if (time_before_eq(nextevt, basej)) { - expires = basem; - base->is_idle = false; - } else { - if (base->timers_pending) - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - /* - * If we expect to sleep more than a tick, mark the base idle. - * Also the tick is stopped so any added timer must forward - * the base clk itself to keep granularity small. This idle - * logic is only maintained for the BASE_STD base, deferrable - * timers may still see large granularity skew (by design). - */ - if ((expires - basem) > TICK_NSEC) - base->is_idle = true; + if (base->timers_pending) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } + + /* + * Base is idle if the next event is more than a tick away. + * + * If the base is marked idle then any timer add operation must forward + * the base clk itself to keep granularity small. This idle logic is + * only maintained for the BASE_STD base, deferrable timers may still + * see large granularity skew (by design). + */ + base->is_idle = time_after(nextevt, basej + 1); + trace_timer_base_idle(base->is_idle, base->cpu); raw_spin_unlock(&base->lock); From patchwork Fri Dec 1 09:26:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172348 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991726vqy; Fri, 1 Dec 2023 01:27:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IF5qFET8aGFrzdyIVibjCJci+Ft6FCKqZOdkZWwH5mDKR9TKTNHNsLGawbNvbkRci6rVkOX X-Received: by 2002:aca:1b1a:0:b0:3b2:e6a4:e177 with SMTP id b26-20020aca1b1a000000b003b2e6a4e177mr2073265oib.51.1701422878068; Fri, 01 Dec 2023 01:27:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422878; cv=none; d=google.com; s=arc-20160816; b=J6Csz8PRhF59mZG1Q81udY9b8EHfzdr+ik/zhu643B6fWL/8SPIXikmrtTcOHvQulN FbijCWcvU0PDgDfHdgCnlY++2RJtwzrmMoWVmp2RriMx3E0rpDlseDI2jgB4dQUjL9yR /5wLudxOgUn4GB96YfYuSjrLHRl/QAbN4iGmmCY856ARnH+TrlA0Senn3VJ9gfruivux VrddHkrYXRx6CxAJ8u9HTvamDuzUBjHUZNKTztKMmcd8fHLzYH//QkUPUV7oBT4oL28d u4ic2ON/MWJklUu7Vi7aKEiV1MWKAN6cI2/E58Ief00hX0550sd5BwZc+cxY+b4dhqko b5Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=nZi/SXfoi2GICSzM+IYBAVSlO8Hm1DXEFr76jQHYOZ0=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=HD1x18lJAYoG/c4S+FLkOjtAx0Cdp/pTcQX3Ro7VlzjkZkLaMgH7tJkfcD6SiBWbwA eqZ7BZc0gGBC3Ub3AgEgKhu+zJeOtfhktvsg2/BJbEA6t2pNCYCF9B3kOlMNe3sexEc9 Wv3rKHxMA2/dss5UUagIaHsjIn6Fl/WC0JjBrVRmIIhgVf3ZLiZDr59QXj9eyAliJELG HdpauBjgD8lzNb9R9QKEKF/cXvyVyT2nSnAFtPgwLKcdtWLOXkOPfdmk7fUl89zXy/kL JSRxRKNeecOv2x78GRnxGZUjSzI0CZo58EuXIxULKiOTLirz2LHGtVWfQS5nyIkkKZYb gdKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="lPUu/u37"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=DGxDnst2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id bx41-20020a056a02052900b005c5e2ac7c3csi3279148pgb.732.2023.12.01.01.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="lPUu/u37"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=DGxDnst2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 24B31833AB02; Fri, 1 Dec 2023 01:27:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378043AbjLAJ1m (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378055AbjLAJ1W (ORCPT ); Fri, 1 Dec 2023 04:27:22 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3A521725 for ; Fri, 1 Dec 2023 01:27:15 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nZi/SXfoi2GICSzM+IYBAVSlO8Hm1DXEFr76jQHYOZ0=; b=lPUu/u3782O76F4tssDfbbwJSu1UYJWRXA1YF1HFvK1m8kbs0zLQsUh/Qrq0SympHt1Xn4 fWs6pLLsziRXsDGGtVSkmgIluUmEPyOt49N0oJuW1GpqQsDrQiTwLCGHDYPxL2sHNl8pO8 uZrfXgDRa3Soql/IDz++WAZHigpe67VrkWm7MrhlkMdDJb1cuTVJRBpNVrj0k0qs67dVi3 SnfBc962XKiwdxw27XZSz4hJUgXOfPYnpxuSOBDbSvDj36Sfa0N/0H08AMFC8K3OePFC/t LsAllTQBY9b+fu99vTafsfRYXEb2PxTXLZsynXnrqJJ34TflfOYmg29flJ7ItA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nZi/SXfoi2GICSzM+IYBAVSlO8Hm1DXEFr76jQHYOZ0=; b=DGxDnst2vjquka22zFxBnL29sukEirkFvC4s2bWFecYQdCP9TQh2tKsQYBTAwCLj3xIhX6 heDteX2CRHR+6lAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 12/32] timers: Fix nextevt calculation when no timers are pending Date: Fri, 1 Dec 2023 10:26:34 +0100 Message-Id: <20231201092654.34614-13-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071195826791695 X-GMAIL-MSGID: 1784071195826791695 When no timer is queued into an empty timer base, the next_expiry will not be updated. It was originally calculated as base->clk + NEXT_TIMER_MAX_DELTA When the timer base stays empty long enough (> NEXT_TIMER_MAX_DELTA), the next_expiry value of the empty base suggests that there is a timer pending soon. This might be more a kind of a theoretical problem, but the fix doesn't hurt. Use only base->next_expiry value as nextevt when timers are pending. Otherwise nextevt will be jiffies + NEXT_TIMER_MAX_DELTA. As all information is in place, update base->next_expiry value of the empty timer base as well. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: New patch --- kernel/time/timer.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0826018d9873..4dffe966424c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1922,8 +1922,8 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; u64 expires = KTIME_MAX; - unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1935,7 +1935,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base->lock); if (base->next_expiry_recalc) next_expiry_recalc(base); - nextevt = base->next_expiry; /* * We have a fresh next event. Check whether we can forward the @@ -1944,10 +1943,20 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) __forward_timer_base(base, basej); if (base->timers_pending) { + nextevt = base->next_expiry; + /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + } else { + /* + * Move next_expiry for the empty base into the future to + * prevent a unnecessary raise of the timer softirq when the + * next_expiry value will be reached even if there is no timer + * pending. + */ + base->next_expiry = nextevt; } /* From patchwork Fri Dec 1 09:26:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172349 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991742vqy; Fri, 1 Dec 2023 01:28:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IEAtl+btLsIUgNevoDh26Mqoo4B5XnOMWxqJkq62LsUxC14X/Cqq7jpj3TrlktidYikZ+RF X-Received: by 2002:a05:6a20:2451:b0:18b:8475:410a with SMTP id t17-20020a056a20245100b0018b8475410amr30500833pzc.7.1701422879725; Fri, 01 Dec 2023 01:27:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422879; cv=none; d=google.com; s=arc-20160816; b=UyYZmjmgN1GCRZoq0F1aOce3/Z4GTW4HANNQJMI4B/PqGEeHP5HcV8wYXWOltwdbCN PBDfueIRnPErblWNW4Koh467whN4ux0UPTaPj7LuE747HBOM/h6vVRtoi1kYTIIs+5BZ 7FwwfvPi39ppe+/tdWlh02y2461NaHN5M9etpnhhsz+LkmR7tSUzQCrQf9iWMzSwRBOy xbJcAl+irfFCSz4Llfor49HqXa1DApVm4B62jX/Q2cBBrChNawFlyySVgmzMexcc7NZ7 uLK8wCpMuTvXjHtSCixYX9mCNfU2nV+oe9LGt7WoZov4AXBiomlbGiug/pEghrUYueYx ZBqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=eWAzXEfpMywLlqp587KxIHRrk44Rp7dtxL6JlN1/eF4=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=Ti+vP5h1fXUeh6tt3wMvrwdv2J1H1FsSyy0mk5EI09VssM5hG7puToJqgEoLNwB3hm V6YSbmSYbWlnxoZVkKgLGJUm1ImEKMWfsZYfzO9YQqUqkqcEdK7S66+3tFItqs5aKRY3 Zx2Kdt/KqRUz9UoR9qsyMEWaAN7Ii6vd6x62HQ5c3Na4GP8bplBFzGQv8DOioqOi0us9 alT8Q5KyAfBjQjmGy6jne7ZrbmH8EWuvnhyfFgjCUfHS6Cf5hVKudWLBw++L0qfVxpxq 28D4mGN1vz6QDfXJzwfCz8gtMPqXae3awfl4KXADIPAuozYUMRXqohB8Su+TR3hJsUEm sQwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=nrPDcWG6; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id pj11-20020a17090b4f4b00b0027739322ce2si3285757pjb.100.2023.12.01.01.27.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:27:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=nrPDcWG6; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 7DEBD80EA97D; Fri, 1 Dec 2023 01:27:54 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378109AbjLAJ1j (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378047AbjLAJ1W (ORCPT ); Fri, 1 Dec 2023 04:27:22 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7992E172E for ; Fri, 1 Dec 2023 01:27:16 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eWAzXEfpMywLlqp587KxIHRrk44Rp7dtxL6JlN1/eF4=; b=nrPDcWG6y+EdgnAwLZJrMGEaTSj4fpLZbHu1F5MYX4pNfm8XwdDmfbaq3+Qfw75U13WXlN Wn/03lvZN5wgl5keYEfUyqe+Ren56ovfCTN73gvsCHyVcgnrOaxR6UU35FeZKoPn9cXLqx HnQJ41b5XrnUqb9KygJlhKObBRbD0CBisK8MgF7pi4EzW1AySz6TKSzWbQeWz/O25sSOGI mUcxSvIcwOCrfPKPMKcjfpRPDQXVMO/6pXJSNwHje+BVCkdOMwdyCG52Y5obMfOKXuqsrz xPK5vVvU53qVkzQ12pNGeND+dF67+YhA8Kfco1uSKz+rFzdJHkWzTHobUjGZJg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eWAzXEfpMywLlqp587KxIHRrk44Rp7dtxL6JlN1/eF4=; b=ejl4p2jWf2hUodAxU46BdlFPsS0jUview2NzC2fTxisTms7SzXjFDnFPfiT065gMEKHznq UDzdpbW485YF6UAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 13/32] timers: Restructure get_next_timer_interrupt() Date: Fri, 1 Dec 2023 10:26:35 +0100 Message-Id: <20231201092654.34614-14-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:27:54 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071197149846098 X-GMAIL-MSGID: 1784071197149846098 get_next_timer_interrupt() contains two parts for the next timer interrupt calculation. Those two parts are separated by forwarding the base clock. But the second part does not depend on the forwarded base clock. Therefore restructure get_next_timer_interrupt() to keep things together which belong together. No functional change. Signed-off-by: Anna-Maria Behnsen --- v9: New patch to eases patch "timers: Split out get next timer functionality" --- kernel/time/timer.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4dffe966424c..9d377ebb7395 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1936,12 +1936,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (base->next_expiry_recalc) next_expiry_recalc(base); - /* - * We have a fresh next event. Check whether we can forward the - * base. - */ - __forward_timer_base(base, basej); - if (base->timers_pending) { nextevt = base->next_expiry; @@ -1959,6 +1953,12 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) base->next_expiry = nextevt; } + /* + * We have a fresh next event. Check whether we can forward the + * base. + */ + __forward_timer_base(base, basej); + /* * Base is idle if the next event is more than a tick away. * From patchwork Fri Dec 1 09:26:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172351 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991841vqy; Fri, 1 Dec 2023 01:28:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHm5E97/IiXE9lAj1KOAhTxbCUj15qlXLOdohwG3/C3qRaCEtTN/3H6nZvsxtksoxRVGTVg X-Received: by 2002:a9d:6743:0:b0:6d8:543b:6cd with SMTP id w3-20020a9d6743000000b006d8543b06cdmr2303275otm.31.1701422894644; Fri, 01 Dec 2023 01:28:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422894; cv=none; d=google.com; s=arc-20160816; b=lMM6yGA+AXlYHjbfuKicbHPH1zq79dmbm//58u801FehQeHDg08KIFzHyL7Ri8NYPm Evhadik36OrAWYMXifaLgRt/KgOyBZuVtciBZ5Dpcx9KM1WY2JqwHiJVXmLBUGxIS2kq FC2+qDaBhRcXC/+OpFpQld6flvHzEwTOmInkO3foDo0ROFjcBE0yEqzxCyds9TEbI2de 7rr2hTk29jzp0q/9Edgvh8qWudmQRMKFWLDGM7G24zQFBkY9x7tTpK9fhKlRzqqj2u44 D/QuEnLkdQ+HgQYHRqU2oaNftfANGTbdtzidndH5+6TiM6UtvxC4KOPQar1qNxGnIlII GxXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=JrX+uOonkdlgOULBTRJ+tPCkJgZ/y8AeNa7bH45nGMA=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=bo5D0MbWy0Nfdl6h417lgnqng/w2JHeafogoaduCy/5KmLP4jZ7fIwNterKtUlaQ8h HcA/h6g2bQCWbmWiMsYzz5kr8zJgitOWj4teE3PiNqJAdeC8dH2fSPgFo7Q2Q6xZzPtW P85MdSFFKJa4m4iVx+ktRWmfLXEYOeYKdCUvI+IZ7xBrXFYVgE7NANqCilclYtUP0OM5 FsohH75ocTXbvhhbk6nxtWkLjRMGA5QCgqI7XKVkxFz9DK5NyFLHvx5rAkqle67P5yqk LTpbRbTm5TpmltINYAezPqVnsLUniDph3UryJ5sOl28JxcCVwiy2Fjox7+hgB7FGjJ9c r3Lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=D5XrFnLB; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Mla+Aech; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id m189-20020a6326c6000000b005bdbda280c2si3077787pgm.364.2023.12.01.01.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=D5XrFnLB; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Mla+Aech; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 4873181552EE; Fri, 1 Dec 2023 01:28:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378134AbjLAJ1t (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378030AbjLAJ1W (ORCPT ); Fri, 1 Dec 2023 04:27:22 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44E0F1731 for ; Fri, 1 Dec 2023 01:27:17 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JrX+uOonkdlgOULBTRJ+tPCkJgZ/y8AeNa7bH45nGMA=; b=D5XrFnLBsbY5lkVSgkpnNLWdRfAh8BRN6Ig8U6EK79iQ7Zz6/XWyZ79d2M5HXR5I74m+Pg S2hARumZwAdv8qw3krKtquSgSCKHqHiLz+BWE4T0RiujFRl5MC0AqCSeVPs3cVP7Qy8QTt BhTFkQaRlJPXDnkOTYvZAkLppDQPesiYyCwodQABI0CGKBDr6ryeU3PvmE3eKTtphN6BN5 vZ6ZXUk6bXGQAKaLiOeCpTyaEQEBor8J2m5oteVUECBzkbP15etoSFY8qeoidjKwOILaA6 vmJKuQ90tjl4f8jjJJDBqc7RPGnDbvC09YC3jWkqB+JV1J73vhnXmj6HVmW2MQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JrX+uOonkdlgOULBTRJ+tPCkJgZ/y8AeNa7bH45nGMA=; b=Mla+AechBwy5Fc5CLuvFJk+kdfUcABs97vqRw9ixz3j4sPEqx0f9eeiOBVWA3gXcymuY6G 1eT8gT1dpYXIoHCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 14/32] timers: Split out get next timer interrupt Date: Fri, 1 Dec 2023 10:26:36 +0100 Message-Id: <20231201092654.34614-15-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071213263580529 X-GMAIL-MSGID: 1784071213263580529 Split out get_next_timer_interrupt() to be able to extend it and make it reusable for other call sites. No functional change. Signed-off-by: Anna-Maria Behnsen --- v9: Move whole get_next_timer_interrupt() into a separate function instead of splitting the bits and pieces. --- kernel/time/timer.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 9d377ebb7395..c9f7f86e95fd 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,15 +1911,7 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -/** - * get_next_timer_interrupt - return the time (clock mono) of the next timer - * @basej: base time jiffies - * @basem: base time clock monotonic - * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. - */ -u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; @@ -1975,6 +1967,19 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return cmp_next_hrtimer_event(basem, expires); } +/** + * get_next_timer_interrupt() - return the time (clock mono) of the next timer + * @basej: base time jiffies + * @basem: base time clock monotonic + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +{ + return __get_next_timer_interrupt(basej, basem); +} + /** * timer_clear_idle - Clear the idle state of the timer base * From patchwork Fri Dec 1 09:26:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172361 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992109vqy; Fri, 1 Dec 2023 01:28:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IEOGHQ2L8v5yymUT8WAOMwEjlm3vl4PpmRBx8psXayXyP1dW2DtRbl56L3I2WgJqrpHzLu3 X-Received: by 2002:a17:903:230c:b0:1cf:c6a3:1652 with SMTP id d12-20020a170903230c00b001cfc6a31652mr17327024plh.12.1701422936424; Fri, 01 Dec 2023 01:28:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422936; cv=none; d=google.com; s=arc-20160816; b=nKX7hYjM0q3rMYws/zMtZccPbOCzbaWEh/p6fNSWguXVTXkR5hoZeKVgzyArLNXe+i Ycsxr/Io019HxkvCmaCyELccim5aDxJ0dOIKCzo01rEjlKwnqLA4pDUKB/clcZo0hoYO vbaa2k9shStNF1kdb/8nQoo9x6POyCTiBiHtx1swLJjaJ1XsRTtwe0uojS6eIQt9aQS4 VNW1Q1CgKLBFIvjiYLL0q8lZtncOtlARAwkn/79NmkZH9zVaWDR2jrFqXeHFH0BwsOQy D43o2wRzrKNVU9elEdC6C521jI7phD8vcY2ECUO8SQRgte+wihK8DK8umDnbgPRndynD opHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=XDkZ99bgjbwfcazN7cvP/yxsFnqYGi5j6mcGkkMlXpc=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=eqEo7QSvT2HVJDyfghZIxdnqwmibpJc4PJhzIxzHKEGkHm3A1bVnEgofMldIUjOyQF y3xFp3L4zTef+mpOArfGmIT8ENR7x6F2aMP9lmNDwV0PYpc1uuzrdXAcDvjjruyAEMPy KKoGPW2FuS+vbGq0B0yC4SeMbC1W2HftSXpMlhJOMIzI0pCDl/e2esTnMorqFPZ8W6RP tqIa4X3DeBPT1L0f/3lIDyGHroVqSeyZqoAB7JRPpTt+7fl+n5NUd/e3G7hXHJYFEsgO SBp44kKyIm5TcUGXO8Ox9suUtvq4QfjmEm8k73HBV4fCWtDjpOZ/guwqWIqp5x7wLiCg GmnQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=i5sS1zVb; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id t1-20020a17090340c100b001d05cf08ac6si337565pld.477.2023.12.01.01.28.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=i5sS1zVb; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 3B7B080BD4E1; Fri, 1 Dec 2023 01:28:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378155AbjLAJ15 (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378084AbjLAJ12 (ORCPT ); Fri, 1 Dec 2023 04:27:28 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69138173A for ; Fri, 1 Dec 2023 01:27:18 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422836; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XDkZ99bgjbwfcazN7cvP/yxsFnqYGi5j6mcGkkMlXpc=; b=i5sS1zVbA72IjyBXtjc3SlWfoRyosMorQZH3qX/weLjOiNry7I/7+4ZgkwI7rmOt1p/UXO 3ww8mo0MOUqO+0Diq1FG02gACn2a5p8pLaKx4fFpd3Yawe9JQAKrWl12KLbQwY1IMXXpAh 5W45BtdaHxyhbNgs8peW4ql2RF8W+ZB6yl2g9nfSMXzk+zG6IaY0ToMyTnMmUuM/iuz3mQ bqLuNhAa5FwwvNfTHeMY4zkeaW0719YMrlCGBsafijBw499DvJ87pFui7vqLqhZttIzHFJ rv22HaG2jZbCCvq2dWwD47XGS1nPvy+Bj4UvuGISVk33iYxk+xxsOgkyHOJrtQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422836; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XDkZ99bgjbwfcazN7cvP/yxsFnqYGi5j6mcGkkMlXpc=; b=pmacNEm05CQnWjnwz89fRs+iAO6fnmS1N9jqQgq915NFXVkq+NFsHJgid3bh8sVzGv0A5B dpET6y0V972a7nCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 15/32] timers: Move marking timer bases idle into tick_nohz_stop_tick() Date: Fri, 1 Dec 2023 10:26:37 +0100 Message-Id: <20231201092654.34614-16-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071256994598446 X-GMAIL-MSGID: 1784071256994598446 The timer base is marked idle when get_next_timer_interrupt() is executed. But the decision whether the tick will be stopped and whether the system is able to go idle is done later. When the timer bases is marked idle and a new first timer is enqueued remote an IPI is raised. Even if it is not required because the tick is not stopped and the timer base is evaluated again at the next tick. To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and get_next_timer_interrupt() is streamlined by only looking for the next timer interrupt. All other work is postponed to timer_base_try_to_set_idle() which is called by tick_nohz_stop_tick(). While at it fix some nearby whitespace damage as well. Signed-off-by: Anna-Maria Behnsen --- v9: - update to the changes of the patch before - Cleanup logic in tick_nohz_stop_tick() after executing timer_base_try_to_set_idle() --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 46 ++++++++++++++++++++++++++--------- kernel/time/timer.c | 48 ++++++++++++++++++++++++++++--------- 3 files changed, 73 insertions(+), 22 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 649f2b48e8f0..dc12a938f00f 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -164,6 +164,7 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem); +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b1b591de781e..3e1cdb7c6966 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -849,11 +849,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) */ delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { - /* - * Tell the timer code that the base is not idle, i.e. undo - * the effect of get_next_timer_interrupt(): - */ - timer_clear_idle(); /* * We've not stopped the tick yet, and there's a timer in the * next period, so no point in stopping it either, bail. @@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) { struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); + unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - u64 expires = ts->timer_expires; + bool timer_idle; + u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; + /* + * Now the tick should be stopped definitely - so the timer base needs + * to be marked idle as well to not miss a newly queued timer. + */ + expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle); + if (!timer_idle) { + /* + * Do not clear tick_stopped here when it was already set - it + * will be retained on the next idle iteration when the tick + * expired earlier than expected. + */ + expires = basemono + TICK_NSEC; + } else if (expires > ts->timer_expires) { + /* + * This path could only happen when the first timer was removed + * between calculating the possible sleep length and now (when + * high resolution mode is not active, timer could also be a + * hrtimer). + * + * We have to stick to the original calculated expiry value to + * not stop the tick for too long with a shallow C-state (which + * was programmed by cpuidle because of an early next expiration + * value). + */ + expires = ts->timer_expires; + } + /* * If this CPU is the one which updates jiffies, then give up * the assignment and let it be taken by the CPU which runs @@ -930,6 +954,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) * in tick_nohz_restart_sched_tick(). */ if (!ts->tick_stopped) { + /* If the timer base is not idle, retain the tick. */ + if (!timer_idle) + return; + calc_load_nohz_start(); quiet_vmstat(); @@ -991,7 +1019,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now) touch_softlockup_watchdog_sched(); /* Cancel the scheduled timer and restore the tick: */ - ts->tick_stopped = 0; + ts->tick_stopped = 0; tick_nohz_restart(ts, now); } @@ -1147,10 +1175,6 @@ void tick_nohz_idle_stop_tick(void) void tick_nohz_idle_retain_tick(void) { tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); - /* - * Undo the effect of get_next_timer_interrupt() called from - * tick_nohz_next_event(). - */ timer_clear_idle(); } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index c9f7f86e95fd..df6558f62e6f 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,7 +1911,8 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; @@ -1921,8 +1922,11 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ - if (cpu_is_offline(smp_processor_id())) + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; return expires; + } raw_spin_lock(&base->lock); if (base->next_expiry_recalc) @@ -1952,16 +1956,24 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) __forward_timer_base(base, basej); /* - * Base is idle if the next event is more than a tick away. - * - * If the base is marked idle then any timer add operation must forward - * the base clk itself to keep granularity small. This idle logic is - * only maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ - base->is_idle = time_after(nextevt, basej + 1); + if (idle) { + /* + * Base is idle if the next event is more than a tick away. + * + * If the base is marked idle then any timer add operation must + * forward the base clk itself to keep granularity small. This + * idle logic is only maintained for the BASE_STD base, + * deferrable timers may still see large granularity skew (by + * design). + */ + if (!base->is_idle) + base->is_idle = time_after(nextevt, basej + 1); + *idle = base->is_idle; + trace_timer_base_idle(base->is_idle, base->cpu); + } - trace_timer_base_idle(base->is_idle, base->cpu); raw_spin_unlock(&base->lock); return cmp_next_hrtimer_event(basem, expires); @@ -1977,7 +1989,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - return __get_next_timer_interrupt(basej, basem); + return __get_next_timer_interrupt(basej, basem, NULL); +} + +/** + * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases + * @basej: base time jiffies + * @basem: base time clock monotonic + * @idle: pointer to store the value of timer_base->is_idle + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) +{ + return __get_next_timer_interrupt(basej, basem, idle); } /** From patchwork Fri Dec 1 09:26:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172352 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991827vqy; Fri, 1 Dec 2023 01:28:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IGKlaoZwBLX5ezGHmdjSHhbWnKWyB5a1wlcS+UlWaeq7N+w/2zhHUOXSv/KKgivq3Fd7uPJ X-Received: by 2002:a17:90b:1d83:b0:285:adca:e8e7 with SMTP id pf3-20020a17090b1d8300b00285adcae8e7mr19146090pjb.5.1701422893134; Fri, 01 Dec 2023 01:28:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422893; cv=none; d=google.com; s=arc-20160816; b=Eo5NKMkVEHpyJ9OyPSA6QJVH3I/nEfSekBTSkEOuP7iWmoXrL/CxPt1g/nh0nvwl9l aZQCWqXWvkpWq4VZkw3VmF4yS2493N2GxmBfx2Teh2dM+7VACuhJ2XbTSgvqo738qz/O 3FLaVR/OondJ7iALkLnckweBOv9CnJ6Zh12TX2JLVbQaMfU2j+eBFTlHMgTXE4twYLgE mo5NzGFGnjzqO/1FGkYOJ4JbvR5VhrHFBQeBmzDkx0BislBXYrGFy8Ktpo75sb/rI4lJ E3WEcZceNbIqsXyycyXejm3D4C5p0KzkuaMCjdKNgBzhIaedYuVx4gddGgNIB1MDQ4Ln Rpaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=nS3of1gsjpadmlTzOeKtxhnS0y1kmX6N3aik41bHBiA=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=i8PWzu8NNUDXsOocGUO3T/ugxKr7iTg8p7eB6G++xAekHByKKFxOUMIQhj5hXjfZfs t5aBBo94lB8wfqskVfU/cF152C3WbMaX7C8rJudx9nFeN32lmWo79U9YR6aFbdFmM/3V 2KDn/9OhW+MMdr5wV0Bg3Kc38WwI3r7bHL/bsxalqmRWSEJaEdEjkyUBSoMM/4wM3LHX 8SfAhk//L09iITize68k53Ay+9XeRoqJsnMkvCLbYLidXKZFLDL7KRiEe96mkluYt8LK VhYUNdEZ+tT2jwhp0Qi6rfFfd4W/09tsX8OBUknj/RSSrl26STXL8YkxNUzOPPOh6zXP pCyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=jd8JdSo6; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id w8-20020a170902904800b001cfd4c10e47si2852892plz.8.2023.12.01.01.28.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=jd8JdSo6; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id D0B1A83D9B66; Fri, 1 Dec 2023 01:28:06 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378069AbjLAJ1y (ORCPT + 99 others); Fri, 1 Dec 2023 04:27:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378071AbjLAJ12 (ORCPT ); Fri, 1 Dec 2023 04:27:28 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE00B173B for ; Fri, 1 Dec 2023 01:27:18 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nS3of1gsjpadmlTzOeKtxhnS0y1kmX6N3aik41bHBiA=; b=jd8JdSo6RSMAzUQGD0eFZtqqXum4T5DNbkhKfHsGgfKrpZeYgvPMRbitqOvajGAVnOXjI6 CUwVNu95rNCy/L5GzK63NOh365EDVDrf9ayAK4msa7dXSpEa80xXUWomRi6duxWLFm+VbM qCr1hrcESNQGxbIdSVpGa6B5nMD7U+enDcqohqFlaqh/MYvEqE3lCzl2P2uDwny8GJ0pZb e+Y7Q5UkpxPIXt7kcmWi+U4AE9QBHs0PlQvwXPRBOBrT+RaB+0SEBxJG882s2281fx76As 0B3s31n61vB24z+VP0fxZkI3ZAtKnfsQ0JzpWsdc4wBnNvNXxSgu/bYlY0W5kw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nS3of1gsjpadmlTzOeKtxhnS0y1kmX6N3aik41bHBiA=; b=XvAVoft42FnLB6t8nWzHoq+keWMNp0u+ZuAOQmacO5++zBupzj5qe3N+KJ0loGqLT7o1ZZ rrBvtWXBA6WYHMAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 16/32] timers: Optimization for timer_base_try_to_set_idle() Date: Fri, 1 Dec 2023 10:26:38 +0100 Message-Id: <20231201092654.34614-17-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071211965782339 X-GMAIL-MSGID: 1784071211965782339 When tick is stopped also the timer base is_idle flag is set. When reentering the timer_base_try_to_set_idle() with the tick stopped, there is no need to check whether the timer base needs to be set idle again. When a timer was enqueued in the meantime, this is already handled by the nohz_get_next_event() call which was executed before tick_nohz_stop_tick(). Signed-off-by: Anna-Maria Behnsen --- v9: New, as this optimization was splitted from the patch before. --- kernel/time/tick-sched.c | 2 +- kernel/time/timer.c | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 3e1cdb7c6966..c6b415052c56 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - bool timer_idle; + bool timer_idle = ts->tick_stopped; u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ diff --git a/kernel/time/timer.c b/kernel/time/timer.c index df6558f62e6f..177bcde8a2c0 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1996,13 +1996,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases * @basej: base time jiffies * @basem: base time clock monotonic - * @idle: pointer to store the value of timer_base->is_idle + * @idle: pointer to store the value of timer_base->is_idle on return; + * *idle contains the information whether tick was already stopped * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer or + * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is + * returned as well. */ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) { + if (*idle) + return KTIME_MAX; + return __get_next_timer_interrupt(basej, basem, idle); } From patchwork Fri Dec 1 09:26:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172354 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991893vqy; Fri, 1 Dec 2023 01:28:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IH6vQsK/d5oQFqQBpbn/Wi3di68sKB9tg3AgZyf++JPXXb7rfL8Z+oT1YIHXfQbu2lXy338 X-Received: by 2002:a17:90b:1bd0:b0:285:ba0a:1c9d with SMTP id oa16-20020a17090b1bd000b00285ba0a1c9dmr18080461pjb.6.1701422900927; Fri, 01 Dec 2023 01:28:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422900; cv=none; d=google.com; s=arc-20160816; b=oQ7/Zz/KTmrKUZP8sxp65z8NwLWSa0vASafJTOS7qGr5SoWzVcEHBqp2mz32qPtrsL ewXtaVIe/Uk5AoXlud54jKFTQi7p4kpZumktISPFweDKClWUTemcrgGd3Ls9Iq3hmAFn Ozg73jCv3ZFF7xYq4cF22r6ewXYTTTCEM6J2uMmwoLjWeYseweK0QVUKc0Mpa290L3ND Zj+b5umvE3iJKrKv7XxYrdEYDet4+TC5eshwFSihfrxnymYZi901MTECMi8G9Dtuj/mV TEksMyrIUw378a64Cl/7qmgI+OJD5dGw4L6GZYIWgcWZXWmPZh39MB2zADiCHN3U5oHC 5czQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=2ndzPV+KTt0XzydVGeObWOLfmNe4sV+QpfiiUpyraAg=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=0OIInY+Jp9PzBvtwexCatAf1A+H3Y6z568ji+PNLEZzZbYSxWlWylf0g/Qn9FtNT1f /tLx7PHFweu25YloIQWCwCuZBKJiyvGOrUnikU0bnC7de8qQjJ1VZWsJAhGIsUStxkgo 47EyzNr7Cyb9KiJq0IvZQ7fSTSdMnt6V10gRJkVVK8XpGTU/GKrl5TLyVo5LkuS3003+ C2CQC8rVtZQU1NFRmCADj8JL/X4HUhjGy4vBZ61X9zhm/DJK97p4udPZK4q6AEI1G3eG Yf+SnIwMAfAS2potoqG+zusGloqsdzivyyGBJCegkjsSqAe0W7uZPMn1PvOrVZi7s1oc xK5Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=DdWrzuSI; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id h24-20020a17090aa89800b00285b3f6bb57si5240740pjq.60.2023.12.01.01.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=DdWrzuSI; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 041C980BD4EC; Fri, 1 Dec 2023 01:28:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378028AbjLAJ2B (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378092AbjLAJ13 (ORCPT ); Fri, 1 Dec 2023 04:27:29 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E43B9198E for ; Fri, 1 Dec 2023 01:27:19 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ndzPV+KTt0XzydVGeObWOLfmNe4sV+QpfiiUpyraAg=; b=DdWrzuSIaDLRs7yzIx91ENx015zY+Bmvcwzkws9SsYYR3IPive1ZN/YjvXz5AIRcb3lUQJ 1eaFD55Yp/84jZkuy4xRdsDKlqi5dUG53CUAHvUuV54SW87J2P5dxCDkXn3pw2SM+IHA+m oliEtTzAMLz+4sI8FKcuCJKbwTxCuIWJgldOAhjkWo/A9xAV+u20RPxcVr6+JQyO5/lN/N eHQB8YWgHLehL8GusjKbxBsc60KV3OMVRv5+KyaETXeoUXX9j4McyXCEaPkbXT+pAVneeI QIo8xmTScm772bnbBNl5nHwoqorRETkJKffNZ6f40vJ/4UY22exgyqwM/wyewQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ndzPV+KTt0XzydVGeObWOLfmNe4sV+QpfiiUpyraAg=; b=gEaTy8k+UeiLTYq8M4pK9aVOhsiw88tdR3F5jM7tqK3wQRMcl5vQgHp9XLAOw+1WTVDKxw WN8aKGqD4x3gUABw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 17/32] timers: Introduce add_timer() variants which modify timer flags Date: Fri, 1 Dec 2023 10:26:39 +0100 Message-Id: <20231201092654.34614-18-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071220017540179 X-GMAIL-MSGID: 1784071220017540179 Timer might be used as pinned timer (using add_timer_on()) and later on as non pinned timers using add_timer(). When the NOHZ timer pull at expiry model is in place, TIMER_PINNED flag is required to be used whenever a timer needs to expire on a dedicated CPU. Flag must no be set, if expiration on a dedicated CPU is not required. add_timer_on()'s behavior will be changed during the preparation patches for the NOHZ timer pull at expiry model to unconditionally set TIMER_PINNED flag. To be able to reset/set the flag when queueing a timer, two variants of add_timer() are introduced. This is a preparatory patch and has no functional change. Signed-off-by: Anna-Maria Behnsen --- v9: Update documentation to match kernel-doc style (missing brackets after function names) New in v6 --- include/linux/timer.h | 2 ++ kernel/time/timer.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/timer.h b/include/linux/timer.h index 26a545bb0153..404bb31a95c7 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -179,6 +179,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires); #define NEXT_TIMER_MAX_DELTA ((1UL << 30) - 1) extern void add_timer(struct timer_list *timer); +extern void add_timer_local(struct timer_list *timer); +extern void add_timer_global(struct timer_list *timer); extern int try_to_del_timer_sync(struct timer_list *timer); extern int timer_delete_sync(struct timer_list *timer); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 177bcde8a2c0..0ce0e6b25482 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1245,6 +1245,40 @@ void add_timer(struct timer_list *timer) } EXPORT_SYMBOL(add_timer); +/** + * add_timer_local() - Start a timer on the local CPU + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is set. + * + * See add_timer() for further details. + */ +void add_timer_local(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags |= TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_local); + +/** + * add_timer_global() - Start a timer without TIMER_PINNED flag set + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is unset. + * + * See add_timer() for further details. + */ +void add_timer_global(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags &= ~TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_global); + /** * add_timer_on - Start a timer on a particular CPU * @timer: The timer to be started From patchwork Fri Dec 1 09:26:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172356 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991978vqy; Fri, 1 Dec 2023 01:28:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IHOhbPfyQbT3Y90MKPCarsV2/Xk7nfNx2/bdEv2KV76/F6lm2dj//ZTAMCNALLoGuSoMGk+ X-Received: by 2002:a05:6830:453:b0:6d8:328e:faa2 with SMTP id d19-20020a056830045300b006d8328efaa2mr2678498otc.26.1701422915323; Fri, 01 Dec 2023 01:28:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422915; cv=none; d=google.com; s=arc-20160816; b=Jn+MLAI4D61Q4zO52CHW8NgxAlS/Q/2NhEb+Z6hlmVxowUcXzyYVRDS+uBoHswnPxj XV0+z2RzLTwBY+OEmg7BYZs9SuPXyzu8s1812Nloj3o2oKXBXO3KZ7J1w7R7XRV5DGoI HzyFJiBuwZyjqv82tsxjF7aT48sLJ8m1Kpizbq7zeK3+LAYodk+O2b+yEL8HCap1JS3t K+T16fvwptpIt+Cw36IjLFnNDYbgl6n0BBgjUj32pn1aH2QtOHnEHvCfNZUm0EelSwyT vG2gEO3n47z17yFpDPc4K/4/GKvwJ0sNJTYWO76a9jMHB999ex5R+GaVulCxUtQ7IGxu L5cA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=fQra51vM0+AF7Pw741oQBrBk7Zt04iCrGLsTZoJApiI=; fh=6m6KAu2D1n2eK6jNJ9BIoRiulvR238oCr7uvY5/A720=; b=ZxfpBNlXvSvFPpYtlJyjufmM/Rgc1HgVuILwRosMPik0a95Uhy/mg2M+97EVTrjH/j iBR1fsNOuAujwB0okD0KdtOtSDlk1yuNfflhShwBtCO3gDjMDJLI5e0CYHwu3ZfpB0i1 jplkmqRXihnFeaxEOY1+9d8ko9Ug0AXgoQWT9zsNSGralI9vm9drEwf+5tyPOSNwfrjd SQtrclWWouAFdLKn4xoh7tZEzIoMEzQDyWh0pbcHTN9Ebf58BjChry+vLB/yTSyh0dJn BFgxH6T3fi0L9T/dvLUT2UpL9V/AFjQTYYiewueAVZObhNDV2tsObORbPYRfxr6jPHbn ezKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N8IXtw37; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=R5Qn2IaG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id f22-20020a63f116000000b00578ea9a0b93si3027008pgi.890.2023.12.01.01.28.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N8IXtw37; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=R5Qn2IaG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 96C61815CDA5; Fri, 1 Dec 2023 01:28:32 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378115AbjLAJ2K (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378110AbjLAJ1j (ORCPT ); Fri, 1 Dec 2023 04:27:39 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A696199F for ; Fri, 1 Dec 2023 01:27:20 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fQra51vM0+AF7Pw741oQBrBk7Zt04iCrGLsTZoJApiI=; b=N8IXtw37hnuRY5f/tQtu7mjNOGcU/i3xD1XD3YN/HcZkRvbEDJTCxJEnzHY59YsAQVlW09 M5V5N1Gaw3UDmlFJvd6Z3vXYCUSobPSzHupk7g2MtzfvPcf0yqMH0bkXkUO4BKdnWaaWag PfqxXUIVhTJ5BMnzCgx3VYAMPd+H+/KSwsWZN44BekJmTMFcHV0S4mjzW1xWSg8njws7ZS je0YZRUi4Hb2chqYKXYLxJo1xR7MWsuTgVXgZs6Gmn/tk8VTmsCEyYfpAlpj+s72ofJ+N+ 3u+B/K5EqYxi8Sxy6Nwv8i8Y0C4kuRCAJObBt669iC2PHpvmplNq5EJtRupBAg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fQra51vM0+AF7Pw741oQBrBk7Zt04iCrGLsTZoJApiI=; b=R5Qn2IaGcKetkUnTnZUQoeXbKXVdaaPRJW0YVxJFh572/+9QYRhJ06YDFf7IlhLPVjNo0B I8kGucuTQGt9uwCw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Tejun Heo , Lai Jiangshan Subject: [PATCH v9 18/32] workqueue: Use global variant for add_timer() Date: Fri, 1 Dec 2023 10:26:40 +0100 Message-Id: <20231201092654.34614-19-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:32 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071234714988913 X-GMAIL-MSGID: 1784071234714988913 The implementation of the NOHZ pull at expiry model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be dropped. This is required for call sites which use the timer alternately as pinned and not pinned timer like workqueues do. Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker Acked-by: Tejun Heo Cc: Tejun Heo Cc: Lai Jiangshan --- v6: - New patch: As v6 provides unconditially setting TIMER_PINNED flag in add_timer_on() workqueue requires new add_timer_global() variant. --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 6e578f576a6f..3a1360794137 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1958,7 +1958,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, if (unlikely(cpu != WORK_CPU_UNBOUND)) add_timer_on(timer, cpu); else - add_timer(timer); + add_timer_global(timer); } /** From patchwork Fri Dec 1 09:26:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172355 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp991959vqy; Fri, 1 Dec 2023 01:28:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5ufW35eftyNCEytzOziyt2m7nLbNsxa3SOhyZ5uYRWpQhdvJejByo5UoygydteUvfJnER X-Received: by 2002:a05:6a21:998e:b0:18c:a9d3:4f96 with SMTP id ve14-20020a056a21998e00b0018ca9d34f96mr18627123pzb.32.1701422912554; Fri, 01 Dec 2023 01:28:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422912; cv=none; d=google.com; s=arc-20160816; b=VGd0feTPqvhNk21mhWoEKwvz6jwuVCU+SYhvmzUpjyhl47nnT0owwRrvGds0klrWl5 LqPgtorx62u7sMUrEmH9ifIgpMnjqQI+gqR8kPzinXMSyAk4dXO0G77pdQu0OcOcskaP 0kLPoKwz7SzKIN4YPf9EhMEyYFhaOLnmhHFz/ziZEdiUT+FyC1/GCrhX+V1tsGsAjCmv YHmHLjIwJ17w8sO3VtyHImT1lvjjuxhXMeZaOBOkAp2xjnBWxgrVJ1dayAbSIrLQT2dj /P0so760fd6IXHepb/ZCurjRwjx4l0tIhCCtRb9jZ8wG4KhmPL9RzoT6ZVNTCYvhz60A FIrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=1gYaVFuPoy5UW7Ox9DtRT+q6BrvDn/y7/UpgVZp13oQ=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=FoXj0jo04RZgSUvDznRuFXh7Hl7lGRHxMpK/VlhFYZzZjB5PKchNAFvMvat5G1kOIu 7Nd2R7xF0bIRIz5EiQuefg6Au56dXZsKxL3N4SaTz1YZZC43fWR1xXxE2D96BsdzzLN/ xCyyUgPF/ELydRN5BfW7IRV13EzlCkyoqVuHuSHDL5Y1LdP+ZgqDnxCUhrVeEmHy4g4N qZ1GDNma6wkCNOJWYkSZs28wPusHCmAi6TsG3dx9/NA4lxMdOtepKBa9xXXiavObkWSH vtQF0zbaFk+DHq4MrweguFen4gFc9fIrQWp0FpZwSm8Rvuxwak1dYIjRV+ucJ6fQ/X91 1PIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="PAq/TcXi"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id gc21-20020a17090b311500b00285b97c7d6fsi5377878pjb.141.2023.12.01.01.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="PAq/TcXi"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 19D38815CDA3; Fri, 1 Dec 2023 01:28:30 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378205AbjLAJ2R (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378145AbjLAJ1x (ORCPT ); Fri, 1 Dec 2023 04:27:53 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94EEC19BA for ; Fri, 1 Dec 2023 01:27:21 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1gYaVFuPoy5UW7Ox9DtRT+q6BrvDn/y7/UpgVZp13oQ=; b=PAq/TcXiBr3vZ/bkP9wgwgb+f+unPFLW+vAXzRt+6k89zhx2iT2ghtEO3DDzYolFQNZOSV Dca4vwfGYpZBDms4ilYjqB8hCoVv4jZuEZgU0QzHrKNk/pGZQgajCQo0/Aek2w1bmQfPC/ 5j4+ejwW8gzdI+22p3K0gko4dopIc4xJmfa4JJaKSzGLhapw4mozgr5/AE8c84BjKIWsFW +0VqVw+gdHK/+qHoLWNSRGcBUSLz/GJFhC9yJO/P42yvOvG6j1Jx5SEdqYP1ysDymvxjUo YmERn21/1bO1ajNJRbIUovB2JPf81tx2v0ls0z/9nr6Djonmo1rE3Eqao/xqcQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1gYaVFuPoy5UW7Ox9DtRT+q6BrvDn/y7/UpgVZp13oQ=; b=CMqrDpwUb7eN+9HpKDodq8mpQY1W2MPkSZ2JjOUoBa5afMYeE9Fk/f8rGRM2JQgxmB4BPk YINYi1ThYLanF4CA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 19/32] timers: add_timer_on(): Make sure TIMER_PINNED flag is set Date: Fri, 1 Dec 2023 10:26:41 +0100 Message-Id: <20231201092654.34614-20-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:30 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071232353346232 X-GMAIL-MSGID: 1784071232353346232 When adding a timer to the timer wheel using add_timer_on(), it is an implicitly pinned timer. With the timer pull at expiry time model in place, TIMER_PINNED flag is required to make sure timers end up in proper base. Add TIMER_PINNED flag unconditionally when add_timer_on() is executed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0ce0e6b25482..ea94479ee7e2 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1284,7 +1284,10 @@ EXPORT_SYMBOL(add_timer_global); * @timer: The timer to be started * @cpu: The CPU to start it on * - * Same as add_timer() except that it starts the timer on the given CPU. + * Same as add_timer() except that it starts the timer on the given CPU and + * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in + * the next round, add_timer_global() should be used instead as it unsets + * the TIMER_PINNED flag. * * See add_timer() for further details. */ @@ -1298,6 +1301,9 @@ void add_timer_on(struct timer_list *timer, int cpu) if (WARN_ON_ONCE(timer_pending(timer))) return; + /* Make sure timer flags have TIMER_PINNED flag set */ + timer->flags |= TIMER_PINNED; + new_base = get_timer_cpu_base(timer->flags, cpu); /* From patchwork Fri Dec 1 09:26:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172358 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992033vqy; Fri, 1 Dec 2023 01:28:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IG1/Cy9HI/rhZbHh7t5J6jkIWlOECkFsvgxSMRdiRvAEXeLnto3HiREup+VW97A0XmTa2lz X-Received: by 2002:a9d:4d0a:0:b0:6d8:2843:8887 with SMTP id n10-20020a9d4d0a000000b006d828438887mr2200368otf.34.1701422923889; Fri, 01 Dec 2023 01:28:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422923; cv=none; d=google.com; s=arc-20160816; b=M3gqpUEerIvu5+sZ2yD1ozOg9e9c1YzSv3cJyHwy3J8NOVO6dMaqHjQgRoQhq5DlO2 5YtX5fsGmEk7vUzYziMpuzVnyVbdbQrtunPCM1rbJijcNGi2FSKYPfGSvcPru/eSBcob dQQnz4SjVXkaB9HbisQuf9tInAz/kkbn2P/t5oefkAV9G1DKnHLk9KrTNCFwzQkjYO5R mqCJdePLKL7lbxZM08Ld39P2wAnfaQ9Pq7gr65L2PpBQ5Tne++aLYNG3Pz1Zc+1TLp2F w752K+zH5j2jiez9F7RscNX6e7ytxQGREp1fiXs06BmXErhrEX1PO9tdcQ/GM9343B7h KxCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=HptRrZ5M792iqw/iN2UweCexRRIRvwfiQKQYLI/3wHc=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=wZIMc+8uHJ+wWSwNrNzhNJfE1DEWLZ2ntiqzk3Qo+KKA2phP1aRYk2E+uhifQHyruz MeKVpEm1YRIhOo3vQNyc62t6Mm8PJEJS5G4KAkKxpe8vXxR8bcgBw9MLeiYE68shNlLU yKzdN7tb4/3Jri6PUHaz+bZVwkafW88vSCZiXS7/oPf4eVMkUHQO8MkL/PSPPAwzCDsp xNP518I7Fi28OPmYdpq123kTtieW/X7go9Jfc0LayBbxTu5PvOAmxJhiJM7GAtT/rUY2 8g01b5geGqF0jWY2qwZYz5GTuLCBIu3/Uhf0zH1OtNZqPsffwZSZMrmoUpaEqkHjJEaf qpEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="yTBAHA/p"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=L5L6rigw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id g4-20020a635204000000b005be18ddf8b4si3087418pgb.646.2023.12.01.01.28.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="yTBAHA/p"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=L5L6rigw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 248BB80EAD53; Fri, 1 Dec 2023 01:28:37 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378227AbjLAJ2V (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378144AbjLAJ1x (ORCPT ); Fri, 1 Dec 2023 04:27:53 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A161BC0 for ; Fri, 1 Dec 2023 01:27:22 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HptRrZ5M792iqw/iN2UweCexRRIRvwfiQKQYLI/3wHc=; b=yTBAHA/pD9Zi3YxjUh/mQPwH80wDQzu4NBOMSc/YlI4oybqc9oa+avtTTMz8Kf+WValTlA dlgISBVCf286D9fbca7mOEhZHUF/fAFl+SrOGDRZtFGcwqMIAvDyHD2cBjo+ebBS1CLvEY BXKYL4sowwjTmrKc4/uWkT8jC/FBfRqYPQGcgHMkXH8GrcgMASx+z1ttb8dMp23D7Ifp86 dwNPYPaYfxtbVjvmKlyr213mfH7G4EKdLJTnLXs1a3ogr5IK4ts3I8LmE4F7Oh8nBih1j6 8/itFyFnOyam5q9D2UY7ueoJkVx4/vtYSK3ZE08jUy2PONXP1WmfA9cIoISLVw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HptRrZ5M792iqw/iN2UweCexRRIRvwfiQKQYLI/3wHc=; b=L5L6rigwBQq2b2USGQ9rEjf9ZzpKMx4OAE8/3HInusvYXAisr+mmjXdoDjX4CVFp3WP+ma yfayh02N64lbiAAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 20/32] timers: Ease code in run_local_timers() Date: Fri, 1 Dec 2023 10:26:42 +0100 Message-Id: <20231201092654.34614-21-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:37 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071244168657898 X-GMAIL-MSGID: 1784071244168657898 The logic for raising a softirq the way it is implemented right now, is readable for two timer bases. When increasing numbers of timer bases, code gets harder to read. With the introduction of the timer migration hierarchy, there will be three timer bases. Therefore ease the code. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v5: New patch to decrease patch size of follow up patches --- kernel/time/timer.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index ea94479ee7e2..b14d84f1fe50 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2132,16 +2132,14 @@ static void run_local_timers(void) struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Fri Dec 1 09:26:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172367 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992312vqy; Fri, 1 Dec 2023 01:29:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IHr+9WnzkTZxYk8lAirW7qvNpKENvEAKXipzcz597nAuffEvCWZRcJlSgOJuZtWDEs9gtZI X-Received: by 2002:a05:6a20:e119:b0:18c:3260:e221 with SMTP id kr25-20020a056a20e11900b0018c3260e221mr26960880pzb.62.1701422965831; Fri, 01 Dec 2023 01:29:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422965; cv=none; d=google.com; s=arc-20160816; b=VD44VaTne83Cj0rp5uwppw8YeKixCfdpcejl4H4QiacIoFJtdgkPja0D2xpIOhZzE7 IBk7se9qylHjucVFaYg7JR+WrcXzUBDDHbDdcdQquZQjf4KSzPts/VgV9Oio/41Nbb8L GQm05OmV2d8/9h6gLL1uENciIr1qucwgO7KtmqGk8eAXCCXm3piZFY02O8mtQAxlvUdi 6l0SJFPLzAqNlnhzM3TlP6xnJ/gK0qJZ35ZtApsuZBO2oZzs12+DOLm4c9rRoyyTy8gW H9cdkfYkdVPT52O208+PSxUZvZVsKynaFhuwycF9AthHBmYz0zpRHrMFFWTAwLRXnHf6 /+bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=j1C4FMe7pCnDmgmGqNMXsgIVbPWX+pQUl8aT2cvTpxE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=JGKlG66jHudN1S8o5Id1qY30OOwkKntJTxb7xVlBExkZKyb7IQfN+M9Qypc1WtvEf1 IYqRQ83dhgRZFyOQ2S0ir4Odu81qHSalG2lEM8Y0clyeoyJqkgav8p/tGyd08rBlrRxJ hYMkQ2AqOA9ozd64xus2/Vs6xTXdjFaubu2+iZ+xaTJB264Tbn0u0maJyysjUezW42YM V3oeFkVzPknvGD6cDI1aNj/rpFZ60IpIgENFGacVYSblcdiItXLyq1yOmbJtCQI80iAi ubLIkyX9EhbqVpzWS94eIPLdcgJolfi0z73Cpk1+uloVyaEYwWnJHqk5STjZNbTz7cpx WCpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=F20CyeVD; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id k18-20020a170902c41200b001d04933efddsi1901955plk.39.2023.12.01.01.29.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:29:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=F20CyeVD; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 9186E825F173; Fri, 1 Dec 2023 01:29:17 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378069AbjLAJ2X (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378151AbjLAJ1y (ORCPT ); Fri, 1 Dec 2023 04:27:54 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E40561BC6 for ; Fri, 1 Dec 2023 01:27:22 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j1C4FMe7pCnDmgmGqNMXsgIVbPWX+pQUl8aT2cvTpxE=; b=F20CyeVDuw7jm58dTRpNRqAbUkr6ff/ZKa8JWcMJjS1xEGOEeyHQaEWSIyjb9dtKsJPOl7 J51VieHtILw3lrPjVx/SxFIrkBv3xVrmzu018wirkiPPuCMkentQlQjSk0LX9NJeQXsS2a wiXf9rUilA3GKSDN/as4IYPlBFlBhTLb1VnnECsJcr8VK398l1Si0a67e1JFxOfsB7gxRw HGSINAF+Bw0YFNzOQTmLcxncwk05bTQmawg21ZZ8DFGSw+gUqURM0GR5+jlOxTabKjFv1s mMLPgavKnCIkKjeowAwjCsW9Xe46YZ1AfAWpO/dpgDEj8y0jJyegvsvCsPTEgw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j1C4FMe7pCnDmgmGqNMXsgIVbPWX+pQUl8aT2cvTpxE=; b=zimkCzW5Hc5BNNqsTiWhrRNjS1Qz+YQP0rnArDywmOir6z0bmDO27jgmpr+TsxqqusG/PV rfRDRQ7UyL85FzDg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 21/32] timers: Split next timer interrupt logic Date: Fri, 1 Dec 2023 10:26:43 +0100 Message-Id: <20231201092654.34614-22-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:17 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071287828070763 X-GMAIL-MSGID: 1784071287828070763 Logic for getting next timer interrupt (no matter of recalculated or already stored in base->next_expiry) is split into a separate function "next_timer_interrupt()" to make it available for new call sites. No functional change. Signed-off-by: Anna-Maria Behnsen --- v9: Adapt to the fix for empty timer bases. --- kernel/time/timer.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index b14d84f1fe50..eda4972ca862 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1951,12 +1951,29 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } +static unsigned long next_timer_interrupt(struct timer_base *base, + unsigned long basej) +{ + if (base->next_expiry_recalc) + next_expiry_recalc(base); + + /* + * Move next_expiry for the empty base into the future to prevent a + * unnecessary raise of the timer softirq when the next_expiry value + * will be reached even if there is no timer pending. + */ + if (!base->timers_pending) + base->next_expiry = basej + NEXT_TIMER_MAX_DELTA; + + return base->next_expiry; +} + static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; u64 expires = KTIME_MAX; + unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1969,24 +1986,13 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, } raw_spin_lock(&base->lock); - if (base->next_expiry_recalc) - next_expiry_recalc(base); + nextevt = next_timer_interrupt(base, basej); if (base->timers_pending) { - nextevt = base->next_expiry; - /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - } else { - /* - * Move next_expiry for the empty base into the future to - * prevent a unnecessary raise of the timer softirq when the - * next_expiry value will be reached even if there is no timer - * pending. - */ - base->next_expiry = nextevt; } /* From patchwork Fri Dec 1 09:26:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172359 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992050vqy; Fri, 1 Dec 2023 01:28:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IH/wFjbFOr+NtAoMvXgY99oYmHsrajAUIoM4kD2uyVISu1MpbAyMbprdvVvgCMf+yxpV0vd X-Received: by 2002:a17:90b:38cd:b0:286:56de:3ff2 with SMTP id nn13-20020a17090b38cd00b0028656de3ff2mr1397975pjb.48.1701422927373; Fri, 01 Dec 2023 01:28:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422927; cv=none; d=google.com; s=arc-20160816; b=KdWFkrOTjnIPVV1aNNffceVCVLrBVNKQnP/v/0Oj+3aoaSbiFZuZ5LfSFntfS0/vC3 RhS2KEwCajmJJrYoBmA3Mc5sBhhB56sr9VQPneXkDuq4lG9oT8DYnylfJcE3rqFu2P1A iyppdwOnZrYlRmPDvOSfxtrtX4K51A61Sxq4Y/54xR/Biw7GML9F+DoI9G0GqQOZQN7w d/nslVyXhIobnPiXKWlpH0Rl3wT43973mXscoEpnWjo6dAvP8rMncGikkpvsRnjvsN4t +jnpCzeYwoJMFNIPcwlh8mb4xuuwJGv+3Ejfetl8xUm7HHc5b1xXr4IUDCiwS0T4R+7g TV6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=+2gcknKks11qfAlzDRSwTUEgmadWT0lIiDr7uy6c0kk=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=TfHngNN4Br+6NYCriBcn8JBmv9pf1XB2qX4TWWuMCBSwquAPX4KjLWuV4pskiQPDsO LfSQ0wbSktMh3TOd8eq6Xw/0fy+4OAdUC3NcurFgTuruLzuEE1CReZpJmqyU5R7q08Wg hdZKKq0XEBm1lA52R9GloqthXhyuF47F7w5rUp4S9u/ENTjLZ2cr0JKpVEtqIXfjO/jR UOikzrrxoHHHdel8Mod/uyC/qki3fS0Au66jr5ikr6jjC9VSahLXEkQaHBfrk9nwRbM/ 9QUskON+0GiEiyoIsmPb64u453MValUPZ9SEvOke8NmHu2ho7l97H6NkMt3MsvRz8ktq 0tJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=LEMhqb0+; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id t3-20020a17090abc4300b002800b0fa4f2si5339242pjv.104.2023.12.01.01.28.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=LEMhqb0+; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 3813780ECF38; Fri, 1 Dec 2023 01:28:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378175AbjLAJ2d (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378166AbjLAJ16 (ORCPT ); Fri, 1 Dec 2023 04:27:58 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C2841BD1 for ; Fri, 1 Dec 2023 01:27:23 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+2gcknKks11qfAlzDRSwTUEgmadWT0lIiDr7uy6c0kk=; b=LEMhqb0+RZ9Gahc8ukopEQMLtEPRI62P0m7xNOrYz0rhqfu+vWZo3RG2EukSZ+b4aILNJN Yt7OSurt0uLEc1ScjlC+583KJOOXDkZpgyuITxu/0HE7H6ZY1wdl4hIwJD8jLzv8LFe797 vrWwKxDyq9SNTbLfK4c8A+vfwHQEtEhwfgCGWHf8LcG3R84VSm93dQpSpDEBmc04ResOKE NGiZCg3KbbEMTPf+wYbgUa4qbpDoI4P5Ib/CjgYZ/aBidwJm1jF6MfNV1NiAPAoaOrO2eO Wx3EAMeQ+zJ6mrqwRL8qQW47qAPRiJ7HwhISN9FHOsnhCS5FmvfsfEhQTc/baA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+2gcknKks11qfAlzDRSwTUEgmadWT0lIiDr7uy6c0kk=; b=M+AlmYf4Cp0yyTXzsnGSUkZHOjn1+YGbV8GtxZbglGJrFRSQu0jnF674YBl3aeatoLgQQR V1wLFUJDfAwiRHBQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v9 22/32] timers: Keep the pinned timers separate from the others Date: Fri, 1 Dec 2023 10:26:44 +0100 Message-Id: <20231201092654.34614-23-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071246965139973 X-GMAIL-MSGID: 1784071246965139973 Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: - Update was required (change of preceding patches) v6: - Drop set TIMER_PINNED flag in add_timer_on() and drop related warning. add_timer_on() fix is splitted into a separate patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker v5: - Add WARN_ONCE() in add_timer_on() - Decrease patch size by splitting into three patches (this patch and the two before) v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 95 ++++++++++++++++++++++++++++++++------------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index eda4972ca862..366ea26ce3ba 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1971,9 +1983,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt, nextevt_local, nextevt_global; + struct timer_base *base_local, *base_global; u64 expires = KTIME_MAX; - unsigned long nextevt; + bool local_first; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1985,10 +1998,31 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, return expires; } - raw_spin_lock(&base->lock); - nextevt = next_timer_interrupt(base, basej); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt_local = next_timer_interrupt(base_local, basej); + nextevt_global = next_timer_interrupt(base_global, basej); - if (base->timers_pending) { + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + nextevt = local_first ? nextevt_local : nextevt_global; + + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; @@ -1999,28 +2033,33 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * We have a fresh next event. Check whether we can forward the * base. */ - __forward_timer_base(base, basej); + __forward_timer_base(base_local, basej); + __forward_timer_base(base_global, basej); /* * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ if (idle) { /* - * Base is idle if the next event is more than a tick away. + * Bases are idle if the next event is more than a tick away. * * If the base is marked idle then any timer add operation must * forward the base clk itself to keep granularity small. This - * idle logic is only maintained for the BASE_STD base, - * deferrable timers may still see large granularity skew (by - * design). + * idle logic is only maintained for the BASE_LOCAL and + * BASE_GLOBAL base, deferrable timers may still see large + * granularity skew (by design). */ - if (!base->is_idle) - base->is_idle = time_after(nextevt, basej + 1); - *idle = base->is_idle; - trace_timer_base_idle(base->is_idle, base->cpu); + if (!base_local->is_idle) { + bool is_idle = time_after(nextevt, basej + 1); + + base_local->is_idle = base_global->is_idle = is_idle; + } + *idle = base_local->is_idle; + trace_timer_base_idle(base_local->is_idle, base_local->cpu); } - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return cmp_next_hrtimer_event(basem, expires); } @@ -2064,15 +2103,15 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); + trace_timer_base_idle(0, smp_processor_id()); } #endif @@ -2123,11 +2162,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -2135,7 +2176,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); From patchwork Fri Dec 1 09:26:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172357 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992026vqy; Fri, 1 Dec 2023 01:28:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IFlGFcySYzBXdFZ0/zEBJz0Z9eZh5BkLG/BZiYOXzprY8ZeHc6dwQ7aiq6i8bG3gopGRp8H X-Received: by 2002:a05:6a00:3989:b0:6c3:1b90:8552 with SMTP id fi9-20020a056a00398900b006c31b908552mr26241198pfb.17.1701422922204; Fri, 01 Dec 2023 01:28:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422922; cv=none; d=google.com; s=arc-20160816; b=iEUOfA6d0A1WChgbNu7nUW86pfbhwraSeTeKpb6s7NJGvzeoPc1KHePDqllZ6tAQJg BqhHPFvlnTW95booII7ofnLzb1EktBqzgN8bq1QyEWGlxIOeuhy/hCa64bpTmzIXwqrD UEQr1oNiocZMqyu4/Hq9kiRxOSmH46iT+oAh8bli106DcbxBBm2Rt9KDyE7szrWX68kj jX2f4NtcKBMr7jFylIkVsLTWmlRIvWttXE36X1nLE8O6PYk2+MAeeUQyCULUfw5Y6CFC kyDzbojh252inJGXVPL750HK1QNE7wYraXg8s7GF2zQ7cvNrSD2Pr5QVhNOCE6saBVPG hdgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=IdBDR0/tYNJhJm6cSGNxPr1uzJXFKTmkB6lItnLpMig=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=u9ccqCIQs3wwTcxx4nbf2lYyJhAgwoD48E2nvzK5QfBz8Jrx69VJdT76wvTwWwzilF o8I4GfeXj7HPSr7F85MVTo3vEalsSCgddAsraTnXUO+md469FGJfs+vzUNHm+jmbIQFq dnEuY13Z7oiwvM4FpbgmRq+AU5y0Yd9vHl02m2sGN0Skk6aCfz739SdLHSxCGr9AZuNJ Q1RK8ltPyzrXKJ0rO7Yn7lJ/1rf3hyYUSEPD0oIsG6WDma3ujBXTjQKro223LALWaXgR IaJO1BH6hbZhgQgWNEMFHZx8osl2UIpUd1ed2jvjOnKcFXIFiBacoVFcvx56lggEsa4b PeKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=3VH7dqj9; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=ObRXt5dM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id f7-20020a056a00228700b006cdef1e9ab8si2545617pfe.245.2023.12.01.01.28.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=3VH7dqj9; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=ObRXt5dM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 7ED28815CD84; Fri, 1 Dec 2023 01:28:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378057AbjLAJ21 (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378167AbjLAJ16 (ORCPT ); Fri, 1 Dec 2023 04:27:58 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 439561BD6 for ; Fri, 1 Dec 2023 01:27:24 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IdBDR0/tYNJhJm6cSGNxPr1uzJXFKTmkB6lItnLpMig=; b=3VH7dqj99SCQxtKQ1cSHqIAwzHimk0La70Y55WoifDDH6FVLlQTD7yElldyys8j5ak6Ljq YM2qwCAcIoxkO3hle8SiRG4mhPdiNBC/EL0AwtvgMX4Kqp1sbL/KJu9cLEaFC1rpO7xrQr D42gEzHFUKk9udutpeiMZHyEuXmMHcC/OsdsNl1AYyKkSb8mRAHkzDl6n6TU3xEWUhsY3K usjBau79n5bQPwp4uEWs/boCxqsThIyOccM08vg6aBAC9p8HXqfiG9EV+O2fxtx/1XSuGd 0lItm/ND1gNEZOUCslMqCpUE3+It9I2+8K80cO721GJVXTEDDgT7kZwG9JbByA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IdBDR0/tYNJhJm6cSGNxPr1uzJXFKTmkB6lItnLpMig=; b=ObRXt5dMA2n4hQG6FSjAmMYBdk0L7rGzSvM/iGKlEi37gmbtj9rRCfhmDpjRCr0+YuSwYw jMbfdkMFH/57TNBQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v9 23/32] timers: Retrieve next expiry of pinned/non-pinned timers separately Date: Fri, 1 Dec 2023 10:26:45 +0100 Message-Id: <20231201092654.34614-24-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:39 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071241643567206 X-GMAIL-MSGID: 1784071241643567206 For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have separate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Update was required (change of preceding patches) --- kernel/time/timer.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 366ea26ce3ba..0d53d853ae22 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,6 +221,11 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -1983,10 +1988,11 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; - u64 expires = KTIME_MAX; bool local_first; + u64 expires; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1995,7 +2001,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, if (cpu_is_offline(smp_processor_id())) { if (idle) *idle = true; - return expires; + return tevt.local; } base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); @@ -2022,13 +2028,31 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, nextevt = local_first ? nextevt_local : nextevt_global; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the @nextevt is at max. one tick away, use @nextevt and store + * it in the local expiry value. The next global event is irrelevant in + * this case and can be left as KTIME_MAX. + */ + if (time_before_eq(nextevt, basej + 1)) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; + goto unlock; } + /* + * Update tevt.* values: + * + * If the local queue expires first, then the global event can be + * ignored. If the global queue is empty, nothing to do either. + */ + if (!local_first && base_global->timers_pending) + tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + /* * We have a fresh next event. Check whether we can forward the * base. @@ -2058,9 +2082,12 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, trace_timer_base_idle(base_local->is_idle, base_local->cpu); } +unlock: raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); + expires = min_t(u64, tevt.local, tevt.global); + return cmp_next_hrtimer_event(basem, expires); } From patchwork Fri Dec 1 09:26:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172360 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992082vqy; Fri, 1 Dec 2023 01:28:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IGqHS4jeIkZ312k26JxI+M5VF8P/8v6u3n+ThKiXJCK0VlDBFi2T6V9XMQ/etLTHzhI8UfL X-Received: by 2002:a05:6870:d1ca:b0:1fa:f19c:6803 with SMTP id b10-20020a056870d1ca00b001faf19c6803mr864638oac.16.1701422933027; Fri, 01 Dec 2023 01:28:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422932; cv=none; d=google.com; s=arc-20160816; b=b6eZoUe+x5eEt+QXGc3ZrNev2SCsubePkNjeZx9PuPvJQcwBpgGv4mMkyhwjr5mkBq pb5Cod5m0didcRCP6fOABduKu0JcmGg0XK3wL6j27m6AikPsLEDMTVM6p9v19/RNSZ4/ HsIwhbmSOFytOlLpKcXjq8YvO5vgrg0j+wdjcRkeCvUxkWpbRQR/UQQwREvP0gd4k/4U CqJ7kYSswhCLaA8HIwLjnGa41sfWkjDHjYTDdYfrFmssyeOj+FuMvyD+J6ncByUJ/C2/ IHRoeuM7Z5jfKNAVUPYUrfSS0UPRuIlscwh24uJTniAP1zKoxDupclu1Kazoc80Mxz/o kPdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=Uvz4yBwpNSi6fkT+oXIwoWMRBUwA4W1/FjhD6j61hx8=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=Obn6yJqIXLOdb7ejf5tbrCbCMtO2rxqfDGANUcyzkO55PY7i926uYknJ6u+CXwAZax xKDnJGlHe9M2fmGC1xVE4PafKyclZwcCh2RyXrwTpShOcrUL9/dPLMm1ELtnD9u9DfZ7 JpNRjOyoXUPOD5m+C1y3l7GWTrtbbO1kzZSqCdDzGggixSWVBUH/F0RZPNy8ZpBHSzId O0sjp3MSDNN4AxB9jApPMU0+Pv4AC0BcYGeIcMgRxp2i8FL640A7b4An+wljKCFwAxFL oufK16QvZK2hRPlCKQMCs63usY54dcCZcFvwXkPJcebfJhmt8iljwBNWAJGg+bfyWCJS PFiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qMsRwsr+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=L2MRucXc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id u22-20020a63d356000000b005c5e24c7e8csi3183932pgi.414.2023.12.01.01.28.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qMsRwsr+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=L2MRucXc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id BBD0880FA854; Fri, 1 Dec 2023 01:28:51 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378181AbjLAJ2i (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378025AbjLAJ2I (ORCPT ); Fri, 1 Dec 2023 04:28:08 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 530B51BE2 for ; Fri, 1 Dec 2023 01:27:25 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uvz4yBwpNSi6fkT+oXIwoWMRBUwA4W1/FjhD6j61hx8=; b=qMsRwsr+tm2NABELd3xGyoIvs9Q3nKC7dA+P1bwnlmoDT+38yK2pUvPofS2GsHFXVt1295 HLaoAEa2mvXXxrwoBYDdvGli/++GZR+ZlkxXl6ewYb2ZbP77HoKfsEkiSiERJVGHfPUfkc sIvtIy0a3bwOVEnQVH/GBNHgGNiouQ70U2o+efM9AwjPrNKIXSnFQaQEn2QGhVLg+OwriT 86PuSupt2sL8tiDwd/TwQjJlybwK5x5Ya1ZGxjsBN4mZucPh9YxFI+GIi1FKFGgieAXDMv BAx2JlsD2U8C5nD8a1aLR8JCCNgq+5CP2wfh9Thj/lLlvXL2ISNL4GV0Go+zKw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uvz4yBwpNSi6fkT+oXIwoWMRBUwA4W1/FjhD6j61hx8=; b=L2MRucXc33ide4+3K4Tl8jxU6vJKiCjEajaGrRk0LYhWnwHLfvl45rw41XC9DgrZmjhbtH ODBBm8bO2hjr5RAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 24/32] timers: Split out "get next timer interrupt" functionality Date: Fri, 1 Dec 2023 10:26:46 +0100 Message-Id: <20231201092654.34614-25-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:52 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071253328026704 X-GMAIL-MSGID: 1784071253328026704 The functionality for getting the next timer interrupt in get_next_timer_interrupt() is split into a separate function fetch_next_timer_interrupt() to be usable by other call sites. This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: Update was required (change of preceding patches) v6: s/splitted/split v5: Update commit message v4: Fix typo in comment --- kernel/time/timer.c | 64 +++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0d53d853ae22..fc376e06980e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1985,30 +1985,13 @@ static unsigned long next_timer_interrupt(struct timer_base *base, return base->next_expiry; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, - bool *idle) +static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_base *base_local, + struct timer_base *base_global, + struct timer_events *tevt) { - struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; - struct timer_base *base_local, *base_global; bool local_first; - u64 expires; - - /* - * Pretend that there is no timer pending if the cpu is offline. - * Possible pending timers will be migrated later to an active cpu. - */ - if (cpu_is_offline(smp_processor_id())) { - if (idle) - *idle = true; - return tevt.local; - } - - base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - - raw_spin_lock(&base_local->lock); - raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); nextevt_local = next_timer_interrupt(base_local, basej); nextevt_global = next_timer_interrupt(base_global, basej); @@ -2037,8 +2020,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; - goto unlock; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + return nextevt; } /* @@ -2048,10 +2031,40 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * ignored. If the global queue is empty, nothing to do either. */ if (!local_first && base_global->timers_pending) - tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; if (base_local->timers_pending) - tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + return nextevt; +} + +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) +{ + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; + struct timer_base *base_local, *base_global; + unsigned long nextevt; + u64 expires; + + /* + * Pretend that there is no timer pending if the cpu is offline. + * Possible pending timers will be migrated later to an active cpu. + */ + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; + return tevt.local; + } + + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt = fetch_next_timer_interrupt(basej, basem, base_local, + base_global, &tevt); /* * We have a fresh next event. Check whether we can forward the @@ -2082,7 +2095,6 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, trace_timer_base_idle(base_local->is_idle, base_local->cpu); } -unlock: raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); From patchwork Fri Dec 1 09:26:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172362 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992141vqy; Fri, 1 Dec 2023 01:29:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTW23JXuq/vxmGh3bkW1TlBDfSMfwEQlEgDh2ZdGNqLquaX1uhIHhFeenjLX546wrbUdrV X-Received: by 2002:a05:6358:c8f:b0:16e:2762:cdca with SMTP id o15-20020a0563580c8f00b0016e2762cdcamr22153250rwj.3.1701422939725; Fri, 01 Dec 2023 01:28:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422939; cv=none; d=google.com; s=arc-20160816; b=QVU+8mNBpdITcYC3uWcKbAsXAkOltCrq8Y/+2Mqp2sfM3ulLPd5Zx7a/G+hrtbcnTP kJshR6rjiJab79DoDZGetmCRlqyNTaC7wDZShjtCkbdNe4ps7PvQnxjZevPbASGZ4gzO QQNfbRX3gOdmlb0TF84nN4C3KSGFWlQywNJvqckQGt1fr0StzJivRG6nbC4A7RFUEsJb ZPYLclOpqn1QJbPvu2a4i0FYCRuPouagkY2ZX67CWHPEBp6CVtw9ZfT3Yv225/pZWr0O VxpN0gTcqZvqxKrG4wgahdQ98Yy8Ytg40j2CZ4EhXGkyqbA6aMSeGcabq773Bncz38A+ 8few== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=EMy/vrIt0wUl/L/Xwvre8ZwMi27M9d4t8qpR3rpCe2k=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=1LqsDlqM2IdcqXZhfeUgRW8ve6l3Y3fwJHjy7zk1w0cKxuEzCZY2NCxv+cTrvK+v94 Hz87DH1NmeX2wV77xFDuXh09ZDFFJBt2unvPbJsWZ1odE5WhrIKd8siai1sTrlfY7+n4 s5DHqjMOto2n1Jfe+EFXxWGaYDjY/Y1KYNj5ya/gcKUVDcA72GTbNkt3iOoSIea51QqN Lf8B5VElcn35CKTNDfyOjTOv+BAMtqcPjFbZuLA7I5dW9ahENcEBc4tc4zbS2xzvQ8UN QTKvHGz6cuu4yZs4mmadjQPax/Sn+pQrVcTUJv6RmG1B8WYqin5ZE6Eh/GarXf8mGfeB dPrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=yCIUgpS6; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id y22-20020a656c16000000b005be03f0da7csi3243687pgu.61.2023.12.01.01.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:28:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=yCIUgpS6; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6941D815F347; Fri, 1 Dec 2023 01:28:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378127AbjLAJ2o (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378171AbjLAJ2J (ORCPT ); Fri, 1 Dec 2023 04:28:09 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2FF21BF0 for ; Fri, 1 Dec 2023 01:27:25 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EMy/vrIt0wUl/L/Xwvre8ZwMi27M9d4t8qpR3rpCe2k=; b=yCIUgpS6DAyFZ8C+W8wpV+zk8mIstno8Qba+0sPG3PSxDJo+5BUB5F473xYkcVjgPP3dqD VXjM6gDKpd56m7SVWXJuVMkXwInZHcUxydH+gKyAWwMriE/h/7KF3lMYHr6DR5no3LxqaZ uu4OqVGV5fFmOBxfkTQvWy47cH74awVqLpGZjR2Mz+lCJ8ZQGLagORM4Agy4kf7/06VM/1 nBPTuvltdE2/XdGcl0pE/+sxVTES6ed2yHTYxln+d084ZGenBbqFLcyBExYdlxj2eDXZ+i aD8o5bbmJBzEW/BEvoH+717FWClTXFQVy1XfaCOY8yunMWNY7Mec/EXorOuv6w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EMy/vrIt0wUl/L/Xwvre8ZwMi27M9d4t8qpR3rpCe2k=; b=LwTgEJB/4Z47VOr90GUt4Dm/KDzYcbxGm3Lp9yeRN6wbMT0NGFec55NSe5xhWf9zfwsoL4 P0W4Qs23JbuKXoAQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 25/32] timers: Add get next timer interrupt functionality for remote CPUs Date: Fri, 1 Dec 2023 10:26:47 +0100 Message-Id: <20231201092654.34614-26-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:28:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071260586274361 X-GMAIL-MSGID: 1784071260586274361 To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Locking of the timer bases and getting the information for the next timer interrupt functionality is split into separate functions. This is required to be compliant with lock ordering when the new model is in place. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v8: - Update comment v7: - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section - change lock, fetch functions to be unconditional - split out unlock function into a separate function v6: - introduce timer_lock_remote_bases() to fix race --- kernel/time/tick-internal.h | 10 +++++ kernel/time/timer.c | 76 ++++++++++++++++++++++++++++++++++--- 2 files changed, 81 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index dc12a938f00f..183ad32330fb 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -155,6 +160,11 @@ extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; +extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); +extern void timer_lock_remote_bases(unsigned int cpu); +extern void timer_unlock_remote_bases(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index fc376e06980e..2cff43c10329 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,11 +221,6 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); -struct timer_events { - u64 local; - u64 global; -}; - #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -2039,6 +2034,77 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, return nextevt; } +# ifdef CONFIG_SMP +/** + * fetch_next_timer_interrupt_remote() - Store next timers into @tevt + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + * + * Caller needs to make sure timer base locks are held (use + * timer_lock_remote_bases() for this purpose). + */ +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_held(&base_local->lock); + lockdep_assert_held(&base_global->lock); + + fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt); +} + +/** + * timer_unlock_remote_bases - unlock timer bases of cpu + * @cpu: Remote CPU + * + * Unlocks the remote timer bases. + */ +void timer_unlock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); +} + +/** + * timer_lock_remote_bases - lock timer bases of cpu + * @cpu: Remote CPU + * + * Locks the remote timer bases. + */ +void timer_lock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_irqs_disabled(); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); +} +# endif /* CONFIG_SMP */ + static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { From patchwork Fri Dec 1 09:26:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172372 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp993232vqy; Fri, 1 Dec 2023 01:31:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5Z+Gu+FVg0f8yBxtZtGpOVBreKCNfZtPoM/CXUUXHCvfIm3Cyfo5+L7Q7MMbZOYWXfC7p X-Received: by 2002:a05:6a00:14c8:b0:68f:c078:b0c9 with SMTP id w8-20020a056a0014c800b0068fc078b0c9mr33578360pfu.11.1701423092553; Fri, 01 Dec 2023 01:31:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701423092; cv=none; d=google.com; s=arc-20160816; b=Olp7rjvgyNLGIXN0A7y4HVO5E4SObBDIq5cEn3iVvnOkttNU9jjsJq2krvQVqNMQsk fFDxRXarqj1o72K9EYwzOSVf0B1vMeDGRp/2pyRoAei9NsIV2BwxaTff9RDODSNk3kzW zmm1AmgvVfw/DflgA1OamieI/64Vs/2h2MFyyM6GXiR4wkGMc+CSr8+j/ipixgFXpgBZ ZaJDYeKYsfVoiJBaKiW6xKQNRSTkMk2DOIzUIrMAAopqe3akPeWL3SsDgns8chF6OMwQ kbcSEsc1baGryNgG7emZPcTcHpyFbOd5OOr2f0E+jA/1eAbkSn7xdmTLLAG1mSmDB1kB 1fTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=bWHMK6r5kBUJ9iV8cP/TU68Vsy3sDvddkiKO4TwUZlc=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=xfQtrpRx06GENA1+vLwDY10Idn7puI3XILwiq1uCw1ViYS5m8/VxMuQMojLJwWIHRH D7bklEsoJxGE76RRKJN3TZj5Mr3in4n+wA13PETckZrTw9zCeAxr4v8zZvbKF0ZOIa2K RO0gjSiR1G+A8d3HqWhm7q7i7N27D0mApV9VKkIK+pjsBhKdoWdvTvoUbKUSy/xeBJsC 9Or1bfx5gKbEfuQzv5q0LbRp8DJsOzy7h2lFrOQfUE5hOLkkGUgOMsP2yWSxlSPMn896 G84zTZ0STZB4pMqKtiMiv5OMnuxDxoh0sHlIkd3UEveVru9jdstfPIlEPExnA+9qOZn3 Q+OQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=QMhs93p2; dkim=neutral (no key) header.i=@linutronix.de header.b=xZTklOBz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id d3-20020a63ed03000000b005a9fb6bf73esi3030608pgi.293.2023.12.01.01.31.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:31:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=QMhs93p2; dkim=neutral (no key) header.i=@linutronix.de header.b=xZTklOBz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id AE79E808BCB7; Fri, 1 Dec 2023 01:30:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378259AbjLAJ2w (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378126AbjLAJ2K (ORCPT ); Fri, 1 Dec 2023 04:28:10 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC6641BFE for ; Fri, 1 Dec 2023 01:27:26 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bWHMK6r5kBUJ9iV8cP/TU68Vsy3sDvddkiKO4TwUZlc=; b=QMhs93p2ZCm6/MJxUMJdkxY/fBy9yYF7JUtsxSHtc9PYue8MZ0SoTHBdgGncOOxORk0ako vL2tnp0171mhN/4n/ReEYLl/a3R6Z/zdScU2QBslAWcu1KSe4vZ+h5iv0arm6+pF4t+FeB NjTheIExH1dDjaJXiRJsT4vg4FMpVT27Wiuihfby106yl2uG6MuiyJfHPXoRVnihspDN6Y SFZd19aQtV1uHqBTmaAUH80vC9KQsqfY6IqczyD0Sp+K85mgP6sQgx6Te5oltDezegw1H0 luV9XbDyrXLtx5BGc7S0R3jPEnug9EJnE2fhRH49c0HGea35EpV2UQ/Z1rlS9A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bWHMK6r5kBUJ9iV8cP/TU68Vsy3sDvddkiKO4TwUZlc=; b=xZTklOBzQvP69gCnFqvs8Dq2tLlMJcoPnlLizYK7DoNtiWj3CrGJ5Pba/DvLj/dMeYpxV6 VDw0ycerZCa3TDCQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v9 26/32] timers: Restructure internal locking Date: Fri, 1 Dec 2023 10:26:48 +0100 Message-Id: <20231201092654.34614-27-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:30:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071420815626448 X-GMAIL-MSGID: 1784071420815626448 From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2cff43c10329..b0fa8afe9059 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2230,11 +2230,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -2258,21 +2254,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Fri Dec 1 09:26:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172363 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992160vqy; Fri, 1 Dec 2023 01:29:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IGWYTBMOjUdJaeg/nhj4Q69K+dQJP54xMrzHvFPsInWteo0TgZgAtYQbnfixU9CD/1OylQL X-Received: by 2002:a05:6830:33f6:b0:6b7:6e07:4951 with SMTP id i22-20020a05683033f600b006b76e074951mr2341285otu.25.1701422943042; Fri, 01 Dec 2023 01:29:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422943; cv=none; d=google.com; s=arc-20160816; b=eBtAlEm2HL3j3suZJIfCSbxLAFmpFKXzB7wo4FWwTUI1cPqiIWV086NhiqIqxnQV4T sojxWxCwo/oTOXpHtfp9JfZGjhIqm0uwuprP3kspqdav4839hsB0Jmwu9DFUd7HF8Dm2 fZbFTonfJO9A6EvcIjHteDNS2ldOV4v/EPP704y31mG1LJ3OROHvr65rowoP4DLmUBfx rAAZUOwmsPG0vOwK88l1jRlSM2X8NseMSVx8LRaXJp899ERZNRIgiXvZ86N6QycKJS5H dUgTawsguJVB9VRHidU7L/al+2GPgp8ch2leUya8eAMdzBwR3aDPUOozCrHzWXOMRiS0 JmCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=2bCi4tjY8tpa6/56LAAq7Ts1AT6CM22+SLDU0+3P94Q=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=G4IKbXSJPi9c2kngtZOOjJn/iJRtR4ntP7LCD6YAmSAPio+wMjiFUt/rNvlwS0wfl9 fQ1pWLX8Cuua+qyDIq9BN3yImhlf30WV/lKl6PShusx1lVxKTUDUXdSLm5IYj2F1T8nr AhDzgZ+w9vdUG5FwQtl6eENdGI+bMontKj2zn84Xn3OARgMP341gAHsttB2Z97w/+r2E FdKRQZDNydTKnYegpbihwRVIgILSl4TjrIVwezaZAU97X7j88tkp5m6JPIm1+oaE49rM nWi/5BKRdqzyJgtlJ6y5p1whzYihCUx++EsxegqVc40vW8FgmGG1XVXfNLG6NcYPnlmJ 3BFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ahMjG98K; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id y16-20020a656c10000000b005c24fb6927csi3362812pgu.656.2023.12.01.01.29.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:29:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ahMjG98K; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8588380FA860; Fri, 1 Dec 2023 01:29:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378247AbjLAJ2s (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378045AbjLAJ2K (ORCPT ); Fri, 1 Dec 2023 04:28:10 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDBB21FC3 for ; Fri, 1 Dec 2023 01:27:27 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2bCi4tjY8tpa6/56LAAq7Ts1AT6CM22+SLDU0+3P94Q=; b=ahMjG98K3D4qOpghguU0ozTb5iyD0FeZ0e5qaVghe+bnsovvsp9OfHlCCOcSaqeATT2Vkh c1SjyHLSvCTPSGf11wOQ2zyxFpjVtFxMniVVw3F/Q52kHz0HrUtZlSBaE/xRYKYjhSSElS WxFViLi9vbKezJgTeWYLvyjOfEEJcuD9Tq0lrBkeaj4TP53vIdwFRvlg3MfeGjugPF8D06 +ybZr+fk81iM6Q8TgOKUYnWfazS4hFI6RmskRMMxWUfFwGZINJlsSbuAxriqyYib5oQAjk gi395JEUfMV5MhMhK8rJplN0PV51wsrhLxKt703ruRVq5tCXBavriNC/6VmjQg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2bCi4tjY8tpa6/56LAAq7Ts1AT6CM22+SLDU0+3P94Q=; b=kRRyCgAuOmpklgvzYnSGwKqnsdjYnb8yb20cvQrna6iuwxMVhjGeLMMflSoErV8LCPv6EQ BP7JgzBcRXdMQuCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 27/32] timers: Check if timers base is handled already Date: Fri, 1 Dec 2023 10:26:49 +0100 Message-Id: <20231201092654.34614-28-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071263818785473 X-GMAIL-MSGID: 1784071263818785473 Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local cpu, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen --- v6: Drop double negation --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index b0fa8afe9059..a797603dfd49 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2232,6 +2232,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Fri Dec 1 09:26:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172364 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992209vqy; Fri, 1 Dec 2023 01:29:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHgQJLNecCyGxCU3OtL8N1OVuixduk+vOSBj/ev0b3YsUUfJL44S68DW5CpAHTCOJ+cw8E X-Received: by 2002:a05:6871:8912:b0:1fa:1ef2:3c3 with SMTP id ti18-20020a056871891200b001fa1ef203c3mr28320952oab.29.1701422951338; Fri, 01 Dec 2023 01:29:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422951; cv=none; d=google.com; s=arc-20160816; b=P8H2t7+TA8jsB6qO7cSt0A8zCWZONUmsy6BODvjd+Qr5x88Q9xlShl6ll/cYaaSpRV 9g37YyfrXi9oSfMj64sMjXxKqXV6I6BEf2VCSg/wcDWbVYnSaaNpcABRbLoORE+0YdOT Yjwx5rBbx0ZeTNkHdHLJc61LSUbm0oPksUShOK+3miqWZF3TDuXGTq3lysJf86vnKnEa 6NUs0vdcYfqf26CuoV9JQR0fSax9VdiwMKSJx354UQyLIfH7W/xgyOTyv9OOE4FQqE6m kbj3FydhiwFOZmJIajgzy4VhMTVwwebxRG8ZvfOjklSjOe4+OCJ9diXZ62zLUGp2Tovh dfvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=eJA4rUNXEYpQORMvzo1itCOqghLSNPP1k7n+/ZfaqQw=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=bOTzszmW05I7EduIApKFw5hBSHOsj5THWj+gJT1zbKc7j2StyIOMZMJTei8HhEj60e QH8PmaxTGyQswTheqTMr441ezCUao5xDutiDLnMO88gZqszAsjEvlv9R7BhlNursR1Iu O+nDQbT1b1VAiIdHeNrtUEMA8993UUTi1lmPARVvcZJ+q6lfYTPw9poQC1SnSq9CGw/N e/qE0fG77DnXjHRxRlQUJe8O6O+ww0gDZoZpIYMExPreDBbYkSoWZgozuaSmcBOJ4UZf 0KBpneArHHVaYgYOpcFrchY2tkBfYbXdAQ5lGHvVd+EsthfpYux3SN1XHloxVrHaUVkd uArg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=kEhWR6Bp; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id b16-20020a6567d0000000b005bd043751ccsi3013930pgs.748.2023.12.01.01.29.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:29:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=kEhWR6Bp; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 33EAA80FA875; Fri, 1 Dec 2023 01:29:10 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378141AbjLAJ2z (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378182AbjLAJ2M (ORCPT ); Fri, 1 Dec 2023 04:28:12 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17AF510F9 for ; Fri, 1 Dec 2023 01:27:27 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eJA4rUNXEYpQORMvzo1itCOqghLSNPP1k7n+/ZfaqQw=; b=kEhWR6BpILpXBPleNuixEgrOHoHU+tSLpntkJXiwMC8Z3g/64p2VWGhRaqc+DO6Mn4EZt3 SO9QnO91MCwqCP4ALrC++C3NiTHGCv2Xj2yvjQiWf+xpDcizD47QTatabumYvok/s049QC Tz+T+2CbvsMSpwCdbPqfi5wVGiTaSj9thk/F3OpNgnFwDG4Vl09HX5UORn+Vz1m8M0spqJ 3HbiuMuItqnoTfzPiaP0bEMg1GB4c4GAs6osXO1jBr3jpHrsQaR2qP0WevQp63Mz1Z9o7M DymzebXlPg5BwTTWtHVUbz0zYCVu9+a2GQmWontxhQ9njwtYQ6QIra/RoFmvUQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eJA4rUNXEYpQORMvzo1itCOqghLSNPP1k7n+/ZfaqQw=; b=PwBCPeViNGgkYRsSqSmBYt+Pw1wS+y9k7jm6ttbX8IIbrYXD4/2A5lmwrnK83nic0Y2ESy WLfgLJyWRc/AmGBA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v9 28/32] tick/sched: Split out jiffies update helper function Date: Fri, 1 Dec 2023 10:26:50 +0100 Message-Id: <20231201092654.34614-29-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071272513082949 X-GMAIL-MSGID: 1784071272513082949 From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global function in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 183ad32330fb..e0e58dd18919 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -158,6 +158,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index c6b415052c56..aca0e133ba09 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -799,18 +799,30 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - u64 basemono, next_tick, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + u64 basemono, next_tick, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Fri Dec 1 09:26:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992264vqy; Fri, 1 Dec 2023 01:29:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IHOg1iyzzjRzIU0k22qG0Mo4gWudieXj0vmOWC7QeKzBXy16sp62Bivz2gOmdyFbNfwKjT7 X-Received: by 2002:a05:6a00:130f:b0:6cb:4d36:6f9c with SMTP id j15-20020a056a00130f00b006cb4d366f9cmr23934915pfu.11.1701422958877; Fri, 01 Dec 2023 01:29:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422958; cv=none; d=google.com; s=arc-20160816; b=kUMCM6gqQOH/GHatWfBzNoGpXBbffUx7B4xh0IwE+eK/Mz0Icomz/r/C52S2+CH9dQ CSU2GkP005R3sIWEeC6w1mZaV26s1/0q83Gukk93bBnfpVw/vzVQd5VEss2BKITaY/DN nflpWKPRO4LreFXmXxC22wrM0tPxCWdTlI7ZwnENWPamaTr3sRtg12TLBlglQc0ofrXq qFWiZFoh/1DQTz++IKBB0szXvDhJw6CZzDIPFPV0MUs1wvyoazuV+72UIw7eXhuhAd8r Zau3J59wbrk1mVrpI7hl1/aIcDVSfIWIfgxQyMfLshw9AhxMTLP8HwodH90VAMTBqUeG VLCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=Lz8mvggzvpUJEzbVXb4zUQlWwIeSFK/TCyXr++4TCPE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=rC/ZFfTjLU3xSre5KLNrguspizm1Ja/hHqiKKQ0FuVJrE7Hqc/rvfiWl9LQINAbxNl L8oFuD4/I19K6VvRII3lIAyncuWwtRXuvyEj55WZKwWEScYVX538TsaxuMVhHxxJ/deM BUWd8eHt83MEGgkGBOZCeClWqo6a17L1UHqHZshkBJfpzcKI/ZPg5Td5G7XV6fafb5AO GcpAARBikfd/Pk3Kz6s6n/rqrQP4NeOIYNjBqfDGCC/cgo3WA3ky9s6+7vlPQusAnDJi BzziyZ3OmK8n1mqMPDcHcprlH6hoPexoAh7dFdrLm3k25sjsp7Sl7zVbqOiGUsRRYw6d Yj3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=EdTH1vhM; dkim=neutral (no key) header.i=@linutronix.de header.b=7AWmEIid; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id b12-20020a056a000ccc00b006cd82b4d0c4si3116736pfv.89.2023.12.01.01.29.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:29:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=EdTH1vhM; dkim=neutral (no key) header.i=@linutronix.de header.b=7AWmEIid; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8D1F280FA9CC; Fri, 1 Dec 2023 01:29:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378194AbjLAJ27 (ORCPT + 99 others); Fri, 1 Dec 2023 04:28:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378081AbjLAJ2M (ORCPT ); Fri, 1 Dec 2023 04:28:12 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FB571FCB for ; Fri, 1 Dec 2023 01:27:28 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lz8mvggzvpUJEzbVXb4zUQlWwIeSFK/TCyXr++4TCPE=; b=EdTH1vhM3NGV6kmiUinGxLJ1Pm0M49zT313ppufUyWkX5Xlv7reO3Fk/sLVkQhQK7lDkKO OKyWIf49mKkvRSRvEz9LZ5jjWbVDcj8jaEtC5xYrxylR/wiK7coOeLfGlMNmNN21nXNxHf sLA55soY106JsKuC2y5XYkXpChswmiJF6O35gv4lDjmZE9ckO49OZq9IJPQgcUO9H2pjss N8ZOv+qp88KBvyM5BxqOCJu8mB64sQctsoqxBAJ1AjggNxBxfqRnfryYYOwsaBJgc3yW80 DJAqa12Niwukm8v9KHr9jy9njV6OEiO4TlibrrJzd9x+L3mFmztY9Gk/tbWvqg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lz8mvggzvpUJEzbVXb4zUQlWwIeSFK/TCyXr++4TCPE=; b=7AWmEIidBRmfR/ZiDTR0IYKQlxe2r1jHuUlbpKg2IhmL9oFRBHb5oBeNAoamIAoHtCdXMK z8HthAkNaKBDp5Dg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 29/32] timers: Introduce function to check timer base is_idle flag Date: Fri, 1 Dec 2023 10:26:51 +0100 Message-Id: <20231201092654.34614-30-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071280638137258 X-GMAIL-MSGID: 1784071280638137258 To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have a function that returns the value of the is_idle flag of the timer base to keep the hierarchy states during online in sync with timer base state. No functional change. Signed-off-by: Anna-Maria Behnsen --- v9: new in v9 --- kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index e0e58dd18919..2d1a44850c20 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -166,6 +166,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, unsigned int cpu); extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); +extern bool timer_base_is_idle(void); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index a797603dfd49..b6c9ac0c3712 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2201,6 +2201,16 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) return __get_next_timer_interrupt(basej, basem, idle); } +/** + * timer_base_is_idle() - Return whether timer base is set idle + * + * Returns value of local timer base is_idle value. + */ +bool timer_base_is_idle(void) +{ + return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle); +} + /** * timer_clear_idle - Clear the idle state of the timer base * From patchwork Fri Dec 1 09:26:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172368 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992545vqy; Fri, 1 Dec 2023 01:30:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHhxGAad3d/y9qtzVLORDrNWEKBqADS5YmbKhqzdBxABhKuaw4qUlRn6+myvLegOxVCs2lP X-Received: by 2002:a05:6871:42c7:b0:1fa:a4f0:ab29 with SMTP id lt7-20020a05687142c700b001faa4f0ab29mr8984822oab.16.1701423003768; Fri, 01 Dec 2023 01:30:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701423003; cv=none; d=google.com; s=arc-20160816; b=gI+EqltDxarHoXQVLweN0dtIRnuDCwNszPlhIXwRmimCUyxnRRLOW9ih2NITrQPfyJ dhlLyEbJmzNq0JT/trOoHeuBBsu/14Ja/caBn65+wAenOkO5ytrv3VjyAgBnDucD4SF8 h4hxugvUb4Tqakr6eZmWXdKeTDGyUQbUF8g71MMm5AeZY+IGlHPHaHRC/Nntb2ndmDo1 qwgdgTE/1Ma6moh8jIZy2ldcwnvVp5BR1Dc1yNOGPgPcBYTc5gvAeBLR9LL/+KWh3YnO zhvL/iEmdT1XxxeaRriIdFLW5i5FNr7Ny+khCRmEbbebdAVsejImYkTT1sUgmxc4BmMJ 6E3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=KXMsmyJyufEgL2ebmuiCg2SwPgKd/zMJW6cpMta/sXs=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=T4ZgXcFMUQZKitrQC2NE45Oui8pbr9Q9tCUBavXy2mRIilCCRXxzz+/pd8xPIkDikB s8epkB85TyESutuds4L8S0UD7a+h5pYW94rqW2ubfe3AyVhlUiwmI6HeWwcyTNPWmn1w WAB1n04ed49UD7aDckP7KmceNJ3uAFUX7igYvo4L2ODWc0EXiVJHFwoUksZOEZMjplr5 c5ISXGMrraapiwYGiaB0tLFnXIrTBonoaO07GS+0I+GglCK5Ep34D4GDNwQi3TEigliK ShGgulgfYhakiSyXnuypwDw3CJZhzeN4jUjDhnyA891xLMdDQxvtXiVEBLf1yvG5Rhdl nm+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=wxVPUVts; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id w15-20020a63d74f000000b005bdbd683601si3124227pgi.57.2023.12.01.01.30.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:30:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=wxVPUVts; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 29EB7825F1D3; Fri, 1 Dec 2023 01:29:38 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378213AbjLAJ3S (ORCPT + 99 others); Fri, 1 Dec 2023 04:29:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378163AbjLAJ2T (ORCPT ); Fri, 1 Dec 2023 04:28:19 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45CCC1FD5 for ; Fri, 1 Dec 2023 01:27:29 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KXMsmyJyufEgL2ebmuiCg2SwPgKd/zMJW6cpMta/sXs=; b=wxVPUVtsCWsVS7pdkFzWxLvQMDp07QSFTczhOBOvCdH270RAdlV5leOboe7jXRZuz74+9w fMkcLwKKaNGJ9/JDiigBmL0BxJWjU2VjK/V7/TdO34VfbMWFXFm4BXNdzaPZr7fuWgO33y sS1gqbaag5+WNk3NJwM6JncYPt+1KVMLdswUTU/queLktc5xNyp7+pLvNfghdKkJST/tq1 dIMBEdMzp1SNJEMjJ4PDVMBqbl9FvQ/luCZEKs7SlCzB9VMzm/2/HEy8E60M2o+Q45ouTK 7tM9lraD9Hu1skZrsbiYl2Ow/nXNXrz1BhiwEJqSvXKPC8M1kf1Xryf2rIw2Qg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KXMsmyJyufEgL2ebmuiCg2SwPgKd/zMJW6cpMta/sXs=; b=9YkZVZayylTHiBIfC+OG3OfnipCjB8b0mTo7ZpwtFl9tqvjKgWhB1wPbYXNbOCRQukQGWQ E0EwaDIELCZfjLDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 30/32] timers: Implement the hierarchical pull model Date: Fri, 1 Dec 2023 10:26:52 +0100 Message-Id: <20231201092654.34614-31-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:38 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071327554268496 X-GMAIL-MSGID: 1784071327554268496 Placing timers at enqueue time on a target CPU based on dubious heuristics does not make any sense: 1) Most timer wheel timers are canceled or rearmed before they expire. 2) The heuristics to predict which CPU will be busy when the timer expires are wrong by definition. So placing the timers at enqueue wastes precious cycles. The proper solution to this problem is to always queue the timers on the local CPU and allow the non pinned timers to be pulled onto a busy CPU at expiry time. Therefore split the timer storage into local pinned and global timers: Local pinned timers are always expired on the CPU on which they have been queued. Global timers can be expired on any CPU. As long as a CPU is busy it expires both local and global timers. When a CPU goes idle it arms for the first expiring local timer. If the first expiring pinned (local) timer is before the first expiring movable timer, then no action is required because the CPU will wake up before the first movable timer expires. If the first expiring movable timer is before the first expiring pinned (local) timer, then this timer is queued into a idle timerqueue and eventually expired by some other active CPU. To avoid global locking the timerqueues are implemented as a hierarchy. The lowest level of the hierarchy holds the CPUs. The CPUs are associated to groups of 8, which are separated per node. If more than one CPU group exist, then a second level in the hierarchy collects the groups. Depending on the size of the system more than 2 levels are required. Each group has a "migrator" which checks the timerqueue during the tick for remote expirable timers. If the last CPU in a group goes idle it reports the first expiring event in the group up to the next group(s) in the hierarchy. If the last CPU goes idle it arms its timer for the first system wide expiring timer to ensure that no timer event is missed. Signed-off-by: Anna-Maria Behnsen --- v9: - Adapt to the changes of the preceding patches - Fix state inconsitency (when timer base is idle, cpu must also be marked as idle in hierarchy) - Make sure new timers are considered, when timer base is idle and a timer is enqueued into global queue (e.g. during interrupt) -> timer_use_tmigr() - Changes which are required due to the timer code change of marking the timer base idle in tick_nohz_stop_tick() v8: - Review of Frederic: - Fix hotplug race (introduction of wakeup_recalc) - Make wakeup and wakeup_recalc logic consistent all over the place - Fix child/group state race and read it with locks held - Add more clarifying comments - Fix grammar all over the place - change integers which act as boolean value into bool - rewrite condition in tmigr_check_migrator() without negation - Improve update events logic with a check of the first event - Implement a quick forecast which is called when get_next_timer_interrupt() is executed. v7: - Review remarks of Frederic and bigeasy: - change logic in tmigr_handle_remote_cpu() - s/kzalloc/kcalloc - move timer_expire_remote() into NO_HZ_COMMON && SMP config section - drop DBG_BUG_ON() makro and use only WARN_ON_ONCE() - remove leftovers from sibling logic during setup - Move timer_expire_remote() into tick-internal.h - Add documentation section about "Required event and timerqueue update after remote expiry" - Fix fallout of kernel test robot v6: - Fix typos - Review remarks of Peter Zijlstra (locking, struct member cleanup, use atomic_try_cmpxchg(), update struct member descriptions) - Fix race in tmigr_handle_remote_cpu() (Frederic Weisbecker) v5: - Review remarks of Frederic - Return nextevt when CPU is marked offline in timer migration hierarchy instead of KTIME_MAX - Fix update of group events issue, after remote expiring v4: - Fold typo fix in comment into proper patch "timer: Split out "get next timer interrupt" functionality" - Update wrong comment for tmigr_state union definition - Fix fallout of kernel test robot --- include/linux/cpuhotplug.h | 1 + kernel/time/Makefile | 3 + kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 111 ++- kernel/time/timer_migration.c | 1636 +++++++++++++++++++++++++++++++++ kernel/time/timer_migration.h | 144 +++ 6 files changed, 1888 insertions(+), 8 deletions(-) create mode 100644 kernel/time/timer_migration.c create mode 100644 kernel/time/timer_migration.h diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index efc0c0b07efb..85a78c5a7c01 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -245,6 +245,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, + CPUHP_AP_TMIGR_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, CPUHP_AP_RANDOM_ONLINE, diff --git a/kernel/time/Makefile b/kernel/time/Makefile index 7e875e63ff3b..4af2a264a160 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -17,6 +17,9 @@ endif obj-$(CONFIG_GENERIC_SCHED_CLOCK) += sched_clock.o obj-$(CONFIG_TICK_ONESHOT) += tick-oneshot.o tick-sched.o obj-$(CONFIG_LEGACY_TIMER_TICK) += tick-legacy.o +ifeq ($(CONFIG_SMP),y) + obj-$(CONFIG_NO_HZ_COMMON) += timer_migration.o +endif obj-$(CONFIG_HAVE_GENERIC_VDSO) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 2d1a44850c20..9cec0c0c314f 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -167,6 +167,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); extern bool timer_base_is_idle(void); +extern void timer_expire_remote(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index b6c9ac0c3712..ac3e888d053f 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -53,6 +53,7 @@ #include #include "tick-internal.h" +#include "timer_migration.h" #define CREATE_TRACE_POINTS #include @@ -2103,6 +2104,64 @@ void timer_lock_remote_bases(unsigned int cpu) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); } + +static void __run_timer_base(struct timer_base *base); + +/** + * timer_expire_remote() - expire global timers of cpu + * @cpu: Remote CPU + * + * Expire timers of global base of remote CPU. + */ +void timer_expire_remote(unsigned int cpu) +{ + struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + __run_timer_base(base); +} + +static void timer_use_tmigr(unsigned long basej, u64 basem, + unsigned long *nextevt, bool *tick_stop_path, + bool timer_base_idle, struct timer_events *tevt) +{ + u64 next_tmigr; + + if (timer_base_idle) + next_tmigr = tmigr_cpu_new_timer(tevt->global); + else if (tick_stop_path) + next_tmigr = tmigr_cpu_deactivate(tevt->global); + else + next_tmigr = tmigr_quick_check(); + + /* + * If the CPU is the last going idle in timer migration hierarchy, make + * sure the CPU will wake up in time to handle remote timers. + * next_tmigr == KTIME_MAX if other CPUs are still active. + */ + if (next_tmigr < tevt->local) { + u64 tmp; + + /* If we missed a tick already, force 0 delta */ + if (next_tmigr < basem) + next_tmigr = basem; + + tmp = div_u64(next_tmigr - basem, TICK_NSEC); + + *nextevt = basej + (unsigned long)tmp; + tevt->local = next_tmigr; + } +} +# else +static void timer_use_tmigr(unsigned long basej, u64 basem, + unsigned long *nextevt, bool *tick_stop_path, + bool timer_base_idle, struct timer_events *tevt) +{ + /* + * Make sure first event is written into tevt->local to not miss a + * timer on !SMP systems. + */ + tevt->local = min_t(u64, tevt->local, tevt->global); +} # endif /* CONFIG_SMP */ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, @@ -2111,7 +2170,6 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; struct timer_base *base_local, *base_global; unsigned long nextevt; - u64 expires; /* * Pretend that there is no timer pending if the cpu is offline. @@ -2132,6 +2190,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, nextevt = fetch_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); + /* + * When the when the next event is only one jiffie ahead there is no + * need to call timer migration hierarchy related + * functions. @tevt->global will be KTIME_MAX, nevertheless if the next + * timer is a global timer. This is also true, when the timer base is + * idle. + * + * The proper timer migration hierarchy function depends on the callsite + * and whether timer base is idle or not. @nextevt will be updated when + * this CPU needs to handle the first timer migration hierarchy event. + */ + if (time_after(nextevt, basej + 1)) + timer_use_tmigr(basej, basem, &nextevt, idle, + base_local->is_idle, &tevt); + /* * We have a fresh next event. Check whether we can forward the * base. @@ -2144,7 +2217,10 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, */ if (idle) { /* - * Bases are idle if the next event is more than a tick away. + * Bases are idle if the next event is more than a tick + * away. Caution: @nextevt could have changed by enqueueing a + * global timer into timer migration hierarchy. Therefore a new + * check is required here. * * If the base is marked idle then any timer add operation must * forward the base clk itself to keep granularity small. This @@ -2159,14 +2235,23 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, } *idle = base_local->is_idle; trace_timer_base_idle(base_local->is_idle, base_local->cpu); + + /* + * When timer base is not set idle, undo the effect of + * tmigr_cpu_deactivate() to prevent inconsitent states - active + * timer base but inactive timer migration hierarchy. + * + * When timer base was already marked idle, nothing will be + * changed here. + */ + if (!base_local->is_idle) + tmigr_cpu_activate(); } raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - expires = min_t(u64, tevt.local, tevt.global); - - return cmp_next_hrtimer_event(basem, expires); + return cmp_next_hrtimer_event(basem, tevt.local); } /** @@ -2174,8 +2259,11 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * @basej: base time jiffies * @basem: base time clock monotonic * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer or + * KTIME_MAX if no timer is pending. If timer of global base was queued into + * timer migration hierarchy, first global timer is not taken into account. If + * it was the last CPU of timer migration hierarchy going idle, first global + * event is taken into account. */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { @@ -2228,6 +2316,9 @@ void timer_clear_idle(void) __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(0, smp_processor_id()); + + /* Activate without holding the timer_base->lock */ + tmigr_cpu_activate(); } #endif @@ -2297,6 +2388,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { run_timer_base(BASE_GLOBAL); run_timer_base(BASE_DEF); + + if (is_timers_nohz_active()) + tmigr_handle_remote(); } } @@ -2311,7 +2405,8 @@ static void run_local_timers(void) for (int i = 0; i < NR_BASES; i++, base++) { /* Raise the softirq only if required. */ - if (time_after_eq(jiffies, base->next_expiry)) { + if (time_after_eq(jiffies, base->next_expiry) || + (i == BASE_DEF && tmigr_requires_handle_remote())) { raise_softirq(TIMER_SOFTIRQ); return; } diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c new file mode 100644 index 000000000000..05cd8f1bc45d --- /dev/null +++ b/kernel/time/timer_migration.c @@ -0,0 +1,1636 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Infrastructure for migratable timers + * + * Copyright(C) 2022 linutronix GmbH + */ +#include +#include +#include +#include +#include +#include + +#include "timer_migration.h" +#include "tick-internal.h" + +/* + * The timer migration mechanism is built on a hierarchy of groups. The + * lowest level group contains CPUs, the next level groups of CPU groups + * and so forth. The CPU groups are kept per node so for the normal case + * lock contention won't happen across nodes. Depending on the number of + * CPUs per node even the next level might be kept as groups of CPU groups + * per node and only the levels above cross the node topology. + * + * Example topology for a two node system with 24 CPUs each. + * + * LVL 2 [GRP2:0] + * GRP1:0 = GRP1:M + * + * LVL 1 [GRP1:0] [GRP1:1] + * GRP0:0 - GRP0:2 GRP0:3 - GRP0:5 + * + * LVL 0 [GRP0:0] [GRP0:1] [GRP0:2] [GRP0:3] [GRP0:4] [GRP0:5] + * CPUS 0-7 8-15 16-23 24-31 32-39 40-47 + * + * The groups hold a timer queue of events sorted by expiry time. These + * queues are updated when CPUs go in idle. When they come out of idle + * ignore flag of events is set. + * + * Each group has a designated migrator CPU/group as long as a CPU/group is + * active in the group. This designated role is necessary to avoid that all + * active CPUs in a group try to migrate expired timers from other CPUs, + * which would result in massive lock bouncing. + * + * When a CPU is awake, it checks in it's own timer tick the group + * hierarchy up to the point where it is assigned the migrator role or if + * no CPU is active, it also checks the groups where no migrator is set + * (TMIGR_NONE). + * + * If it finds expired timers in one of the group queues it pulls them over + * from the idle CPU and runs the timer function. After that it updates the + * group and the parent groups if required. + * + * CPUs which go idle arm their CPU local timer hardware for the next local + * (pinned) timer event. If the next migratable timer expires after the + * next local timer or the CPU has no migratable timer pending then the + * CPU does not queue an event in the LVL0 group. If the next migratable + * timer expires before the next local timer then the CPU queues that timer + * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0 + * group. + * + * When CPU comes out of idle and when a group has at least a single active + * child, the ignore flag of the tmigr_event is set. This indicates, that + * the event is ignored even if it is still enqueued in the parent groups + * timer queue. It will be removed when touching the timer queue the next + * time. This spares locking in active path as the lock protects (after + * setup) only event information. For more information about locking, + * please read the section "Locking rules". + * + * If the CPU is the migrator of the group then it delegates that role to + * the next active CPU in the group or sets migrator to TMIGR_NONE when + * there is no active CPU in the group. This delegation needs to be + * propagated up the hierarchy so hand over from other leaves can happen at + * all hierarchy levels w/o doing a search. + * + * When the last CPU in the system goes idle, then it drops all migrator + * duties up to the top level of the hierarchy (LVL2 in the example). It + * then has to make sure, that it arms it's own local hardware timer for + * the earliest event in the system. + * + * + * Lifetime rules: + * --------------- + * + * The groups are built up at init time or when CPUs come online. They are + * not destroyed when a group becomes empty due to offlining. The group + * just won't participate in the hierarchy management anymore. Destroying + * groups would result in interesting race conditions which would just make + * the whole mechanism slow and complex. + * + * + * Locking rules: + * -------------- + * + * For setting up new groups and handling events it's required to lock both + * child and parent group. The lock ordering is always bottom up. This also + * includes the per CPU locks in struct tmigr_cpu. For updating the migrator and + * active CPU/group information atomic_try_cmpxchg() is used instead and only + * the per CPU tmigr_cpu->lock is held. + * + * During the setup of groups tmigr_level_list is required. It is protected by + * @tmigr_mutex. + * + * When @timer_base->lock as well as tmigr related locks are required, the lock + * ordering is: first @timer_base->lock, afterwards tmigr related locks. + * + * + * Protection of the tmigr group state information: + * ------------------------------------------------ + * + * The state information with the list of active children and migrator needs to + * be protected by a sequence counter. It prevents a race when updates in a + * child groups are propagated in changed order. The following scenario + * describes what happens without updating the sequence counter: + * + * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well + * as GRP0:1 will not change during the scenario): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU0 migrator = CPU2 + * active = CPU0 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * active idle active idle + * + * + * 1. CPU0 goes idle (changes are updated in GRP0:0; afterwards the current + * states of GRP0:0 and GRP1:0 are stored in the data for walking the + * hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = TMIGR_NONE migrator = CPU2 + * --> active = active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * --> idle idle active idle + * + * 2. CPU1 comes out of idle (changes are update in GRP0:0; afterwards the + * current states of GRP0:0 and GRP1:0 are stored in the data for walking the + * hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = CPU1 migrator = CPU2 + * --> active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle --> active active idle + * + * 3. Here comes the change of the order: Propagating the changes of step 2 + * through the hierarchy to GRP1:0 - nothing to be done, because GRP0:0 + * is already up to date. + * + * 4. Propagating the changes of step 1 through the hierarchy to GRP1:0 + * + * LVL 1 [GRP1:0] + * --> migrator = GRP0:1 + * --> active = GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU1 migrator = CPU2 + * active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle active active idle + * + * Now there is a inconsistent overall state because GRP0:0 is active, but + * it is marked as idle in the GRP1:0. This is prevented by incrementing + * sequence counter whenever changing the state. + * + * + * Required event and timerqueue update after a remote expiry: + * ----------------------------------------------------------- + * + * After a remote expiry of a CPU, a walk through the hierarchy updating the + * events and timerqueues has to be done when there is a 'new' global timer of + * the remote CPU (which is obvious) but also if there is no new global timer, + * but the remote CPU is still idle: + * + * 1. CPU2 is the migrator and does the remote expiry in GRP1:0; expiry of + * evt-CPU0 and evt-CPU1 are equal: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 2. Remove the first event of the timerqueue in GRP1:0 and expire the timers + * of CPU0 (see evt-GRP0:0->cpu value): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 3. After the remote expiry CPU0 has no global timer that needs to be + * enqueued. When skipping the walk, the global timer of CPU1 is not handled, + * as the group event of GRP0:0 is not updated and not enqueued into GRP1:0. The + * walk has to be done to update the group events and timerqueues: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU1 groupevt.cpu = + * --> timerqueue = evt-CPU1 timerqueue = + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * Now CPU2 (migrator) is able to handle the timer of CPU1 as CPU2 only scans + * the timerqueues of GRP0:1 and GRP1:0. + * + * The update of step 3 is valid to be skipped, when the remote CPU went offline + * in the meantime because an update was already done during inactive path. When + * CPU became active in the meantime, update isn't required as well, because + * GRP0:0 is now longer idle. + */ + +static DEFINE_MUTEX(tmigr_mutex); +static struct list_head *tmigr_level_list __read_mostly; + +static unsigned int tmigr_hierarchy_levels __read_mostly; +static unsigned int tmigr_crossnode_level __read_mostly; + +static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu); + +#define TMIGR_NONE 0xFF +#define BIT_CNT 8 + +static inline bool tmigr_is_not_available(struct tmigr_cpu *tmc) +{ + return !(tmc->tmgroup && tmc->online); +} + +/* + * Returns true, when @childmask corresponds to the group migrator or when the + * group is not active - so no migrator is set. + */ +static bool tmigr_check_migrator(struct tmigr_group *group, u8 childmask) +{ + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + return true; + + return false; +} + +static bool tmigr_check_migrator_and_lonely(struct tmigr_group *group, u8 childmask) +{ + bool lonely, migrator = false; + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + migrator = true; + + active = s.active; + lonely = bitmap_weight(&active, BIT_CNT) <= 1; + + return (migrator && lonely); +} + +static bool tmigr_check_lonely(struct tmigr_group *group) +{ + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + active = s.active; + + return bitmap_weight(&active, BIT_CNT) <= 1; +} + +typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *); + +static void __walk_groups(up_f up, void *data, + struct tmigr_cpu *tmc) +{ + struct tmigr_group *child = NULL, *group = tmc->tmgroup; + + do { + WARN_ON_ONCE(group->level >= tmigr_hierarchy_levels); + + if (up(group, child, data)) + break; + + child = group; + group = group->parent; + } while (group); +} + +static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc) +{ + lockdep_assert_held(&tmc->lock); + + __walk_groups(up, data, tmc); +} + +/** + * struct tmigr_walk - data required for walking the hierarchy + * @evt: Pointer to tmigr_event which needs to be queued (of idle + * child group) + * @childmask: childmask of child group + * @nextexp: Next CPU event expiry information which is handed into + * the timer migration code by the timer code + * (get_next_timer_interrupt()); it is furthermore used for + * the first event which is queued, if the timer migration + * hierarchy is completely idle + * @childstate: tmigr_group->migr_state of the child - will be only + * reread when cmpxchg in the group fails (is required for + * the deactivate path and the new timer path) + * @groupstate: tmigr_group->migr_state of the group - will be only + * reread when cmpxchg in the group fails (is required for + * the active, the deactivate and the new timer path) + * @remote: Is set, when the new timer path is executed in + * tmigr_handle_remote_cpu() + */ +struct tmigr_walk { + struct tmigr_event *evt; + u8 childmask; + u64 nextexp; + union tmigr_state childstate; + union tmigr_state groupstate; + bool remote; +}; + +/** + * struct tmigr_remote_data - data required for remote expiry hierarchy walk + * @basej: timer base in jiffies + * @now: timer base monotonic + * @nextexp: returns expiry of the first timer in the idle timer + * migration hierarchy to make sure the timer is handled in + * time; it is stored in the per CPU tmigr_cpu struct of + * CPU which expires remote timers + * @childmask: childmask of child group + * @check: is set if there is the need to handle remote timers; + * required in tmigr_check_handle_remote() only + * @tmc_active: this flag indicates, whether the CPU which triggers + * the hierarchy walk is !idle in the timer migration + * hierarchy. When the CPU is idle and the whole hierarchy is + * idle, only the first event of the top level has to be + * considered. + */ +struct tmigr_remote_data { + unsigned long basej; + u64 now; + u64 nextexp; + u8 childmask; + bool check; + bool tmc_active; +}; + +/* + * Returns the next event of the timerqueue @group->events + * + * Removes timers with ignore flag and update next_expiry of the group. Values + * of the group event are updated in tmigr_update_events() only. + */ +static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group) +{ + struct timerqueue_node *node = NULL; + struct tmigr_event *evt = NULL; + + lockdep_assert_held(&group->lock); + + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + while ((node = timerqueue_getnext(&group->events))) { + evt = container_of(node, struct tmigr_event, nextevt); + + if (!evt->ignore) { + WRITE_ONCE(group->next_expiry, evt->nextevt.expires); + return evt; + } + + /* + * Remove next timers with ignore flag, because the group lock + * is held anyway + */ + if (!timerqueue_del(&group->events, node)) + break; + } + + return NULL; +} + +/* + * Return the next event which is already expired of the group timerqueue + * + * Event, which is returned, is also removed from the queue. + */ +static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group, + u64 now) +{ + struct tmigr_event *evt = tmigr_next_groupevt(group); + + if (!evt || now < evt->nextevt.expires) + return NULL; + + /* + * The event is already expired. Remove it. If it's not the last event, + * then update all group event related information. + */ + if (timerqueue_del(&group->events, &evt->nextevt)) + tmigr_next_groupevt(group); + else + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + return evt; +} + +static u64 tmigr_next_groupevt_expires(struct tmigr_group *group) +{ + struct tmigr_event *evt; + + evt = tmigr_next_groupevt(group); + + if (!evt) + return KTIME_MAX; + else + return evt->nextevt.expires; +} + +static bool tmigr_active_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + if (newstate.migrator == TMIGR_NONE) { + newstate.migrator = childmask; + + /* Changes need to be propagated */ + walk_done = false; + } + + newstate.active |= childmask; + + newstate.seq++; + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + goto retry; + } + + if (group->parent && (walk_done == false)) { + data->groupstate.state = atomic_read(&group->parent->migr_state); + data->childmask = group->childmask; + } + + /* + * The group is active and the event will be ignored - the ignore flag is + * updated without holding the lock. In case the bit is set while + * another CPU already handles remote events, nothing happens, because + * it is clear that the CPU became active just in this moment, or in + * worst case the event is handled remote. Nothing to worry about. + */ + group->groupevt.ignore = true; + + return walk_done; +} + +static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) +{ + struct tmigr_walk data; + + data.childmask = tmc->childmask; + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + tmc->cpuevt.ignore = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + tmc->wakeup_recalc = false; + + walk_groups(&tmigr_active_up, &data, tmc); +} + +/** + * tmigr_cpu_activate() - set this CPU active in timer migration hierarchy + * + * Call site timer_clear_idle() is called with interrupts disabled. + */ +void tmigr_cpu_activate(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + if (tmigr_is_not_available(tmc)) + return; + + if (WARN_ON_ONCE(!tmc->idle)) + return; + + raw_spin_lock(&tmc->lock); + tmc->idle = false; + __tmigr_cpu_activate(tmc); + raw_spin_unlock(&tmc->lock); +} + +/* + * Returns true, if there is nothing to be propagated to the next level + * + * @data->nextexp is reset to KTIME_MAX; it is reused for the first global event + * which needs to be handled by the migrator (in the top level group). + * + * This is the only place where the group event expiry value is set. + */ +static bool tmigr_update_events(struct tmigr_group *group, + struct tmigr_group *child, + struct tmigr_walk *data) +{ + struct tmigr_event *evt, *first_childevt; + bool walk_done, remote = data->remote; + bool leftmost_change = false; + u64 nextexp; + + if (child) { + raw_spin_lock(&child->lock); + raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING); + + data->childstate.state = atomic_read(&child->migr_state); + data->groupstate.state = atomic_read(&group->migr_state); + + if (data->childstate.active) { + walk_done = true; + goto unlock; + } + + first_childevt = tmigr_next_groupevt(child); + nextexp = child->next_expiry; + evt = &child->groupevt; + } else { + nextexp = data->nextexp; + + /* + * Set @data->nextexp to KTIME_MAX; it is reused for the first + * global event which needs to be handled by the migrator (in + * the top level group). + */ + data->nextexp = KTIME_MAX; + + first_childevt = evt = data->evt; + + /* + * Walking the hierarchy is required in any case when a + * remote expiry was done before. This ensures to not lose + * already queued events in non active groups (see section + * "Required event and timerqueue update after remote + * expiry" in the documentation at the top). + * + * The two call sites which are executed without a remote expiry + * before, are not prevented from propagating changes through + * the hierarchy by the return: + * - When entering this path by tmigr_new_timer(), @evt->ignore + * is never set. + * - tmigr_inactive_up() takes care of the propagation by + * itself and ignores the return value. But an immediate + * return is required because nothing has to be done in this + * level as the event could be ignored. + */ + if (evt->ignore && !remote) + return true; + + raw_spin_lock(&group->lock); + data->groupstate.state = atomic_read(&group->migr_state); + } + + if (nextexp == KTIME_MAX) { + evt->ignore = true; + + /* + * When the next child event could be ignored (nextexp is + * KTIME_MAX) and there was no remote timer handling before or + * the group is already active, there is no need to walk the + * hierarchy even if there is a parent group. + * + * The other way round: even if the event could be ignored, but + * if a remote timer handling was executed before and the group + * is not active, walking the hierarchy is required to not miss + * an enqueued timer in the non active group. The enqueued timer + * of the group needs to be propagated to a higher level to + * ensure it is handled. + */ + if (!remote || data->groupstate.active) { + walk_done = true; + goto unlock; + } + } else { + /* + * An update of @evt->cpu and @evt->ignore flag is required only + * when @child is set (the child is equal or higher than lvl0), + * but it doesn't matter if it is written once more to the per + * CPU event; make the update unconditional. + */ + evt->cpu = first_childevt->cpu; + evt->ignore = false; + } + + walk_done = !group->parent; + + /* + * If the child event is already queued in the group, remove it from the + * queue when the expiry time changed only. + */ + if (timerqueue_node_queued(&evt->nextevt)) { + if (evt->nextevt.expires == nextexp) + goto check_toplvl; + + leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt; + if (!timerqueue_del(&group->events, &evt->nextevt)) + WRITE_ONCE(group->next_expiry, KTIME_MAX); + } + + evt->nextevt.expires = nextexp; + + if (timerqueue_add(&group->events, &evt->nextevt)) { + leftmost_change = true; + WRITE_ONCE(group->next_expiry, nextexp); + } + +check_toplvl: + if (walk_done && (data->groupstate.migrator == TMIGR_NONE)) { + /* + * Nothing to do when first event didn't changed and update was + * done during remote timer handling. + */ + if (remote && !leftmost_change) + goto unlock; + /* + * The top level group is idle and it has to be ensured the + * global timers are handled in time. (This could be optimized + * by keeping track of the last global scheduled event and only + * arming it on the CPU if the new event is earlier. Not sure if + * its worth the complexity.) + */ + data->nextexp = tmigr_next_groupevt_expires(group); + } + +unlock: + raw_spin_unlock(&group->lock); + + if (child) + raw_spin_unlock(&child->lock); + + return walk_done; +} + +static bool tmigr_new_timer_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_walk *data = ptr; + + return tmigr_update_events(group, child, data); +} + +/* + * Returns the expiry of the next timer that needs to be handled. KTIME_MAX is + * returned, when an active CPU will handle all the timer migration hierarchy + * timers. + */ +static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .evt = &tmc->cpuevt, + .nextexp = nextexp }; + + lockdep_assert_held(&tmc->lock); + + if (tmc->remote) + return KTIME_MAX; + + tmc->cpuevt.ignore = false; + data.remote = false; + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + /* If there is a new first global event, make sure it is handled */ + return data.nextexp; +} + +static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, + unsigned long jif) +{ + struct timer_events tevt; + struct tmigr_walk data; + struct tmigr_cpu *tmc; + u64 next = KTIME_MAX; + + tmc = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_lock_irq(&tmc->lock); + + /* + * The remote CPU is offline or the CPU event does not has to be handled + * (the CPU is active or there is no longer an event to expire) or + * another CPU handles the CPU timers already or the next event was + * already expired - return! + */ + if (!tmc->online || tmc->remote || tmc->cpuevt.ignore || + now < tmc->cpuevt.nextevt.expires) { + raw_spin_unlock_irq(&tmc->lock); + return next; + } + + tmc->remote = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* Drop the lock to allow the remote CPU to exit idle */ + raw_spin_unlock_irq(&tmc->lock); + + if (cpu != smp_processor_id()) + timer_expire_remote(cpu); + + /* + * Lock ordering needs to be preserved - timer_base locks before tmigr + * related locks (see section "Locking rules" in the documentation at + * the top). During fetching the next timer interrupt, also tmc->lock + * needs to be held. Otherwise there is a possible race window against + * the CPU itself when it comes out of idle, updates the first timer in + * the hierarchy and goes back to idle. + * + * timer base locks are dropped as fast as possible: After checking + * whether the remote CPU went offline in the meantime and after + * fetching the next remote timer interrupt. Dropping the locks as fast + * as possible keeps the locking region small and prevents holding + * several (unnecessary) locks during walking the hierarchy for updating + * the timerqueue and group events. + */ + local_irq_disable(); + timer_lock_remote_bases(cpu); + raw_spin_lock(&tmc->lock); + + /* + * When the CPU went offline in the meantime, no hierarchy walk has to + * be done for updating the queued events, because the walk was + * already done during marking the CPU offline in the hierarchy. + * + * When the CPU is no longer idle, the CPU takes care of the timers and + * also of the timers in the path to the top. + * + * (See also section "Required event and timerqueue update after + * remote expiry" in the documentation at the top) + */ + if (!tmc->online || !tmc->idle) { + timer_unlock_remote_bases(cpu); + goto unlock; + } else { + /* next event of CPU */ + fetch_next_timer_interrupt_remote(jif, now, &tevt, cpu); + } + + timer_unlock_remote_bases(cpu); + + data.evt = &tmc->cpuevt; + data.nextexp = tevt.global; + data.remote = true; + + /* + * The update is done even when there is no 'new' global timer pending + * on the remote CPU (see section "Required event and timerqueue update + * after remote expiry" in the documentation at the top) + */ + walk_groups(&tmigr_new_timer_up, &data, tmc); + + next = data.nextexp; + +unlock: + tmc->remote = false; + raw_spin_unlock_irq(&tmc->lock); + + return next; +} + +static bool tmigr_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u64 now, next = KTIME_MAX; + struct tmigr_event *evt; + unsigned long jif; + u8 childmask; + + jif = data->basej; + now = data->now; + + childmask = data->childmask; + +again: + /* + * Handle the group only if @childmask is the migrator or if the + * group has no migrator. Otherwise the group is active and is + * handled by its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + raw_spin_lock_irq(&group->lock); + + evt = tmigr_next_expired_groupevt(group, now); + + if (evt) { + unsigned int remote_cpu = evt->cpu; + + raw_spin_unlock_irq(&group->lock); + + next = tmigr_handle_remote_cpu(remote_cpu, now, jif); + + /* check if there is another event, that needs to be handled */ + goto again; + } else { + raw_spin_unlock_irq(&group->lock); + } + + /* Update of childmask for the next level */ + data->childmask = group->childmask; + data->nextexp = next; + + return false; +} + +/** + * tmigr_handle_remote() - Handle global timers of remote idle CPUs + * + * Called from the timer soft interrupt with interrupts enabled. + */ +void tmigr_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (tmigr_is_not_available(tmc)) + return; + + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + + /* + * NOTE: This is a doubled check because the migrator test will be done + * in tmigr_handle_remote_up() anyway. Keep this check to fasten the + * return when nothing has to be done. + */ + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return; + + data.now = get_jiffies_update(&data.basej); + + /* + * Update @tmc->wakeup only at the end and do not reset @tmc->wakeup to + * KTIME_MAX. Even if tmc->lock is not held during the whole remote + * handling, tmc->wakeup is fine to be stale as it is called in + * interrupt context and tick_nohz_next_event() is executed in interrupt + * exit path only after processing the last pending interrupt. + */ + + __walk_groups(&tmigr_handle_remote_up, &data, tmc); + + raw_spin_lock_irq(&tmc->lock); + WRITE_ONCE(tmc->wakeup, data.nextexp); + raw_spin_unlock_irq(&tmc->lock); +} + +static bool tmigr_requires_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u8 childmask; + + childmask = data->childmask; + + /* + * Handle the group only if the child is the migrator or if the group + * has no migrator. Otherwise the group is active and is handled by its + * own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + /* + * When there is a parent group and the CPU which triggered the + * hierarchy walk is not active, proceed the walk to reach the top level + * group before reading the next_expiry value. + */ + if (group->parent && !data->tmc_active) + goto out; + + /* + * On 32 bit systems the racy lockless check for next_expiry will + * turn into a random number generator. Therefore do the lockless + * check only on 64 bit systems. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + data->nextexp = READ_ONCE(group->next_expiry); + if (data->now >= data->nextexp) { + data->check = true; + return true; + } + } else { + raw_spin_lock(&group->lock); + data->nextexp = group->next_expiry; + if (data->now >= group->next_expiry) { + data->check = true; + raw_spin_unlock(&group->lock); + return true; + } + raw_spin_unlock(&group->lock); + } + +out: + /* Update of childmask for the next level */ + data->childmask = group->childmask; + return false; +} + +/** + * tmigr_requires_handle_remote() - Check the need of remote timer handling + * + * Must be called with interrupts disabled. + */ +int tmigr_requires_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + unsigned int ret = 0; + unsigned long jif; + + if (tmigr_is_not_available(tmc)) + return ret; + + data.now = get_jiffies_update(&jif); + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + data.tmc_active = !tmc->idle; + data.check = false; + + /* + * When the CPU is active, walking the hierarchy to check whether a + * remote expiry is required. + * + * Check is done lockless as interrupts are disabled and @tmc->idle is + * set only by the local CPU. + */ + if (!tmc->idle) { + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + if (data.nextexp != KTIME_MAX) + ret = 1; + + return ret; + } + + /* + * When the CPU is idle, check whether the recalculation of @tmc->wakeup + * is required. @tmc->wakeup_recalc is set by a remote CPU which is + * about to go offline, was the last active CPU in the whole timer + * migration hierarchy and now delegates handling of the hierarchy to + * this CPU. + * + * Racy lockless check is valid: + * - @tmc->wakeup_recalc is set by the remote CPU before it issues + * reschedule IPI. + * - As interrupts are disabled here this CPU will either observe + * @tmc->wakeup_recalc set before the reschedule IPI can be handled or + * it will observe it when this function is called again on return + * from handling the reschedule IPI. + */ + if (tmc->wakeup_recalc) { + raw_spin_lock(&tmc->lock); + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + if (data.nextexp != KTIME_MAX) + ret = 1; + + WRITE_ONCE(tmc->wakeup, data.nextexp); + tmc->wakeup_recalc = false; + raw_spin_unlock(&tmc->lock); + + return ret; + } + + /* + * When the CPU is idle and @tmc->wakeup is reliable, compare it with + * @data.now. On 64 bit it is valid to do this lockless. On 32 bit + * systems, holding the lock is required to get valid data on concurrent + * writers. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + if (data.now >= READ_ONCE(tmc->wakeup)) + ret = 1; + } else { + raw_spin_lock(&tmc->lock); + if (data.now >= tmc->wakeup) + ret = 1; + raw_spin_unlock(&tmc->lock); + } + + return ret; +} + +/** + * tmigr_cpu_new_timer() - enqueue next global timer into hierarchy (idle tmc) + * @nextexp: Next expiry of global timer (or KTIME_MAX if not) + * + * The CPU is already deactivated in the timer migration + * hierarchy. tick_nohz_get_sleep_length() calls tick_nohz_next_event() + * and thereby the timer idle path is executed once more. @tmc->wakeup + * holds the first timer, when the timer migration hierarchy is + * completely idle. + * + * Returns the first timer that needs to be handled by this CPU or KTIME_MAX if + * nothing needs to be done. + */ +u64 tmigr_cpu_new_timer(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (tmigr_is_not_available(tmc)) + return nextexp; + + raw_spin_lock(&tmc->lock); + + ret = READ_ONCE(tmc->wakeup); + if (nextexp != KTIME_MAX) { + if (nextexp != tmc->cpuevt.nextevt.expires || + tmc->cpuevt.ignore) { + ret = tmigr_new_timer(tmc, nextexp); + } + } else if (tmc->wakeup_recalc) { + struct tmigr_remote_data data; + + data.now = KTIME_MAX; + data.childmask = tmc->childmask; + data.nextexp = KTIME_MAX; + data.tmc_active = false; + data.check = false; + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + ret = data.nextexp; + } + tmc->wakeup_recalc = false; + + /* + * Make sure the reevaluation of timers in idle path will not miss an + * event. + */ + WRITE_ONCE(tmc->wakeup, ret); + + raw_spin_unlock(&tmc->lock); + return ret; +} + +static bool tmigr_inactive_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + /* Reset active bit when the child is no longer active */ + if (!data->childstate.active) + newstate.active &= ~childmask; + + if (newstate.migrator == childmask) { + /* + * Find a new migrator for the group, because the child group is + * idle! + */ + if (!data->childstate.active) { + unsigned long new_migr_bit, active = newstate.active; + + new_migr_bit = find_first_bit(&active, BIT_CNT); + + if (new_migr_bit != BIT_CNT) { + newstate.migrator = BIT(new_migr_bit); + } else { + newstate.migrator = TMIGR_NONE; + + /* Changes need to be propagated */ + walk_done = false; + } + } + } + + newstate.seq++; + + WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active)); + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + + /* + * Something changed in the child/parent group in the meantime, + * reread the state of the child and parent; Update of + * data->childstate is required for event handling; + */ + if (child) + data->childstate.state = atomic_read(&child->migr_state); + + goto retry; + } + + data->groupstate = newstate; + data->remote = false; + + /* Event Handling */ + tmigr_update_events(group, child, data); + + if (group->parent && (walk_done == false)) { + data->childmask = group->childmask; + data->childstate = newstate; + data->groupstate.state = atomic_read(&group->parent->migr_state); + } + + /* + * data->nextexp was set by tmigr_update_events() and contains the + * expiry of the first global event which needs to be handled + */ + if (data->nextexp != KTIME_MAX) { + WARN_ON_ONCE(group->parent); + /* + * Top level path - If this CPU is about going offline, wake + * up some random other CPU so it will take over the + * migrator duty and program its timer properly. Ideally + * wake the CPU with the closest expiry time, but that's + * overkill to figure out. + * + * Set wakeup_recalc of remote CPU, to make sure the complete + * idle hierarchy with enqueued timers is reevaluated. + */ + if (!(this_cpu_ptr(&tmigr_cpu)->online)) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned int cpu = smp_processor_id(); + struct tmigr_cpu *tmc_resched; + + cpu = cpumask_any_but(cpu_online_mask, cpu); + tmc_resched = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_unlock(&tmc->lock); + + raw_spin_lock(&tmc_resched->lock); + tmc_resched->wakeup_recalc = true; + raw_spin_unlock(&tmc_resched->lock); + + raw_spin_lock(&tmc->lock); + smp_send_reschedule(cpu); + } + } + + return walk_done; +} + +static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .childmask = tmc->childmask, + .evt = &tmc->cpuevt, + .nextexp = nextexp, + .childstate.state = 0 }; + + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + /* + * If nextexp is KTIME_MAX, the CPU event will be ignored because the + * local timer expires before the global timer, no global timer is set + * or CPU goes offline. + */ + if (nextexp != KTIME_MAX) + tmc->cpuevt.ignore = false; + + walk_groups(&tmigr_inactive_up, &data, tmc); + return data.nextexp; +} + +/** + * tmigr_cpu_deactivate() - Put current CPU into inactive state + * @nextexp: The next timer event expiry set in the current CPU + * + * Must be called with interrupts disabled. + * + * Return: the next event expiry of the current CPU or the next event expiry + * from the hierarchy if this CPU is the top level migrator or the hierarchy is + * completely idle. + */ +u64 tmigr_cpu_deactivate(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (tmigr_is_not_available(tmc)) + return nextexp; + + raw_spin_lock(&tmc->lock); + + ret = __tmigr_cpu_deactivate(tmc, nextexp); + + tmc->idle = true; + + /* + * Make sure the reevaluation of timers in idle path will not miss an + * event. + */ + WRITE_ONCE(tmc->wakeup, ret); + + raw_spin_unlock(&tmc->lock); + return ret; +} + +/** + * tmigr_quick_check() - Quick forecast of next tmigr event when CPU wants to + * go idle + * + * Returns KTIME_MAX, when it is probable that nothing has to be done (not the + * only one in the level 0 group; and if it is the only one in level 0 group, + * but there are more than a single group active in top level) + * + * Returns first expiry of the top level group, when it is the only one in level + * 0 and top level also only has a single active child. + */ +u64 tmigr_quick_check(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_group *topgroup; + struct list_head lvllist; + + if (tmigr_is_not_available(tmc)) + return KTIME_MAX; + + if (WARN_ON_ONCE(tmc->idle)) + return KTIME_MAX; + + if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask)) + return KTIME_MAX; + + for (int i = tmigr_hierarchy_levels; i > 0 ; i--) { + lvllist = tmigr_level_list[i - 1]; + if (list_is_singular(&lvllist)) { + topgroup = list_first_entry(&lvllist, struct tmigr_group, list); + + if (tmigr_check_lonely(topgroup)) + return READ_ONCE(topgroup->next_expiry); + } else { + continue; + } + } + + return KTIME_MAX; +} + +static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl, + int node) +{ + union tmigr_state s; + + raw_spin_lock_init(&group->lock); + + group->level = lvl; + group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE; + + group->num_children = 0; + + s.migrator = TMIGR_NONE; + s.active = 0; + s.seq = 0; + atomic_set(&group->migr_state, s.state); + + timerqueue_init_head(&group->events); + timerqueue_init(&group->groupevt.nextevt); + group->groupevt.nextevt.expires = KTIME_MAX; + WRITE_ONCE(group->next_expiry, KTIME_MAX); + group->groupevt.ignore = true; +} + +static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, + unsigned int lvl) +{ + struct tmigr_group *tmp, *group = NULL; + + lockdep_assert_held(&tmigr_mutex); + + /* Try to attach to an existing group first */ + list_for_each_entry(tmp, &tmigr_level_list[lvl], list) { + /* + * If @lvl is below the cross numa node level, check whether + * this group belongs to the same numa node. + */ + if (lvl < tmigr_crossnode_level && tmp->numa_node != node) + continue; + + /* Capacity left? */ + if (tmp->num_children >= TMIGR_CHILDREN_PER_GROUP) + continue; + + /* + * TODO: A possible further improvement: Make sure that all CPU + * siblings end up in the same group of the lowest level of the + * hierarchy. Rely on the topology sibling mask would be a + * reasonable solution. + */ + + group = tmp; + break; + } + + if (group) + return group; + + /* Allocate and set up a new group */ + group = kzalloc_node(sizeof(*group), GFP_KERNEL, node); + if (!group) + return ERR_PTR(-ENOMEM); + + tmigr_init_group(group, lvl, node); + + /* Setup successful. Add it to the hierarchy */ + list_add(&group->list, &tmigr_level_list[lvl]); + return group; +} + +static void tmigr_connect_child_parent(struct tmigr_group *child, + struct tmigr_group *parent) +{ + union tmigr_state childstate; + + raw_spin_lock_irq(&child->lock); + raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING); + + child->parent = parent; + child->childmask = BIT(parent->num_children++); + + raw_spin_unlock(&parent->lock); + raw_spin_unlock_irq(&child->lock); + + /* + * To prevent inconsistent states, active children need to be active in + * the new parent as well. Inactive children are already marked inactive + * in the parent group. + */ + childstate.state = atomic_read(&child->migr_state); + if (childstate.migrator != TMIGR_NONE) { + struct tmigr_walk data; + + data.childmask = child->childmask; + data.groupstate.state = atomic_read(&parent->migr_state); + + /* + * There is only one new level per time. When connecting the + * child and the parent and set the child active when the parent + * is inactive, the parent needs to be the uppermost + * level. Otherwise there went something wrong! + */ + WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent); + } +} + +static int tmigr_setup_groups(unsigned int cpu, unsigned int node) +{ + struct tmigr_group *group, *child, **stack; + int top = 0, err = 0, i = 0; + struct list_head *lvllist; + + stack = kcalloc(tmigr_hierarchy_levels, sizeof(*stack), GFP_KERNEL); + if (!stack) + return -ENOMEM; + + do { + group = tmigr_get_group(cpu, node, i); + if (IS_ERR(group)) { + err = PTR_ERR(group); + break; + } + + top = i; + stack[i++] = group; + + /* + * When booting only less CPUs of a system than CPUs are + * available, not all calculated hierarchy levels are required. + * + * The loop is aborted as soon as the highest level, which might + * be different from tmigr_hierarchy_levels, contains only a + * single group. + */ + if (group->parent || i == tmigr_hierarchy_levels || + (list_empty(&tmigr_level_list[i]) && + list_is_singular(&tmigr_level_list[i - 1]))) + break; + + } while (i < tmigr_hierarchy_levels); + + do { + group = stack[--i]; + + if (err < 0) { + list_del(&group->list); + kfree(group); + continue; + } + + WARN_ON_ONCE(i != group->level); + + /* + * Update tmc -> group / child -> group connection + */ + if (i == 0) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&group->lock); + + tmc->tmgroup = group; + tmc->childmask = BIT(group->num_children++); + + raw_spin_unlock_irq(&group->lock); + + /* There are no children that need to be connected */ + continue; + } else { + child = stack[i - 1]; + tmigr_connect_child_parent(child, group); + } + + /* check if uppermost level was newly created */ + if (top != i) + continue; + + WARN_ON_ONCE(top == 0); + + lvllist = &tmigr_level_list[top]; + if (group->num_children == 1 && list_is_singular(lvllist)) { + lvllist = &tmigr_level_list[top - 1]; + list_for_each_entry(child, lvllist, list) { + if (child->parent) + continue; + + tmigr_connect_child_parent(child, group); + } + } + } while (i > 0); + + kfree(stack); + + return err; +} + +static int tmigr_add_cpu(unsigned int cpu) +{ + int node = cpu_to_node(cpu); + int ret; + + mutex_lock(&tmigr_mutex); + ret = tmigr_setup_groups(cpu, node); + mutex_unlock(&tmigr_mutex); + + return ret; +} + +static int tmigr_cpu_online(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned int ret; + + /* First online attempt? Initialize CPU data */ + if (!tmc->tmgroup) { + raw_spin_lock_init(&tmc->lock); + + ret = tmigr_add_cpu(cpu); + if (ret < 0) + return ret; + + if (tmc->childmask == 0) + return -EINVAL; + + timerqueue_init(&tmc->cpuevt.nextevt); + tmc->cpuevt.nextevt.expires = KTIME_MAX; + tmc->cpuevt.ignore = true; + tmc->cpuevt.cpu = cpu; + + tmc->remote = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + } + raw_spin_lock_irq(&tmc->lock); + if (timer_base_is_idle()) + tmc->idle = true; + else + __tmigr_cpu_activate(tmc); + tmc->online = true; + raw_spin_unlock_irq(&tmc->lock); + return 0; +} + +static int tmigr_cpu_offline(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&tmc->lock); + tmc->online = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* + * CPU has to handle the local events on his own, when on the way to + * offline; Therefore nextevt value is set to KTIME_MAX + */ + __tmigr_cpu_deactivate(tmc, KTIME_MAX); + raw_spin_unlock_irq(&tmc->lock); + + return 0; +} + +static int __init tmigr_init(void) +{ + unsigned int cpulvl, nodelvl, cpus_per_node, i; + unsigned int nnodes = num_possible_nodes(); + unsigned int ncpus = num_possible_cpus(); + int ret = -ENOMEM; + + /* Nothing to do if running on UP */ + if (ncpus == 1) + return 0; + + /* + * Calculate the required hierarchy levels. Unfortunately there is no + * reliable information available, unless all possible CPUs have been + * brought up and all numa nodes are populated. + * + * Estimate the number of levels with the number of possible nodes and + * the number of possible CPUs. Assume CPUs are spread evenly across + * nodes. We cannot rely on cpumask_of_node() because there only already + * online CPUs are considered. + */ + cpus_per_node = DIV_ROUND_UP(ncpus, nnodes); + + /* Calc the hierarchy levels required to hold the CPUs of a node */ + cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + /* Calculate the extra levels to connect all nodes */ + nodelvl = DIV_ROUND_UP(order_base_2(nnodes), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + tmigr_hierarchy_levels = cpulvl + nodelvl; + + /* + * If a numa node spawns more than one CPU level group then the next + * level(s) of the hierarchy contains groups which handle all CPU groups + * of the same numa node. The level above goes across numa nodes. Store + * this information for the setup code to decide when node matching is + * not longer required. + */ + tmigr_crossnode_level = cpulvl; + + tmigr_level_list = kcalloc(tmigr_hierarchy_levels, sizeof(struct list_head), GFP_KERNEL); + if (!tmigr_level_list) + goto err; + + for (i = 0; i < tmigr_hierarchy_levels; i++) + INIT_LIST_HEAD(&tmigr_level_list[i]); + + pr_info("Timer migration: %d hierarchy levels; %d children per group;" + " %d crossnode level\n", + tmigr_hierarchy_levels, TMIGR_CHILDREN_PER_GROUP, + tmigr_crossnode_level); + + ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online", + tmigr_cpu_online, tmigr_cpu_offline); + if (ret) + goto err; + + return 0; + +err: + pr_err("Timer migration setup failed\n"); + return ret; +} +late_initcall(tmigr_init); diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h new file mode 100644 index 000000000000..260b87e5708d --- /dev/null +++ b/kernel/time/timer_migration.h @@ -0,0 +1,144 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _KERNEL_TIME_MIGRATION_H +#define _KERNEL_TIME_MIGRATION_H + +/* Per group capacity. Must be a power of 2! */ +#define TMIGR_CHILDREN_PER_GROUP 8 + +/** + * struct tmigr_event - a timer event associated to a CPU + * @nextevt: The node to enqueue an event in the parent group queue + * @cpu: The CPU to which this event belongs + * @ignore: Hint whether the event could be ignored; it is set when + * CPU or group is active; + */ +struct tmigr_event { + struct timerqueue_node nextevt; + unsigned int cpu; + bool ignore; +}; + +/** + * struct tmigr_group - timer migration hierarchy group + * @lock: Lock protecting the event information and group hierarchy + * information during setup + * @migr_state: State of the group (see union tmigr_state) + * @parent: Pointer to the parent group + * @groupevt: Next event of the group which is only used when the + * group is !active. The group event is then queued into + * the parent timer queue. + * Ignore bit of @groupevt is set when the group is active. + * @next_expiry: Base monotonic expiry time of the next event of the + * group; It is used for the racy lockless check whether a + * remote expiry is required; it is always reliable + * @events: Timer queue for child events queued in the group + * @childmask: childmask of the group in the parent group; is set + * during setup and will never change; could be read + * lockless + * @level: Hierarchy level of the group; Required during setup + * @list: List head that is added to the per level + * tmigr_level_list; is required during setup when a + * new group needs to be connected to the existing + * hierarchy groups + * @numa_node: Is set to numa node when level < tmigr_crossnode_level; + * otherwise it is set to NUMA_NO_NODE; Required for + * setup only to make sure CPUs and groups are per + * numa node as long as level < tmigr_crossnode_level + * @num_children: Counter of group children to make sure the group is only + * filled with TMIGR_CHILDREN_PER_GROUP; Required for setup + * only + */ +struct tmigr_group { + raw_spinlock_t lock; + atomic_t migr_state; + struct tmigr_group *parent; + struct tmigr_event groupevt; + u64 next_expiry; + struct timerqueue_head events; + u8 childmask; + unsigned int level; + struct list_head list; + int numa_node; + unsigned int num_children; +}; + +/** + * struct tmigr_cpu - timer migration per CPU group + * @lock: Lock protecting the tmigr_cpu group information + * @online: Indicates whether the CPU is online; In deactivate path + * it is required to know whether the migrator in the top + * level group is on the way to go offline when a timer is + * pending. Then another online CPU needs to be rescheduled + * to make sure the timers are handled properly; + * Furthermore the information is required in CPU hotplug + * path as the CPU is able to go idle before the timer + * migration hierarchy hotplug AP is reached. During this + * phase, the CPU has to handle the global timers by its + * own and does not act as a migrator. + * @idle: Indicates whether the CPU is idle in the timer migration + * hierarchy + * @remote: Is set when timers of the CPU are expired remote + * @wakeup_recalc: Indicates, whether a recalculation of the @wakeup value + * is required. It is only used when the CPU is marked idle + * in the timer migration hierarchy. + * @tmgroup: Pointer to the parent group + * @childmask: childmask of tmigr_cpu in the parent group + * @wakeup: Stores the first timer when the timer migration + * hierarchy is completely idle and remote expiry was done; + * is returned to timer code in the idle path; it is only + * valid, when @wakeup_recalc is not set + * @cpuevt: CPU event which could be queued into the parent group + */ +struct tmigr_cpu { + raw_spinlock_t lock; + bool online; + bool idle; + bool remote; + bool wakeup_recalc; + struct tmigr_group *tmgroup; + u8 childmask; + u64 wakeup; + struct tmigr_event cpuevt; +}; + +/** + * union tmigr_state - state of tmigr_group + * @state: Combined version of the state - only used for atomic + * read/cmpxchg function + * @struct: Split version of the state - only use the struct members to + * update information to stay independent of endianness + */ +union tmigr_state { + u32 state; + /** + * struct - split state of tmigr_group + * @active: Contains each childmask bit of the active children + * @migrator: Contains childmask of the child which is migrator + * @seq: Sequence counter needs to be increased when an update + * to the tmigr_state is done. It prevents a race when + * updates in the child groups are propagated in changed + * order. Detailed information about the scenario is + * given in the documentation at the begin of + * timer_migration.c. + */ + struct { + u8 active; + u8 migrator; + u16 seq; + } __packed; +}; + +#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) +extern void tmigr_handle_remote(void); +extern int tmigr_requires_handle_remote(void); +extern void tmigr_cpu_activate(void); +extern u64 tmigr_cpu_deactivate(u64 nextevt); +extern u64 tmigr_cpu_new_timer(u64 nextevt); +extern u64 tmigr_quick_check(void); +#else +static inline void tmigr_handle_remote(void) { } +static inline int tmigr_requires_handle_remote(void) { return 0; } +static inline void tmigr_cpu_activate(void) { } +#endif + +#endif From patchwork Fri Dec 1 09:26:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172371 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp993136vqy; Fri, 1 Dec 2023 01:31:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGMfmvXWFUGi3OxAYBTIsgF9x8Sel/fih/lZRG370L9CbG8iQbd8d04iYCJnWkC0kCVm8HS X-Received: by 2002:a17:90b:380b:b0:285:da91:69d6 with SMTP id mq11-20020a17090b380b00b00285da9169d6mr15861296pjb.36.1701423079081; Fri, 01 Dec 2023 01:31:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701423079; cv=none; d=google.com; s=arc-20160816; b=N6BpNtudkQxh6fUhIt2lQTpBw+iSvgrY+Fgt5qYDWRXd35f+gXuLyzJxLkAmM2hPGh WwbTWkCQtU1+tp9ynCwFVFatPpf2grZGqI+QvSnV2vTyk5qi9cR45uk65SNZK1pSlZE3 9TuaXQMaUJcKJn6pgwOGLOLdsD0445gLhpryC/5tCajrgsAoPsMtOwRSFiG8g2HLbp88 jQcKXieh722jfUCwdBebOoPyxDtSGZ+fr58PDdS6LaCe5ek5DxqO3WgOoVGkH7pIxA5e 63Y2QWHGyVWykLZSdcjbngq2/U0FxbKfXV6CNLMrSpFdGP1dOa7OAFVXlL7eQ8/b8tU3 /cgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=ZXDeEjIBNXYe4LZ65KgmZyDO9TwDwGVuKUpoOVmxOWc=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=XdHT0VXU4ASxRETbFDvBTqvWTa7e3rOV36MvjRSJi4w1D39vyO+94zjD4j5hbOP89c 91XssIASdpQUnM0qBvTEc504MMn6KCcxxhniDEtaPsTCNjSXpc8eAeiJvaNyxAk+hyEE DnPVu2Tamg37pmHTz7VBGeez13QlCmp7nMU3DSjaj0eq0vTNiKhon6Rg2TQ83I76+210 ZfMyCNCtDIYK7Hzi9Ru6h9WY6vI14CJTnnkS4A+YvhsPgQ6IEsKdEmfWUGQ+oybVOsNg 8skZ3NYX6Yuj7vBQMoicT7boURO/YWW6VY1SKvxEn1KZcryOqT51Rk55LYrDwnUQA/1x fYCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=zm31mmUk; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id me17-20020a17090b17d100b0027d74e8a50bsi3288531pjb.69.2023.12.01.01.31.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:31:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=zm31mmUk; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 3EBAB810FBD7; Fri, 1 Dec 2023 01:29:59 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378279AbjLAJ3J (ORCPT + 99 others); Fri, 1 Dec 2023 04:29:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378148AbjLAJ2R (ORCPT ); Fri, 1 Dec 2023 04:28:17 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FFB11FD9 for ; Fri, 1 Dec 2023 01:27:30 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZXDeEjIBNXYe4LZ65KgmZyDO9TwDwGVuKUpoOVmxOWc=; b=zm31mmUk2IjTpxPYsC6/ialV0ay+XIkmitMWTaydrVJdz6B3TKLisJeKPPGvg5dErxwEHD v9zjHLfgWJxRLchQkayhTIjIgJQm/FPDfari+Kp5ipgw/Bd3gJWDxQGWHh0YDoMQPEKwDg Yp3Z7YUzWpgAluAUJSReQlykqtm+8koH1B1inrAlm4GuOAYcbezZm+j5BquS5UQF+f5JRG hNbP91HjSVbq2cS6E00VIvm/lnapYOuMjyAhFAsICEd3oJJcPGzk9BTnCxkSUh6BpHdcYp DDHbfH/zwH04KxwfIf5j5Wzz+Pm7l1nEHNcC1NJwDRMGund+pT0fueJ1hrOsIA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZXDeEjIBNXYe4LZ65KgmZyDO9TwDwGVuKUpoOVmxOWc=; b=p9mSNC+mdFKvEq4pxhCmglgSSShce5hxYxXZE6i/056/w2VrmsQ8Me+Ms3lu+DRytfJ61c 7JzJMkCgoePaajBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v9 31/32] timer_migration: Add tracepoints Date: Fri, 1 Dec 2023 10:26:53 +0100 Message-Id: <20231201092654.34614-32-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071406535326493 X-GMAIL-MSGID: 1784071406535326493 The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen --- v9: Add tmigr_cpu_new_timer_idle tracepoint v8: Add wakeup value to tracepoints --- include/trace/events/timer_migration.h | 297 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 26 +++ 2 files changed, 323 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..a2e7e32058f8 --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->numa_node = child->parent->num_children; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->numa_node = tmc->tmgroup->num_children; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p wakeup=%llu", __entry->cpu, __entry->parent, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DECLARE_EVENT_CLASS(tmigr_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu wakeup=%llu", + __entry->cpu, __entry->parent, __entry->nextevt, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_new_timer_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index 05cd8f1bc45d..54ab18ccc62b 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -14,6 +14,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -511,6 +514,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ group->groupevt.ignore = true; + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -521,6 +526,8 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) data.childmask = tmc->childmask; data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + trace_tmigr_cpu_active(tmc); + tmc->cpuevt.ignore = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); tmc->wakeup_recalc = false; @@ -688,6 +695,9 @@ static bool tmigr_update_events(struct tmigr_group *group, data->nextexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, data->childstate, + data->groupstate, nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -721,6 +731,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + tmc->cpuevt.ignore = false; data.remote = false; @@ -754,6 +766,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return next; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); @@ -838,6 +852,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -1101,6 +1116,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_new_timer_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1210,6 +1226,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, } } + trace_tmigr_group_set_cpu_inactive(group, newstate, childmask); + return walk_done; } @@ -1264,6 +1282,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1376,6 +1395,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -1393,6 +1413,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irq(&child->lock); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active children need to be active in * the new parent as well. Inactive children are already marked inactive @@ -1474,6 +1496,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irq(&group->lock); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no children that need to be connected */ continue; } else { @@ -1541,6 +1565,7 @@ static int tmigr_cpu_online(unsigned int cpu) WRITE_ONCE(tmc->wakeup, KTIME_MAX); } raw_spin_lock_irq(&tmc->lock); + trace_tmigr_cpu_online(tmc); if (timer_base_is_idle()) tmc->idle = true; else @@ -1563,6 +1588,7 @@ static int tmigr_cpu_offline(unsigned int cpu) * offline; Therefore nextevt value is set to KTIME_MAX */ __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); return 0; From patchwork Fri Dec 1 09:26:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 172366 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp992296vqy; Fri, 1 Dec 2023 01:29:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IEL7bRi+MhStebSO7ExDT+QzdrzNBKeBx9c6d1Fz8sNwFs3l+nfgx3g+BBc6fsYMQ2r3EE+ X-Received: by 2002:a17:90b:38d2:b0:285:b99a:a932 with SMTP id nn18-20020a17090b38d200b00285b99aa932mr20281129pjb.17.1701422963013; Fri, 01 Dec 2023 01:29:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701422962; cv=none; d=google.com; s=arc-20160816; b=y5JrzGtIIQBoyYbKakG8Cr1Le6NOirFgUpUrJ96EwsGdi9qPSs+YpDcCQMviUgDyih kGfyGHtNbNAesRMIVrTOzWbjhr/0brLO5CL4DlUF3pFAUFxLIBN6OgV7dF8qkDGNjJsf wTM65eDglJUEyVfDXYS2IfzIfcmLDY/wJjgAzRdwAU8VhEA47cYsHhv5kBdBqEVT5F0V RhdrYiSR26YW587aJ++k9T8kDQHNEQWumtNhm6CnI0Wm5pirG1SaO9O/QUpSdgu64u2+ XF/aoWtIeVRn1KBc33yUtlTO0qNLvlSJQJq71pxuLbM8+HgCqA0xXXKpq5MNPj485uXu 1KlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=QYexXRgS2m5/dffYZ2zGFXnxjxh6wHD+OeXAnIrdHS4=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=1Bwn5EaNc+XzNzZxEr7Hu782avrmwfOPX1eAFu6FmIRdK9KUPxtJ7X0aFkSnaDZ8eo vR33n34K49jTvR8OkE0SZSIBkcbaeVSBsC25PRx8QSIMPMJwsAq64W0uF1XBUUT0w8X9 YfqBPfmXfUfbEEMw8nC16XIQ4FN0KlXnidQJBh/I8IYFpM7h4trgEGWfsJKGnTSr43Rd jKJ43b41FqyWsnlovB6S5kc0vluCdfmHLYctSb5My9L24xxqPwGw8PxvbrZwQkOKUOLn VMVCdFKOIpLYGff7IWSm4Hb6zw6wLn//D9iJVL4oqYJZDTrL/6s3SeUALp0tc95Bqir5 +bew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=CaKDJWrM; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id y22-20020a17090aa41600b00283a2ae9f65si5326002pjp.92.2023.12.01.01.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 01:29:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=CaKDJWrM; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 520278160385; Fri, 1 Dec 2023 01:29:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378141AbjLAJ3G (ORCPT + 99 others); Fri, 1 Dec 2023 04:29:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378201AbjLAJ2R (ORCPT ); Fri, 1 Dec 2023 04:28:17 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D88E1FDD for ; Fri, 1 Dec 2023 01:27:30 -0800 (PST) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1701422849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QYexXRgS2m5/dffYZ2zGFXnxjxh6wHD+OeXAnIrdHS4=; b=CaKDJWrMIQ2XDsHebkDLVj6vhfpY1bVbxlutZsI2FXCWK5GlSqOp9ycw/krKlnZcFKGsaX jSZcx3uZRM97ThMaXYFFFJmCWI864VKBNRpf92bkHuN3DykbJ7cDnWITVhwupQzdBiMR3C VDhuTVto3q6v34p/0MU4EEmoa+i01EcMnxju8dE+Lb9C7Qubv7nrrElqNIBRevESh3OjqH Ky8ewq1ml+IcXQspz+Wrw1MZ0tUyQ2JHWZuX5fn8Vc/JxL9l60YggYqiXRQsnsAftvNM5f g9gDlKPSCoZlnABDbZ1fGARLiMBLefQ5l8uNx3F8nCiWoJIR+yEnQIH+TaC+cQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1701422849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QYexXRgS2m5/dffYZ2zGFXnxjxh6wHD+OeXAnIrdHS4=; b=A0lpCutDv4FDkeQp+mxGZD/cO52q4pGv2KnxDGTXksG1RQRM0/MTNKD+h5e0wys3W7C0zA IZU3vnjknhOnYjAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v9 32/32] timers: Always queue timers on the local CPU Date: Fri, 1 Dec 2023 10:26:54 +0100 Message-Id: <20231201092654.34614-33-anna-maria@linutronix.de> In-Reply-To: <20231201092654.34614-1-anna-maria@linutronix.de> References: <20231201092654.34614-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 01 Dec 2023 01:29:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784071284794854477 X-GMAIL-MSGID: 1784071284794854477 The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the separate storage and eventually pulled at expiry time to a remote CPU. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- v9: - Update to the changes of the preceding patches v6: - Update TIMER_PINNED flag description. v5: - Move WARN_ONCE() in add_timer_on() into a previous patch - Fold crystallball magic related hunks into this patch v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 14 ++++--------- kernel/time/timer.c | 46 +++++++++++++++++++++---------------------- 2 files changed, 26 insertions(+), 34 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 404bb31a95c7..4dd59e4e5681 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -50,16 +50,10 @@ struct timer_list { * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. - * - * Note: Because enqueuing of timers can migrate the timer from one - * CPU to another, pinned timers are not guaranteed to stay on the - * initialy selected CPU. They move to the CPU on which the enqueue - * function is invoked via mod_timer() or add_timer(). If the timer - * should be placed on a particular CPU, then add_timer_on() has to be - * used. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the + * timer was enqueued. When a particular CPU is required, add_timer_on() + * has to be used. Enqueue via mod_timer() and add_timer() is always done + * on the local CPU. */ #define TIMER_CPUMASK 0x0003FFFF #define TIMER_MIGRATING 0x00040000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index ac3e888d053f..6e9e1d852438 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle && timer->flags & TIMER_PINNED) wake_up_nohz_cpu(base->cpu); } @@ -941,17 +944,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void __forward_timer_base(struct timer_base *base, unsigned long basej) { @@ -1106,7 +1098,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -2228,11 +2220,17 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * BASE_GLOBAL base, deferrable timers may still see large * granularity skew (by design). */ - if (!base_local->is_idle) { - bool is_idle = time_after(nextevt, basej + 1); - base_local->is_idle = base_global->is_idle = is_idle; - } + /* + * base->is_idle information is required to wakeup an idle CPU + * when a new timer was enqueued. Only pinned timers could be + * enqueued remotely into a idle base. Therefore do maintain + * only base_local->is_idle information and ignore + * base_global->is_idle information. + */ + if (!base_local->is_idle) + base_local->is_idle = time_after(nextevt, basej + 1); + *idle = base_local->is_idle; trace_timer_base_idle(base_local->is_idle, base_local->cpu); @@ -2307,13 +2305,13 @@ bool timer_base_is_idle(void) void timer_clear_idle(void) { /* - * We do this unlocked. The worst outcome is a remote enqueue sending - * a pointless IPI, but taking the lock would just make the window for - * sending the IPI a few instructions smaller for the cost of taking - * the lock in the exit from idle path. + * We do this unlocked. The worst outcome is a remote pinned timer + * enqueue sending a pointless IPI, but taking the lock would just + * make the window for sending the IPI a few instructions smaller + * for the cost of taking the lock in the exit from idle + * path. Required for BASE_LOCAL only. */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); - __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(0, smp_processor_id());