From patchwork Mon Jan 15 14:37:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188179 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1742778dyc; Mon, 15 Jan 2024 06:38:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IGIKJG39DIJPda29Kn6o3KM6rCIT5Jhh9ViZx6Aa8XmlXrUHbdOifgAufmsksYcmN3neyLQ X-Received: by 2002:a05:620a:a95:b0:783:539a:f30e with SMTP id v21-20020a05620a0a9500b00783539af30emr3970919qkg.83.1705329524476; Mon, 15 Jan 2024 06:38:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329524; cv=none; d=google.com; s=arc-20160816; b=NHmHVtnwDkkpQc1mZt8mS5AlF2LEoqZ857atIFkxBdXfXK0NIgkyeApPNCRwLp/49x zYGgpTE4xrPZQ7hwoYslltuzlI24ahJwx3B6sib7RfKTkmBGKElK1V8bMSUT1zmQHKd/ vNWV5/zbVK6vJpPqNrZB8nodO8N+3Fy/8ehL3Y1lUmyro7ra4MaYJi95O1riA53g+ZsA cYj0+npaYTNxkT5tJQP27UUeDw4GIWWVNnvwcKTLvOXzFCyHNB5Acq086Es/Rc6iiKk9 6C89cL1yEjx/yWQ+RQLMiEolGz3IDCzKK/FbdEKNLJOo/uAu3MhflEWvorPNGpTilakH Wm6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=MI9g9MDbA2NFp7jgDttymOb2sC6nFxhp2a/RZS0Cubo=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=itdn2L5Hvy3e95axhRwwGfM0f2UgK8KFpuHKdc3QcmO7j0bZS3XrFTDLQOirydrAc6 iczp6VxQbrK5GT4GdBX1yc4NXzBjFqbQmEIGiGer5W3L6zpUp5Lsg/izviaxkyZ1AWq+ VLALHX9UdIqOWZNlT4OEhDC0OQoRXD2VtWLaC1q439iIBV3m2Ch3byOgEz9iCP4Wuo0Z f5ra2akmi7jAv99d3oHf47eS7/fIx9O92eU0tJlQ3Rsmb4w0LfEE11bJzdbSrWrdk2dQ DcNcwDOIqMPhEK3XX7KY6Fly7DcQuWL9mjFunjlDEW7/IcfKYc5MGdFtYrL05aaa4VUu 16Xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dKME5u9Y; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26096-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26096-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id g18-20020a05620a40d200b00783420fb6f3si8582469qko.314.2024.01.15.06.38.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:38:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26096-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dKME5u9Y; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26096-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26096-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 0178F1C21B61 for ; Mon, 15 Jan 2024 14:38:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D2BB317BA5; Mon, 15 Jan 2024 14:38:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="dKME5u9Y"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="BZs3wXhs" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D62DD17738 for ; Mon, 15 Jan 2024 14:38:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MI9g9MDbA2NFp7jgDttymOb2sC6nFxhp2a/RZS0Cubo=; b=dKME5u9YJRG9rJBXAK+YQgOciaQpeurM1q2gGo7H/jfxRnKrlC/PnxxVZXurGGwTY+rmXc Ldmk2exeCLEOEpJhE3PzGps785U+vNDBLqTn/GQfF9YZnpR4Do53XoTrYwb6bwo31kTeQq h5Qe4It9XifK89MSwYxtYcSB9Nzs91HFVu99rFZ/9GwT3OAZ4vD+fCvCr/ZJnam3Bv8wIc Gk+QuEUicMziS1+XSLjIyuAyU3+vIOjzIEnA5A01l4IoBk1rbkhl2sfMlrJnmB4Qpqgjgy +b1bwYF0jt3dKgrbnyUkhtZei+IqjaSTu/NHB2mzluaCybNsFaJTg6xgV+AxnQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MI9g9MDbA2NFp7jgDttymOb2sC6nFxhp2a/RZS0Cubo=; b=BZs3wXhsMMmkrUob5t5tgA7S+fbdHDyNyUea31/wUAg5nASEUBX78LfwVe21ZJgJzXKJGw ftF9HXeATwEkVvCQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 01/20] timers: Restructure get_next_timer_interrupt() Date: Mon, 15 Jan 2024 15:37:24 +0100 Message-Id: <20240115143743.27827-2-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167611546109073 X-GMAIL-MSGID: 1788167611546109073 get_next_timer_interrupt() contains two parts for the next timer interrupt calculation. Those two parts are separated by forwarding the base clock. But the second part does not depend on the forwarded base clock. Therefore restructure get_next_timer_interrupt() to keep things together which belong together. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 352b161113cd..5f21db4fa3ca 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1937,12 +1937,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (base->next_expiry_recalc) next_expiry_recalc(base); - /* - * We have a fresh next event. Check whether we can forward the - * base. - */ - __forward_timer_base(base, basej); - if (base->timers_pending) { nextevt = base->next_expiry; @@ -1960,6 +1954,12 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) base->next_expiry = nextevt; } + /* + * We have a fresh next event. Check whether we can forward the + * base. + */ + __forward_timer_base(base, basej); + /* * Base is idle if the next event is more than a tick away. * From patchwork Mon Jan 15 14:37:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188180 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1742852dyc; Mon, 15 Jan 2024 06:38:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IGBGhNksu0kjRfQSygXBhIN7WX9+twqM8Z3O5MrliQ/IO8AVfAOkO9PISpwMbxle7js6f0U X-Received: by 2002:a05:622a:44b:b0:428:3cc5:1350 with SMTP id o11-20020a05622a044b00b004283cc51350mr7660252qtx.7.1705329532847; Mon, 15 Jan 2024 06:38:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329532; cv=none; d=google.com; s=arc-20160816; b=RdtqLfiDQQl2wYTrsgICfB/C5ptqUYEYuj309EgEyGqr9zZHMnh1XiX306ytcpWrsO QYRHYEI2AU6JXydShYKKAPHMP8BwhixWJZ4f8UhGTlazH7rqqoUIUKevTWYS6pS//5g2 OSHIHFmdz5IY36bzYB9s70jgGRV9Ge2D09pc7EVGuD4bblbEknwouhSPlZK6lTuxMhkp HfZ+ZDZxbQR3RJHPtBd3tyvA4wm7o34QtEBDlHwu970HU4V1ypfR2dgs5anoiD85jAyE Tdu7/5qm9shUQzxWxn6p6sp3ZCxvH3GPB7/QQS3XqC2WI5/RRw8nB01roRKu8uuUWtzJ f2iA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=TrRJNa/4dVg7BTloDs+oOr0KGfNkWbg12FR3FDPf3kI=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=z2fmAhWn6tlYB0s1Xc9HGCT/OafAg30kBL2h8IjhPbzJS4l932kI641KFAnZX2D+Ll 7umAQWmQh+BM4FnWfXDZcApHHfgf+UeJA2JwXblHN74zclcWI9vY0QpiTiKevtA3Ps1V Oc1yR07hdv1y1nGcR2aW8cNN8gfWj2jgVgjIJ3Yx6R6ljfLii/4LG+2rd16qDeQRp/nU YKRUMZiywnvI4csSa7iTyHDRxVwWzzB2zyeuFpJyonxMI5u3DCh2tzHFcKY5kcKKXQy+ Qfzs1Q9KwtfXgZqE0QZzDiYkIZlH3GZXtniJ4UvKvJuS5Lb4PLT/zhCHyy4/LYJWIRrX thPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=iqQYd40g; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26098-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26098-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id s5-20020ac85cc5000000b00429a53e70aesi8185807qta.140.2024.01.15.06.38.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:38:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26098-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=iqQYd40g; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26098-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26098-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id A329C1C21B30 for ; Mon, 15 Jan 2024 14:38:52 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 638C717BBA; Mon, 15 Jan 2024 14:38:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="iqQYd40g"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="jpYB6+8/" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A87C1773A for ; Mon, 15 Jan 2024 14:38:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TrRJNa/4dVg7BTloDs+oOr0KGfNkWbg12FR3FDPf3kI=; b=iqQYd40g6jjZ8kvPdnidNQc8zqE5Zq5Oyd0ENex12OAECMGf7KJPZOxPlpgecMDOfNqmjp iSy92IYa4iyCzHf9wnuCjjdxhwFefVqTiZhARgqzAOiIsvFWKHCR6BK/KNe2vdP5Ua7sB1 vAXuf2/mj9R/MgJpjnqHHWbls4jIzdmidCZzDt3zJQFSV0knI5FtJFmMI3/Hii6OM9QZIy jvNw99w1eN8BfV4GjgNbYAjA/7fvkozUcKJa1qjp+T8O9TDalR84pVax7b9P/1FamlfGCe ma7GDWmdCIter2i5Q6ndvM5UikKP/QzDrf8+OcsFKhDYPIOPWSymcysUKyLXGg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TrRJNa/4dVg7BTloDs+oOr0KGfNkWbg12FR3FDPf3kI=; b=jpYB6+8/2KkjAvRHid8a5at40ByDrZXjwLFmJvXJPlY2UnGF/8Tkauw4/zFeqbbofToFQd OONQDpUIoneCDnBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 02/20] timers: Split out get next timer interrupt Date: Mon, 15 Jan 2024 15:37:25 +0100 Message-Id: <20240115143743.27827-3-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167620007384832 X-GMAIL-MSGID: 1788167620007384832 Split out get_next_timer_interrupt() to be able to extend it and make it reusable for other call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 5f21db4fa3ca..2aea55d53416 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,15 +1911,7 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -/** - * get_next_timer_interrupt - return the time (clock mono) of the next timer - * @basej: base time jiffies - * @basem: base time clock monotonic - * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. - */ -u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; @@ -1978,6 +1970,19 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return cmp_next_hrtimer_event(basem, expires); } +/** + * get_next_timer_interrupt() - return the time (clock mono) of the next timer + * @basej: base time jiffies + * @basem: base time clock monotonic + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +{ + return __get_next_timer_interrupt(basej, basem); +} + /** * timer_clear_idle - Clear the idle state of the timer base * From patchwork Mon Jan 15 14:37:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188182 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743306dyc; Mon, 15 Jan 2024 06:39:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTSbYxgIAkrfuEu16vseZkmoUsuII5nGiG0VaYtZf+Mo92yF2fQT909OzlFP8CHBDpNoKo X-Received: by 2002:a17:90a:1b8b:b0:286:a089:c3dd with SMTP id w11-20020a17090a1b8b00b00286a089c3ddmr3124371pjc.62.1705329585494; Mon, 15 Jan 2024 06:39:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329585; cv=none; d=google.com; s=arc-20160816; b=Nm9f6VcMZ/+A6oGH5xFFqcOBDHBRVmU6zo7AjAZtnlG22Ixz/NNh5HqoSNqSJ+yTwh dIDLFehrL7ZWmmcIAt2yLbTJcnqdDuGggPCE7HYlgWR5X3TdmSqiXY/jGIAJYVYQPhBR BALjPzQeuTkkmPmvvnFUl1QuNEH4JKjWPY4oe8PjP423YoA5y/fA1+XKkRytl5wSBVNL cHQyR4T7V4/TVwQOQUFY3sXkARQScnA5gNeRmHJ8LwQTIBTCvWrcokPUhcsyBjmc4dMU oprL5mPhjXtg6FeEeAlUfze4/YUebQmRVw1jzM3uy34SFAb8aZsTzVL0LWosisGmvv2j E1zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=cawu2qRxujfpAJkXtS4G5pM5Zis15l5+RosRC2CHij0=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=j6ri9IYc12BVAWLs03t6/K1apvqWYxvbRXBJjudn5vhHHZ98MRPBCxTq3WWKQvaTCb LMxxw4shDdJVq7TUmM4OF+MKpZgpF+b86s+pWigfeDTxfTj3RpCrIk3yLmWnjFxpm2nj STuPqXuUKS73CIeJrs0OyO3G9R80DpDelVxK89KTh/0V0qcItP6fjsnKzFze/Vl4PfLw ArxUTPGIyDLLamXaYHp4THEASKxWwxJRHZsmy1FHxJ7KGXSadMFbHyxtSOap1/5Hug4f IW1NN6d88J4ps+ozXJoam5qMShu8aaT82gQIce1Z7lrmDxKzeqCdIWTcf5OdKNY6JiXm yCZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=W85lh00Z; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26099-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26099-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x21-20020a17090aca1500b0028d29091591si12140301pjt.144.2024.01.15.06.39.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:39:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26099-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=W85lh00Z; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26099-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26099-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 448302827BC for ; Mon, 15 Jan 2024 14:39:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CC708182B9; Mon, 15 Jan 2024 14:38:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="W85lh00Z"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="rwASbY7z" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C791F17732 for ; Mon, 15 Jan 2024 14:38:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cawu2qRxujfpAJkXtS4G5pM5Zis15l5+RosRC2CHij0=; b=W85lh00Z8Bd2TwnSXwhuTY7NxPs3Pbd0tRT8/E7ERMFhpeJOMKCEFl+agh81alVUlY95ZR Rf5eohvgqFiyP4Mu3j5y41QEJIxSNET9l/4PU4HMy7mgnL2Xh4fPKmgpsb3sIdHu/3l9wT 2kfMA3WS3y7yji4FbFMuuUx1+aOyXmy2lPGFO81HWDa/sR0JL2JcY3DDp5vhFrj64ImSZg AazX2KiBDfv3KZl6v4TFxVjj1UnIRgSYEokDS2/1bk71nsfM0Y4BVZ6G3jNZJCWFnBsfNj W0M+YGjEiQ55WBxl33gPg9rT+2eaQpDmIP84MWG/34sjTAtZaYKcNvSkBLhkaA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cawu2qRxujfpAJkXtS4G5pM5Zis15l5+RosRC2CHij0=; b=rwASbY7zJUfUnjudKyyiqJI4YgwJVGLnhdA9+Rn+0LHFic2vInNiDAfCkIVWw0hOj70W9b iGrWq4H2zN3z0VDg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick() Date: Mon, 15 Jan 2024 15:37:26 +0100 Message-Id: <20240115143743.27827-4-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167675449071043 X-GMAIL-MSGID: 1788167675449071043 tick_nohz_stop_tick() The timer base is marked idle when get_next_timer_interrupt() is executed. But the decision whether the tick will be stopped and whether the system is able to go idle is done later. When the timer bases is marked idle and a new first timer is enqueued remote an IPI is raised. Even if it is not required because the tick is not stopped and the timer base is evaluated again at the next tick. To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and get_next_timer_interrupt() is streamlined by only looking for the next timer interrupt. All other work is postponed to timer_base_try_to_set_idle() which is called by tick_nohz_stop_tick(). With this, tick_sched::tick_stopped and timer_base::is_idle is always in sync. So there is no longer the need to execute timer_clear_idle() in tick_nohz_idle_retain_tick(). This was required before, as tick_nohz_next_event() set timer_base::is_idle even if the tick would not be stopped. So timer_clear_idle() is only executed, when timer base is idle. So the check whether timer base is idle, is now no longer required as well. While at it fix some nearby whitespace damage as well. Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 47 +++++++++++++++++++++-------- kernel/time/timer.c | 60 ++++++++++++++++++++++++++----------- 3 files changed, 78 insertions(+), 30 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 481b7ab65e2c..47df30b871e4 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -163,6 +163,7 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem); +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index a17d26002831..c6223afc801f 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -849,11 +849,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) */ delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { - /* - * Tell the timer code that the base is not idle, i.e. undo - * the effect of get_next_timer_interrupt(): - */ - timer_clear_idle(); /* * We've not stopped the tick yet, and there's a timer in the * next period, so no point in stopping it either, bail. @@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) { struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); + unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - u64 expires = ts->timer_expires; + bool timer_idle; + u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; + /* + * Now the tick should be stopped definitely - so the timer base needs + * to be marked idle as well to not miss a newly queued timer. + */ + expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle); + if (!timer_idle) { + /* + * Do not clear tick_stopped here when it was already set - it + * will be retained on the next idle iteration when the tick + * expired earlier than expected. + */ + expires = basemono + TICK_NSEC; + } else if (expires > ts->timer_expires) { + /* + * This path could only happen when the first timer was removed + * between calculating the possible sleep length and now (when + * high resolution mode is not active, timer could also be a + * hrtimer). + * + * We have to stick to the original calculated expiry value to + * not stop the tick for too long with a shallow C-state (which + * was programmed by cpuidle because of an early next expiration + * value). + */ + expires = ts->timer_expires; + } + /* * If this CPU is the one which updates jiffies, then give up * the assignment and let it be taken by the CPU which runs @@ -930,6 +954,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) * scheduler tick in tick_nohz_restart_sched_tick(). */ if (!ts->tick_stopped) { + /* If the timer base is not idle, retain the tick. */ + if (!timer_idle) + return; + calc_load_nohz_start(); quiet_vmstat(); @@ -991,7 +1019,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now) touch_softlockup_watchdog_sched(); /* Cancel the scheduled timer and restore the tick: */ - ts->tick_stopped = 0; + ts->tick_stopped = 0; tick_nohz_restart(ts, now); } @@ -1147,11 +1175,6 @@ void tick_nohz_idle_stop_tick(void) void tick_nohz_idle_retain_tick(void) { tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); - /* - * Undo the effect of get_next_timer_interrupt() called from - * tick_nohz_next_event(). - */ - timer_clear_idle(); } /** diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2aea55d53416..3a668060692e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,19 +1911,22 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; u64 expires = KTIME_MAX; - bool was_idle; /* * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ - if (cpu_is_offline(smp_processor_id())) + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; return expires; + } raw_spin_lock(&base->lock); if (base->next_expiry_recalc) @@ -1953,17 +1956,26 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) __forward_timer_base(base, basej); /* - * Base is idle if the next event is more than a tick away. - * - * If the base is marked idle then any timer add operation must forward - * the base clk itself to keep granularity small. This idle logic is - * only maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ - was_idle = base->is_idle; - base->is_idle = time_after(nextevt, basej + 1); - if (was_idle != base->is_idle) - trace_timer_base_idle(base->is_idle, base->cpu); + if (idle) { + /* + * Base is idle if the next event is more than a tick away. + * + * If the base is marked idle then any timer add operation must + * forward the base clk itself to keep granularity small. This + * idle logic is only maintained for the BASE_STD base, + * deferrable timers may still see large granularity skew (by + * design). + */ + if (!base->is_idle) { + if (time_after(nextevt, basej + 1)) { + base->is_idle = true; + trace_timer_base_idle(true, base->cpu); + } + } + *idle = base->is_idle; + } raw_spin_unlock(&base->lock); @@ -1980,7 +1992,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - return __get_next_timer_interrupt(basej, basem); + return __get_next_timer_interrupt(basej, basem, NULL); +} + +/** + * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases + * @basej: base time jiffies + * @basem: base time clock monotonic + * @idle: pointer to store the value of timer_base->is_idle + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) +{ + return __get_next_timer_interrupt(basej, basem, idle); } /** @@ -1998,10 +2024,8 @@ void timer_clear_idle(void) * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - if (base->is_idle) { - base->is_idle = false; - trace_timer_base_idle(false, smp_processor_id()); - } + base->is_idle = false; + trace_timer_base_idle(false, smp_processor_id()); } #endif From patchwork Mon Jan 15 14:37:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188181 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743142dyc; Mon, 15 Jan 2024 06:39:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IFg9rOMODl/oefOLwD2iqdWPPt1fl8FrpZCC9AeXURVf1084l5nmt01SHV5Ts9UEYWfyDfU X-Received: by 2002:a05:6a00:1745:b0:6d8:e153:f884 with SMTP id j5-20020a056a00174500b006d8e153f884mr6589643pfc.30.1705329564448; Mon, 15 Jan 2024 06:39:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329564; cv=none; d=google.com; s=arc-20160816; b=BQNiCioPJACGATjfSGAyUNohtSzr0g2DXtmgaVyXLdtVxtA5hWZXmHxi55bFy+QTgk AqWuAiec1ycgu3cy8SPOCsYDlbH5ryuI92Gt/O+xzGXMe2ZZ+pPjCwYld+Apy4OwKe8d fnC23VcAvIiZ2IZ3pf3AoyTgVuSokwuiWAKEORGHc3ZsfRoEwt0CbSdSS2UnTNpAYBlL nHQv0JhrfSUqdQPFUZln0kJ5SNmZ3BsSShDJqN7e39RSyJCerW92O+z5alUmizu7gGcl ljzBBn0xTu7RP6GTnlZI5QDrXKbVLMSvCvM7IPWjNDO7Q8/Bf98HfkTo7cnQOKZ9RRGS 19Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=8tyreN3+pRRCj9fddFni7Yjd9oa5LLsmYxigcZC8rcg=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=NkhARobf+VbIT0UX3taVBeBGGjBKjXRbsltzrL8pBBcYANfXlKjvNUrfFcUz8Y99yI UaS3GwJ/qzLw41Z+ucFLJXpgCW4fHCz0eSDpeBmqyGONMj6JOvrmdj7YG36p3RSYauYz hiZo9pAitvy4PibkNoukk5fEqRDVPfDe90MwJC9S8JiAB+NfJBkMVn3sVkjmfQ1Fax8n UkhgKw+nY7G1r4y65E6gQy/B1VbeWqulq3bvQ2OFuKZIt4n/hV6BhO2tw/rblAonPlz2 NnUZKv3NuTJYXtPoFOedJy1YWlcDPR1ytXdUBKcL8MyezKaN/4zJp9NLVBg2ZJww/Zxg JIGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=NlXab2cz; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel+bounces-26100-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26100-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id u9-20020a056a00098900b006d96bb5db4bsi9034194pfg.168.2024.01.15.06.39.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:39:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26100-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=NlXab2cz; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel+bounces-26100-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26100-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3B8DD282845 for ; Mon, 15 Jan 2024 14:39:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DA8421802E; Mon, 15 Jan 2024 14:38:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="NlXab2cz"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="WukEzm0D" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A933F17743 for ; Mon, 15 Jan 2024 14:38:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8tyreN3+pRRCj9fddFni7Yjd9oa5LLsmYxigcZC8rcg=; b=NlXab2czETZp0xP+IzFHm6awdOka425Kzdzign5GPkoImvVgJWFPet0O1EW09TvG7M0L3m Y+dn4iDZS+MzOESzvSjVs7M4BV37zoPhtQ299jVWZRsGqLkETRvccoIw2UdMEDJfaZFOgH uaGiiYdetxgsHsjKab2KMYxJka1GNu0bxN/NhKYHll6hFm7OldcGToYy9qqCV4dySwibWU 5i/fqIwQZrKElFuVBQCNTxGL83OT1hcWi1xGQlI5YvU4lDXzREJ5e2qRnx5S3OdFbC6SWG 075hrzrLNhmiTPkEsV2Fok+cgT+UxrqcOozjePeIX5Xtghz9xkzAMgUO9HFhGQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8tyreN3+pRRCj9fddFni7Yjd9oa5LLsmYxigcZC8rcg=; b=WukEzm0DqpDbF2CDUItBPNFQBpgiXis0UKMUH26OhxA6gVei/m2F7wqKm8NtJaTG3Li47F oGqLobTfz0ErcmCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 04/20] timers: Optimization for timer_base_try_to_set_idle() Date: Mon, 15 Jan 2024 15:37:27 +0100 Message-Id: <20240115143743.27827-5-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167653553125860 X-GMAIL-MSGID: 1788167653553125860 When tick is stopped also the timer base is_idle flag is set. When reentering the timer_base_try_to_set_idle() with the tick stopped, there is no need to check whether the timer base needs to be set idle again. When a timer was enqueued in the meantime, this is already handled by the tick_nohz_next_event() call which was executed before tick_nohz_stop_tick(). Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 2 +- kernel/time/timer.c | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index c6223afc801f..27f1a2ae7f39 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - bool timer_idle; + bool timer_idle = ts->tick_stopped; u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3a668060692e..2f69a485a070 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1999,13 +1999,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases * @basej: base time jiffies * @basem: base time clock monotonic - * @idle: pointer to store the value of timer_base->is_idle + * @idle: pointer to store the value of timer_base->is_idle on return; + * *idle contains the information whether tick was already stopped * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer or + * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is + * returned as well. */ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) { + if (*idle) + return KTIME_MAX; + return __get_next_timer_interrupt(basej, basem, idle); } From patchwork Mon Jan 15 14:37:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188184 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743464dyc; Mon, 15 Jan 2024 06:40:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IEDaLD4AmBp2WgurIqN9E7iKQEOXGaAvSjnm2BVos5DCj5WhtIkrcidlxF/WVqOpGoHK7Cn X-Received: by 2002:a17:90a:cf85:b0:28c:c616:3dc8 with SMTP id i5-20020a17090acf8500b0028cc6163dc8mr3047521pju.79.1705329602965; Mon, 15 Jan 2024 06:40:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329602; cv=none; d=google.com; s=arc-20160816; b=ntVFbQfpQ1ITlJ9fQv/88o+/geBya9qbetr3dXXZbfkqSsu8vuk3v4K6RilujaV6lt sn6oYH0EnwblxWruANQB9MZdVCW15QnyosouGVAZ3HEGp6xnmJxtoSRBS3A12eZQE3SM FROHGLup0eMo0ei64kstX4vHtigeA90pnoWjzoBviTLBeMbKx7cWfAbnc+3oNJr+C+pV dsYJ2ONAROUk+JIAIIF3E9wIHrJFS0sYwsFCB49pSlPGhMMu3xgISbxg7fXiw5Q0ypwi SLKDkfrj6S9T56SvPs47rY2luz0Ior18aiMjVijoqCwVG0O3nThbA4Kru6990UEL5mRq gC9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=dd7ZSY9uCCvXsEjv5hsri8WxYcwJShixETTpt1BLsE4=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=U++ranzWhPC4rr7LbxzSMl+LTX9Nn0geegxsmM8O7vmmB/rRTywMGqHhf54qVDjnZW xMXXdRfsbXaaQ1YE18j9SuUlJkoPpGPIyWaQteAhCkHuUc5NTFjBxH56lF8RyrUguD+U 1RO6ceOyuNmFQFY01UIyx3Nt++tLQ4T/x4cbY1w5kcQnuMUYDz6hrOxZwybtbZDu6chA b8aKQoiDApKWx+luWrhu92lXOWOJ0RbhWPEJ1YbcjAvy5KFMi9HS73wk0ZrQvKGM+Kab 9aQFDqnKZw4Sg332EGhSHi3ZehUyp5nhvPSObyQR6s7x9eGkFHRWU6oDAh0/XOc7rkiH g/ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="f57/0sSB"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26102-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26102-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x21-20020a17090aca1500b0028d29091591si12140301pjt.144.2024.01.15.06.40.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:40:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26102-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="f57/0sSB"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26102-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26102-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id B8DCD2828EB for ; Mon, 15 Jan 2024 14:40:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9F5A018643; Mon, 15 Jan 2024 14:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="f57/0sSB"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="WqiDL90+" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0776F17999 for ; Mon, 15 Jan 2024 14:38:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dd7ZSY9uCCvXsEjv5hsri8WxYcwJShixETTpt1BLsE4=; b=f57/0sSBXZ1JRAX3RVEVr5HFQv6Lqb5w3CD5uk8FY+PR9dgPCaun9CyEOq7cdBwCYK3TwZ QV4OykKeDthyz2Q5gL0/pysVAVrZy/qsIrc4ePzSvIqEJRO3QTh+3TB4JIbsuPoKgb4SaI 98gmtJ/nGofxAzq1BWJ1HpmbxTdutIV7/1oo3Yis2h0Ht3WkFR8EVxHEyaXi7cfzn4hbyz dPubmZNV6uAofAmMhnWHw+CP8T7VzwJGSEh9KGL6SGLwpR5gNKzzLdiQYNhcEDRrZw+efJ pUBOOZ+rHMo0345vx2R+yOthEkzdKNUNqtr74EEpTZDQsNVy7ZcMvCVgxdCG9g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dd7ZSY9uCCvXsEjv5hsri8WxYcwJShixETTpt1BLsE4=; b=WqiDL90+9AC1nMXbkvsV88JCQ77hfxzaVccWYpl01ws4yhT+L5Gkum9jw4wwsNZ9ofshe6 ukC2UFG/zu2l6KAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 05/20] timers: Introduce add_timer() variants which modify timer flags Date: Mon, 15 Jan 2024 15:37:28 +0100 Message-Id: <20240115143743.27827-6-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167693346337186 X-GMAIL-MSGID: 1788167693346337186 timer flags A timer might be used as a pinned timer (using add_timer_on()) and later on as non-pinned timer using add_timer(). When the "NOHZ timer pull at expiry model" is in place, the TIMER_PINNED flag is required to be used whenever a timer needs to expire on a dedicated CPU. Otherwise the flag must not be set if expiration on a dedicated CPU is not required. add_timer_on()'s behavior will be changed during the preparation patches for the "NOHZ timer pull at expiry model" to unconditionally set TIMER_PINNED flag. To be able to clear/ set the flag when queueing a timer, two variants of add_timer() are introduced. This is a preparatory patch and has no functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Commit message reworded as suggested by bigeasy v9: Update documentation to match kernel-doc style (missing brackets after function names) New in v6 --- include/linux/timer.h | 2 ++ kernel/time/timer.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/timer.h b/include/linux/timer.h index 26a545bb0153..404bb31a95c7 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -179,6 +179,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires); #define NEXT_TIMER_MAX_DELTA ((1UL << 30) - 1) extern void add_timer(struct timer_list *timer); +extern void add_timer_local(struct timer_list *timer); +extern void add_timer_global(struct timer_list *timer); extern int try_to_del_timer_sync(struct timer_list *timer); extern int timer_delete_sync(struct timer_list *timer); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2f69a485a070..3cf016d6fa59 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1245,6 +1245,40 @@ void add_timer(struct timer_list *timer) } EXPORT_SYMBOL(add_timer); +/** + * add_timer_local() - Start a timer on the local CPU + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is set. + * + * See add_timer() for further details. + */ +void add_timer_local(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags |= TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_local); + +/** + * add_timer_global() - Start a timer without TIMER_PINNED flag set + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is unset. + * + * See add_timer() for further details. + */ +void add_timer_global(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags &= ~TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_global); + /** * add_timer_on - Start a timer on a particular CPU * @timer: The timer to be started From patchwork Mon Jan 15 14:37:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188183 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743392dyc; Mon, 15 Jan 2024 06:39:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IHcfxsdjSkXoVlPMbrs7W46IF2Hh0R7qtghEsgXykw8WUj3U1XEI/ZjBaVAGNdp0hAzuovt X-Received: by 2002:a17:902:c40a:b0:1d4:e0f:e4f3 with SMTP id k10-20020a170902c40a00b001d40e0fe4f3mr3143240plk.101.1705329595888; Mon, 15 Jan 2024 06:39:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329595; cv=none; d=google.com; s=arc-20160816; b=ohY3uoiK2RRDs+oPtUXhT3zUwWgO1c82O/3xGpcKOvdb0Da3XN9d2O1cXQ7j5WPOrM zqmPRvpDg6FaguIGfqPWZ9Nu/nDu1uSkr9LSyovl3cVH2sFzO19hkf7Hi6ZeB67OAGmD fY1p2BxwCLEDxAEV5Cr3kDn0NOWhPGM5cVv08AEa9z4Ev/NvPwI1yYQ2dAtN6jA/4A2l WfSAnvVyUaHZ85p8620yLuO1ZDEx3rbVbgYNeFct4h6E2WLXfU3SMwVWkSm70o/TFsBT 6ObQLE/RFncgLpH8DtBO/I+vyhUEPovVjNlXWNOPv0mbmdlPqeKp1EDxV+pLSnWU47lk vLbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=3N/e3ayce4s+S+psHat1NbmR2VoGUdUGd0vnbZups1E=; fh=6m6KAu2D1n2eK6jNJ9BIoRiulvR238oCr7uvY5/A720=; b=g4vHlwrLlM+VNzUCoqgqKGr6FYUENM4QAnQtApIt1YkuB8ZQC0y1k/vvy7Mvf71Hzu GLD0CcXnvDheu158P2SU3nUCDKUeHbl1nzlrmuzFr7H3UloSV11iqIX9c8LvFi+Fa8F/ lRdHmAs1Ryrc8DuyQp1ISh1kWrjmb/R1OzlKND3w1q1E/aWNDEdBA/L8TM65lDMj2hQb HKC2HdWIvZFZooS5VbIu2BrUDeGYQOfTXmsZ2NNU/SNF4MaTrYX6RLdqhSRENpBZwhWE SACdVj+XVlzisCpCfi3dyiFI8xPrv+RPqL6cBIZhY1L7lpgXqIMPPuxBNHdid2mDg4Q6 jI/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=RyWK6cnq; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=nI9H8aRP; spf=pass (google.com: domain of linux-kernel+bounces-26101-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26101-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id iy7-20020a170903130700b001cfe02fa89esi8981930plb.117.2024.01.15.06.39.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:39:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26101-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=RyWK6cnq; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=nI9H8aRP; spf=pass (google.com: domain of linux-kernel+bounces-26101-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26101-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id A810A28284A for ; Mon, 15 Jan 2024 14:39:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5F3651862B; Mon, 15 Jan 2024 14:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="RyWK6cnq"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="nI9H8aRP" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08E211799A for ; Mon, 15 Jan 2024 14:38:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3N/e3ayce4s+S+psHat1NbmR2VoGUdUGd0vnbZups1E=; b=RyWK6cnqw0ZSnAWy/fuDH3dP0nS91x8d3ZEE8sFVaN+Vwyu+FNf6Eo8e15EGEiU0TVhbBP CEiUEWxOavweTRQ5m0Az1upcpffWC2rkjj1sjatSDdEBJ3v2H/uwX97TZ+kDziSmES9dPj oQ0z9ycusV9RXflK9FcIJEv3PAO1wO405l7j3tR3MOq6Q1A8GZfSywxZJ54WOnXBdaxU5U jifuboiqZw+UW2BdbBgpEV2Wa2WrBuq57KGcfP53oo5YC9B0bViLiLuuhE3rFoDv/Ln+gW t7odEXusxS4BmTv+2pDa7dzXH8jx7ahZCyNgV5jvZgJmFqO1ROABdBOk49NG+Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3N/e3ayce4s+S+psHat1NbmR2VoGUdUGd0vnbZups1E=; b=nI9H8aRP8bIz2zRhY61S/vNnS4ySyBwwN2VLZ3UTM6sC9x7IvA1EKT80kGdS6DDSY3JvvZ JIxNDTZi3oKmaoAQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Tejun Heo , Lai Jiangshan Subject: [PATCH v10 06/20] workqueue: Use global variant for add_timer() Date: Mon, 15 Jan 2024 15:37:29 +0100 Message-Id: <20240115143743.27827-7-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167686314042851 X-GMAIL-MSGID: 1788167686314042851 The implementation of the NOHZ pull at expiry model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be dropped. This is required for call sites which use the timer alternately as pinned and not pinned timer like workqueues do. Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker Acked-by: Tejun Heo Cc: Tejun Heo Cc: Lai Jiangshan --- v6: - New patch: As v6 provides unconditially setting TIMER_PINNED flag in add_timer_on() workqueue requires new add_timer_global() variant. --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 2989b57e154a..31a6b0831318 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1955,7 +1955,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, if (unlikely(cpu != WORK_CPU_UNBOUND)) add_timer_on(timer, cpu); else - add_timer(timer); + add_timer_global(timer); } /** From patchwork Mon Jan 15 14:37:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188185 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743533dyc; Mon, 15 Jan 2024 06:40:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IFwSHOUAi+cIMejDiQtgWQoGmBtoxdbR2MkFThQec2weKc5ZIW/RZdoV9gKtSv+DV+9p1aO X-Received: by 2002:a05:620a:384:b0:783:4f59:666f with SMTP id q4-20020a05620a038400b007834f59666fmr4171119qkm.14.1705329611318; Mon, 15 Jan 2024 06:40:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329611; cv=none; d=google.com; s=arc-20160816; b=eurVeXE376JSuyDQ5qtOjNyrOsgMyq2p0rBz42McI5P2YAPeMFl2HQVznkvr43m7Ej lLQpqvoZdfyAZW9a/IZmsegLD/9zjUZFtIs20DEX3tXI/iJRuUa7QGqoxd6LrYYDlNLr TkU5VhouJV6TWmrubGn3NDWCiBN4kQGxFMypW0IUKkUpDanxmRZ6pWAuPvHxAPxEP1NK IVNHGqS6pY1Bz9glpbR4VmBf3Ac/1extK91xtog0AS3UTgMUAxail2nD/NzvqtWwrCHE j+ys79EDtjAdSRyxHZXBShu/J5ZjmX6gMvk3bw5rcfIKBCyFRsvbbAha0CtzreD9TuBc GZOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=ID/05caAQ7viG6IvKvJwH4SWuLM7iMQOtASJkbsjYUE=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=pQB/FipujNXiN+zRecJCq+z7DncCCVjdoGyLGgvSs2MhetpRpL2lLSp8P1EQewugYA IY0QwQbhO//HPRNTg1cd4NEJ3KuLCgvQEhc9wFDwLruCKmSsW+wpPML/f+2G7vYeWgKq yAB/Nqmo581dAeW52rGcVMXkxdrru4OGyGy15bAoxAascm1/ap5gvRf3jp4cnYAi/n6D s3PFxcqYzCMZon6jkyum1PBuZvvKVN0DxhBu0qe8eput5Aea/q7xGNRDE4GYetYt9aK8 jVxOBuVSC7P5SeWJuWIuGYFB5usg7Dw8296S7RybU8SjA6Ma7B0pAQJh9pBnrcM2+8nQ vxZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Kd6kcmWr; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=NJleEKH8; spf=pass (google.com: domain of linux-kernel+bounces-26103-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26103-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id n22-20020ae9c316000000b007834a7b552fsi5995477qkg.289.2024.01.15.06.40.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:40:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26103-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Kd6kcmWr; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=NJleEKH8; spf=pass (google.com: domain of linux-kernel+bounces-26103-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26103-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 1EB701C21B17 for ; Mon, 15 Jan 2024 14:40:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id ED7FE18AE1; Mon, 15 Jan 2024 14:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Kd6kcmWr"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="NJleEKH8" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68F96179AD for ; Mon, 15 Jan 2024 14:38:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ID/05caAQ7viG6IvKvJwH4SWuLM7iMQOtASJkbsjYUE=; b=Kd6kcmWrMdGquSj5bS/s2WvOQTwxrPJ29PzQN8y3YkLHLxB8kkZYrV0QHBCCkD3W8/SUJ+ CvCmeqwAR8j5N5psDYLVrIspyBcre5NiN1VDnxxPczpjh274XKrO6Putyl9p8bpYevuSyi MhKXjwjPJmUFV9OfUePjAjZI3eFB/D3ZzeVzlM5uWusnWfYMSzt20rnurcXZiCM5mpf9Ff gHquLdZkSPZL2tTWjih6hj+Uw2d2Nw/Dq20Hzmzq20zei053PApLVeVDdYJFOAcQ4Oi6+M lw2UuXQuWUolN1qXre4fomZA5Mi57m9FRoCFSTOq3i/B/3ybnN7ykOuwcLCYnA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ID/05caAQ7viG6IvKvJwH4SWuLM7iMQOtASJkbsjYUE=; b=NJleEKH8Ghy/Gbl6g2GgfGmK8C28DDGE4e44xPDO+2QBqPAAAdluGRrg4G0VbHCGvYR03Y Y2/ELZ67he0XRhAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 07/20] timers: add_timer_on(): Make sure TIMER_PINNED flag is set Date: Mon, 15 Jan 2024 15:37:30 +0100 Message-Id: <20240115143743.27827-8-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167702834782174 X-GMAIL-MSGID: 1788167702834782174 set When adding a timer to the timer wheel using add_timer_on(), it is an implicitly pinned timer. With the timer pull at expiry time model in place, TIMER_PINNED flag is required to make sure timers end up in proper base. Add TIMER_PINNED flag unconditionally when add_timer_on() is executed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3cf016d6fa59..fc4c406c9ec7 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1284,7 +1284,10 @@ EXPORT_SYMBOL(add_timer_global); * @timer: The timer to be started * @cpu: The CPU to start it on * - * Same as add_timer() except that it starts the timer on the given CPU. + * Same as add_timer() except that it starts the timer on the given CPU and + * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in + * the next round, add_timer_global() should be used instead as it unsets + * the TIMER_PINNED flag. * * See add_timer() for further details. */ @@ -1298,6 +1301,9 @@ void add_timer_on(struct timer_list *timer, int cpu) if (WARN_ON_ONCE(timer_pending(timer))) return; + /* Make sure timer flags have TIMER_PINNED flag set */ + timer->flags |= TIMER_PINNED; + new_base = get_timer_cpu_base(timer->flags, cpu); /* From patchwork Mon Jan 15 14:37:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188186 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1743548dyc; Mon, 15 Jan 2024 06:40:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IH97oNJTecAHjz8Jdb718GkSb+RexTYSWHKxaR/6OklcK/qqMSOav/dGBtN+jiyBPLC+YQe X-Received: by 2002:a05:6214:1243:b0:67a:a721:b1b5 with SMTP id r3-20020a056214124300b0067aa721b1b5mr8103130qvv.112.1705329612970; Mon, 15 Jan 2024 06:40:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329612; cv=none; d=google.com; s=arc-20160816; b=ThxT1nlebP8zZCXgj97ovhX2QzbagNI+ei7ga4vaKwsXFQTBp7TGI+DjHUne3T+p// iFiu9dhQpshCc52oateLZqsSxbwKBIIwo7mbmmGdfpOx9FH/vfwma0ocBogmRku14Eay pSphumpOFmNBQVGFGejj/fXn+m7fyb/MEWfI4fChHXYzZvTybX4wLmlHHYtA1YcDL37u 66LW4UsRrwtLvTynQGzFYRhshwuxSokiTtVROLbBMhSldRILYo4cwTqcgpmBo8xihbpj 55Xf2qKLjGSFARAfJCWAkjxvinV3RrCow9S8flL9hoDcgglUAIznTPgVY/H/C+uMAruh FWtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=QRl9bsgrm2/Z3NpMddYIe4IdlsRsgfApHihxcCMeQvaDONvqUzXQgRGzAJrMDVAS5r 3YojjwJyA0eyB7ywD7kY7WsQaiixbnIvIvM6oTJDx8b7o6QTLzr31LAZBxmQWXTkn/Hz mF4+wrdEB32UYed/FVOVBnHcr3jt2oS5p8ztR0zN/owTaEhlr0EZIgSNeA7Ylo4UphXy Nc6zc60ohWqJRw4o5mtZOY84ndOQPJzW3vRejqv/Zh0eArgS2dodPNR1jZ9gtjiTOPHs PJvaWqdw9FvCVwO2kes6cGX4Qh88ShIexoFGVODHZ5VCIc9MWAFdvMPXiqhJ2z5rRHO4 3iAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=xLLcLooz; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26104-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26104-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id j10-20020a0c9cca000000b0067f65b3857bsi7939259qvf.15.2024.01.15.06.40.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:40:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26104-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=xLLcLooz; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26104-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26104-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id BDCE11C2088C for ; Mon, 15 Jan 2024 14:40:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 082AB18AED; Mon, 15 Jan 2024 14:38:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="xLLcLooz"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="p1SCOwxx" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C678179BC for ; Mon, 15 Jan 2024 14:38:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; b=xLLcLoozEBINigPBjKzXzLZqUlO4iXEH+bnYjLA8s9NsrzOM8ZdRuhFAooFDJdtoI7/Xzd aMwRjpebAWtxtl0NSl3/wnCFU5Ygagq0zg7hqzyyLeWBLhxnc7oI8qMYJECx97wGadrptu 2No2XTcbRpniHVJ/O7Y3cWbkSpPu94dyWNhFVcuKzWxOLIKy1iC3GjlBq05Gpm0YtRe+Sc PqmExnHcc5n0fOTTQfGChQssf8GUqb7mMsQLGnHw+MkWLLxGOCZ0Mi3dgw+PO3q3ix/y29 gP6fOiahz/2/xp5AP4hSrJTKmKsvnOH4KlAkHk01BLejV5ImgxJd8CUaozruLw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; b=p1SCOwxxFeN90cEaCj5jTm16QNT7JNWHQUUHeBYm8MRqtexuXM2Hn/RJJ7TY6rd4sgZiTK WJpMEkxX9GZiTdAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 08/20] timers: Ease code in run_local_timers() Date: Mon, 15 Jan 2024 15:37:31 +0100 Message-Id: <20240115143743.27827-9-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167704159799925 X-GMAIL-MSGID: 1788167704159799925 The logic for raising a softirq the way it is implemented right now, is readable for two timer bases. When increasing numbers of timer bases, code gets harder to read. With the introduction of the timer migration hierarchy, there will be three timer bases. Therefore ease the code. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v5: New patch to decrease patch size of follow up patches --- kernel/time/timer.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index fc4c406c9ec7..793848167852 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2135,16 +2135,14 @@ static void run_local_timers(void) struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Mon Jan 15 14:37:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188188 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744210dyc; Mon, 15 Jan 2024 06:41:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IHhcj88p4ttADDWUGux348qEaA+l8n+U5dGWjD1UGtmkEG00JhwfIrw8bwokHlTy2/jF5Ux X-Received: by 2002:a05:6214:1c4a:b0:681:117d:827a with SMTP id if10-20020a0562141c4a00b00681117d827amr8522056qvb.2.1705329686876; Mon, 15 Jan 2024 06:41:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329686; cv=none; d=google.com; s=arc-20160816; b=0alAkGDZ74258597Fy+yHPFqHWvJgoBQIBqNI8PziB+6+IUj97+6V0r/ptSts1J2bG h18EsX70IjSK9fJQt4UNeAlrZ/u0QkknSPSoOnrsqTz/GcaLbWyvGaHm7dhwgRwmczmd +4m/pmKDGeY/ZRcQCg8jVSUU3iBmNHo+5VXOC4mXrA+cT/PRgWbvn4IjTwmAC1ZxW+EO 1lB6yL9TgWk8r9PyoCrMnfTtRWc4ZQEK56tUlsj6moAf8G/G0GMq0jVqWZdx5ICTtj0m WrlMQfSRiFcmWnhiH7ZuTm9TBgKfpOHljoP/SaEa+uv5AQ2jSbB9DIWcMm7hpQHLPbbf bGbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=dsz69NezZ9nKAiTPhrMIzy1xjK2BcbrJi4MD0PI0gp5hx2PpET2RFh9+uwBCliF5ZT e/qxoFael9tTcV4jDGSfk8m4UzNvRNsSvrqb28EdShjBc4J6KFHpcikFemMMQCfWrDCJ wa88aBuoaAWXqToqU4/HIjgsEd4phWfTf3Iy0TfOuIw2qGX54yIFxvk3l3ul9wbjFRQf ftSJYpxNct0oIaug3G3lVpmW/ubGdMp2CN79zn5y1XfTvncpUDC/sLzPQJa5eatgNtAf Dll8C2UejnbfCm0ueKnfbzgWYv4Fqp2Sh3CMocU2+reKnSgYQhnZHshvBqVRLuOoVjDd UTeg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=J4BV4qZI; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26105-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26105-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id n18-20020a0cdc92000000b0067fe06ad843si8000354qvk.518.2024.01.15.06.41.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:41:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26105-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=J4BV4qZI; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26105-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26105-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 9BC901C21B07 for ; Mon, 15 Jan 2024 14:41:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E91AC19451; Mon, 15 Jan 2024 14:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="J4BV4qZI"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="W0G/z6ks" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E617717BCF for ; Mon, 15 Jan 2024 14:38:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; b=J4BV4qZIGNkj8PoY+9gJVrsUyn59/ZbtjLKONPlhyjLHe8PiQUhMowDH7fnDdWUIMfhSI0 +iriNvefclVeWwlLF/OIIvMtT14VTKzTt6YsMdRJBfHykZSTJTS2JVJzlYjcoypCsJC80r P0VKfR7AXcY3x21+gq9fsY7hElMk/RXIpy2f+36oblwL8T7jJhf+F5QVwHFI3pXrrqTbT/ UG0ijLd2rIid5dIxeRbXLNxG2PIp2qHLxlRHgwBVrYtlQYWwAj0SMJtWCl1PKvHiOfNtQv /nR5G4zDH4A9W25zvOwcowxY2kXeLphXUHMs0/Hwmqm4yYzoxGtobICDAj3dnQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; b=W0G/z6kszirkGE7Zrej8QBE5kexwzHFENdBlYZGz20lj0wLnMxE2xFum0OHLRKHAcyOsiL r6D+T569b9wZ4VCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v10 10/20] timers: Keep the pinned timers separate from the others Date: Mon, 15 Jan 2024 15:37:33 +0100 Message-Id: <20240115143743.27827-11-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167781780010288 X-GMAIL-MSGID: 1788167781780010288 Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: - Simplify local_first check in __get_next_timer_interrupt() due to updated next_expiry values of empty timer bases v9: - Update was required (change of preceding patches) v6: - Drop set TIMER_PINNED flag in add_timer_on() and drop related warning. add_timer_on() fix is splitted into a separate patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker v5: - Add WARN_ONCE() in add_timer_on() - Decrease patch size by splitting into three patches (this patch and the two before) v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 85 +++++++++++++++++++++++++++++---------------- 1 file changed, 56 insertions(+), 29 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4d6cf49a2fd1..5ca831444954 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1961,6 +1973,9 @@ static unsigned long next_timer_interrupt(struct timer_base *base, * Move next_expiry for the empty base into the future to prevent an * unnecessary raise of the timer softirq when the next_expiry value * will be reached even if there is no timer pending. + * + * This update is also required to make timer_base::next_expiry values + * easy comparable to find out which base holds the first pending timer. */ if (!base->timers_pending) base->next_expiry = basej + NEXT_TIMER_MAX_DELTA; @@ -1971,9 +1986,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt, nextevt_local, nextevt_global; + struct timer_base *base_local, *base_global; u64 expires = KTIME_MAX; - unsigned long nextevt; + bool local_first; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1985,10 +2001,20 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, return expires; } - raw_spin_lock(&base->lock); - nextevt = next_timer_interrupt(base, basej); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - if (base->timers_pending) { + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt_local = next_timer_interrupt(base_local, basej); + nextevt_global = next_timer_interrupt(base_global, basej); + + local_first = time_before_eq(nextevt_local, nextevt_global); + + nextevt = local_first ? nextevt_local : nextevt_global; + + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; @@ -1999,31 +2025,31 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * We have a fresh next event. Check whether we can forward the * base. */ - __forward_timer_base(base, basej); + __forward_timer_base(base_local, basej); + __forward_timer_base(base_global, basej); /* * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ if (idle) { /* - * Base is idle if the next event is more than a tick away. + * Bases are idle if the next event is more than a tick away. * * If the base is marked idle then any timer add operation must * forward the base clk itself to keep granularity small. This - * idle logic is only maintained for the BASE_STD base, - * deferrable timers may still see large granularity skew (by - * design). + * idle logic is only maintained for the BASE_LOCAL and + * BASE_GLOBAL base, deferrable timers may still see large + * granularity skew (by design). */ - if (!base->is_idle) { - if (time_after(nextevt, basej + 1)) { - base->is_idle = true; - trace_timer_base_idle(true, base->cpu); - } + if (!base_local->is_idle && time_after(nextevt, basej + 1)) { + base_local->is_idle = base_global->is_idle = true; + trace_timer_base_idle(true, base_local->cpu); } - *idle = base->is_idle; + *idle = base_local->is_idle; } - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return cmp_next_hrtimer_event(basem, expires); } @@ -2067,15 +2093,14 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(false, smp_processor_id()); } #endif @@ -2126,11 +2151,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -2138,7 +2165,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); From patchwork Mon Jan 15 14:37:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744116dyc; Mon, 15 Jan 2024 06:41:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IEgJQBkTjpuKuvU5S7Fnco9J3NB1MIm3hVJSIZ0mZbGhYcDvEXrFPrtMBd3pOrqGB7U8b0D X-Received: by 2002:aa7:cf04:0:b0:554:3ba8:f804 with SMTP id a4-20020aa7cf04000000b005543ba8f804mr2427622edy.28.1705329675428; Mon, 15 Jan 2024 06:41:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329675; cv=none; d=google.com; s=arc-20160816; b=RVKeHh9q5gm44+ljjybMuHT7XCLd2qE+m6UWRHrnwiC5LLsnO/Sr9UiJ0XW55F/9Ym ZztAbPC9TsEZqmC148Fsbaq7AiuH0kAsBGDIIv1ULRgCTWUyA5CWAaL39LIHbv2yVzGj WyUWk1QHoDBWiCgVLvbHDzNRZMyDMdNloFgtUGRHdZkDK6ron7N3nzHynCBsCmhL885T IAKfF0+CzU0YdCTXQZE1PL+OdixOFvi5sjI/e+LtJ1slLJZdzD6UD9Qv7qusgsYRjzSO rmo8TIxj59jGAxcfo6Ze/gc5BVkm2bUrqG2pZk+V4GzOhcKIKZGWZkBjlW+w0WPFx/by Q+3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=Fr+0oSb+AndOblwVvQzleNBKd5oIVJXXyPBNDPB6wYY=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=s4p4w2zznOee0w9qV30JUL6nspIDqlLnuRIRIIyWK/7ifUOOsc51mxKsra4vmsAQT2 bOJO7lDRRCydwF+jTbXi2SsVsqnJXRoabeDmf5N/Q6Dr5bfxWurHNmNJt/IuSy0ULk11 VU8+ilmxYkLyhvKYS9Z4CJjlqJSpP0YGIe2Z7dtzAPCkyon9XZmoR/09BVjS+KSjVOUd /tAhaezsugyawwAdLJlJSBRxiZxmwmrjZ72EiaAp5ZzYlxZXo00uvJtnUhryBhzCAEyH 6wJlF9o1Rt+0bhme0lIJ3IBiyl356KJxQ07iUHzNqrBGlQdDNVUvkXCppUsuFDCns5e9 MUDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=YbSTa+5p; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26107-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26107-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id fj15-20020a0564022b8f00b005581ef9ccdesi4079914edb.318.2024.01.15.06.41.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:41:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26107-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=YbSTa+5p; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26107-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26107-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DBFC41F2289C for ; Mon, 15 Jan 2024 14:41:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9A7B818ED4; Mon, 15 Jan 2024 14:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YbSTa+5p"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tgJIiLML" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9314A17C7C for ; Mon, 15 Jan 2024 14:38:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fr+0oSb+AndOblwVvQzleNBKd5oIVJXXyPBNDPB6wYY=; b=YbSTa+5pAcCvS4Wu/iTQy+7Efu4QkhWpb+LdmNJp3T/95XWrgehdESVtjUbLKPcQ8dqMlU xTtHaf5bF+8GKvsYaB8Zc4mQY3fFJllBcgiuJwZJVmpMbgjbBL6lGTx55h54r4OezUZbhJ IRfBUP3Fa9GZ2DU7soBptOC0lTWKHmFgDVNIl+gktGqF+yxvkowCu4DaEYTF5aWLhwyADV rhtl+MyMqATNTeqPe6pdPEZnmoqfzEb9MOzS/vp1sBQraZjg5f3rfPf69ytEIbn8MvtVPa JfKEhZ75t3n/omQyjat4E3DSeqNK2T2SXL16Hd0hAvPIDhx+wuEz99NmmAcH4Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fr+0oSb+AndOblwVvQzleNBKd5oIVJXXyPBNDPB6wYY=; b=tgJIiLMLs5RmyS9MgTYmslgvJZvQpm6vHXIxjarUqg90tW0RG5XhKAJ8G2UwoJYq3URsi7 On9mOU4z1EnxBhAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v10 11/20] timers: Retrieve next expiry of pinned/non-pinned timers separately Date: Mon, 15 Jan 2024 15:37:34 +0100 Message-Id: <20240115143743.27827-12-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167769884791416 X-GMAIL-MSGID: 1788167769884791416 timers separately For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have separate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Fix no functional change message v9: Update was required (change of preceding patches) --- kernel/time/timer.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 5ca831444954..f119b44e44e0 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,6 +221,11 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -1986,10 +1991,11 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; - u64 expires = KTIME_MAX; bool local_first; + u64 expires; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1998,7 +2004,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, if (cpu_is_offline(smp_processor_id())) { if (idle) *idle = true; - return expires; + return tevt.local; } base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); @@ -2014,13 +2020,32 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, nextevt = local_first ? nextevt_local : nextevt_global; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the @nextevt is at max. one tick away, use @nextevt and store + * it in the local expiry value. The next global event is irrelevant in + * this case and can be left as KTIME_MAX. + */ + if (time_before_eq(nextevt, basej + 1)) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; + goto forward; } + /* + * Update tevt.* values: + * + * If the local queue expires first, then the global event can be + * ignored. If the global queue is empty, nothing to do either. + */ + if (!local_first && base_global->timers_pending) + tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + +forward: /* * We have a fresh next event. Check whether we can forward the * base. @@ -2051,6 +2076,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); + expires = min_t(u64, tevt.local, tevt.global); + return cmp_next_hrtimer_event(basem, expires); } From patchwork Mon Jan 15 14:37:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188189 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744319dyc; Mon, 15 Jan 2024 06:41:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IGaEaMM68cFomwSSrlzae6/Q0/OrwDI4O46vOGAvHHlqRt7hob0kZxOARVK8TxtXRN1BBrp X-Received: by 2002:a17:906:88a:b0:a28:b71d:6801 with SMTP id n10-20020a170906088a00b00a28b71d6801mr2604580eje.149.1705329704963; Mon, 15 Jan 2024 06:41:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329704; cv=none; d=google.com; s=arc-20160816; b=o/IuYCcESOy81xKKrRZWjlnjlgWJDFpku+8kvPUzZjzFJqM2FDHJDlVDrg+2tf5dW2 FwDRXjCHyI2qS1hnkF86kf5ifGQFcF+o+Op+I7qzLwm5Gu9mZrTaqGT8zQdmxYj1Wvt7 mevZkzRu+LT4Kq3g7OkfYdJGJ+1Qh1wW6WayAFBMzZeqAuyY4d+b7jeNImXeIfqrcpZ1 lP2/jzQ621m/3I7T5JWln+4lplq8Y2xqJLPYnewMQbAKLLSNjRUJnj/dxbwtcmODwrMG REVWazcpuGw0LAKTmEGCTOW0s7qRNBEloX5IocVkIS+j3qv2CnW9jz4C++5fyXkSfhIP CI5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=L0MYhdyqdCnNF2sikCCC+B6J3YzfhlbKGb8NzRYHY1Q=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=IOHjSyOwltD0udpe+roSlOtBz+6kyMSOhxWQP44dd9WEgmdsL59anWbzjU/jrjxU9n ufSSxqtu1BSK5/JQXK48LTKxwr3orXZYCMtFR9PSSSePNHH2vUU3+15fKVofk2NaOkZn nM0syucbbRy928EWqyeL37yVq6Uj247qJkCevd/nza4G4gB+TH2uoLxUuuLPWSdHpO7h IX67aqkmabXp3PXUx1yug9Obksv7rPU3Bb0c7hbvHBypMi1wiCGi4t2d436nJ3m6tf6h j8sVPHpMBsKpyYqWloyG9lLWy2qm4ClOpX3sW8o6WhqpEOSKmD2sv5M+Spb6MnyFfoCF p3Pw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=XoTSyhcG; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=g08rqGsE; spf=pass (google.com: domain of linux-kernel+bounces-26108-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26108-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id jx24-20020a170906ca5800b00a2cfe9dd275si2444276ejb.440.2024.01.15.06.41.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:41:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26108-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=XoTSyhcG; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=g08rqGsE; spf=pass (google.com: domain of linux-kernel+bounces-26108-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26108-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 9084E1F2292A for ; Mon, 15 Jan 2024 14:41:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8252C1947F; Mon, 15 Jan 2024 14:38:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="XoTSyhcG"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="g08rqGsE" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B36C1805E for ; Mon, 15 Jan 2024 14:38:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L0MYhdyqdCnNF2sikCCC+B6J3YzfhlbKGb8NzRYHY1Q=; b=XoTSyhcGXL3jewDT8HeyR35QvUNIwwU5TQmxnt8IqAsASAAa7s6QlLsGaP9rjri5CVBtkh bt2NmkVvbHCYMfA/RffmV+1YNGNRkBrsVc8Pm1XOJ4FM119w8CHQ3bwbGy6alkxKGxwEw8 9MlVF1ZGDuTkS2bItWoFLKsJR+dy+cww2ThJvEhGlXJO1u0C1qubSvBFZrXxEONbwbtvBR 8ywQDFqZhlpTf/T0yPXCkLgffWz0kGEOVoQMQtSgNiYlbbRJULMChJ1BZDTWgEGiVSx3U9 NnCB2ToDM1XZaJR/KjNCRSuto1povy5FzB1hQlCt9pMumKbyKmYBkjYvH+AJ0Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L0MYhdyqdCnNF2sikCCC+B6J3YzfhlbKGb8NzRYHY1Q=; b=g08rqGsEbLNBGwipTVvkMSon7N5/imR58bml+pm9R18WMYCyhxI3JHagKL/BjtMNScic1Z jMV806HMjOLAOZDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 12/20] timers: Split out "get next timer interrupt" functionality Date: Mon, 15 Jan 2024 15:37:35 +0100 Message-Id: <20240115143743.27827-13-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167800447462430 X-GMAIL-MSGID: 1788167800447462430 functionality The functionality for getting the next timer interrupt in get_next_timer_interrupt() is split into a separate function fetch_next_timer_interrupt() to be usable by other call sites. This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Update was required (change of preceding patches) v9: Update was required (change of preceding patches) v6: s/splitted/split v5: Update commit message v4: Fix typo in comment --- kernel/time/timer.c | 64 +++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index f119b44e44e0..9fa759dd80f5 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1988,30 +1988,13 @@ static unsigned long next_timer_interrupt(struct timer_base *base, return base->next_expiry; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, - bool *idle) +static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_base *base_local, + struct timer_base *base_global, + struct timer_events *tevt) { - struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; - struct timer_base *base_local, *base_global; bool local_first; - u64 expires; - - /* - * Pretend that there is no timer pending if the cpu is offline. - * Possible pending timers will be migrated later to an active cpu. - */ - if (cpu_is_offline(smp_processor_id())) { - if (idle) - *idle = true; - return tevt.local; - } - - base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - - raw_spin_lock(&base_local->lock); - raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); nextevt_local = next_timer_interrupt(base_local, basej); nextevt_global = next_timer_interrupt(base_global, basej); @@ -2029,8 +2012,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; - goto forward; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + return nextevt; } /* @@ -2040,12 +2023,41 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * ignored. If the global queue is empty, nothing to do either. */ if (!local_first && base_global->timers_pending) - tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; if (base_local->timers_pending) - tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + return nextevt; +} + +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) +{ + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; + struct timer_base *base_local, *base_global; + unsigned long nextevt; + u64 expires; + + /* + * Pretend that there is no timer pending if the cpu is offline. + * Possible pending timers will be migrated later to an active cpu. + */ + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; + return tevt.local; + } + + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt = fetch_next_timer_interrupt(basej, basem, base_local, + base_global, &tevt); -forward: /* * We have a fresh next event. Check whether we can forward the * base. From patchwork Mon Jan 15 14:37:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188191 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744483dyc; Mon, 15 Jan 2024 06:42:01 -0800 (PST) X-Google-Smtp-Source: AGHT+IE++ZfeRb7QamxWj7NRZY4ac4zcusI7EbBOWuLyvun1w+Liq2uebytAfoBdsWlAAmHeAZJj X-Received: by 2002:a05:620a:a0c:b0:783:6460:b851 with SMTP id i12-20020a05620a0a0c00b007836460b851mr621661qka.9.1705329721358; Mon, 15 Jan 2024 06:42:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329721; cv=none; d=google.com; s=arc-20160816; b=FWwtqar1/9kTjwWS1OyThs6sHQdbPXAtQ+Yzesa2NjuSPXC/b788IcTPqOBlYiEceC B0304KQhgn/yFJaA2asJHLhRim6mhA39i9iUpphERS/Z9O0a9ZN2NRgtx/6XuI0Rdswl KLw9fzfw5m8rVBNasPyQvC/Ggd543P81qegkyC6jtXfJumLU/aJi+HR77VbS0k3yient O9b9tHdSLRWn6GRg0fxsfzx5Hfo09eW753XSN44A9ooX0bPuWYOJerpugtvA4//B+kDm /I4uV1WoMzAmqVpHihoIKtZFN+YJe3MKp+bSxuqrL0MXRNuhqCQVuN/ILp5km+lX0S0q A7Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=ia4sddnbK0Qx+BWCOb+xOAdGjZ34vg4G3iXN8bRXi4s=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=e/GB/hlctdaVbauQ8tXI7fEd0yybFKdG9O9kyG2A2s3tUyaSruhVTOPSGYNZ7PZ7co ZbSPk1fz+zUxBzyJuZ2FI0gEIQnsAG0z6Ss5NbmJ3JuuLBbSN32Mpoi1iiaSftxY+j7Z Q4cqJtehZwc/u8azZiA8gYYF0Li+luhWSlyvbLU8j/h1EnR9Reg0QSF3s2kks+xldL9X IxTanPSHJZhbHpxGwKG4O+H/64qYhmnbOvWIjy44jgrLZVDpeTsFf2JUwU8BzO7Geb/S iihp2srCKFGD7Cn1X1HYoHTmiZKW0jS5RvwfeLpqAiF9FxVrFi1TSPBB/rcUllEnmDuh AfeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=b8GGphrl; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=j+t04NEL; spf=pass (google.com: domain of linux-kernel+bounces-26109-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26109-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id dc35-20020a05620a522300b0078160dc34cfsi7896082qkb.20.2024.01.15.06.42.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26109-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=b8GGphrl; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=j+t04NEL; spf=pass (google.com: domain of linux-kernel+bounces-26109-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26109-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 1B5121C219C8 for ; Mon, 15 Jan 2024 14:42:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2D9C3EEB4; Mon, 15 Jan 2024 14:38:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="b8GGphrl"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="j+t04NEL" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF23D182C3 for ; Mon, 15 Jan 2024 14:38:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ia4sddnbK0Qx+BWCOb+xOAdGjZ34vg4G3iXN8bRXi4s=; b=b8GGphrlLbDvJc/pDdyXU2kGAPnBE+bfQAiN++hh9wO8KyqhoTl94U9CrJSuGmSpPDyFZl cLIe0e0k4amIcy5S/NJU1uzMOZvDYVocGPAJ0Mejw3owPepGbbfbh1p0KI7My6IJG2A6a6 oHwdiNYrIW6rAmolipJKKpjx/cBeYpKIOFLnBZ8ojnpA89TnqzcOnv14q89e1Y41EAaG9E DU8pnAwafNmnZIqkH2lJ8HS5k3yqCimeJGbB54bllC7IIEDGpfk+jSOL7DyqK/ZUlzMLVr vNCzqb5rmrN2pLujw2Tik888Lvz7W6SXTgr9qm6hOEl51qitzauy9SiGlvVC9A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ia4sddnbK0Qx+BWCOb+xOAdGjZ34vg4G3iXN8bRXi4s=; b=j+t04NELCR9FOn+3Ltaz1pEStFnL5V8hESrqaOUd7BCwDRzN1BJgcvFY9O/2TamDwtwKWQ 5IJD6N6eDzQki4Bw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 13/20] timers: Add get next timer interrupt functionality for remote CPUs Date: Mon, 15 Jan 2024 15:37:36 +0100 Message-Id: <20240115143743.27827-14-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167817466880382 X-GMAIL-MSGID: 1788167817466880382 remote CPUs To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Locking of the timer bases and getting the information for the next timer interrupt functionality is split into separate functions. This is required to be compliant with lock ordering when the new model is in place. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: - sparse annotations for locks v8: - Update comment v7: - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section - change lock, fetch functions to be unconditional - split out unlock function into a separate function v6: - introduce timer_lock_remote_bases() to fix race --- kernel/time/tick-internal.h | 10 +++++ kernel/time/timer.c | 80 ++++++++++++++++++++++++++++++++++--- 2 files changed, 85 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 47df30b871e4..8b0c28edbd09 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -154,6 +159,11 @@ extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; +extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); +extern void timer_lock_remote_bases(unsigned int cpu); +extern void timer_unlock_remote_bases(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 9fa759dd80f5..3e2adfc15f3a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,11 +221,6 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); -struct timer_events { - u64 local; - u64 global; -}; - #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -2031,6 +2026,81 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, return nextevt; } +# ifdef CONFIG_SMP +/** + * fetch_next_timer_interrupt_remote() - Store next timers into @tevt + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + * + * Caller needs to make sure timer base locks are held (use + * timer_lock_remote_bases() for this purpose). + */ +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_held(&base_local->lock); + lockdep_assert_held(&base_global->lock); + + fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt); +} + +/** + * timer_unlock_remote_bases - unlock timer bases of cpu + * @cpu: Remote CPU + * + * Unlocks the remote timer bases. + */ +void timer_unlock_remote_bases(unsigned int cpu) + __releases(timer_bases[BASE_LOCAL]->lock) + __releases(timer_bases[BASE_GLOBAL]->lock) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); +} + +/** + * timer_lock_remote_bases - lock timer bases of cpu + * @cpu: Remote CPU + * + * Locks the remote timer bases. + */ +void timer_lock_remote_bases(unsigned int cpu) + __acquires(timer_bases[BASE_LOCAL]->lock) + __acquires(timer_bases[BASE_GLOBAL]->lock) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_irqs_disabled(); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); +} +# endif /* CONFIG_SMP */ + static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { From patchwork Mon Jan 15 14:37:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188190 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744384dyc; Mon, 15 Jan 2024 06:41:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IFnjCMGgiOzu6tHYxdyOCLcHVgPx6O1Z2LofeRUkir1FXbmmIJnEZa0Ma7Eh+fuiwmF9Iaz X-Received: by 2002:a05:6a21:1692:b0:19b:250a:57f9 with SMTP id np18-20020a056a21169200b0019b250a57f9mr291934pzb.26.1705329712120; Mon, 15 Jan 2024 06:41:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329712; cv=none; d=google.com; s=arc-20160816; b=CwM+ETU/9AMKGjGcsqTOmFAuy5nbNq9Kdn7s8pVDSbLzLIdMSW4qlKhHXAA4CVlVdY XCBBVaBhPdOzv4DEqpVNaKwPg3OSLQbMgMzLt0CCFL+0r1nujFIcUCvQon2PXg8iqzKH e8ygm/m8R037EXsvK3CLxiZIkGLbTSGC5lsQdnZCGu9I4T9YmRx2N0911PoSdyBd9PMi 2qFjzbIbriUuKWsKji5RKmskjM3933jtgP6+DJZtupgCeWuanovjOUN5PiH/fxvnrxOF I7SoodhU44bvbzFu/+TufPUPfVXwj7LLsiAsMxwlvMiiFlPwyRWuMSUFk26XH4mjk71i X+SA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=Nb48Ec9P9wN/ba+vLQTCWoxnTPCDbsQyo2JqCHOzxss=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=oBS0IfqiLuga1fQt/UsWzd/Puai72mBtholtsHXAjD1Jikpih2lYQ9gfDBeJ8vePxH z0Cv3ZM5lXuDUEIizmY43ihC79u05EBplTR58odWJLgqb770ev2sQyuw0FThihKz5IHc dmKu97CyUL/q41Id1UAC+5UfG39BcbBCnloBf/QYymUqTtkwy8TfUM/X88o+eIE0zcBX KuvWvPmWFegueFgs16zMMiWwrXBz1/6qtJqDNm6D+p4nooOeHyhD5TGamGsyQhTKNB3w st2rl3ZSQos3BgcK6KtCMQQlaBCUypAtccgihlevYnVw2RgdzPM1aJPky6b56TlwQDF9 YEDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=RpM+MlQ9; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26110-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26110-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id q11-20020a17090a9f4b00b0028ded0d5690si8789063pjv.95.2024.01.15.06.41.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:41:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26110-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=RpM+MlQ9; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26110-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26110-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id E0C402820BD for ; Mon, 15 Jan 2024 14:41:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C70C7199B3; Mon, 15 Jan 2024 14:38:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="RpM+MlQ9"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ktZOmXHD" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64EEC18632 for ; Mon, 15 Jan 2024 14:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nb48Ec9P9wN/ba+vLQTCWoxnTPCDbsQyo2JqCHOzxss=; b=RpM+MlQ98engTxxgTZb3m1VEMYSW7zPbVdt9EtlYVA92ZO5VCBuGDLe2A6/9XL672oLhz8 hXSYjXGyXyvVjkbJn77lqdYHPrwJpatsb0XPTubsfT3bbsF+gBeLyYExxM9kiBcT09cRPe VfykTG5s77jURoIqGMsIiZpqAOkXGlARCipzI2ye5tp7GtgtkVj65uQ2RjEzym4Ewf64wB zywAwo3/qBIHBCyruCWmsDcEJC20gf1zfxK6h2ay031x717BwY+VWkhuy5d5FP8cMGnoIv jebL2aKqB99ATJBOMsN49nuHlggiEt9QxCdISlAHCOg1E8i/hTWq+FwR1Zyn2g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nb48Ec9P9wN/ba+vLQTCWoxnTPCDbsQyo2JqCHOzxss=; b=ktZOmXHD5e+0jPN1zjCs1yytQQtNHNYadVzQ83G1DCIneFUyqWPZpmcomr3cjqK/SuTg0n iPzH1Z1ilW5vU1AQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v10 14/20] timers: Restructure internal locking Date: Mon, 15 Jan 2024 15:37:37 +0100 Message-Id: <20240115143743.27827-15-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167808075560150 X-GMAIL-MSGID: 1788167808075560150 From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3e2adfc15f3a..ccb2a5c03b87 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2223,11 +2223,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -2251,21 +2247,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Mon Jan 15 14:37:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744563dyc; Mon, 15 Jan 2024 06:42:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEy2E4A2No5tAvo41H1jt0Vcn2cdrNTke6PAFJHnWWhSi7MpRFopOaAILjofluYjasXjbf X-Received: by 2002:a05:6a00:1409:b0:6d9:a0da:4fbc with SMTP id l9-20020a056a00140900b006d9a0da4fbcmr2879217pfu.36.1705329729382; Mon, 15 Jan 2024 06:42:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329729; cv=none; d=google.com; s=arc-20160816; b=N8JK8h6yDGWADiBZ+p6kjFt0/2nNOgF2sSvOvCwtcf2rWPtposToPKGB2edqSYkvZP Nji1Zy7n1G7dQ6rS2esbhTFqBgoUZTcrkYB/z38LOYC+l2gD3ZYXuBwNlPKuwvlMWfCz 8c7Bvf3CluQO0igMX7PXfmJ8eTBSaViXFBhgGaJ81ZP5na+Vp+XMwOZdwuO38bt7oTXt Jc9cemW0SDG2cip8w9LEld/bFtUo+pEQPcdq4WCDkF8PcTSAbS9YkrkJ2E6Hwq6zD1UU GtfdRK02FmVtW1f2VKfVfgWy3Ly2OT3FDHWJ8KNa8w6ztwWN8WYthITu127hbnAaxU2O +rhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=Opkow5gPOaaBUb3NbiN3AOkG3DcAm/G1qJk4x5I78GI=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=Q0yS29QogItp5TXQ5qGZgkvuNG8Q3+QfZWTG6gOFYEJ3O6lQwccNNQFoxcmbZpTxW6 bVk0CLZO9ODEVU/UnB0fbyon7s3EO8fVg+KNjWxUyaowMVVo1wZaU0x2l5B4VK5qxFzI TgRu4xdpqYUWwtXxesK541SARG+DcXPHfixo3DB9qKW4+cTVVsEgnbXdwwNoi/9KnBb0 qE1R+Oog6+Oe0d0iGJxhWOWvbWb18t2rGOMMmk2OfvYbGHOXAa3HnRPueAyMK8ok6rKy ANi4cPDlYj0Xol1fC6JHU9ejjyjMt8bRDq8oI90eNNZU55x/yZZwQva79mBQ88BFAX35 KYng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ie8gWsdU; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26111-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26111-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id d10-20020a65424a000000b005c2786b7e32si9036382pgq.812.2024.01.15.06.42.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26111-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ie8gWsdU; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26111-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26111-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 1EB3928264F for ; Mon, 15 Jan 2024 14:42:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 94833175B2; Mon, 15 Jan 2024 14:38:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ie8gWsdU"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="atGElf/B" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DE9E18C01 for ; Mon, 15 Jan 2024 14:38:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329491; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Opkow5gPOaaBUb3NbiN3AOkG3DcAm/G1qJk4x5I78GI=; b=ie8gWsdUYa+mDSLuzV7dk7msmvEw1vRr4MX7/KUDaezICOJHY5kGXthCkRRrq2V/Cm3QwT M/Ia4z+ICY3uEuKwzf+KJlwoCRioqjhG5E13ONLvcsGwXLtI6hdf3qlCZlTmqAHITBQnV/ oCmnmfKS4Ec15QGfvdk/K10X3oZ5J+4UrWvzf6cl/l+KT72NR1/6rUxenmm0HuPSDIFOEd Ig9fkQEMR2/Exz+mAs2mKoK63fKL4Y7v3bWXdFksZb7DIJ1wO7gDBLfTGXB6yWhCu8Isgb hoGlq1Q7rouQh+yiYK2hDV43qJ91UebEjVLWxcgDnugQhnfUmp/oz2iJ8qyzCg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329491; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Opkow5gPOaaBUb3NbiN3AOkG3DcAm/G1qJk4x5I78GI=; b=atGElf/B15NatGtc5QWfPX7yXEKL5ON23WrledoHFHNeTOnwOonj+qkfzRdjJFyPyW6n21 PvNi9dInYpmACHDg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 15/20] timers: Check if timers base is handled already Date: Mon, 15 Jan 2024 15:37:38 +0100 Message-Id: <20240115143743.27827-16-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167825950150130 X-GMAIL-MSGID: 1788167825950150130 Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local CPU, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: s/cpu/CPU/ in commit message v6: Drop double negation --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index ccb2a5c03b87..eb99297a96fe 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2225,6 +2225,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Mon Jan 15 14:37:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744651dyc; Mon, 15 Jan 2024 06:42:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IGbYfl261Ztz2FSv8pJbtQ/FAjBOBzeZ2WneTdu6Fl5QPq3xUmfM5GtCuJ/Vsoi/E7y3dfH X-Received: by 2002:a05:6214:c6e:b0:681:5d1a:883c with SMTP id t14-20020a0562140c6e00b006815d1a883cmr2683987qvj.43.1705329740018; Mon, 15 Jan 2024 06:42:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329740; cv=none; d=google.com; s=arc-20160816; b=fHUVmc5elKpwLcyXjvkqyKZ16x3ga6jv0TdO8YM99WvXvpGoZKBsp/lkfxHB3Bcy09 HuRNLZXUkrGV1lrC0RdbdU6dEqi3wc8z3wefEannBlIEovU3A+AZymt4D+l2tlGeM7Db BrFU7oQsuVct4AtIM/y1NgKdrFhKYNn3C/QuU2YkvckjD0mwVzT+nf+Hp/nrhVSicqNW MSLij1aDJhRDkVlCJqN/wIrng44FMVF4POCLhafAJITUFf5OWPPF29jPCT5WdQ9H23oa qZyl6cfZ+EgruiiSzD2rOSSJgvfwXOZoxMyFhST1qARsJ5WHsyKEsAYQgdhZEaGpwAFG QMVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=FfYsNU3nxkN76DhpfApxecdpLCdc0FbmIjtWoc6sI04=; fh=H6hHKnScvHhKAB7z8Wb75LBNJxB0aNyPhv3ZCgyEYhY=; b=zpCi1+JUOcabKg3PItV+DAfpddjZ+nlowDvEuatofCwSQd4SZYnDLjjHVQ8deC556q Lw94nvqP63273okSyGS3koka+f8yFu1Bljlwr/0cIp5VXH3eRL+mys7NAgfRdJRijVGN 7OQjRnljQpxSHeuge6Q7b0c2ixEWcxGHPmWbEoS4rO/bG7jXha+K8CbhHGSNNwivyslA BanPOwIpGYaC4k2bxGv47FnYMg6oqnY0Hc42jLQVCZzEetaqurrNJ5vaivYDI8x83jtu dUbNlY1+vVHLZ6LwAqkPV1GgO9gdR9YlLs5aB/TBJFd5y8aHrwt/rnp/kdxPnxANP6bG 65Ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Y12vRgDO; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26112-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26112-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id n18-20020a0cdc92000000b0067fe06ad843si8000354qvk.518.2024.01.15.06.42.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26112-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Y12vRgDO; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26112-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26112-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id CB1E51C21BF0 for ; Mon, 15 Jan 2024 14:42:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 34E7417744; Mon, 15 Jan 2024 14:38:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Y12vRgDO"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="YRhjk5FM" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE71118E08 for ; Mon, 15 Jan 2024 14:38:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FfYsNU3nxkN76DhpfApxecdpLCdc0FbmIjtWoc6sI04=; b=Y12vRgDO9utSd0YcJu5fn9UXbjopBvL7kF/0XnYRwwON7Bf/G3o/T4W3sD6QcoHlq1AIDs lNwLRS+UBIPbe8NXsen4ktizBy/saBwcZ4SwljENCh+vx/9bucVeQlNuWqytSP/TkZlm8r iPuOqBb9hPwkH3QgqBlYEUhj6jtuPtlLPqRPNObeRhoXY4lMXzD+/V72lcIcHzfQMiCJcQ z1scyh6FAzsxuawiVN+csfo6MNNBJZvd3dM0WG2KjzlsR/Zb6VU07VrMqt+H4omH3hOhlH Wam9gwWACpqT6lZkLnGh6HZSvhN5Or9R57BW6gfIQzomE7vFRjpnbqKo7wGBNQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FfYsNU3nxkN76DhpfApxecdpLCdc0FbmIjtWoc6sI04=; b=YRhjk5FMhkO0wLgA/Vbh2OR1jdhNeTFpGp1jFIrxYWwr0WG9QxF6CHHnyEgwptnnUr01IK Pt1zN3hGlWm30bDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v10 16/20] tick/sched: Split out jiffies update helper function Date: Mon, 15 Jan 2024 15:37:39 +0100 Message-Id: <20240115143743.27827-17-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167837399939791 X-GMAIL-MSGID: 1788167837399939791 From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global function in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 8b0c28edbd09..ccf39befde85 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -157,6 +157,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 27f1a2ae7f39..a3eb5165d439 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -799,18 +799,30 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - u64 basemono, next_tick, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + u64 basemono, next_tick, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Mon Jan 15 14:37:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188194 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744687dyc; Mon, 15 Jan 2024 06:42:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IHa0LuXGVWkRppeoiW8z1YSrkcbHhJiISTP00TyAriZ0X0gBDFaHsqAmg4U27XaBaBbRqc+ X-Received: by 2002:ac8:5c4a:0:b0:429:9e25:ac3 with SMTP id j10-20020ac85c4a000000b004299e250ac3mr7393106qtj.20.1705329742333; Mon, 15 Jan 2024 06:42:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329742; cv=none; d=google.com; s=arc-20160816; b=aAX1z9s+ma9q3m8vmwdsJO0cQFIAcm0RPcp83Aput0ORuy+gXExYdwlVPCvQGFcvNg 4htDP764q8Bu9U5Gxrc6fetX9RGFdgmWmYLtn2d8IAtpIakec2O8nvfMe3delKR5sogr 5r7CXcRQRclvPyXBkPNGAEye/7HQX8xXFVif2hZQBgd++eLOtBbAMNjj9Sv0KAKUABHh YUIXDxvOpQgXEJ9SjUU4vt35kvaichkcqId6bWNMy/FCInowPf2sap+B6FBoSjkYwW3M XiHKLO1JglmiJPtPpsuxr9hHPUab4gEk3uYquPohpeYazDgx9QSbjDJol7UjKLtaZgj1 yUYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=WgXCRky76cIAK5QBFsb3qOUmiS36hjINUCdmIygML+w=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=iPUa98Y6lRielOwww8w/+kMln7fyXa61bzs1HyUNvITHMAEArEdiq8vGwS85rsEdlC Kh06KqAVgNkos6vVJr8HgCFfzP/cYWBO8JipR1UfmMqpu+kIYqCj1OXCZDedenGz2PTE rGvBnMpi2No8gdfZeD1ItVG72QsJ2+1w8Yx6xHQUfdy/EsefHEK5GlvPbNs+CBwTobWc Se0oqkLNzrVbrPv9SRM0JOu6CT5ZMVCKMR/UrznMDth2uOHGS7M9PrjnZ5q9rN/ZxCK1 cvsY7gxw+NtoNJ6om4ioxpSN/VmCNBbTDBX6yoQXDh5kHlkZ/Zsr6ybWk3hBKC/KyL+4 wfwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=VJ0ZdCXs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26113-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26113-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id o13-20020ac85a4d000000b00429957730dbsi8441489qta.141.2024.01.15.06.42.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26113-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=VJ0ZdCXs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26113-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26113-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 23C7D1C21B9E for ; Mon, 15 Jan 2024 14:42:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 590291774A; Mon, 15 Jan 2024 14:38:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="VJ0ZdCXs"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="MHGgPXm8" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33CD418E1E for ; Mon, 15 Jan 2024 14:38:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WgXCRky76cIAK5QBFsb3qOUmiS36hjINUCdmIygML+w=; b=VJ0ZdCXsTwC3rHgQvht3mFDy8gMSW7AsbulTkecRFJ2Z+Ju9JPZXmawgbflukXbt5vgBWe KoAdUAQiIYZj9c3G+ACdgevN3KtEj59A6YXhOwB3PxjpgD0w3wuOPKlr6g+G2vhRTCFTN0 KXDI2rTeKIjmNTpyNDzBBtDVqevBouoMOswftP5rL36OeUJAIEFu9YUdUUWggGpC7pwSIZ IIORwcCgDMWlkQ+mRZc0pvNWNNWj7hTiUqt1eBvTNfP8MohcZzchWXpz3DX8xGDQ6Iq+0L pjE+a4/quqWC/j7/TNb5IbGcfTmKB5R5LVodaUb7pwqqkShObFwWnxGH//uuMQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WgXCRky76cIAK5QBFsb3qOUmiS36hjINUCdmIygML+w=; b=MHGgPXm8qWbBjcbzLFlIsP+V77UUBvpMyV8XjM6R3228s7QVVTwUJQ0sqMlPvj3iFVm+Sn mBcwtPddeKZe2PCQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 17/20] timers: Introduce function to check timer base is_idle flag Date: Mon, 15 Jan 2024 15:37:40 +0100 Message-Id: <20240115143743.27827-18-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167839482025401 X-GMAIL-MSGID: 1788167839482025401 flag To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have a function that returns the value of the is_idle flag of the timer base to keep the hierarchy states during online in sync with timer base state. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Fix fallout of 0day: Move function definition of timer_base_is_idle() into SMP && NO_HZ_COMMON ifdef section v9: new in v9 --- kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index ccf39befde85..7e3090109e33 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -165,6 +165,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, unsigned int cpu); extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); +extern bool timer_base_is_idle(void); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index eb99297a96fe..3c49d8fdfd53 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2099,6 +2099,16 @@ void timer_lock_remote_bases(unsigned int cpu) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); } + +/** + * timer_base_is_idle() - Return whether timer base is set idle + * + * Returns value of local timer base is_idle value. + */ +bool timer_base_is_idle(void) +{ + return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle); +} # endif /* CONFIG_SMP */ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, From patchwork Mon Jan 15 14:37:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188196 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1745031dyc; Mon, 15 Jan 2024 06:43:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IGPSISZyLJpSeF1D58T7uYEr4YezE9hj0KMHehCNO7C6OLNYMyU1GQn/JnCY/LZ2fbLVxqq X-Received: by 2002:a17:906:bf4c:b0:a2c:cdd8:af91 with SMTP id ps12-20020a170906bf4c00b00a2ccdd8af91mr1481846ejb.264.1705329779741; Mon, 15 Jan 2024 06:42:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329779; cv=none; d=google.com; s=arc-20160816; b=rT7TbSCQwGz5BQqi/bnyjJzAOtfl9N4qZivNiqaY8Usqzo7X+7cP1190BKmMSumZDr SmbDJcDQYmxAaHUyodh/iON/tmrwWrwiaFxy7b4oom0jehJuEhE5tm+cLVn1SpXl46iS QtwhuaIUjaZNNhfFsQAGyRq/ATj9/x8eBkp1BK626mkPQ/bFJtTXXW2FCAMSRQGq/jXQ SqnidZg+Sb+roK+KK88d7XWiB1MPiWpPuAr9IeqTnDtpwuGvRu/PFPVSRA6G6zYvK8RH RkRORuzPCBHoYR1gmHodWvcRhIB03sXLgHc3xlIAgz8ccIWhEd0s78YKYNpIqleddGwG YhFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=Cm9ubgE2S8LXnJlXOk04it5H/fBOtaMthr0f/KhtUW4=; fh=PG6uS4TiiUSDyl8D/joYkWbwCgDm4ug0ir2h7tHBJXQ=; b=zIOmam1vM/j1hHIGp3GIormsq4MJSOMDwDMTEFepwE2Rafuv6ZmoOdnY3Tlh+U8BMu T0LHbR0wUfFUZGvd7SyUev97cTvEKGekkjdqEnw4bvGoCWLZnQPcgRn966Nz781WBmWZ roF2KassLOTq5etjRNcLAWQC/BQPB53Y9Z2kpZYVB6GSsiC/06pFK6vN9eGaP6xGiXca d419K9D8qp3MizyPlIZw14fr4BsXm0BOfo+cwoZYMMmE2XfKwg5ZYSvz1NdqhYoFRFLB 2VciL5ZyPU/d29JbBUBSzDiTQj9YL/wu+5ES/DQ5sJB5bE93zfowTmdeTcDeg0uvr9b0 H9Fg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2DRMDITF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26114-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26114-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id t19-20020a1709060c5300b00a2b1f0f2ef4si3793772ejf.707.2024.01.15.06.42.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26114-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2DRMDITF; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26114-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26114-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 339061F21FD0 for ; Mon, 15 Jan 2024 14:42:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 85A6519BCA; Mon, 15 Jan 2024 14:38:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="2DRMDITF"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="/JAX5835" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E605B1944E for ; Mon, 15 Jan 2024 14:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329494; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cm9ubgE2S8LXnJlXOk04it5H/fBOtaMthr0f/KhtUW4=; b=2DRMDITF5UybmTlVWk6gbbm2t4vIeSkHOcUurkwV7YpktOGCzTlCCh3ZAah2BOvxVTUt35 s28u+l2Nrdf6OH/dTiqHMdDtApojeHNR8tEdP4EEjc9WmAj0c1nRb37CoJvBmAR9+QZPCQ AJKy7Y2RUQA3Wjg+wCxwDOXlCbsGsrBrLTlcQKkVZk8G55+a4YZWnSuYCE298sEJ7xguEY KtQ6k3Bc/bpXxzQcqx/zM6a/72kN+plAJb4LwA54rg983xVVT3YFlA4NpxUzI9clEKLmiO 75jmtz7pr9fX3V4CW/P5tS2eloK8OybLJCefsfsAOJ+zITWAlZGHX9MpbnQ5Xw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329494; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cm9ubgE2S8LXnJlXOk04it5H/fBOtaMthr0f/KhtUW4=; b=/JAX58356Wn3eGipK9Y6eLDwDlXr1PO6NkOuv8E/VD2z6yhBY067mF9p0EwSdbiStsl2A5 p0stlNxo5CzdW+Cg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen Subject: [PATCH v10 19/20] timer_migration: Add tracepoints Date: Mon, 15 Jan 2024 15:37:42 +0100 Message-Id: <20240115143743.27827-20-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167879423358473 X-GMAIL-MSGID: 1788167879423358473 The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen --- v10: Make an entry in MAINTAINERS file v9: Add tmigr_cpu_new_timer_idle tracepoint v8: Add wakeup value to tracepoints --- --- MAINTAINERS | 1 + include/trace/events/timer_migration.h | 297 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 26 +++ 3 files changed, 324 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/MAINTAINERS b/MAINTAINERS index a7c4cf8201e0..9f9f2a695082 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17244,6 +17244,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core F: fs/timerfd.c F: include/linux/time_namespace.h F: include/linux/timer* +F: include/trace/events/timer* F: kernel/time/*timer* F: kernel/time/namespace.c diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..a2e7e32058f8 --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->numa_node = child->parent->num_children; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->numa_node = tmc->tmgroup->num_children; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p wakeup=%llu", __entry->cpu, __entry->parent, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DECLARE_EVENT_CLASS(tmigr_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu wakeup=%llu", + __entry->cpu, __entry->parent, __entry->nextevt, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_new_timer_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index de1905b0bae7..d3b7200a9f4a 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -14,6 +14,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -537,6 +540,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ group->groupevt.ignore = true; + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -546,6 +551,8 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) data.childmask = tmc->childmask; + trace_tmigr_cpu_active(tmc); + tmc->cpuevt.ignore = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); tmc->wakeup_recalc = false; @@ -703,6 +710,9 @@ bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child, data->firstexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, childstate, groupstate, + nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -745,6 +755,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + tmc->cpuevt.ignore = false; data.remote = false; @@ -787,6 +799,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return next; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); @@ -872,6 +886,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -1140,6 +1155,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_new_timer_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1243,6 +1259,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, } } + trace_tmigr_group_set_cpu_inactive(group, newstate, childmask); + return walk_done; } @@ -1295,6 +1313,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1407,6 +1426,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -1424,6 +1444,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irq(&child->lock); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active children need to be active in * the new parent as well. Inactive children are already marked inactive @@ -1504,6 +1526,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irq(&group->lock); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no children that need to be connected */ continue; } else { @@ -1571,6 +1595,7 @@ static int tmigr_cpu_online(unsigned int cpu) WRITE_ONCE(tmc->wakeup, KTIME_MAX); } raw_spin_lock_irq(&tmc->lock); + trace_tmigr_cpu_online(tmc); tmc->idle = timer_base_is_idle(); if (!tmc->idle) __tmigr_cpu_activate(tmc); @@ -1592,6 +1617,7 @@ static int tmigr_cpu_offline(unsigned int cpu) * offline; Therefore nextevt value is set to KTIME_MAX */ __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); return 0; From patchwork Mon Jan 15 14:37:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 188195 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1744957dyc; Mon, 15 Jan 2024 06:42:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IGUXPHPGw832QU+mUxeUL5jkfAO+U+VpU1XaLiAJUYnqFMJnrwjGmkDQsyPkkx4ZYdRxYRM X-Received: by 2002:a05:6a20:d80a:b0:19a:e925:5e00 with SMTP id iv10-20020a056a20d80a00b0019ae9255e00mr842570pzb.101.1705329773230; Mon, 15 Jan 2024 06:42:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705329773; cv=none; d=google.com; s=arc-20160816; b=HcwWKeriWFrB4LSpWcbUoSAV71CdHKLPlSaFTr4O3R3Pn6Rft/E85JVe8IYW6lgBKW Qj4k9aTzfXK5Z3f8M1Q1m0wSjG3qoDES8t534aNJavCWwWJwAREDeBRisUMfBVejVbou 52XC4m6Hk2BxMScOKwam/q5tiQc7qQQorpkxx1UutDu99JajbpX5C1h9sv86V7NlSQa2 zUQUS8A/zUomO4lA+jm+kUlrDf6s2n8AjT7yGOaJ4BsXyuuLLRRSq+k6E2cKd9MIfrIy KNuvWg37YHPo0j1wgr3XbBJd0GaNy27NVltgroXsEheKPfBXKkQHQO5EaAeRJPE0uyKS BuYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=m8FYkcVHVLyMjEugh4Zzvv2eVrdB5BKbeH71HJmbkiM=; fh=Qsvhj/y6vhUDpzzDJgUZ1ZrqVjoqsWAHfqtsSs/uzrU=; b=W6mjaSZzgbrf8iKCgavuwsrqtb0niu8jaOE50K2MI4uon3krWTPmHbdicruMNF+mF5 woFnY3i4rujXC9lvcOea7k0SFB1mIG5znbQ+IXJs7K9fKeD9gmFPDnN2o+0gahJFNiOO HR1eFK4jzXg59x4FwRe9z0e4xw7SGPHnZkEWMDKtRHytxiQlVJyEwMynaExXX/JxeuRq R/wcal04hNQPKook+sSHs0j//F2be+zO/1E3T/5FtemwiXXwEfKIfpLiCdfAhQdTKccD lYuRXAkZqzZGPOJ7vB6O1WjIMqJvjGB6cEaMHAuM0/rzapItaZvwRIu+pQP6XhRUIjKn n2bQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=sxM7WZm+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26115-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26115-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id v5-20020a632f05000000b005c6f9aca013si8725724pgv.597.2024.01.15.06.42.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 06:42:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26115-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=sxM7WZm+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel+bounces-26115-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26115-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 21B14B216CB for ; Mon, 15 Jan 2024 14:42:52 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7244819BB1; Mon, 15 Jan 2024 14:38:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="sxM7WZm+"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="V0te2fdr" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EBC51947A for ; Mon, 15 Jan 2024 14:38:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1705329494; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m8FYkcVHVLyMjEugh4Zzvv2eVrdB5BKbeH71HJmbkiM=; b=sxM7WZm+5FtzNzexxDgXWNCqEwS3r5nCxUYiLWgomIg2Ni0lZoVAW/wPdMA1M+Bc7VBtYq s9Oz76cdORwbPA5moxhUmqsWdQDhfbyeKywlqDVpggedSs/840JvenzNjNW/5eGJMml7DJ TYdSobFWHVKkFa46LjoPr18at0O3FAKrL1peY1NhJTbiDJyO5U4rWHoEPGc0ZSqfiBjuTB ky0hbXqmr529qRmjolKKkW26ufgXmJTkyVqTeIPobZvpL6c/T6UQLYc1HKQHSijO7Ih/jB KKqlYj7gI5KNeB/ZyeNzmq8vMxHgW8g4qEwm85C2XubaSeoQgO/VbwkJFbN2qA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1705329494; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m8FYkcVHVLyMjEugh4Zzvv2eVrdB5BKbeH71HJmbkiM=; b=V0te2fdrVPmijf/2dLYsUg5/Bqkk1AzUt4raxBy2PrteUn3wgDmpMfVN5HtU6cYKkAigRF 83v9ZUg5aHm1BGDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v10 20/20] timers: Always queue timers on the local CPU Date: Mon, 15 Jan 2024 15:37:43 +0100 Message-Id: <20240115143743.27827-21-anna-maria@linutronix.de> In-Reply-To: <20240115143743.27827-1-anna-maria@linutronix.de> References: <20240115143743.27827-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788167872185973835 X-GMAIL-MSGID: 1788167872185973835 The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the separate storage and eventually pulled at expiry time to a remote CPU. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v9: - Update to the changes of the preceding patches v6: - Update TIMER_PINNED flag description. v5: - Move WARN_ONCE() in add_timer_on() into a previous patch - Fold crystallball magic related hunks into this patch v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 14 ++++---------- kernel/time/timer.c | 34 +++++++++++++--------------------- 2 files changed, 17 insertions(+), 31 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 404bb31a95c7..4dd59e4e5681 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -50,16 +50,10 @@ struct timer_list { * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. - * - * Note: Because enqueuing of timers can migrate the timer from one - * CPU to another, pinned timers are not guaranteed to stay on the - * initialy selected CPU. They move to the CPU on which the enqueue - * function is invoked via mod_timer() or add_timer(). If the timer - * should be placed on a particular CPU, then add_timer_on() has to be - * used. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the + * timer was enqueued. When a particular CPU is required, add_timer_on() + * has to be used. Enqueue via mod_timer() and add_timer() is always done + * on the local CPU. */ #define TIMER_CPUMASK 0x0003FFFF #define TIMER_MIGRATING 0x00040000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index a52f68be8dfd..8d53718af21d 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle && timer->flags & TIMER_PINNED) wake_up_nohz_cpu(base->cpu); } @@ -941,17 +944,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void __forward_timer_base(struct timer_base *base, unsigned long basej) { @@ -1106,7 +1098,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -2237,7 +2229,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * granularity skew (by design). */ if (!base_local->is_idle && time_after(nextevt, basej + 1)) { - base_local->is_idle = base_global->is_idle = true; + base_local->is_idle = true; trace_timer_base_idle(true, base_local->cpu); } *idle = base_local->is_idle; @@ -2303,13 +2295,13 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) void timer_clear_idle(void) { /* - * We do this unlocked. The worst outcome is a remote enqueue sending - * a pointless IPI, but taking the lock would just make the window for - * sending the IPI a few instructions smaller for the cost of taking - * the lock in the exit from idle path. + * We do this unlocked. The worst outcome is a remote pinned timer + * enqueue sending a pointless IPI, but taking the lock would just + * make the window for sending the IPI a few instructions smaller + * for the cost of taking the lock in the exit from idle + * path. Required for BASE_LOCAL only. */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); - __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(false, smp_processor_id()); /* Activate without holding the timer_base->lock */