From patchwork Wed Feb 21 09:05:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204025 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918114dyc; Wed, 21 Feb 2024 01:06:57 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXM5aZibhEEyrsF7EMun1MyLuw/jky4PL3p8bt6Y54Nz4Fjt4DNIxdyblalo/qoJZV0xprngtAkjKfoO5qCPydNnNkGtQ== X-Google-Smtp-Source: AGHT+IGSMD7vTTJmuduntVgEUbVPKROlb/tlTGbUz1WfZbLp9seHUElP1pVF/r0qHIyl99sJ4L7W X-Received: by 2002:a17:90a:cf03:b0:299:c5a0:868a with SMTP id h3-20020a17090acf0300b00299c5a0868amr8211901pju.11.1708506416961; Wed, 21 Feb 2024 01:06:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506416; cv=pass; d=google.com; s=arc-20160816; b=CrjtP8INKT5t30TGQJTUq/WIBKgTlEsvX7jVJXQHY1jFS/IRCwcCbTuJ/kQxK96+jl hBTuG/Sig1lxbqlfoo2vMKJZb5fyxCQfkU2m9hAH05LXcagBLSMJ7le13HAa0+aFqy4l tXaQN6jBtF7afxjY2rayjILTWZQBdUllfPm9ZfAAeX32bmd9aUeaNSt9EtOlInqDrTHn 1REPS2C2UbanzTdhLHT/RVVAz9g1TbgYFIJKnf30P4t4nfR++aRLeywfBzQXEYDY3DQf QbLZapbEYxlFea50gGXkNSYYA5QAlB2p3pAbI6J8oNo7e6QXTKsvCi8pKbo7jy9XV7jM Cdng== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=vneXKeJPs1ClvnSuuOqDrAMvkymNvjYj3Z1H5l2g9Sg=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=IMSxaH+DxQoYQlQ0ITviHbIq0XCDSjfioO+tnZqLWSNlCAlGm/TMr0wjMZtBWXKIXv ctUFLZYIVlGQgMEne5PzDNL4u9blyXgSMWJ2MsLOspCr03oFBy5kOuGngbbKu2OU0W9W LSAC20B4xLtw9lABAPASQvk5mAjRtO2P61X36x+0y1TNFRW41ah92kSmyWa7nORwec7o 0x3yrwfwTnw9yGqgDB1YbMu3aTpFu+JDG8HqJihxJyI1TtlpBHPrUVrul67ny0+yKj86 yEssad+PBwSI43SbKqBOuykn2/wSB/6Chdef97HgFEvZ1zVP9t4vwCWemD0RuZGIQCdp YkHg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KEmESPwo; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74345-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74345-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id kb16-20020a17090ae7d000b0029909da2167si1181017pjb.32.2024.02.21.01.06.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:06:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74345-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KEmESPwo; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74345-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74345-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id BBAAB284C6D for ; Wed, 21 Feb 2024 09:06:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 406AA3D578; Wed, 21 Feb 2024 09:06:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="KEmESPwo"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="GHH97aoc" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFB8E3C493 for ; Wed, 21 Feb 2024 09:06:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506376; cv=none; b=cVAmObVeCc8nxceKq/0kN2+PmKItfeQ7kKJGxHKiFv3tIx9K3V1P6PjOSZoDLKmor+AFf4WeXIXjO7o9d1IL5fll2uYN7ka6+atnxu7D7JB2jx3PCe5X3Zo5GJwk9tTlGkv4BuBi4wGUir5igV5bOD6k3v5LyDpjecWS4SxvEt4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506376; c=relaxed/simple; bh=0AtunAhm4yop3m4xvmzGd+hzZGncHjxpu0P4GmAnNxo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UZ4S318LGdcsmuDth14Bi82WxLcnjWUld8k2cP/OGtmMnne4GKtlgoOlHEEgsbqOhyebPX+o5RJQvKUGs6ux8KvYj5rn/lidVD6AqvGFdcTpUPfARHlrV6ZktX2lxSnAzpfy2eqlR4dE7YIde8jqVHQSDu0SuJ/Wetu+v/RpbP4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=KEmESPwo; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=GHH97aoc; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vneXKeJPs1ClvnSuuOqDrAMvkymNvjYj3Z1H5l2g9Sg=; b=KEmESPwodaFkrKmhUVstJaC78bDPtNz/TfDJM1wMDkyhH3OQr4WPHShzCaRQ2REOUVWJna 4uCF9pSWasqFawBs8UhAfbcDP22hl8t2rgR/kVY9g/Tvc7XsYAI/jA4atC/fI9fIhCQFTB CUk0bZme5i6NAPiQIHlTLZhQ0PZNyDQWMAOpdKK4qNIrPV6ZaQUL5oXf/Ji8GN6yR1IJ/V PxMWfVUlguiWWSTeXxyXzbTfsZHTgGa7mqh6uPKuxrG0OuxUtCARP1OKHXUIqg/XFdd73d 36AIENSxuwnRmkRzct9rXaHnTRn8WXF9S/oXGrnn79jAsaIsd5Ci0l7XSIjErQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vneXKeJPs1ClvnSuuOqDrAMvkymNvjYj3Z1H5l2g9Sg=; b=GHH97aocnaVxt5bPD7KsD/FW0E4YZPLwb1A62txKWJjEd5Riz/Ot57Y5J+vogyUreQL9cr HKbTbxelpfjQmZBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 01/20] timers: Restructure get_next_timer_interrupt() Date: Wed, 21 Feb 2024 10:05:29 +0100 Message-Id: <20240221090548.36600-2-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498824338254026 X-GMAIL-MSGID: 1791498824338254026 get_next_timer_interrupt() contains two parts for the next timer interrupt calculation. Those two parts are separated by forwarding the base clock. But the second part does not depend on the forwarded base clock. Therefore restructure get_next_timer_interrupt() to keep things together which belong together. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 352b161113cd..5f21db4fa3ca 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1937,12 +1937,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (base->next_expiry_recalc) next_expiry_recalc(base); - /* - * We have a fresh next event. Check whether we can forward the - * base. - */ - __forward_timer_base(base, basej); - if (base->timers_pending) { nextevt = base->next_expiry; @@ -1960,6 +1954,12 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) base->next_expiry = nextevt; } + /* + * We have a fresh next event. Check whether we can forward the + * base. + */ + __forward_timer_base(base, basej); + /* * Base is idle if the next event is more than a tick away. * From patchwork Wed Feb 21 09:05:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204026 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918399dyc; Wed, 21 Feb 2024 01:07:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWPmz1VhsrGKZvr/dqLNiYju56dUoNFnAd7I+P2Pub/ZaOnn3+voZWeGVFs9cgKu+9PPc8bqbflWNcaY01hdDcbnm4nVA== X-Google-Smtp-Source: AGHT+IHzpdT3EN2Gy402c8Zr5oP6GvKzjAAgg+oBq2+R5ukaZJxyDG/bUyDe0s6pjnFW6kFCb2HI X-Received: by 2002:a17:902:ea06:b0:1db:ed09:db97 with SMTP id s6-20020a170902ea0600b001dbed09db97mr10247023plg.6.1708506460058; Wed, 21 Feb 2024 01:07:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506460; cv=pass; d=google.com; s=arc-20160816; b=ztGNXDq7MNvadSJHLVP0LY0c1EIavOZV7rRfRsACDJO8Ik3/SX3X76NITzXOdQMxvx dZpTBvhsEOmEgRScAO9tX1GcT37BkuKoouz4Umf72k8IxVpLZk7Ya5+ItMUr04wFztPm J0M1oLso/L5l8juwXS67ijxh6uC1BtVAiS/P595vwb8lo9vCN63kCXxz492wON9/rtTQ NuPAm3yzjLWfu1ZKIX4u3/xf01kqGw8Ntvo9AQbY6ts0wtrMAkHVUZrbyeXgVfL4GFnR PFHlOizhGO/ZEFvdCk6NZ/rXANiFMU2s3jLb7YM0yxt6UrureN/HVKXhvX99hJcddq4n Pmrg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=ZnbOpzCKzJr8GVlvDiuyFIftQ7kDyRb60zxIswiDeKE=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=jf7YG2LvXcjrRGGs19pcU6FcZf6lLpUTzXG4879gT9Y5ajyQQj8KL7IqHj2QQzIViU AsTE2olOUvxRtg5+IHwf1sOZ64tV8hOZbxn2lFVs5dX7XlMXwFyaeIlcIaPsvTd3JoNE EbV2gCC644TtSrUodhz7E9gc65Iu2oHOheYtjB7aLgtKwkD09+rMZzfB/ePSHsMApAgf vbNfJi6z20SEnSRyD0XiJJvoy1NT4ykZlCj6r3nkpOBi4PJfrUF2kHwBkT+bvYNavS1q jN0G3nkktFxc6YxKyWFc9HQ2oUnsTi5EWrAb+TIci2GXRG3upawHCaIKakUmk5UIPK5X JD1A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=bW6c6uDK; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74347-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74347-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id n12-20020a170902d2cc00b001d94c1d01besi7961232plc.276.2024.02.21.01.07.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:07:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74347-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=bW6c6uDK; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74347-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74347-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 8A3D4B22531 for ; Wed, 21 Feb 2024 09:07:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 53E9C3D98E; Wed, 21 Feb 2024 09:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="bW6c6uDK"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="NS4V0cz9" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A58F3CF7C for ; Wed, 21 Feb 2024 09:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506377; cv=none; b=JABC1poGEa3Mj0h6IWjHvYtQUDWUFsmwMTL/QKkQ8Z62gsCycSGQLFkbp4Y+YCG+zw0krU2IdASj8Q4AIxOZ6M3JW/O4LUaeh5miZd4JhcuacrXUOd7l7h8e+J9imP30sFKClF0YBd17c7Bo/lnNfLesgFJx4t0b8yQvW2x/NcA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506377; c=relaxed/simple; bh=cx7W79/0JOWapdXA61gJNfkyrPTg/Tmal7MREy72obs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hJtnK4S2PnqIMip9I5ZLMDkFjzkV2SRMwJtojivOexWGcvxMAS5VXtnxiunVEJQJ+gLzpn+ApnHvF/0wQ/6TgX8s5ljpdAaBImRBjkxWxRjn4KMc2sWX56N1lLw5sA0gSA2UDKjavxkFmco73QKZM009xJBde7/iEcWbgDPrZVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=bW6c6uDK; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=NS4V0cz9; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZnbOpzCKzJr8GVlvDiuyFIftQ7kDyRb60zxIswiDeKE=; b=bW6c6uDKTlx+T/vPLW1QjAk5/ewWlAA3hBGqgC9UyIcgRmbVKwt8YMB1ME9UbPI1X+adBZ bpZNeGpLBPsv8hISm32Ex5iKN1Us0mBVRcPt/s0dx9I2FAF2sRKwfWdnHTKadaJu2Sq9MQ NU8svm0LqSj+jtzDp3/Nr5a9zdBYsuQ9JqvFk/98VfqLWmOAelKS1Cylt8SQP01uAHuRhy 6le2M5D/nGeoo9tZltDBWE/OOfxfFEx6DR+/RPUKhgEjiSrGqt7PVbJRyxnd/AYwXTTZYG DmYgq/08QwA3BlV9qooRhwlLTMai0bsDEgRUf8BPorQNXpO8wWzuLkFsgc+7nw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZnbOpzCKzJr8GVlvDiuyFIftQ7kDyRb60zxIswiDeKE=; b=NS4V0cz9P1b+jOQ8gLhCmi/ndTdsC5DQxDsMA7Yt/YaXthsygD5Ayo469VqNwBqmfIiZTO j4SdOyTl0bcTMUAQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 02/20] timers: Split out get next timer interrupt Date: Wed, 21 Feb 2024 10:05:30 +0100 Message-Id: <20240221090548.36600-3-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498869473328975 X-GMAIL-MSGID: 1791498869473328975 Split out get_next_timer_interrupt() to be able to extend it and make it reusable for other call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 5f21db4fa3ca..2aea55d53416 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,15 +1911,7 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -/** - * get_next_timer_interrupt - return the time (clock mono) of the next timer - * @basej: base time jiffies - * @basem: base time clock monotonic - * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. - */ -u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; @@ -1978,6 +1970,19 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return cmp_next_hrtimer_event(basem, expires); } +/** + * get_next_timer_interrupt() - return the time (clock mono) of the next timer + * @basej: base time jiffies + * @basem: base time clock monotonic + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +{ + return __get_next_timer_interrupt(basej, basem); +} + /** * timer_clear_idle - Clear the idle state of the timer base * From patchwork Wed Feb 21 09:05:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204028 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918511dyc; Wed, 21 Feb 2024 01:07:56 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV7N4ZUP7EcogMoxAsKwlsbAT9zZP9SwOVoqq2t5WCBo0K7S0ScqREJgKaToU5LRMW6z+ZhOgxDGSa6uDz9n4+hA6tJqQ== X-Google-Smtp-Source: AGHT+IFshxv7dLIVM/tqL6/1T0MzaRyYzBe9jOqMooWnBmX/ClUdzAmhoW8JH172V6hMEngL8ia6 X-Received: by 2002:a05:6a20:9191:b0:1a0:c564:99e with SMTP id v17-20020a056a20919100b001a0c564099emr885719pzd.16.1708506476469; Wed, 21 Feb 2024 01:07:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506476; cv=pass; d=google.com; s=arc-20160816; b=lEq13VKajaH8r/MqTxZWcVIadeqas9mhutwikyVynmSXdnwXdXuABVhSWRcYgsPucS sgFqPZYKg4qiW4SjfEFtkgUtrinOg+3RKr9I1w/OmMrHAF96R0GOfpMMhkzH/5V1crDV KyPkKYut9bcZGHAEtrhMSLmv2/Fm96AslNXnb4Hei7+JaX+Qz8G0vtNMSwBz713YL1vr CJMB5sVbrFyeXt7NGfJBYtTV7oHTNgcIapoodhomZQi0NUcXF0ST2WQyUzZhjT1H7ySQ pFKUBZTnoD3jEHFGzPP+qbhBxzcjK25JJ7EW4XACdygvdMt0hsrJ4S3gif8dzm77dbbM h41w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=9iSI+fFKaZsEg09o5KpmvrClYbUXst2Mqatycuvus4k=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=xN07kNoufTPlec228J373qFqvt2HBB8bvZ5OYYnkz7UXvv3v0/HLQJ3daQwcf9uxMF HZxxZSK/fVRsP5RyElsA5li/kitx/fWpdwpg2U/gN819HnsnwmmfhVASTzSy6fIwZmP2 U9k6DmBM6ISGNhEigsCD/6+aRuCJNNdWxBq/Rxc+qHRrd8JzYn+F0lbsebleFJZ4oT/u m8GLGPVcxHkiGWBnB4/5Ric8wjbD6Rspldy3DUIr6NnFfvvUgplAL46Ks+o4Qu3Nc+ik DZjYhcxhfctNVq7eZadH30q6Xk/OuzpW6T6ts7eapUg3aGZ5y136izyGLiRw46Dfsedt NLPQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="zLzoLg/K"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74348-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74348-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id w186-20020a6382c3000000b005dc93b2bc4asi7581304pgd.885.2024.02.21.01.07.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:07:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74348-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="zLzoLg/K"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74348-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74348-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 43CC1284FE3 for ; Wed, 21 Feb 2024 09:07:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 68EFE3EA97; Wed, 21 Feb 2024 09:06:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="zLzoLg/K"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="p8390MXB" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A5CB3D0A8 for ; Wed, 21 Feb 2024 09:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506377; cv=none; b=M6s/vaWTd3tMdJpfWwW5zk0j/l7fiYg5c1eAJ4HsUtS77pJ0UO6/lwjoxMiMC0T/TZyQqgik3I3+3YcuoHH4mr7pa4wLeKXDKuEuXBiGdHI5PbtEv2WOZmPj8xtpJUfctYzQuz9/dAYqYFvdU6kd4ROoSeMN2ZSc9pc+ENg1GHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506377; c=relaxed/simple; bh=CoA1DalDvnRQocmU4jzgfw8sIMWdry+4nJS7Y4YZlxo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rCg4FDuNflX+8PuNM8uQi4o1CmCbeZvFf6Kum5Ka4rbYhpG48CfUYSnUHMMVvRYV61LT8Crch22cFj7mbcQZdmpF1S++eF0KACVWE418bCSRkbkriL0vvRGPGBZN+dbO1mSFz7/k++7AoXTbhSnRsWr/VGiFJbldEPqicyJnB90= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=zLzoLg/K; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=p8390MXB; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9iSI+fFKaZsEg09o5KpmvrClYbUXst2Mqatycuvus4k=; b=zLzoLg/KycHu4YwafMTB2UkhepnmxPhFEU0uvY1lWf8jpdoy6UySKiPgFxdEXhhJyViXIR PB8hl7FeSq9WxklOnxHR/+BACLJzfs8g8+nxhQYy2wrEPKoH1ge42WqOjDn6spxEyL3e4l vo4u8bGCQ+DRrkWFFatweSp9SwBmfZLHT2ugP7UT7VpY9bLxna9DYqI1t3AE8RfWMHtXtx EIPlwIaF2Plcs48a8WMgJUNwpGalHLtVnqK0Yt3lQ0cTahTQMTDi/bx26/+xAzKQaa6nWt 1AR0HJ/IvlCNDFEAncM+TGpzsEgMhR5Oh4YzbRW/6/nsNcuyVCOHoIp5yEX6jA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9iSI+fFKaZsEg09o5KpmvrClYbUXst2Mqatycuvus4k=; b=p8390MXBavD7rE72Y1+1cYUD60wfPt3atraITeLJ8nqT60CdwQ7jv4UO+MkjTnN9sOF3B8 caUJh94Hgq0/yOCw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick() Date: Wed, 21 Feb 2024 10:05:31 +0100 Message-Id: <20240221090548.36600-4-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498886676607844 X-GMAIL-MSGID: 1791498886676607844 The timer base is marked idle when get_next_timer_interrupt() is executed. But the decision whether the tick will be stopped and whether the system is able to go idle is done later. When the timer bases is marked idle and a new first timer is enqueued remote an IPI is raised. Even if it is not required because the tick is not stopped and the timer base is evaluated again at the next tick. To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and get_next_timer_interrupt() is streamlined by only looking for the next timer interrupt. All other work is postponed to timer_base_try_to_set_idle() which is called by tick_nohz_stop_tick(). timer_base_try_to_set_idle() never resets timer_base::is_idle state. This is done when the tick is restarted via tick_nohz_restart_sched_tick(). With this, tick_sched::tick_stopped and timer_base::is_idle are always in sync. So there is no longer the need to execute timer_clear_idle() in tick_nohz_idle_retain_tick(). This was required before, as tick_nohz_next_event() set timer_base::is_idle even if the tick would not be stopped. So timer_clear_idle() is only executed, when timer base is idle. So the check whether timer base is idle, is now no longer required as well. While at it fix some nearby whitespace damage as well. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v11: - early return in tick_nohz_stop_tick() when timer base is not idle v10a: - Drop the unnecessary if branch which handles return value of timer_base_try_to_set_idle() - Do not open code 'tick_nohz_retain_tick()' and keep tick_nohz_idle_retain_tick() as is. --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 40 +++++++++++++++++-------- kernel/time/timer.c | 60 ++++++++++++++++++++++++++----------- 3 files changed, 71 insertions(+), 30 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 481b7ab65e2c..47df30b871e4 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -163,6 +163,7 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem); +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 01fb50c1b17e..72c80f6e7bed 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -849,11 +849,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) */ delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { - /* - * Tell the timer code that the base is not idle, i.e. undo - * the effect of get_next_timer_interrupt(): - */ - timer_clear_idle(); /* * We've not stopped the tick yet, and there's a timer in the * next period, so no point in stopping it either, bail. @@ -889,12 +884,38 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) { struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); + unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - u64 expires = ts->timer_expires; + bool timer_idle; + u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ ts->timer_expires_base = 0; + /* + * Now the tick should be stopped definitely - so the timer base needs + * to be marked idle as well to not miss a newly queued timer. + */ + expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle); + if (expires > ts->timer_expires) { + /* + * This path could only happen when the first timer was removed + * between calculating the possible sleep length and now (when + * high resolution mode is not active, timer could also be a + * hrtimer). + * + * We have to stick to the original calculated expiry value to + * not stop the tick for too long with a shallow C-state (which + * was programmed by cpuidle because of an early next expiration + * value). + */ + expires = ts->timer_expires; + } + + /* If the timer base is not idle, retain the not yet stopped tick. */ + if (!timer_idle) + return; + /* * If this CPU is the one which updates jiffies, then give up * the assignment and let it be taken by the CPU which runs @@ -991,7 +1012,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now) touch_softlockup_watchdog_sched(); /* Cancel the scheduled timer and restore the tick: */ - ts->tick_stopped = 0; + ts->tick_stopped = 0; tick_nohz_restart(ts, now); } @@ -1147,11 +1168,6 @@ void tick_nohz_idle_stop_tick(void) void tick_nohz_idle_retain_tick(void) { tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); - /* - * Undo the effect of get_next_timer_interrupt() called from - * tick_nohz_next_event(). - */ - timer_clear_idle(); } /** diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2aea55d53416..3a668060692e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1911,19 +1911,22 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; u64 expires = KTIME_MAX; - bool was_idle; /* * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ - if (cpu_is_offline(smp_processor_id())) + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; return expires; + } raw_spin_lock(&base->lock); if (base->next_expiry_recalc) @@ -1953,17 +1956,26 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) __forward_timer_base(base, basej); /* - * Base is idle if the next event is more than a tick away. - * - * If the base is marked idle then any timer add operation must forward - * the base clk itself to keep granularity small. This idle logic is - * only maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ - was_idle = base->is_idle; - base->is_idle = time_after(nextevt, basej + 1); - if (was_idle != base->is_idle) - trace_timer_base_idle(base->is_idle, base->cpu); + if (idle) { + /* + * Base is idle if the next event is more than a tick away. + * + * If the base is marked idle then any timer add operation must + * forward the base clk itself to keep granularity small. This + * idle logic is only maintained for the BASE_STD base, + * deferrable timers may still see large granularity skew (by + * design). + */ + if (!base->is_idle) { + if (time_after(nextevt, basej + 1)) { + base->is_idle = true; + trace_timer_base_idle(true, base->cpu); + } + } + *idle = base->is_idle; + } raw_spin_unlock(&base->lock); @@ -1980,7 +1992,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem) */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - return __get_next_timer_interrupt(basej, basem); + return __get_next_timer_interrupt(basej, basem, NULL); +} + +/** + * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases + * @basej: base time jiffies + * @basem: base time clock monotonic + * @idle: pointer to store the value of timer_base->is_idle + * + * Returns the tick aligned clock monotonic time of the next pending + * timer or KTIME_MAX if no timer is pending. + */ +u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) +{ + return __get_next_timer_interrupt(basej, basem, idle); } /** @@ -1998,10 +2024,8 @@ void timer_clear_idle(void) * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - if (base->is_idle) { - base->is_idle = false; - trace_timer_base_idle(false, smp_processor_id()); - } + base->is_idle = false; + trace_timer_base_idle(false, smp_processor_id()); } #endif From patchwork Wed Feb 21 09:05:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204027 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918434dyc; Wed, 21 Feb 2024 01:07:44 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWN61rP8Hs6o+81US6ScHVYXqyxzXXEBKKva4ck3hLkhB4rp8OKRE/jWs1AAO/BK+g/gct82yLuOU5ma+n0TSzlwRl+SQ== X-Google-Smtp-Source: AGHT+IFp318GK1mfCI61KXzizq9vMRPit86BohABoDx4fAdZhFTcRWmo+EjenmznEfymKkRVNzWa X-Received: by 2002:a17:906:ca59:b0:a3e:48bd:93ed with SMTP id jx25-20020a170906ca5900b00a3e48bd93edmr6902135ejb.61.1708506464622; Wed, 21 Feb 2024 01:07:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506464; cv=pass; d=google.com; s=arc-20160816; b=UjScY01mjT0HWIA2ZtKa6/eo5nOby2L15c4Ro1948vD9DWWikAOnkYSw/4a9BMWCxq PQ/OagcV12OTAxMqrDGuX67pDa8VZBtPNu2pTaDKZirRTN1OTiOc76tZAH2qs9DCWx4T /pa53oHJ5fBUvGgX4ULM0U1DkzlsOGNZWMdHuc1RBs2CtCCITL4B4jOzH5DDQtxIGjjB UXMDyCSwhWtGQAnK5ZC5a0dPMiiqNIPRytlAQ6FI9B/ueEHu9SXqj+AtDeoLfVWPRO51 zVpWDC4gSXqeBvBuqHeXoXsqIC/d5c80IhtY1mdzL6smUBT6K37VY9/+q8tQn+oMz8hg JuAw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=8x8V66oedAQdLiHPaHBR+VdwFWZt1XkVYVwYivWxpoM=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=kETjzURQT3UmdP2w/fkdq4GpeVAZwDmlSeSDyR/R/g/wornhkfLgqohGE2fFt3HTYH dzNapjCRmSD2CnTLsYflbn9clpQx0x8yN7cc1T3jxuGIR7kv3IBrRVd0q9aGlrYiZHtJ peTXDWhJmfpJlD/hwiPxltYn5fm3U1zCHgKHiCkSQ86nhF1uczqMWIX/u3rNRx0cytm2 A1Jr/Q6dYXOZtN6sO+Ch2xDikObq7gBH7HH+NrQVU3NPRXlSdLlczyTjLq4jIGU/AXis 6dDVlvAfe6Ac76EjCKdHy3TcI2GHvtbXuUNzr5PGUYfvzRAjUMiwEjaVzYsR0ciF0yia sSoQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Hx8+Knr8; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=tf7n+Sll; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74349-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74349-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id kq2-20020a170906abc200b00a3e461d567asi3413507ejb.666.2024.02.21.01.07.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:07:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74349-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Hx8+Knr8; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=tf7n+Sll; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74349-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74349-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 3BD011F21D20 for ; Wed, 21 Feb 2024 09:07:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B29833EA71; Wed, 21 Feb 2024 09:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Hx8+Knr8"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tf7n+Sll" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D68D3D0B4 for ; Wed, 21 Feb 2024 09:06:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506378; cv=none; b=MWgKu9LMEJHDy6uRQxViUyLk7zV59o+EpAJxujkbQ0yL8Tiw+liT7MSGT3jbzTR0BeOPW+hjoPr/ybLtjKZ/uz5SGoHibore1PqBxnYb341eA+WpaSoLv7rWQn/LEMA4Zm/4D349nKrLQoDjHd0lzRioqN+XZkjzOzN07o2ubZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506378; c=relaxed/simple; bh=ws4UsvqTVPEJwlRcuRRnPZASwZHWSXxgj2cFVwZJl0g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VuhR7wKrB2ogeC8svnWDZA969Ac2KTroZ9zaZxUSqhD/zPTgReGocAlYgXsB2FPfvRyMyOKjN3YyYLWNJyRGrCjqI509gY6Np/VOOhsZt5Iq9ReenyFPbEDEA677Kzc3GLDKLcfy/aG2YgTShYpR82yoMmmIE+0PVr3D01Lb1I0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Hx8+Knr8; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=tf7n+Sll; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8x8V66oedAQdLiHPaHBR+VdwFWZt1XkVYVwYivWxpoM=; b=Hx8+Knr8Z4dJ4chBC2NafZksHEPbKTCpR5IGhR5LRCe/JaVTRP1EVRAFVm/VJOl7MBivfe CPlw95EYEMRha70OszFXL+AVP8hFXZvntTYUYhybEgjSL5i3kmhQp7jFVcIZXwns/6tYGk q41Akso+7a0DWowVMQ3bmFU9ahEdtwN0UzW39j5mrSGuZrYeWnqnaMPlbWS3/e5rMXTftv vUPeQ9XL5zaV5IPOK+3s4AZJH2yqCvgWtbAoJ8ZbUQf2ROC3DW22hZe2oqukjmo/F3ggA6 Tncb3XnmNgp98CMmz3BrCv7Q9FsIO9MNd7eAnYKhhIOGIRlWbnXmRRiLeltMkw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8x8V66oedAQdLiHPaHBR+VdwFWZt1XkVYVwYivWxpoM=; b=tf7n+SllvBHbEQu6a0UoCrngxBM8lQ8i8OnvMAJYCfZztbbKhMtwZZwKcgArYXicK4Ufuj Q4BGYcmQOmDy/TAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 04/20] timers: Optimization for timer_base_try_to_set_idle() Date: Wed, 21 Feb 2024 10:05:32 +0100 Message-Id: <20240221090548.36600-5-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498874670065018 X-GMAIL-MSGID: 1791498874670065018 When tick is stopped also the timer base is_idle flag is set. When reentering the timer_base_try_to_set_idle() with the tick stopped, there is no need to check whether the timer base needs to be set idle again. When a timer was enqueued in the meantime, this is already handled by the tick_nohz_next_event() call which was executed before tick_nohz_stop_tick(). Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 2 +- kernel/time/timer.c | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 72c80f6e7bed..384723314c1f 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu) struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev); unsigned long basejiff = ts->last_jiffies; u64 basemono = ts->timer_expires_base; - bool timer_idle; + bool timer_idle = ts->tick_stopped; u64 expires; /* Make sure we won't be trying to stop it twice in a row. */ diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3a668060692e..2f69a485a070 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1999,13 +1999,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases * @basej: base time jiffies * @basem: base time clock monotonic - * @idle: pointer to store the value of timer_base->is_idle + * @idle: pointer to store the value of timer_base->is_idle on return; + * *idle contains the information whether tick was already stopped * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer or + * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is + * returned as well. */ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) { + if (*idle) + return KTIME_MAX; + return __get_next_timer_interrupt(basej, basem, idle); } From patchwork Wed Feb 21 09:05:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204032 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919243dyc; Wed, 21 Feb 2024 01:09:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWAOt6bjbVG8iVQsYxuMk1oqr5us70CW+gtjvnxWC4uCFQUxCoye+YFGIstNARmRwnM3hKw241J2wBj6TrJtgRrLOYBjw== X-Google-Smtp-Source: AGHT+IFbprqPeh9RmdLR4NTSU4HjhsKIUFBBee60BZgJtAqXylH2RpEMA+ebJGTvrbS3maBilEK5 X-Received: by 2002:a05:6871:7411:b0:21e:b43c:8166 with SMTP id nw17-20020a056871741100b0021eb43c8166mr9298642oac.9.1708506580111; Wed, 21 Feb 2024 01:09:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506580; cv=pass; d=google.com; s=arc-20160816; b=E0JBU7y+FMMlI3nkgOdNtmegZ7a1gpZL7Ezs6bDZXtKhhkNb3pHvjLtnbcChQJ2gky gSWwuacP4bhW0pL0lCijVw0aEsmuXw21EHHqKerpDuVFPFKjEUSONPn8pun0LMRT1ceG AFotc9nO4gJR2IFsfjtm6EGMMThQJZLA6v9YA1vZsQNZ9jt2nVTn0TssyNMhSUt98fVS R5hdqK3S167UyAV1rAoF15Jn0BTs1zX/pKM+l3KyP6bIPOglwE+1nwDg3dTSk2lyXqrc UaFoqQ1f54NUdxfMq1yXiMugZL0KjhDc0OV6dRy5hfoWncFQyN64l/M4SdbbSwqJD/Lf NNcw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=QisrBEswo4w1Tqs9XJzYOTpU4VJaqEco1Cayy0t85z0=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=iYZJFCfztvMSePNYoTg5n4p/Qc9iGGrGcU5KYuMAo2xJibmc1abpc/GpzQ8yo7xmqq XW+UBK91TahUz0fNn0wPxAmsWiprcJIBJNzI3GJq0E+K8OTy2t4e6kpFgrtcQ2v4kgIb 1+sukiM8KvXRN2ZQGGGhEGHM4Y1ksvENpJTEUul1F7Ot1Gf/lCuWrwOfcmCPy34JO8L5 RcNB5UVKzBh6ZOOpfY4QFV8j/TrBQNRzE2kcPConwbn5+YrnJN7EKFjwjtCEIRYCYvXB MQ51x9AwUFq2733EvbMoWFa6QgBEkf29YhSL42Kd/D8SsMPZA5WA3KQJT1NTsqM0hV13 0sng==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=cxbjYNYY; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74351-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74351-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id n21-20020a638f15000000b005d8d56cd646si7781213pgd.130.2024.02.21.01.09.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:09:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74351-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=cxbjYNYY; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74351-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74351-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id A84F3B22CB8 for ; Wed, 21 Feb 2024 09:08:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B61E73F9E0; Wed, 21 Feb 2024 09:06:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="cxbjYNYY"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Sc1xvsMr" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D870E3D3A0 for ; Wed, 21 Feb 2024 09:06:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506379; cv=none; b=A3EJSyW5mC/USX1/Mv3EFRiJSFP61jjgLOSN8FsASmSCosgiB1FvuPSTcJ/JJYTnVslMlc8gCJ2UReF6lDX8i9wEaLo+Qtu35yob7Iy5rAKBMtyhNFciEHOtJ7+bwnjmrCRllPZU5kOR1kGUYHR+myO5AU5vIFqV6hKh5xkb5m4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506379; c=relaxed/simple; bh=fAB5xEIJhUT2xY7i5T7pqKYPBAt3GNmgTzm/jf5bOqc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nPloMWIQcl+0i6pcCp1uSmpLkoV+3m1wRxuK9rgkEQDlSBe0BQxPoArVQBJL0Cc4LHRypgsVEtRASp9kfLJFru9/NFEM22cxoj6f0Ajui/aL9UumljXwUsC4FrbE3bD1ky7XzfHq4YQvHs2MwnAFWgIiQf9xPgj4EJkK0kqqIcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=cxbjYNYY; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Sc1xvsMr; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QisrBEswo4w1Tqs9XJzYOTpU4VJaqEco1Cayy0t85z0=; b=cxbjYNYYvJE7nTIXnHt66nKWJdXP5gSGl1ZHzbW7WkQraHBGShczZFlaWC8cCHv0Nlal1u Fgamecy9zmNMhW2C0uGmgDvM69iP/PZMvYe2bvDUdZ08ArHoNyjsPaehvmFa/0QBvxF6iS xOUURNmNr2dh+yA2+0/MaUVZKJltKwceAY4+JdI1yCxjeo16RZRhbE9tH7KIV/PCW2YvZx sFQoYjOCGp/mMBCgUTlccfwSKqlvkWxIyO6PRn0Ic7SDK26DnHSkLJD2EXet6/LPX7QUZ7 riN0pjbWCiDzxPCOwMrkoQwFfaCn+onpcsvvD4zfpZdl997LzAQlqnAU5HHBXg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QisrBEswo4w1Tqs9XJzYOTpU4VJaqEco1Cayy0t85z0=; b=Sc1xvsMrcCn5y3enYMmEDN21px5+dzmS0F3umaqNg/3PyQcflgyKmwhjWEo13yzGEfeU1K lFRuIkDk5XebubDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 05/20] timers: Introduce add_timer() variants which modify timer flags Date: Wed, 21 Feb 2024 10:05:33 +0100 Message-Id: <20240221090548.36600-6-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498995939382279 X-GMAIL-MSGID: 1791498995939382279 A timer might be used as a pinned timer (using add_timer_on()) and later on as non-pinned timer using add_timer(). When the "NOHZ timer pull at expiry model" is in place, the TIMER_PINNED flag is required to be used whenever a timer needs to expire on a dedicated CPU. Otherwise the flag must not be set if expiration on a dedicated CPU is not required. add_timer_on()'s behavior will be changed during the preparation patches for the "NOHZ timer pull at expiry model" to unconditionally set TIMER_PINNED flag. To be able to clear/ set the flag when queueing a timer, two variants of add_timer() are introduced. This is a preparatory patch and has no functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Commit message reworded as suggested by bigeasy v9: Update documentation to match kernel-doc style (missing brackets after function names) New in v6 --- include/linux/timer.h | 2 ++ kernel/time/timer.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/timer.h b/include/linux/timer.h index f18a2f1eb79e..2be8be6dd317 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -165,6 +165,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires); #define NEXT_TIMER_MAX_DELTA ((1UL << 30) - 1) extern void add_timer(struct timer_list *timer); +extern void add_timer_local(struct timer_list *timer); +extern void add_timer_global(struct timer_list *timer); extern int try_to_del_timer_sync(struct timer_list *timer); extern int timer_delete_sync(struct timer_list *timer); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2f69a485a070..3cf016d6fa59 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1245,6 +1245,40 @@ void add_timer(struct timer_list *timer) } EXPORT_SYMBOL(add_timer); +/** + * add_timer_local() - Start a timer on the local CPU + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is set. + * + * See add_timer() for further details. + */ +void add_timer_local(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags |= TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_local); + +/** + * add_timer_global() - Start a timer without TIMER_PINNED flag set + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is unset. + * + * See add_timer() for further details. + */ +void add_timer_global(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags &= ~TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_global); + /** * add_timer_on - Start a timer on a particular CPU * @timer: The timer to be started From patchwork Wed Feb 21 09:05:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204029 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918729dyc; Wed, 21 Feb 2024 01:08:23 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXa+DgQlq0hnTfmgojbtI3qzVQHHYCr7RsVksjlL+aUJQNEKdsQH7rCJvAQNteq9E2k6lUTXaCmnbrkJzzRAtqhg51bCw== X-Google-Smtp-Source: AGHT+IF8AY3BDWxU/9hzJdJxvXdfJTonwxyF+IFE26ASdXv2bD3B4AAwW0CpLjhFFngHYaTrUNBk X-Received: by 2002:ac8:5bc3:0:b0:42e:2866:6874 with SMTP id b3-20020ac85bc3000000b0042e28666874mr4699850qtb.21.1708506503380; Wed, 21 Feb 2024 01:08:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506503; cv=pass; d=google.com; s=arc-20160816; b=DBLJ4t0vCSzQxVD7iATG66Mk2KfG42OTzekl4K/jB4nY9qdJb4Ol0TATm7ce3F47vu IVVcRDd0IHlOwHlN1Wsbc1v2TY6UQw4dbB0QFPCa20mWoqFfnCxnyfT/g1ha9KWZ08Si jRTBsa2MWwYRxNaT5JbsZITFYyjrCb0KjqXbX4eABf3jfF+vv8YLnYL1qOQzEYOUeRrB ExNAE2TxYWlYVtFJgME9A2Go/mGVkJdqSNO8DZaLjpXd+l4pJCyoQA4N235XYXgqLDa8 prnWbm/CiGtrza1fPd2WOCp9Odxs5xE2WvPuH9Rsagucg1FWxqkTWWJF+Ua+qsua6roW kxQg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=AqA3e23FiOrvTnHWocrTEQRtHmd8ahLc00lmTygo058=; fh=jF6pf1T4w8nC/jaLeIBj6pWxFnEKdE6OeoCUkrV7kR8=; b=QgSXww8FXNV1yUV4NVyBnjd5til802MlA0gYKLlUErGcunUL5TDZ7uOvYoilCOkimg Y4qyrW3v+s7fOkxJHFu+gQd8yPeSVjqPwRDvqfpOuGog30+lsFGB95XpQFxJOJIt7Lmp gEGqx8lhOtUXhSZo/7Y7bEhTc1P//j48rOS/QZYef6AOaN27TyQ/1twyp3Epx2a1jFcn diMf9+chGc1+xUbCa9P28VBM+o7+NUwSWDCy/yBNCWoYoSvVYLWPd5JWL2/YMOXD7AMF /MIknjAp2vcWljvZpVapi/8lJ/QbGvLHr0swFZZg7e9Dap2jRHuLN5XiCEs7ZCPTTXIh Cr/w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=GhQruKgY; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74350-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74350-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 4-20020a05620a04c400b00785c0c8579fsi9781991qks.632.2024.02.21.01.08.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:08:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74350-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=GhQruKgY; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74350-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74350-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 303AB1C21CB9 for ; Wed, 21 Feb 2024 09:08:23 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 661553F9CA; Wed, 21 Feb 2024 09:06:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="GhQruKgY"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="TizbBauo" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D87383D3A1 for ; Wed, 21 Feb 2024 09:06:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506379; cv=none; b=RE/ixYo7oQAxGCfhFirN9Egq0ylrRrZS8aP2uUQBdUhj4aLBboRP15+IVD4jBnoOvMIxCZymuOZ4SFwgkDl/c6GnuqTh+0dJWspmY9hOQOlHgDjnRoc5Jy5+qnjRoBhhxEeThU6nv1/SRuRBHjcF8HcPS8Rs9VEoCZmLRg/Qz48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506379; c=relaxed/simple; bh=6M52TCV6rNFV1WzKYn148scsFyR2YNgc31imWTJffU8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JC1JXQNhxqY821YawZqQOYT7pQmKm8HcLxvLECxzHP73B9MkhMqWUJgdCbY7JjueAImJ+CGVmG2BoA9vwOYRf51Roac0JRtF0NE50ueLHoZEx8GrPHXuYJG+tsd+Zt37wrlOBwb8GKBO9SQnKBFvO/zQ17m9lKyQD2BO64+CuEY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=GhQruKgY; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=TizbBauo; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AqA3e23FiOrvTnHWocrTEQRtHmd8ahLc00lmTygo058=; b=GhQruKgYXp9tDYmM14Y9I+xhqF1Q59YA672dmlhTBZp1CJMj7WDPorb8REzcfUCaVe4RIG iffSQqSqPbOLe9b92l+wEuK1oxzdnuHycskTcOuaJKBeDNQAaihXYbZJsH9FF/OpdSxj7y /lrLZ0VAYfujB5Ig174uKx1IIt1GeSPKOzxymvD5k4NzwEFfZv8MoKzsGGsiXYWeImDcwo Wp6RUCyRNcJnGNXnE8t5kTpAUkoSDFqx/RbbfArHfWAFYw5QBHj4fmPgBzoAVm5+qEXmr6 ripng9czy/cvlTIIhAAVhFD/Ait2na426IiQ3hz2tD+2sBkziRIFjBuPWFb1ng== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AqA3e23FiOrvTnHWocrTEQRtHmd8ahLc00lmTygo058=; b=TizbBauoqo7JHxtaY42PsIlf6HZKEpUo+7SAD3XmsMeQj+oP+89cNahyYm38xfijITDRT6 1uAmPXh7bs63fGDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen , Tejun Heo , Lai Jiangshan Subject: [PATCH v11 06/20] workqueue: Use global variant for add_timer() Date: Wed, 21 Feb 2024 10:05:34 +0100 Message-Id: <20240221090548.36600-7-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498915454560313 X-GMAIL-MSGID: 1791498915454560313 The implementation of the NOHZ pull at expiry model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be dropped. This is required for call sites which use the timer alternately as pinned and not pinned timer like workqueues do. Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker Acked-by: Tejun Heo Cc: Tejun Heo Cc: Lai Jiangshan --- v6: - New patch: As v6 provides unconditially setting TIMER_PINNED flag in add_timer_on() workqueue requires new add_timer_global() variant. --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7b482a26d741..78eaea2e5d72 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1961,7 +1961,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, if (unlikely(cpu != WORK_CPU_UNBOUND)) add_timer_on(timer, cpu); else - add_timer(timer); + add_timer_global(timer); } /** From patchwork Wed Feb 21 09:05:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204031 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918977dyc; Wed, 21 Feb 2024 01:09:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVNAhiJWSr0MEpMXf6K+SkghencROyrs2yPpwgzsYgPOkEWbqPXUkTk9t0TOhmFImuGOldnK6luZoI62r/iEyZnaybISQ== X-Google-Smtp-Source: AGHT+IECxCbJxMO7tLGzTcMuwUc80qqf5oiXabnxxvCPhjy0KFy6r9vYIbhfQOz2FrUGPj+zq1y1 X-Received: by 2002:a17:906:2a97:b0:a38:916e:a4fe with SMTP id l23-20020a1709062a9700b00a38916ea4femr12289322eje.39.1708506542703; Wed, 21 Feb 2024 01:09:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506542; cv=pass; d=google.com; s=arc-20160816; b=zinX9cUIBs3QYBS1peY7ojCvbrOfti0N641QzwThhe19G4NRP69PTnbpwMLxykJBm6 zIZdgPkGoh8MahgKcGvu3qL4onLTokoN6ehdrUp+zF//gH7YpJHlKYaFneSUKf+Z0BBh CSI4kQoOfqMUK+RIApAYG65xNSex2+QREOwB0oIqw6A72nbnOZhJLM1XgC502s2UtXFd ZzaWBI6Y3N1Dz0KVEjfJYdOE/TcZ0DBmqP7bp6nlj9fVCExaum5ey6YeOu1myX5PyB3Z gj4t/EuWylulYVkbFBLqlI9dvjm6LmpmjStItpDpAaLE269hnKu+vfUyonkW0n1Jkjib w88g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=BSKskFBA3hNZnL8AqDErkBhwO140lD1i26cDtauIZEw=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=SejtJUzjmD8UDSCDABrnHOwz9AQZyXfDhyHoPuGtiRKtSKEdwIf4eYrcxebr6CY/Ls euQBUr/YDK0WBDln0uWCJSC8DjeTfyKDEpGPIg99l7XfACLYDiEQI+m246oHwt/a8FUf u5pmehxoWUEjuGXloL/h7qrBG9BXBhEuCe19NqE7uiSHOytqnzNNDVdFozUjbJ0AOI5z ZcgIrcscRjzeB/a37jySukCEtljXr/GtUUj76t+yOzQSVYU5akI5Wt504GMS6qUFER9w cc5hq1uA9hENvDfbgCkJsBaOMO+zLCDLkBLc7vAAKj1VGBN8qNRDXE+nN7zhCDxhupEq BdTA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N+iH1EA+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74352-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74352-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id h17-20020a170906261100b00a3f4f713966si155745ejc.538.2024.02.21.01.09.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:09:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74352-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N+iH1EA+; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74352-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74352-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4C0971F23061 for ; Wed, 21 Feb 2024 09:09:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 73AF93FB3B; Wed, 21 Feb 2024 09:06:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="N+iH1EA+"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="s9SUakEH" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 689143D547 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; cv=none; b=Z2F4luefy8t3l8If0tenMcCrA1X54ewKs7/AKFmFWiaJTvzytCn9lb8nucOGA9Jt2NzXeG9PDtI9vIXOVST3Uflm8jm8fSDL76cIcK5odCcqyzFdWb4WzB9PpGb7MJOdjQbq8OWgzT+VcBX9FKZmiT3UKf+TA8HGI9p7pY0KkEg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; c=relaxed/simple; bh=V1CbKsW0rcYBYg8RiJijs3cSGDAOjzbv3oQTZO9JY7Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NNQwv1mLzutSYh+uwH7KVym7ryAM8vEXn/+AOIl9lWsSOi0vqUMkY4sXInJ8WsfrvunGzIlhBcoBSkVtzBeKlarGaQFcuknqDjTkKmyhTgssEiNfRKJUlNP/iJSgMY2AXQZo7ixVNnAXKw6uAR6dNIS+92Q93YE/ARKloXCU8jY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=N+iH1EA+; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=s9SUakEH; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BSKskFBA3hNZnL8AqDErkBhwO140lD1i26cDtauIZEw=; b=N+iH1EA+Q9CrW9USEa9Lre+aORR0RMKuhVcaHRNSEUiOMsemgtGOAfodGmv1rzBpTfvXgT 6qusX02SD9UMY2rZ4g0bQ3pcedlUjJWAnrF3XB5osCHnOGtkn6CCIPRW782IZdOLbLlhWm eqNfp/mmcACeg3FPdt8VAa8ODhXhyJn4ZEgfnp2m/9mSL7d308dYaUbjcT1RngK13Kkv0G uVtf7UE0TBCL+w+xHSwF0LqPo7Zrn/QRuTMR3oML7qUAKvksGJfLUc9vkG9piZgMhAQeaN s3J9JlGdO8sA1S0dFEh8nSSVVGOGykitdu/jsMkV0IKd/RLJa95EzQyOgS1EJQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BSKskFBA3hNZnL8AqDErkBhwO140lD1i26cDtauIZEw=; b=s9SUakEHFB8KfG5IVhDcJfSAriaMylPMWg2po4rwWWAUBOw3GaG0+PNTrQjlt9gOeIdBKO pi8ytFiJk4eY3VBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 07/20] timers: add_timer_on(): Make sure TIMER_PINNED flag is set Date: Wed, 21 Feb 2024 10:05:35 +0100 Message-Id: <20240221090548.36600-8-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498956841601725 X-GMAIL-MSGID: 1791498956841601725 When adding a timer to the timer wheel using add_timer_on(), it is an implicitly pinned timer. With the timer pull at expiry time model in place, TIMER_PINNED flag is required to make sure timers end up in proper base. Add TIMER_PINNED flag unconditionally when add_timer_on() is executed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3cf016d6fa59..fc4c406c9ec7 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1284,7 +1284,10 @@ EXPORT_SYMBOL(add_timer_global); * @timer: The timer to be started * @cpu: The CPU to start it on * - * Same as add_timer() except that it starts the timer on the given CPU. + * Same as add_timer() except that it starts the timer on the given CPU and + * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in + * the next round, add_timer_global() should be used instead as it unsets + * the TIMER_PINNED flag. * * See add_timer() for further details. */ @@ -1298,6 +1301,9 @@ void add_timer_on(struct timer_list *timer, int cpu) if (WARN_ON_ONCE(timer_pending(timer))) return; + /* Make sure timer flags have TIMER_PINNED flag set */ + timer->flags |= TIMER_PINNED; + new_base = get_timer_cpu_base(timer->flags, cpu); /* From patchwork Wed Feb 21 09:05:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204030 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp918974dyc; Wed, 21 Feb 2024 01:09:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV7UpFBVyG2GqLLpQZzC474IWottEymME7n1ARaqgDhhdMTP5eMoMxJR4kUG88CJu/3ZcOVJ3l13uxzRcT1ErdPr+S+qA== X-Google-Smtp-Source: AGHT+IFULAWI9YxD5VfJUS51bKbbMXD8LK3vm3BksE5cPxtsLxaq2Bu0pixtLMrJMh8YohKAU8es X-Received: by 2002:a05:6870:7818:b0:21f:1679:d6a4 with SMTP id hb24-20020a056870781800b0021f1679d6a4mr4618604oab.30.1708506542472; Wed, 21 Feb 2024 01:09:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506542; cv=pass; d=google.com; s=arc-20160816; b=oMaHgR16MvrFpDwo4j6PB1Q5auGLxCGBnrJQGtl9LyBujddg/Bos9xQkueBU6NlU4O IttSDCow6iULTxCawDfKC0VQYEkOfzRsdnLJDV15leNF69ac9bZffDp9MhUYEbShfQ35 Z6p56HEpKVP9BboMNaYmD9ey9ARRZbJL0rX/QY+1OrOsNskfh5ZEhEk8R5pUWy/zoJM4 Fp0nBaC8v2NOPM6i+rypW/tJzVNlD7+TFJfARAlBz8KxI5tb3ns86eRdul2n8VwVKSYj qiZh8lvGefGRcLi3UDxn1fVOJD/UbavzFDggEeshBRWqICUSZqcYKDsvGHWRTKT4qk5t InWQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=q7jQeTSpBWpkiINXH6oUI3YXYRcVo3N/p00lgUgbfOV1i5DMg3cYXYPmIS7xj2Q178 J2pqO8Vu89zqyH9mW0LYyCkQnV3Aq8z0olFE3jFblVYgBgkw+g0ClCltAFcIf2NZexiO ZmOAzWMjgLxT8EFXVz8IzDwAi/Ii3qgTYOFeXwnE1Az5btgGqJ7M/MRJPE7eF5fNunMS SQgMQK4+XOAfILrmUslNUfBCKSs07l5u5f584JApPtIuiw3seB6gB8wRfploUgAnLEVz mB8rwKFIatgqyJKBcvJO4fv0YYLDiqEgUcOo9ty7TdeRGbaNPNfCUDyrNnf+LlG+O1c2 NWpA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=HvRZ57j6; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74353-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74353-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id k6-20020a635a46000000b005dbee5249fcsi7792389pgm.688.2024.02.21.01.09.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:09:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74353-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=HvRZ57j6; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74353-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74353-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2DA83285069 for ; Wed, 21 Feb 2024 09:09:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 79F913FB3D; Wed, 21 Feb 2024 09:06:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="HvRZ57j6"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="pE7xSGt4" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 689733D548 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; cv=none; b=fZ2uX6U+nPk8Z1pB+ZeGzYjJNtlm4oq/RbIwvegTJQn2bitKvQTAa/g3qid5pudxYbYaLTBkXBDkXOitR5paqSYOgy281D0Wxipm26GM4zg75ChYzeIpK53WbCzflSdzxg/MnNq2gUEsAU4trxpBIJKlPev6CCAgB87gtt1FOFc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; c=relaxed/simple; bh=fMnUip/MpnSW36woG3gUZ5/i2vpiyMU5OgA7NkdHqf8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IBiTXXBdCNTcXd0uZNK8AMTVUXcBCX1Hb1Bb8OQxXBgU03Wy6TzB5sKo/MFXQIj9HU/bifO0oEkvt4QkQcQY6DCi8B8rOmvLS6vp9LplcSYhIDIAGfqNS0FUAsLXcK6WPaGQdWrO2Nocy0nf13PF1us7ASJYVHbRnqbkUrU3rNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=HvRZ57j6; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=pE7xSGt4; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506371; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; b=HvRZ57j6yC35ZAi87VIvPdSqQPxGKA37Fw5X7GMyMMKu1PfClEnNQrcXGlFTP/Ncd5Ggko WkOe7WnAQEIKhr6iCuAE4crKzZew/srk5b/5yAxWmKWIezrXGPmNElWemBRCRijklkzP0C MxrzAFxL+iyYoivIEGQfXV3ztsm5CghxWCE0mjMNz5AnuJZkUsNFb+Ih4DrIAtcnT7Wcj4 SDZL+9mwyIHjckotLCYp3EcOZ8vWvZZcs5fvkfjWfVjL93vqXWZFK91V6kplXSUy5AP2dZ u3BLvRGgA3X6W+1iBKKSNzjnakCsR/+ggBJNTivzCJYbLinCv54nJ2a/laagGw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506371; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JNIN8XCUDE1sXjDFOsAqNC/m2dPziZPst5V05kZh90Q=; b=pE7xSGt45DLeitGoDdAVEkrpLrx8syoI2M2zBt+QD+vQJM+h6MjCvTUuyVdQJ8Z904MgQQ 2Renbh2v4LfXzFBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 08/20] timers: Ease code in run_local_timers() Date: Wed, 21 Feb 2024 10:05:36 +0100 Message-Id: <20240221090548.36600-9-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791498955901242262 X-GMAIL-MSGID: 1791498955901242262 The logic for raising a softirq the way it is implemented right now, is readable for two timer bases. When increasing numbers of timer bases, code gets harder to read. With the introduction of the timer migration hierarchy, there will be three timer bases. Therefore ease the code. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v5: New patch to decrease patch size of follow up patches --- kernel/time/timer.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index fc4c406c9ec7..793848167852 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2135,16 +2135,14 @@ static void run_local_timers(void) struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Wed Feb 21 09:05:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204033 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919368dyc; Wed, 21 Feb 2024 01:10:03 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVlE6ry17A33H3d1Cvogz4vUX6sosZ4mlpvI3IxNc4tY+TxdfZeXNxjO4oMujNQodY40D/dF5GKiwkgy9pJIRX+fl3o8w== X-Google-Smtp-Source: AGHT+IFNbOekbbtztlFoCpJBmWzzSIxgqNpI9RDCe4yfzm94Hf9/ChnbdQIk8X9KJyPQiypn/Mtb X-Received: by 2002:a05:622a:1747:b0:42d:a88f:1cf6 with SMTP id l7-20020a05622a174700b0042da88f1cf6mr21388514qtk.5.1708506603182; Wed, 21 Feb 2024 01:10:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506603; cv=pass; d=google.com; s=arc-20160816; b=E1uA7sD8pkwNdFMIFSjv67Iu3jysW+zWGeiVgsr9Azd42uKdXKSkpFboaUHLBsTQG3 PkjjgfpHTbuz7h9/cLfs1I5Mut5Y6q23VcJeaw2JAmleOe5jGZPPTxxVQn/VqKfGXwaH 53+goMscO4rlWs7Rvtm7L07esYOcjkkh41TAM7EAxXosx2+Jjhsvpkb+O6UsLt5NUy/H 8nz8UXVexFrXvWEWLFxCY/3xFdDbnd5yyiYHqkdzm0gHhWCWvPB8B44WG5zKALO6+1hv RrlEfj0V1YVcVmB8pfHpl5sGxtezs6DqaMefCKV+0TKNtLWhYHuVQ3kA67jKTeFwAAP/ 9pbA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=tVCrlgF3mrMVRn6VCmTujRoLZ7e8Pe7/0fX1s066Dis=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=wQcBS+HRZVarbstobtG91B99+/4sir4gwfVD4vnpnD2ulVHbDnH1y3zrwCmg7wd68n kvIcul/6LrTDSrqCdDRg2SYWLSNu3brepOBz1JrYZ0CSpBPcMLGqKlbPc9E3VPfou0PA 6viOtxK3h54O4yWlwN1T4McXQq1Fwq0aDBmS3A4vC3yvQ3MqnPTKNhj5ZS2drRvSdZoE slwDzFcxkvjevAbVlC9r05RGNeSlBzdQP5+F12/VVR/tHB6i6p8KP7fyKyMdOUmTidAF j2VYjNCtYiUKO6TaRpHIBVfz0R8HXzRPzrKMjSNE+pMa5e1kJ0M8iqWLZRB8yaJBpWBA v3jg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UauhkFtN; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74354-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74354-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id j13-20020ac85f8d000000b0042e3fe7ff8dsi59211qta.203.2024.02.21.01.10.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:10:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74354-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UauhkFtN; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74354-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74354-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E43181C22F0C for ; Wed, 21 Feb 2024 09:10:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 08E0745014; Wed, 21 Feb 2024 09:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="UauhkFtN"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Xw6B4fp4" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B23503D556 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; cv=none; b=Zo3NATDVZPtJdv8mo6cY4e05W0dZ31wj0JqNGo4R/PKES/Uboa9qsxcS7kYgd1UGm3ANpRYQlVMOR2l3rk4TdsIb0mO+NizE0dmWbtzN2IVPO6TZQ/fbYvPVy5+UQX8o3lCRaNCmj8Jxlgc+wEMBY6q/i3sZCACMDKh6GD8acqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; c=relaxed/simple; bh=qUk4S9h8Q0/zI9/a+pkUeXQEJ/CkK2JDvQly8qMxVXw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qqfC7OusqtyObnBKZLGpQ1snxXqK0SjrF1m8VntcNagCf4sqfIBv+nk8NWMIzjmWcEyYb/+H6t7EFrCCiIGNw+UOhkGN5Tc3i1uW+V5MPjTi+cR+zC2KYvdZnqow21niI+Y0Eq+tXSxy5PrQDzWQd7lQyZC+PwTVpZ6U5uS1TZY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=UauhkFtN; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Xw6B4fp4; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506372; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tVCrlgF3mrMVRn6VCmTujRoLZ7e8Pe7/0fX1s066Dis=; b=UauhkFtNugJUVAcU2XBXqHgJpEFFzmmYJbalN9Bx5r6lqOQF5mi+ok6kGZQRczwcuMp45o TdYGS2AuQqd+26Yqo7OEgLOgB32vjtdHAK9XmFr8x8yFFuRfkq2IVI84ZuwF3gBZGmuo6Y jkEXZIqwsEHiU5NaQjndBffrWrOcTglHHVvXCGP0VRR+eOPFd3tvDyW8TVo6VOIaieqUrt KNBAyPpuaUtkdwoTd5ClwibvvsxmGq7pgYcF1HJ+oM6CForFA9X5u9gZywu+ykjpXXECis dgIfI6+l1BZ3DJjISCpOseksYX68Vfij+nZlSl/FbIm68tBkQey57PcXddhWwg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506372; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tVCrlgF3mrMVRn6VCmTujRoLZ7e8Pe7/0fX1s066Dis=; b=Xw6B4fp4kN3Srl3jj/CMgV47Uasf86D4P80s5n23HE3qzLv2k0wZP+p/jaKVkkGBIuH1Dn itPtcKAGxKRghNCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 09/20] timers: Split next timer interrupt logic Date: Wed, 21 Feb 2024 10:05:37 +0100 Message-Id: <20240221090548.36600-10-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499019927847242 X-GMAIL-MSGID: 1791499019927847242 Split the logic for getting next timer interrupt (no matter of recalculated or already stored in base->next_expiry) into a separate function named next_timer_interrupt(). Make it available to local call sites only. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Reword commit message v9: Adapt to the fix for empty timer bases. --- kernel/time/timer.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 793848167852..4d6cf49a2fd1 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1951,12 +1951,29 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } +static unsigned long next_timer_interrupt(struct timer_base *base, + unsigned long basej) +{ + if (base->next_expiry_recalc) + next_expiry_recalc(base); + + /* + * Move next_expiry for the empty base into the future to prevent an + * unnecessary raise of the timer softirq when the next_expiry value + * will be reached even if there is no timer pending. + */ + if (!base->timers_pending) + base->next_expiry = basej + NEXT_TIMER_MAX_DELTA; + + return base->next_expiry; +} + static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA; u64 expires = KTIME_MAX; + unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1969,24 +1986,13 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, } raw_spin_lock(&base->lock); - if (base->next_expiry_recalc) - next_expiry_recalc(base); + nextevt = next_timer_interrupt(base, basej); if (base->timers_pending) { - nextevt = base->next_expiry; - /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - } else { - /* - * Move next_expiry for the empty base into the future to - * prevent a unnecessary raise of the timer softirq when the - * next_expiry value will be reached even if there is no timer - * pending. - */ - base->next_expiry = nextevt; } /* From patchwork Wed Feb 21 09:05:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204041 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919957dyc; Wed, 21 Feb 2024 01:11:32 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW9ZijnkwiypCmOhdg7L50H8HwB1YH2gcfMTICPMCtQbitDazo60V2e+ZOkCmz0ZINO72aoCyiPEacPHGs7ivDO/VQnLg== X-Google-Smtp-Source: AGHT+IEoQ4kaZ0aM+NbdbfUzX2DMcLQqD7pFiMxE0kYr+JZ6DZrgyJFmg7QBQF1m5jk19yFHOj6K X-Received: by 2002:a05:6a00:4fd1:b0:6e0:f6ed:cf19 with SMTP id le17-20020a056a004fd100b006e0f6edcf19mr18567232pfb.18.1708506692685; Wed, 21 Feb 2024 01:11:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506692; cv=pass; d=google.com; s=arc-20160816; b=Je3nC0klkO+vzXp2wYD7uxVnUrWnPWlH8nquFwZ8wn9wcedRX+ELUhK+0BizAPhKmg BsbokE5IqjwhGxLmNXfgmC4dhhIOpbQpn9wgn4RQ5GaeVzVM/X21zLw5Ulu+THCdmxtY Uo6bQC/Ix2riXCLieYoG1IVKUDDPRWg2l6yLxx9djyxZryxsew7XfqY54pqN8utXkdRm M+Z/4Hf9P4veMqku5vuA6plGCDNN9yBYvyFIDN/KnDeQq9lVdsB9QyTNJVpQlUPkUsXv M1/Ozbh/JB/JEW2pebGMGYSBeKs207Uu6E/NGl4qGd6ppFXHZcNqgoLUtWhDPGn1zCJ/ 2sSw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; fh=UyYALGSm//Y7venVD8hN44UibCXnYUm1mVNGtcl9feo=; b=kxj95yeNuis/hjyonO7L5m99URKQoq7SOkq9hp6LhoFGys5YPZPzgkaxybgIhKVHrl EI6v1joAd3ETkxyzvbTz2/wYoUzCS+W9l4FqvDCKEuPTwY4KaRL3yz/Bxdr6i9pUw93g Doltgta4okDeyn/tI58aQOyjwvbbq3UCWUdqWeICk16OJcRcayaW64H6I/NwDFen2Hm6 H2j+ZrMCodR4ndTNUeK3fp+pT1hewVSV9ySE98m/7xfc4VOw0xZib8XbInK+s5iJ6wNF bcV0aZdO7woYpxh9CcxkSbGq2PZSSmEnb/HGMeDjepsfq4P+Lz3YATcIA78MZTv+/REs G9pA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=bhJCVGIR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74355-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74355-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id 71-20020a63004a000000b005cd8b5dfe9csi8003459pga.491.2024.02.21.01.11.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74355-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=bhJCVGIR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74355-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74355-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id C07DB286A9C for ; Wed, 21 Feb 2024 09:10:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 56B9B46549; Wed, 21 Feb 2024 09:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="bhJCVGIR"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="vzezfTvZ" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B23173D555 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; cv=none; b=gFk546+GjgDhdkBqnRqhm17ovgf1AUJASbcngun9wuirXY1xjxlgWFTo8tR4p6xcON70+c7GYVxsVyqaD+3935bTPkarIMsZH0WQNwmJLHF8hfti9riE7XDqmQw5t0v7AT981276Jo6OxA0QfYbTy4hPNdvlTOSR+yoPdVJuBVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506380; c=relaxed/simple; bh=NKQqux4Wr2JtrfLkCkoLP8yZaKcp+XfdbugCAa+8pIE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Fgi1smnfAixYA65yMI0EFvOeHkDm8h8MPD46j6vxVXeSOzSdnKBGjEWB72KctKngVIhCpueH64I74qms00e63b32iijYYG5iL9xao/AaMEC6i1xSJs1H+Q0oD0puL0VJo7Pygp7fVyHrJZTIGm5t2gCnKshVmcaC6F9S95ryLgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=bhJCVGIR; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=vzezfTvZ; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506372; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; b=bhJCVGIRnZW5nAb+MFHLSIXlvn8q2m7FJaOZLtBWZMNfNwFPIiYDIJKJqTKXk6mSGiiG2i 9JWochTRL81HVTQQ1BZLtr0RVgG8kk2D/pJx5eWjFMMChce0FXBxYcpBbge1KxVqAhYif2 k/0qGm1eGbmph+yFsfz10pk3nf9RTbLdfbLp6CTGdTGizr720SnFShAg3nyqa2kIlStZGf ahYQkRnuKsyOuS6vo44TMUQ3P++puSNRVhedU5mV0Z7n7NoorcZOlvkY2KSASgT7DN8PCr F9WogggMSm8/1Uvzqsr3qP8IAssQL7p+WlqL21OR4W9+mrsynu9Rtlc4r86kZA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506372; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RQOeVJJgEWbHRW0ZjUOmxv5PwcZZzcUZxTpzkcXB+IU=; b=vzezfTvZ41R9PIB5ATY7KiiCPyXomXAqaa8iw9Ah00ue7GuKChysnRLHA/f++2b8iavEOB 9RyGtvtQ9fkUSEAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v11 10/20] timers: Keep the pinned timers separate from the others Date: Wed, 21 Feb 2024 10:05:38 +0100 Message-Id: <20240221090548.36600-11-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499114158142801 X-GMAIL-MSGID: 1791499114158142801 Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: - Simplify local_first check in __get_next_timer_interrupt() due to updated next_expiry values of empty timer bases v9: - Update was required (change of preceding patches) v6: - Drop set TIMER_PINNED flag in add_timer_on() and drop related warning. add_timer_on() fix is splitted into a separate patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker v5: - Add WARN_ONCE() in add_timer_on() - Decrease patch size by splitting into three patches (this patch and the two before) v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 85 +++++++++++++++++++++++++++++---------------- 1 file changed, 56 insertions(+), 29 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4d6cf49a2fd1..5ca831444954 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1961,6 +1973,9 @@ static unsigned long next_timer_interrupt(struct timer_base *base, * Move next_expiry for the empty base into the future to prevent an * unnecessary raise of the timer softirq when the next_expiry value * will be reached even if there is no timer pending. + * + * This update is also required to make timer_base::next_expiry values + * easy comparable to find out which base holds the first pending timer. */ if (!base->timers_pending) base->next_expiry = basej + NEXT_TIMER_MAX_DELTA; @@ -1971,9 +1986,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt, nextevt_local, nextevt_global; + struct timer_base *base_local, *base_global; u64 expires = KTIME_MAX; - unsigned long nextevt; + bool local_first; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1985,10 +2001,20 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, return expires; } - raw_spin_lock(&base->lock); - nextevt = next_timer_interrupt(base, basej); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - if (base->timers_pending) { + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt_local = next_timer_interrupt(base_local, basej); + nextevt_global = next_timer_interrupt(base_global, basej); + + local_first = time_before_eq(nextevt_local, nextevt_global); + + nextevt = local_first ? nextevt_local : nextevt_global; + + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; @@ -1999,31 +2025,31 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * We have a fresh next event. Check whether we can forward the * base. */ - __forward_timer_base(base, basej); + __forward_timer_base(base_local, basej); + __forward_timer_base(base_global, basej); /* * Set base->is_idle only when caller is timer_base_try_to_set_idle() */ if (idle) { /* - * Base is idle if the next event is more than a tick away. + * Bases are idle if the next event is more than a tick away. * * If the base is marked idle then any timer add operation must * forward the base clk itself to keep granularity small. This - * idle logic is only maintained for the BASE_STD base, - * deferrable timers may still see large granularity skew (by - * design). + * idle logic is only maintained for the BASE_LOCAL and + * BASE_GLOBAL base, deferrable timers may still see large + * granularity skew (by design). */ - if (!base->is_idle) { - if (time_after(nextevt, basej + 1)) { - base->is_idle = true; - trace_timer_base_idle(true, base->cpu); - } + if (!base_local->is_idle && time_after(nextevt, basej + 1)) { + base_local->is_idle = base_global->is_idle = true; + trace_timer_base_idle(true, base_local->cpu); } - *idle = base->is_idle; + *idle = base_local->is_idle; } - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return cmp_next_hrtimer_event(basem, expires); } @@ -2067,15 +2093,14 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(false, smp_processor_id()); } #endif @@ -2126,11 +2151,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -2138,7 +2165,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); From patchwork Wed Feb 21 09:05:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204046 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp921032dyc; Wed, 21 Feb 2024 01:14:10 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXVXs67wSNiT99VPh6FlqxfnRPI+F9FicTMbJLVwAVKKGh2QzFp+8qbjC+5fZ2mHF56VU4+XBbrlLx/Krm/Nm7JVQIILg== X-Google-Smtp-Source: AGHT+IF4/OEf4QwKakhf9XRPeQYHIW7blOHhky49lgaAT5IJg6/oooKATRe4MWeelihppRFQ+YxQ X-Received: by 2002:a17:903:486:b0:1db:a2e7:1b94 with SMTP id jj6-20020a170903048600b001dba2e71b94mr13672590plb.19.1708506850001; Wed, 21 Feb 2024 01:14:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506849; cv=pass; d=google.com; s=arc-20160816; b=S1TFHoBWm5TSzhXnaTT00Z4fYXGVrOxQpDCguyBhj+vnqdB8xEmbNG/24E3tD0ZqIF qQyqaxHgu1lvuYWV2ymOW0Y4lzio5dFGlTq1aIHdDLcLY/3cQc7G2nB9rhyadojwssOd vbqwpLYVZXxtsZCO/vk3EfUOFVxBClF9HwmR6JvQQvItll5b5lUUH4vL1rY9ej6r8QBC MacYoqeKIw755yhigUzPOcqisq+383c7ZXFErgpmJNTk96KHvz+zcpB6Urh2G5A4L/bt NoQ8Wui52+iW3GH7EOm0SPx/L46ULSaFFO5Hoz46uweRkU+5ayLhrbp+vqUx4mpaDKnM 0IXg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=0rdV5ipuCK7k881Y4gsklU0XL1atEsQC6N/1j4oq3gU=; fh=UyYALGSm//Y7venVD8hN44UibCXnYUm1mVNGtcl9feo=; b=phMvVaLjwlq886+deR1DyG/iW4xODjALBn1OUFp7DIw5NUQcQ5EVqPJtWk5Lesa9cC HOHjc5Z0nCMzP+mxyBtV+zERjqxAXATMl39OLmpcNiQCB8uR0+Bv3v5/guIaSzR1FZ4i cQUnjhQ08snkPC7PCdiwYL3Bdg1366Z/yj0YJjXa1xVXh55yxEcXwhCQwH1l65YZEnR2 tcurkNtvWXfq/PwABvaVpuwpOGWkjkBQmp6E9GXfuPggeAjg5j0AkO0QYRd+5UmEVHGC 4qcXgOaCzBU5JefkjrnKqNmaBBEV7+vzpvQLndQRNiUonj2czSy+PfLpTZYqYwNVQfOl 2tEQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=erx+rjUr; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74359-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74359-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id x10-20020a170902ec8a00b001db40e91bf8si7814557plg.249.2024.02.21.01.14.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:14:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74359-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=erx+rjUr; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74359-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74359-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 5DE11B21548 for ; Wed, 21 Feb 2024 09:10:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EB40446441; Wed, 21 Feb 2024 09:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="erx+rjUr"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ALe/TX7h" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29F9E3D570 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; cv=none; b=j/jFcqXtP24xwM9p9WR3TSGQeBJHlB/IBmLwHamsGtSdle6+OKFQPBCWjc0N3oVjrDBwDyCfw8RNsdjsPrMRJWZKcj+TE5niGZswmQaouUeYC6MQRfORmIXLOS3IeDYmzckrvP7GwQJcH5FUplmD+Y8mttYHlB7K0BBwsKFKO1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; c=relaxed/simple; bh=+PFtgywkqe7EU2O4wsgXtoFK8yKu231t3eYiycwv0as=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l2ggfdbk5fAl+h3Qw58r8ScCzH/SCpMdYqLiqxoeSSpvBE5wReMoUASKrF7I+1BRTyrXQXfeASM5Zvigr/pQ2Zx5R/UzZEp1rfKXFiLFxu0bho//Mmv63RpyZDKo/f3pu/1PLSGtpguGg9waG6tFVkFItWG4abfxbQ08s9LR3y8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=erx+rjUr; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ALe/TX7h; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506373; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rdV5ipuCK7k881Y4gsklU0XL1atEsQC6N/1j4oq3gU=; b=erx+rjUrJdkST1kCUsNdV3qpWn4r06LFiSqvkL7kmB4JHRJ9LiuzUpm9g63+X8yOzssUjl sk3O9UMZK+QH6OrOfToxhnMwwKJv1ABH8CinEpiw5cqz+SH+qjq3cEKUIa8C8BNxiXl3LV wZI8VvXSqNZ10X9/mTy02OFuQDVje6Diz8yLYnTqbyTr+wNtM/EO2CxMa/5At/82+G7Rq+ Tha+7iNgOUlyRPFqLDHPzjo2Xpy6f5jkAiVFTAjNPNKkt+B6MBN+ofr4agIfeSP5ongHnE vVuefPq1ocPlR932unA/U6qIYuFxxbCeHZyDVw/cufC81ligWvamgylWIbkKgg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506373; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rdV5ipuCK7k881Y4gsklU0XL1atEsQC6N/1j4oq3gU=; b=ALe/TX7h7OMA4pyk46Rv7ZfsqFG3JzoYB32BJWYlgCRvXZET1YkekoXmmuFdcUshUynjrc 9kP8mFmyhR+x/VCw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v11 11/20] timers: Retrieve next expiry of pinned/non-pinned timers separately Date: Wed, 21 Feb 2024 10:05:39 +0100 Message-Id: <20240221090548.36600-12-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499278745587550 X-GMAIL-MSGID: 1791499278745587550 For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have separate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Fix no functional change message v9: Update was required (change of preceding patches) --- kernel/time/timer.c | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 5ca831444954..f119b44e44e0 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,6 +221,11 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -1986,10 +1991,11 @@ static unsigned long next_timer_interrupt(struct timer_base *base, static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; - u64 expires = KTIME_MAX; bool local_first; + u64 expires; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1998,7 +2004,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, if (cpu_is_offline(smp_processor_id())) { if (idle) *idle = true; - return expires; + return tevt.local; } base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); @@ -2014,13 +2020,32 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, nextevt = local_first ? nextevt_local : nextevt_global; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the @nextevt is at max. one tick away, use @nextevt and store + * it in the local expiry value. The next global event is irrelevant in + * this case and can be left as KTIME_MAX. + */ + if (time_before_eq(nextevt, basej + 1)) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; + goto forward; } + /* + * Update tevt.* values: + * + * If the local queue expires first, then the global event can be + * ignored. If the global queue is empty, nothing to do either. + */ + if (!local_first && base_global->timers_pending) + tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + +forward: /* * We have a fresh next event. Check whether we can forward the * base. @@ -2051,6 +2076,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); + expires = min_t(u64, tevt.local, tevt.global); + return cmp_next_hrtimer_event(basem, expires); } From patchwork Wed Feb 21 09:05:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204034 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919553dyc; Wed, 21 Feb 2024 01:10:32 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVb2Q1eD3jIkxTZ8vVPuSh6x9kNpF7sWz01RGSUOlqZw2T2+7HjbMb3wKDh3okUre5WVCmiif7GG2r953TyZ5VI6gGiOQ== X-Google-Smtp-Source: AGHT+IGqOhPNbQStEP7rWrwp8wion9VM0CD9CBJI8e7HkBHypJDOamN9+MNco1wUBTtfBN9fhYdp X-Received: by 2002:a05:6402:5518:b0:561:ee53:504f with SMTP id fi24-20020a056402551800b00561ee53504fmr20034076edb.5.1708506632162; Wed, 21 Feb 2024 01:10:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506632; cv=pass; d=google.com; s=arc-20160816; b=BLJWQgtjLsfOUfpVHFQC7f6HXRTQRH4jGkwVkGKS/gI8ORfCxQ816/ry02cAgkctcg aBp38dl90HyjrPSoESu9dIzxXKq5qVIsfV+ZzTPFWj2u2a1pVFLOFDxd/TZDsoweJb4M TCtBdHbhpLZnVSzk3P+FaTOMqhuYGuWT7ur+MJod4mJK94OcPDOufu4t3F3JSigx2Vxh XPPIJFlWh/DfdYwZRoRfbWwlCQ2mUnTK+051iC9MTMfCujwxKto+z5ZTf3676vhZD0i5 FjW6Q0H0YLmxoz0b0fkJX7c1h90hnkqPWv5nVcZ8ZkoGAxDEZPBe2F6xnToENcACJEav 8UxQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=1b+XJv886jQpeQSImvPTzFbojcDlDb9MA0gl6E6c3yw=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=R4OvZfveO282oydonrKjvpSyRE4/1VXNgTd9cQAMAUUlvJV9eaCKUwMYVHNJj01WB+ 7Phz1IWu6TN2HS2Od6nZXCMYoKpFw5VoyL6XNXVP5aBbByrZeV9IPMdadhZbdsOOugRP pU8VokkLUFQlgH1UOMXj/8s7HCQRj+vINEl0U+Mp4iYiydK81hj1Wh60/fIV0bg3uS7F LIXGn6eYsa3osWiaX+ZxAMmb0nHOt6zdAHE62DMHwcK5+JHzCDcoYzRW4uliU0LTXCbg 5X7zOcgSouOObdSN17jq+kNvJ34yEaT+/0W0wROL4bWlWNXTQjQAKmvv4NMjvS6xFKuQ 3hgw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="u/sfdH+V"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74356-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74356-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id t28-20020a508d5c000000b00564511aa945si3164645edt.266.2024.02.21.01.10.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:10:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74356-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="u/sfdH+V"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74356-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74356-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 9885D1F2343C for ; Wed, 21 Feb 2024 09:10:31 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C293C46420; Wed, 21 Feb 2024 09:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="u/sfdH+V"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="XuF4oFDy" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29FE23D571 for ; Wed, 21 Feb 2024 09:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; cv=none; b=DvHS4KkkQGOQk9DySHMpkZPaU5ilabeKSQ7v2HReqyMnt49OQ0qsVpm7Loqi2UXHucrh5Eeckjdc9lHZaTRdoTDCvwl/xWFt6LjrwDfkBmKzk0CAPstGnsOQ7HiVtR5+s3KQvPbOqJArbw72UbEfmln2Z/wdVLkfEy5YxBVrHFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; c=relaxed/simple; bh=8GAi+clf4ARudGxl8gQObKgEbjDfootfzyMlgIma4rc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a9VrJItPEUE3KtO3MbcvcnDQCyIgBUQverrB/H3Oh1B47dCxXD1pL3qDspTVKRmU26p+hvs2VhUmxoGmGloeEpSJZ8NGHAgLLbAeXSocYhlI5mWkcphAdCcNPs9M80F6YRmaqJcgOWSum9swo+XEBfaPOWEACvctc+pZ7Nhk7YE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=u/sfdH+V; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=XuF4oFDy; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506373; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1b+XJv886jQpeQSImvPTzFbojcDlDb9MA0gl6E6c3yw=; b=u/sfdH+Vk7kDr9sz2nE4bHm9dCTIDU49w4pS88q+ts5uf4fH8H8L9lFaZKNsBGTLXHGi4h hybiBIpDkjFGsxd4qda7fWnmJI+X+iXI57e8SZuQLSi3CmfUOOG3YVFR4fLuuPnY/+NS5t gQRuFX6tmxSkPo7r3+mNyY2jYx++aL4c0HR3fMRakftpGMhYBEyDtgNXkNOv7r6zIY2yw6 OzpFoq8EkguASNq+nF9MCLb99IVzQGss52yziACJ2/PBf2KFR+Ofipqo0NdkkdezWFen/Y avz7QIbHo0Cql5QN2efQi+T78JUUfSGIofpmOE6QXb0D0KqLoXFGEc2Q3O91nA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506373; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1b+XJv886jQpeQSImvPTzFbojcDlDb9MA0gl6E6c3yw=; b=XuF4oFDy234l2NaSKHRP6EJO2dzYvZlOG2UwL/ppCrMgrcelmBfVunKnh0A+OOesZY0xda UM3Qf53oKdQpfkBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 12/20] timers: Split out "get next timer interrupt" functionality Date: Wed, 21 Feb 2024 10:05:40 +0100 Message-Id: <20240221090548.36600-13-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499050186804880 X-GMAIL-MSGID: 1791499050186804880 The functionality for getting the next timer interrupt in get_next_timer_interrupt() is split into a separate function fetch_next_timer_interrupt() to be usable by other call sites. This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Update was required (change of preceding patches) v9: Update was required (change of preceding patches) v6: s/splitted/split v5: Update commit message v4: Fix typo in comment --- kernel/time/timer.c | 64 +++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index f119b44e44e0..9fa759dd80f5 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1988,30 +1988,13 @@ static unsigned long next_timer_interrupt(struct timer_base *base, return base->next_expiry; } -static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, - bool *idle) +static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_base *base_local, + struct timer_base *base_global, + struct timer_events *tevt) { - struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; - struct timer_base *base_local, *base_global; bool local_first; - u64 expires; - - /* - * Pretend that there is no timer pending if the cpu is offline. - * Possible pending timers will be migrated later to an active cpu. - */ - if (cpu_is_offline(smp_processor_id())) { - if (idle) - *idle = true; - return tevt.local; - } - - base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - - raw_spin_lock(&base_local->lock); - raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); nextevt_local = next_timer_interrupt(base_local, basej); nextevt_global = next_timer_interrupt(base_global, basej); @@ -2029,8 +2012,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; - goto forward; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + return nextevt; } /* @@ -2040,12 +2023,41 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * ignored. If the global queue is empty, nothing to do either. */ if (!local_first && base_global->timers_pending) - tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; if (base_local->timers_pending) - tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + return nextevt; +} + +static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, + bool *idle) +{ + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; + struct timer_base *base_local, *base_global; + unsigned long nextevt; + u64 expires; + + /* + * Pretend that there is no timer pending if the cpu is offline. + * Possible pending timers will be migrated later to an active cpu. + */ + if (cpu_is_offline(smp_processor_id())) { + if (idle) + *idle = true; + return tevt.local; + } + + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt = fetch_next_timer_interrupt(basej, basem, base_local, + base_global, &tevt); -forward: /* * We have a fresh next event. Check whether we can forward the * base. From patchwork Wed Feb 21 09:05:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204036 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919706dyc; Wed, 21 Feb 2024 01:10:58 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUiA4DX9smBXDb0Xy/jUtrEtM9pja6jK8Ii+6hdU0zNatlGfJaLUpYwjpU2kPNHM8kBzkswRs74wGPM4QyOHoOKSpsL3g== X-Google-Smtp-Source: AGHT+IFq8kpn7m9ilpbe/LyodWiojtR8nE5WRqq6D8Kr7L0G6x5OcMSXdMlMrY0+8+6eYHlQPAzo X-Received: by 2002:ac2:544c:0:b0:511:9e5a:922d with SMTP id d12-20020ac2544c000000b005119e5a922dmr11203431lfn.14.1708506658303; Wed, 21 Feb 2024 01:10:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506658; cv=pass; d=google.com; s=arc-20160816; b=n3mQfKKtTV9nACAqFVHldsxsEMhpH9QOavaA20DgSpO0PJOThyOvPDN3CF1WgzxYiw BJbaYEZPuols6WsPOsdh/MEGXf+2xpPhmd3eMMrtJonxleFiNl31spK4ZKZnyAIOArA6 X2dMZvXqUehwx3g3iDR3Pg3NNz5VWCXS1J1sd49WLdxce4jSJcbfnKk7NuTX05xmrIpy AaRzSwzt1UE7EoUPiJGdUoEMYlWGbbSCUwFvlDvZBeRUgDNhtlt0p8d4nv7RI8o+0Jk0 o+hgH7lekp26uLvR/ednguQ6rPO1Bcz/vAdmvdrvcf48Z0+5WCf3L9Em9AaFYUT9Hh49 Q/Lg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=7M+cLD3z/7BiBVC+tTh5mmcK9KmfYufY+d1NGfjIUd0=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=TN8bT5X4VZOV0mX1jLDgBc+9cIYtiEZICH0gju7OrYPwtgjsfnhGUEpkajimtWRANP xo6Iu71cZ1PGqN+LvJtZ02i1qCAKPatBvBlCmRYi6WAMdUH8ZSFtPBB9ZS1C2gUnc6kN zz2fPwoszeR5QmPYYmxKmV3BCWbixZ16cWgOJM/+5y0chFXy9Na/HHR7nGhcIdL5G+yg vZ5koFHPxtUy5hsnbuS7H3HHPOZaesanCsC909Xy602Ybdg6anZOmC25DetYf0fG43AS HUxyRkgm3FDT02h2DxBURW5JA63KZO1J6B+mobpGNd8n/PwhIa9YOrO/OyKmE8oQ6cgT d4Mg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=nyXedZ4U; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74358-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74358-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id g38-20020a056402322600b005649bb0e7c4si2138067eda.221.2024.02.21.01.10.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:10:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74358-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=nyXedZ4U; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74358-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74358-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id B986F1F23B3D for ; Wed, 21 Feb 2024 09:10:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 95B1A46BA6; Wed, 21 Feb 2024 09:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="nyXedZ4U"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tGyZD5Ps" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B83503D967 for ; Wed, 21 Feb 2024 09:06:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; cv=none; b=ugUN//gixsh+jNJYQp9kFRx+r5x/5vgmd8pDnys8iGXS4uATdssViXO8nGXdVs1RW52PNgL4jR2yHzNDSobIxkZnHVVYeym8cj6uEKHEudr9eOFRsduDdUbwV1ztLOb1qGHPyqKh/EF+hj5hNXvMEaklfQdreiTeqpKJl4PsyZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; c=relaxed/simple; bh=A1ak4c05RsMMc85EHNnnsqJhV5a1qxUJ9DF2MBZohj8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X8n3bo0ZeJKuAhgjynaj07HZuQPaUzUYHkL8FCHPSE5mfDtuXpL1PEDqnw3EHa9o1yUd7JfTqHA+hFspNn7Kbv63Zc02IUG6zWhTI8ku7OLncPEWIRkRVM4pgpzhAaG8m4B2cH0jEhem353twPHBnqZGVKLftZLJFyxwG9FemTY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=nyXedZ4U; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=tGyZD5Ps; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7M+cLD3z/7BiBVC+tTh5mmcK9KmfYufY+d1NGfjIUd0=; b=nyXedZ4ULMKfhslx9q3MBFhD3FvdgJXdxLmS7rdCWmnguxXTcrGasMApsH5yxBaJmDV0bM 4zGkijLbX5rjWdTXQ4xW4TgfEx5XXugDVd82s+HPg9k+p3cQSIjTmcOHYF5cobkjo/CRld uw5A1NYLuXE0km8oXvt0ULuhxY4m12QW4qbTi20kWYS9u4x4nW/kBIK0qWrRqvU21GP2E/ dUVV0zCP6N598KXHNQSsLKWrOD/tvimhPiSMv8Ctl7go9r1a1oncyClCIezMfwdz0OVuFu DfLJUjJ9NUmHQ/V8dMelUO3AKziMWJPwQLVpSQkg2wexHwOA6Ud/JGmIVGrzSg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7M+cLD3z/7BiBVC+tTh5mmcK9KmfYufY+d1NGfjIUd0=; b=tGyZD5Pslo4rbM64ULduxSkn9Zve72LNFv+ut6WRELNpeDBPyG/Mq/S2mEYluwGVPbn8MH /WGMnqEd2EyulpAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 13/20] timers: Add get next timer interrupt functionality for remote CPUs Date: Wed, 21 Feb 2024 10:05:41 +0100 Message-Id: <20240221090548.36600-14-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499077529977926 X-GMAIL-MSGID: 1791499077529977926 To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Locking of the timer bases and getting the information for the next timer interrupt functionality is split into separate functions. This is required to be compliant with lock ordering when the new model is in place. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v11: - Update tevt.global when global is first event to make sure during remote timer handling, this global timer does not get lost v10: - sparse annotations for locks v8: - Update comment v7: - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section - change lock, fetch functions to be unconditional - split out unlock function into a separate function v6: - introduce timer_lock_remote_bases() to fix race --- kernel/time/tick-internal.h | 10 ++++ kernel/time/timer.c | 95 +++++++++++++++++++++++++++++++++++-- 2 files changed, 100 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 47df30b871e4..8b0c28edbd09 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -154,6 +159,11 @@ extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; +extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); +extern void timer_lock_remote_bases(unsigned int cpu); +extern void timer_unlock_remote_bases(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 9fa759dd80f5..88160b3461e0 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,11 +221,6 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); -struct timer_events { - u64 local; - u64 global; -}; - #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -2013,6 +2008,21 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, if (time_before(nextevt, basej)) nextevt = basej; tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + + /* + * This is required for the remote check only but it doesn't + * hurt, when it is done for both call sites: + * + * * The remote callers will only take care of the global timers + * as local timers will be handled by CPU itself. When not + * updating tevt->global with the already missed first global + * timer, it is possible that it will be missed completely. + * + * * The local callers will ignore the tevt->global anyway, when + * nextevt is max. one tick away. + */ + if (!local_first) + tevt->global = tevt->local; return nextevt; } @@ -2031,6 +2041,81 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem, return nextevt; } +# ifdef CONFIG_SMP +/** + * fetch_next_timer_interrupt_remote() - Store next timers into @tevt + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + * + * Caller needs to make sure timer base locks are held (use + * timer_lock_remote_bases() for this purpose). + */ +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_held(&base_local->lock); + lockdep_assert_held(&base_global->lock); + + fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt); +} + +/** + * timer_unlock_remote_bases - unlock timer bases of cpu + * @cpu: Remote CPU + * + * Unlocks the remote timer bases. + */ +void timer_unlock_remote_bases(unsigned int cpu) + __releases(timer_bases[BASE_LOCAL]->lock) + __releases(timer_bases[BASE_GLOBAL]->lock) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); +} + +/** + * timer_lock_remote_bases - lock timer bases of cpu + * @cpu: Remote CPU + * + * Locks the remote timer bases. + */ +void timer_lock_remote_bases(unsigned int cpu) + __acquires(timer_bases[BASE_LOCAL]->lock) + __acquires(timer_bases[BASE_GLOBAL]->lock) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_irqs_disabled(); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); +} +# endif /* CONFIG_SMP */ + static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, bool *idle) { From patchwork Wed Feb 21 09:05:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204038 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919849dyc; Wed, 21 Feb 2024 01:11:18 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWw8orz72PqxeQQuE5TpoLjj/LGoHTPga+CCFav8rf0fvCRTuInm2+dVrNkoflrauIxBgbaxYA55BGPuVsfCEm46uOcCQ== X-Google-Smtp-Source: AGHT+IEzpxfZbEmsWpNb1niy76CiPbBDYF1nk9dzz5osVHnc9Z+EpEURihuDuso7vvNExoCSNRs2 X-Received: by 2002:a05:6808:bc6:b0:3c1:652a:93d3 with SMTP id o6-20020a0568080bc600b003c1652a93d3mr5525148oik.29.1708506678317; Wed, 21 Feb 2024 01:11:18 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506678; cv=pass; d=google.com; s=arc-20160816; b=AcfQU9ISSuNjWmzsXIzj+wqug2ZUPPv9Ez70lJAZKYdV+jJgsDi0cDiYtD6iR9dnx3 6ebDKJI50ISbzFRgPIIJ3AvH9rBydKvjjaR7VR3uzK8Bv4LLsfOLD2FLI6zkoEcyx8Wt sf+jsebLyR9f19AMY7hEgDYKa3NNrPEPXAh9DM8Xb6OYAD851aRaK+OxEaW+KO5D9P3G H3VYVALZDWve4OWstgCONuPLPk1Oy6erQ9cWAZceRwxz8wHc1Ine0ZtAX25ZOI5ZZjjN F0ci/Y4D+FuiXu5lnPf0ZVWw4xJEpbWHVVDvUyticTjZ6vVRNgM4+NkW6OtNTBtG90Cf bbIQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=8Rk56jS4ox3LCyUSnYHvNlLdqZIvRSv1jWR0mBi/KNM=; fh=kuIFmsLavJbIiXGfj+OQqzrKlGRdum47/II5WvdxEO4=; b=Vw99UK17GOWMVjslUKFq8a5E3Bm38rl5hB3cpLER7lOv5saaSXyINQWcup0uhc1bWX ijoH9y8dts6xnBaXBXqZCkOR5kBj+kNgXOtKk6WPU7E+QTLaX+qnyiOn2Q9NzoyFeClT zd9MF8+DcatIaDOqomU7LwZsvu1Q6ShnUq99M6dDBbjCKaBP5bEgtqJI30sxFh1YOIk2 c3DU9DtwSRBFKKA0JY78fTlnlyhN22XreuULe4XfTnjDl8X6i+GuzKPPSrk0ZpST8vbw mL7BCRA411n3N8rhpBsAVD/JyG37y77tGlJgBVbc/Q00s55mdjFKtY2xnVuqeJt+HVpU ZrNg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tvSifHza; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74357-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74357-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id u71-20020a63794a000000b005dc8762dce5si7904089pgc.51.2024.02.21.01.11.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74357-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tvSifHza; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74357-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74357-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2A4032851AE for ; Wed, 21 Feb 2024 09:10:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6EC1B45BE6; Wed, 21 Feb 2024 09:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tvSifHza"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="wtOujEsX" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B83323D965 for ; Wed, 21 Feb 2024 09:06:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; cv=none; b=ns+ZXF1IyDH01ChP2U1t5bdBIQroyvxcKIrPs+AGXeIhtUM1ge5sEa8V6BHDKyIFJzRxSqu9jzDIXFhgbfN6MdpRA0EhrgdWsMUqXfmrzFMjWD/0HPwKE0V/S0cW4tZH9hbczuHaq9jz3JLBSEwI2+xvw/fDzMeN3vB3aO9tvCA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506381; c=relaxed/simple; bh=Y2acepoGiFBus/o+gYKEuBFXzXXX1WvysSEaYzKXOYg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G5BZGaSMCKY2MFUqrZ4xK116RqM3+KzZRmmNqp3872MaPx4GVjCj4npfIWhrga/o8x72RigndSnd3yGKVPL7No+EOb7M/gaSc621wJ4OjFAwxMH8RqsD4KKHYMSVMmrU4Ouyt2zm7xgqUo5D8BysRObIePEaEQz9z791vcAZGOY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=tvSifHza; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=wtOujEsX; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8Rk56jS4ox3LCyUSnYHvNlLdqZIvRSv1jWR0mBi/KNM=; b=tvSifHza9FJpYdPGN14ZKC6V3XR8MQF6d1u8sRX1M3nXQkeBVoGTK3ed45Tp+Wn6Pxl34P 6IeT7hO5vshrkdsYEa11CvDY2TV2OumynAdBsHrz+zGnGhKt5939Qqixkem4dRqkNsoTPI YijYecLSRUiZQ/jzOkiN2Y7VXUeNBznGVaQYdGRZzLjCRSkyMvzFXo1YZTF+Sy+89QbpAf ghnWE/HFPgZJN+LMRiSk7WFx/ajpy7z3E4F4L3QiApGCdFBmTmQ2n/qD8Get8icU8Hgkg8 cgOk0xiWQf1V2On6T0jm+96lB03fPKwNMt8JjwbzQW1sSlWDG8drZ4x0dqlDnA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8Rk56jS4ox3LCyUSnYHvNlLdqZIvRSv1jWR0mBi/KNM=; b=wtOujEsXq56FVsf4pAgUz4UY2mcmXcLUCrvFSKEik1is8Jtqy3EJq05R0YiQNLzYwXRU1a TUIrAQ29Ip7CV2BA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v11 14/20] timers: Restructure internal locking Date: Wed, 21 Feb 2024 10:05:42 +0100 Message-Id: <20240221090548.36600-15-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499098836356220 X-GMAIL-MSGID: 1791499098836356220 From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 88160b3461e0..52af50d00ae6 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2238,11 +2238,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -2266,21 +2262,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Wed Feb 21 09:05:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204040 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919924dyc; Wed, 21 Feb 2024 01:11:27 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUt8AD4By7UVmWLE8EU340hJSLiTwtAsADZpyamOe3MoxizYCC4pvC8ro42y9T1P+Ms618huad2WXst5g0+DIgHQwQETw== X-Google-Smtp-Source: AGHT+IE+wDq0Egl74JUYkqjuWXCxKV2IRUTKJrOu8ZH/FAfMTuYq4POHJ1ulkhsHQSPhDfvPcRNP X-Received: by 2002:a05:6808:d53:b0:3c1:55ba:7d42 with SMTP id w19-20020a0568080d5300b003c155ba7d42mr9896335oik.11.1708506687450; Wed, 21 Feb 2024 01:11:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506687; cv=pass; d=google.com; s=arc-20160816; b=drJL/IyXzt5fL3Hjrc9FaaLFmMkZJd674JsEttI+ZVXZhp+tLPDYscc7zhCGkb91R2 1AnNykE4v8FXBYDngGwF25HEyw4LzbX1jz0/2L75FxKrpJRVzbR2N63bVhjzRApuPcXQ xhAN+Y1h7BoGu0/15LrG2m8Nzj/Aj956Rp4bb+7WR/p5RIlNi2eU/mWedKGSYyVzhS7y iyBwtbWchXYqLhz0rLtNXy+Pd5cx/J2N6d25BLyM6/DOu/TO8GykzEQg5bN7bPDj9W4f 0Ob/h3vWvoiCCy2ogWAbt/nu3Z+JVviHrjWxEdLL04jCYiFqm2dRD4pmfG04fcNf9x0E PNWg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=84mTYhbxH69oUoy6SwRp7poh6N58HyKxHp1SCrdM7O4=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=suDbtvb6ZTQ/BJXEcaJceRS+9Sy36fDzKuvLi3PXuebW15+YrroaOJrPVwzroOVUPX 8wS3rEIRIaHL/ammPqqq6f+psbn84KpmxQeJAl9C3ZweyFtVbFX0ur65cm0q/OsUrvdL HS+4MINIOL6MEaXt5KrM6WoDAevuqGTLwXpuhaQuRNlpqVEQgX8FZ5JKi2d8Eqh6mJEg 7xlrPagFH42vINnqEKkP2scDtahKb64F8Wt+8RfRnEWemAsria9NqUbJRhCQWaZgQEqa GDYX1TzUpF6F4RrAoZFunmpC43mbBgXfWdwppty4ln8BHhhrS9mIp8Y0dHsYFK0Bmytw NPtA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ZKiwyV2z; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74361-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74361-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id t14-20020a056a0021ce00b006e05de14bd2si4660609pfj.318.2024.02.21.01.11.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74361-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ZKiwyV2z; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74361-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74361-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 114F52868FD for ; Wed, 21 Feb 2024 09:10:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 439D146542; Wed, 21 Feb 2024 09:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ZKiwyV2z"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="eIB2QWWz" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E65903E468 for ; Wed, 21 Feb 2024 09:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; cv=none; b=GmSCiox3g8nc0OnBrYNi6sOBNHb5WoH0oo8CZ96D1EIUsZhn096hCJW3Ia3sGVcW+iUDjWaMRvZLVY3LveI5MvG2j50vS78DBQUYfFpPl0rgFolcxYGGfB0Qk4RMU8SbZpRNJTaiYFBUIsldQV7CyKgR9fnbBcbY7fW7PXSecYE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; c=relaxed/simple; bh=Ud9pgOn2B/0m50aOpIF31SSVvpgQRaOUxbGunhq51UI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pGMoyrZOuJddnwOWCFilyaEROf5iqNNiGYFJmp2bbBiR9uf85kj42iynxSLRBQyxYOxUnC5+0vs0C0CGsIo1A7cqyrj5bBEtQnBZtBNtmCipXK2BiDdhExlyWLavWp1oYaY1A6/Usa58EjaYtoq1qtscYbuwvPPBchhI86nyFJg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ZKiwyV2z; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=eIB2QWWz; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=84mTYhbxH69oUoy6SwRp7poh6N58HyKxHp1SCrdM7O4=; b=ZKiwyV2zkZ+h+SdEAzRyGq5O9LN9CHu1eeIzWsHowWJu6xsGLtpPL44lQ4YBsCFe8D8cwy zrenZpCWFkeb9u4ZxmK2v+tJNJ9J2qmrOY5jyCHgWcf8zBZGh+nsPVcERkERUy+TWpWT/m YQruHpAStbSFmkVs8rkOd3Rs7KQF6LOSmpubfL2ukplL/lVvaKcd8HoceXpy3Pujk9nKcC udmHmsSrpVB7uNBPmNXUEk/W9YoiMguuEdhDkPXqC8Q2R4NoARbheI7LEQC5179OikwtqY RWWy6jTQuiqsmpNFvQBxkB3NJA4KOyhSXnWIVgvYwQM2LfjyhiKaVPvBBQkIZw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=84mTYhbxH69oUoy6SwRp7poh6N58HyKxHp1SCrdM7O4=; b=eIB2QWWzcZQPTgaK4YkWqHopYbdU70bbtzBJsBK1v6l+WFLnw8dlk2GujZTaEKWDnqOHqp RC7nGW6K89ngf5AQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 15/20] timers: Check if timers base is handled already Date: Wed, 21 Feb 2024 10:05:43 +0100 Message-Id: <20240221090548.36600-16-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499107831881095 X-GMAIL-MSGID: 1791499107831881095 Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local CPU, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: s/cpu/CPU/ in commit message v6: Drop double negation --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 52af50d00ae6..7b9f9ed25fc2 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2240,6 +2240,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Wed Feb 21 09:05:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204035 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919688dyc; Wed, 21 Feb 2024 01:10:53 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU2T4qz81vlHl5dDQ0JsVOAXUvFjaHbR9Lgg9aDcHiFFdsGaf6s7p6+xWPdJ1TaArSBg5YdZxVjy2wK27yhU69xs51CNQ== X-Google-Smtp-Source: AGHT+IGb6suWu6uHfgSc6hvlmuSsKHCFtODGwqoALNmfTbAIkWrefwD5EhoZvJdBNJpNnWeWKmx9 X-Received: by 2002:a17:906:751:b0:a3f:5144:ada2 with SMTP id z17-20020a170906075100b00a3f5144ada2mr361293ejb.2.1708506652998; Wed, 21 Feb 2024 01:10:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506652; cv=pass; d=google.com; s=arc-20160816; b=eA6uLUwZLqsOL94VP12jRhKs/zrZoMFuSsZBzvzNOBHkpiO+hcLbSaUKn89lxAXpH9 TuH8hipoHO5TnKwRAHFJ6NHs4KszgKkmS1j62uu5DXUuzwd6JXhz5mMlvyKPAHPSCk7S ormjmtOMdQHi57Hc1sKvR1xFQEOdmL5yfEBN4VcjrILoj8MfkOAm7cVv173bqdam3bCC 9STiHi8JknvfherIBLpNAyF0e4Zx8/aza7LPA9knsXmAhzX421VaEs6aaSwiayuIDqKI 61pW2okDbcVczyiTr3uLcyswu27XL6YCshkaxK4TowGqy14JwtW736U3J8hnZNg4DBR+ Cijg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=be6MfeAvNPKcsJz3AUGh7ZqxvewZ3xXmvk3rMk/0KI0=; fh=kuIFmsLavJbIiXGfj+OQqzrKlGRdum47/II5WvdxEO4=; b=QAujv3GUkXS0l184zq1fA84x6mfIP2DCwK/za61hzTkCsECqc9d8DBIdhEKcNW0Vve EiNlHTPRSfKp7x19DP4y02EFNitA573AaRLSl3LmoGuAet0nMp4SBQDhj9VIE7QSpv1W ikHbyKpDGEj6SOuMdfKW+94GdzOiqbE/o9C4r28hTAygP77LnGQa6xtlVxhqQAUvrflG mERl0uY10EPvUFdV7XZkTKRMnMqOlwE4/GWJjMYFkgEU8lZfO6bcEcthMHGb25IYK6jS j6oCKPjbsJxaeuVTkr8FgqTlCz0vxC8EA35TgsU6lHYHxD/t3TcnAHwigBD5v+955eNS v/jA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=QJvHFeIS; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74360-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74360-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id z8-20020a1709060f0800b00a3d01994ccasi4134168eji.268.2024.02.21.01.10.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:10:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74360-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=QJvHFeIS; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74360-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74360-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 9ABC71F21E75 for ; Wed, 21 Feb 2024 09:10:52 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7308A46B80; Wed, 21 Feb 2024 09:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="QJvHFeIS"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="yvbnrV+A" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E67883E46D for ; Wed, 21 Feb 2024 09:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; cv=none; b=knk0ZleVpAQeT//aenlplHpnkm1npuEpkyFowJbHl5Eq6hc3gm5i3IStEbCkLbzTnmq1DVQgYebuWgiPzHuMy/H6x+bdjJ5xoEDLN8kRC/NMeG4BbwtuW7eZ2h+bsxQgy84gPki/D49NC+nUAXH7YjBMoggsIFI7SHY2oQibpu8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506382; c=relaxed/simple; bh=9uucUzWlPsh+XllbURepFkpBIfJjjTcmG/VWFvI4CmQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b2cfufiTcKCGIyv4D05M0Qt4DKZAWWmIneTCa2J7DphNIFW1M0QW/8pLAY8HUVniJjTREtZLJdg/LhwbwUa29bEk6Td2Y8b9qfIYrYvdmPc/pyBCuQg7mMT17KstTrqCb8M7T6VLWlJXogDiXzDom2auSuyxIglm4Nzyt0BKvgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=QJvHFeIS; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=yvbnrV+A; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=be6MfeAvNPKcsJz3AUGh7ZqxvewZ3xXmvk3rMk/0KI0=; b=QJvHFeISZgLzbY1jZ3y0IhU8jkZ+0LhZmq+s7G4/AOekOCTZi1/wjN22VXOsJNliK6tuLm 0wHvb3iSi339wRsPYOJ3gzW2Ebd6qSQFXmkKmWDD6ckd41Lo9PeVnE4pvjvpLieNA1oYfp dR5EEpvbQ0FYE3yoBodOAbO8XErwmdtGdHxEAahYx1YbTyE4XTRB3ZbRe9y/6xfZ3ph4yk mbrJbtm5OtB61r5TEVW1aODJmxdotz6lFUmxcAKI/B8MqEPnNqdJ/BugG2N92tluwCVHsU aeOvtjEmcu1Aaqd77R7EDvSQzyUbSzo2xK1UMe+rukjN21sRO0simJzyGOmpDw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=be6MfeAvNPKcsJz3AUGh7ZqxvewZ3xXmvk3rMk/0KI0=; b=yvbnrV+AKWKKUEos8S13cg3G2cgGT4GYWmL6cQXXFWozXgPfEEyUq/AeL0nBSa8rZjfz/H PGGS/MoyPAq9hOCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v11 16/20] tick/sched: Split out jiffies update helper function Date: Wed, 21 Feb 2024 10:05:44 +0100 Message-Id: <20240221090548.36600-17-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499072031615585 X-GMAIL-MSGID: 1791499072031615585 From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global function in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 8b0c28edbd09..ccf39befde85 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -157,6 +157,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 384723314c1f..c4d80d73bcfb 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -799,18 +799,30 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - u64 basemono, next_tick, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + u64 basemono, next_tick, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Wed Feb 21 09:05:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204037 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919723dyc; Wed, 21 Feb 2024 01:11:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWOzXHQVGhqyXcjpqx3Ppp8/AHMxslVX/YitnW5gU/qaZzUanjeDS3bnHosThusfpFLN8OST8D57+ktcoFE7Fzn5JRBjg== X-Google-Smtp-Source: AGHT+IE7DYiMaeWJOxwJzo3FiaJsZUAwg928YyrIjP7w6tF1nQWyVyBFcyYvH3rT0dsDgjKlwbSd X-Received: by 2002:a25:8404:0:b0:dbe:4f15:b5cf with SMTP id u4-20020a258404000000b00dbe4f15b5cfmr17201366ybk.15.1708506662259; Wed, 21 Feb 2024 01:11:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506662; cv=pass; d=google.com; s=arc-20160816; b=gQSOZARE2Rv2ApipXQd1G8kycoWX8DdaQNSBZ/RTiea4BRFfWmcCIEfaL25tF2Dy1o ZIQFX2VOjIVvjtewGwldLrylfsTOVK9iNYhAlzE8Q0KVs+BHAmO+yFf8maKuR83SW+3h dPX3tTLB85uWluIFnqcTMfnkXgRic5Aw4KQ0RpNvZBcGm/fF3WpRWVhvb7CHUKOIQNFh 2x0nwFWqFFvKMk3WMqiPGFb/xnqdKLguM9YdiHGWyhNO5Ih/QPJQNbpRqlo1UKjqtxaV VVWgasyZWrCoygn1jS49UfeNDfK0GJcA/a6OqV18yLkbJKhDxa6VYVEFz5O4dkeF7PLQ 66RA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=vEgEniDt+i600hdl6sVMoPHqdN5DurBPCgTXgC6Hb5c=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=yCLxbEuCZ5xfwCv48e8fc+/PwCPoe03pF4/zY6jG+YhiEEz0u8/Yg6XQVdOk1X4HI/ m66oQa68Itgh1+4Rr3OpdFCjvkQyukzJYHwbkLYyvQz16QIXFPwNokctTO60jYUOlIw6 NjrmHdPEP0EL70D2n/A+yTO9YC8MnZw/YO7oRBni63w/dzFR7iaTcOn9SXIZgJ4bdCfQ F/i79mRhNurMxWhzFox/8HzLMX3QKnjB/V0nwR9utSsaUgNN5rYlhEOcA39sHg4yDOSl CvoAh0M5BBONoUHbpFFFfncyIUjfCHqo3eHQ/Ip1EpvW+kYKIkxqx8gvCmbqLRZsWmmQ Z4Cg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=hLXk0gF4; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74362-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74362-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id o19-20020a05622a139300b0042c7dabc997si10782503qtk.715.2024.02.21.01.11.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74362-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=hLXk0gF4; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74362-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74362-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 107B31C214B2 for ; Wed, 21 Feb 2024 09:11:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D1D914778C; Wed, 21 Feb 2024 09:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="hLXk0gF4"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="BCTijobW" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DBAB3C6AC for ; Wed, 21 Feb 2024 09:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506383; cv=none; b=Etz5KuowYNPEvRanB598RN4rrn8zywTeWdcvlboy0dAsx3CecT9ddIzil4Aq/YmDjeek5I2RMp9QqLpWUQDAo80aYkp0XFQzN4axZptAbPYU+W2PZ9W6mkLLzrbNWa9JlW5pSEmoIimCRacSYLeR3ZTJIPhKlNjPvUxtgbLto/w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506383; c=relaxed/simple; bh=mYpTTHDiPnAjk6i2Zz3Vido8E6MS2R/eipR3R6cwMmc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qzeyLj0pDFsw6Qc/1E37gkfC0WiUvynxA5ZgLrHH2XDqiDqhBQ4V9Rv4m6ycjYce9FFx8rDb/o2UDZ0moS8nq27ac3pUcc6uota0Nit5meLgLT/BYJmeOFQ0OMAN3Ts6YtfUznPRmXuafKk8GB3ONZa79sW4FIxzBZZkrTDSDUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=hLXk0gF4; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=BCTijobW; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEgEniDt+i600hdl6sVMoPHqdN5DurBPCgTXgC6Hb5c=; b=hLXk0gF4IuRTJS+9uUgPogjJFyNzDfUNvkSKriQpARBOh30P8VO2ouGuBfxZ/JiWw0wxi6 Dvoaz2podQ57TjNf2qCsDxrKWF2BMBXaOg7GCIPHJYND2vZFnXV4wujD0zyYmk0qiCnYdq 0E2Q7fvHZC9YetINuOoVpCc2CSIU0/RShJ3NN48SBvEooM4KD9rxxRnWnRYqVSkqlQYvdu tKTKuFdQTJcAKxgOO9z3YX4Do7Sxj8abAUnqzV8eh8M4FMIa95fZNRdACCtAZKmjLCeu5+ p3M9wt2nGcTqq67jKezzF/V3v2Y7fKMsz+hsZrLOo0gc8ovOMjTaZmAK7JrLTg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEgEniDt+i600hdl6sVMoPHqdN5DurBPCgTXgC6Hb5c=; b=BCTijobWit1AkWSXjv1qMwGTLE5QKTde/y5krj2WFmv1C2QMSXVoBAb3C3CxnXEmNbrXiS Xl5ghdy5RHvYZHAQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 17/20] timers: Introduce function to check timer base is_idle flag Date: Wed, 21 Feb 2024 10:05:45 +0100 Message-Id: <20240221090548.36600-18-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499081751944992 X-GMAIL-MSGID: 1791499081751944992 To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have a function that returns the value of the is_idle flag of the timer base to keep the hierarchy states during online in sync with timer base state. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v10: Fix fallout of 0day: Move function definition of timer_base_is_idle() into SMP && NO_HZ_COMMON ifdef section v9: new in v9 --- kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index ccf39befde85..7e3090109e33 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -165,6 +165,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, unsigned int cpu); extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); +extern bool timer_base_is_idle(void); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7b9f9ed25fc2..3f618ee39763 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2114,6 +2114,16 @@ void timer_lock_remote_bases(unsigned int cpu) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); } + +/** + * timer_base_is_idle() - Return whether timer base is set idle + * + * Returns value of local timer base is_idle value. + */ +bool timer_base_is_idle(void) +{ + return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle); +} # endif /* CONFIG_SMP */ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, From patchwork Wed Feb 21 09:05:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204043 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp920091dyc; Wed, 21 Feb 2024 01:11:51 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUpAxwtxvpeI4mYec/6y+n99zTVs6ejh4LWuKfV4eq9WfXKK/GJ6uQoFdLKT1nvZFEIaqagPpE2ZO9eXXnedfZSF3aa1g== X-Google-Smtp-Source: AGHT+IHeZyGVep1iore+bcZhOTPacw8c4I4DvbX9E8EZtNzTl7V+MOaHvGz9rtOfKDEt0TMoBZrp X-Received: by 2002:a17:906:b18e:b0:a3e:8972:4404 with SMTP id w14-20020a170906b18e00b00a3e89724404mr5317359ejy.9.1708506711720; Wed, 21 Feb 2024 01:11:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506711; cv=pass; d=google.com; s=arc-20160816; b=c3Adi5n8uQwgrkJIRIOJRpLbIsiXZVknn7T8eidAH1+bklFKwZ/KmRHJIylrpkHsqf AuWPxBG85uGuwxE4EmprCFSiQBdQ4PSnTg7vXahxN5Ld71GJtjbdRUllR1qAYp/Dl2LN veAelv3Z+QdKqXReWqkeCmnrw8b5TPOpEvUR4g/rgUSHO+cunVwTvht27HTpmCTCS2P1 vbwKnRDpqvpryXfXLXxC+ZCPB/JioNbQx0JOwxRNn4fV20mSrVBcOH504cW2w7xyyo8T vJI25eKgBNQmqV719WveFBuL2y765B8ZS3ZZxGvi1OeykJJjbBQZexz/U45fI+NgyJwm UEDw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=jfuPUgUV3mra9xtSojeJst/48ODepDXAMWIY43hz8AY=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=cRH73JAO4IPMYQZWvsitcesPEykyeG1wasW9UOrrskeLWEVdu+1FTHB/Z2iPrwjZM6 uDSyV70iDryytJ9LrBm0+feGwWOSBD6jn8EMzCY8kRFGLYL9phuTIbav6v0k7SBUJh8w tX2YuZ3jRcKm1Dkc07zzkTXT5zd9fhoUB9vkWTru2gKvOxyzAHecINzPk6xQiQT4uP2w ImdxCDC8Zx+i8q0jnkYJr+9qtqn1JN3WaZ+axxaCN5Ar89Cw3sdSt03JzkTBZOUSsGwa XtygPlz2hTTxsUDFws/hrX6cm4gRfriEy8lAFxek8s7yH8B94LzuwWElCH4W383HZV0S f1xg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=t5FfMKHA; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=I2lWB5O8; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74365-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74365-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id 27-20020a170906311b00b00a3e57ae39dfsi3447860ejx.899.2024.02.21.01.11.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74365-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=t5FfMKHA; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=I2lWB5O8; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74365-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74365-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 8CCC71F230A3 for ; Wed, 21 Feb 2024 09:11:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CFD8A481DE; Wed, 21 Feb 2024 09:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="t5FfMKHA"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="I2lWB5O8" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DBEA3CF7C for ; Wed, 21 Feb 2024 09:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506386; cv=none; b=I5g2Ru8xS2LCpxQLHqpn11UfpJfs4H0vd2G8nTi6+5dg9TKy6cY/Z0v9hV/t6mh/hjYo+t7IVwsaGPm8a+UVsl2eKJeo8fhaRbnWSDXkjfJVK00Q/hCqKWGgFrBQL3A2bJpYzUYTy3dmnA0x0+2vdXHx3aitufIhdQE0VY92c98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506386; c=relaxed/simple; bh=5J7zUY2aOAQ1QUycFDuGQIEAgCXCUDLUqfAeF5vZUAo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=m6yZ2sMSozqTJYAUEbEQeWeOBI9ppPaWNuMFsjKzllgSv23RMw9D/cRdOyRHFMCu8XQTKFeenX+B0hkMGdFM9avy07+KoCHPkbLzXTaJOOsGrYUzD74QeHXDmDv1yDAzzNPGOElNVTkLt8hlkq4Hsb0YOGlKlLuEsGuo0vUrlcs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=t5FfMKHA; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=I2lWB5O8; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jfuPUgUV3mra9xtSojeJst/48ODepDXAMWIY43hz8AY=; b=t5FfMKHAmerxdf6N3iFa9SjNYDl26MopJ8z+JLemcbwdTAlRdbI+eoWuDfItatLDVnsL3o Q4BEfeVbLNkviXw0vM8o+7Z0igbLXaxw8ZQF3J7fYvEQHtUhfZFIdl9s2CVxJVk4WQnnOs KAB3DvoRFU0OdpU4c6n7C5U7LdLllm4X0J02aFNspN+bto1oNJNeNS4hgrOR9e9+XbLIvG DABAdc85UKz6Fx940qcaUD7X7X+KNke25eHbk9CStNtJsq3gSHcb+raNF3V7xRmVdlsxJb SunOiBMRlxVoW6Tg+kZgdUg8dWbxTzA1c/zS13dexfNm1R3gaxVdO5kqB2Ixkg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jfuPUgUV3mra9xtSojeJst/48ODepDXAMWIY43hz8AY=; b=I2lWB5O8R0zGUKqR8qLEP443HYtiR2NZhMB7tX9QtS1DI+HSjF/k10QhExT8HD/JgtoYe6 nbDO1pQ3PoYMD4Ag== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 18/20] timers: Implement the hierarchical pull model Date: Wed, 21 Feb 2024 10:05:46 +0100 Message-Id: <20240221090548.36600-19-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499133768841884 X-GMAIL-MSGID: 1791499133768841884 Placing timers at enqueue time on a target CPU based on dubious heuristics does not make any sense: 1) Most timer wheel timers are canceled or rearmed before they expire. 2) The heuristics to predict which CPU will be busy when the timer expires are wrong by definition. So placing the timers at enqueue wastes precious cycles. The proper solution to this problem is to always queue the timers on the local CPU and allow the non pinned timers to be pulled onto a busy CPU at expiry time. Therefore split the timer storage into local pinned and global timers: Local pinned timers are always expired on the CPU on which they have been queued. Global timers can be expired on any CPU. As long as a CPU is busy it expires both local and global timers. When a CPU goes idle it arms for the first expiring local timer. If the first expiring pinned (local) timer is before the first expiring movable timer, then no action is required because the CPU will wake up before the first movable timer expires. If the first expiring movable timer is before the first expiring pinned (local) timer, then this timer is queued into an idle timerqueue and eventually expired by another active CPU. To avoid global locking the timerqueues are implemented as a hierarchy. The lowest level of the hierarchy holds the CPUs. The CPUs are associated to groups of 8, which are separated per node. If more than one CPU group exist, then a second level in the hierarchy collects the groups. Depending on the size of the system more than 2 levels are required. Each group has a "migrator" which checks the timerqueue during the tick for remote expirable timers. If the last CPU in a group goes idle it reports the first expiring event in the group up to the next group(s) in the hierarchy. If the last CPU goes idle it arms its timer for the first system wide expiring timer to ensure that no timer event is missed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v11: - Memory barriers for reading the group and child states - Fix handling of 'check' member of struct tmigr_walk when executing tmigr_requires_handle_remote() - Fix race when updating events by holding the lock when reading the group and child states - hand in next event as argument of tmigr_quick_check() to make sure first global timer of CPU is also taken into account when doing the forecast - fix race against failed CPU hotplug in tmigr_quick_check() by walking the hierarchy instead of relying on the per level hlist - Update missing and outdated comments - Drop return value of tmigr_handle_remote_cpu() as group->next_expiry is updated and can be used - Frederics fix: Rework corner case, when last active CPU goes offline. v10: - Fix fallout of 0day (s/unsigned int/int) - Adress review remarks of bigeasy v9: - Adapt to the changes of the preceding patches - Fix state inconsitency (when timer base is idle, cpu must also be marked as idle in hierarchy) - Make sure new timers are considered, when timer base is idle and a timer is enqueued into global queue (e.g. during interrupt) -> timer_use_tmigr() - Changes which are required due to the timer code change of marking the timer base idle in tick_nohz_stop_tick() v8: - Review of Frederic: - Fix hotplug race (introduction of wakeup_recalc) - Make wakeup and wakeup_recalc logic consistent all over the place - Fix child/group state race and read it with locks held - Add more clarifying comments - Fix grammar all over the place - change integers which act as boolean value into bool - rewrite condition in tmigr_check_migrator() without negation - Improve update events logic with a check of the first event - Implement a quick forecast which is called when get_next_timer_interrupt() is executed. v7: - Review remarks of Frederic and bigeasy: - change logic in tmigr_handle_remote_cpu() - s/kzalloc/kcalloc - move timer_expire_remote() into NO_HZ_COMMON && SMP config section - drop DBG_BUG_ON() makro and use only WARN_ON_ONCE() - remove leftovers from sibling logic during setup - Move timer_expire_remote() into tick-internal.h - Add documentation section about "Required event and timerqueue update after remote expiry" - Fix fallout of kernel test robot v6: - Fix typos - Review remarks of Peter Zijlstra (locking, struct member cleanup, use atomic_try_cmpxchg(), update struct member descriptions) - Fix race in tmigr_handle_remote_cpu() (Frederic Weisbecker) v5: - Review remarks of Frederic - Return nextevt when CPU is marked offline in timer migration hierarchy instead of KTIME_MAX - Fix update of group events issue, after remote expiring v4: - Fold typo fix in comment into proper patch "timer: Split out "get next timer interrupt" functionality" - Update wrong comment for tmigr_state union definition - Fix fallout of kernel test robot --- include/linux/cpuhotplug.h | 1 + kernel/time/Makefile | 3 + kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 113 ++- kernel/time/timer_migration.c | 1747 +++++++++++++++++++++++++++++++++ kernel/time/timer_migration.h | 140 +++ 6 files changed, 1997 insertions(+), 8 deletions(-) create mode 100644 kernel/time/timer_migration.c create mode 100644 kernel/time/timer_migration.h diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 172d0a743e5d..7651904c6db5 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -231,6 +231,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, + CPUHP_AP_TMIGR_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, CPUHP_AP_RANDOM_ONLINE, diff --git a/kernel/time/Makefile b/kernel/time/Makefile index 7e875e63ff3b..4af2a264a160 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -17,6 +17,9 @@ endif obj-$(CONFIG_GENERIC_SCHED_CLOCK) += sched_clock.o obj-$(CONFIG_TICK_ONESHOT) += tick-oneshot.o tick-sched.o obj-$(CONFIG_LEGACY_TIMER_TICK) += tick-legacy.o +ifeq ($(CONFIG_SMP),y) + obj-$(CONFIG_NO_HZ_COMMON) += timer_migration.o +endif obj-$(CONFIG_HAVE_GENERIC_VDSO) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 7e3090109e33..a3243c4ac45f 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -166,6 +166,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); extern bool timer_base_is_idle(void); +extern void timer_expire_remote(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3f618ee39763..0339273f9365 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -53,6 +53,7 @@ #include #include "tick-internal.h" +#include "timer_migration.h" #define CREATE_TRACE_POINTS #include @@ -2124,6 +2125,64 @@ bool timer_base_is_idle(void) { return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle); } + +static void __run_timer_base(struct timer_base *base); + +/** + * timer_expire_remote() - expire global timers of cpu + * @cpu: Remote CPU + * + * Expire timers of global base of remote CPU. + */ +void timer_expire_remote(unsigned int cpu) +{ + struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + __run_timer_base(base); +} + +static void timer_use_tmigr(unsigned long basej, u64 basem, + unsigned long *nextevt, bool *tick_stop_path, + bool timer_base_idle, struct timer_events *tevt) +{ + u64 next_tmigr; + + if (timer_base_idle) + next_tmigr = tmigr_cpu_new_timer(tevt->global); + else if (tick_stop_path) + next_tmigr = tmigr_cpu_deactivate(tevt->global); + else + next_tmigr = tmigr_quick_check(tevt->global); + + /* + * If the CPU is the last going idle in timer migration hierarchy, make + * sure the CPU will wake up in time to handle remote timers. + * next_tmigr == KTIME_MAX if other CPUs are still active. + */ + if (next_tmigr < tevt->local) { + u64 tmp; + + /* If we missed a tick already, force 0 delta */ + if (next_tmigr < basem) + next_tmigr = basem; + + tmp = div_u64(next_tmigr - basem, TICK_NSEC); + + *nextevt = basej + (unsigned long)tmp; + tevt->local = next_tmigr; + } +} +# else +static void timer_use_tmigr(unsigned long basej, u64 basem, + unsigned long *nextevt, bool *tick_stop_path, + bool timer_base_idle, struct timer_events *tevt) +{ + /* + * Make sure first event is written into tevt->local to not miss a + * timer on !SMP systems. + */ + tevt->local = min_t(u64, tevt->local, tevt->global); +} # endif /* CONFIG_SMP */ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, @@ -2132,7 +2191,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; struct timer_base *base_local, *base_global; unsigned long nextevt; - u64 expires; + bool idle_is_possible; /* * Pretend that there is no timer pending if the cpu is offline. @@ -2153,6 +2212,22 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, nextevt = fetch_next_timer_interrupt(basej, basem, base_local, base_global, &tevt); + /* + * If the next event is only one jiffie ahead there is no need to call + * timer migration hierarchy related functions. The value for the next + * global timer in @tevt struct equals then KTIME_MAX. This is also + * true, when the timer base is idle. + * + * The proper timer migration hierarchy function depends on the callsite + * and whether timer base is idle or not. @nextevt will be updated when + * this CPU needs to handle the first timer migration hierarchy + * event. See timer_use_tmigr() for detailed information. + */ + idle_is_possible = time_after(nextevt, basej + 1); + if (idle_is_possible) + timer_use_tmigr(basej, basem, &nextevt, idle, + base_local->is_idle, &tevt); + /* * We have a fresh next event. Check whether we can forward the * base. @@ -2165,7 +2240,10 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, */ if (idle) { /* - * Bases are idle if the next event is more than a tick away. + * Bases are idle if the next event is more than a tick + * away. Caution: @nextevt could have changed by enqueueing a + * global timer into timer migration hierarchy. Therefore a new + * check is required here. * * If the base is marked idle then any timer add operation must * forward the base clk itself to keep granularity small. This @@ -2178,14 +2256,23 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, trace_timer_base_idle(true, base_local->cpu); } *idle = base_local->is_idle; + + /* + * When timer base is not set idle, undo the effect of + * tmigr_cpu_deactivate() to prevent inconsitent states - active + * timer base but inactive timer migration hierarchy. + * + * When timer base was already marked idle, nothing will be + * changed here. + */ + if (!base_local->is_idle && idle_is_possible) + tmigr_cpu_activate(); } raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - expires = min_t(u64, tevt.local, tevt.global); - - return cmp_next_hrtimer_event(basem, expires); + return cmp_next_hrtimer_event(basem, tevt.local); } /** @@ -2193,8 +2280,11 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * @basej: base time jiffies * @basem: base time clock monotonic * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer or + * KTIME_MAX if no timer is pending. If timer of global base was queued into + * timer migration hierarchy, first global timer is not taken into account. If + * it was the last CPU of timer migration hierarchy going idle, first global + * event is taken into account. */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { @@ -2236,6 +2326,9 @@ void timer_clear_idle(void) __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(false, smp_processor_id()); + + /* Activate without holding the timer_base->lock */ + tmigr_cpu_activate(); } #endif @@ -2305,6 +2398,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { run_timer_base(BASE_GLOBAL); run_timer_base(BASE_DEF); + + if (is_timers_nohz_active()) + tmigr_handle_remote(); } } @@ -2319,7 +2415,8 @@ static void run_local_timers(void) for (int i = 0; i < NR_BASES; i++, base++) { /* Raise the softirq only if required. */ - if (time_after_eq(jiffies, base->next_expiry)) { + if (time_after_eq(jiffies, base->next_expiry) || + (i == BASE_DEF && tmigr_requires_handle_remote())) { raise_softirq(TIMER_SOFTIRQ); return; } diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c new file mode 100644 index 000000000000..9d6d5e5def5b --- /dev/null +++ b/kernel/time/timer_migration.c @@ -0,0 +1,1747 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Infrastructure for migratable timers + * + * Copyright(C) 2022 linutronix GmbH + */ +#include +#include +#include +#include +#include +#include + +#include "timer_migration.h" +#include "tick-internal.h" + +/* + * The timer migration mechanism is built on a hierarchy of groups. The + * lowest level group contains CPUs, the next level groups of CPU groups + * and so forth. The CPU groups are kept per node so for the normal case + * lock contention won't happen across nodes. Depending on the number of + * CPUs per node even the next level might be kept as groups of CPU groups + * per node and only the levels above cross the node topology. + * + * Example topology for a two node system with 24 CPUs each. + * + * LVL 2 [GRP2:0] + * GRP1:0 = GRP1:M + * + * LVL 1 [GRP1:0] [GRP1:1] + * GRP0:0 - GRP0:2 GRP0:3 - GRP0:5 + * + * LVL 0 [GRP0:0] [GRP0:1] [GRP0:2] [GRP0:3] [GRP0:4] [GRP0:5] + * CPUS 0-7 8-15 16-23 24-31 32-39 40-47 + * + * The groups hold a timer queue of events sorted by expiry time. These + * queues are updated when CPUs go in idle. When they come out of idle + * ignore flag of events is set. + * + * Each group has a designated migrator CPU/group as long as a CPU/group is + * active in the group. This designated role is necessary to avoid that all + * active CPUs in a group try to migrate expired timers from other CPUs, + * which would result in massive lock bouncing. + * + * When a CPU is awake, it checks in it's own timer tick the group + * hierarchy up to the point where it is assigned the migrator role or if + * no CPU is active, it also checks the groups where no migrator is set + * (TMIGR_NONE). + * + * If it finds expired timers in one of the group queues it pulls them over + * from the idle CPU and runs the timer function. After that it updates the + * group and the parent groups if required. + * + * CPUs which go idle arm their CPU local timer hardware for the next local + * (pinned) timer event. If the next migratable timer expires after the + * next local timer or the CPU has no migratable timer pending then the + * CPU does not queue an event in the LVL0 group. If the next migratable + * timer expires before the next local timer then the CPU queues that timer + * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0 + * group. + * + * When CPU comes out of idle and when a group has at least a single active + * child, the ignore flag of the tmigr_event is set. This indicates, that + * the event is ignored even if it is still enqueued in the parent groups + * timer queue. It will be removed when touching the timer queue the next + * time. This spares locking in active path as the lock protects (after + * setup) only event information. For more information about locking, + * please read the section "Locking rules". + * + * If the CPU is the migrator of the group then it delegates that role to + * the next active CPU in the group or sets migrator to TMIGR_NONE when + * there is no active CPU in the group. This delegation needs to be + * propagated up the hierarchy so hand over from other leaves can happen at + * all hierarchy levels w/o doing a search. + * + * When the last CPU in the system goes idle, then it drops all migrator + * duties up to the top level of the hierarchy (LVL2 in the example). It + * then has to make sure, that it arms it's own local hardware timer for + * the earliest event in the system. + * + * + * Lifetime rules: + * --------------- + * + * The groups are built up at init time or when CPUs come online. They are + * not destroyed when a group becomes empty due to offlining. The group + * just won't participate in the hierarchy management anymore. Destroying + * groups would result in interesting race conditions which would just make + * the whole mechanism slow and complex. + * + * + * Locking rules: + * -------------- + * + * For setting up new groups and handling events it's required to lock both + * child and parent group. The lock ordering is always bottom up. This also + * includes the per CPU locks in struct tmigr_cpu. For updating the migrator and + * active CPU/group information atomic_try_cmpxchg() is used instead and only + * the per CPU tmigr_cpu->lock is held. + * + * During the setup of groups tmigr_level_list is required. It is protected by + * @tmigr_mutex. + * + * When @timer_base->lock as well as tmigr related locks are required, the lock + * ordering is: first @timer_base->lock, afterwards tmigr related locks. + * + * + * Protection of the tmigr group state information: + * ------------------------------------------------ + * + * The state information with the list of active children and migrator needs to + * be protected by a sequence counter. It prevents a race when updates in child + * groups are propagated in changed order. The state update is performed + * lockless and group wise. The following scenario describes what happens + * without updating the sequence counter: + * + * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well + * as GRP0:1 will not change during the scenario): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU0 migrator = CPU2 + * active = CPU0 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * active idle active idle + * + * + * 1. CPU0 goes idle. As the update is performed group wise, in the first step + * only GRP0:0 is updated. The update of GRP1:0 is pending as CPU0 has to + * walk the hierarchy. + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = TMIGR_NONE migrator = CPU2 + * --> active = active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * --> idle idle active idle + * + * 2. While CPU0 goes idle and continues to update the state, CPU1 comes out of + * idle. CPU1 updates GRP0:0. The update for GRP1:0 is pending as CPU1 also + * has to walk the hierarchy. Both CPUs (CPU0 and CPU1) now walk the + * hierarchy to perform the needed update from their point of view. The + * currently visible state looks the following: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = CPU1 migrator = CPU2 + * --> active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle --> active active idle + * + * 3. Here is the race condition: CPU1 managed to propagate its changes (from + * step 2) through the hierarchy to GRP1:0 before CPU0 (step 1) did. The + * active members of GRP1:0 remain unchanged after the update since it is + * still valid from CPU1 current point of view: + * + * LVL 1 [GRP1:0] + * --> migrator = GRP0:1 + * --> active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU1 migrator = CPU2 + * active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle active active idle + * + * 4. Now CPU0 finally propagates its changes (from step 1) to GRP1:0. + * + * LVL 1 [GRP1:0] + * --> migrator = GRP0:1 + * --> active = GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU1 migrator = CPU2 + * active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle active active idle + * + * + * The race of CPU0 vs. CPU1 led to an inconsistent state in GRP1:0. CPU1 is + * active and is correctly listed as active in GRP0:0. However GRP1:0 does not + * have GRP0:0 listed as active, which is wrong. The sequence counter has been + * added to avoid inconsistent states during updates. The state is updated + * atomically only if all members, including the sequence counter, match the + * expected value (compare-and-exchange). + * + * Looking back at the previous example with the addition of the sequence + * counter: The update as performed by CPU0 in step 4 will fail. CPU1 changed + * the sequence number during the update in step 3 so the expected old value (as + * seen by CPU0 before starting the walk) does not match. + * + * Prevent race between new event and last CPU going inactive + * ---------------------------------------------------------- + * + * When the last CPU is going idle and there is a concurrent update of a new + * first global timer of an idle CPU, the group and child states have to be read + * while holding the lock in tmigr_update_events(). The following scenario shows + * what happens, when this is not done. + * + * 1. Only CPU2 is active: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * next_expiry = KTIME_MAX + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * next_expiry = KTIME_MAX next_expiry = KTIME_MAX + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 2. Now CPU 2 goes idle (and has no global timer, that has to be handled) and + * propagates that to GRP0:1: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * next_expiry = KTIME_MAX + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE --> migrator = TMIGR_NONE + * active = --> active = + * next_expiry = KTIME_MAX next_expiry = KTIME_MAX + * / \ / \ + * CPUs 0 1 2 3 + * idle idle --> idle idle + * + * 3. Now the idle state is propagated up to GRP1:0. As this is now the last + * child going idle in top level group, the expiry of the next group event + * has to be handed back to make sure no event is lost. As there is no event + * enqueued, KTIME_MAX is handed back to CPU2. + * + * LVL 1 [GRP1:0] + * --> migrator = TMIGR_NONE + * --> active = + * next_expiry = KTIME_MAX + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = TMIGR_NONE + * active = active = + * next_expiry = KTIME_MAX next_expiry = KTIME_MAX + * / \ / \ + * CPUs 0 1 2 3 + * idle idle --> idle idle + * + * 4. CPU 0 has a new timer queued from idle and it expires at TIMER0. CPU0 + * propagates that to GRP0:0: + * + * LVL 1 [GRP1:0] + * migrator = TMIGR_NONE + * active = + * next_expiry = KTIME_MAX + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = TMIGR_NONE + * active = active = + * --> next_expiry = TIMER0 next_expiry = KTIME_MAX + * / \ / \ + * CPUs 0 1 2 3 + * idle idle idle idle + * + * 5. GRP0:0 is not active, so the new timer has to be propagated to + * GRP1:0. Therefore the GRP1:0 state has to be read. When the stalled value + * (from step 2) is read, the timer is enqueued into GRP1:0, but nothing is + * handed back to CPU0, as it seems that there is still an active child in + * top level group. + * + * LVL 1 [GRP1:0] + * migrator = TMIGR_NONE + * active = + * --> next_expiry = TIMER0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = TMIGR_NONE + * active = active = + * next_expiry = TIMER0 next_expiry = KTIME_MAX + * / \ / \ + * CPUs 0 1 2 3 + * idle idle idle idle + * + * This is prevented by reading the state when holding the lock (when a new + * timer has to be propagated from idle path):: + * + * CPU2 (tmigr_inactive_up()) CPU0 (tmigr_new_timer_up()) + * -------------------------- --------------------------- + * // step 3: + * cmpxchg(&GRP1:0->state); + * tmigr_update_events() { + * spin_lock(&GRP1:0->lock); + * // ... update events ... + * // hand back first expiry when GRP1:0 is idle + * spin_unlock(&GRP1:0->lock); + * // ^^^ release state modification + * } + * tmigr_update_events() { + * spin_lock(&GRP1:0->lock) + * // ^^^ acquire state modification + * group_state = atomic_read(&GRP1:0->state) + * // .... update events ... + * // hand back first expiry when GRP1:0 is idle + * spin_unlock(&GRP1:0->lock) <3> + * // ^^^ makes state visible for other + * // callers of tmigr_new_timer_up() + * } + * + * When CPU0 grabs the lock directly after cmpxchg, the first timer is reported + * back to CPU0 and also later on to CPU2. So no timer is missed. A concurrent + * update of the group state from active path is no problem, as the upcoming CPU + * will take care of the group events. + * + * Required event and timerqueue update after a remote expiry: + * ----------------------------------------------------------- + * + * After expiring timers of a remote CPU, a walk through the hierarchy and + * update of events and timerqueues is required. It is obviously needed if there + * is a 'new' global timer but also if there is no new global timer but the + * remote CPU is still idle. + * + * 1. CPU0 and CPU1 are idle and have both a global timer expiring at the same + * time. So both have an event enqueued in the timerqueue of GRP0:0. CPU3 is + * also idle and has no global timer pending. CPU2 is the only active CPU and + * thus also the migrator: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 2. CPU2 starts to expire remote timers. It starts with LVL0 group + * GRP0:1. There is no event queued in the timerqueue, so CPU2 continues with + * the parent of GRP0:1: GRP1:0. In GRP1:0 it dequeues the first event. It + * looks at tmigr_event::cpu struct member and expires the pending timer(s) + * of CPU0. + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 3. Some work has to be done after expiring the timers of CPU0. If we stop + * here, then CPU1's pending global timer(s) will not expire in time and the + * timerqueue of GRP0:0 has still an event for CPU0 enqueued which has just + * been processed. So it is required to walk the hierarchy from CPU0's point + * of view and update it accordingly. CPU0's event will be removed from the + * timerqueue because it has no pending timer. If CPU0 would have a timer + * pending then it has to expire after CPU1's first timer because all timers + * from this period were just expired. Either way CPU1's event will be first + * in GRP0:0's timerqueue and therefore set in the CPU field of the group + * event which is then enqueued in GRP1:0's timerqueue as GRP0:0 is still not + * active: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = false groupevt.ignore = true + * --> groupevt.cpu = CPU1 groupevt.cpu = + * --> timerqueue = evt-CPU1 timerqueue = + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * Now CPU2 (migrator) will continue step 2 at GRP1:0 and will expire the + * timer(s) of CPU1. + * + * The hierarchy walk in step 3 can be skipped if the migrator notices that a + * CPU of GRP0:0 is active again. The CPU will mark GRP0:0 active and take care + * of the group as migrator and any needed updates within the hierarchy. + */ + +static DEFINE_MUTEX(tmigr_mutex); +static struct list_head *tmigr_level_list __read_mostly; + +static unsigned int tmigr_hierarchy_levels __read_mostly; +static unsigned int tmigr_crossnode_level __read_mostly; + +static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu); + +#define TMIGR_NONE 0xFF +#define BIT_CNT 8 + +static inline bool tmigr_is_not_available(struct tmigr_cpu *tmc) +{ + return !(tmc->tmgroup && tmc->online); +} + +/* + * Returns true, when @childmask corresponds to the group migrator or when the + * group is not active - so no migrator is set. + */ +static bool tmigr_check_migrator(struct tmigr_group *group, u8 childmask) +{ + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + return true; + + return false; +} + +static bool tmigr_check_migrator_and_lonely(struct tmigr_group *group, u8 childmask) +{ + bool lonely, migrator = false; + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE)) + migrator = true; + + active = s.active; + lonely = bitmap_weight(&active, BIT_CNT) <= 1; + + return (migrator && lonely); +} + +static bool tmigr_check_lonely(struct tmigr_group *group) +{ + unsigned long active; + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + active = s.active; + + return bitmap_weight(&active, BIT_CNT) <= 1; +} + +typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *); + +static void __walk_groups(up_f up, void *data, + struct tmigr_cpu *tmc) +{ + struct tmigr_group *child = NULL, *group = tmc->tmgroup; + + do { + WARN_ON_ONCE(group->level >= tmigr_hierarchy_levels); + + if (up(group, child, data)) + break; + + child = group; + group = group->parent; + } while (group); +} + +static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc) +{ + lockdep_assert_held(&tmc->lock); + + __walk_groups(up, data, tmc); +} + +/** + * struct tmigr_walk - data required for walking the hierarchy + * @nextexp: Next CPU event expiry information which is handed into + * the timer migration code by the timer code + * (get_next_timer_interrupt()) + * @firstexp: Contains the first event expiry information when last + * active CPU of hierarchy is on the way to idle to make + * sure CPU will be back in time. + * @evt: Pointer to tmigr_event which needs to be queued (of idle + * child group) + * @childmask: childmask of child group + * @remote: Is set, when the new timer path is executed in + * tmigr_handle_remote_cpu() + */ +struct tmigr_walk { + u64 nextexp; + u64 firstexp; + struct tmigr_event *evt; + u8 childmask; + bool remote; +}; + +/** + * struct tmigr_remote_data - data required for remote expiry hierarchy walk + * @basej: timer base in jiffies + * @now: timer base monotonic + * @firstexp: returns expiry of the first timer in the idle timer + * migration hierarchy to make sure the timer is handled in + * time; it is stored in the per CPU tmigr_cpu struct of + * CPU which expires remote timers + * @childmask: childmask of child group + * @check: is set if there is the need to handle remote timers; + * required in tmigr_requires_handle_remote() only + * @tmc_active: this flag indicates, whether the CPU which triggers + * the hierarchy walk is !idle in the timer migration + * hierarchy. When the CPU is idle and the whole hierarchy is + * idle, only the first event of the top level has to be + * considered. + */ +struct tmigr_remote_data { + unsigned long basej; + u64 now; + u64 firstexp; + u8 childmask; + bool check; + bool tmc_active; +}; + +/* + * Returns the next event of the timerqueue @group->events + * + * Removes timers with ignore flag and update next_expiry of the group. Values + * of the group event are updated in tmigr_update_events() only. + */ +static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group) +{ + struct timerqueue_node *node = NULL; + struct tmigr_event *evt = NULL; + + lockdep_assert_held(&group->lock); + + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + while ((node = timerqueue_getnext(&group->events))) { + evt = container_of(node, struct tmigr_event, nextevt); + + if (!evt->ignore) { + WRITE_ONCE(group->next_expiry, evt->nextevt.expires); + return evt; + } + + /* + * Remove next timers with ignore flag, because the group lock + * is held anyway + */ + if (!timerqueue_del(&group->events, node)) + break; + } + + return NULL; +} + +/* + * Return the next event (with the expiry equal or before @now) + * + * Event, which is returned, is also removed from the queue. + */ +static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group, + u64 now) +{ + struct tmigr_event *evt = tmigr_next_groupevt(group); + + if (!evt || now < evt->nextevt.expires) + return NULL; + + /* + * The event is ready to expire. Remove it and update next group event. + */ + timerqueue_del(&group->events, &evt->nextevt); + tmigr_next_groupevt(group); + + return evt; +} + +static u64 tmigr_next_groupevt_expires(struct tmigr_group *group) +{ + struct tmigr_event *evt; + + evt = tmigr_next_groupevt(group); + + if (!evt) + return KTIME_MAX; + else + return evt->nextevt.expires; +} + +static bool tmigr_active_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + curstate.state = atomic_read(&group->migr_state); + + do { + newstate = curstate; + walk_done = true; + + if (newstate.migrator == TMIGR_NONE) { + newstate.migrator = childmask; + + /* Changes need to be propagated */ + walk_done = false; + } + + newstate.active |= childmask; + newstate.seq++; + + } while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)); + + if ((walk_done == false) && group->parent) + data->childmask = group->childmask; + + /* + * The group is active (again). The group event might be still queued + * into the parent group's timerqueue but can now be handled by the + * migrator of this group. Therefore the ignore flag for the group event + * is updated to reflect this. + * + * The update of the ignore flag in the active path is done lockless. In + * worst case the migrator of the parent group observes the change too + * late and expires remotely all events belonging to this group. The + * lock is held while updating the ignore flag in idle path. So this + * state change will not be lost. + */ + group->groupevt.ignore = true; + + return walk_done; +} + +static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) +{ + struct tmigr_walk data; + + data.childmask = tmc->childmask; + + tmc->cpuevt.ignore = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + walk_groups(&tmigr_active_up, &data, tmc); +} + +/** + * tmigr_cpu_activate() - set this CPU active in timer migration hierarchy + * + * Call site timer_clear_idle() is called with interrupts disabled. + */ +void tmigr_cpu_activate(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + if (tmigr_is_not_available(tmc)) + return; + + if (WARN_ON_ONCE(!tmc->idle)) + return; + + raw_spin_lock(&tmc->lock); + tmc->idle = false; + __tmigr_cpu_activate(tmc); + raw_spin_unlock(&tmc->lock); +} + +/* + * Returns true, if there is nothing to be propagated to the next level + * + * @data->firstexp is set to expiry of first gobal event of the (top level of + * the) hierarchy, but only when hierarchy is completely idle. + * + * The child and group states need to be read under the lock, to prevent a race + * against a concurrent tmigr_inactive_up() run when the last CPU goes idle. See + * also section "Prevent race between new event and last CPU going inactive" in + * the documentation at the top. + * + * This is the only place where the group event expiry value is set. + */ +static +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child, + struct tmigr_walk *data) +{ + struct tmigr_event *evt, *first_childevt; + union tmigr_state childstate, groupstate; + bool leftmost_change = false; + bool remote = data->remote; + bool walk_done = false; + u64 nextexp; + + if (child) { + raw_spin_lock(&child->lock); + raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING); + + childstate.state = atomic_read(&child->migr_state); + groupstate.state = atomic_read(&group->migr_state); + + if (childstate.active) { + walk_done = true; + goto unlock; + } + + first_childevt = tmigr_next_groupevt(child); + nextexp = child->next_expiry; + evt = &child->groupevt; + + evt->ignore = (nextexp == KTIME_MAX) ? true : false; + } else { + nextexp = data->nextexp; + + first_childevt = evt = data->evt; + + /* + * Walking the hierarchy is required in any case when a + * remote expiry was done before. This ensures to not lose + * already queued events in non active groups (see section + * "Required event and timerqueue update after a remote + * expiry" in the documentation at the top). + * + * The two call sites which are executed without a remote expiry + * before, are not prevented from propagating changes through + * the hierarchy by the return: + * - When entering this path by tmigr_new_timer(), @evt->ignore + * is never set. + * - tmigr_inactive_up() takes care of the propagation by + * itself and ignores the return value. But an immediate + * return is required because nothing has to be done in this + * level as the event could be ignored. + */ + if (evt->ignore && !remote) + return true; + + raw_spin_lock(&group->lock); + + childstate.state = 0; + groupstate.state = atomic_read(&group->migr_state); + } + + /* + * If the child event is already queued in the group, remove it from the + * queue when the expiry time changed only or when it could be ignored. + */ + if (timerqueue_node_queued(&evt->nextevt)) { + if ((evt->nextevt.expires == nextexp) && !evt->ignore) + goto check_toplvl; + + leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt; + if (!timerqueue_del(&group->events, &evt->nextevt)) + WRITE_ONCE(group->next_expiry, KTIME_MAX); + } + + if (evt->ignore) { + /* + * When the next child event could be ignored (nextexp is + * KTIME_MAX) and there was no remote timer handling before or + * the group is already active, there is no need to walk the + * hierarchy even if there is a parent group. + * + * The other way round: even if the event could be ignored, but + * if a remote timer handling was executed before and the group + * is not active, walking the hierarchy is required to not miss + * an enqueued timer in the non active group. The enqueued timer + * of the group needs to be propagated to a higher level to + * ensure it is handled. + */ + if (!remote || groupstate.active) + walk_done = true; + } else { + evt->nextevt.expires = nextexp; + evt->cpu = first_childevt->cpu; + + if (timerqueue_add(&group->events, &evt->nextevt)) { + leftmost_change = true; + WRITE_ONCE(group->next_expiry, nextexp); + } + } + +check_toplvl: + if (!group->parent && (groupstate.migrator == TMIGR_NONE)) { + walk_done = true; + + /* + * Nothing to do when update was done during remote timer + * handling. First timer in top level group which needs to be + * handled when top level group is not active, is calculated + * directly in tmigr_handle_remote_up(). + */ + if (remote) + goto unlock; + + /* + * The top level group is idle and it has to be ensured the + * global timers are handled in time. (This could be optimized + * by keeping track of the last global scheduled event and only + * arming it on the CPU if the new event is earlier. Not sure if + * its worth the complexity.) + */ + data->firstexp = tmigr_next_groupevt_expires(group); + } + +unlock: + raw_spin_unlock(&group->lock); + + if (child) + raw_spin_unlock(&child->lock); + + return walk_done; +} + +static bool tmigr_new_timer_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_walk *data = ptr; + + return tmigr_update_events(group, child, data); +} + +/* + * Returns the expiry of the next timer that needs to be handled. KTIME_MAX is + * returned, if an active CPU will handle all the timer migration hierarchy + * timers. + */ +static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .nextexp = nextexp, + .firstexp = KTIME_MAX, + .evt = &tmc->cpuevt }; + + lockdep_assert_held(&tmc->lock); + + if (tmc->remote) + return KTIME_MAX; + + tmc->cpuevt.ignore = false; + data.remote = false; + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + /* If there is a new first global event, make sure it is handled */ + return data.firstexp; +} + +static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now, + unsigned long jif) +{ + struct timer_events tevt; + struct tmigr_walk data; + struct tmigr_cpu *tmc; + + tmc = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_lock_irq(&tmc->lock); + + /* + * If the remote CPU is offline then the timers have been migrated to + * another CPU. + * + * If tmigr_cpu::remote is set, at the moment another CPU already + * expires the timers of the remote CPU. + * + * If tmigr_event::ignore is set, then the CPU returns from idle and + * takes care of its timers. + * + * If the next event expires in the future, then the event has been + * updated and there are no timers to expire right now. The CPU which + * updated the event takes care when hierarchy is completely + * idle. Otherwise the migrator does it as the event is enqueued. + */ + if (!tmc->online || tmc->remote || tmc->cpuevt.ignore || + now < tmc->cpuevt.nextevt.expires) { + raw_spin_unlock_irq(&tmc->lock); + return; + } + + tmc->remote = true; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* Drop the lock to allow the remote CPU to exit idle */ + raw_spin_unlock_irq(&tmc->lock); + + if (cpu != smp_processor_id()) + timer_expire_remote(cpu); + + /* + * Lock ordering needs to be preserved - timer_base locks before tmigr + * related locks (see section "Locking rules" in the documentation at + * the top). During fetching the next timer interrupt, also tmc->lock + * needs to be held. Otherwise there is a possible race window against + * the CPU itself when it comes out of idle, updates the first timer in + * the hierarchy and goes back to idle. + * + * timer base locks are dropped as fast as possible: After checking + * whether the remote CPU went offline in the meantime and after + * fetching the next remote timer interrupt. Dropping the locks as fast + * as possible keeps the locking region small and prevents holding + * several (unnecessary) locks during walking the hierarchy for updating + * the timerqueue and group events. + */ + local_irq_disable(); + timer_lock_remote_bases(cpu); + raw_spin_lock(&tmc->lock); + + /* + * When the CPU went offline in the meantime, no hierarchy walk has to + * be done for updating the queued events, because the walk was + * already done during marking the CPU offline in the hierarchy. + * + * When the CPU is no longer idle, the CPU takes care of the timers and + * also of the timers in the hierarchy. + * + * (See also section "Required event and timerqueue update after a + * remote expiry" in the documentation at the top) + */ + if (!tmc->online || !tmc->idle) { + timer_unlock_remote_bases(cpu); + goto unlock; + } + + /* next event of CPU */ + fetch_next_timer_interrupt_remote(jif, now, &tevt, cpu); + timer_unlock_remote_bases(cpu); + + data.nextexp = tevt.global; + data.firstexp = KTIME_MAX; + data.evt = &tmc->cpuevt; + data.remote = true; + + /* + * The update is done even when there is no 'new' global timer pending + * on the remote CPU (see section "Required event and timerqueue update + * after a remote expiry" in the documentation at the top) + */ + walk_groups(&tmigr_new_timer_up, &data, tmc); + +unlock: + tmc->remote = false; + raw_spin_unlock_irq(&tmc->lock); + + return; +} + +static bool tmigr_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + struct tmigr_event *evt; + unsigned long jif; + u8 childmask; + u64 now; + + jif = data->basej; + now = data->now; + + childmask = data->childmask; + +again: + /* + * Handle the group only if @childmask is the migrator or if the + * group has no migrator. Otherwise the group is active and is + * handled by its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + raw_spin_lock_irq(&group->lock); + + evt = tmigr_next_expired_groupevt(group, now); + + if (evt) { + unsigned int remote_cpu = evt->cpu; + + raw_spin_unlock_irq(&group->lock); + + tmigr_handle_remote_cpu(remote_cpu, now, jif); + + /* check if there is another event, that needs to be handled */ + goto again; + } + + /* + * Update of childmask for the next level and keep track of the expiry + * of the first event that needs to be handled (group->next_expiry was + * updated by tmigr_next_expired_groupevt(), next was set by + * tmigr_handle_remote_cpu()). + */ + data->childmask = group->childmask; + data->firstexp = group->next_expiry; + + raw_spin_unlock_irq(&group->lock); + + return false; +} + +/** + * tmigr_handle_remote() - Handle global timers of remote idle CPUs + * + * Called from the timer soft interrupt with interrupts enabled. + */ +void tmigr_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (tmigr_is_not_available(tmc)) + return; + + data.childmask = tmc->childmask; + data.firstexp = KTIME_MAX; + + /* + * NOTE: This is a doubled check because the migrator test will be done + * in tmigr_handle_remote_up() anyway. Keep this check to speed up the + * return when nothing has to be done. + */ + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return; + + data.now = get_jiffies_update(&data.basej); + + /* + * Update @tmc->wakeup only at the end and do not reset @tmc->wakeup to + * KTIME_MAX. Even if tmc->lock is not held during the whole remote + * handling, tmc->wakeup is fine to be stale as it is called in + * interrupt context and tick_nohz_next_event() is executed in interrupt + * exit path only after processing the last pending interrupt. + */ + + __walk_groups(&tmigr_handle_remote_up, &data, tmc); + + raw_spin_lock_irq(&tmc->lock); + WRITE_ONCE(tmc->wakeup, data.firstexp); + raw_spin_unlock_irq(&tmc->lock); +} + +static bool tmigr_requires_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u8 childmask; + + childmask = data->childmask; + + /* + * Handle the group only if the child is the migrator or if the group + * has no migrator. Otherwise the group is active and is handled by its + * own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + /* + * When there is a parent group and the CPU which triggered the + * hierarchy walk is not active, proceed the walk to reach the top level + * group before reading the next_expiry value. + */ + if (group->parent && !data->tmc_active) + goto out; + + /* + * The lock is required on 32bit architectures to read the variable + * consistently with a concurrent writer. On 64bit the lock is not + * required because the read operation is not split and so it is always + * consistent. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + data->firstexp = READ_ONCE(group->next_expiry); + if (data->now >= data->firstexp) { + data->check = true; + return true; + } + } else { + raw_spin_lock(&group->lock); + data->firstexp = group->next_expiry; + if (data->now >= group->next_expiry) { + data->check = true; + raw_spin_unlock(&group->lock); + return true; + } + raw_spin_unlock(&group->lock); + } + +out: + /* Update of childmask for the next level */ + data->childmask = group->childmask; + return false; +} + +/** + * tmigr_requires_handle_remote() - Check the need of remote timer handling + * + * Must be called with interrupts disabled. + */ +bool tmigr_requires_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + unsigned long jif; + bool ret = false; + + if (tmigr_is_not_available(tmc)) + return ret; + + data.now = get_jiffies_update(&jif); + data.childmask = tmc->childmask; + data.firstexp = KTIME_MAX; + data.tmc_active = !tmc->idle; + data.check = false; + + /* + * If the CPU is active, walk the hierarchy to check whether a remote + * expiry is required. + * + * Check is done lockless as interrupts are disabled and @tmc->idle is + * set only by the local CPU. + */ + if (!tmc->idle) { + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + return data.check; + } + + /* + * When the CPU is idle, compare @tmc->wakeup with @data.now. The lock + * is required on 32bit architectures to read the variable consistently + * with a concurrent writer. On 64bit the lock is not required because + * the read operation is not split and so it is always consistent. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + if (data.now >= READ_ONCE(tmc->wakeup)) + return true; + } else { + raw_spin_lock(&tmc->lock); + if (data.now >= tmc->wakeup) + ret = true; + raw_spin_unlock(&tmc->lock); + } + + return ret; +} + +/** + * tmigr_cpu_new_timer() - enqueue next global timer into hierarchy (idle tmc) + * @nextexp: Next expiry of global timer (or KTIME_MAX if not) + * + * The CPU is already deactivated in the timer migration + * hierarchy. tick_nohz_get_sleep_length() calls tick_nohz_next_event() + * and thereby the timer idle path is executed once more. @tmc->wakeup + * holds the first timer, when the timer migration hierarchy is + * completely idle. + * + * Returns the first timer that needs to be handled by this CPU or KTIME_MAX if + * nothing needs to be done. + */ +u64 tmigr_cpu_new_timer(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (tmigr_is_not_available(tmc)) + return nextexp; + + raw_spin_lock(&tmc->lock); + + ret = READ_ONCE(tmc->wakeup); + if (nextexp != KTIME_MAX) { + if (nextexp != tmc->cpuevt.nextevt.expires || + tmc->cpuevt.ignore) { + ret = tmigr_new_timer(tmc, nextexp); + } + } + /* + * Make sure the reevaluation of timers in idle path will not miss an + * event. + */ + WRITE_ONCE(tmc->wakeup, ret); + + raw_spin_unlock(&tmc->lock); + return ret; +} + +static bool tmigr_inactive_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate, childstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + curstate.state = atomic_read_acquire(&group->migr_state); + childstate.state = 0; + + for (;;) { + if (child) + childstate.state = atomic_read(&child->migr_state); + + newstate = curstate; + walk_done = true; + + /* Reset active bit when the child is no longer active */ + if (!childstate.active) + newstate.active &= ~childmask; + + if (newstate.migrator == childmask) { + /* + * Find a new migrator for the group, because the child + * group is idle! + */ + if (!childstate.active) { + unsigned long new_migr_bit, active = newstate.active; + + new_migr_bit = find_first_bit(&active, BIT_CNT); + + if (new_migr_bit != BIT_CNT) { + newstate.migrator = BIT(new_migr_bit); + } else { + newstate.migrator = TMIGR_NONE; + + /* Changes need to be propagated */ + walk_done = false; + } + } + } + + newstate.seq++; + + WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active)); + + if (atomic_try_cmpxchg(&group->migr_state, &curstate.state, + newstate.state)) + break; + smp_mb__after_atomic(); + } + + data->remote = false; + + /* Event Handling */ + tmigr_update_events(group, child, data); + + if (group->parent && (walk_done == false)) + data->childmask = group->childmask; + + /* + * data->firstexp was set by tmigr_update_events() and contains the + * expiry of the first global event which needs to be handled. It + * differs from KTIME_MAX if: + * - group is the top level group and + * - group is idle (which means CPU was the last active CPU in the + * hierarchy) and + * - there is a pending event in the hierarchy + */ + WARN_ON_ONCE(data->firstexp != KTIME_MAX && group->parent); + + return walk_done; +} + +static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .nextexp = nextexp, + .firstexp = KTIME_MAX, + .evt = &tmc->cpuevt, + .childmask = tmc->childmask }; + + /* + * If nextexp is KTIME_MAX, the CPU event will be ignored because the + * local timer expires before the global timer, no global timer is set + * or CPU goes offline. + */ + if (nextexp != KTIME_MAX) + tmc->cpuevt.ignore = false; + + walk_groups(&tmigr_inactive_up, &data, tmc); + return data.firstexp; +} + +/** + * tmigr_cpu_deactivate() - Put current CPU into inactive state + * @nextexp: The next global timer expiry of the current CPU + * + * Must be called with interrupts disabled. + * + * Return: the next event expiry of the current CPU or the next event expiry + * from the hierarchy if this CPU is the top level migrator or the hierarchy is + * completely idle. + */ +u64 tmigr_cpu_deactivate(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (tmigr_is_not_available(tmc)) + return nextexp; + + raw_spin_lock(&tmc->lock); + + ret = __tmigr_cpu_deactivate(tmc, nextexp); + + tmc->idle = true; + + /* + * Make sure the reevaluation of timers in idle path will not miss an + * event. + */ + WRITE_ONCE(tmc->wakeup, ret); + + raw_spin_unlock(&tmc->lock); + return ret; +} + +/** + * tmigr_quick_check() - Quick forecast of next tmigr event when CPU wants to + * go idle + * @nextevt: The next global timer expiry of the current CPU + * + * Return: + * * KTIME_MAX - when it is probable that nothing has to be done (not + * the only one in the level 0 group; and if it is the + * only one in level 0 group, but there are more than a + * single group active on the way to top level) + * * nextevt - when CPU is offline and has to handle timer on his own + * or when on the way to top in every group only a single + * child is active and but @nextevt is before next_expiry + * of top level group + * * next_expiry (top) - value of top level group, when on the way to top in + * every group only a single child is active and @nextevt + * is after this value active child. + */ +u64 tmigr_quick_check(u64 nextevt) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_group *group = tmc->tmgroup; + + if (tmigr_is_not_available(tmc)) + return nextevt; + + if (WARN_ON_ONCE(tmc->idle)) + return nextevt; + + if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask)) + return KTIME_MAX; + + do { + if (!tmigr_check_lonely(group)) { + return KTIME_MAX; + } else if (!group->parent) { + u64 first_global = READ_ONCE(group->next_expiry); + return min_t(u64, nextevt, first_global); + } + group = group->parent; + } while (group); + + return KTIME_MAX; +} + +static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl, + int node) +{ + union tmigr_state s; + + raw_spin_lock_init(&group->lock); + + group->level = lvl; + group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE; + + group->num_children = 0; + + s.migrator = TMIGR_NONE; + s.active = 0; + s.seq = 0; + atomic_set(&group->migr_state, s.state); + + timerqueue_init_head(&group->events); + timerqueue_init(&group->groupevt.nextevt); + group->groupevt.nextevt.expires = KTIME_MAX; + WRITE_ONCE(group->next_expiry, KTIME_MAX); + group->groupevt.ignore = true; +} + +static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, + unsigned int lvl) +{ + struct tmigr_group *tmp, *group = NULL; + + lockdep_assert_held(&tmigr_mutex); + + /* Try to attach to an existing group first */ + list_for_each_entry(tmp, &tmigr_level_list[lvl], list) { + /* + * If @lvl is below the cross NUMA node level, check whether + * this group belongs to the same NUMA node. + */ + if (lvl < tmigr_crossnode_level && tmp->numa_node != node) + continue; + + /* Capacity left? */ + if (tmp->num_children >= TMIGR_CHILDREN_PER_GROUP) + continue; + + /* + * TODO: A possible further improvement: Make sure that all CPU + * siblings end up in the same group of the lowest level of the + * hierarchy. Rely on the topology sibling mask would be a + * reasonable solution. + */ + + group = tmp; + break; + } + + if (group) + return group; + + /* Allocate and set up a new group */ + group = kzalloc_node(sizeof(*group), GFP_KERNEL, node); + if (!group) + return ERR_PTR(-ENOMEM); + + tmigr_init_group(group, lvl, node); + + /* Setup successful. Add it to the hierarchy */ + list_add(&group->list, &tmigr_level_list[lvl]); + return group; +} + +static void tmigr_connect_child_parent(struct tmigr_group *child, + struct tmigr_group *parent) +{ + union tmigr_state childstate; + + raw_spin_lock_irq(&child->lock); + raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING); + + child->parent = parent; + child->childmask = BIT(parent->num_children++); + + raw_spin_unlock(&parent->lock); + raw_spin_unlock_irq(&child->lock); + + /* + * To prevent inconsistent states, active children need to be active in + * the new parent as well. Inactive children are already marked inactive + * in the parent group: + * + * * When new groups were created by tmigr_setup_groups() starting from + * the lowest level (and not higher then one level below the current + * top level), then they are not active. They will be set active when + * the new online CPU comes active. + * + * * But if a new group above the current top level is required, it is + * mandatory to propagate the active state of the already existing + * child to the new parent. So tmigr_connect_child_parent() is + * executed with the formerly top level group (child) and the newly + * created group (parent). + */ + childstate.state = atomic_read(&child->migr_state); + if (childstate.migrator != TMIGR_NONE) { + struct tmigr_walk data; + + data.childmask = child->childmask; + + /* + * There is only one new level per time. When connecting the + * child and the parent and set the child active when the parent + * is inactive, the parent needs to be the uppermost + * level. Otherwise there went something wrong! + */ + WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent); + } +} + +static int tmigr_setup_groups(unsigned int cpu, unsigned int node) +{ + struct tmigr_group *group, *child, **stack; + int top = 0, err = 0, i = 0; + struct list_head *lvllist; + + stack = kcalloc(tmigr_hierarchy_levels, sizeof(*stack), GFP_KERNEL); + if (!stack) + return -ENOMEM; + + do { + group = tmigr_get_group(cpu, node, i); + if (IS_ERR(group)) { + err = PTR_ERR(group); + break; + } + + top = i; + stack[i++] = group; + + /* + * When booting only less CPUs of a system than CPUs are + * available, not all calculated hierarchy levels are required. + * + * The loop is aborted as soon as the highest level, which might + * be different from tmigr_hierarchy_levels, contains only a + * single group. + */ + if (group->parent || i == tmigr_hierarchy_levels || + (list_empty(&tmigr_level_list[i]) && + list_is_singular(&tmigr_level_list[i - 1]))) + break; + + } while (i < tmigr_hierarchy_levels); + + do { + group = stack[--i]; + + if (err < 0) { + list_del(&group->list); + kfree(group); + continue; + } + + WARN_ON_ONCE(i != group->level); + + /* + * Update tmc -> group / child -> group connection + */ + if (i == 0) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&group->lock); + + tmc->tmgroup = group; + tmc->childmask = BIT(group->num_children++); + + raw_spin_unlock_irq(&group->lock); + + /* There are no children that need to be connected */ + continue; + } else { + child = stack[i - 1]; + tmigr_connect_child_parent(child, group); + } + + /* check if uppermost level was newly created */ + if (top != i) + continue; + + WARN_ON_ONCE(top == 0); + + lvllist = &tmigr_level_list[top]; + if (group->num_children == 1 && list_is_singular(lvllist)) { + lvllist = &tmigr_level_list[top - 1]; + list_for_each_entry(child, lvllist, list) { + if (child->parent) + continue; + + tmigr_connect_child_parent(child, group); + } + } + } while (i > 0); + + kfree(stack); + + return err; +} + +static int tmigr_add_cpu(unsigned int cpu) +{ + int node = cpu_to_node(cpu); + int ret; + + mutex_lock(&tmigr_mutex); + ret = tmigr_setup_groups(cpu, node); + mutex_unlock(&tmigr_mutex); + + return ret; +} + +static int tmigr_cpu_online(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + int ret; + + /* First online attempt? Initialize CPU data */ + if (!tmc->tmgroup) { + raw_spin_lock_init(&tmc->lock); + + ret = tmigr_add_cpu(cpu); + if (ret < 0) + return ret; + + if (tmc->childmask == 0) + return -EINVAL; + + timerqueue_init(&tmc->cpuevt.nextevt); + tmc->cpuevt.nextevt.expires = KTIME_MAX; + tmc->cpuevt.ignore = true; + tmc->cpuevt.cpu = cpu; + + tmc->remote = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + } + raw_spin_lock_irq(&tmc->lock); + tmc->idle = timer_base_is_idle(); + if (!tmc->idle) + __tmigr_cpu_activate(tmc); + tmc->online = true; + raw_spin_unlock_irq(&tmc->lock); + return 0; +} + +/* + * tmigr_trigger_active() - trigger a CPU to become active again + * + * This function is executed on a CPU which is part of cpu_online_mask, when the + * last active CPU in the hierarchy is offlining. With this, it is ensured that + * the other CPU is active and takes over the migrator duty. + */ +static long tmigr_trigger_active(void *unused) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + WARN_ON_ONCE(!tmc->online || tmc->idle); + + return 0; +} + +static int tmigr_cpu_offline(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + int migrator; + u64 firstexp; + + raw_spin_lock_irq(&tmc->lock); + tmc->online = false; + WRITE_ONCE(tmc->wakeup, KTIME_MAX); + + /* + * CPU has to handle the local events on his own, when on the way to + * offline; Therefore nextevt value is set to KTIME_MAX + */ + firstexp = __tmigr_cpu_deactivate(tmc, KTIME_MAX); + raw_spin_unlock_irq(&tmc->lock); + + if (firstexp != KTIME_MAX) { + migrator = cpumask_any_but(cpu_online_mask, cpu); + work_on_cpu(migrator, tmigr_trigger_active, NULL); + } + + return 0; +} + +static int __init tmigr_init(void) +{ + unsigned int cpulvl, nodelvl, cpus_per_node, i; + unsigned int nnodes = num_possible_nodes(); + unsigned int ncpus = num_possible_cpus(); + int ret = -ENOMEM; + + BUILD_BUG_ON_NOT_POWER_OF_2(TMIGR_CHILDREN_PER_GROUP); + + /* Nothing to do if running on UP */ + if (ncpus == 1) + return 0; + + /* + * Calculate the required hierarchy levels. Unfortunately there is no + * reliable information available, unless all possible CPUs have been + * brought up and all NUMA nodes are populated. + * + * Estimate the number of levels with the number of possible nodes and + * the number of possible CPUs. Assume CPUs are spread evenly across + * nodes. We cannot rely on cpumask_of_node() because it only works for + * online CPUs. + */ + cpus_per_node = DIV_ROUND_UP(ncpus, nnodes); + + /* Calc the hierarchy levels required to hold the CPUs of a node */ + cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + /* Calculate the extra levels to connect all nodes */ + nodelvl = DIV_ROUND_UP(order_base_2(nnodes), + ilog2(TMIGR_CHILDREN_PER_GROUP)); + + tmigr_hierarchy_levels = cpulvl + nodelvl; + + /* + * If a NUMA node spawns more than one CPU level group then the next + * level(s) of the hierarchy contains groups which handle all CPU groups + * of the same NUMA node. The level above goes across NUMA nodes. Store + * this information for the setup code to decide in which level node + * matching is no longer required. + */ + tmigr_crossnode_level = cpulvl; + + tmigr_level_list = kcalloc(tmigr_hierarchy_levels, sizeof(struct list_head), GFP_KERNEL); + if (!tmigr_level_list) + goto err; + + for (i = 0; i < tmigr_hierarchy_levels; i++) + INIT_LIST_HEAD(&tmigr_level_list[i]); + + pr_info("Timer migration: %d hierarchy levels; %d children per group;" + " %d crossnode level\n", + tmigr_hierarchy_levels, TMIGR_CHILDREN_PER_GROUP, + tmigr_crossnode_level); + + ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online", + tmigr_cpu_online, tmigr_cpu_offline); + if (ret) + goto err; + + return 0; + +err: + pr_err("Timer migration setup failed\n"); + return ret; +} +late_initcall(tmigr_init); diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h new file mode 100644 index 000000000000..6c37d94a37d9 --- /dev/null +++ b/kernel/time/timer_migration.h @@ -0,0 +1,140 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _KERNEL_TIME_MIGRATION_H +#define _KERNEL_TIME_MIGRATION_H + +/* Per group capacity. Must be a power of 2! */ +#define TMIGR_CHILDREN_PER_GROUP 8 + +/** + * struct tmigr_event - a timer event associated to a CPU + * @nextevt: The node to enqueue an event in the parent group queue + * @cpu: The CPU to which this event belongs + * @ignore: Hint whether the event could be ignored; it is set when + * CPU or group is active; + */ +struct tmigr_event { + struct timerqueue_node nextevt; + unsigned int cpu; + bool ignore; +}; + +/** + * struct tmigr_group - timer migration hierarchy group + * @lock: Lock protecting the event information and group hierarchy + * information during setup + * @parent: Pointer to the parent group + * @groupevt: Next event of the group which is only used when the + * group is !active. The group event is then queued into + * the parent timer queue. + * Ignore bit of @groupevt is set when the group is active. + * @next_expiry: Base monotonic expiry time of the next event of the + * group; It is used for the racy lockless check whether a + * remote expiry is required; it is always reliable + * @events: Timer queue for child events queued in the group + * @migr_state: State of the group (see union tmigr_state) + * @level: Hierarchy level of the group; Required during setup + * @numa_node: Required for setup only to make sure CPU and low level + * group information is NUMA local. It is set to NUMA node + * as long as the group level is per NUMA node (level < + * tmigr_crossnode_level); otherwise it is set to + * NUMA_NO_NODE + * @num_children: Counter of group children to make sure the group is only + * filled with TMIGR_CHILDREN_PER_GROUP; Required for setup + * only + * @childmask: childmask of the group in the parent group; is set + * during setup and will never change; can be read + * lockless + * @list: List head that is added to the per level + * tmigr_level_list; is required during setup when a + * new group needs to be connected to the existing + * hierarchy groups + */ +struct tmigr_group { + raw_spinlock_t lock; + struct tmigr_group *parent; + struct tmigr_event groupevt; + u64 next_expiry; + struct timerqueue_head events; + atomic_t migr_state; + unsigned int level; + int numa_node; + unsigned int num_children; + u8 childmask; + struct list_head list; +}; + +/** + * struct tmigr_cpu - timer migration per CPU group + * @lock: Lock protecting the tmigr_cpu group information + * @online: Indicates whether the CPU is online; In deactivate path + * it is required to know whether the migrator in the top + * level group is to be set offline, while a timer is + * pending. Then another online CPU needs to be notified to + * take over the migrator role. Furthermore the information + * is required in CPU hotplug path as the CPU is able to go + * idle before the timer migration hierarchy hotplug AP is + * reached. During this phase, the CPU has to handle the + * global timers on its own and must not act as a migrator. + * @idle: Indicates whether the CPU is idle in the timer migration + * hierarchy + * @remote: Is set when timers of the CPU are expired remotely + * @tmgroup: Pointer to the parent group + * @childmask: childmask of tmigr_cpu in the parent group + * @wakeup: Stores the first timer when the timer migration + * hierarchy is completely idle and remote expiry was done; + * is returned to timer code in the idle path and is only + * used in idle path. + * @cpuevt: CPU event which could be enqueued into the parent group + */ +struct tmigr_cpu { + raw_spinlock_t lock; + bool online; + bool idle; + bool remote; + struct tmigr_group *tmgroup; + u8 childmask; + u64 wakeup; + struct tmigr_event cpuevt; +}; + +/** + * union tmigr_state - state of tmigr_group + * @state: Combined version of the state - only used for atomic + * read/cmpxchg function + * @struct: Split version of the state - only use the struct members to + * update information to stay independent of endianness + */ +union tmigr_state { + u32 state; + /** + * struct - split state of tmigr_group + * @active: Contains each childmask bit of the active children + * @migrator: Contains childmask of the child which is migrator + * @seq: Sequence counter needs to be increased when an update + * to the tmigr_state is done. It prevents a race when + * updates in the child groups are propagated in changed + * order. Detailed information about the scenario is + * given in the documentation at the begin of + * timer_migration.c. + */ + struct { + u8 active; + u8 migrator; + u16 seq; + } __packed; +}; + +#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) +extern void tmigr_handle_remote(void); +extern bool tmigr_requires_handle_remote(void); +extern void tmigr_cpu_activate(void); +extern u64 tmigr_cpu_deactivate(u64 nextevt); +extern u64 tmigr_cpu_new_timer(u64 nextevt); +extern u64 tmigr_quick_check(u64 nextevt); +#else +static inline void tmigr_handle_remote(void) { } +static inline bool tmigr_requires_handle_remote(void) { return false; } +static inline void tmigr_cpu_activate(void) { } +#endif + +#endif From patchwork Wed Feb 21 09:05:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204039 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp919875dyc; Wed, 21 Feb 2024 01:11:22 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXTxUlcTJkhhtnXm4NLrNsKqG9bGTwbh4GgM36WGO6Wfcgqi/IMBShnECZ9t+zmnAkMLFHsBgzLZgvBSOYzpK1Qn4aAQA== X-Google-Smtp-Source: AGHT+IF/732jkSsJ+o6Q1ntIMswwW8zTgmzTksfPWnJV9253wFZBoiC04/CWR9Mc/A2/yo6qcPxz X-Received: by 2002:a05:622a:40b:b0:42e:1577:7980 with SMTP id n11-20020a05622a040b00b0042e15777980mr11298794qtx.1.1708506681809; Wed, 21 Feb 2024 01:11:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506681; cv=pass; d=google.com; s=arc-20160816; b=DYW5wY+rw8cwIagH+EsODcfnbe4o5z9Kbz+q8h60MRmIg0kGi61aWlENPOXPBdrxp5 wlgjCa6TwhKvkuKABjTPIEh/OHNsJqxv0+CGh8InatenSlsciwsM6AH4tlElQzIHHnZf XqpS+2hxJQXvraxhZg5mag1Xm6eZ38jvsms3Py4g33TRMacLZE38zF9hd4tW8w7E6Ehr YGQS62RkjkkouhvgTFQE4741wjXB2QALcIYPlf47rSBOTAAMenYzCa1Dd2yRIJ14BAxZ Bku/imIXqMJ0OSAmCy1mjjwitCqfe6P67c+CY3+h87epuwsXlnZP0WuogpEiFtZLCq1q lZ7Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=LfUPkbsUGavaFMAgteB30U/f1KDL95Bl2oCba8MYkuY=; fh=N5fTIFf2i7nvF8Om7kdprA8Zy2dleTNav/f0ArIISzo=; b=dOZN/tyj2OVuj0xqX9s7J3cnjv30t+ThX13Fu4gnFZKZEo7EU2GnmhIPXvS6NFbIPF wrlOP/BhQOj73j1+vE0GqA2fYUiv2gn9tYvogPCzzEIRZ685dMvt7Rjfr9BIEN+D7ZpE ONDV6HAUafRrW0TZbuwM7ER1a0tQIVSH+vOMQtz0P8wS4yOuP2i1kdwc2RS0pV+8mbzF JwCYu7nvIRlgv4lt1eVMnVKXn7IGYJjf20/EVLOn/IjYvrkzqqZ9Y0ecCbEiPAfuNRQG UGgVPCzt6FuM2erGa9tuuXNQNj2kr8geP33jMtLCMfy22i7Ziz+3WzlSszeYTs4iSLz2 1Kxg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="h/ybSua0"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74364-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74364-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id p9-20020ac87409000000b0042e286668fbsi3033969qtq.246.2024.02.21.01.11.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74364-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="h/ybSua0"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74364-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74364-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 8B8E01C2237C for ; Wed, 21 Feb 2024 09:11:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 501EF47F59; Wed, 21 Feb 2024 09:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="h/ybSua0"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ez2c8G4c" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A489E3E49B for ; Wed, 21 Feb 2024 09:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506384; cv=none; b=h/0sWz0X7+Pex+1oQgxO3zZka3RdRHaNchWl+O6WIT/YyPIBJ+EOc4mFyvDthUi/ThfOIg9mGkKBZkEJybezNtYLle11q6dzotAGhYZ/OUMLL1RTS4Xj9Fb7Hf3diJ+TF48cBZ1D09+GdrHUzf06ZVmG7umVgdHBhTsn0p5ETVI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506384; c=relaxed/simple; bh=lqzo/64LZxbO3BlWTISc+9iP7bdmS3XtmCo7nVHw3Y0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Vgpqcr2eLQftqGgQhwh6AAsOslseCARGzYcsrvTRo4L2XvrfGJxh8siKLyo3SOuebTNQgqDU6qzX631CvYAGNsAYpasXOW+rMBnCoORNrlAqDhtr2U1MNu0hZlPPoWDSVOL26NCARlcFOVHkmlsuM1y2DigN0G5up0Xy71daZIw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=h/ybSua0; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ez2c8G4c; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LfUPkbsUGavaFMAgteB30U/f1KDL95Bl2oCba8MYkuY=; b=h/ybSua0ryW5qplaASkFnIjwDLiYS4DLuYlT+Gw+pvmXJX4vCHw6KtrdZjVskhqP3xxY18 cTBNQdWRiSPn04i9mLC1axFzhImsh6L3/jrDGb04qZebRVcnAVujlebxvbnNIdnXPoHSjF YHGqiahTiExD6G+urhPOThwrG21RBpjB00XDrGbCFIuLCT5wYceI9AxASRD0EYYZQ00I9e 1On8NHkx7IA0R9DwbfIOPndbirxr2GP4IiaFKdN5nwrtct2NtWV3rN7BuHpfrhs2I/vjzc ThiXFrftqx13441HWBLy4ixGZPxVWDZGUhtQRc6yz7c/WHtAShu732AJmeSZgQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LfUPkbsUGavaFMAgteB30U/f1KDL95Bl2oCba8MYkuY=; b=ez2c8G4co/0Mq3cNqicCO5BHd/vN4zzIH0VhwLlrGV8saAmtkcV8lqyZHPZADLNdLKXaea Sn1ZtnnDv4hOesDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen Subject: [PATCH v11 19/20] timer_migration: Add tracepoints Date: Wed, 21 Feb 2024 10:05:47 +0100 Message-Id: <20240221090548.36600-20-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499102020024710 X-GMAIL-MSGID: 1791499102020024710 The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen Acked-by: Frederic Weisbecker --- v11: 's/numa_node/num_children/' when num_children value is assigned v10: Make an entry in MAINTAINERS file v9: Add tmigr_cpu_new_timer_idle tracepoint v8: Add wakeup value to tracepoints --- MAINTAINERS | 1 + include/trace/events/timer_migration.h | 297 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 26 +++ 3 files changed, 324 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/MAINTAINERS b/MAINTAINERS index 9ed4d3868539..70c07ae6e584 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17499,6 +17499,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core F: fs/timerfd.c F: include/linux/time_namespace.h F: include/linux/timer* +F: include/trace/events/timer* F: kernel/time/*timer* F: kernel/time/namespace.c diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..3f6e9502c41e --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->num_children = child->parent->num_children; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->num_children = tmc->tmgroup->num_children; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p wakeup=%llu", __entry->cpu, __entry->parent, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DECLARE_EVENT_CLASS(tmigr_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + __field( u64, wakeup) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + __entry->wakeup = tmc->wakeup; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu wakeup=%llu", + __entry->cpu, __entry->parent, __entry->nextevt, __entry->wakeup) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +DEFINE_EVENT(tmigr_idle, tmigr_cpu_new_timer_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index 9d6d5e5def5b..291cfa2bc08d 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -14,6 +14,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -658,6 +661,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ group->groupevt.ignore = true; + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -667,6 +672,8 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) data.childmask = tmc->childmask; + trace_tmigr_cpu_active(tmc); + tmc->cpuevt.ignore = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); @@ -828,6 +835,9 @@ bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child, data->firstexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, childstate, groupstate, + nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -862,6 +872,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + tmc->cpuevt.ignore = false; data.remote = false; @@ -903,6 +915,8 @@ static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = true; WRITE_ONCE(tmc->wakeup, KTIME_MAX); @@ -985,6 +999,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -1207,6 +1222,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_new_timer_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1285,6 +1301,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, */ WARN_ON_ONCE(data->firstexp != KTIME_MAX && group->parent); + trace_tmigr_group_set_cpu_inactive(group, newstate, childmask); + return walk_done; } @@ -1337,6 +1355,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) */ WRITE_ONCE(tmc->wakeup, ret); + trace_tmigr_cpu_idle(tmc, nextexp); raw_spin_unlock(&tmc->lock); return ret; } @@ -1453,6 +1472,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -1470,6 +1490,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irq(&child->lock); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active children need to be active in * the new parent as well. Inactive children are already marked inactive @@ -1561,6 +1583,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irq(&group->lock); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no children that need to be connected */ continue; } else { @@ -1628,6 +1652,7 @@ static int tmigr_cpu_online(unsigned int cpu) WRITE_ONCE(tmc->wakeup, KTIME_MAX); } raw_spin_lock_irq(&tmc->lock); + trace_tmigr_cpu_online(tmc); tmc->idle = timer_base_is_idle(); if (!tmc->idle) __tmigr_cpu_activate(tmc); @@ -1667,6 +1692,7 @@ static int tmigr_cpu_offline(unsigned int cpu) * offline; Therefore nextevt value is set to KTIME_MAX */ firstexp = __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); if (firstexp != KTIME_MAX) { From patchwork Wed Feb 21 09:05:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 204042 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp920022dyc; Wed, 21 Feb 2024 01:11:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUEvxX+GtYkaOh6O5x9vVe5J/YZ5x3dVZFnKieftdL8NglyJyxjq6/ykQyvdCBjYHRgXr1ROJaQcDmkInnVobEXfGL6JQ== X-Google-Smtp-Source: AGHT+IFFsexpD4fxAHQBC+j+tp3MgydTdw2BRHPOzub9JsjDkupxeTZ3Jae2f5ds3Oey2YHCs0JQ X-Received: by 2002:aa7:8a03:0:b0:6e4:62ff:d74 with SMTP id m3-20020aa78a03000000b006e462ff0d74mr7967940pfa.3.1708506700600; Wed, 21 Feb 2024 01:11:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708506700; cv=pass; d=google.com; s=arc-20160816; b=zcb3DQBbOtK/sagcvEtOW3g6d4h0Lo64JB19hbGeEAi64/lxphz+s6BakgnQ/JHZfy LPbqV3w3AgTMS2v+fIwgMHtW94lPARYlRge19HGbz9O+YEwUSPINEdlQB0+L6z0HQOz3 Y7QQisBN39RYE2Lh68mkWSj5uU2Hi//PqBJqPe+N0i31ydraTuzoVKazMyAYHk62/8AP 2+rLPaJ48kxDsuVnzb2pe30i2tKeTrxxBzqRyw+EGlRfWuM8CTOG2IBKmdhrKZqO+k58 7/MQ3sxVZBkNknDKWqr1rp7sakKnBC1ukab3/QKG7oIYe+fDDH8Qscz5zy6vLeaefAn7 l/tw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:dkim-signature:dkim-signature:from; bh=jeKqlLMkWK4hDy9uLHYSPeSLgMoip1gsYk9scrtIqMg=; fh=UyYALGSm//Y7venVD8hN44UibCXnYUm1mVNGtcl9feo=; b=sZCtqo3AgLqTLqka7cGfi3VS8QsAYugDWT6mpRu/KZfcgl9V4DFVe4+MayxyWTHHAi EdpslQDvD2UjxHgP5mezeKKeZgoNWj4OVr67XkXPHSc1sLPpn59NtS6fxu6N5QWB6xc2 xVZyoKS3Sfu1IXP4wkj7ONNx6YXddbYDrU7Nicas7pdPwpqjv08FuvdL5lx6nTRaX4Sr sRTwhj9RECe3UYrHc2aneJQlAtxSeMKMjaKYRHcSXq43ocy3KqNEnpT83mnqqligbETk yHdhXOmEdx2zlCPwEwfyYuc3ca7YyomCbk3CsRC7PumaCZSMAtx06zRp/PqDaqWx2Trf wxAg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=i+8YJTPe; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74363-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74363-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id cn9-20020a056a00340900b006e4892148d1si1463362pfb.355.2024.02.21.01.11.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 01:11:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74363-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=i+8YJTPe; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-74363-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74363-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id CF601281616 for ; Wed, 21 Feb 2024 09:11:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5F02147A6B; Wed, 21 Feb 2024 09:06:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="i+8YJTPe"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Wa8EvOGU" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A48CD3E49C for ; Wed, 21 Feb 2024 09:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506384; cv=none; b=DC8SHdd5AD9acPcZCX3x3LmXd9f9smqbKeLgDZpLqd3RtHRVvNrnPBJgdcTEo0vwqI6q+MfpXb0eVF8saUkWGYlus1t3lOWZT4GJP0+bb7qiQ90XzBLQFY+KxCDpXVZ+CdNDMRzpK0KZZukvpaveQIXqb/LPAa/vsPEGbRYnbFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708506384; c=relaxed/simple; bh=KMdGvqAqNunk9f49u3SAzHLnc4ZmsFKul3Ua8+evWDE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TYAlJ2PEJuSOHDQpax3LCPfcPF95x0MytOyqmdhoSME6E5jk6n+OGGYnIXhWS01HkLUa/QZKNbX5iqZBw8BKgrorzyeClYCE8v1rFmw+jYQTkkDLFe+yzIf+T2J7ixVg6BzpvPtfMKZI1s2fdk7jdpNpEtBUpqw4DrlP1ESSI+Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=i+8YJTPe; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Wa8EvOGU; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708506379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jeKqlLMkWK4hDy9uLHYSPeSLgMoip1gsYk9scrtIqMg=; b=i+8YJTPebGk1WVJrX3XNgQONZjwIY4Iv+gVXLqBX3z9f92R5yM3rcIeT1GxvVOu7NMJsDh 1EqBTECCn/SA2aXJ4DuGeyBCiMKDDkuzLMxUCrihkQFAuSHdYbqLg1stINWsMzBLfeossT EygREtcXqgdsDqoySEvmKUTVwoIbv2CoFzGNWME6pv6mBZRrP+QpWE8TnrKcmM+/6kC7Oh LH0UdZiCoAvTbsQ3vl8nCRa+JnbnFDZE1CZd3bYfgMeRJ1/6/sMihg6cYKIbTmq2CAz7CM qBIEHIHwjT+sqtonq2A7kRT1S7dUKNNKYLqu0Ht0gj1SMHFoPalgZ7izku8SnA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708506379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jeKqlLMkWK4hDy9uLHYSPeSLgMoip1gsYk9scrtIqMg=; b=Wa8EvOGUcNwj0imH5TRhabVRJOouUi/urXpTv/E0Us3Ng/dWEGrES3E7V+Zb34Rg0Qmb8S DwzAVjq+mL7DGLAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Srinivas Pandruvada , K Prateek Nayak , Christian Loehle , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v11 20/20] timers: Always queue timers on the local CPU Date: Wed, 21 Feb 2024 10:05:48 +0100 Message-Id: <20240221090548.36600-21-anna-maria@linutronix.de> In-Reply-To: <20240221090548.36600-1-anna-maria@linutronix.de> References: <20240221090548.36600-1-anna-maria@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791499121848999410 X-GMAIL-MSGID: 1791499121848999410 The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the separate storage and eventually pulled at expiry time to a remote CPU. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v11: - drop check for TIMER_PINNED flag and replace it with a WARN_ON_ONCE() v9: - Update to the changes of the preceding patches v6: - Update TIMER_PINNED flag description. v5: - Move WARN_ONCE() in add_timer_on() into a previous patch - Fold crystallball magic related hunks into this patch v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 14 ++++---------- kernel/time/timer.c | 36 +++++++++++++++--------------------- 2 files changed, 19 insertions(+), 31 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 2be8be6dd317..14a633ba61d6 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -36,16 +36,10 @@ * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. - * - * Note: Because enqueuing of timers can migrate the timer from one - * CPU to another, pinned timers are not guaranteed to stay on the - * initialy selected CPU. They move to the CPU on which the enqueue - * function is invoked via mod_timer() or add_timer(). If the timer - * should be placed on a particular CPU, then add_timer_on() has to be - * used. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the + * timer was enqueued. When a particular CPU is required, add_timer_on() + * has to be used. Enqueue via mod_timer() and add_timer() is always done + * on the local CPU. */ #define TIMER_CPUMASK 0x0003FFFF #define TIMER_MIGRATING 0x00040000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0339273f9365..9ca921383497 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -590,11 +590,16 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle) { + WARN_ON_ONCE(!(timer->flags & TIMER_PINNED)); wake_up_nohz_cpu(base->cpu); + } } /* @@ -941,17 +946,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void __forward_timer_base(struct timer_base *base, unsigned long basej) { @@ -1106,7 +1100,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -2252,7 +2246,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem, * granularity skew (by design). */ if (!base_local->is_idle && time_after(nextevt, basej + 1)) { - base_local->is_idle = base_global->is_idle = true; + base_local->is_idle = true; trace_timer_base_idle(true, base_local->cpu); } *idle = base_local->is_idle; @@ -2318,13 +2312,13 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle) void timer_clear_idle(void) { /* - * We do this unlocked. The worst outcome is a remote enqueue sending - * a pointless IPI, but taking the lock would just make the window for - * sending the IPI a few instructions smaller for the cost of taking - * the lock in the exit from idle path. + * We do this unlocked. The worst outcome is a remote pinned timer + * enqueue sending a pointless IPI, but taking the lock would just + * make the window for sending the IPI a few instructions smaller + * for the cost of taking the lock in the exit from idle + * path. Required for BASE_LOCAL only. */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); - __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); trace_timer_base_idle(false, smp_processor_id()); /* Activate without holding the timer_base->lock */