From patchwork Fri Nov 4 14:57:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15613 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453612wru; Fri, 4 Nov 2022 07:59:04 -0700 (PDT) X-Google-Smtp-Source: AMsMyM53FXxqlFGioYfenCBoqWRAqzIGr9yIWoA+KL4kcu9JefBEOfGzhUsF9GNusnFHBNef/Z1/ X-Received: by 2002:a17:906:2bc7:b0:72f:dc70:a3c6 with SMTP id n7-20020a1709062bc700b0072fdc70a3c6mr35596546ejg.645.1667573943973; Fri, 04 Nov 2022 07:59:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573943; cv=none; d=google.com; s=arc-20160816; b=ymFe5LCoFiL6H2av542FvxnMfNxV4yiSNSUmR3z1gRN8mYPdsxQgPS2Ai0KZseZSld pvLZTF5k/sQ8hMDRbQnd63iHhJmeIWZubh53z1DN1cg8qDOO84GMjmgpkEtg98POnClX rfrpoxwX+CuM/xM3oZjn+IYxCOhzcVMhZVhDHBkGbtGYweddPHhjhiZcf8YAZ7B3XwaR yctywOxk3WTYeXU8uZG/ftm21Ezi/6LMcDCDS1JtsYSXJ798eh9ZQqd1x3xXPoUCeV4l +THCC7t7DCSurc5AwiBXTZK67wt/edNG704qh5h3U3S9RdRV4ZI6JM/6u1miXagXrsdK MjUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=BOCDJ0nGoVQscn5FVoqed4VCDME+xrNe/Iu7/rdK+w0=; b=jh8wyDvLojO2fxI8R0H5LbttR8+k0N6pirXYm7X9skX79fi53H2+RSOEBH60AXfoYr 8unyUMeTtKQ9txvZqyp2eAKZpo9vc6mRE8to8wk4BIS7aqSjVIUiDVXRMNjEA2pToJPd vTYQa+EpKgeCpbDFLLd7vRX3QcIm35JXpEfq+eODAOCf+uDxfUdO5cHyQFPRlAMGO6wk niS1U2PeRc+CNhPGcUuAiYFgjZF9zq/aPJ+AQvNKNVEX2k9nK8Im0S11JG4H5MumT7qJ 7dapXtiCVOEdOOSYB7WGhSU7YvtbCwja9oId6te1luXqDgRqQBOF/SB59g6Dhmebemtk cWPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=VlCDY+iL; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f24-20020a056402151800b0045d523bee48si4504660edw.178.2022.11.04.07.58.39; Fri, 04 Nov 2022 07:59:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=VlCDY+iL; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232291AbiKDO6L (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231549AbiKDO55 (ORCPT ); Fri, 4 Nov 2022 10:57:57 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3057F2EF33 for ; Fri, 4 Nov 2022 07:57:55 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BOCDJ0nGoVQscn5FVoqed4VCDME+xrNe/Iu7/rdK+w0=; b=VlCDY+iLQ1c9Xo1HDq0mYn612Fpuhq7nD0iZqweE6V81oNaB6DPVYXlePKsWJ6PQMAPcSV PnYMxkcR/YdR2y793jGdN5YSVM5g8XRSRddtsJusdYhgOR5htE6idsk0sSvUGDkN3a5h2t sMYWhhIso3zjRlc0Fq5ZLakCM6uHmpo37uEX7JHF6HVh+jxwPQelTgHJnEei9781vejUrC 07zov+nDBNEx8sSvQyd04XnXne+yoVQnU6/Gl8ExisRc8leW9oF8Ssb5F46CYwOLaGFJAs 98EodKCY7iS3dzmpttSgSRGF/fEuZjQdL86R5rWsbXpG4PKePAQmhWitI9yH7w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BOCDJ0nGoVQscn5FVoqed4VCDME+xrNe/Iu7/rdK+w0=; b=OghQyD2QYWXdAbcVu7hxOV1er79fWzh+jW//Lpvc8U3bgXcuKperpThSfh9X+TIHUC02OK 0NfnPLt8UNMIFdAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v4 01/16] tick-sched: Warn when next tick seems to be in the past Date: Fri, 4 Nov 2022 15:57:22 +0100 Message-Id: <20221104145737.71236-2-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578015855712999?= X-GMAIL-MSGID: =?utf-8?q?1748578015855712999?= When the next tick is in the past, the delta between basemono and the next tick gets negativ. But the next tick should never be in the past. The negative effect of a wrong next tick might be a stop of the tick and timers might expire late. To prevent expensive debugging when changing underlying code, add a WARN_ON_ONCE into this code path. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..7ffdc7ba19b4 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -826,6 +826,8 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * If the tick is due in the next period, keep it ticking or * force prod the timer. */ + WARN_ON_ONCE(basemono > next_tick); + delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { /* From patchwork Fri Nov 4 14:57:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15612 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453529wru; Fri, 4 Nov 2022 07:58:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6wyY1mEfH7HR9e6vkjQDCGZ91Bn8ACgzDUj+bRtnhoS5hW9j5weZck8kbI/JawKINKtiD7 X-Received: by 2002:a05:6402:847:b0:45c:c716:4b76 with SMTP id b7-20020a056402084700b0045cc7164b76mr36132116edz.113.1667573935653; Fri, 04 Nov 2022 07:58:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573935; cv=none; d=google.com; s=arc-20160816; b=yb0dKhVBr9t0fBCOq/nmKqOlh8MHklEofshC/NQL8+q1c51xy8MT9EztnNAdSqXCl/ vBKwpjfULCC9x7fT4RZoLPYIrxbYzspiH/p7iDC2sytLCcjllbHiHr5riz2txpCjG+i+ 8ONPTfYwYemQmyGHEcqFDT/9oj/CMmvMWs2gJtHBC1wArIwHRnNSVBKsFI2tdWGi/8Y+ QrSE3PJzCzQHEb0CYu8ckWE6Ro9NWH+VfnYxEkiqzYR1EANuMk/WEjT1C1YjPEdu70B2 0EL/ICUg5H4oYZmX0tj2ftP9DYX0KvdJEx0yPIELvzFt3DjYwE+kry44cndvPZfqbnsn lWvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=5SYFJLwFPZaNWscBGLUeDFCSexiFqWz1WiWI/WoTB14=; b=X4irYEBHZNySIiTbnFouDCyP29xMyDkqwh2RdDI4+g1SxufanK/7L9xp0BVRSqBjRx 47/Lgk0GKDFhEW6JI67bjwZ6ED1R1BQoOyk1UJfCK3Ld3s6jbVLnks7v9KVG6nHn8YDY a6zaiWXBF0iX8CN6DfnuoQDYX1jYyjta8JkVCPbYC0jQ327lgYRKOJyX3zS1LPjO69yo HpN517SIz1BtdYKHrDVNvmDqA255cKbCht1r+kZ/w1E5f4PjOM3jvVwx6hNT2GkvH8EB 0/6zSCa0g8kSsZ0O77jgxY1uHiBkqIyqLJ4i+pC0ihJ4vabtVgF2KcO7P0Fn0hzRzq4N CmOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=HepTyuV7; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j14-20020a05640211ce00b00458ac23f0b8si6177845edw.399.2022.11.04.07.58.31; Fri, 04 Nov 2022 07:58:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=HepTyuV7; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232199AbiKDO56 (ORCPT + 99 others); Fri, 4 Nov 2022 10:57:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbiKDO54 (ORCPT ); Fri, 4 Nov 2022 10:57:56 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3042A2EF24 for ; Fri, 4 Nov 2022 07:57:55 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5SYFJLwFPZaNWscBGLUeDFCSexiFqWz1WiWI/WoTB14=; b=HepTyuV7cID1C6WhAFE5nDOjWovpqnf2mAD+QoOGGtwSB2pzdpjQpFbtanayuzYfIxZBlj DpjP8aKCE+zvmVQ7CEQRmzL3mQfgMC8fOS+nNb7FCdGVHb3PgPEOKF7TAzFpYQtHWtZuKC FTHIXytnit5sNnDqSc7ulMD9nSDY/bOydhleEbiRAYfLgoM0T7II3MAsghJGPVlz18fK37 r7UCrr6jiNtGfBAjzjgBrjdsuFKWlcNcD5/IiyyQ8TDuAJhUEsTmjDdjjjcxllUt829kyr +iVxsJiyAMhktP2JMtuqtNIcF0TZ4kKKobS4qg4SMfbSm5fzWRtdZufUlc3UoA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5SYFJLwFPZaNWscBGLUeDFCSexiFqWz1WiWI/WoTB14=; b=y36jNLIMEgsBqxw3naxkOK6Uo0wMHAOxboaACW0IAppPXCX8uFHWUmBI9B1Zl7G/MZG7aj FA/TKfCCFzaIsyCw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 02/16] timer: Move store of next event into __next_timer_interrupt() Date: Fri, 4 Nov 2022 15:57:23 +0100 Message-Id: <20221104145737.71236-3-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578006934558106?= X-GMAIL-MSGID: =?utf-8?q?1748578006934558106?= Both call sides of __next_timer_interrupt() store return value directly in base->next_expiry. Move the store into __next_timer_interrupt() and to make purpose more clear, rename function to next_expiry_recalc(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner --- v4: rename function as suggested by Frederic Weisbecker --- kernel/time/timer.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 717fcb9fb14a..2fd5de648d7c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1571,8 +1571,10 @@ static int next_pending_bucket(struct timer_base *base, unsigned offset, /* * Search the first expiring timer in the various clock levels. Caller must * hold base->lock. + * + * Store next expiry time in base->next_expiry. */ -static unsigned long __next_timer_interrupt(struct timer_base *base) +static void next_expiry_recalc(struct timer_base *base) { unsigned long clk, next, adj; unsigned lvl, offset = 0; @@ -1638,10 +1640,11 @@ static unsigned long __next_timer_interrupt(struct timer_base *base) clk += adj; } + base->next_expiry = next; base->next_expiry_recalc = false; base->timers_pending = !(next == base->clk + NEXT_TIMER_MAX_DELTA); - return next; + return; } #ifdef CONFIG_NO_HZ_COMMON @@ -1701,7 +1704,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base->lock); if (base->next_expiry_recalc) - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); nextevt = base->next_expiry; /* @@ -1784,7 +1787,7 @@ static inline void __run_timers(struct timer_base *base) WARN_ON_ONCE(!levels && !base->next_expiry_recalc && base->timers_pending); base->clk++; - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); while (levels--) expire_timers(base, heads + levels); From patchwork Fri Nov 4 14:57:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15614 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453697wru; Fri, 4 Nov 2022 07:59:13 -0700 (PDT) X-Google-Smtp-Source: AMsMyM44mAI4TTCnb4+HR61RWwyg97dzcU1cjwMnhU6BNR/vmogT+Ckt+8vpnj3ZbDh7vMoqxAy6 X-Received: by 2002:a17:907:1de6:b0:7a5:ea4b:ddbb with SMTP id og38-20020a1709071de600b007a5ea4bddbbmr35143281ejc.757.1667573953288; Fri, 04 Nov 2022 07:59:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573953; cv=none; d=google.com; s=arc-20160816; b=jLqoXqlfGD5SjRYasmjKN7qB834/f03fPjgwI6VCXzQ+kjbKEobarbuvNs0wXSt+6D YAoMkZA9+jBsVupxsZwpsGYB0f1CymNclR7RLxJjcgdPb6d1IX3+F4jwAYYecj4+8Nyn DlNmCYeAGPU+s3/fi6ukSLtrbj6fRjmVCFqdaLD3dgq/yTEQthUflx1mdRe5NHtRWt+/ Ay1rVxhBm0N/baCbiLv83vRWH3vxMlzh3qiBKv89a8Cw/W4lqozW9RLwbos2t4JfjCxR 8Cb0SNi5CF48wyeIJuOSpI7NdRR3x1S+QtDO/FW5veosJzREvtmRNs7s3JzP10JNo7VF Gk4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=TfKpQBhTGidhSKvCEygfZZ4AX4EB6wM0N8LXrGnUD24=; b=USlQCzqAcl12sSJLJo1IxfcSELrBpA24EvTuUOxe68HJvNTSy4l8FGGDxUjJphyfHW 2NEEXh8fqkscTEQD9PUaZzgzoJlUHdDstnIcfEthJjisCP11gNcLI7ZLaWOWEHvYPKnN W9SNoBQKS7hhB8KdRo3HY9Cb3tIlU4Hyd7VE0dKaaBxixbBJpg8FRTuKC3hyrOI3g+AN jsnVgaZyE2OqG4XmYpRccizCeWMDS8z784k6kihSb9t6El3szMZqlwQqKeTtzO+HROH3 DMWkfhfGHGQ9aaSIWX5pV8ccfbEo3WIUyCn6Jvh1D4MajZw10LMTSZsZJXH58wRgoOiL TZXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Li15vs6H; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u1-20020a1709064ac100b0077cfec3a52fsi3891705ejt.839.2022.11.04.07.58.49; Fri, 04 Nov 2022 07:59:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Li15vs6H; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231903AbiKDO6R (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231901AbiKDO55 (ORCPT ); Fri, 4 Nov 2022 10:57:57 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A76DA2EF67 for ; Fri, 4 Nov 2022 07:57:55 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TfKpQBhTGidhSKvCEygfZZ4AX4EB6wM0N8LXrGnUD24=; b=Li15vs6HYf2iWUIGpE4I1OaOJmwT7R+ogU+5n+djOv+2u2zdPW7B4EQp6vAiS9RuJOlFId enH2ahXusKWnGN+ALobtamjmrupAMOAXNBXnXu1CCxGlzPKMFvh7/+It5kKUnLPh6aSTTE Yzu+y9b7+n74NxalFS69SPWHKNaM2UOVzOxu0irdhf7EDkw5YfHQ/U3O4/QmBXzUghf0t4 wcq+r9MBiobk9xeDuxt/QzBrCxR6DkuVrW4uET+VO1ffPunu8QrkIvV5SXzmQrW4z7yO2u DqtONqF9xIhaTzOodPOx2f1VxME4m8tTvzsX36KicLE5b0T86UdFKXy/Jh4MHQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TfKpQBhTGidhSKvCEygfZZ4AX4EB6wM0N8LXrGnUD24=; b=/ldAXVLBpjPFC9LLcqYeRHoQ5QvVVt44qSu/1rWNPnSiE+Cp1FRl8h2wIXyzbaE1eR9iWy wE8xaPJVKR6jhuAg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v4 03/16] timer: Split next timer interrupt logic Date: Fri, 4 Nov 2022 15:57:24 +0100 Message-Id: <20221104145737.71236-4-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578025679528303?= X-GMAIL-MSGID: =?utf-8?q?1748578025679528303?= Logic for getting next timer interrupt (no matter of recalculated or already stored in base->next_expiry) is split into a separate function "next_timer_interrupt()" to make it available for new call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 2fd5de648d7c..4547d2efcb86 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1681,6 +1681,15 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } + +static unsigned long next_timer_interrupt(struct timer_base *base) +{ + if (base->next_expiry_recalc) + next_expiry_recalc(base); + + return base->next_expiry; +} + /** * get_next_timer_interrupt - return the time (clock mono) of the next timer * @basej: base time jiffies @@ -1703,9 +1712,8 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return expires; raw_spin_lock(&base->lock); - if (base->next_expiry_recalc) - next_expiry_recalc(base); - nextevt = base->next_expiry; + + nextevt = next_timer_interrupt(base); /* * We have a fresh next event. Check whether we can forward the From patchwork Fri Nov 4 14:57:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15615 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453786wru; Fri, 4 Nov 2022 07:59:21 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4QMM+uQwLco+QWe0gEx79QcvfYF6bVtS6qsfPoalT573uKffSUnH4DKRyGwUHSVB7Cvq2k X-Received: by 2002:a17:906:5dda:b0:78d:ed96:2127 with SMTP id p26-20020a1709065dda00b0078ded962127mr33518056ejv.373.1667573961145; Fri, 04 Nov 2022 07:59:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573961; cv=none; d=google.com; s=arc-20160816; b=c+/yiu7Hz5zy1WKYbDxM29/kyfSFiyjW29zHDmlMPC6+jyV8Q2pB6Al3hgMa81Ku4O q4Gk04K4NCyNnA4/FEiP4yHFITx7IJ9naqm/zj7sJ9dG7KkRHBEFPZqLTgNT/6YabkF0 VxXdt49Ce5wOJ+OOp6W6vDNmSukqFWEEpTp3/GFimuHl00zvfjomcG4TKNL7nH8WWxyw BQbP77jeCk+uMZn2OaAZmOO84cFJESVgw6g1T+CLatJKI2URu1fYYXj4I42AFwxQnDR5 SUUmWOXtXOGEWwyeEssej8QUhJn2G6HtzLouD0isoLC7RRtGngQxUUPCRZA/0CNUFo7r xoWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=FruOUvpKeK+X7swaFNp01mmQJvyOWTcwLkHl8M9RVAo=; b=e66u/81V6qmZGjcGTkliXmUHHAHeRWKNicT0hNjsBmkwu4t8lG4TV1gO5Ow2IcLVyS GJ+eGOC21qpgCrL9BNNdNhFSTIGClUnRMYuDr77uwRZGx1OVG1K+Uk8H7Djcn+v8px/a apyNmFU4Zq1rYkCD67Z73/GBwWBBgSNPduaPTbamTrFUs22reuARfvnncQTBP/yEJOn2 k0jt2dOwHIFBa0vfV4o1M/kGCHgNeskIQvCMd/l2k39CMUZs2C30+SpD/H76tBplKEK+ 5fTbbreosTuT9qp2Xa/kNX53icgDi79QuAhzjsb0uoqfh7YL/1F75pgh3ph1J5cMSwb3 O2zA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WHasSckl; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=2wS8GEB4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dr7-20020a170907720700b0073d8830e4c7si5648272ejc.954.2022.11.04.07.58.57; Fri, 04 Nov 2022 07:59:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WHasSckl; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=2wS8GEB4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231948AbiKDO6W (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231814AbiKDO55 (ORCPT ); Fri, 4 Nov 2022 10:57:57 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AB4E303C4 for ; Fri, 4 Nov 2022 07:57:56 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FruOUvpKeK+X7swaFNp01mmQJvyOWTcwLkHl8M9RVAo=; b=WHasScklTX2l+M94cuiAXTiHWhyYQZKUYPXN+0WrU04i3B2wIeHRAL7p5P2Vd0qKsdWBmJ NTXHiGgqYviBgWE/hJeIxrxiUxrEBTxr6ji/Cc9zINbCyDMl7QTV71eU7yob/aLskz2qJk pSg+OMtjNkwgInWSfJsvF8kywdnPzkpjbhy7dqwNc5RR5d3Uldy3cSDaR9TIuBYT0xvhDN HK1IuK6I6a1qeJcc8OrKZ9ESrEiUuavv2PjdjyX+g/KcBf4A3Pwv4GKupUFUe69rK0F3Zy Lsr0HrkqYijt+h6onL8LKHZ1AmbwfzpMVdHxezK1nGyujGqAnyP5zKKEdgkkpw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FruOUvpKeK+X7swaFNp01mmQJvyOWTcwLkHl8M9RVAo=; b=2wS8GEB4PEbBXo6txsaCfHEF1g+vJEVKaQl6R3/fMHiP+lPDtKGvgPQJ+ghQetYhnuWU5z QFpqJjEwwkTYl8Dw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v4 04/16] timer: Rework idle logic Date: Fri, 4 Nov 2022 15:57:25 +0100 Message-Id: <20221104145737.71236-5-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578033950705500?= X-GMAIL-MSGID: =?utf-8?q?1748578033950705500?= From: Thomas Gleixner To improve readability of the code, split base->idle calculation and expires calculation into separate parts. Thereby the following subtle change happens if the next event is just one jiffy ahead and the tick was already stopped: Originally base->is_idle remains true in this situation. Now base->is_idle turns to false. This may spare an IPI if a timer is enqueued remotely to an idle CPU that is going to tick on the next jiffy. Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v4: Change condition to force 0 delta and update commit message (Frederic) --- kernel/time/timer.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4547d2efcb86..0e9f609f5a38 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1727,21 +1727,20 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) base->clk = nextevt; } - if (time_before_eq(nextevt, basej)) { - expires = basem; - base->is_idle = false; - } else { - if (base->timers_pending) - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - /* - * If we expect to sleep more than a tick, mark the base idle. - * Also the tick is stopped so any added timer must forward - * the base clk itself to keep granularity small. This idle - * logic is only maintained for the BASE_STD base, deferrable - * timers may still see large granularity skew (by design). - */ - if ((expires - basem) > TICK_NSEC) - base->is_idle = true; + /* + * Base is idle if the next event is more than a tick away. Also + * the tick is stopped so any added timer must forward the base clk + * itself to keep granularity small. This idle logic is only + * maintained for the BASE_STD base, deferrable timers may still + * see large granularity skew (by design). + */ + base->is_idle = time_after(nextevt, basej + 1); + + if (base->timers_pending) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } raw_spin_unlock(&base->lock); From patchwork Fri Nov 4 14:57:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15626 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456672wru; Fri, 4 Nov 2022 08:03:36 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5kAKNOQumG6PMIfIcMOlS/AZkiGB2wgNhRINrCymjKTaSbg22e+PyjClss47QPKUWmFdRf X-Received: by 2002:a17:906:6a27:b0:7a6:c537:ba4 with SMTP id qw39-20020a1709066a2700b007a6c5370ba4mr33318777ejc.517.1667574216258; Fri, 04 Nov 2022 08:03:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574216; cv=none; d=google.com; s=arc-20160816; b=gA9EY4q2ohtYlT59CW9tIWCCo2YR6Toh6ZmJCQC5mM8YdnSOzZf25LvRO15SQ2IjQp JlGMo2149FAqnYhP/zH7C/DlMNOeBFU0Q7dgaDYyJrinxxxHJzHxPkRAkbBAI45LEE4+ 5Z5Dx8jRiSanb9m67VHtlj+2SrwRHwRiYOTjVd0r5cFNpPAYNMuWeyNeOxtjuYUyK2I8 /fBAl7SVa2J8adsnd2dSWGhdIpIkzRua15Dp1yEibK6ye5eOy0p95TsL7xG7JU2qFs97 yH/CrTxLGRPCyhSG4a5SG9SCfBcUlK+8/9SJlRT8et+VxJm3hXDh7oj2gXw+cPTALXbD RODw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=SJQXT73M5EqpgN/OaScjUbemA9h45EuN/DjSPRp7+B8=; b=lko5gW1klxKHxhWOJGl/AD5tWg7nLGTOP75gst9z1vPZUpUhg4dXBnmiyPHkAi+MJ8 4zlzwNwq4EpYWQMzZjcA9GENnyYkKqfOf8QB0xMpiD89vghh/2OFRCNWcrdv6s5kzJDz tuSWpHTz5ptm40eWDraZT86ngzXiq0KAAbVXLEKBnb5JbIOEDSZ7SAl/RIoVmCNiK2pQ f3x70mUlB4m0v+DFRMLiOLBfr9TnuEVs+B3nHTFCkUDKPNECmzZrdtSR1qpI0LvvVs4L gT1njcIDAFp5psn/GclxtYMOUWwlm0sbW3GaP8rQUvlkCy0HwNzzo0rGj8V57QrObaOA 2W7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=opSSMs8+; dkim=neutral (no key) header.i=@linutronix.de header.b=Qw00fyiu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d14-20020a50f68e000000b004608c0b9a8asi4956463edn.201.2022.11.04.08.03.10; Fri, 04 Nov 2022 08:03:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=opSSMs8+; dkim=neutral (no key) header.i=@linutronix.de header.b=Qw00fyiu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232298AbiKDO6t (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232263AbiKDO6C (ORCPT ); Fri, 4 Nov 2022 10:58:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C7B52ED53 for ; Fri, 4 Nov 2022 07:57:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SJQXT73M5EqpgN/OaScjUbemA9h45EuN/DjSPRp7+B8=; b=opSSMs8+INQWcYQu5UHb3U05ouat0+yID7SNf3MMsxOGnyiItb/TVF8NFamERP/PRs9WS/ QWiFzEGdPN/gDJwLu1w2iZ2XRZ8GRyJn3I3oCyzc/Nl/PYbJMmTnoanSIQTN+fBTorjgoK dbJVSy4yNQxYIFHfHhUIbtR+XyXGAKEyBdg+DvZLEXBIPBgMXpta3mEiCbv2hUXANnRDxm +ZJ264kASAAprhV9amckn3jET3X8I/cL32JGSblkbgX6pHJ4hb5j3SHni7EPbfMn6tA5PZ 88Tk1AUB2GexmeMPAEmKJpEL57n6qdZFJHtdUx4rlzq7frqsvT/zcOLSnB2Xsg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SJQXT73M5EqpgN/OaScjUbemA9h45EuN/DjSPRp7+B8=; b=Qw00fyiuSOq+8nMDyzrrq/BLOY+V56inCGAMwhxbouDigpsmgPMJJjh8IN+Ez30tZDJ+3x c55XuXSl6NCE/HCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , "Theodore Ts'o" , "Jason A. Donenfeld" , Stephen Boyd , Tejun Heo , Lai Jiangshan Subject: [PATCH v4 05/16] add_timer_on(): Make sure callers have TIMER_PINNED flag Date: Fri, 4 Nov 2022 15:57:26 +0100 Message-Id: <20221104145737.71236-6-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578301556558906?= X-GMAIL-MSGID: =?utf-8?q?1748578301556558906?= The implementation of the hierachical timer pull model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. Otherwise they will be queued on the dedicated CPU but in global timer base and those timers could also expire on other CPUs. Timers with TIMER_DEFERRABLE flag end up in a separate base anyway and are executed on the local CPU only. Therefore add the missing TIMER_PINNED flag for those callers who use add_timer_on() without the flag. No functional change. Signed-off-by: Anna-Maria Behnsen Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Cc: "H. Peter Anvin" Cc: "Theodore Ts'o" Cc: "Jason A. Donenfeld" Cc: John Stultz Cc: Stephen Boyd Cc: Tejun Heo Cc: Lai Jiangshan --- v4: - Move patch before local and global base are introduced - Add missing user (drivers/char/random.c) of add_timer_on() without TIMER_PINNED flag (kernel test robot) --- arch/x86/kernel/tsc_sync.c | 3 ++- drivers/char/random.c | 2 +- kernel/time/clocksource.c | 2 +- kernel/workqueue.c | 7 +++++-- 4 files changed, 9 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/tsc_sync.c b/arch/x86/kernel/tsc_sync.c index 9452dc9664b5..eab827288e0f 100644 --- a/arch/x86/kernel/tsc_sync.c +++ b/arch/x86/kernel/tsc_sync.c @@ -110,7 +110,8 @@ static int __init start_sync_check_timer(void) if (!cpu_feature_enabled(X86_FEATURE_TSC_ADJUST) || tsc_clocksource_reliable) return 0; - timer_setup(&tsc_sync_check_timer, tsc_sync_check_timer_fn, 0); + timer_setup(&tsc_sync_check_timer, tsc_sync_check_timer_fn, + TIMER_PINNED); tsc_sync_check_timer.expires = jiffies + SYNC_CHECK_INTERVAL; add_timer(&tsc_sync_check_timer); diff --git a/drivers/char/random.c b/drivers/char/random.c index 69754155300e..2cae98dc86dc 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -949,7 +949,7 @@ static DEFINE_PER_CPU(struct fast_pool, irq_randomness) = { #define FASTMIX_PERM HSIPHASH_PERMUTATION .pool = { HSIPHASH_CONST_0, HSIPHASH_CONST_1, HSIPHASH_CONST_2, HSIPHASH_CONST_3 }, #endif - .mix = __TIMER_INITIALIZER(mix_interrupt_randomness, 0) + .mix = __TIMER_INITIALIZER(mix_interrupt_randomness, TIMER_PINNED) }; /* diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index 8058bec87ace..f8c310e62758 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -523,7 +523,7 @@ static inline void clocksource_start_watchdog(void) { if (watchdog_running || !watchdog || list_empty(&watchdog_list)) return; - timer_setup(&watchdog_timer, clocksource_watchdog, 0); + timer_setup(&watchdog_timer, clocksource_watchdog, TIMER_PINNED); watchdog_timer.expires = jiffies + WATCHDOG_INTERVAL; add_timer_on(&watchdog_timer, cpumask_first(cpu_online_mask)); watchdog_running = 1; diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7cd5f5e7e0a1..a0f7bf7be6f2 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1670,10 +1670,13 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, dwork->cpu = cpu; timer->expires = jiffies + delay; - if (unlikely(cpu != WORK_CPU_UNBOUND)) + if (unlikely(cpu != WORK_CPU_UNBOUND)) { + timer->flags |= TIMER_PINNED; add_timer_on(timer, cpu); - else + } else { + timer->flags &= ~TIMER_PINNED; add_timer(timer); + } } /** From patchwork Fri Nov 4 14:57:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15619 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp454017wru; Fri, 4 Nov 2022 07:59:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM580BM3K/UGMQu7qtBTvi6HLUzMevZFAKQC68Xm35scyh7mZhN4tfog7VWLuhYLtRedjaah X-Received: by 2002:a17:907:74e:b0:74f:83d4:cf58 with SMTP id xc14-20020a170907074e00b0074f83d4cf58mr35041632ejb.178.1667573988266; Fri, 04 Nov 2022 07:59:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573988; cv=none; d=google.com; s=arc-20160816; b=RxIYiIb8UO9bV0O2UB0YThh0ZBJXx5q9O72Y1BDLE0PL9u7Zgb5vOFjGxrI7MYrtOk JUo4eRAA1Jej6W1+pqxxv+0GLq6ZiqrVO2OuO7FhIvGPv3K+nbSOExHEei7CrEmvnWa6 0AkX+HfUYCZyIH9SP1NvpGx4xajav6aBrFgbVWrBoNzRPjmf3QOymbkn83UoRnv6BJgP p2l2sQxufQrR0/HN93L3xfrRSuoPoarjsHsQmVE4UsUISXZS84CLZUGQYhnLcusQnnOx eYk4HCC4dQHcN0xcHY2aFEFBkiRxH2ydf6T/p2tPJdnFMcqVn4rWSr54yaLeBu7zgkLD dI2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=sIev2RMtWMuvWP7lonqH3nX8IVRQOAlTrRdPt/ll48c=; b=s2hRTJrAaqp8fCMZVbZLVqcKv1OQw7ZBgGykjvjNCMcPwOgfH5lvMuel7FhFgvvQJY pCHMEAqF+4N/UdyGNV2BdOyAWWN6tSwcPv1Pju0ix8CvHyGbIb4i6EX0Mh1MsD9FMUI2 0LivMKtBk0VsoJ2AFCwop0XMYm38Ui6v0fajJnZyPpKnqFhvVGoYRblCUMCeaQHFAwzZ IW0gnkQ/tc/5j3nIP7/MglD7d0BWZz20CIDf5TJ1WtUXPC5tPoS8qJxL8kIxkB0yVhpO MIM5uYT2ShQ/zaBV0RliYigsn1kpd3fKMbHMh+Idq+pT8Rajwnd/8IiuM4sX6By/knOB Be/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=TvaXTh87; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l21-20020a170906795500b007ae0e8f59a6si5240533ejo.821.2022.11.04.07.59.23; Fri, 04 Nov 2022 07:59:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=TvaXTh87; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232300AbiKDO6n (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232259AbiKDO6C (ORCPT ); Fri, 4 Nov 2022 10:58:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A17372EF24 for ; Fri, 4 Nov 2022 07:57:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sIev2RMtWMuvWP7lonqH3nX8IVRQOAlTrRdPt/ll48c=; b=TvaXTh87nppEXEmP3O5q3L9aGcjK6yMVgBfjAbUDoWYCZjMFJgVU95lrFegl/6mC42GqtA DGRmH7aj01OEVYcwozYT/g76n0maqC8sMftfEpQeoEtVnTLfAcwBQ4de7ZWeIH45K3CfA5 a9qMNS9cnf3bNlxjOFEQr5X9OX2Y67nJRuMqCE0VfnfgQj2EJoEZQCAOKGVFv3TVWrJaN7 yjbOykULEb/WLlK3qzWrMuUUeGX5ZRPi8tZ1pF8T3QuHLK1bwwo9Tse5WVP9OZOn2NsC/h guBTWgIdv+GBenGABPh2camC7I3MwDFfL9Gei9Q8Stu52Z8fvOoMd40j7BhzLA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sIev2RMtWMuvWP7lonqH3nX8IVRQOAlTrRdPt/ll48c=; b=Se5OT/jN3qjTWdL4ku+vPeUuOXiv6Nzaxx95RGPS0avgChF9K0gePwZGbzXbaTwZ01Jh66 27lG3HaATg20fRAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Richard Cochran , Frederic Weisbecker Subject: [PATCH v4 06/16] timer: Keep the pinned timers separate from the others Date: Fri, 4 Nov 2022 15:57:27 +0100 Message-Id: <20221104145737.71236-7-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578062086247063?= X-GMAIL-MSGID: =?utf-8?q?1748578062086247063?= Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. No functional change. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 122 ++++++++++++++++++++++++++++++-------------- 1 file changed, 84 insertions(+), 38 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0e9f609f5a38..0e2abd906916 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -902,7 +908,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -915,7 +924,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1690,6 +1702,21 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } +/* + * Forward base clock is done only when @basej is past base->clk, otherwise + * base-clk might be rewind. + */ +static void forward_base_clk(struct timer_base *base, unsigned long nextevt, + unsigned long basej) +{ + if (time_after(basej, base->clk)) { + if (time_after(nextevt, basej)) + base->clk = basej; + else if (time_after(nextevt, base->clk)) + base->clk = nextevt; + } +} + /** * get_next_timer_interrupt - return the time (clock mono) of the next timer * @basej: base time jiffies @@ -1700,9 +1727,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base) */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt, nextevt_local, nextevt_global; + struct timer_base *base_local, *base_global; + bool local_first, is_idle; u64 expires = KTIME_MAX; - unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1711,38 +1739,57 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (cpu_is_offline(smp_processor_id())) return expires; - raw_spin_lock(&base->lock); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - nextevt = next_timer_interrupt(base); + nextevt_local = next_timer_interrupt(base_local); + nextevt_global = next_timer_interrupt(base_global); /* * We have a fresh next event. Check whether we can forward the - * base. We can only do that when @basej is past base->clk - * otherwise we might rewind base->clk. + * base. */ - if (time_after(basej, base->clk)) { - if (time_after(nextevt, basej)) - base->clk = basej; - else if (time_after(nextevt, base->clk)) - base->clk = nextevt; - } + forward_base_clk(base_local, nextevt_local, basej); + forward_base_clk(base_global, nextevt_global, basej); + + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + nextevt = local_first ? nextevt_local : nextevt_global; /* - * Base is idle if the next event is more than a tick away. Also + * Bases are idle if the next event is more than a tick away. Also * the tick is stopped so any added timer must forward the base clk * itself to keep granularity small. This idle logic is only - * maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * maintained for the BASE_LOCAL and BASE_GLOBAL base, deferrable + * timers may still see large granularity skew (by design). */ - base->is_idle = time_after(nextevt, basej + 1); + is_idle = time_after(nextevt, basej + 1); - if (base->timers_pending) { + /* We need to mark both bases in sync */ + base_local->is_idle = base_global->is_idle = is_idle; + + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return cmp_next_hrtimer_event(basem, expires); } @@ -1754,15 +1801,14 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); } #endif @@ -1808,11 +1854,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -1820,19 +1868,17 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Fri Nov 4 14:57:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15620 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp454187wru; Fri, 4 Nov 2022 08:00:05 -0700 (PDT) X-Google-Smtp-Source: AMsMyM63kHCa3E7U+xYPSSXGn+OkipoE3ydoIMCjb7gMV/zjJqfK1nInEhs+20J/wshsFVx+6/vo X-Received: by 2002:a17:907:708:b0:77e:ff47:34b1 with SMTP id xb8-20020a170907070800b0077eff4734b1mr32630005ejb.493.1667574005030; Fri, 04 Nov 2022 08:00:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574005; cv=none; d=google.com; s=arc-20160816; b=pWeVEYKPCSFopHr88N2FBxUUtyZvB5PfkqoOwBdBEAY5gzNj6kY5KCfs4ZabB+bSsc 9YnMP6cDCmgWLBAEG5R8FguGB3IJuZZAX3pHxXEkLBXJ2d373fsKC1Xq9p3UrSPeg6Bh ILjPjgyu4ZcpyrCCkjK/mZ7Ft3G7UBwkgDVPTCqG4WdEunLDpLJOgjJcCr7+vc8bI9kk PedUKXREsy7U3C0XsDHgWgBVvpelmPWVYrFxYEt2zU85CQMUcwZIBRWnf5cNQayGP+CV NLp93KKybJBN1Iv741/tKeM/PKkGsKGPyftuThWn3QNxfL/nNSRBcILI2skuoRWvlQ7X Ni4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=Q2yemYw0b9XnqUBChrfmowbVg4uqr41YiwOJoEf/0lc=; b=hjatYCg0ArULXlhmBv4m6i4uveXi6kB1F8eoNXOOYYdDelCAcr8QORHsxYjDvhGe3y W+xG0OXB7zeBWRodh+ZYhYXnDqfEYAFMjkgFKZamOpHclx8OAQgAUR44aEV0DAiF8V/r nC9QGcZLy+7GQ6kk8uMyrxo8uI7NGt29eHu5P4legrVvaFg8gAK+lqlhatVK/qAe67OD mkz6KV5FMnrBvWEsXZgWSOT35smUPdaHv9lnYzwHmonCeJqE4ZnT88T2SdsxqA/PJpYB BQyjxt8Mmjgal2WfPLIWF1FpefmtwoQpxXUFCXbaWjpZYZJ+Tl7xjeDN0XhoVN4hs/qO G/SA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2AgDEWlS; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hc6-20020a170907168600b007a6384d506csi6242350ejc.643.2022.11.04.07.59.39; Fri, 04 Nov 2022 08:00:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2AgDEWlS; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232343AbiKDO6r (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232261AbiKDO6C (ORCPT ); Fri, 4 Nov 2022 10:58:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7A972EF48 for ; Fri, 4 Nov 2022 07:57:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q2yemYw0b9XnqUBChrfmowbVg4uqr41YiwOJoEf/0lc=; b=2AgDEWlSAz6fhOfklAMqgW0XMnuKTy2ItAGHIZZmaD+jMZY2GU0IH9xWiBJPSMe9dM6XTv 4PKdVUEa/9RAad2nV7G9D46CqftglC+ghdEiJbwV6RXy/z5g37R191LrMS8SOIcUvY27Ph ajReU/RgJKWCc11DUJ5u+AAiUkRF8V7mDDsuNMpXHExhHxMJSmT1u+WXAYO32igvkiEMKD AWyHVC0dCgSabXuYfYoo9JPNL793iiprU3F/xhZpk3z9WomVZXylrv3wkc4CQ2bRZFdurJ i/k+K/cHGA0RMG2RooeWii1rUIYYEKoBgIIs0jCvOSRZwI7LaqFH98I5ErMR6A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q2yemYw0b9XnqUBChrfmowbVg4uqr41YiwOJoEf/0lc=; b=worRQsbG4fafVfswdvOZvJ0ny2Co+XEBy2/Rd0+/I4Iu8NgUWIkYzyWwTTJbMf38v+7Mg/ YQDEx5nYnzfhimCw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v4 07/16] timer: Retrieve next expiry of pinned/non-pinned timers seperately Date: Fri, 4 Nov 2022 15:57:28 +0100 Message-Id: <20221104145737.71236-8-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578079981147594?= X-GMAIL-MSGID: =?utf-8?q?1748578079981147594?= For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have seperate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/tick-internal.h | 8 ++++- kernel/time/tick-sched.c | 20 ++++++++---- kernel/time/timer.c | 65 ++++++++++++++++++++++++++++--------- 3 files changed, 70 insertions(+), 23 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 649f2b48e8f0..fcb2d45c2934 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -163,7 +168,8 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); -extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem); +extern void get_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_events *tevt); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 7ffdc7ba19b4..78f172d1f3d2 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -784,7 +784,8 @@ static inline bool local_timer_softirq_pending(void) static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) { - u64 basemono, next_tick, delta, expires; + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; + u64 basemono, delta, expires; unsigned long basejiff; unsigned int seq; @@ -809,7 +810,11 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) */ if (rcu_needs_cpu() || arch_needs_cpu() || irq_work_needs_cpu() || local_timer_softirq_pending()) { - next_tick = basemono + TICK_NSEC; + /* + * If anyone needs the CPU, treat this as a local + * timer expiring in a jiffy. + */ + tevt.local = basemono + TICK_NSEC; } else { /* * Get the next pending timer. If high resolution @@ -818,17 +823,18 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * disabled this also looks at the next expiring * hrtimer. */ - next_tick = get_next_timer_interrupt(basejiff, basemono); - ts->next_timer = next_tick; + get_next_timer_interrupt(basejiff, basemono, &tevt); + tevt.local = min_t(u64, tevt.local, tevt.global); + ts->next_timer = tevt.local; } /* * If the tick is due in the next period, keep it ticking or * force prod the timer. */ - WARN_ON_ONCE(basemono > next_tick); + WARN_ON_ONCE(basemono > tevt.local); - delta = next_tick - basemono; + delta = tevt.local - basemono; if (delta <= (u64)TICK_NSEC) { /* * Tell the timer code that the base is not idle, i.e. undo @@ -861,7 +867,7 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) else expires = KTIME_MAX; - ts->timer_expires = min_t(u64, expires, next_tick); + ts->timer_expires = min_t(u64, expires, tevt.local); out: return ts->timer_expires; diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0e2abd906916..d272672935da 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1664,7 +1664,7 @@ static void next_expiry_recalc(struct timer_base *base) * Check, if the next hrtimer event is before the next timer wheel * event: */ -static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) +static void cmp_next_hrtimer_event(u64 basem, struct timer_events *tevt) { u64 nextevt = hrtimer_get_next_event(); @@ -1672,15 +1672,17 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) * If high resolution timers are enabled * hrtimer_get_next_event() returns KTIME_MAX. */ - if (expires <= nextevt) - return expires; + if (tevt->local <= nextevt) + return; /* * If the next timer is already expired, return the tick base * time so the tick is fired immediately. */ - if (nextevt <= basem) - return basem; + if (nextevt <= basem) { + tevt->local = basem; + return; + } /* * Round up to the next jiffie. High resolution timers are @@ -1690,7 +1692,7 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) * * Use DIV_ROUND_UP_ULL to prevent gcc calling __divdi3 */ - return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; + tevt->local = DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } @@ -1718,26 +1720,31 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, } /** - * get_next_timer_interrupt - return the time (clock mono) of the next timer + * get_next_timer_interrupt * @basej: base time jiffies * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding field + * is set to KTIME_MAX. */ -u64 get_next_timer_interrupt(unsigned long basej, u64 basem) +void get_next_timer_interrupt(unsigned long basej, u64 basem, + struct timer_events *tevt) { unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; bool local_first, is_idle; - u64 expires = KTIME_MAX; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; /* * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ if (cpu_is_offline(smp_processor_id())) - return expires; + return; base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); @@ -1782,16 +1789,44 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the bases are not marked idle, i.e one of the events is at + * max. one tick away, then the CPU can't go into a NOHZ idle + * sleep. Use the earlier event of both and store it in the local + * expiry value. The next global event is irrelevant in this case + * and can be left as KTIME_MAX. CPU will wakeup on time. + */ + if (!is_idle) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; + goto unlock; } + + /* + * If the bases are marked idle, i.e. the next event on both the + * local and the global queue are farther away than a tick, + * evaluate both bases. No need to check whether one of the bases + * has an already expired timer as this is caught by the !is_idle + * condition above. + */ + if (base_local->timers_pending) + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + /* + * If the local queue expires first, then the global event can be + * ignored. The CPU wakes up before that. If the global queue is + * empty, nothing to do either. + */ + if (!local_first && base_global->timers_pending) + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + +unlock: raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - return cmp_next_hrtimer_event(basem, expires); + cmp_next_hrtimer_event(basem, tevt); } /** From patchwork Fri Nov 4 14:57:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15616 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453836wru; Fri, 4 Nov 2022 07:59:26 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7MsNwrznora1t9qWNlrooZ1kZiyPQfDQsuRRWpPtJFVJf/RTeSJMZ8ZUXK3I7AutOKk2+i X-Received: by 2002:a17:906:eecb:b0:73c:5bcb:8eb3 with SMTP id wu11-20020a170906eecb00b0073c5bcb8eb3mr35438773ejb.284.1667573966472; Fri, 04 Nov 2022 07:59:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573966; cv=none; d=google.com; s=arc-20160816; b=mKOa2NNcn2KAd3CyXWAyl7+M9X5naYPLHvsmP6R7qPo9bNWSXChiN8STxVw27/PX2S LU2e0ItqJ5KRzvuuQne/b+hEMZrgSNLza5KWHODc3Kpbc18bWEtU9CIFVAo4RHkGttC/ XKDn9jjwgfgc3bvF+zLCtQwCgMykJtdXgTlK+r4vTMEUTutZwB0iy/ASMljBUMNUbATU yP++q7cYEh5Qp2xxnp0VrlgWXPHsgfm+5hmaAeFbB1DWj7/uNC/yzMwEiglpSXRYVaDm CbgPjTJTqM+Dt2u7WKnyaZ7zLuiq6999E1pJvBOLSinf/1HkSkejY8nI6jSbOY+3DLzj 8p8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=mWl/nVix9xdTC/DXxH80uO1VlOYyblDq6iPvE5oroB4=; b=rPbGixS3tXSlWlWhMuBLy5PoNsd9dPyNoVacYp8tyQM5tRTC1SHUEyVqkmGWme0hQ+ JxQcLt2gIOdraZEuOkN4HbX0R47jWwGNcuKgMOYNc5tEAYONCW+WPoTHy2VHn7JyE2PE Yo+m2CDplNLJb/OR+n6ni4HkoHiHVTj1BeveXVADpOA6Dfl2C93AjEhWlfXXPFA+X2Us drzKMO73EKvdqvHgdVB9wxyDYFflXlpO7/PZ0YjOoKAOtm+8njhHeMfQk2yZyXT2mZu1 o6luYHoFlIke+SFml06hI03vs4n5yghB2WQWmhWQ/vMdfq4PhPFXFSWUpVizNGfDIO2C LAPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="eOggDz/4"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=XkMRk8Ci; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dd11-20020a1709069b8b00b0073d81935416si5699370ejc.749.2022.11.04.07.59.03; Fri, 04 Nov 2022 07:59:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="eOggDz/4"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=XkMRk8Ci; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231979AbiKDO60 (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232250AbiKDO6C (ORCPT ); Fri, 4 Nov 2022 10:58:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A730A2EF33 for ; Fri, 4 Nov 2022 07:57:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mWl/nVix9xdTC/DXxH80uO1VlOYyblDq6iPvE5oroB4=; b=eOggDz/4sdP8PfMV8UqQ5XfEdVGj0V/aH9mzAlKjh3iqNn2KdD/cSkT1XzbvR6XDDokQ7G QaLfbsijzclrBUyfkhJU4zi6bnVQL/3EtvhIUV2dOWXJV1uq1YZGh3eRpKNytpzVtbEo7o z/B2VmdVulSWuDk/GcwLv8i1X9LwdH3ihD88JmQIAE87Hq0M+DiC7IOoAzQkO9tSuf4t2F Z7ytzsOGqM/FMOqwD1ZTt7LmnhoVxMVhOlIrROZsv1/xzxiyRo9HAVtfFoooROx0cd/KC/ wBqP/uE1IoAClsQSn+e5jHzJ/AOKBwxBDDiv+JmHcGVcyyf5/EBLj2o3Je4J4g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mWl/nVix9xdTC/DXxH80uO1VlOYyblDq6iPvE5oroB4=; b=XkMRk8CizaM6HLaixTRMwyYTeh2Y5N3KBr9QJsuWsP6jiaGIWMYQ3n80NY8op8ahTMMFWZ /DMifY/BqjCBlDBQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 08/16] timer: Rename get_next_timer_interrupt() Date: Fri, 4 Nov 2022 15:57:29 +0100 Message-Id: <20221104145737.71236-9-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578039661012135?= X-GMAIL-MSGID: =?utf-8?q?1748578039661012135?= get_next_timer_interrupt() does more than simply getting the next timer interrupt. The timer bases are forwarded and also marked as idle when possible and the next timer interrupt information is required for this. To get not confused, rename function to a more descriptive name. No functional change. Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 4 ++-- kernel/time/tick-sched.c | 16 ++++++++-------- kernel/time/timer.c | 10 +++++++--- 3 files changed, 17 insertions(+), 13 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index fcb2d45c2934..6f5f164506d5 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -168,8 +168,8 @@ static inline void timers_update_nohz(void) { } DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); -extern void get_next_timer_interrupt(unsigned long basej, u64 basem, - struct timer_events *tevt); +extern void forward_and_idle_timer_bases(unsigned long basej, u64 basem, + struct timer_events *tevt); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 78f172d1f3d2..7f7bfe8b498d 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -802,11 +802,11 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * Keep the periodic tick, when RCU, architecture or irq_work * requests it. * Aside of that check whether the local timer softirq is - * pending. If so its a bad idea to call get_next_timer_interrupt() - * because there is an already expired timer, so it will request - * immediate expiry, which rearms the hardware timer with a - * minimal delta which brings us back to this place - * immediately. Lather, rinse and repeat... + * pending. If so its a bad idea to call + * forward_and_idle_timer_bases() because there is an already + * expired timer, so it will request immeditate expiry, which + * rearms the hardware timer with a minimal delta which brings us + * back to this place immediately. Lather, rinse and repeat... */ if (rcu_needs_cpu() || arch_needs_cpu() || irq_work_needs_cpu() || local_timer_softirq_pending()) { @@ -823,7 +823,7 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * disabled this also looks at the next expiring * hrtimer. */ - get_next_timer_interrupt(basejiff, basemono, &tevt); + forward_and_idle_timer_bases(basejiff, basemono, &tevt); tevt.local = min_t(u64, tevt.local, tevt.global); ts->next_timer = tevt.local; } @@ -838,7 +838,7 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) if (delta <= (u64)TICK_NSEC) { /* * Tell the timer code that the base is not idle, i.e. undo - * the effect of get_next_timer_interrupt(): + * the effect of forward_and_idle_timer_bases(): */ timer_clear_idle(); /* @@ -1141,7 +1141,7 @@ void tick_nohz_idle_retain_tick(void) { tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); /* - * Undo the effect of get_next_timer_interrupt() called from + * Undo the effect of forward_and_idle_timer_bases() called from * tick_nohz_next_event(). */ timer_clear_idle(); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index d272672935da..680a0760e30d 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1720,7 +1720,7 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, } /** - * get_next_timer_interrupt + * forward_and_idle_timer_bases * @basej: base time jiffies * @basem: base time clock monotonic * @tevt: Pointer to the storage for the expiry values @@ -1728,9 +1728,13 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, * Stores the next pending local and global timer expiry values in the * struct pointed to by @tevt. If a queue is empty the corresponding field * is set to KTIME_MAX. + * + * If required, base->clk is forwarded and base is also marked as + * idle. Idle handling of timer bases is allowed only to be done by + * CPU itself. */ -void get_next_timer_interrupt(unsigned long basej, u64 basem, - struct timer_events *tevt) +void forward_and_idle_timer_bases(unsigned long basej, u64 basem, + struct timer_events *tevt) { unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; From patchwork Fri Nov 4 14:57:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15618 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453967wru; Fri, 4 Nov 2022 07:59:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7hNT9U13xTvj/lG7AbzsJE87MJRQoQmx2DbZfX77mWQDSwwokU8FvXSD+6sJR7ZpTdRxg9 X-Received: by 2002:a17:906:fe49:b0:73d:70c5:1a52 with SMTP id wz9-20020a170906fe4900b0073d70c51a52mr35640029ejb.469.1667573981634; Fri, 04 Nov 2022 07:59:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573981; cv=none; d=google.com; s=arc-20160816; b=JmzyhzA4vpFMpiW++9zOEyNLAE7wotTurL4aFZXTapf2kspRBdCPDZ3fMMd4bAfDSX JYlGnpfAa6DdwPhALgn/Gbnc+3SKMTNdfGvyXBGeJqPZ5LdI5674Zmz91ntCig3qvobG zv/SU6JAuVjGKnvdreDckdEEbll0ebSFjROQCd0PIUY/l4Ha+0OKMNUty9Zo0zoHSbEd KLamZDiw3eZu0/YviGkEm4gzcOSPkZFhdscb3pmRNKlU7NJB70v/4pPwXQxPi+GtcINk IvdcCmc0y9DrtRU9tMGi85QoVugS1owOjPIMOEOKdZa93dzlMJSJCqA4x4HKnCzHCjOL IUWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=q4/MLAhLTPtV7qoK3GH2WeOlioSjvDorYqg4PRgG1nU=; b=mDqF1z/hFqVAebWRllAXcpR0br3xz8LZmAgxR3unABMTfP2whB3LKcXsUYO2WlUhlQ VDMlNr7C6m5/09b731qIbMH6LgjjgZfhbRCg1m1iYdd1A8OI4qqW4AgvfAw3X6Cm0PnT i7bKcmQEcTBt2GykKpJ7kV2+S72QIMbSpMh93A7HJQJjr41ClA4tD9V93B9/O3fhCXxv A+pmTLO3IBQanNDOonqhSJ1tqu5vIj8T2aQfSKAIZXSrk4i6Y/b0oC2cYT+zpTkzg1Hz iP65kDe1vktJMZDd1z3L3ucokZFs+XS0vBolqODupCRRy52/jBRt4CzWEeu771WzkGxl E9YA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ORe+HzPL; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=slqSY2OQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a21-20020a1709066d5500b007835897050asi4002356ejt.83.2022.11.04.07.59.18; Fri, 04 Nov 2022 07:59:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ORe+HzPL; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=slqSY2OQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbiKDO6i (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232267AbiKDO6E (ORCPT ); Fri, 4 Nov 2022 10:58:04 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7C8A3123E for ; Fri, 4 Nov 2022 07:57:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4/MLAhLTPtV7qoK3GH2WeOlioSjvDorYqg4PRgG1nU=; b=ORe+HzPLzL0nYvHtB3lHHqfioy+48qZXQY2id+lpHiXJJhU+BwM4+kWGtv6u+S97KiQJkn 4BdBCSa4rAwpq9kZBBZVtCTfND7UPMN9UecEEsXegYh1HoJL5ehbs4ZWNFURiF9cCtPfQ6 NRZ34sYy2FdLYDbH+54TpnWclgCUMdNMNsHxLh1QVL2qkj/hRCz/byInwPAJSg2s3R5TPv j6QJ5wBcReSquPG+r/kNzewx/dVJbeEELZ05HnjiX9AjrZtCelaYTg5ZYi7YFhJ+WhAbZq jD69f6+9Gp00w8g/lUg1c4AJGcX0EgwvGR1EIpcu+U7sq/T8TQva0hS47eVQvA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4/MLAhLTPtV7qoK3GH2WeOlioSjvDorYqg4PRgG1nU=; b=slqSY2OQGC/wGbjGP+bBoCvIJNORpq3RLGJTsO5feZ/WYf8OifeUvQ1cMWG6hGCNUB75os tK4QctkJVcXdotCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 09/16] timer: Split out "get next timer interrupt" functionality Date: Fri, 4 Nov 2022 15:57:30 +0100 Message-Id: <20221104145737.71236-10-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578055206052418?= X-GMAIL-MSGID: =?utf-8?q?1748578055206052418?= forward_and_idle_timer_bases() includes the functionality for getting the next timer interrupt. To reuse it, it is splitted into an separate function "get_next_timer_interrupt()". This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen --- v4: Fix typo in comment --- kernel/time/timer.c | 93 +++++++++++++++++++++++++-------------------- 1 file changed, 51 insertions(+), 42 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 680a0760e30d..853089febf83 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1704,6 +1704,46 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } +static unsigned long get_next_timer_interrupt(struct timer_base *base_local, + struct timer_base *base_global, + unsigned long basej, u64 basem, + struct timer_events *tevt) +{ + unsigned long nextevt_local, nextevt_global; + bool local_first; + + nextevt_local = next_timer_interrupt(base_local); + nextevt_global = next_timer_interrupt(base_global); + + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + /* + * Update tevt->* values: + * + * If the local queue expires first, then the global event can + * be ignored. If the global queue is empty, nothing to do + * either. + */ + if (!local_first && base_global->timers_pending) + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + return local_first ? nextevt_local : nextevt_global; +} + /* * Forward base clock is done only when @basej is past base->clk, otherwise * base-clk might be rewind. @@ -1738,7 +1778,7 @@ void forward_and_idle_timer_bases(unsigned long basej, u64 basem, { unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; - bool local_first, is_idle; + bool is_idle; /* Preset local / global events */ tevt->local = tevt->global = KTIME_MAX; @@ -1756,8 +1796,11 @@ void forward_and_idle_timer_bases(unsigned long basej, u64 basem, raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - nextevt_local = next_timer_interrupt(base_local); - nextevt_global = next_timer_interrupt(base_global); + nextevt = get_next_timer_interrupt(base_local, base_global, + basej, basem, tevt); + + nextevt_local = base_local->next_expiry; + nextevt_global = base_global->next_expiry; /* * We have a fresh next event. Check whether we can forward the @@ -1766,21 +1809,6 @@ void forward_and_idle_timer_bases(unsigned long basej, u64 basem, forward_base_clk(base_local, nextevt_local, basej); forward_base_clk(base_global, nextevt_global, basej); - /* - * Check whether the local event is expiring before or at the same - * time as the global event. - * - * Note, that nextevt_global and nextevt_local might be based on - * different base->clk values. So it's not guaranteed that - * comparing with empty bases results in a correct local_first. - */ - if (base_local->timers_pending && base_global->timers_pending) - local_first = time_before_eq(nextevt_local, nextevt_global); - else - local_first = base_local->timers_pending; - - nextevt = local_first ? nextevt_local : nextevt_global; - /* * Bases are idle if the next event is more than a tick away. Also * the tick is stopped so any added timer must forward the base clk @@ -1793,43 +1821,24 @@ void forward_and_idle_timer_bases(unsigned long basej, u64 basem, /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); + /* * If the bases are not marked idle, i.e one of the events is at * max. one tick away, then the CPU can't go into a NOHZ idle * sleep. Use the earlier event of both and store it in the local * expiry value. The next global event is irrelevant in this case - * and can be left as KTIME_MAX. CPU will wakeup on time. + * and can be reset as KTIME_MAX. CPU will wakeup on time. */ if (!is_idle) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC; - goto unlock; + tevt->global = KTIME_MAX; } - /* - * If the bases are marked idle, i.e. the next event on both the - * local and the global queue are farther away than a tick, - * evaluate both bases. No need to check whether one of the bases - * has an already expired timer as this is caught by the !is_idle - * condition above. - */ - if (base_local->timers_pending) - tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; - - /* - * If the local queue expires first, then the global event can be - * ignored. The CPU wakes up before that. If the global queue is - * empty, nothing to do either. - */ - if (!local_first && base_global->timers_pending) - tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; - -unlock: - raw_spin_unlock(&base_global->lock); - raw_spin_unlock(&base_local->lock); - cmp_next_hrtimer_event(basem, tevt); } From patchwork Fri Nov 4 14:57:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15617 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp453904wru; Fri, 4 Nov 2022 07:59:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM656kU0W2Y+Dk7tiiS/ZEi1BTPbIRYuv51jX2V1kYN9wkRBUiR9qgy8OkY8VR+SUiRTmlDf X-Received: by 2002:a17:907:70a:b0:750:bf91:caa3 with SMTP id xb10-20020a170907070a00b00750bf91caa3mr35200406ejb.711.1667573974175; Fri, 04 Nov 2022 07:59:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667573974; cv=none; d=google.com; s=arc-20160816; b=Cqyn9qVwCeqBEZl/HxkplC0jCOFtvXiMrPkYdlH2YtjryPKBZ+z5IFJ/N5fWdji9Bh lKAElnLyR6a6YC9e1YudqQ7VqgJmhb44zwTYv/gRMJ0/mUXrd22vRteL81yhaq4LeBms JiY6gnz3oWSLsEJWHWSHxq8ggTHfPm0X/EH3EOhSDuer/Iaz+VobqzaZaQ91wdWzIuPm LpNNeGDkXlIrZWKEyLznU8Wa86HX/GJjm5YAmPZFqChXKCc1oQsJI3vKdUf42q5sEKdR CjFjwm+KL8hSxaObyGxNJ5lCBA+VLnY/qLVHWgjRkrECYE1HsRFqOgKSxuUU/nFRt3BE gJDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=VJsZTf9koJv37AGxRjIxMnvDNnKc+RAv8HmlgWGKkig=; b=KWdhJQqqpzENYhtL3kQKslL5SVH80D8VzLc1vqkCJBUwOWOHyHlpQZWf7mbiSZkVfk vQ6MQPK37L4N1akJAt2imsOrBZ9nw2+4Hb0AbmJD9qOC3nNXqDP1OiKLjGnPv8qvMrOj nunEi8Z3jgBeBryIxj+K9QL5/PYv2hFvwsVHkoEMUphAKVqEPoLpzvkZP4OqSSqDE7N0 sW4jQHfR8kJRp29P1h+SOapFtl8yIg84FvaX8381eruZLp07NfdRW+INNEHbB7MH7f23 0ikhjH8xHyB8tPwWmlXkJ+pGr+dtR6ijvErL6Jw7x4BL8cmFr2sK/iDRMevmGk4yy+Ui BMVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="o0/2YzYr"; dkim=neutral (no key) header.i=@linutronix.de header.b=Vj6IDZ1Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hd14-20020a170907968e00b007316ac034acsi5900930ejc.834.2022.11.04.07.59.10; Fri, 04 Nov 2022 07:59:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="o0/2YzYr"; dkim=neutral (no key) header.i=@linutronix.de header.b=Vj6IDZ1Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231873AbiKDO6d (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230324AbiKDO6E (ORCPT ); Fri, 4 Nov 2022 10:58:04 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6275C317ED for ; Fri, 4 Nov 2022 07:57:59 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VJsZTf9koJv37AGxRjIxMnvDNnKc+RAv8HmlgWGKkig=; b=o0/2YzYr/C5Go2NsXHPKNZJGD+J2HxmJl+Kr/NLzpNWobjibKzEfXWod7WDzTIbEX8BLhj 5hPmG++t7DcmlxmGkVzOKwT5aXSaM5j+BwzcLZ1/hLvYzfcMq3in9p4VMNXTfvMfH2yRRg xnyiksuALIrSMX1z0jwfJU1gykJlRTHX3qxCKdpH6I7OKMin+UVrVjukJnOskvf+9xHa4W MBwGfHXN2uw0LYv1tYT75/tvYh63ffHCrwR0mgXvC57DTX7i91WlNx8XTlh4S487iPSODy KwKPEGZn5vB+geOzDHyW3OfCEErhOEJ1SeEX29wmNCV2QvDdZ2SVvCEgCjMFrw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VJsZTf9koJv37AGxRjIxMnvDNnKc+RAv8HmlgWGKkig=; b=Vj6IDZ1QZdyOaQNVKe/daLQVKtBKVcMoIZ7FfaXD0S387W6BRXx7Bu36atSulSuCOu3jzk RawRexfhN0phzmDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 10/16] timer: Add get next timer interrupt functionality for remote CPUs Date: Fri, 4 Nov 2022 15:57:31 +0100 Message-Id: <20221104145737.71236-11-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578047852492764?= X-GMAIL-MSGID: =?utf-8?q?1748578047852492764?= To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 3 +++ kernel/time/timer.c | 41 +++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 6f5f164506d5..7b65abc9d803 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -170,6 +170,9 @@ DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases); extern void forward_and_idle_timer_bases(unsigned long basej, u64 basem, struct timer_events *tevt); +extern void get_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); void timer_clear_idle(void); #define CLOCK_SET_WALL \ diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 853089febf83..1b82661f6172 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1759,6 +1759,47 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, } } +/** + * get_next_timer_interrupt_remote + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + */ +void get_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + unsigned long flags; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + /* + * Pretend that there is no timer pending if the cpu is offline. + * Possible pending timers will be migrated later to an active cpu. + */ + if (cpu_is_offline(cpu)) + return; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_lock_irqsave(&base_local->lock, flags); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + get_next_timer_interrupt(base_local, base_global, basej, basem, tevt); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock_irqrestore(&base_local->lock, flags); +} + /** * forward_and_idle_timer_bases * @basej: base time jiffies From patchwork Fri Nov 4 14:57:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15623 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456176wru; Fri, 4 Nov 2022 08:02:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6Js3GEjfhu6FWlTj16R3P8t9MzHSF/nO2TYHY5skLSRN/AGSfIy4IByDjqQEm/PGC6Pyzj X-Received: by 2002:a05:6402:530d:b0:463:b0cb:50e5 with SMTP id eo13-20020a056402530d00b00463b0cb50e5mr20231522edb.45.1667574175306; Fri, 04 Nov 2022 08:02:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574175; cv=none; d=google.com; s=arc-20160816; b=ez2YJoE+F02PiSF1XUW9VAOxLc/zcDXMquFVvRLEW8n3+2tjuV6ifF6pUYPQ1uNjdk vPS1hJ3blaI6Ov1HCo3a/V3kw9/m6k7/JNzqT1KpO0A91ZczPIsFlmC7geoGKb1wRDkc UzTE5Emi0q8QviMpOE6jzlfe9MzuDfGCxpG19p7DxnrpUw4fvlhzl5oSRiYFywVrkXWe 33N1zgYofKuBN8N+laMPZ0cNRo1NT2upvyTjzQzbeeiRZ/S22xvaofcVLN0h6gln474i aI2DKIjuDJsC1lxrbMm1QQgAN79CklH0Kfl82J7mtTnC+Vjbyb1Xu166GNlcAtvn15Oe tbyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=bR/mftSP+c00ARZaKpp17SnhKoPd5BhUk3++d2T/vCA=; b=qgbEfnV5UBpj4wvzm+Cemd4e+vJq3rame12TFmyurOzMIXazMFh0kMaYNoH+2JrODl kbL5Juv3CkmeC9TgxZ/hDeXCELJ8lVrh/RFA3P3Uy3Kwz8et9oSaazZwngj7GgzJDmeM peR035ZndlckzAJdMqpgycmTVQKWIFiyqj9POMA5yJYUsPgT9qbUzeMH5nDCN4rO5UA8 t4YyjMPz98yCfowimsqTQQiu3dkKnzlxQ0hJuAZ3pFc5QPIxV54lsLf5WXucwHOCYcso GT8hMIQZdnOtoczafaFAN7X27qBqEcltv+18fI1gFmhoacnrxKJWtDNv9cZoqSU/Q+ck 88QQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=pfuMn0tB; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hv9-20020a17090760c900b007adc8d6b77esi6007936ejc.308.2022.11.04.08.02.31; Fri, 04 Nov 2022 08:02:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=pfuMn0tB; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232362AbiKDO7H (ORCPT + 99 others); Fri, 4 Nov 2022 10:59:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232248AbiKDO6F (ORCPT ); Fri, 4 Nov 2022 10:58:05 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBBA7317EF for ; Fri, 4 Nov 2022 07:57:59 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bR/mftSP+c00ARZaKpp17SnhKoPd5BhUk3++d2T/vCA=; b=pfuMn0tB4XvaXaxjLKLEA7eLNd19BHMNYFkZjtkLMrL7U7qZpJ83I0x0iTzN3ophyHF8sK Y32iIHbmCxJ1zUzVhUOSmjwBCkoQkZWAWRiWNtCNuFOwc87yYDPC9c2WKbRcJzCuQFAnDE bDvh8mrUjJh9ypeXTE4GI87WhQm4k3lDk4zoQ4Mt+O8HUmVeKOEhromVWuQgTZZLoJGX89 1xXA52iEzuMXsBI5QPEmLCYdaFgGwqxC5rAZzaZzSFFRIC6gsfsqq6zbzVZ37Gag5W2wgh 5r5qUzbw15jPZL5GYCRUHcSKzaRvb8FL7i5f0tlrmL82hM+/qv2bzMJc6L5k/Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bR/mftSP+c00ARZaKpp17SnhKoPd5BhUk3++d2T/vCA=; b=9FEpIwyJCazovQUC5fZa2Ev4q0i98lV3OV5rm+y2sgZ/wTepYpgtZsSEbEaPEIZQvzmlHt O4Yo32UnlmCSnPBw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v4 11/16] timer: Restructure internal locking Date: Fri, 4 Nov 2022 15:57:32 +0100 Message-Id: <20221104145737.71236-12-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578258599385501?= X-GMAIL-MSGID: =?utf-8?q?1748578258599385501?= From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1b82661f6172..641e7284f0d7 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1910,11 +1910,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -1934,21 +1930,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Fri Nov 4 14:57:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15621 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456132wru; Fri, 4 Nov 2022 08:02:52 -0700 (PDT) X-Google-Smtp-Source: AMsMyM50LV0qulQZxvtqj8qiDZrHafaREV62Y9Pghl4/5+shM2X2x/T7mlXYy6cDdlOxFL5rh4k+ X-Received: by 2002:a50:fd12:0:b0:463:cb99:5b1 with SMTP id i18-20020a50fd12000000b00463cb9905b1mr18344705eds.256.1667574172405; Fri, 04 Nov 2022 08:02:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574172; cv=none; d=google.com; s=arc-20160816; b=EGpwWNEwiLe+e76suYwlnY8GcG04jYXq+br5yWgKcJjv1MPxP99FNiZCS7f31lSLtH W8942sHK8b8dcQOZzHImvgrjqQhIMTf35q8OK3652gJMbYMhtznJ9LA+Y24IYUhxzV1T fr1mQMRQcRB+4IA2axPdgagHuncGprxr+VV90Yoo1c4dPTNWpTRhXznUyv498rKgp0Mi D0uxrQJ3XAlVc8VBVuIP7KYvvxUZ0E0ELF1niI53ivMtQ7OWlfoK0Me5C4QTJzLx8RkM kySK3niQOV8iWdnt7nK/hkHGk++491m86Q0gtRPfA4523qS1AseTdJgsV+TGVnjB1RP8 Qf7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=og4Fmjqc4BhAa6zqoKeZ4IC2HuQC1SBhqBX9c57/ezA=; b=GKnLD2khTXmVmGwFtaLyMF0LwDAWgiYK6xffXFGLL/gC1NM3/Dz8vkoci4Ori4fQ2h vp0L2AsdxJ3JIi07NvmB4yXJf61DunqYJpxSeL8YwuMoaMNN4NH4QHUBn2m4SeBx7Ttm JstD7NDL1iRllHkHkMasKbnXq6wHWBBgWtV+YovS2ileZeRVwkZ3pNiJpNSKHPspQ3o/ ohKaK0sWdScezQTVmSkGoPZKxHzjRYAWEh73wkWV7xpt1+ytB+DKlAfbUfZSC4fm0pjs zl48gDvLFtX7XjpWkHk1a+kr/z/VykhYyWxlrbCZp8LgFV6ZKfXwWqSol/OIbp0MOODX 1M1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="BG0dJQS/"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dm5-20020a170907948500b007826de24087si5763076ejc.228.2022.11.04.08.02.27; Fri, 04 Nov 2022 08:02:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="BG0dJQS/"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232355AbiKDO6z (ORCPT + 99 others); Fri, 4 Nov 2022 10:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232278AbiKDO6G (ORCPT ); Fri, 4 Nov 2022 10:58:06 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54E4E31DCE for ; Fri, 4 Nov 2022 07:58:00 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=og4Fmjqc4BhAa6zqoKeZ4IC2HuQC1SBhqBX9c57/ezA=; b=BG0dJQS/41uoSVCg4fsUBsZRj3FHe0a2pi920fPBprxNiIUtEzblTBMLbF1Y7v+G0+usMQ XF6HUNpzMCq32JKBMQn/f+i8Bqt5svsuKxSMp0wYHnryE3voSKYtuwCLK3BkQvhktjUMpH f4mtmQfjKU0PYYCSGXv/qYT9kbv3DzUCMgIIusxV/CxaXTV1lk2Sa9aMqOmRyVIUgsz+nk yJ6RKQKzyxfZpU3/Wdsfhh7VKTN5c32YVRfYkQ5a6o87lvc6iioImPEwXGKNdqTJEks0kf O0WoX4NUkeuiiJfES3D9HjP64NFDFwdLK2wkfyd0w01M9+YwiPGy1cDVweZ4ZA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=og4Fmjqc4BhAa6zqoKeZ4IC2HuQC1SBhqBX9c57/ezA=; b=sHaIgiqK10fb5yXmKO+wxs2Y4tbS5fk61OmJgtwBTpBRAMibYAZqVws1dyIiungZE9HQuj jMvsK86yboVj15Bw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 12/16] timer: Check if timers base is handled already Date: Fri, 4 Nov 2022 15:57:33 +0100 Message-Id: <20221104145737.71236-13-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578255436144204?= X-GMAIL-MSGID: =?utf-8?q?1748578255436144204?= Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local cpu, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 641e7284f0d7..f8b2065df79b 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1912,6 +1912,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (!!base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Fri Nov 4 14:57:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15622 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456168wru; Fri, 4 Nov 2022 08:02:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7dpjHcRj9yxXlYrOcIhsXbhMNh6Hr/D5vIlR5Bd2Dxahd2BjDs5L3NJjPFTVl41oYCxdb1 X-Received: by 2002:a17:906:6a24:b0:7ad:f2df:51b7 with SMTP id qw36-20020a1709066a2400b007adf2df51b7mr18813954ejc.54.1667574175197; Fri, 04 Nov 2022 08:02:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574175; cv=none; d=google.com; s=arc-20160816; b=grEn36tM5GxpH+Ar8ZkKPe8+1jPOH9+28oBpcqYveX13HMWJVhNwdYbzLJu7Y+WzLD xda31Uvn62xAjrckguf8aOTqwbW9N2W/iSr/f47WgS4I5jsWRzZzz8c8QaZqQURD9njS uQAGirokDS/b0mE1Sr3CQIsjRg4Y3lSYCvgVsO2W6pOyz1Jp4Ov/bU5IVc9xD4uPn9c1 VH9WjRK1gUtg6vyEzz3/z3Tn6weHh7A5snfzcc1NXZkxQXMnpwewLNz8USJ/putbJKOo H0zejLKbWrCg87htjKUy+RgtPiW1S13hJVRSy+JcWI6mRXFbDI0Y6tndP7VTVOmTgc6P UlHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=7kvCElohZyUyNJGM2JneGjqwOan7vBCE2nbaLgbbE24=; b=ako+KdBKaoeHtbNjNl4i1Ig515Ahk+Ur6m2OWDL6g2e5PsMWwzZViVL1hAd4EnmF/N 2R1aZYpi+lQxiD24DO+T0nMBZ49X4M2iOqL+f6SZxwbOu1HJKSfXgTIPT42USJejNYtD M/E8yNSm08FlPaOdBDoirst3BpRK692nEY1UxNujMB+rNjkLrM+QH5Spa2o9/C7EcQ3+ nr7hiT+RyPCXvqa4hy2t3GmBr7ftrN44vNvH4CoFKQqIfNaEdxVoxz3fXsPQ9U2fktgX Ha1YrjJce01EGe9mZsF2W0haPnELeB3vXd+vPkTTsnjsgv+dT5LP4Pm9M9owOBUMbJv2 CLEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0tuI8buT; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=XvrogLHg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y7-20020a056402270700b0045cca8f9a0asi6228985edd.580.2022.11.04.08.02.31; Fri, 04 Nov 2022 08:02:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0tuI8buT; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=XvrogLHg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232357AbiKDO7C (ORCPT + 99 others); Fri, 4 Nov 2022 10:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232282AbiKDO6G (ORCPT ); Fri, 4 Nov 2022 10:58:06 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 555122EF7B for ; Fri, 4 Nov 2022 07:58:00 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7kvCElohZyUyNJGM2JneGjqwOan7vBCE2nbaLgbbE24=; b=0tuI8buTxcQd683GsIeTB2XnvF+yOTjJCOWEzNfR1pfDAmxN20GCUlC36qmFby74aWOo0b Jp1pe1Q7/zkmL1fCF/v8jjHO/UnzxZdHgcX30pQ1tKzk6X8wc2gLBVlzbzXJjm2b2EIBat 3X/5b3IX+EnjHS62nyFo7zcHS6aH4obDW2HKWCTptp8wZmLJ7xSwvEUJWpmr2mrBsM4TWl UGgmRmBnZYt7IehlzhAi2yyc0IIs/QMzUu8oK2fbAVu2DNu+Eks43tEEvRG6Zjhkz8Tn8l Ytxjkl84wObu9x6ASDBdLLk7xl4tToDtB8igF2/YNn2ShanmE65Dc86xxB7fUg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7kvCElohZyUyNJGM2JneGjqwOan7vBCE2nbaLgbbE24=; b=XvrogLHg8qBegViDEiUEOPcLE9nQosfe7dI6/vK+t49B/gbCFUaDJ8+tS/0F7HBSAJperm ZSI2edb40Ftv/oCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v4 13/16] tick/sched: Split out jiffies update helper function Date: Fri, 4 Nov 2022 15:57:34 +0100 Message-Id: <20221104145737.71236-14-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578258354513480?= X-GMAIL-MSGID: =?utf-8?q?1748578258354513480?= From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global funtion in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 20 ++++++++++++++++---- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 7b65abc9d803..8f99e35e75b4 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -158,6 +158,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; # endif diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 7f7bfe8b498d..868679ff3421 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -782,19 +782,31 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; - u64 basemono, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; + u64 basemono, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Fri Nov 4 14:57:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15625 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456220wru; Fri, 4 Nov 2022 08:02:58 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4fyfFLVSZl+sUPh6ZA9r4LZZAOq2si/q0V2abLaZ6HfXHxpC5Sd2NqPdoR+2WpjQTtkuNz X-Received: by 2002:a05:6402:2685:b0:462:73aa:c23d with SMTP id w5-20020a056402268500b0046273aac23dmr35231591edd.390.1667574178600; Fri, 04 Nov 2022 08:02:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574178; cv=none; d=google.com; s=arc-20160816; b=LpLZvHQnHI3DbVdvg6F63jCt21nQdg9k9KD6AbuQ8CAOSIIFYaWLGwInSzTUTQko8q xiJZW9MgloWgIzFzJ1toxMZMkF1T405pqe8InT/I98pUshkn2XV9CCRyC5htkF1TyZYy m8+SxjYrsJB6cdJ5vj6H3a3943jLTK1/bF33n/84KkVdcziejDGIydZMEvc8w+gdBmsE if7b0/OSQwSU2OZ17xMayUP70nl0Tha1ef48+rrMkxGSSMsNqChIArsjEj/rM7RWX4Wm FiMwKgaRN2fMgESu3TS2lBhIM/vfD9gt9Op2xJFNiG+9XmiTQnp05Fxf5SayayxLVTtm nycw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=At3lq94QvZboM8UQgclQrt7aPXPckkCESTRfz3Ldtxo=; b=r3cQ4eTlqnvGIofKlKh8FRZiHmvb+MgdwNg29kl2EpLjG1OMOd84ZJJsXSBgOYonQq UEGsS2cz2CpNGVe/BX7sVo57fZ3wz0RKfSE0RX5IErfNAp6v/I4lFkOEitYZ1H7b7Wvg fvG86tMBeCkIozvZt65YwwjhGsytqE0SOL1R6eiR1z6dWWH5ARWT5pbLuQOa7F34qLDy smxK4fIDa/XlfdIr45b5drvnc7LTk3yjYF3CAb/XEdKztNH6k/QDTCbf+MOEsAZqkivV fN8PSoi7jg06yWMELyzze0kMjmlXCxvnOSkUwlYO4etfxYDTsC+XUMFRAl/Uh0UUQ51E 5+9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=IJhHodgc; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qk31-20020a1709077f9f00b0078db89b2566si5642546ejc.699.2022.11.04.08.02.33; Fri, 04 Nov 2022 08:02:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=IJhHodgc; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232367AbiKDO7M (ORCPT + 99 others); Fri, 4 Nov 2022 10:59:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232287AbiKDO6I (ORCPT ); Fri, 4 Nov 2022 10:58:08 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 559C331DE5 for ; Fri, 4 Nov 2022 07:58:01 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=At3lq94QvZboM8UQgclQrt7aPXPckkCESTRfz3Ldtxo=; b=IJhHodgcOYpRYzo19TA+TjrB886V/wkn6vHANLP41V8FtJnnpSzUmD9ihT5p5Rk4NnWbcq VXqfbgL8oYk5KVuSrgJx+OV5jN9W2XtqZ6kd8Jmef10/36LZWzuQtbt5t/Lko54GU8Kax/ gFyO3CTEctLUpt28bC6sEAvWYpqF30J4vNan5PPx8p0qYZT6XMpY4gHVaObMIZL955ZA9X s+xkgmGAyON/dHOwMZ6DtHSE8D5HYZDCwr1vV6Q1RtRd+YxABwYmscsxZVIblx/2NOHFX1 5IKltg5Ua3Z63qvj68YiLJNMfSsUvLOreYYBPLeLXHlHmURS3cqFB1BEhep64Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=At3lq94QvZboM8UQgclQrt7aPXPckkCESTRfz3Ldtxo=; b=J2aKW2CDtRUW1YF3Hc0NvqQpkYq4Z/ignwUJcB59J7yR/mk89po45x5Gd+sttXUOsT3k9S qPZAAnaxz8ZCfNBA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 14/16] timer: Implement the hierarchical pull model Date: Fri, 4 Nov 2022 15:57:35 +0100 Message-Id: <20221104145737.71236-15-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578261668337049?= X-GMAIL-MSGID: =?utf-8?q?1748578261668337049?= Placing timers at enqueue time on a target CPU based on dubious heuristics does not make any sense: 1) Most timer wheel timers are canceled or rearmed before they expire. 2) The heuristics to predict which CPU will be busy when the timer expires are wrong by definition. So placing the timers at enqueue wastes precious cycles. The proper solution to this problem is to always queue the timers on the local CPU and allow the non pinned timers to be pulled onto a busy CPU at expiry time. Therefore split the timer storage into local pinned and global timers: Local pinned timers are always expired on the CPU on which they have been queued. Global timers can be expired on any CPU. As long as a CPU is busy it expires both local and global timers. When a CPU goes idle it arms for the first expiring local timer. If the first expiring pinned (local) timer is before the first expiring movable timer, then no action is required because the CPU will wake up before the first movable timer expires. If the first expiring movable timer is before the first expiring pinned (local) timer, then this timer is queued into a idle timerqueue and eventually expired by some other active CPU. To avoid global locking the timerqueues are implemented as a hierarchy. The lowest level of the hierarchy holds the CPUs. The CPUs are associated to groups of 8, which are seperated per node. If more than one CPU group exist, then a second level in the hierarchy collects the groups. Depending on the size of the system more than 2 levels are required. Each group has a "migrator" which checks the timerqueue during the tick for remote expirable timers. If the last CPU in a group goes idle it reports the first expiring event in the group up to the next group(s) in the hierarchy. If the last CPU goes idle it arms its timer for the first system wide expiring timer to ensure that no timer event is missed. Signed-off-by: Anna-Maria Behnsen --- v4: - Fold typo fix in comment into proper patch "timer: Split out "get next timer interrupt" functionality" - Update wrong comment for tmigr_state union definition - Fix fallout of kernel test robot --- include/linux/cpuhotplug.h | 1 + kernel/time/Makefile | 3 + kernel/time/timer.c | 58 +- kernel/time/timer_migration.c | 1239 +++++++++++++++++++++++++++++++++ kernel/time/timer_migration.h | 123 ++++ 5 files changed, 1420 insertions(+), 4 deletions(-) create mode 100644 kernel/time/timer_migration.c create mode 100644 kernel/time/timer_migration.h diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f61447913db9..95e89495251a 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -244,6 +244,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, + CPUHP_AP_TMIGR_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, CPUHP_AP_RANDOM_ONLINE, diff --git a/kernel/time/Makefile b/kernel/time/Makefile index 7e875e63ff3b..4af2a264a160 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -17,6 +17,9 @@ endif obj-$(CONFIG_GENERIC_SCHED_CLOCK) += sched_clock.o obj-$(CONFIG_TICK_ONESHOT) += tick-oneshot.o tick-sched.o obj-$(CONFIG_LEGACY_TIMER_TICK) += tick-legacy.o +ifeq ($(CONFIG_SMP),y) + obj-$(CONFIG_NO_HZ_COMMON) += timer_migration.o +endif obj-$(CONFIG_HAVE_GENERIC_VDSO) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o diff --git a/kernel/time/timer.c b/kernel/time/timer.c index f8b2065df79b..214397d84747 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -53,6 +53,7 @@ #include #include "tick-internal.h" +#include "timer_migration.h" #define CREATE_TRACE_POINTS #include @@ -592,10 +593,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle && timer->flags & TIMER_PINNED) wake_up_nohz_cpu(base->cpu); } @@ -1859,6 +1863,36 @@ void forward_and_idle_timer_bases(unsigned long basej, u64 basem, */ is_idle = time_after(nextevt, basej + 1); + if (is_idle) { + u64 next_tmigr; + + next_tmigr = tmigr_cpu_deactivate(tevt->global); + + tevt->global = KTIME_MAX; + + /* + * If CPU is the last going idle in timer migration + * hierarchy, make sure CPU will wake up in time to handle + * remote timers. next_tmigr == KTIME_MAX if other CPUs are + * still active. + */ + if (next_tmigr < tevt->local) { + u64 tmp; + + /* If we missed a tick already, force 0 delta */ + if (next_tmigr < basem) + next_tmigr = basem; + + tmp = div_u64(next_tmigr - basem, TICK_NSEC); + + nextevt = basej + (unsigned long)tmp; + tevt->local = next_tmigr; + is_idle = time_after(nextevt, basej + 1); + } else { + nextevt = nextevt_local; + } + } + /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; @@ -1898,6 +1932,9 @@ void timer_clear_idle(void) */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); + + /* Activate without holding the timer_base->lock */ + tmigr_cpu_activate(); } #endif @@ -1954,6 +1991,15 @@ static void run_timer_base(int index) __run_timer_base(base); } +#ifdef CONFIG_SMP +void timer_expire_remote(unsigned int cpu) +{ + struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + __run_timer_base(base); +} +#endif + /* * This function runs timers and the timer-tq in bottom half context. */ @@ -1963,6 +2009,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { run_timer_base(BASE_GLOBAL); run_timer_base(BASE_DEF); + + if (is_timers_nohz_active()) + tmigr_handle_remote(); } } @@ -1977,7 +2026,8 @@ static void run_local_timers(void) for (int i = 0; i < NR_BASES; i++, base++) { /* Raise the softirq only if required. */ - if (time_after_eq(jiffies, base->next_expiry)) { + if (time_after_eq(jiffies, base->next_expiry) || + (i == BASE_DEF && tmigr_requires_handle_remote())) { raise_softirq(TIMER_SOFTIRQ); return; } diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c new file mode 100644 index 000000000000..bb0eecae3390 --- /dev/null +++ b/kernel/time/timer_migration.c @@ -0,0 +1,1239 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Infrastructure for migrateable timers + * + * Copyright(C) 2022 linutronix GmbH + */ +#include +#include +#include +#include +#include + +#include "timer_migration.h" +#include "tick-internal.h" + +/* + * The timer migration mechanism is built on a hierarchy of groups. The + * lowest level group contains CPUs, the next level groups of CPU groups + * and so forth. The CPU groups are kept per node so for the normal case + * lock contention won't happen accross nodes. Depending on the number of + * CPUs per node even the next level might be kept as groups of CPU groups + * per node and only the levels above cross the node topology. + * + * Example topology for a two node system with 24 CPUs each. + * + * LVL 2 [GRP2:0] + * GRP1:0 = GRP1:M + * + * LVL 1 [GRP1:0] [GRP1:1] + * GRP0:0 - GRP0:2 GRP0:3 - GRP0:5 + * + * LVL 0 [GRP0:0] [GRP0:1] [GRP0:2] [GRP0:3] [GRP0:4] [GRP0:5] + * CPUS 0-7 8-15 16-23 24-31 32-39 40-47 + * + * The groups hold a timer queue of events sorted by expiry time. These + * queues are updated when CPUs go in idle. When they come out of idle + * ignore bit of events is set. + * + * Each group has a designated migrator CPU/group as long as a CPU/group is + * active in the group. This designated role is necessary to avoid that all + * active CPUs in a group try to migrate expired timers from other cpus, + * which would result in massive lock bouncing. + * + * When a CPU is awake, it checks in it's own timer tick the group + * hierarchy up to the point where it is assigned the migrator role or if + * no CPU is active, it also checks the groups where no migrator is set + * (TMIGR_NONE). + * + * If it finds expired timers in one of the group queues it pulls them over + * from the idle CPU and runs the timer function. After that it updates the + * group and the parent groups if required. + * + * CPUs which go idle arm their CPU local timer hardware for the next local + * (pinned) timer event. If the next migrateable timer expires after the + * next local timer or the CPU has no migrateable timer pending then the + * CPU does not queue an event in the LVL0 group. If the next migrateable + * timer expires before the next local timer then the CPU queues that timer + * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0 + * group. + * + * If the CPU is the migrator of the group then it delegates that role to + * the next active CPU in the group or sets migrator to TMIGR_NONE when + * there is no active CPU in the group. This delegation needs to be + * propagated up the hierarchy so hand over from other leaves can happen at + * all hierarchy levels w/o doing a search. + * + * When the last CPU in the system goes idle, then it drops all migrator + * duties up to the top level of the hierarchy (LVL2 in the example). It + * then has to make sure, that it arms it's own local hardware timer for + * the earliest event in the system. + * + * Lifetime rules: + * + * The groups are built up at init time or when CPUs come online. They are + * not destroyed when a group becomes empty due to offlining. The group + * just won't participate in the hierachary management anymore. Destroying + * groups would result in interesting race conditions which would just make + * the whole mechanism slow and complex. + * + * Locking rules: + * + * For setting up new groups and handling events it's required to lock both + * child and parent group. The lock odering is always bottom up. This also + * includes the per CPU locks in struct tmigr_cpu. For updating migrator + * and active CPU/group information atomic_cmpxchg() is used instead and + * only per CPU tmigr_cpu->lock is held. + * + * During setup of groups tmigr_level_list is required. It is protected by + * tmigr_mutex. + */ + +#ifdef DEBUG +# define DBG_BUG_ON(x) BUG_ON(x) +#else +# define DBG_BUG_ON(x) +#endif + +static DEFINE_MUTEX(tmigr_mutex); +static struct list_head *tmigr_level_list __read_mostly; + +static unsigned int tmigr_cores_per_group __read_mostly; +static unsigned int tmigr_hierarchy_levels __read_mostly; +static unsigned int tmigr_crossnode_level __read_mostly; + +static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu); + +#define TMIGR_NONE 0xFF +#define BIT_CNT 8 + +static DEFINE_STATIC_KEY_FALSE(tmigr_enabled); + +static inline bool is_tmigr_enabled(void) +{ + return static_branch_unlikely(&tmigr_enabled); +} + +/* + * Returns true, when @childmask corresponds to group migrator or when group + * is not active - so no migrator is set. + */ +static bool tmigr_check_migrator(struct tmigr_group *group, u32 childmask) +{ + union tmigr_state s; + + s.state = atomic_read(group->migr_state); + + if ((s.migrator != (u8)childmask) && (s.migrator != TMIGR_NONE)) + return false; + + return true; +} + +typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *); + +static void __walk_groups(up_f up, void *data, + struct tmigr_cpu *tmc) +{ + struct tmigr_group *child = NULL, *group = tmc->tmgroup; + + do { + DBG_BUG_ON(group->level >= tmigr_hierarchy_levels); + + if (up(group, child, data)) + break; + + child = group; + group = group->parent; + } while (group); +} + +static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc) +{ + lockdep_assert_held(&tmc->lock); + + __walk_groups(up, data, tmc); +} + +/** + * struct tmigr_walk - data required for walking the hierarchy + * @evt: Pointer to tmigr_event which needs to be queued (of idle + * child group) + * @childmask: childmask of child group + * @nextexp: Next CPU event expiry information which is handed + * into tmigr code by timer code + * (forward_and_idle_timer_bases()); it is furthermore + * used for next event which is queued if timer + * migration hierarchy is completely idle + * @childstate: tmigr_group->migr_state of child - will be only reread + * when cmpxchg in group fails (is required for deactive + * path and new timer path) + * @groupstate: tmigr_group->migr_state of group - will be only reread + * when cmpxchg in group fails (is required for active, + * deactive and new timer path) + */ +struct tmigr_walk { + struct tmigr_event *evt; + u32 childmask; + u64 nextexp; + union tmigr_state childstate; + union tmigr_state groupstate; +}; + +/** + * struct tmigr_remote_data - data required for (check) remote expiry + * hierarchy walk + * @basej: timer base in jiffies + * @now: timer base monotonic + * @childmask: childmask of child group + * @check: is set to 1 if there is the need to handle remote timers; + * required in tmigr_check_handle_remote() only + * @wakeup: returns expiry of first timer in idle timer migration hierarchy + * to make sure timer is handled in time; it is stored in per CPU + * tmigr_cpu struct of CPU which expires remote timers + */ +struct tmigr_remote_data { + unsigned long basej; + u64 now; + u32 childmask; + int check; + u64 wakeup; +}; + +/* + * Returns next event of timerqueue @group->events + * + * Removes timers with ignore bits and update next_expiry and event cpu + * value in group. Expiry value of group event is updated in + * tmigr_update_events() only. + */ +static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group) +{ + struct timerqueue_node *node = NULL; + struct tmigr_event *evt = NULL; + + lockdep_assert_held(&group->lock); + + group->next_expiry = KTIME_MAX; + + while ((node = timerqueue_getnext(&group->events))) { + evt = container_of(node, struct tmigr_event, nextevt); + + if (!test_bit(0, &evt->ignore)) { + group->next_expiry = evt->nextevt.expires; + return evt; + } + + /* + * Remove next timers with ignore bits, because group lock + * is held anyway + */ + if (!timerqueue_del(&group->events, node)) + break; + } + + return NULL; +} + +/* + * Return next event which is already expired of group timerqueue + * + * Event is also removed from queue. + */ +static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group, + u64 now) +{ + struct tmigr_event *evt = tmigr_next_groupevt(group); + + if (!evt || now < evt->nextevt.expires) + return NULL; + + /* + * Event is already expired. Remove it. If it's not the last event, + * then update all group event related information. + */ + if (timerqueue_del(&group->events, &evt->nextevt)) + tmigr_next_groupevt(group); + else + group->next_expiry = KTIME_MAX; + + return evt; +} + +static u64 tmigr_next_groupevt_expires(struct tmigr_group *group) +{ + struct tmigr_event *evt; + + evt = tmigr_next_groupevt(group); + + if (!evt) + return KTIME_MAX; + else + return evt->nextevt.expires; +} + +static bool tmigr_active_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u32 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + if (newstate.migrator == TMIGR_NONE) { + newstate.migrator = (u8)childmask; + + /* Changes need to be propagated */ + walk_done = false; + } + + newstate.active |= (u8)childmask; + + newstate.seq++; + + if (atomic_cmpxchg(group->migr_state, curstate.state, newstate.state) != curstate.state) { + newstate.state = curstate.state = atomic_read(group->migr_state); + goto retry; + } + + if (group->parent && (walk_done == false)) { + data->groupstate.state = atomic_read(group->parent->migr_state); + data->childmask = group->childmask; + } + + /* + * Group is active, event will be ignored - bit is updated without + * holding the lock. In case bit is set while another CPU already + * handles remote events, nothing happens, because it is clear that + * CPU became active just in this moment, or in worst case event is + * handled remote. Nothing to worry about. + */ + set_bit(0, &group->groupevt.ignore); + + return walk_done; +} + +static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) +{ + struct tmigr_walk data; + data.childmask = tmc->childmask; + data.groupstate.state = atomic_read(tmc->tmgroup->migr_state); + + set_bit(0, &tmc->cpuevt.ignore); + + walk_groups(&tmigr_active_up, &data, tmc); +} + +void tmigr_cpu_activate(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online || !tmc->idle) + return; + + raw_spin_lock(&tmc->lock); + tmc->idle = 0; + tmc->wakeup = KTIME_MAX; + __tmigr_cpu_activate(tmc); + raw_spin_unlock(&tmc->lock); +} + +/* + * Returns true, if there is nothing to be propagated to the next level + * + * @data->nextexp is reset to KTIME_MAX; it is reused for first global + * event which needs to be handled by migrator (in toplevel group) + * + * This is the only place where group event expiry value is set. + */ +static bool tmigr_update_events(struct tmigr_group *group, + struct tmigr_group *child, + struct tmigr_walk *data) +{ + struct tmigr_event *evt, *first_childevt; + bool walk_done; + u64 nextexp; + + if (child) { + if (data->childstate.active) + return true; + + raw_spin_lock(&child->lock); + raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING); + + first_childevt = tmigr_next_groupevt(child); + nextexp = child->next_expiry; + evt = &child->groupevt; + } else { + nextexp = data->nextexp; + + /* + * Set @data->nextexp to KTIME_MAX; it is reused for first + * global event which needs to be handled by migrator (in + * toplevel group) + */ + data->nextexp = KTIME_MAX; + + first_childevt = evt = data->evt; + if (test_bit(0, &evt->ignore)) + return true; + + raw_spin_lock(&group->lock); + } + + if (nextexp == KTIME_MAX) { + set_bit(0, &evt->ignore); + walk_done = true; + goto unlock; + } + + walk_done = !group->parent; + + /* + * Update of event cpu and ignore bit is required only when @child + * is set (child is equal or higher than lvl0), but it doesn't + * matter if it is written once more to per cpu event; make the + * update unconditional. + */ + evt->cpu = first_childevt->cpu; + clear_bit(0, &evt->ignore); + + /* + * If child event is already queued in group, remove it from queue + * when expiry time changed only + */ + if (timerqueue_node_queued(&evt->nextevt)) { + if (evt->nextevt.expires == nextexp) + goto check_toplvl; + + if (evt->nextevt.expires != nextexp && + !timerqueue_del(&group->events, &evt->nextevt)) + group->next_expiry = KTIME_MAX; + } + + evt->nextevt.expires = nextexp; + + if (timerqueue_add(&group->events, &evt->nextevt)) + group->next_expiry = nextexp; + +check_toplvl: + if (walk_done && (data->groupstate.migrator == TMIGR_NONE)) { + /* + * Toplevel group is idle and it has to be ensured global + * timers are handled in time. (This could be optimized by + * keeping track of the last global scheduled event and + * only arming it on CPU if the new event is earlier. Not + * sure if its worth the complexity.) + */ + data->nextexp = tmigr_next_groupevt_expires(group); + } + +unlock: + raw_spin_unlock(&group->lock); + + if (child) + raw_spin_unlock(&child->lock); + + return walk_done; +} + +static bool tmigr_new_timer_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_walk *data = ptr; + bool walk_done; + + walk_done = tmigr_update_events(group, child, data); + + if (!walk_done) { + /* Update state information for next iteration */ + data->childstate.state = atomic_read(group->migr_state); + if (group->parent) + data->groupstate.state = atomic_read(group->parent->migr_state); + } + + return walk_done; +} + +/* + * Returns expiry of next timer that needs to be handled. KTIME_MAX is + * returned, when an active CPU will handle all timer migration hierarchy + * timers. + */ +static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .evt = &tmc->cpuevt, + .nextexp = nextexp }; + + lockdep_assert_held(&tmc->lock); + + if (tmc->remote) + return KTIME_MAX; + + clear_bit(0, &tmc->cpuevt.ignore); + + data.groupstate.state = atomic_read(tmc->tmgroup->migr_state); + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + /* If there is a new first global event, make sure it is handled */ + return data.nextexp; +} + +static bool tmigr_inactive_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u32 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + /* Reset active bit when child is no longer active */ + if (!data->childstate.active) + newstate.active &= ~(u8)childmask; + + if (newstate.migrator == (u8)childmask) { + /* + * Find a new migrator for the group, because child group + * is idle! + */ + if (!data->childstate.active) { + unsigned long new_migr_bit, active = newstate.active; + + new_migr_bit = find_first_bit(&active, BIT_CNT); + + /* Changes need to be propagated */ + walk_done = false; + data->childmask = group->childmask; + + if (new_migr_bit != BIT_CNT) + newstate.migrator = BIT(new_migr_bit); + else + newstate.migrator = TMIGR_NONE; + } + } + + newstate.seq++; + + DBG_BUG_ON((newstate.migrator != TMIGR_NONE) && !(newstate.active)); + + if (atomic_cmpxchg(group->migr_state, curstate.state, newstate.state) != curstate.state) { + /* + * Something changed in child/parent group in the meantime, + * reread the state of child and parent; Update of + * data->childstate is required for event handling; + */ + if (child) + data->childstate.state = atomic_read(child->migr_state); + newstate.state = curstate.state = atomic_read(group->migr_state); + + goto retry; + } + + data->groupstate = newstate; + + /* Event Handling */ + tmigr_update_events(group, child, data); + + if (group->parent && (walk_done == false)) { + data->childstate = newstate; + data->groupstate.state = atomic_read(group->parent->migr_state); + } + + /* + * data->nextexp was set by tmigr_update_events() and contains the + * expiry of first global event which needs to be handled + */ + if (data->nextexp != KTIME_MAX) { + DBG_BUG_ON(group->parent); + /* + * Toplevel path - If this cpu is about going offline wake + * up some random other cpu so it will take over the + * migrator duty and program its timer properly. Ideally + * wake the cpu with the closest expiry time, but that's + * overkill to figure out. + */ + if (!(this_cpu_ptr(&tmigr_cpu)->online)) { + unsigned int cpu = smp_processor_id(); + + cpu = cpumask_any_but(cpu_online_mask, cpu); + smp_send_reschedule(cpu); + } + } + + return walk_done; +} + +static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .childmask = tmc->childmask, + .evt = &tmc->cpuevt, + .nextexp = nextexp, + .childstate.state = 0 }; + + data.groupstate.state = atomic_read(tmc->tmgroup->migr_state); + + /* + * If nextexp is KTIME_MAX, CPU event will be ignored because, + * local timer expires before global timer, no global timer is set + * or CPU goes offline. + */ + if (nextexp != KTIME_MAX) + clear_bit(0, &tmc->cpuevt.ignore); + + walk_groups(&tmigr_inactive_up, &data, tmc); + return data.nextexp; +} + +/** + * tmigr_cpu_deactivate - Put current CPU into inactive state + * @nextexp: The next timer event expiry set in the current CPU + * + * Must be called with interrupts disabled. + * + * Return: next event of the current CPU or next event from the hierarchy + * if this CPU is the top level migrator or hierarchy is completely idle. + */ +u64 tmigr_cpu_deactivate(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret = KTIME_MAX; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return ret; + + raw_spin_lock(&tmc->lock); + + /* + * CPU is already deactivated in timer migration + * hierarchy. tick_nohz_get_sleep_length() calls + * tick_nohz_next_event() and thereby timer idle path is + * executed once more. tmc->wakeup holds the first timer, when + * timer migration hierarchy is completely idle and remote + * expiry was done. If there is no new next expiry value + * handed in which should be inserted into the timer migration + * hierarchy, wakeup value is returned. + */ + if (tmc->idle) { + ret = tmc->wakeup; + + tmc->wakeup = KTIME_MAX; + + if (nextexp != KTIME_MAX) { + if (nextexp != tmc->cpuevt.nextevt.expires || + test_bit(0, &tmc->cpuevt.ignore)) + ret = tmigr_new_timer(tmc, nextexp); + } + + goto unlock; + } + + /* + * When tmigr_remote is active, set cpu inactive path and queuing of + * nextexp is done by handle remote path. + */ + ret = __tmigr_cpu_deactivate(tmc, nextexp); + + tmc->idle = 1; + +unlock: + raw_spin_unlock(&tmc->lock); + return ret; +} + +static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, + unsigned long jif) +{ + struct timer_events tevt; + struct tmigr_walk data; + struct tmigr_cpu *tmc; + u64 next = KTIME_MAX; + unsigned long flags; + + tmc = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_lock_irqsave(&tmc->lock, flags); + /* + * Remote CPU is offline or no longer idle or other cpu handles cpu + * timers already or next event was already expired - return! + */ + if (!tmc->online || tmc->remote || test_bit(0, &tmc->cpuevt.ignore) || + now < tmc->cpuevt.nextevt.expires) { + raw_spin_unlock_irqrestore(&tmc->lock, flags); + return next; + } + + tmc->remote = 1; + + /* Drop the lock to allow the remote CPU to exit idle */ + raw_spin_unlock_irqrestore(&tmc->lock, flags); + + if (cpu != smp_processor_id()) + timer_expire_remote(cpu); + + /* next event of cpu */ + get_next_timer_interrupt_remote(jif, now, &tevt, cpu); + + raw_spin_lock_irqsave(&tmc->lock, flags); + /* + * Nothing more to do when CPU came out of idle in the meantime - needs + * to be checked when holding the base lock to prevent race. + */ + if (!tmc->idle) + goto unlock; + + data.evt = &tmc->cpuevt; + data.nextexp = tevt.global; + data.groupstate.state = atomic_read(tmc->tmgroup->migr_state); + clear_bit(0, &tmc->cpuevt.ignore); + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + next = data.nextexp; + +unlock: + tmc->remote = 0; + raw_spin_unlock_irqrestore(&tmc->lock, flags); + + return next; +} + +static bool tmigr_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u64 now, next = KTIME_MAX; + unsigned long flags, jif; + struct tmigr_event *evt; + u32 childmask; + + jif = data->basej; + now = data->now; + + childmask = data->childmask; + +again: + /* + * Handle the group only if @childmask is the migrator or if the + * group has no migrator. Otherwise the group is active and is + * handled by its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + raw_spin_lock_irqsave(&group->lock, flags); + + evt = tmigr_next_expired_groupevt(group, now); + + if (evt) { + unsigned int remote_cpu; + + remote_cpu = READ_ONCE(evt->cpu); + + raw_spin_unlock_irqrestore(&group->lock, flags); + + next = tmigr_handle_remote_cpu(remote_cpu, now, jif); + + /* check if there is another event, that needs to be handled */ + goto again; + } else { + raw_spin_unlock_irqrestore(&group->lock, flags); + } + + /* Update of childmask for next level */ + data->childmask = group->childmask; + data->wakeup = next; + + return false; +} + +/** + * tmigr_handle_remote - Handle migratable timers on remote idle CPUs + * + * Called from the timer soft interrupt with interrupts enabled. + */ +void tmigr_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + unsigned long flags; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return; + + /* + * NOTE: This is a doubled check because migrator test will be done + * in tmigr_handle_remote_up() anyway. Keep this check to fasten + * the return when nothing has to be done. + */ + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return; + + data.now = get_jiffies_update(&data.basej); + data.childmask = tmc->childmask; + data.wakeup = KTIME_MAX; + + __walk_groups(&tmigr_handle_remote_up, &data, tmc); + + raw_spin_lock_irqsave(&tmc->lock, flags); + if (tmc->idle) + tmc->wakeup = data.wakeup; + + raw_spin_unlock_irqrestore(&tmc->lock, flags); + + return; +} + +static bool tmigr_requires_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u32 childmask; + + childmask = data->childmask; + + /* + * Handle the group only if child is the migrator or if the group + * has no migrator. Otherwise the group is active and is handled by + * its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + /* + * Racy lockless check for next_expiry + */ + if (data->now >= group->next_expiry) { + data->check = 1; + return true; + } + + /* Update of childmask for next level */ + data->childmask = group->childmask; + return false; +} + +int tmigr_requires_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return 0; + + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return 0; + + data.now = get_jiffies_update(&data.basej); + data.childmask = tmc->childmask; + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + return data.check; +} + +static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl, + unsigned int node, atomic_t *migr_state) +{ + union tmigr_state s; + + raw_spin_lock_init(&group->lock); + + group->level = lvl; + group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE; + + group->num_childs = 0; + + /* + * num_cores is required for level=0 groups only during setup and + * when siblings exists but it doesn't matter if this value is set + * in other groups as well + */ + group->num_cores = 1; + + s.migrator = TMIGR_NONE; + s.active = 0; + s.seq = 0; + atomic_set(migr_state, s.state); + + group->migr_state = migr_state; + + timerqueue_init_head(&group->events); + timerqueue_init(&group->groupevt.nextevt); + group->groupevt.nextevt.expires = KTIME_MAX; + group->next_expiry = KTIME_MAX; + set_bit(0, &group->groupevt.ignore); +} + +static bool sibling_in_group(int newcpu, struct tmigr_group *group) +{ + int i, cpu; + + /* Find a sibling of newcpu in group members */ + for (i = 0; i < group->num_childs; i++) { + cpu = group->cpus[i]; + + if (cpumask_test_cpu(newcpu, topology_sibling_cpumask(cpu))) + return true; + } + return false; +} + +static struct tmigr_group *tmigr_get_group(unsigned int cpu, unsigned int node, + unsigned int lvl) +{ + struct tmigr_group *tmp, *group = NULL; + bool first_loop = true; + atomic_t *migr_state; + +reloop: + /* Try to attach to an exisiting group first */ + list_for_each_entry(tmp, &tmigr_level_list[lvl], list) { + /* + * If @lvl is below the cross numa node level, check whether + * this group belongs to the same numa node. + */ + if (lvl < tmigr_crossnode_level && tmp->numa_node != node) + continue; + + /* Capacity left? */ + if (tmp->num_childs >= TMIGR_CHILDS_PER_GROUP) + continue; + + /* + * If this is the lowest level of the hierarchy, make sure + * that thread siblings share a group. It is only executed + * when siblings exist. ALL groups of lowest level needs to + * be checked for thread sibling, before thread cpu is + * added to a random group with capacity. When all groups + * are checked and no thread sibling was found, reloop of + * level zero groups is required to get a group with + * capacity. + */ + if (!lvl && (tmigr_cores_per_group != TMIGR_CHILDS_PER_GROUP)) { + if (first_loop == true && !sibling_in_group(cpu, tmp)) { + continue; + } else if (first_loop == false) { + if (tmp->num_cores >= tmigr_cores_per_group) + continue; + else + tmp->num_cores++; + } + } + + group = tmp; + break; + } + + if (group) { + return group; + } else if (first_loop == true) { + first_loop = false; + goto reloop; + } + + /* Allocate and set up a new group with corresponding migr_state */ + group = kzalloc_node(sizeof(*group), GFP_KERNEL, node); + if (!group) + return ERR_PTR(-ENOMEM); + + migr_state = kzalloc_node(sizeof(atomic_t), GFP_KERNEL, node); + if (!migr_state) { + kfree(group); + return ERR_PTR(-ENOMEM); + } + + tmigr_init_group(group, lvl, node, migr_state); + /* Setup successful. Add it to the hierarchy */ + list_add(&group->list, &tmigr_level_list[lvl]); + return group; +} + +static void tmigr_connect_child_parent(struct tmigr_group *child, + struct tmigr_group *parent) +{ + union tmigr_state childstate; + unsigned long flags; + + raw_spin_lock_irqsave(&child->lock, flags); + raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING); + + child->parent = parent; + child->childmask = BIT(parent->num_childs++); + + raw_spin_unlock(&parent->lock); + raw_spin_unlock_irqrestore(&child->lock, flags); + + /* + * To prevent inconsistent states, active childs needs to be active + * in new parent as well. Inactive childs are already marked + * inactive in parent group. + */ + childstate.state = atomic_read(child->migr_state); + if (childstate.migrator != TMIGR_NONE) { + struct tmigr_walk data; + + data.childmask = child->childmask; + data.groupstate.state = atomic_read(parent->migr_state); + + /* + * There is only one new level per time. When connecting + * child and parent and set child active when parent is + * inactive, parent needs to be the upperst + * level. Otherwise there went something wrong! + */ + WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent); + } +} + +static int tmigr_setup_groups(unsigned int cpu, unsigned int node) +{ + struct tmigr_group *group, *child, **stack; + int top = 0, err = 0, i = 0; + struct list_head *lvllist; + size_t sz; + + sz = sizeof(struct tmigr_group *) * tmigr_hierarchy_levels; + stack = kzalloc(sz, GFP_KERNEL); + if (!stack) + return -ENOMEM; + + do { + group = tmigr_get_group(cpu, node, i); + if (IS_ERR(group)) { + err = IS_ERR(group); + break; + } + + top = i; + stack[i++] = group; + + /* + * When booting only less CPUs of a system than CPUs are + * available, not all calculated hierarchy levels are required. + * + * The loop is aborted as soon as the highest level, which might + * be different from tmigr_hierarchy_levels, contains only a + * single group. + */ + if (group->parent || i == tmigr_hierarchy_levels || + (list_empty(&tmigr_level_list[i]) && + list_is_singular(&tmigr_level_list[i - 1]))) + break; + + } while (i < tmigr_hierarchy_levels); + + do { + group = stack[--i]; + + if (err < 0) { + list_del(&group->list); + kfree(group); + continue; + } + + DBG_BUG_ON(i != group->level); + + /* + * Update tmc -> group / child -> group connection + */ + if (i == 0) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned long flags; + + raw_spin_lock_irqsave(&group->lock, flags); + + tmc->tmgroup = group; + tmc->childmask = BIT(group->num_childs); + + group->cpus[group->num_childs++] = cpu; + + raw_spin_unlock_irqrestore(&group->lock, flags); + + /* There are no childs that needs to be connected */ + continue; + } else { + child = stack[i - 1]; + tmigr_connect_child_parent(child, group); + } + + /* check if upperst level was newly created */ + if (top != i) + continue; + + DBG_BUG_ON(top == 0); + + lvllist = &tmigr_level_list[top]; + if (group->num_childs == 1 && list_is_singular(lvllist)) { + lvllist = &tmigr_level_list[top - 1]; + list_for_each_entry(child, lvllist, list) { + if (child->parent) + continue; + + tmigr_connect_child_parent(child, group); + } + } + } while (i > 0); + + kfree(stack); + + return err; +} + +static int tmigr_add_cpu(unsigned int cpu) +{ + unsigned int node = cpu_to_node(cpu); + int ret; + mutex_lock(&tmigr_mutex); + ret = tmigr_setup_groups(cpu, node); + mutex_unlock(&tmigr_mutex); + + return ret; +} + +static int tmigr_cpu_online(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned long flags; + unsigned int ret; + + /* First online attempt? Initialize CPU data */ + if (!tmc->tmgroup) { + raw_spin_lock_init(&tmc->lock); + + ret = tmigr_add_cpu(cpu); + if (ret < 0) + return ret; + + if (tmc->childmask == 0) + return -EINVAL; + + timerqueue_init(&tmc->cpuevt.nextevt); + tmc->cpuevt.nextevt.expires = KTIME_MAX; + set_bit(0, &tmc->cpuevt.ignore); + tmc->cpuevt.cpu = cpu; + + tmc->remote = 0; + tmc->idle = 0; + tmc->wakeup = KTIME_MAX; + } + raw_spin_lock_irqsave(&tmc->lock, flags); + __tmigr_cpu_activate(tmc); + tmc->online = 1; + raw_spin_unlock_irqrestore(&tmc->lock, flags); + return 0; +} + +static int tmigr_cpu_offline(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&tmc->lock); + tmc->online = 0; + __tmigr_cpu_deactivate(tmc, KTIME_MAX); + raw_spin_unlock_irq(&tmc->lock); + + return 0; +} + +static int __init tmigr_init(void) +{ + unsigned int cpulvl, nodelvl, cpus_per_node, i, ns; + unsigned int nnodes = num_possible_nodes(); + unsigned int ncpus = num_possible_cpus(); + int ret = -ENOMEM; + size_t sz; + + /* Nothing to do if running on UP */ + if (ncpus == 1) + return 0; + + /* + * Unfortunately there is no reliable way to determine the number of SMT + * siblings in a generic way. tmigr_init() is called after SMP bringup, + * so for the normal boot case it can be assumed that all siblings have + * been brought up and the number of siblings of the current cpu can be + * used. If someone booted with 'maxcpus=N/2' on the kernel command line + * and (at least x86) bring up the siblings later then the siblings will + * end up in different groups. Bad luck. + */ + ns = cpumask_weight(topology_sibling_cpumask(raw_smp_processor_id())); + tmigr_cores_per_group = TMIGR_CHILDS_PER_GROUP; + if (ns >= 2 && ns < TMIGR_CHILDS_PER_GROUP) + tmigr_cores_per_group /= ns; + + /* + * Calculate the required hierarchy levels. Unfortunately there is no + * reliable information available, unless all possible CPUs have been + * brought up and all numa nodes are populated. + * + * Estimate the number of levels with the number of possible nodes and + * the number of possible cpus. Assume CPUs are spread evenly accross + * nodes. We cannot rely on cpumask_of_node() because there only already + * online CPUs are considered. + */ + cpus_per_node = DIV_ROUND_UP(ncpus, nnodes); + + /* Calc the hierarchy levels required to hold the CPUs of a node */ + cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node), + ilog2(TMIGR_CHILDS_PER_GROUP)); + + /* Calculate the extra levels to connect all nodes */ + nodelvl = DIV_ROUND_UP(order_base_2(nnodes), + ilog2(TMIGR_CHILDS_PER_GROUP)); + + tmigr_hierarchy_levels = cpulvl + nodelvl; + + /* + * If a numa node spawns more than one CPU level group then the next + * level(s) of the hierarchy contains groups which handle all CPU groups + * of the same numa node. The level above goes accross numa nodes. Store + * this information for the setup code to decide when node matching is + * not longer required. + */ + tmigr_crossnode_level = cpulvl; + + sz = sizeof(struct list_head) * tmigr_hierarchy_levels; + tmigr_level_list = kzalloc(sz, GFP_KERNEL); + if (!tmigr_level_list) + goto err; + + for (i = 0; i < tmigr_hierarchy_levels; i++) + INIT_LIST_HEAD(&tmigr_level_list[i]); + + pr_info("Timer migration: %d hierarchy levels; %d childs per group;" + " %d cores_per_group; %d crossnode level\n", + tmigr_hierarchy_levels, TMIGR_CHILDS_PER_GROUP, + tmigr_cores_per_group, tmigr_crossnode_level); + + ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online", + tmigr_cpu_online, tmigr_cpu_offline); + if (ret) + goto err; + + static_branch_enable(&tmigr_enabled); + + return 0; + +err: + pr_err("Timer migration setup failed\n"); + return ret; +} +late_initcall(tmigr_init); diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h new file mode 100644 index 000000000000..ab2953c89ef2 --- /dev/null +++ b/kernel/time/timer_migration.h @@ -0,0 +1,123 @@ +#ifndef _KERNEL_TIME_MIGRATION_H +#define _KERNEL_TIME_MIGRATION_H + +/* Per group capacity. Must be a power of 2! */ +#define TMIGR_CHILDS_PER_GROUP 8 + +/** + * struct tmigr_event - a timer event associated to a CPU + * @nextevt: The node to enqueue an event in the parent group queue + * @cpu: The CPU to which this event belongs + * @ignore: Hint whether the event could be ignored; it is set when + * CPU or group is active; + */ +struct tmigr_event { + struct timerqueue_node nextevt; + unsigned int cpu; + unsigned long ignore; +}; + +/** + * struct tmigr_group - timer migration hierarchy group + * @lock: Lock protecting the event information + * @cpus: Array with CPUs which are member of group; required for + * sibling CPUs; used only when level == 0 + * @parent: Pointer to parent group + * @list: List head that is added to per level tmigr_level_list + * @level: Hierarchy level of group + * @numa_node: Is set to numa node when level < tmigr_crossnode_level; + * otherwise it is set to NUMA_NO_NODE; Required for setup + * only + * @num_childs: Counter of group childs; Required for setup only + * @num_cores: Counter of cores per group; Required for setup only when + * level == 0 and siblings exist + * @migr_state: State of group (see struct tmigr_state) + * @childmask: childmask of group in parent group; is set during setup + * never changed; could be read lockless + * @events: Timer queue for child events queued in the group + * @groupevt: Next event of group; it is only reliable when group is + * !active (ignore bit is set when group is active) + * @next_expiry: Base monotonic expiry time of next event of group; + * Used for racy lockless check whether remote expiry is + * required; it is always reliable + */ +struct tmigr_group { + raw_spinlock_t lock; + unsigned int cpus[TMIGR_CHILDS_PER_GROUP]; + struct tmigr_group *parent; + struct list_head list; + unsigned int level; + unsigned int numa_node; + unsigned int num_childs; + unsigned int num_cores; + atomic_t *migr_state; + u32 childmask; + struct timerqueue_head events; + struct tmigr_event groupevt; + u64 next_expiry; +}; + +/** + * struct tmigr_cpu - timer migration per CPU group + * @lock: Lock protecting tmigr_cpu group information + * @online: Indicates wheter CPU is online + * @idle: Indicates wheter CPU is idle in timer migration hierarchy + * @remote: Is set when timers of CPU are expired remote + * @tmgroup: Pointer to parent group + * @childmask: childmask of tmigr_cpu in parent group + * @cpuevt: CPU event which could be queued into parent group + * @wakeup: Stores the first timer when the timer migration hierarchy is + * completely idle and remote expiry was done; is returned to + * timer code when tmigr_cpu_deactive() is called and group is + * idle; afterwards a reset to KTIME_MAX is required; + */ +struct tmigr_cpu { + raw_spinlock_t lock; + int online; + int idle; + int remote; + struct tmigr_group *tmgroup; + u32 childmask; + struct tmigr_event cpuevt; + u64 wakeup; +}; + +/** + * union tmigr_state - state of tmigr_group + * @state: Combined version of the state - only used for atomic + * read/cmpxchg function + * @struct: Splitted version of the state - only use the struct members to + * update information to stay independant of endianess + */ +union tmigr_state { + u32 state; + /** + * struct - splitted state of tmigr_group + * @active: Contains each childmask bit of active childs + * @migrator: Contains childmask of child which is migrator + * @seq: Seqence number to prevent race when update in child + * group are propagated in wrong order (especially when + * migrator changes are involved) + */ + struct { + u8 active; + u8 migrator; + u16 seq; + } __packed; +}; + +#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) +extern void tmigr_handle_remote(void); +extern int tmigr_requires_handle_remote(void); +extern void tmigr_cpu_activate(void); +extern u64 tmigr_cpu_deactivate(u64 nextevt); +extern void timer_expire_remote(unsigned int cpu); +#else +static inline void tmigr_handle_remote(void) { } +extern inline int tmigr_requires_handle_remote(void) { return 0; } +static inline void tmigr_cpu_activate(void) { } +static inline u64 tmigr_cpu_deactivate(u64 nextevt) { return KTIME_MAX; } +extern inline void timer_expire_remote(unsigned int cpu) { } +#endif + +#endif From patchwork Fri Nov 4 14:57:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15624 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456263wru; Fri, 4 Nov 2022 08:03:01 -0700 (PDT) X-Google-Smtp-Source: AMsMyM78lG8qUef9vVuqpC4xDOsCxt8V9ey3ZAzhYtopdF5NxW/kwGVYoUbQLB4aWsHd3+AmC2gl X-Received: by 2002:a17:907:da0:b0:7ad:9760:e1d2 with SMTP id go32-20020a1709070da000b007ad9760e1d2mr33961667ejc.419.1667574181093; Fri, 04 Nov 2022 08:03:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574181; cv=none; d=google.com; s=arc-20160816; b=yIzcYgmad+xfcw546VgGPsuqUB4uUdn9cwzLm7UuIdY47VhJ4d9/tnF6bEJMx5kKPm V3BUkU1t0lTpt4R7S8uD0l5jBgTuI+wkVP3Qi61h5ijaJ6CuePQea+Ut3R++lGo8pZLN tqFgECkNKoZp/BeAtCTJRy4fH5egIuouoZCR8iw7LTTK9GuQm5rYrLU1567yEuJcrq0+ HUfuUlRGRGJlW/WEvhIotfYu7NE4Zq+IvxgqutTuzOaqYKyqLw8/mu977hRqRxm1D8vv yOLWqqx2XbShDYNmDJQagdk0jxpyfo/uxp/v5kSmgLApXBQX8h+mgLCY2XqdSnhvAdvm 1yWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=W2Z5wfLOO31dLK5ATp28FZtKqoTdnygaDEB1lQ4ljsQ=; b=y6lOnUzrgpEUuorYFT1w9H3VoPHl00SIRLirJhUuXkmYBcsnBwGYOF4/nsQ1opEYZJ 10/ZqaI6BnIM4MyN4MIP5hqGz96/OnURziVihxxKfWxmS4GMHTLwRdz8zHn6GJLav0Jy t+nWgBfmPaiH0ezSxupRovNjgu2oUA5cc2kosKwKJSOU6EOYj6Pa/EsiVsFumj6EKVY1 T0Wi7sBmE1HKFGNUANO5YmwxdNkKJgcky4npzxFOZGzl9imxP/FUACDIob/9b0+CFDjY wy8EjphRysCQMHEp97NNAISLlvHEQGi6Xpste760NYQwwDO1xAXr1JUKgCFpBR7M546w EdlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=O+Jhekuv; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hk18-20020a170906c9d200b0078d484e0e7esi4028794ejb.488.2022.11.04.08.02.35; Fri, 04 Nov 2022 08:03:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=O+Jhekuv; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232382AbiKDO7P (ORCPT + 99 others); Fri, 4 Nov 2022 10:59:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232303AbiKDO6L (ORCPT ); Fri, 4 Nov 2022 10:58:11 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F53131DFC for ; Fri, 4 Nov 2022 07:58:03 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W2Z5wfLOO31dLK5ATp28FZtKqoTdnygaDEB1lQ4ljsQ=; b=O+Jhekuvj/Vs4O0Fv0wiCcvtfwVMvDL0sRceseCHyQvUjhbuznkDst7HIHIrgTBLV/HLm+ 2HP2ZHBMa2+eqr1xCtob5g9VKjwTg8YxSDMKHyJuVTNRN82L+IVoT5s2VOlq/AO3Q4N5gp aZ+RV5F+fFfr2h/yQCuG8zqEH476gV/iWi9ZmUhNJaNCZuWgiPgYCiVJvopbzI73BnVwnw Kt449AvMJDMtCG66eVaynLB4DD+YKxJobeEBe8o6nJUgJlNx4tYyf9kCpw0oCvBqlUnQSk LitsbFaEZn8f5ze1EQvoK05o9LNVWSaa5G48ZxeB0QT2Uv5GNk56bCj5DcwAgQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W2Z5wfLOO31dLK5ATp28FZtKqoTdnygaDEB1lQ4ljsQ=; b=omgiynJPjD0N3iEG6eWVA6drD4UhDBaTeuoXxIBdBLjTi+3VtaglinHCVglCDMkKSWgPb1 OM32fL5ycpH6zHCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen Subject: [PATCH v4 15/16] timer_migration: Add tracepoints Date: Fri, 4 Nov 2022 15:57:36 +0100 Message-Id: <20221104145737.71236-16-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578264563512858?= X-GMAIL-MSGID: =?utf-8?q?1748578264563512858?= The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen --- include/trace/events/timer_migration.h | 277 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 24 +++ 2 files changed, 301 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..0c4824056930 --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,277 @@ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_childs ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->numa_node = child->parent->num_childs; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_childs=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_childs) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_childs ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->numa_node = tmc->tmgroup->num_childs; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_childs=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_childs) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + ), + + TP_printk("cpu=%d parent=%p", __entry->cpu, __entry->parent) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +TRACE_EVENT(tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu", + __entry->cpu, __entry->parent, __entry->nextevt) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index bb0eecae3390..838fc4a6d62e 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -13,6 +13,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -317,6 +320,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ set_bit(0, &group->groupevt.ignore); + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -341,6 +346,7 @@ void tmigr_cpu_activate(void) raw_spin_lock(&tmc->lock); tmc->idle = 0; tmc->wakeup = KTIME_MAX; + trace_tmigr_cpu_active(tmc); __tmigr_cpu_activate(tmc); raw_spin_unlock(&tmc->lock); } @@ -435,6 +441,9 @@ static bool tmigr_update_events(struct tmigr_group *group, data->nextexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, data->childstate, + data->groupstate, nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -478,6 +487,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + clear_bit(0, &tmc->cpuevt.ignore); data.groupstate.state = atomic_read(tmc->tmgroup->migr_state); @@ -576,6 +587,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, } } + trace_tmigr_group_set_cpu_inactive(group, data->childstate, childmask); + return walk_done; } @@ -652,6 +665,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) tmc->idle = 1; unlock: + trace_tmigr_cpu_idle(tmc, ret); raw_spin_unlock(&tmc->lock); return ret; } @@ -678,6 +692,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return next; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = 1; /* Drop the lock to allow the remote CPU to exit idle */ @@ -728,6 +744,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -963,6 +980,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, unsigned int node, tmigr_init_group(group, lvl, node, migr_state); /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -981,6 +999,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irqrestore(&child->lock, flags); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active childs needs to be active * in new parent as well. Inactive childs are already marked @@ -1067,6 +1087,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irqrestore(&group->lock, flags); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no childs that needs to be connected */ continue; } else { @@ -1135,6 +1157,7 @@ static int tmigr_cpu_online(unsigned int cpu) tmc->wakeup = KTIME_MAX; } raw_spin_lock_irqsave(&tmc->lock, flags); + trace_tmigr_cpu_online(tmc); __tmigr_cpu_activate(tmc); tmc->online = 1; raw_spin_unlock_irqrestore(&tmc->lock, flags); @@ -1148,6 +1171,7 @@ static int tmigr_cpu_offline(unsigned int cpu) raw_spin_lock_irq(&tmc->lock); tmc->online = 0; __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); return 0; From patchwork Fri Nov 4 14:57:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 15627 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp456847wru; Fri, 4 Nov 2022 08:03:51 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5TEpq7g+eYN6zcHh+81z2DRV/RV7SuQpokY3NXD12XqGxbEQ84wjkDazAROuwDKOwmmeec X-Received: by 2002:a17:906:ee86:b0:741:89bc:27a1 with SMTP id wt6-20020a170906ee8600b0074189bc27a1mr35890299ejb.725.1667574231015; Fri, 04 Nov 2022 08:03:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667574231; cv=none; d=google.com; s=arc-20160816; b=fIk54FOE/BF5Q0+xl+21T19BFGiXtLkGR+gXPz3O+DWcHcr1yLWeuP2I98VQUTThID AaEvNcza+LRtmKaJuQVHTRu7MkbHwkU3YQ2byl3cLW6twpsskUpjU3tteovOLXchGbPO hXP9UKBXDMUrK+ehkt1wC+sM5ppwlpf6tdseVLoY2Z2lh0YhC38Lwypg3fZVCZr6J+ld 5xjHgxh1JkOkaNqVrcIHakN0jfvBTgdIM/ZmG72f7g7MJJiiKBppvHI/H2PheV6cwbXk FxLs7UqNVp1+EKI//rdsQNP1XaGf7lujqwf2F/aEAaH50GutYExOwQkRP/7qZdFJb/tO yIjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=FdVcoBx7/tuIOiTEjcb989uDKzmjA/NfDt6Q7fyRUUU=; b=VzefIm/6QG5TPf622cVc/OCXJyi1fboIQxhjidHf9XPt0NTTdwzsd4D1KY5jmuKl9g EVxX29JtxYdiuXQxG5Ph89lGV8HnoFm0SL/ScQl1mtc4wubTZEQRFCy1yYihngGSR953 /v+/XDlQ1ldqFxDflbQ17Dl7lKAQgiqiYIPaEMLZfiJaWuGzzSKprc/lMQPGMuwCyKt9 eoktjTZeQffTpQLVvsISyWZ3TF4qsEyfazTKI9AaqZ8EoBqndKCp0u3ipLuIQfEYWsh7 l0bAE5OEyegKOSbdJovWOjx6KpXF0LIMfhHUaZIxKizbm+kRiND1qKHqyT+Q84fh7qpp bpZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0g22eLEg; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w17-20020a056402269100b00462750bbe08si6034012edd.522.2022.11.04.08.03.26; Fri, 04 Nov 2022 08:03:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0g22eLEg; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232385AbiKDO7W (ORCPT + 99 others); Fri, 4 Nov 2022 10:59:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232305AbiKDO6M (ORCPT ); Fri, 4 Nov 2022 10:58:12 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F5E131ED0 for ; Fri, 4 Nov 2022 07:58:03 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1667573881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FdVcoBx7/tuIOiTEjcb989uDKzmjA/NfDt6Q7fyRUUU=; b=0g22eLEgtf+duujb8mx91XeZxs3w2lmo2ok9sJqTz2qndFHIDwU3ND4lBw/yvb4ozJPAjZ lHmijG/mdXeNls5aMN5YUTPHyTNdxGmwASeDPF+pzPK6km2aTNX1tVRoVQGUZoGiSFO6Xk K5XDGgrsd1V2J64kvfH7tp1o8gcLzGJ0wJQR+xX5+xE9IPUQ4/bRGjJ8ipdrO6ulCdtXC0 O6eUfTWLlz+1G5nmHJlCc5Zd1kjgrbNJAj18Ta/qLQFRnDBF6im1WC56JFrAPvsmgwEIoV BUGJ69P/+9dM9YQ0AtHniaxG+brnxPJ+G7CYXVh1VQ6nL77/rjyKI5jXZDWKgQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1667573881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FdVcoBx7/tuIOiTEjcb989uDKzmjA/NfDt6Q7fyRUUU=; b=dAIjc2Y3hCQFk7g+gXz6kF/aSMr0MpEjlw+MrN+F/R5HuISKPvp2SarBBoTcpf2vPR5FyX s/m7EwNWsIDwwIDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v4 16/16] timer: Always queue timers on the local CPU Date: Fri, 4 Nov 2022 15:57:37 +0100 Message-Id: <20221104145737.71236-17-anna-maria@linutronix.de> In-Reply-To: <20221104145737.71236-1-anna-maria@linutronix.de> References: <20221104145737.71236-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748578316521563639?= X-GMAIL-MSGID: =?utf-8?q?1748578316521563639?= The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the seperate storage and eventually pulled at expiry time to a remote CPU. When a timer is added via add_timer_on(), TIMER_PINNED flag is required to ensure it expires on the specified CPU. Otherwise it will be enqueued in the global timer base which could be expired by a remote CPU. WARN_ONCE() is added to prevent misuse. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 5 ++--- kernel/time/timer.c | 17 +++++------------ 2 files changed, 7 insertions(+), 15 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 648f00105f58..181a3d3fa45a 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -50,9 +50,8 @@ struct timer_list { * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which + * the timer was enqueued. * * Note: Because enqueuing of timers can migrate the timer from one * CPU to another, pinned timers are not guaranteed to stay on the diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 214397d84747..bdff4be066e4 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -947,17 +947,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void forward_timer_base(struct timer_base *base) { unsigned long jnow = READ_ONCE(jiffies); @@ -1093,7 +1082,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -1228,6 +1217,10 @@ void add_timer_on(struct timer_list *timer, int cpu) BUG_ON(timer_pending(timer) || !timer->function); + WARN_ONCE(!(timer->flags & TIMER_PINNED), "TIMER_PINNED flag for " + "add_timer_on() is missing: timer=%p function=%ps", + timer, timer->function); + new_base = get_timer_cpu_base(timer->flags, cpu); /*