From patchwork Wed May 24 07:06:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98341 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2653416vqo; Wed, 24 May 2023 00:16:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7UIdh8IjUeIXN+LWRMWnvwOQCcX3Z70PPg2myJVw+Im5OzU3a9Rt6j/s/q7mYuSERjknKF X-Received: by 2002:a05:6a20:4315:b0:ef:7aa2:3cfd with SMTP id h21-20020a056a20431500b000ef7aa23cfdmr15815376pzk.50.1684912568752; Wed, 24 May 2023 00:16:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912568; cv=none; d=google.com; s=arc-20160816; b=IPY71zRj06qzz3R86kv7l5wE6WiFR8JhCNevWqlqknRi4btLS/y0p2z2qNPalSKQDn 1Oy+7/uWsOXY4YTydwStlZRj/B7MCitLgPQfguhlCRyGBBsjbQDZvGr/PxGlLC2lPwsv cLMa5gKU57TzXj1WgBNILRpa9fz3J31JTzVFnDSeC6uDCSrJBVAC2gEIEolebo/H2iCj pL2bVJUObiGv1LmWqdfWPBODee3Ly9S7owLYOqKhfd5/3bGPGpE9VFa6fXMwjvpRwr5f VtEPx5pEItK7pUaqHBM9sDF/VhkvyUKAor3a4PJ+qhnbaGu0wY41aUu29wBiTE57ZUqO UV5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=PQ3NXQMtig0Ydn+DZ27wQ54KKOjUtVfs2xndfW98O6U=; b=DirZ4MO7MtsQyHB312OWHdPHQ4fRo+ezZJmImfLbIqtwNTB7g826aE4koPt+BLi8PW jgrUkAuVgrHBQmMtjOzJjJSrcq6JbKzs7XazyuuRW5NjTwers/F8ceun/rz+ERvLsKq1 wu9nUGO8yfsyyIhlCJgFhb8COwBM/imoNJCYMQPDqdMjVURGqlR9dK86ZWGOywU0udej bA6TPvr0HqGNtGVqqSrr5eDAr19Og9cecynekfPachRQG8uYXNuDxMBG4wPZBvAHRUQ/ Y/FsKLmprtZ+iPRz557EtIvwGH2ZjvegKAx2yQG93w06yKLG1nlvk5Q0f+QzbvODv40k owKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=G5siFWrs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k70-20020a638449000000b0051b423d966csi1234300pgd.280.2023.05.24.00.15.54; Wed, 24 May 2023 00:16:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=G5siFWrs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239806AbjEXHHF (ORCPT + 99 others); Wed, 24 May 2023 03:07:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233779AbjEXHG6 (ORCPT ); Wed, 24 May 2023 03:06:58 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B567597 for ; Wed, 24 May 2023 00:06:57 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQ3NXQMtig0Ydn+DZ27wQ54KKOjUtVfs2xndfW98O6U=; b=G5siFWrsT9q+G+iRVJWKholCzvsqc7TcdsDMp/QlYXgteNVLws2xuw8XtztHTAlarVBIof 9R4n7fd2KpJa7lAKm52rkRFmKgX5VMLZYHXiOOiAFxygm9H3a5kAaPIXBZAx8YcOAqRYni hWhSPSjbodoOfMTqjOPTyPEXJeeF9dGq+xS3LVlVCNbfl6OkzyrZQDn7QkXMuXLMjBxCz6 xN6lqBFTb9014+NhdixwYua53dJ3whHrSLRF9Ai9xG6AwDQ04x+bgaREo6uy2saDj100mZ rHhO2W2n1xeBmWYosuF+nALYy7NuCBGDMbvyeQ5CUrucHaWscjr29ucikDrrsA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQ3NXQMtig0Ydn+DZ27wQ54KKOjUtVfs2xndfW98O6U=; b=/rAPckayV2VcrKm9f5e7XUlQZYPYkrm3FYI+hfFAH8ns7bpLqp+w7ah8pQixFXqjffY4Rv p3PJ+rr35QetIOAQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 01/21] tick-sched: Warn when next tick seems to be in the past Date: Wed, 24 May 2023 09:06:09 +0200 Message-Id: <20230524070629.6377-2-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758881732966550?= X-GMAIL-MSGID: =?utf-8?q?1766758881732966550?= When the next tick is in the past, the delta between basemono and the next tick gets negativ. But the next tick should never be in the past. The negative effect of a wrong next tick might be a stop of the tick and timers might expire late. To prevent expensive debugging when changing underlying code, add a WARN_ON_ONCE into this code path. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 52254679ec48..62836490ba4d 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -831,6 +831,8 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) * If the tick is due in the next period, keep it ticking or * force prod the timer. */ + WARN_ON_ONCE(basemono > next_tick); + delta = next_tick - basemono; if (delta <= (u64)TICK_NSEC) { /* From patchwork Wed May 24 07:06:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98344 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2653659vqo; Wed, 24 May 2023 00:16:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7Ik6AQJhYkTIrIIFAzkNc+skNSLMsD1TVXU6WdRMY6TGDslUD9o8sabt4kWOVCYJPTw6OE X-Received: by 2002:a05:6a21:6811:b0:e9:5b0a:deff with SMTP id wr17-20020a056a21681100b000e95b0adeffmr15169790pzb.22.1684912595873; Wed, 24 May 2023 00:16:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912595; cv=none; d=google.com; s=arc-20160816; b=pinWLgrh1ZYIxb/AcK/+k5NISjBviSwjEeqG18Uhw9RXTZPtky2erkdFTg0NoFGd7d fqEAfGHrTH0jMnmnM5mOz9JUorc+TmQ3fwGbk7+xxoE3DPIUcwEIlL/1fE9mojGlrxr/ 0JoYlD5Vx88zxsMmKpsP7NKU+/ejgUEpdm2EfMHLspXDjtXvrKtvdrr15cw7F/Rjt9x7 9+bnU+waP+1ZKlKVeEIgS9ZYDvHJlTlIAUd8ZMbz3KNnBCKNhVv1JQ3NptR5lX2VEXg2 WElpm33zbsM3OzsKeQBr+BC/nqPtrmwjjMXTMAAiBLN5AYB/NcWtr/D1rEBdWIPaOxdJ jBIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=GmrO3nNzYfXwzb1RvZTjX+af+BCpvVn7WiRHJl4Z/10=; b=w0ROPh0/iEJPSwUfaJKiyvgmDHjBudTrFPMNNRNz+2VOjmcei0+TEpDyt12JQBGDGt vJuXnQ+ivTtSUeugiHc6OVDruZveCwUaWpuyuClbD+JcVkT6ynpxYFkGqSjbJLSiSA6e QrynlzoH3ZtETokHjTIsA6lP2TDwUZJzIEbPqP8j7WzNI72vODp0fVEpviI+5jQob7bk dL6XhpLU2V5qycEtOKi8FkDljVna5DUJNWTQUf/x50NO4tNnJZpfxe7ZOG4747hMqvus cKGw6AgbZlqZrBllsszfrfHMPK4m2poW1nW/kVTpLxbGhd6rnKCnUehfW3MrGJtsEfa4 OKpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=oQFmzl0h; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h4-20020aa79f44000000b006281ec442absi61846pfr.309.2023.05.24.00.16.17; Wed, 24 May 2023 00:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=oQFmzl0h; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239847AbjEXHHI (ORCPT + 99 others); Wed, 24 May 2023 03:07:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239746AbjEXHG6 (ORCPT ); Wed, 24 May 2023 03:06:58 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B660199 for ; Wed, 24 May 2023 00:06:57 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GmrO3nNzYfXwzb1RvZTjX+af+BCpvVn7WiRHJl4Z/10=; b=oQFmzl0hCgOVqO0CCwNu0rsAtbzxlfYbvPv/MREvfhtRZB1g8L68j32VDfZF3deIIpftGL Oj81rEJyxdz8wXJInIWretBl0CLUWAM9+DbMLieeu5mG9u2s4tzmBwkAExbqO89g67hOtc 0vjdETFq5rwod2wu988mY/G0AYBFksm+ChoWX6iLHGAji7CAqKhkwO02viZHdOMcAjrbFH yFfy1qb6pwTwe9UcfXpyjneBoW/R+DQ5AG0FDfWUGXTWqWVgv9NKmfJh0cAgdmeQGLUR6Z 8YXP6N0BjHDtjX8uWNKD1hLF5hwsCO8ukNhNnwlfWjklUzNF+8b9vZ2Uuupg8w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GmrO3nNzYfXwzb1RvZTjX+af+BCpvVn7WiRHJl4Z/10=; b=OXnNcBY2JcZtsufQ7vxjReORO+MhFEQ2p/jHSbw/U1CzMcGn0NihckMjNahMgxFwWYKZQt gc8tTdaxcQGKCgDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 02/21] timer: Do not IPI for deferrable timers Date: Wed, 24 May 2023 09:06:10 +0200 Message-Id: <20230524070629.6377-3-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758909767029837?= X-GMAIL-MSGID: =?utf-8?q?1766758909767029837?= Deferrable timers do not prevent CPU from going idle and are not taken into account on idle path. Sending an IPI to a remote CPU when a new first deferrable timer was enqueued will wake up the remote CPU and but nothing will be done regarding the deferrable timers. Drop IPI completely when a new first deferrable timer was enqueued. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: new patch --- kernel/time/timer.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 63a8ce7177dd..6e251e3cf659 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -571,18 +571,15 @@ static int calc_wheel_index(unsigned long expires, unsigned long clk, static void trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) { - if (!is_timers_nohz_active()) - return; - /* - * TODO: This wants some optimizing similar to the code below, but we - * will do that when we switch from push to pull for deferrable timers. + * Deferrable timers do not prevent CPU from going idle and are not + * taken into account on idle path. An IPI when a new deferrable + * timer is enqueued will wake up the remote CPU but nothing will + * be done with the deferrable timer base. Therefore skip remote + * IPI for deferrable timers completely. */ - if (timer->flags & TIMER_DEFERRABLE) { - if (tick_nohz_full_cpu(base->cpu)) - wake_up_nohz_cpu(base->cpu); + if (!is_timers_nohz_active() || timer->flags & TIMER_DEFERRABLE) return; - } /* * We might have to IPI the remote CPU if the base is idle and the From patchwork Wed May 24 07:06:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98328 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650182vqo; Wed, 24 May 2023 00:09:11 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6CiW+8Csd2VEeKQidljGLKbmSx2SU5anQj70EH+aI8bbx+ZmisTHNpXdGDSYL5NmorNI3C X-Received: by 2002:a05:6a00:181f:b0:635:7fb2:2ab4 with SMTP id y31-20020a056a00181f00b006357fb22ab4mr1673953pfa.6.1684912150834; Wed, 24 May 2023 00:09:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912150; cv=none; d=google.com; s=arc-20160816; b=aDtJfxya0lDuJ47j9aEjFENcps+TGm/aZ4jOCVpIPvoQCGhg4VSPzPenM5qqSGYI9B hS/LiYlXfydqk8+OoiXabpJBU8Z/nU8reDSyrexnrWwsi7jqqcBxPNJegyPNvrcbLyQY kn53on4qHhnWcsC4gn7/uQK/5SaE7k0IGEPdpoJBWiBYRJ9GIQx8oPU24MArLDV9MCnX 9qmtr3amTiL4uuUT+BSta6f+xXCSKx/qklpDG09yM6KfNxYIE5O9Rj/ghWMazLWl3X2W IYw4qhWRPFPFACdg+Bm564gV/dardjKRiRk0vhIz7dhsnK3Xi13jTD1YL3WocRuKLQrU GZuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=ypjjo6K+ZjX/gE7dPsSFsIuC37ELchmIlQCCnVAGgp8=; b=BOcDTJFpQyMwM44vjAPwmADFvY2p/Xk8sGBsGMedq+z6xAOc478056Ks1pb4K1KqOd pDzfOdVErc3pzpD9ZvbpvEwlnr23qRPGWVtM0XyhSPxyNhEB6IwSd6bxonqfVsWiDO1l QxVwZpkA3HJydQsqFjI2A/bVucXeEQ/e7YdGSzEFybPL7zkk6SWp4X0dpCCH+K1FR52a 2BYNTws//tK5u7OP5yYv8lBlFEPjjY3+EArTp/gg/4KpqThhNwMmlNCe0DYg5a9TVsjW BV17oIS0HhdQ7yRupRUdT0RWCyCBrVShp3jvZm7nIuDbdXf60F8Yy8tN1GP2vhDArYfz udgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=TksK2xEV; dkim=neutral (no key) header.i=@linutronix.de header.b=sULkadB9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i7-20020a639d07000000b00534913980acsi1914530pgd.854.2023.05.24.00.08.57; Wed, 24 May 2023 00:09:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=TksK2xEV; dkim=neutral (no key) header.i=@linutronix.de header.b=sULkadB9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239853AbjEXHHL (ORCPT + 99 others); Wed, 24 May 2023 03:07:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239766AbjEXHG7 (ORCPT ); Wed, 24 May 2023 03:06:59 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48BFD93 for ; Wed, 24 May 2023 00:06:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ypjjo6K+ZjX/gE7dPsSFsIuC37ELchmIlQCCnVAGgp8=; b=TksK2xEVUsi1ZjbS5dg7wQVa7/RDG+ido0cCXWrd2wuzAE0GTXbAlFQ+fQSZdj4OgbOQqr YGaLWyBc2shTu4ofKXUcN5suBde5xSKwu3yfiX2TeUlgShn7UQ0FeuhzSqNxFoLWo3aVXi iMdEGWrTukCb4PiYbk7fXQk4lBSrHd7wXaPk2mbr0Q1NtbfE+4QYnPw539X0N1urWkaqEn MIpgF5iwQPE0ID0dnJz2dHSMfH6qcSK8Oe6CWfrDXw7/ZJ8aPcRw/yvcFxBuqHHqVO7LIM oZCUQ9Zwrac11S+D2SFEIKD5M17OtDHCksvaVjc/8oGCbCSEDdXS/6aUT4GcoQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ypjjo6K+ZjX/gE7dPsSFsIuC37ELchmIlQCCnVAGgp8=; b=sULkadB9nFvaj+x9zAbIGMko8DCx3pDokGMDNJLPFtj8hXCBWb7L7Ou7x6L/j5Jdjv7JCU +/Z2ukjhNtAJ+PAw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 03/21] timer: Add comment to get_next_timer_interrupt() description Date: Wed, 24 May 2023 09:06:11 +0200 Message-Id: <20230524070629.6377-4-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758443529171367?= X-GMAIL-MSGID: =?utf-8?q?1766758443529171367?= get_next_timer_interrupt() does more than simply getting the next timer interrupt. The timer bases are forwarded and also marked as idle whenever possible. To get not confused, add a comment to function description. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: s/handling/marking v5: New patch, which adds only a comment to get_next_timer_interrupt() instead of changing the function name --- kernel/time/timer.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 6e251e3cf659..0699fd14d00a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1912,6 +1912,10 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) * @basej: base time jiffies * @basem: base time clock monotonic * + * If required, base->clk is forwarded and base is also marked as + * idle. Idle marking of timer bases is allowed only to be done by CPU + * itself. + * * Returns the tick aligned clock monotonic time of the next pending * timer or KTIME_MAX if no timer is pending. */ From patchwork Wed May 24 07:06:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98329 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650183vqo; Wed, 24 May 2023 00:09:11 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5GQWnxYEtZafF5x8jLjUFiW0i+eiMbsMSFR+lGsUWNrpEnt33DkCt2INjLjxObVvkz8nt6 X-Received: by 2002:a05:6a00:391b:b0:64d:632a:d0ac with SMTP id fh27-20020a056a00391b00b0064d632ad0acmr1910066pfb.25.1684912151059; Wed, 24 May 2023 00:09:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912151; cv=none; d=google.com; s=arc-20160816; b=ETrKzTupz65/ipLsAKWCsVWOuh+8gXPSypcgRAw9PLDn0ZUF3mk2b8tiBnqlSgor9u s9bc5tO2iv2wi3gs0KuiciFYwhthcjOaVGT+D5lMFT6EwqkQE5uBvMfNXD6j6vQ2QHNj QwAM2SVhj9kvWzo4NewGcHatJlf2M55C6bcJghQ/VOBz3GR3O74vRKuqZYGW707USEHG boRURleuNmzYIuogc7u1wD2fcZpxAIjgQyVSCxWhjiDRteIT24jcQwQoxcHVgOXPPXBX tVl/f4EvWR0IfjVElPnIwzBTsozwmRTJ3FAvxfeAnOH3qkkN9ARW7Ca0zWvR7iW7bPeq LBrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=zI+0rerNf99sxfZIFO3XWHcHmkxmJchxL7sCWUUXfOI=; b=LqsWNy58Uah8QaumFYLFy/5sRS2Ji8W5P7/QrkQycchV1Ip1UlnliC+fvqaP0ASjNR LSieS1+XGwURsM4D2LrM6Q1SZjJ+1n2GUAM/EoIDq0rNOGJPdXfzfICLo3Gt7hepX+bP l4pL4R+XNl9X1YcnY/AaCGqD6/MJfL5zx+ppkxs6BgP24OxO4BaXQ3obX+p062oPdt+c RibLA0QoIWkDlSjWH/GefZ4+GcqH7Qd58z/TLKJFXHrbCMVtzq+ju+yUcMOw87tX4uDk QNSBBalmBuD7gy3eX9XGrF96Lflp4VhoWT2o06sB2ejtTzQPagsH5JVzlzqd0+UkahhK oJuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Wr2VyIci; dkim=neutral (no key) header.i=@linutronix.de header.b=aFOpeYDb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b21-20020aa79515000000b00643a2b0c6c8si6116887pfp.106.2023.05.24.00.08.58; Wed, 24 May 2023 00:09:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=Wr2VyIci; dkim=neutral (no key) header.i=@linutronix.de header.b=aFOpeYDb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239869AbjEXHHR (ORCPT + 99 others); Wed, 24 May 2023 03:07:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239779AbjEXHHC (ORCPT ); Wed, 24 May 2023 03:07:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09269A1 for ; Wed, 24 May 2023 00:06:58 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zI+0rerNf99sxfZIFO3XWHcHmkxmJchxL7sCWUUXfOI=; b=Wr2VyIci+1KGANfU0hvgLY2W+RSW/uSFDqAG77CiZcGF0EX+8Inx/EeYDJH+qjjSC8b6u+ tyqo9DnQDvUCdpqpmAKjoSCK+FKETKHw5EuoZDNVyXHSjkN3qTvVcuGlkhh5R4yIB51de2 wPcsDoNmoMU/4U1PA2fmlWPi+HN6cTR3QdWsWUezlXzif/XSTdXnx0rn0HbtvjGKXWyr/H T+e+bFO5iliSvLB65XRgUQeBow/Fb8D9yNEjA0UjpwWVm8Ju0aLrGJVzDiMd3hA/AV4fY0 ncyhUIgJ7Irw06WRJnciyIIflNFvAdX0chwhZsHYgkrd4VEVS5TbPgB6/8h3yA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zI+0rerNf99sxfZIFO3XWHcHmkxmJchxL7sCWUUXfOI=; b=aFOpeYDb/IZ7oNaaLSwyRIRSdcLlye2JIH4mnkGNUr1VsXgheAXY8TzfHIZuEhN8Ih0q6W p0sc0DN6o98G3+CA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 04/21] timer: Move store of next event into __next_timer_interrupt() Date: Wed, 24 May 2023 09:06:12 +0200 Message-Id: <20230524070629.6377-5-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758444063086228?= X-GMAIL-MSGID: =?utf-8?q?1766758444063086228?= Both call sites of __next_timer_interrupt() store return value directly in base->next_expiry. Move the store into __next_timer_interrupt() and to make its purpose more clear, rename function to next_expiry_recalc(). No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- v6: Fix typos in commit message and drop not required return as suggested by Peter Zijlstra) v4: rename function as suggested by Frederic Weisbecker --- kernel/time/timer.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 0699fd14d00a..59f2e15733ab 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1800,8 +1800,10 @@ static int next_pending_bucket(struct timer_base *base, unsigned offset, /* * Search the first expiring timer in the various clock levels. Caller must * hold base->lock. + * + * Store next expiry time in base->next_expiry. */ -static unsigned long __next_timer_interrupt(struct timer_base *base) +static void next_expiry_recalc(struct timer_base *base) { unsigned long clk, next, adj; unsigned lvl, offset = 0; @@ -1867,10 +1869,9 @@ static unsigned long __next_timer_interrupt(struct timer_base *base) clk += adj; } + base->next_expiry = next; base->next_expiry_recalc = false; base->timers_pending = !(next == base->clk + NEXT_TIMER_MAX_DELTA); - - return next; } #ifdef CONFIG_NO_HZ_COMMON @@ -1934,7 +1935,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base->lock); if (base->next_expiry_recalc) - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); nextevt = base->next_expiry; /* @@ -2017,7 +2018,7 @@ static inline void __run_timers(struct timer_base *base) WARN_ON_ONCE(!levels && !base->next_expiry_recalc && base->timers_pending); base->clk++; - base->next_expiry = __next_timer_interrupt(base); + next_expiry_recalc(base); while (levels--) expire_timers(base, heads + levels); From patchwork Wed May 24 07:06:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98345 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2653788vqo; Wed, 24 May 2023 00:16:53 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4JHLo8S5Qlh9n1DK5xXquwqyg3xqTbqpSrDfrLoQCXCE05xJgqiGNIV3WzClUWMbr93KbG X-Received: by 2002:a17:90b:1644:b0:255:e753:dcaf with SMTP id il4-20020a17090b164400b00255e753dcafmr5393pjb.16.1684912612917; Wed, 24 May 2023 00:16:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912612; cv=none; d=google.com; s=arc-20160816; b=aQDn974hNsD3moGsL0Uz0QUcwgGTgMdabtrFuveSE37MNGUjTUChHBw7QV7kS/3lmu R91yy8rErU+DUJ7zTmogrpbb4DGRXKdIM1YXhARSqW+/oUPsqJIMMCybttwy8QTzOjHW gO6aoxHsxuEX6qLe7UuYe1nCDXeyTtYoSkdnzePhklbmhAYKgIdy1OEVWl4pUCk8dg4y ZcFEbEA0qeDvbKsuLsAmNyg44TjWrYijRYTJeW6UdkpxTyqs846C7cHvuDpoGauCOgpk dxT4WJxvMiHCnbekSQLMD4rfqyQDkS9Iea2nb/0EMnngYYMVy1+8tqhMNgH1T4b5JSDK NSow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=HxmCND6qgUOL8dSwqTz+jvdhC0vJDN9Q/b7YApMcna0=; b=b1Yw8h/XsuAtPNU2nSHSfXEvl+gmksfDfGjd4qb0Y5ni8pSUwsINHkvh5U67PBcG1a lJPkRkhApnJ0si/rqVwE/65uRY+y8Vleib6qDJncvrg+gBFTtaQbHqc+amyl1pNKyO5r K7P71gGTuhdsjpbHPSUjyQ1vmkI2vxMioYiF78bRoKOOVPknergKZBi15XUEJZg9SSww 4hgXOC39+YLCdRCNOjNv2e2MGVvJeQDz0ZXbZw1g0jWiV5jcRNU1otDEYAo/rjjWSyAb IIh0glVTkSFKPphGerv758Ate0/O3R+pObrzNMTYKLGD1L/Kq8is1lD39QjaEsUny5m3 Jgsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UqSy0m3N; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=aXODCXcf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m13-20020a17090ade0d00b0024e3527e8f1si834067pjv.60.2023.05.24.00.16.38; Wed, 24 May 2023 00:16:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UqSy0m3N; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=aXODCXcf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239887AbjEXHHZ (ORCPT + 99 others); Wed, 24 May 2023 03:07:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239796AbjEXHHC (ORCPT ); Wed, 24 May 2023 03:07:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39E84B6 for ; Wed, 24 May 2023 00:07:00 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HxmCND6qgUOL8dSwqTz+jvdhC0vJDN9Q/b7YApMcna0=; b=UqSy0m3N8Wqw9BX3M7onpKiverK90EhM6obouvh6WE77o1fYsssDtYGcukRNa2uQOs6y3v lMXPYkGzRUdutJjZY2p0zykh7P3Lef0opyP21vN9qbaEJFxGh/nWNZoVl2iepcrzjmOkuN eocA83ihRgF8Sp5hgQRSQKMMTe5PTJhxAvs3Bp3zh0ah9Ms9multYarnqFd2Y2yqu8jd2z zUurqJnHxKHW8o8KVVc505hfygDLPmOvcLrKpzUaCmIzCEYcUORyUR1nYviCLcJUCEVYF1 F4SVLVmjFnNNPKj1crknpWxcSoUhOWOAH0RFfu1UE48J9jZgcU/INczn9zjH9Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HxmCND6qgUOL8dSwqTz+jvdhC0vJDN9Q/b7YApMcna0=; b=aXODCXcf92QefXn6aVZ7336BGvru5y8XgcoqyIDc1SfIGCvQxyXffdu+85D9MO1L2K4ODv I9FQh/rssYnDUfCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 05/21] timer: Split next timer interrupt logic Date: Wed, 24 May 2023 09:06:13 +0200 Message-Id: <20230524070629.6377-6-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758928412495968?= X-GMAIL-MSGID: =?utf-8?q?1766758928412495968?= Logic for getting next timer interrupt (no matter of recalculated or already stored in base->next_expiry) is split into a separate function "next_timer_interrupt()" to make it available for new call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Thomas Gleixner Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 59f2e15733ab..1a61977f9efa 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1908,6 +1908,14 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires) return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC; } +static unsigned long next_timer_interrupt(struct timer_base *base) +{ + if (base->next_expiry_recalc) + next_expiry_recalc(base); + + return base->next_expiry; +} + /** * get_next_timer_interrupt - return the time (clock mono) of the next timer * @basej: base time jiffies @@ -1934,9 +1942,8 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) return expires; raw_spin_lock(&base->lock); - if (base->next_expiry_recalc) - next_expiry_recalc(base); - nextevt = base->next_expiry; + + nextevt = next_timer_interrupt(base); /* * We have a fresh next event. Check whether we can forward the From patchwork Wed May 24 07:06:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98348 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2654725vqo; Wed, 24 May 2023 00:19:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4Fkqm8qVeF/YWXW6albtNI9iRDewcq4C5d9rMD4kySZtk2UMr3neQaBPhzdqSzGWymxi4k X-Received: by 2002:a17:902:6bc2:b0:1ae:6003:2de0 with SMTP id m2-20020a1709026bc200b001ae60032de0mr14368475plt.68.1684912755399; Wed, 24 May 2023 00:19:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912755; cv=none; d=google.com; s=arc-20160816; b=gjl9KX9nKIZwEmIfKwpSYlI6IBa3E2qvtW8QmE6gH5Lcbr1YIDHt9YxtXzwxCy9bel g//zwitq0qp4J5x//EnTYQVGm3sYfBoRbpaUDNmpoPaynUGVaN45uCcohNPonw702Pir E4YIxJzKNasP37Ow+ecjkG/xaym4q2kv4ToH9d2PvHANJPNZ2697NegDQWqR9Ojb9xhr XHO2J9Mu+8emYKpVY3ymsCo3lcIS329x8Y69Rv4rEkC8vF07nUOBqHQAtBZQ0IiHD2vY jx+PF7icbEjd9MyeT+lhgt7j5nlDjJn3zwtHlAc1TSkH2lYj8c7Vv5MgIWyM+1YONr2x s3yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=3nPoc4ex9ONP2BmQpTyzqK0BECnFowWD2Bg10S9sUhQ=; b=ilOjG5OMxqCvAqDj+GfncPT0EiVgUxGa1ODohIiq6Qgd1UG15RgKog1p1pZGHJd1BS uFB5QWNOUUFlV+TKwucUxq0ygiW6b1uXiSO6nrKN1FZJb8GCM2LZrj8CITIfnqZt1JKz Gt00SMXserAqf0ldnZRRmOPuxo+yk17FKIGitAWEa0nk4etm2NV6hm0HlIfqJ4mYfuIt IwSWYJRv5qtiMZscoGs3nqE0VHnURwTNZrmtgV56x69MXUNRRk1QgX7NAwSWWbbQFC8V Qxv2xDQwhZ+Hh2jmVvoDpl7FhnbHb9udoIMRDSSDdn2jCBfrqb588XsIWc6oUrcCMAH0 +oyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ygmvl73R; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=nyt4KIqC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id je11-20020a170903264b00b001ae5b59462asi7620855plb.309.2023.05.24.00.19.03; Wed, 24 May 2023 00:19:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ygmvl73R; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=nyt4KIqC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239878AbjEXHHX (ORCPT + 99 others); Wed, 24 May 2023 03:07:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239793AbjEXHHC (ORCPT ); Wed, 24 May 2023 03:07:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FC47E6 for ; Wed, 24 May 2023 00:07:00 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3nPoc4ex9ONP2BmQpTyzqK0BECnFowWD2Bg10S9sUhQ=; b=ygmvl73RIKtg2CvGUhk/vWDDvl4mJ296sHFwke/NlcFYGUeimMzJVFINJlSkfCGe9tiExJ djEqcpnCw8xR0lTyXMg3hlI9V4Mo3Idol2TFcDFpgwnSjVHcMoB/LrG5IQPL+jbbrzbRyT IAiOZuEWqLH26XKLgRDNcYc7rSPGK/QJobgGL/co7LsFVcqY3M46RTYIC85fp8My1OB+4y 3PN4dydZktD7gqvWs2ubcK5VK4tMRC16SLC07VXXjzLxfHSMBlsagGS2T5IEFQjTFBtaVp cJeqU6NRotBD0Z9Ry/UTaVYStzAwHHgyG+lqelGd3wFZooDCP0hZjnuedx6guQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3nPoc4ex9ONP2BmQpTyzqK0BECnFowWD2Bg10S9sUhQ=; b=nyt4KIqCFq0Er9OWDgdt9jN0tYGg+h+0Ojga3OXErmHJ3T8wH27jM7FPc9tt//rkAMhiMx Yd9vKAN+xKRWpVBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 06/21] timer: Rework idle logic Date: Wed, 24 May 2023 09:06:14 +0200 Message-Id: <20230524070629.6377-7-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766759077553284197?= X-GMAIL-MSGID: =?utf-8?q?1766759077553284197?= From: Thomas Gleixner To improve readability of the code, split base->idle calculation and expires calculation into separate parts. Thereby the following subtle change happens if the next event is just one jiffy ahead and the tick was already stopped: Originally base->is_idle remains true in this situation. Now base->is_idle turns to false. This may spare an IPI if a timer is enqueued remotely to an idle CPU that is going to tick on the next jiffy. Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v4: Change condition to force 0 delta and update commit message (Frederic) --- kernel/time/timer.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1a61977f9efa..1522fb81887e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1957,21 +1957,20 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) base->clk = nextevt; } - if (time_before_eq(nextevt, basej)) { - expires = basem; - base->is_idle = false; - } else { - if (base->timers_pending) - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; - /* - * If we expect to sleep more than a tick, mark the base idle. - * Also the tick is stopped so any added timer must forward - * the base clk itself to keep granularity small. This idle - * logic is only maintained for the BASE_STD base, deferrable - * timers may still see large granularity skew (by design). - */ - if ((expires - basem) > TICK_NSEC) - base->is_idle = true; + /* + * Base is idle if the next event is more than a tick away. Also + * the tick is stopped so any added timer must forward the base clk + * itself to keep granularity small. This idle logic is only + * maintained for the BASE_STD base, deferrable timers may still + * see large granularity skew (by design). + */ + base->is_idle = time_after(nextevt, basej + 1); + + if (base->timers_pending) { + /* If we missed a tick already, force 0 delta */ + if (time_before(nextevt, basej)) + nextevt = basej; + expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } raw_spin_unlock(&base->lock); From patchwork Wed May 24 07:06:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98330 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650215vqo; Wed, 24 May 2023 00:09:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ47OVVvYAk499e2uGP0MmMSaYMa9/FowZYD0Z5jHbBT3VyQjfWNJxTL/F2x/huVlHdli2qO X-Received: by 2002:a05:6a20:3d1a:b0:101:96b0:be40 with SMTP id y26-20020a056a203d1a00b0010196b0be40mr19907222pzi.26.1684912155495; Wed, 24 May 2023 00:09:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912155; cv=none; d=google.com; s=arc-20160816; b=OmfZfzqYIgLQloF47+WlNc1Hk6qZLzdgib3K/PmCU/YiEHNlDAlAxG78n6W+6HUwft OD54zF84QG54j0hvhivNNp8WQFWo+YcXDiWU2FfC6WSLCUtMzZXpNxNQrc1fXLbw0o1b vhw2KgqxBH6ZbokOD0wQgpOpHSsm/CDdm4c+hd/GLsinV6PkLP4HdHa2l8skY8EIuuHh Yli7N2mjGvoMPIxqjcLBEBBYWuehlYqYAdOiao1fukThs+L5lVRYC3GKpWtrckUEB/42 HLCYM37Yh8uQljQBoiYJW+uhYE5PlvZ5pTVQpQKJy50UH9L6LkgHPp0nuEnudgx074j9 VETg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=V5ghRq1p1UzQMeOe2kb9B0X1weJSuecItqx6Ppb2A0s=; b=qOeq/tyvfgXq6SuFdDC7stlFb/VYLhGkEjUzHo+e40DhDaorDHwmYPlb0MZU4ykxKC pW2YnSnPyC5cwI2atVm3sl43OcDQaEFRva0VoNFplEwKPjV43NDkd/THJzh4rn2QptuU CCsHCOG4/EyZEMEfcW4SUr8Vg+NtK3p2Qq/JrZ6+7kUtOzHFRwfFJog+M/T53Y/PlMms qm9YHXXkN/ASnK+Ji68bSUscdXrRawIXsD25qeagaSj1V6Ua3+wsHXZqBFcapAORpuSS Qw8Jxt8bi8Hlx7vXWHwys+afV2ehKdkiNzkqMmZbTeUFLyWrwfgu2E6EsnjgLyK2pwCW MnVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2WjbYAAV; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v8-20020a63b648000000b00534769ac7acsi7711413pgt.879.2023.05.24.00.09.00; Wed, 24 May 2023 00:09:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=2WjbYAAV; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239891AbjEXHH3 (ORCPT + 99 others); Wed, 24 May 2023 03:07:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239804AbjEXHHE (ORCPT ); Wed, 24 May 2023 03:07:04 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A503DFC for ; Wed, 24 May 2023 00:07:00 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V5ghRq1p1UzQMeOe2kb9B0X1weJSuecItqx6Ppb2A0s=; b=2WjbYAAV6lbiLJ7V0U3sea1cyaYFl/TwuQaIWaGpI+OU2Z7fWTKSIC80eyHe7aLx1fL9kD jPpHpVJMr6UJIt15OB9iEZJIFXv1ZTS+jnkaoFk8YgGjh4B8CKX7N+BOvvMt7N6LtZcOyM kZ11fSAH3zJY1NuCS7d3Ms4jjw17WweDB8lWYUQMcJIQA+wLrQ36SehwsuiY96QAb00ROR c2RGtT7WJG4T7ODze3SjqrVV8ez5psTj/LHMHDChPJrWJnl5SliMnrQI8qcnLBQuxLlXos wzxFw2x2z02YRz1x9W+6OxvSARzVl1LzXL332lDXqMBuwW2fvZ/XVN1CtN9LZw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V5ghRq1p1UzQMeOe2kb9B0X1weJSuecItqx6Ppb2A0s=; b=JVr0tIqLMcWr2kJTnV1Dpt+Euh05MXC73CvKJbxWlZr9gT7yeSneT2L6w6+7X1drsNAr7U b1hvvIv2dm5FiUDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 07/21] timers: Introduce add_timer() variants which modify timer flags Date: Wed, 24 May 2023 09:06:15 +0200 Message-Id: <20230524070629.6377-8-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758448093627781?= X-GMAIL-MSGID: =?utf-8?q?1766758448093627781?= Timer might be used as pinned timer (using add_timer_on()) and later on as non pinned timers using add_timer(). When the NOHZ timer pull at expiry model is in place, TIMER_PINNED flag is required to be used whenever a timer needs to expire on a dedicated CPU. Flag must no be set, if expiration on a dedicated CPU is not required. add_timer_on()'s behavior will be changed during the preparation patches for the NOHZ timer pull at expiry model to unconditionally set TIMER_PINNED flag. To be able to reset/set the flag when queueing a timer, two variants of add_timer() are introduced. This is a preparatory patch and has no functional change. Signed-off-by: Anna-Maria Behnsen --- New in v6 --- include/linux/timer.h | 2 ++ kernel/time/timer.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/timer.h b/include/linux/timer.h index 9162f275819a..6f96661480dd 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -180,6 +180,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires); #define NEXT_TIMER_MAX_DELTA ((1UL << 30) - 1) extern void add_timer(struct timer_list *timer); +extern void add_timer_local(struct timer_list *timer); +extern void add_timer_global(struct timer_list *timer); extern int try_to_del_timer_sync(struct timer_list *timer); extern int timer_delete_sync(struct timer_list *timer); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1522fb81887e..ab9a8bb11a8a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1242,6 +1242,40 @@ void add_timer(struct timer_list *timer) } EXPORT_SYMBOL(add_timer); +/** + * add_timer_local - Start a timer on the local CPU + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is set. + * + * See add_timer() for further details. + */ +void add_timer_local(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags |= TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_local); + +/** + * add_timer_global - Start a timer without TIMER_PINNED flag set + * @timer: The timer to be started + * + * Same as add_timer() except that the timer flag TIMER_PINNED is unset. + * + * See add_timer() for further details. + */ +void add_timer_global(struct timer_list *timer) +{ + if (WARN_ON_ONCE(timer_pending(timer))) + return; + timer->flags &= ~TIMER_PINNED; + __mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING); +} +EXPORT_SYMBOL(add_timer_global); + /** * add_timer_on - Start a timer on a particular CPU * @timer: The timer to be started From patchwork Wed May 24 07:06:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98343 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2653614vqo; Wed, 24 May 2023 00:16:30 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5Ob1vAcDxRkedc3RrlpZ+THln1uoQ6c/FDitCXa03xewQL1lzulRG3KrlUqfBExQIiuTIi X-Received: by 2002:a17:903:1212:b0:1ae:2013:4bc8 with SMTP id l18-20020a170903121200b001ae20134bc8mr22993344plh.18.1684912590172; Wed, 24 May 2023 00:16:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912590; cv=none; d=google.com; s=arc-20160816; b=Z9AvwOoWWc9mhKEXSx/AID6KzihbOyG2zrDeysBbYD6k6nNryb9SDlU66eqiBBcrRl Q1jsbG2lgTPK/eswZOATjlO8UvIyC8GA+3o9tjOUzwO3FfTlICPpezFjdAwoKdEYVUcc fgr5vx+SdhyliTo7OpQdS8WTsrQoQwrEJKB+eqVyP2LC5U/v+gmIMOB2HUUcx1rzZadI qXKlUbZSKXCX+Br/ngs+W36O8289em1yqVNaQXjMNLA3dE4hnqXwa/7QOqnVroWKnYQB zdxIiGat6m5jJ5gZsW1QwM2TrWT2hUKwblR/cWrVnCPz6APPysNtGajvvF3bXRBiupER FukA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=34NWQ1poL6YfGVQCYFvWaIp9I4PRyjufILSdpMSdq8I=; b=DdObUAA0G5k8g0tNh1LReesY9RIB+JIyVJW4LoWI6OOC+saEtZNxYU/4QjZSuXXI66 qCIne1sln56zzrFDMNfab5wmYUq8PpW9oQKg7jcTFhbKaddgGr+VsJ6SyrcnSJ5RoubE 0jcpCEYPi/tvYHPpqgLLmBvB735KQHekaft55XAqjwRg0yE6pKI7q55Hpy28syDMPxV7 Fjrlj5+ax8inK69ThhwjqBNAsmIxmqvAHZOh1O1PHk8Lal85tiJsfyGtYEGFM3tatINs wfapb5ZEo/6JKucuLCbzeQBFEJVQdLjcNs4O1UZlECM9GZdaOusW/BpUCbjl692hsMXY TiUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="b0CxIr/e"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ja13-20020a170902efcd00b001ac6189cac3si517255plb.187.2023.05.24.00.16.12; Wed, 24 May 2023 00:16:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="b0CxIr/e"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239480AbjEXHHc (ORCPT + 99 others); Wed, 24 May 2023 03:07:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239805AbjEXHHE (ORCPT ); Wed, 24 May 2023 03:07:04 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17DD1135 for ; Wed, 24 May 2023 00:07:01 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=34NWQ1poL6YfGVQCYFvWaIp9I4PRyjufILSdpMSdq8I=; b=b0CxIr/eDi7egiRgaO88B9iIMrYhC2oCP8QM9QFMTljCWr3b/SAu9H/ktLSpYe+gctT2SG Tn4sewlgqeGYOazdRxIvHWh6+8X6eItg0Kf/cI4mvtqAvXkiHZhWvWQEMnBJxzR5vwoil0 +iMklh0Hp5g/cRdda6WkU/9S2HLz29/27jmg4ib+HwyMkVWsFl39rXteCcdwZ5s4aUKSe/ 3DL8raHqRJKNnXH8k8XHaAyzZjU/pDt0UruNUDyDp0aOwV6ihV9SOSIpXX0IPubRu3EWZ7 LVmUYa6nyVWgH3v5FXpr1ql+/wttAl5m2rRcHw35zcT3NB1pgnGp50gGLg8W3g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=34NWQ1poL6YfGVQCYFvWaIp9I4PRyjufILSdpMSdq8I=; b=VTB7vrBFMd3rbSh0hYDYptblqNxtbsEGLe4RDkTRIKRbkDlN19M19dXy3qcy+OTUInonDE G0iIl+GZOkU7UoDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Tejun Heo , Lai Jiangshan Subject: [PATCH v7 08/21] workqueue: Use global variant for add_timer() Date: Wed, 24 May 2023 09:06:16 +0200 Message-Id: <20230524070629.6377-9-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758903952634609?= X-GMAIL-MSGID: =?utf-8?q?1766758903952634609?= The implementation of the NOHZ pull at expiry model will change the timer bases per CPU. Timers, that have to expire on a specific CPU, require the TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be dropped. This is required for call sites which use the timer alternately as pinned and not pinned timer like workqueues do. Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped. Signed-off-by: Anna-Maria Behnsen Acked-by: Tejun Heo Cc: Tejun Heo Cc: Lai Jiangshan Reviewed-by: Frederic Weisbecker --- v6: - New patch: As v6 provides unconditially setting TIMER_PINNED flag in add_timer_on() workqueue requires new add_timer_global() variant. --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 4666a1a92a31..5c529f19e709 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1682,7 +1682,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, if (unlikely(cpu != WORK_CPU_UNBOUND)) add_timer_on(timer, cpu); else - add_timer(timer); + add_timer_global(timer); } /** From patchwork Wed May 24 07:06:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98346 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2654048vqo; Wed, 24 May 2023 00:17:30 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5RvL+eTW+7WJOtLOdSRWUVnkyGNMabzEIOtKKxqDAHHKv7Ewg90x9+wdr1LMj7N51hnRuZ X-Received: by 2002:a05:6a20:258c:b0:10c:89cc:bc5f with SMTP id k12-20020a056a20258c00b0010c89ccbc5fmr4730320pzd.20.1684912650235; Wed, 24 May 2023 00:17:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912650; cv=none; d=google.com; s=arc-20160816; b=rKyBpp+vR5snhxQlPV/prUIjVp4PoNIahLJLZb/QEpk5dgJgtJf0GqYxkcKt2HpWao VsEDhNLAEXqfspyNy8NtqMb3RNcBUyT2gTDh/E5xXEHx1qzkFhMnru4Rvn0i3KopQ0Dm g1BuojffLXr0YHOCbS2wYqDM/z0KQ6jiqY/+E6IWXMSWN2nnp2jMurc3GFc2K8AGds+S xcrM8RNwbeLvBPkbTtXAAHhhL+wXO3unH2gOq2EVVdA3rBnv9PV2GUWC8dgsbegAc+g9 55yh4T6sWPiUFyYQIlit+Sdo04SKBX+suyr7ZX539bov415ey/n6wjW83L0+yYC6pjeE MyPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=lT6/GGqloCM8AGirZHb7HkPsMCoi/5vNo/Me8IGb/1s=; b=IjJRNUe+vAjN/a2WFazWKCBcJYXemaEZSomtJOiXQIky4ZBAzLn/jJ2IgjPofvDFxD fZwdIK+rsWavsLD9GhxR0HVdbL0ehVX4rcq0kprgX23NunCIEpdvSyk4RicFnbxWBnf0 w7ZtyCNB8itAUkZaNr7mu6pX1O3yrVci66wHyXeogDlNFxv8RR6joQp7Cu74zYQLTS8A tf/dQrvmhP1aAl6aK6CYSbWQBq+qXZ88Ut2V8XzxI9gjbv/wyu5hUejFfvMB8sGlO2wS wH0wKVXqvBVqv+fCG8m9SX3UW5KZclu/HHAieH9Acdo49Ab579n2PQ3s2sjsB7bvQH0M zxzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=eDOksZ3M; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h4-20020aa79f44000000b006281ec442absi61846pfr.309.2023.05.24.00.17.17; Wed, 24 May 2023 00:17:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=eDOksZ3M; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239903AbjEXHHh (ORCPT + 99 others); Wed, 24 May 2023 03:07:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239807AbjEXHHF (ORCPT ); Wed, 24 May 2023 03:07:05 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A717597 for ; Wed, 24 May 2023 00:07:01 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lT6/GGqloCM8AGirZHb7HkPsMCoi/5vNo/Me8IGb/1s=; b=eDOksZ3MxO3qy4i6Bs04pCvfZX2q4ptoyUl8Gz9gFNZZ62vh9WtXJ8X/Gwz3WGaoJI1T0+ HxRoMNjSX9N28K3ON3+VFKnTlTVLQHXOyo0WCYhiqAwhFYDt07vDucYIUREQ6R5/L9eaZD hJrojJXN0AiogwBML0Z6TJMUZLfblqsmGlGSt7JxsIT8DTS4BB00hEVHniXIDTUWtZB2SQ 0PHEJlfhLLOd7D5QHqrRrK5Q/xs+piZGI6AK4rQkFzTcH0Al1IU35AQHSZsCZrApPVfPNp ZIX1EJrj/BRqoTiD1Ycru7o0IyAyGkrIjwdTfDSlEtP1rQjiaCAyT6Xl3StEpw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lT6/GGqloCM8AGirZHb7HkPsMCoi/5vNo/Me8IGb/1s=; b=rpUktC55F0DExnhjdnVJx9htvbxzw/S5B8SGXzwmH03itM5CtNS4u5pdK8Skp2dwzk6wOa c91OzWG/HZ3IZlDQ== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 09/21] timer: add_timer_on(): Make sure TIMER_PINNED flag is set Date: Wed, 24 May 2023 09:06:17 +0200 Message-Id: <20230524070629.6377-10-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758966932385191?= X-GMAIL-MSGID: =?utf-8?q?1766758966932385191?= When adding a timer to the timer wheel using add_timer_on(), it is an implicitly pinned timer. With the timer pull at expiry time model in place, TIMER_PINNED flag is required to make sure timers end up in proper base. Add TIMER_PINNED flag unconditionally when add_timer_on() is executed. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index ab9a8bb11a8a..b7599216d183 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1281,7 +1281,10 @@ EXPORT_SYMBOL(add_timer_global); * @timer: The timer to be started * @cpu: The CPU to start it on * - * Same as add_timer() except that it starts the timer on the given CPU. + * Same as add_timer() except that it starts the timer on the given CPU and + * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in + * the next round, add_timer_global() should be used instead as it unsets + * the TIMER_PINNED flag. * * See add_timer() for further details. */ @@ -1295,6 +1298,9 @@ void add_timer_on(struct timer_list *timer, int cpu) if (WARN_ON_ONCE(timer_pending(timer))) return; + /* Make sure timer flags have TIMER_PINNED flag set */ + timer->flags |= TIMER_PINNED; + new_base = get_timer_cpu_base(timer->flags, cpu); /* From patchwork Wed May 24 07:06:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98342 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2653460vqo; Wed, 24 May 2023 00:16:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5j6hevHoppxL/foWoS4fJ9TRkOFMTJonsV7+gwrOGRjqC/xZWBa9+/KDCgi22QCdjXuwml X-Received: by 2002:a17:90b:1e49:b0:253:4a0e:87da with SMTP id pi9-20020a17090b1e4900b002534a0e87damr15719744pjb.9.1684912572416; Wed, 24 May 2023 00:16:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912572; cv=none; d=google.com; s=arc-20160816; b=PKzSw6Z16qcegUCF2LB4eHYY8ty3j3QRpoE50SegkAxWGp9l7nz9yJAIuEzhrV80RZ aUAYQcSNp//186WAOMpemwluv+be78CtJBgy9MKyfsnHldzDMQM7iba4Qw41cPkq0SD/ X/FzRs/U/PzYG4Yt7xTrdmY2QRdMXezpOPnb4NK0IcMtvHLJNcG+acakgIHRolWtHuRd FpuSzjcTXB7FnIpKyz9sGJURth7zus5t9LBeslpRPz4SkbEU0pYE5W7/x2NU3miVaQq5 xwK/zEEvvFxT9Je0lnm85CcehRjoa41CMUPEQpOqScMrkFEUuDvh7vHrBbYlEfrkVYIU 8Oqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=LtOGw35GRW9p2fx5MhyE24mUeYiFRQ6uZYmjDalJe0M=; b=CJTREYnDan8Jv1MyMjc1Mr3uVAUDdoF1yatZQ8tkBOWDyJ4wkusp37aSHt0ogFEaIa 8Eg8+4ShIQkkYVuxf8sg1Tl4zHJ9dLXp0/B0y4sFANUjE9gmi422n8c9ldMEMwvRrdKu SD+Yujrqe3AJC14VB6/HQoPIwklkAVRndBWiNyzRsOMEU73vmUUUfLM7++AlFThKGT2G eyqDeMBrdphGSeVuvlaNSvJ31EZIQI2t84AU63udC4mw8ex/1GJ9HkVZ42lhsUaOqdQS uwzSttHGAAgrXi/aJf4hajLIzmRjYdl1JxS8b2twCRSAekYSqIDxa5uBvLVZ6QXAbc8B s9eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=J1ihm9Qi; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s22-20020a17090aa11600b00252b824d13esi851940pjp.30.2023.05.24.00.16.00; Wed, 24 May 2023 00:16:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=J1ihm9Qi; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239917AbjEXHHk (ORCPT + 99 others); Wed, 24 May 2023 03:07:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239820AbjEXHHF (ORCPT ); Wed, 24 May 2023 03:07:05 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BAE518C for ; Wed, 24 May 2023 00:07:02 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LtOGw35GRW9p2fx5MhyE24mUeYiFRQ6uZYmjDalJe0M=; b=J1ihm9QiV4eErwTOiXLtydZzQk3ZK591reJ8uJyZTe8PlOFAGOPpxGKB4PK4S9LUjSiXfx s0IHM/t8ILeKVq8J2Tybkwosbf9Waslq6LF0yICYluaiKWz7tzJgBFRbf/7W/Y3nTbtWtL 9iaRJWDGkF7HxumBGNLnzEeO4WIbq8OgJwF+2Nz7dCXTDT9B6t+YWwHb5NnmF8TkvXiOYu IlMVewgLwFHGSEz38RTO15XSCkKlOMz/7gWCG29FxL3bBFZZkVon6pSgeN6kcXdI1i58sv 0/MDRoCchmw1RCJmf2+fR2AhZBj1MrbscuE9ZqlXm0GN5jI5LQ2kMN9hpd4gOQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LtOGw35GRW9p2fx5MhyE24mUeYiFRQ6uZYmjDalJe0M=; b=soTl8kB1cMiSPqjYY9ZdsFWoSmHOD953MyBsWjikDlSOk0eXI+4qAvO23ttFYxqFpya9TC 3g5Vrw9Fzpu64oAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 10/21] timers: Ease code in run_local_timers() Date: Wed, 24 May 2023 09:06:18 +0200 Message-Id: <20230524070629.6377-11-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758885510414247?= X-GMAIL-MSGID: =?utf-8?q?1766758885510414247?= The logic for raising a softirq the way it is implemented right now, is readable for two timer bases. When increasing numbers of timer bases, code gets harder to read. With the introduction of the timer migration hierarchy, there will be three timer bases. Therefore ease the code. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v5: New patch to decrease patch size of follow up patches --- kernel/time/timer.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index b7599216d183..749b1570bdcd 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2093,16 +2093,14 @@ static void run_local_timers(void) struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); hrtimer_run_queues(); - /* Raise the softirq only if required. */ - if (time_before(jiffies, base->next_expiry)) { - if (!IS_ENABLED(CONFIG_NO_HZ_COMMON)) - return; - /* CPU is awake, so check the deferrable base. */ - base++; - if (time_before(jiffies, base->next_expiry)) + + for (int i = 0; i < NR_BASES; i++, base++) { + /* Raise the softirq only if required. */ + if (time_after_eq(jiffies, base->next_expiry)) { + raise_softirq(TIMER_SOFTIRQ); return; + } } - raise_softirq(TIMER_SOFTIRQ); } /* From patchwork Wed May 24 07:06:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98331 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650432vqo; Wed, 24 May 2023 00:09:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5F/hKNVtj8qPZYYz/K+caLBBXgfIWckcSHAvddzLF7zFp7V38jDXqPR97b8LihkCdrcnmW X-Received: by 2002:a17:902:ecd2:b0:1af:dd26:4a17 with SMTP id a18-20020a170902ecd200b001afdd264a17mr1955902plh.11.1684912179460; Wed, 24 May 2023 00:09:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912179; cv=none; d=google.com; s=arc-20160816; b=Fhiu8y0kr/kSjVI2tfDAGivbUnteZVruwFtRS9U6D2ZMElAqTbOHV7IdmwH81iaKFY 04nyYH2HmuPt/5wFGFH0nsgVDOASAZLL8RcjZFUssUvcpaCNyi4OjeVzhMOtNE5WtGqg cIpMrppDaGUnOUSqbD08YHSpI6aZMDUZ78T3P7TpziAADRm+xH46wI71EQpTRwgkHvoE R8rCQ9u+vA2u4QsW4kFGMe/qGUtPmOJGKij8oE5IiIBQ00wPOkoNPUEhAbCCyZ0cw6Hi BYPag9QbCjPO5VmsszJkwqg7fXIfOHlwan6t2KxJrkjNbwxRSmCeea8DSgMntWgUkgbn h99A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=69v6KAzJ3Kkk+I5jf4FnTJ5YhbczmyAfH/JLKYuJ49U=; b=z9gaiWekmGA92jMcpf6CQZqNgbaMQd2zWAV/iX19IMwZ4MwJmQR5piEPdOyB/+PVpe K150GuhtXyzUbgD85H0ylQDR1b0ADvHJcEbzksjPj4kmDFUmFvStaVfJYGI92IiZUdQn mPyWDvQSPJ4+jhAOW/Sjd+BfWFnpGCJ9T5C6W9mgRVZxV0OknQaTmfxkGJBfkUL2h/QT k9hqfTOElmV5iNLrf0u9cV/bfFWOOa4EGZ+k3OIbiWFOZN62icj3b/LGiFDNFHlb/LLz a6b4kJYn4pFMHvSrTL9V1RFF/9uJAgBpMwiT5jfLSxYU3ZuKwuGRdqQbhSh5AKM7Lxqa xBww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="WML/DefT"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jh19-20020a170903329300b001afb5af367csi318810plb.629.2023.05.24.00.09.27; Wed, 24 May 2023 00:09:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="WML/DefT"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239928AbjEXHHs (ORCPT + 99 others); Wed, 24 May 2023 03:07:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239838AbjEXHHH (ORCPT ); Wed, 24 May 2023 03:07:07 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B41819C for ; Wed, 24 May 2023 00:07:03 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=69v6KAzJ3Kkk+I5jf4FnTJ5YhbczmyAfH/JLKYuJ49U=; b=WML/DefTybhtSenrVmJjckwFy5HhArUrPp5NzIJCOlVIS+tsDYpGGz6XtAL/Tjej+iS+eX rNbbfqXomGylr1TyXGNF9C7oSeCfGOkYaCL6sd1BAOaFGieZNgZw+gx3W72RrGrsEqcGLc j41YELXuAC9lwkibcnANeJAr2yPbqNP9bpeigvcmY9yC12BVpQNQFx7PrtdJqmvtPCaA9q yZUPwZjmnH+97vj8H5kp0G1Ce7qmOae3ftU+CRjdlMHVs2P6sddNhWkRxPuKSsTu5Dd7cM UGNTTHsaFHXiDV0x82Y1FRDIPj9ryBiaIHna/DBFGClomSc/MPfLR+bRKzgpFA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=69v6KAzJ3Kkk+I5jf4FnTJ5YhbczmyAfH/JLKYuJ49U=; b=c1N1GlFettzOJ4SoyCIpwPS+BzrT4zoXcaFKud66e1Fyr8PpCx/7gD+aeuKM8PYcvWUASH RMvesJu/2iq35MAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 11/21] timers: Create helper function to forward timer base clk Date: Wed, 24 May 2023 09:06:19 +0200 Message-Id: <20230524070629.6377-12-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758473504102211?= X-GMAIL-MSGID: =?utf-8?q?1766758473504102211?= The logic for forwarding timer base clock is split into a separate function to make it accessible for other call sites. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: s/splitted/split v5: New patch to simplify next patch --- kernel/time/timer.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 749b1570bdcd..fcff03757641 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1956,6 +1956,21 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } +/* + * Forward base clock is done only when @basej is past base->clk, otherwise + * base-clk might be rewind. + */ +static void forward_base_clk(struct timer_base *base, unsigned long nextevt, + unsigned long basej) +{ + if (time_after(basej, base->clk)) { + if (time_after(nextevt, basej)) + base->clk = basej; + else if (time_after(nextevt, base->clk)) + base->clk = nextevt; + } +} + /** * get_next_timer_interrupt - return the time (clock mono) of the next timer * @basej: base time jiffies @@ -1987,15 +2002,9 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* * We have a fresh next event. Check whether we can forward the - * base. We can only do that when @basej is past base->clk - * otherwise we might rewind base->clk. + * base. */ - if (time_after(basej, base->clk)) { - if (time_after(nextevt, basej)) - base->clk = basej; - else if (time_after(nextevt, base->clk)) - base->clk = nextevt; - } + forward_base_clk(base, nextevt, basej); /* * Base is idle if the next event is more than a tick away. Also From patchwork Wed May 24 07:06:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98332 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650459vqo; Wed, 24 May 2023 00:09:42 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7wrSdzq9nAwrLxr+AuAzLtgL9tgmWe+HoBMRudbdRP605IwYwmc1Ppsgvr6SPF/0AdJMiF X-Received: by 2002:a17:90a:3e03:b0:253:6e6f:f5c5 with SMTP id j3-20020a17090a3e0300b002536e6ff5c5mr20255532pjc.7.1684912182607; Wed, 24 May 2023 00:09:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912182; cv=none; d=google.com; s=arc-20160816; b=rYLg1BTkuil5DKMszqMlDsBoUp6ljL7aj6s5ZaHSjLhMsSYK4ARyv2nU98XzVII3P9 yXAPzMMf2IdQvKxxwOJ3Z+htGZMs72alr2IhD027I3hJznpzh1QAZvGBx2vN+v0Hky64 V9OJh2/vMraTk0vCxCVUMz5gSpEEIW8/taf5r1NFI0B2IYvqhTvtmJ+8k0o+ZszOSYfB j80rMjC2lo5c8/os+eB5XkdbFvy9bbHiEun0GwZnTwP/HtlciuJPnrojpzsC20RylIfw ZW213Vc6Ajhp3MtnHtVXWLiDJZU9CR+eZtqo8wjlcukovclDcX5gBDCw5htFYu9+WNN1 ctZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=2kwO+AnJDXfsw1irBqTyNknQ35rVxg5VAbQ15cO4kOI=; b=u2sVtI3jE7TrKgpJ4roX7ynbGnwkt9GcOO9Rrdqx9pU9mMfcPOoN4Y/jp9Fk03TNd7 j5GR0tCDw+Y3KWCJf+6OqdyZnCGi3a53dd3DdpgmHcwQW24oFOvJL+poHP60qC4tyTVH 7iLkOuhW5LjNuahgiqKHijIesTdnW3OvlqaFNgeL2yliHh8fOWVaxq2H3UT+atBFvuWH v9iwfzvL4YiphSSkgwdHwjOFGVUUOdFMIkX8GpqPnxWjuAhRh5C4wCehUVDyWtERI6Ug QcC62McAlcd1KOxxjWutSQvDZDIQobhb5d7hw6X7afbbjzOFZbLcH5c5ykFEjxtRodvz q19Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AVaNYd3l; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s22-20020a17090aa11600b00252b824d13esi851940pjp.30.2023.05.24.00.09.29; Wed, 24 May 2023 00:09:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=AVaNYd3l; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239935AbjEXHHw (ORCPT + 99 others); Wed, 24 May 2023 03:07:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239789AbjEXHHH (ORCPT ); Wed, 24 May 2023 03:07:07 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3BEB1A1 for ; Wed, 24 May 2023 00:07:03 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2kwO+AnJDXfsw1irBqTyNknQ35rVxg5VAbQ15cO4kOI=; b=AVaNYd3l+yqlQll5hEmAqXTChU3DSe5VnRJy78E/eVFCXdWsnTCqc9yE7m7a8euAvEo1xv B143ELVeTjUa+fLv3RoDY8zbtJV7BqRLEbIPO4+Zc0aBK7catKus46p8/P7x5OWa9ktCXy Ph3B/M9uvGF2TV5wkvbNn7rX5uWBWzl2ZedGBDX1qDGlcm0GqftiG5SSLq9nZ4aQJ12c39 UIt4XLYHd6no2ZX52qj7nwAYf76yqNhk8dkBKecZFPlog31cP8llQlLBmvsjyIYkYqoM1a eT41JlIo23WyITjLOc8fM6iIHxee9TG8KLkKNnGglnEo7e10qalNzP1veTG8Vg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2kwO+AnJDXfsw1irBqTyNknQ35rVxg5VAbQ15cO4kOI=; b=Nj9tYes1l3ulYLYWuwfvbbrojEdkhLDKRYnW6c0JufPf/1w+xn/KG4radWI9aGfYqOuvFW L775AW06NLyWNnDA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v7 12/21] timer: Keep the pinned timers separate from the others Date: Wed, 24 May 2023 09:06:20 +0200 Message-Id: <20230524070629.6377-13-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758476789174354?= X-GMAIL-MSGID: =?utf-8?q?1766758476789174354?= Separate the storage space for pinned timers. Deferrable timers (doesn't matter if pinned or non pinned) are still enqueued into their own base. This is preparatory work for changing the NOHZ timer placement from a push at enqueue time to a pull at expiry time model. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: - Drop set TIMER_PINNED flag in add_timer_on() and drop related warning. add_timer_on() fix is splitted into a separate patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker v5: - Add WARN_ONCE() in add_timer_on() - Decrease patch size by splitting into three patches (this patch and the two before) v4: - split out logic to forward base clock into a helper function forward_base_clk() (Frederic) - ease the code in run_local_timers() and timer_clear_idle() (Frederic) --- kernel/time/timer.c | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index fcff03757641..010c8877fa85 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64); #define WHEEL_SIZE (LVL_SIZE * LVL_DEPTH) #ifdef CONFIG_NO_HZ_COMMON -# define NR_BASES 2 -# define BASE_STD 0 -# define BASE_DEF 1 +/* + * If multiple bases need to be locked, use the base ordering for lock + * nesting, i.e. lowest number first. + */ +# define NR_BASES 3 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 1 +# define BASE_DEF 2 #else # define NR_BASES 1 -# define BASE_STD 0 +# define BASE_LOCAL 0 +# define BASE_GLOBAL 0 # define BASE_DEF 0 #endif @@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base, static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) { - struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = per_cpu_ptr(&timer_bases[index], cpu); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu) static inline struct timer_base *get_timer_this_cpu_base(u32 tflags) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL; + struct timer_base *base; + + base = this_cpu_ptr(&timer_bases[index]); /* * If the timer is deferrable and NO_HZ_COMMON is set then we need @@ -1985,9 +1997,10 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + unsigned long nextevt, nextevt_local, nextevt_global; + struct timer_base *base_local, *base_global; + bool local_first, is_idle; u64 expires = KTIME_MAX; - unsigned long nextevt; /* * Pretend that there is no timer pending if the cpu is offline. @@ -1996,32 +2009,57 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (cpu_is_offline(smp_processor_id())) return expires; - raw_spin_lock(&base->lock); + base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); + base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); - nextevt = next_timer_interrupt(base); + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); + + nextevt_local = next_timer_interrupt(base_local); + nextevt_global = next_timer_interrupt(base_global); /* * We have a fresh next event. Check whether we can forward the * base. */ - forward_base_clk(base, nextevt, basej); + forward_base_clk(base_local, nextevt_local, basej); + forward_base_clk(base_global, nextevt_global, basej); + + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + nextevt = local_first ? nextevt_local : nextevt_global; /* - * Base is idle if the next event is more than a tick away. Also + * Bases are idle if the next event is more than a tick away. Also * the tick is stopped so any added timer must forward the base clk * itself to keep granularity small. This idle logic is only - * maintained for the BASE_STD base, deferrable timers may still - * see large granularity skew (by design). + * maintained for the BASE_LOCAL and BASE_GLOBAL base, deferrable + * timers may still see large granularity skew (by design). */ - base->is_idle = time_after(nextevt, basej + 1); + is_idle = time_after(nextevt, basej + 1); + + /* We need to mark both bases in sync */ + base_local->is_idle = base_global->is_idle = is_idle; - if (base->timers_pending) { + if (base_local->timers_pending || base_global->timers_pending) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; expires = basem + (u64)(nextevt - basej) * TICK_NSEC; } - raw_spin_unlock(&base->lock); + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); return cmp_next_hrtimer_event(basem, expires); } @@ -2033,15 +2071,14 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ void timer_clear_idle(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); - /* * We do this unlocked. The worst outcome is a remote enqueue sending * a pointless IPI, but taking the lock would just make the window for * sending the IPI a few instructions smaller for the cost of taking * the lock in the exit from idle path. */ - base->is_idle = false; + __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); + __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); } #endif @@ -2087,11 +2124,13 @@ static inline void __run_timers(struct timer_base *base) */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); __run_timers(base); - if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) + if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { + __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + } } /* @@ -2099,7 +2138,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) */ static void run_local_timers(void) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]); + struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); hrtimer_run_queues(); From patchwork Wed May 24 07:06:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98333 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2650982vqo; Wed, 24 May 2023 00:10:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6aKqKobIO8fS0lHYrZ5Ykd0coM6eHQKYVr8dSBXUpdMraF0+ZTE68j79Qpi8+/Cf/Kdp9A X-Received: by 2002:a05:6a20:160e:b0:10b:6b1f:acf1 with SMTP id l14-20020a056a20160e00b0010b6b1facf1mr12150746pzj.29.1684912249796; Wed, 24 May 2023 00:10:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912249; cv=none; d=google.com; s=arc-20160816; b=tT7wq1OkBx65XW6d9E4Eqfx1tj9qIVpAxfvIxk34WmdBedUg5RApQu8BEC/nuFYSri aZo1vlNjeuOnn4ajgdgmnaPCd3GUkh8rPH19pZHrIcnoDTI2CqpoGUBwbrkjuCtTP1Tz E/Iq/qOCGitN9a1TeUT/K0lnVJKt1NX+5aKyykaBIn9htT351DDEYua9dJ+IQ3jqRlcg qqOgFLf3eXeOyeq6rJjtoe9EkEYeD9TuA3AtpkVGCDv5X43kuvT35FQTsVsT7Vx5zzf1 RhuxwxTFdW8aWZ5cpo078mvDg2mpbfg9ZWDFKKcxqwgQco/QNKnyWHp1gtNfpVUEjJl3 +2/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=r6aAJpUsFbdAiX81BLzrgrPjSrKyBdcLGR/988cXOog=; b=uMGXTJaCrqDfo3l3CWuERGpICPDPN+ut1Yom26pDEb7XGlm6cqRXzxr12vms2rM4il WB/pbOBuVh0q96dIF3YQRN1heZjR1xZZLzdJ/yGlJXRDmTPtzJZVx9WQov59dM0xzA7C 88VpaOBZS8tLnQY1w2DS7gYIW0D74IcuZZEEEz3c0F4WypPJTOcAIhw9UYOejOTukjSL bXBv8lG3PwtusOBraV4kLi8xhCekrS+u7LHxpGnGsywPbaVSXZ5HqQwwPA735FsEod/v M4PsUBl8daYN5q/WyIFzr0e25U2SKNaTRm5S0Zy1bb4YZpicttpyJHn7yM7fSyTIccH/ cXrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dotju3Hn; dkim=neutral (no key) header.i=@linutronix.de header.b=NY8bKGCP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z28-20020aa79e5c000000b0064354230c2asi429821pfq.367.2023.05.24.00.10.37; Wed, 24 May 2023 00:10:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=dotju3Hn; dkim=neutral (no key) header.i=@linutronix.de header.b=NY8bKGCP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239941AbjEXHHz (ORCPT + 99 others); Wed, 24 May 2023 03:07:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239860AbjEXHHO (ORCPT ); Wed, 24 May 2023 03:07:14 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6261B1A7 for ; Wed, 24 May 2023 00:07:04 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r6aAJpUsFbdAiX81BLzrgrPjSrKyBdcLGR/988cXOog=; b=dotju3Hn6qkQGsykvxfeiHzdDDTEMSn5UJX94gDlmxsBbeo2gk1Uao0P/JJhqVqG2+IQK+ 223ALp6/P+ZME0M4pYKnWqHYUNzikSZ/glwLMO8GRewxoeCA6LB9DGxAxe3MmdaYW5kAgU aAX1cI8VGgxUvBhBIQbeASIZKcvV7pfH1u/kBgavVB/sSwEdFBDyTdr5ZcMYsf1o5Ob/W/ 72OPmK1LeGzrVqsb0kKUJ7i37MpffOa7ACt8MVJ9eWnmvxZKpBYzeUsB3iqBytPbV3y4h/ c59ApQaapOxjyn7nGk8mvsEWT8krbSBGa9egMV5nzv+mx4e18PQXZ8Tpafbncg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r6aAJpUsFbdAiX81BLzrgrPjSrKyBdcLGR/988cXOog=; b=NY8bKGCPTTmbfMpu7VN7uc27lpyCP0SGl5UQuM/cv0smZEQTlQYK5kdoEHNDLLCUgNXMof 15sd8FJPNbnH4PDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Richard Cochran , Frederic Weisbecker Subject: [PATCH v7 13/21] timer: Retrieve next expiry of pinned/non-pinned timers separately Date: Wed, 24 May 2023 09:06:21 +0200 Message-Id: <20230524070629.6377-14-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758546956142121?= X-GMAIL-MSGID: =?utf-8?q?1766758546956142121?= For the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have separate expiry times for the pinned and the non-pinned (movable) timers. Therefore struct timer_events is introduced. No functional change Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- kernel/time/timer.c | 45 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 5 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 010c8877fa85..e4b50760da3f 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,6 +221,11 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -1997,17 +2002,17 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { + struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; bool local_first, is_idle; - u64 expires = KTIME_MAX; /* * Pretend that there is no timer pending if the cpu is offline. * Possible pending timers will be migrated later to an active cpu. */ if (cpu_is_offline(smp_processor_id())) - return expires; + return tevt.local; base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]); base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]); @@ -2052,16 +2057,46 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; - if (base_local->timers_pending || base_global->timers_pending) { + /* + * If the bases are not marked idle, i.e one of the events is at + * max. one tick away, then the CPU can't go into a NOHZ idle + * sleep. Use the earlier event of both and store it in the local + * expiry value. The next global event is irrelevant in this case + * and can be left as KTIME_MAX. CPU will wakeup on time. + */ + if (!is_idle) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; - expires = basem + (u64)(nextevt - basej) * TICK_NSEC; + tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; + goto unlock; } + + /* + * If the bases are marked idle, i.e. the next event on both the + * local and the global queue are farther away than a tick, + * evaluate both bases. No need to check whether one of the bases + * has an already expired timer as this is caught by the !is_idle + * condition above. + */ + if (base_local->timers_pending) + tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + /* + * If the local queue expires first, then the global event can be + * ignored. The CPU wakes up before that. If the global queue is + * empty, nothing to do either. + */ + if (!local_first && base_global->timers_pending) + tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + +unlock: raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); - return cmp_next_hrtimer_event(basem, expires); + tevt.local = min_t(u64, tevt.local, tevt.global); + + return cmp_next_hrtimer_event(basem, tevt.local); } /** From patchwork Wed May 24 07:06:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98335 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651082vqo; Wed, 24 May 2023 00:11:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5AKhsSesOVaIHsIHh6pORKvAtAoUkIlEv4ixD/PI9OipSC3bq0HSxEl3TVhe4a/g5fODu3 X-Received: by 2002:a17:90a:d98f:b0:253:62c2:4e1b with SMTP id d15-20020a17090ad98f00b0025362c24e1bmr14135092pjv.48.1684912263165; Wed, 24 May 2023 00:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912263; cv=none; d=google.com; s=arc-20160816; b=mWpZAuYm16Lou0+bbKmWZvpUi4JyxRypYkdIQ0JCUixIFke0yJOPeRb9vqP/0RO20R nxRcuQTESIURiWPf4h91GDDlkqEUCZKG9feEHVBLuBGYaaHwFrqv9txSDAHSaV70Rg/H wBcVlFjfR85S/bDnwcMLaw34AuqfXeWeqBQ6qFExEMG/NXGppT5NCT344NC1maa5kJRR Ee4tSufrNzcVSSy6sXpREqGQ3faC4ZoTLDhJFjBKAi+QU6x8oS2EsP+lvVI+NINOyX5f Qqq1ZkXt8EfMSIx1pHoSMpHdERGEEUgj5kiigrukcflNhTXhc/CDkXF7diWoJyTZMPD/ 2EPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=toZ3+ufortFTTBI6OBlc3a3bWizxyvQL3TeTeVCWUqU=; b=hvYAMllEu+dAA5MPXf1l8Jn84VGUuVj7FGbSVtg21TDArxedyhQ/TQMkY1ZxLjcdiv TN2j56rexjg60+gcOYADSphBCN5jnvIqFzjWWbMQq5napNm9AbffxoqnoGrbCF4A6DbB /2FJ5R/qEny8YI1gBXs741IAQq7PLp7TTsUwHFjqV9fsGk7CH4NoGrUWx3MPiEZayD9p rZUoyGExibY74yv6Nd0ZrY0bTtWt4WaJU+E84t6QxX6qdi7USg5cLY3IlJWTT/GvzTSb jwKqqn5HYBI0SCoIhc7CjzcHMj54C18vIUm5bpzq1HTFfPG6kxKLkzXr+IIGcH0WHCfV fvEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=U34zMuW1; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d23-20020a17090ab31700b0025352448b95si411036pjr.172.2023.05.24.00.10.50; Wed, 24 May 2023 00:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=U34zMuW1; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239592AbjEXHIL (ORCPT + 99 others); Wed, 24 May 2023 03:08:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239900AbjEXHHf (ORCPT ); Wed, 24 May 2023 03:07:35 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1212D9B for ; Wed, 24 May 2023 00:07:05 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=toZ3+ufortFTTBI6OBlc3a3bWizxyvQL3TeTeVCWUqU=; b=U34zMuW1T+0NnQXAvVtSY/xZY7B7AQ7PK8D6JhE/lZ1vlJJHSyGCug8oKFSSY6GfcUSSem OM4sDozDQ4U0INnjdI3PqmVxW7nh64lVuYkniLfwzSvxwbrJxXCtCa4ZFWTb2i00I6APFe dpt+deuVti3Iz2xiYd7oJ6Tr8tVnItyPTYwLdWzBvigEbXV52YFOFu0at7N7LNQE1zJ653 ouL0EHo+QPmUrip6shzvzXom8mYh+fTYI2YKXsG2zjZBZgGkurWZ2FxKGoLzzUGJG6zU/b pz5DNTg3yX5Y2iKbF2oBAP685kys/zSFqBsNpFRNwB+PeV8ctvXzgGw9Is2Dig== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=toZ3+ufortFTTBI6OBlc3a3bWizxyvQL3TeTeVCWUqU=; b=8TaUbW9PT2Pjl6QzwXtZA7s6O4RZP3SC15HsBKLr6IdomKYrFDQbeYwL8DNCNf+UBNnkEZ zZ7dqhWkv502wlAA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Frederic Weisbecker Subject: [PATCH v7 14/21] timer: Split out "get next timer interrupt" functionality Date: Wed, 24 May 2023 09:06:22 +0200 Message-Id: <20230524070629.6377-15-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758560837519342?= X-GMAIL-MSGID: =?utf-8?q?1766758560837519342?= The functionality for getting the next timer interrupt in get_next_timer_interrupt() is split into a separate function fetch_next_timer_interrupt() to be usable by other call sites. This is preparatory work for the conversion of the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v6: s/splitted/split v5: Update commit message v4: Fix typo in comment --- kernel/time/timer.c | 91 +++++++++++++++++++++++++-------------------- 1 file changed, 50 insertions(+), 41 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index e4b50760da3f..4bb6c168d106 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1973,6 +1973,46 @@ static unsigned long next_timer_interrupt(struct timer_base *base) return base->next_expiry; } +static unsigned long fetch_next_timer_interrupt(struct timer_base *base_local, + struct timer_base *base_global, + unsigned long basej, u64 basem, + struct timer_events *tevt) +{ + unsigned long nextevt_local, nextevt_global; + bool local_first; + + nextevt_local = next_timer_interrupt(base_local); + nextevt_global = next_timer_interrupt(base_global); + + /* + * Check whether the local event is expiring before or at the same + * time as the global event. + * + * Note, that nextevt_global and nextevt_local might be based on + * different base->clk values. So it's not guaranteed that + * comparing with empty bases results in a correct local_first. + */ + if (base_local->timers_pending && base_global->timers_pending) + local_first = time_before_eq(nextevt_local, nextevt_global); + else + local_first = base_local->timers_pending; + + /* + * Update tevt->* values: + * + * If the local queue expires first, then the global event can + * be ignored. If the global queue is empty, nothing to do + * either. + */ + if (!local_first && base_global->timers_pending) + tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; + + if (base_local->timers_pending) + tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; + + return local_first ? nextevt_local : nextevt_global; +} + /* * Forward base clock is done only when @basej is past base->clk, otherwise * base-clk might be rewind. @@ -2005,7 +2045,7 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX }; unsigned long nextevt, nextevt_local, nextevt_global; struct timer_base *base_local, *base_global; - bool local_first, is_idle; + bool is_idle; /* * Pretend that there is no timer pending if the cpu is offline. @@ -2020,8 +2060,11 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); - nextevt_local = next_timer_interrupt(base_local); - nextevt_global = next_timer_interrupt(base_global); + nextevt = fetch_next_timer_interrupt(base_local, base_global, + basej, basem, &tevt); + + nextevt_local = base_local->next_expiry; + nextevt_global = base_global->next_expiry; /* * We have a fresh next event. Check whether we can forward the @@ -2030,21 +2073,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) forward_base_clk(base_local, nextevt_local, basej); forward_base_clk(base_global, nextevt_global, basej); - /* - * Check whether the local event is expiring before or at the same - * time as the global event. - * - * Note, that nextevt_global and nextevt_local might be based on - * different base->clk values. So it's not guaranteed that - * comparing with empty bases results in a correct local_first. - */ - if (base_local->timers_pending && base_global->timers_pending) - local_first = time_before_eq(nextevt_local, nextevt_global); - else - local_first = base_local->timers_pending; - - nextevt = local_first ? nextevt_local : nextevt_global; - /* * Bases are idle if the next event is more than a tick away. Also * the tick is stopped so any added timer must forward the base clk @@ -2057,6 +2085,9 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); + /* * If the bases are not marked idle, i.e one of the events is at * max. one tick away, then the CPU can't go into a NOHZ idle @@ -2069,31 +2100,9 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) if (time_before(nextevt, basej)) nextevt = basej; tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; - goto unlock; + tevt.global = KTIME_MAX; } - /* - * If the bases are marked idle, i.e. the next event on both the - * local and the global queue are farther away than a tick, - * evaluate both bases. No need to check whether one of the bases - * has an already expired timer as this is caught by the !is_idle - * condition above. - */ - if (base_local->timers_pending) - tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC; - - /* - * If the local queue expires first, then the global event can be - * ignored. The CPU wakes up before that. If the global queue is - * empty, nothing to do either. - */ - if (!local_first && base_global->timers_pending) - tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC; - -unlock: - raw_spin_unlock(&base_global->lock); - raw_spin_unlock(&base_local->lock); - tevt.local = min_t(u64, tevt.local, tevt.global); return cmp_next_hrtimer_event(basem, tevt.local); From patchwork Wed May 24 07:06:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98334 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651083vqo; Wed, 24 May 2023 00:11:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7fWvaX33426GbNgg4CiWUhXOV7fxfNQyIBoDPfVNq2of5IG/iyhqvS5H8kLBeF+Zf3/lAK X-Received: by 2002:a05:6a20:158e:b0:10b:92b8:9845 with SMTP id h14-20020a056a20158e00b0010b92b89845mr10724929pzj.7.1684912263210; Wed, 24 May 2023 00:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912263; cv=none; d=google.com; s=arc-20160816; b=YoafrsqysjZqJA9oFVvElV4Auwg2TgIMpBTQzzf0E33ed8Sfyu2XO6vZmVw2J8PDxi MeSzaAcHhimXY/4FfG7ss/geYFWIlw65yDhnAVWRQRFFq0By7i02HZJN2nv73s8gb0Sl I8hrJrCyS3v3DIcb7MzETt03i7DLsSi6VmVXdQBitBfa0qb/8hlj68TDMHWoCrIDdz29 jtmy46wBwXdwI/KOH89EmvRvbO7zXDCeUvrvCW+1tG4/HmslwXlRoMOOgsAROSC97GT1 +D6lzSfTwHaLi+PojDI9S24OKQuT9Zxg7R7t/39YuWQcDH3OYM3bZBeZX/BXMztWjALJ ECXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=JrYsq/KOHDHxpa95h8UCcHB8BfDXtWT1sXcK5p35vZM=; b=gFzzjMNNfzFwKgaVaHlwWBn7tEbD3eme9BBRUtwmzksYYqBfvZT/VRv6S0sfN3cBuO pROjdHK3AndDguEviQ0HG3+gdfZQIsr/9MosXpcTWs3ot1pXS0upT9rRZrDyuXn4CO64 RhKpjxGcEfIYEt4fgZ4GctZF/hds3v4i1ADutiRq4xmzjfr5UMHstGGCsESBBwUfumFp dzuhdPEy5TyBfM7eaysKYblgT2DQriZUubZ0zXxYDnHJBcncO9k7b4VsmuCxJxZQc8z/ 0i1kBhMuS8Tm0hcim41+h8gCbCIiCTrAN4SZoGoxZMGbNKRzPtVGq8lqhg1nJ4rP8/yL sbDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=31ReD5wF; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c11-20020a056a00008b00b00643a730d50bsi289331pfj.389.2023.05.24.00.10.51; Wed, 24 May 2023 00:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=31ReD5wF; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239969AbjEXHIP (ORCPT + 99 others); Wed, 24 May 2023 03:08:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239906AbjEXHHi (ORCPT ); Wed, 24 May 2023 03:07:38 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5B709E for ; Wed, 24 May 2023 00:07:05 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JrYsq/KOHDHxpa95h8UCcHB8BfDXtWT1sXcK5p35vZM=; b=31ReD5wF1GyMOnZMnmOfgPQM7WEEQoGhX1O5LbxX4bUmVF/71U6sRpN4A23TiCNGXevAic yailAyK2ImdqrJ28dzkf5ZqfmjnKYaBfoDnd8TGzLO9suo20o7YNiygCWaIn2lLqjzfW83 8z/UTxYkBRm4oUNoPWjx1AP7XmslDcKduKL/XLwIPT7yg8QVETlt2JWGbOKptXLFyKsNwX /nbjg0D+2dScekQYcPMvC51aycYLBnlLSS5DgJ+dO8x4sbUFYEpCXKHrxcpc7ge68PXCGp 9wfwi6Xl6w+cz/Nt1KfMTGZAT43G7ljCAl8NgIckxJ6053j8KGbvpjGwYg9/Kw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JrYsq/KOHDHxpa95h8UCcHB8BfDXtWT1sXcK5p35vZM=; b=R2ZrH/i1drMQQTxLOEBANtnwfGSA1jgMXZNOPE9nCL4Xoas4Fkq8VOARbqqffTHg3yzP3J rCYXOVq3bVWJ0EBg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 15/21] timer: Add get next timer interrupt functionality for remote CPUs Date: Wed, 24 May 2023 09:06:23 +0200 Message-Id: <20230524070629.6377-16-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758561362715450?= X-GMAIL-MSGID: =?utf-8?q?1766758561362715450?= To prepare for the conversion of the NOHZ timer placement to a pull at expiry time model it's required to have functionality available getting the next timer interrupt on a remote CPU. Locking of the timer bases and getting the information for the next timer interrupt functionality is split into separate functions. This is required to be compliant with lock ordering when the new model is in place. Signed-off-by: Anna-Maria Behnsen Reviewed-by: Frederic Weisbecker --- v7: - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section - change lock, fetch functions to be unconditional - split out unlock function into a separate function v6: - introduce timer_lock_remote_bases() to fix race --- kernel/time/tick-internal.h | 10 +++++ kernel/time/timer.c | 77 ++++++++++++++++++++++++++++++++++--- 2 files changed, 82 insertions(+), 5 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 649f2b48e8f0..9b738117b2ab 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -8,6 +8,11 @@ #include "timekeeping.h" #include "tick-sched.h" +struct timer_events { + u64 local; + u64 global; +}; + #ifdef CONFIG_GENERIC_CLOCKEVENTS # define TICK_DO_TIMER_NONE -1 @@ -155,6 +160,11 @@ extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; +extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu); +extern void timer_lock_remote_bases(unsigned int cpu); +extern void timer_unlock_remote_bases(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 4bb6c168d106..564001b98098 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -221,11 +221,6 @@ struct timer_base { static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]); -struct timer_events { - u64 local; - u64 global; -}; - #ifdef CONFIG_NO_HZ_COMMON static DEFINE_STATIC_KEY_FALSE(timers_nohz_active); @@ -2013,6 +2008,78 @@ static unsigned long fetch_next_timer_interrupt(struct timer_base *base_local, return local_first ? nextevt_local : nextevt_global; } +# ifdef CONFIG_SMP +/** + * fetch_next_timer_interrupt_remote + * @basej: base time jiffies + * @basem: base time clock monotonic + * @tevt: Pointer to the storage for the expiry values + * @cpu: Remote CPU + * + * Stores the next pending local and global timer expiry values in the + * struct pointed to by @tevt. If a queue is empty the corresponding + * field is set to KTIME_MAX. If local event expires before global + * event, global event is set to KTIME_MAX as well. + * + * Caller needs to make sure timer base locks are held (use + * timer_lock_remote_bases() for this purpose). Caller must make sure + * interrupts are reopened, if required. + */ +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, + struct timer_events *tevt, + unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + /* Preset local / global events */ + tevt->local = tevt->global = KTIME_MAX; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_held(&base_local->lock); + lockdep_assert_held(&base_global->lock); + + fetch_next_timer_interrupt(base_local, base_global, basej, basem, tevt); +} + +/** + * timer_unlock_remote_bases - unlock timer bases of cpu + * @cpu: Remote CPU + * + * Unlocks the remote timer bases. + */ +void timer_unlock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + raw_spin_unlock(&base_global->lock); + raw_spin_unlock(&base_local->lock); +} + +/** + * timer_lock_remote_bases - lock timer bases of cpu + * @cpu: Remote CPU + * + * Locks the remote timer bases. + */ +void timer_lock_remote_bases(unsigned int cpu) +{ + struct timer_base *base_local, *base_global; + + base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu); + base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + lockdep_assert_irqs_disabled(); + + raw_spin_lock(&base_local->lock); + raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); +} +# endif /* CONFIG_SMP */ + /* * Forward base clock is done only when @basej is past base->clk, otherwise * base-clk might be rewind. From patchwork Wed May 24 07:06:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98337 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651097vqo; Wed, 24 May 2023 00:11:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6D7OB14/BxqPwGMsP3hMd/2/G6TyX9tZDrYtvHWr6SjFxgLjuyxJzFNR85e3LGOgoK8Kj+ X-Received: by 2002:a17:90a:ee8c:b0:24d:e296:659b with SMTP id i12-20020a17090aee8c00b0024de296659bmr17493211pjz.22.1684912265351; Wed, 24 May 2023 00:11:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912265; cv=none; d=google.com; s=arc-20160816; b=r3uoyGTtgS4iakcjR1XjQQVgGEGgZWFvV+csXKamtDLXoBFcQH8y3KdriiKAPdoK1t 7U2/75v+4NlvtdhnPx82G1RGfELJoeAnbnoH7nG1txAFlK4zDpmSz5V86nMyS3hiZQoe WoaB2KVvua/phJgsl5vfjhL2i5FqGzMaUtRV2MAzBo9BUKN0tiNU4bKG7rHtYL7eQOlK uyUXQhigOv7S+AcI43Mr0BEfhBTqRPnv7SNgQlOK9Dsjg3du5uzxv4FFZsIkY/Y52KjK fH+5AIGGKVvcZ3K3Rbg2uK4ix/HhXIRfDVdD4H5hpWXvjUNwsIhSlSzS/diITb7MLkbs VOVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=5Rgz0XMpFHBcgryFLb9u8XHoyQkZQCERFRJm4RJQ5aM=; b=uYQ33Q+G04VBJT2q8KD46yKu8dMBqIfAvV9Ga8aIYMDAlmjVxiiePPV/pwF/4Y8A8F e5uriafC3RIHOLxQQy8oRoGyk0xMm1ZA+bIIiVdo38dJvNP4qO18H1Az0MPi5kMAe3jb wFsh2oFI5xz7ORwM0z6aJH4mfXk8kNC90wAMUmNbYZIrgpx/95Yk/xhk2jzqKUqcJEOq i49j/9GQ2sZhclvc+BiSkw7JTddeG5wXt1CmRackP4jhsFAKoCJPu6kM4PeceLVETZXn VTS2XbU7QBIxooLyZWOQqc9rfJs7+zI3wEgMNcjyOc8rEAufqLb4uxJGodVcjenQrZ2r mXiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="ZA/565hY"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k70-20020a638449000000b0051b423d966csi1234300pgd.280.2023.05.24.00.10.52; Wed, 24 May 2023 00:11:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="ZA/565hY"; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239737AbjEXHIT (ORCPT + 99 others); Wed, 24 May 2023 03:08:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239908AbjEXHHj (ORCPT ); Wed, 24 May 2023 03:07:39 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50C431B4 for ; Wed, 24 May 2023 00:07:06 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5Rgz0XMpFHBcgryFLb9u8XHoyQkZQCERFRJm4RJQ5aM=; b=ZA/565hYULTzhp4Plywb8Bb+rNQozqthRlQ6lyBZEoVflXtTSl3KMZujNEkfx0dSNcU5/7 trLT0/nXE2rKYK6Cizc0bIhdbBXM44Qg9RlA3fFF0fgq6Vg2HKXAKhkbjZqkf0kruE5vZO nieSIsftVRLVWSu9ja7vlz+8Lbz13x96ybGfJQ4LhtyKs8McVBIfgEj34JufyUpUZ3n3/Q 8Gq8jjCogehMm6wwEdK3YSpTVigv7G4MRAwOBI97dFp9GDoRQvYuylShzLYDRBvp1Lanrf 3NFUmpc85EyY6G9KcUr2YW+U4rnL4i9lHp9GGV0Uqf6oTE9SLjYjbhw44w04iQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5Rgz0XMpFHBcgryFLb9u8XHoyQkZQCERFRJm4RJQ5aM=; b=B10AAADYkHpwVbWkLSNQvdrPRmrH7RAgbM7qllpRuQNJyuypUowUeOG9RoOUqqsfBCeaUt k5vfawz1UXqFJcCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v7 16/21] timer: Restructure internal locking Date: Wed, 24 May 2023 09:06:24 +0200 Message-Id: <20230524070629.6377-17-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758563727744826?= X-GMAIL-MSGID: =?utf-8?q?1766758563727744826?= From: "Richard Cochran (linutronix GmbH)" Move the locking out from __run_timers() to the call sites, so the protected section can be extended at the call site. Preparatory patch for changing the NOHZ timer placement to a pull at expiry time model. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/timer.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 564001b98098..20106dad531a 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2202,11 +2202,7 @@ static inline void __run_timers(struct timer_base *base) struct hlist_head heads[LVL_DEPTH]; int levels; - if (time_before(jiffies, base->next_expiry)) - return; - - timer_base_lock_expiry(base); - raw_spin_lock_irq(&base->lock); + lockdep_assert_held(&base->lock); while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { @@ -2226,21 +2222,36 @@ static inline void __run_timers(struct timer_base *base) while (levels--) expire_timers(base, heads + levels); } +} + +static void __run_timer_base(struct timer_base *base) +{ + if (time_before(jiffies, base->next_expiry)) + return; + + timer_base_lock_expiry(base); + raw_spin_lock_irq(&base->lock); + __run_timers(base); raw_spin_unlock_irq(&base->lock); timer_base_unlock_expiry(base); } +static void run_timer_base(int index) +{ + struct timer_base *base = this_cpu_ptr(&timer_bases[index]); + + __run_timer_base(base); +} + /* * This function runs timers and the timer-tq in bottom half context. */ static __latent_entropy void run_timer_softirq(struct softirq_action *h) { - struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]); - - __run_timers(base); + run_timer_base(BASE_LOCAL); if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { - __run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL])); - __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF])); + run_timer_base(BASE_GLOBAL); + run_timer_base(BASE_DEF); } } From patchwork Wed May 24 07:06:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651096vqo; Wed, 24 May 2023 00:11:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7SL17ugU3CRjjnq7r/Iwvjq1XWLcUy77j268jRt/wlnYF/4HVPfjVbDV9yIsWp+zk42v0W X-Received: by 2002:a17:902:c111:b0:1ab:1b8:8a5f with SMTP id 17-20020a170902c11100b001ab01b88a5fmr15085848pli.33.1684912265173; Wed, 24 May 2023 00:11:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912265; cv=none; d=google.com; s=arc-20160816; b=wL/E+p0K19RJaKwMBvmURnxD4gqN8kgMstfLAeVNBSYNsYSvI2HueJCIZhvFkSFmGb qDBsUu/UQOnDc3oFzmiJCUqRNzMv/5YUCPcAqK9bnvQUw3488Gq2yC55/VfEggxxA13m qhFPP1WPkwWGJQt5tBP5CwItoxwA+fAG7LeudztHAvcyy9o6xT1pWeLS2RT0jUYX0XwB 7ZtOqmVtGfQisayW/WAP/acaxu5TOMNya86m68279PDqHeManBqIVAQ/ON5vHrMaBaOs 9P1pbVS1aOmdAw73wEENjkZbOsYys8Ah3OaMIl4SOUfthoV3qscF6oxIa9rIMvVBQNtV zc/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=qCLAKadcYOZy8B4KFbVXg+LyGdFwzMy0r154VhivR5o=; b=cYJDZeOw48lKhcAR9rEniRCtl+oQkfv1JmB7jMMlsV89eyvf70YDgrK4DTfQSZJ2HS Pqa109tvF/GDIU0/zlu34mnw5w/idVwHnLnLXA4vmBCzZqCSMbHEwvxWU1jxMGIiSry+ QqgoJxN/xQv5S0tvBLubCX7MVT+l6nfPfRxVkkhle2wY0IdX4dHa4E3lWiZznnhb1zyW cx6nlA9Jg6KlA5M8NgNjd/Db55HkIt1IrH/ktamSfSzngK8p8rYzdoIRuYHbmJ8PrI6S 4mygfcpUzd0yK9b4aM0mwnhr8a2087p+Tx/tuxq1x6S1IyjiFkU4jUZ0stXjiWlQvpXf lCUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tHFqHdu2; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=kgRbb+1b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 21-20020a170902c21500b001a6d4eab493si1860640pll.26.2023.05.24.00.10.53; Wed, 24 May 2023 00:11:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=tHFqHdu2; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=kgRbb+1b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239975AbjEXHIW (ORCPT + 99 others); Wed, 24 May 2023 03:08:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239912AbjEXHHk (ORCPT ); Wed, 24 May 2023 03:07:40 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 011A4E6 for ; Wed, 24 May 2023 00:07:06 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qCLAKadcYOZy8B4KFbVXg+LyGdFwzMy0r154VhivR5o=; b=tHFqHdu20MXPAtpTRsfPwWNCbJztGEDXzxOplx9XGBrL3C7BfZWGQtFFfGYTlB//V2DEhR 9MsIKWOSexeT5+cBj4L+WmPYG0q13mwHDfbqMks0hIKqapcuzyyBLpblAXrgfIVPaa2LGL DRBetaRFUiLbawOwdu4+yG+2n/X918H+VEFe691uHIlC5AYM3EiAaP2pr7/cZ5qXMsobgf Bx3QoAsWsQQzqKuHTFKg16c3t1W4Jo2PtjrmXPLz64QR9Lyw8wBdaPaXS11/tQe1jBEcBh lUwlZgGBX+YsDBL5u1LNiyGnNQrKAYceYMCX4OaNU8jRCmpTJlWHvarvOzE73w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qCLAKadcYOZy8B4KFbVXg+LyGdFwzMy0r154VhivR5o=; b=kgRbb+1b+la7MSqatrlHvx+mu/1T9qsoiBvm94x+G6Yg8dKw7GjUSDxT5O+joLO9aWiuCW VIb6eGjqyNj2zSDw== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 17/21] timer: Check if timers base is handled already Date: Wed, 24 May 2023 09:06:25 +0200 Message-Id: <20230524070629.6377-18-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758563105466754?= X-GMAIL-MSGID: =?utf-8?q?1766758563105466754?= Due to the conversion of the NOHZ timer placement to a pull at expiry time model, the per CPU timer bases with non pinned timers are no longer handled only by the local CPU. In case a remote CPU already expires the non pinned timers base of the local cpu, nothing more needs to be done by the local CPU. A check at the begin of the expire timers routine is required, because timer base lock is dropped before executing the timer callback function. This is a preparatory work, but has no functional impact right now. Signed-off-by: Anna-Maria Behnsen --- v6: Drop double negation --- kernel/time/timer.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 20106dad531a..c2d66fd4638f 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2204,6 +2204,9 @@ static inline void __run_timers(struct timer_base *base) lockdep_assert_held(&base->lock); + if (base->running_timer) + return; + while (time_after_eq(jiffies, base->clk) && time_after_eq(jiffies, base->next_expiry)) { levels = collect_expired_timers(base, heads); From patchwork Wed May 24 07:06:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98338 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651133vqo; Wed, 24 May 2023 00:11:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6si2t50grZldt5hEynH+oNwgc57AEPvrrlyEtpoRvg7UAfcw9vnk0ajtkHAViZ3m0MtzI/ X-Received: by 2002:a17:902:c942:b0:1ab:94:1ee4 with SMTP id i2-20020a170902c94200b001ab00941ee4mr17560981pla.2.1684912269887; Wed, 24 May 2023 00:11:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912269; cv=none; d=google.com; s=arc-20160816; b=MIlsVHi2sMpJSyDNnyD3pbbkm3/FaU6eeQ0JRMn2Z/qNDgsqNJpAISqfPAUUVi6LrN Z2jc7M899EByTAFkQwVcwojTmhTfxwrbu3Q40VoqmX982A8I2SJSDooB/T4k80sCYFLv /pIEyqDeertsc5WrHJ2lNJxg72ngd229nXq5AjTRWGGhNPFYSEGTDFqGFPl7W22YYKuC k+T5RvUmZw9LCCwAmxhTFosLM38ysKZXacqcJmNo9oRWITNlsz3NUewWVdjxennjXu3Z Gn2fMsqhM2h2y1TjviV2LM5hwaRzGQN0oj6LfXPcTJRkAaWetKM5wcXW4KbpqGPEDGB7 cPCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=YzExw9CwT/xYQwHozBW+vEKUpVUnPxVAkZD2Ieh/LaM=; b=mpKgso9Zeu8KyOwTwjDIaL/zP4JmoMw6gwrQndbuKMogWJW51NzkO0bL9iBm9e5pnz TmGoyP6R5dWYHbupAuU73IADLfCS9GCiAB2Cmg6HbqqKTBuKBP/ABzAdYgOVyQDE2JPr uSUUzXxzHbTNQYlCLdAsXFxIRJNdYB+nTRmouqBx2fbPgR0FESDjghGnKyDNI+5je/zC KV5ZV8DkW/t7GPheDsGkmB/pHOyGFsuWpDXxJcVT7d8YB/F8+HpyH40oM+73wAwYVPnX LWaGyA4OkpSxwQ+yhgg/qJ5MKuS7I7FJvW9CEwuh+RfjirRS6ms6ypL3arfh6mcFoPTe ej2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lHvpjZ86; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v4-20020a170902b7c400b001ae4c3bbb0dsi299773plz.145.2023.05.24.00.10.56; Wed, 24 May 2023 00:11:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=lHvpjZ86; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239985AbjEXHIb (ORCPT + 99 others); Wed, 24 May 2023 03:08:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239918AbjEXHHk (ORCPT ); Wed, 24 May 2023 03:07:40 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFE691BF for ; Wed, 24 May 2023 00:07:07 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YzExw9CwT/xYQwHozBW+vEKUpVUnPxVAkZD2Ieh/LaM=; b=lHvpjZ86cvEeJSeF5EV5nGiDMEnPGsM89BE8/gGu8yVg0iwxws5hQFXGDpyd+cQdo7AHo0 ekOT2qSeeyCGY5ZkvHwipS464J+zIAumoHihgNH4mDskUP5K6OJCOevsKyW4oGo4PnIC5G ppA/YGfqRvdCBlKpHnUmjkTYbRNVTvxL4GwHekY5YbUNfauajSwzUfkELWAuscsXcFe7O5 9YBWPpUp6dbMCB2z1dYJQ7/AQK7bYalkHkC1+SuO7C3KVuDLaIuiP32MSuBSgcjkyQzMgi 5PRPbfoQWML/L96uVZRpE36vy/ZK2ShzH6aMD28wN4fXADbYN1smfjmHTzk1BA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YzExw9CwT/xYQwHozBW+vEKUpVUnPxVAkZD2Ieh/LaM=; b=XLvSwUscNPoTa358YL9bKhPMT2HSbQXx9gPfZw73Kx0hBid+oWgLDqUkekiFQtw6+0Yc2S MRrH56vJTHmBD4AA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , "Richard Cochran (linutronix GmbH)" , Anna-Maria Behnsen Subject: [PATCH v7 18/21] tick/sched: Split out jiffies update helper function Date: Wed, 24 May 2023 09:06:26 +0200 Message-Id: <20230524070629.6377-19-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758568631021043?= X-GMAIL-MSGID: =?utf-8?q?1766758568631021043?= From: "Richard Cochran (linutronix GmbH)" The logic to get the time of the last jiffies update will be needed by the timer pull model as well. Move the code into a global function in anticipation of the new caller. No functional change. Signed-off-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- kernel/time/tick-internal.h | 1 + kernel/time/tick-sched.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 9b738117b2ab..34d44fb50c81 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -158,6 +158,7 @@ static inline void tick_nohz_init(void) { } #ifdef CONFIG_NO_HZ_COMMON extern unsigned long tick_nohz_active; extern void timers_update_nohz(void); +extern u64 get_jiffies_update(unsigned long *basej); # ifdef CONFIG_SMP extern struct static_key_false timers_migration_enabled; extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 62836490ba4d..847c459c01f2 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -787,18 +787,30 @@ static inline bool local_timer_softirq_pending(void) return local_softirq_pending() & BIT(TIMER_SOFTIRQ); } -static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +/* + * Read jiffies and the time when jiffies were updated last + */ +u64 get_jiffies_update(unsigned long *basej) { - u64 basemono, next_tick, delta, expires; unsigned long basejiff; unsigned int seq; + u64 basemono; - /* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqcount_begin(&jiffies_seq); basemono = last_jiffies_update; basejiff = jiffies; } while (read_seqcount_retry(&jiffies_seq, seq)); + *basej = basejiff; + return basemono; +} + +static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu) +{ + u64 basemono, next_tick, delta, expires; + unsigned long basejiff; + + basemono = get_jiffies_update(&basejiff); ts->last_jiffies = basejiff; ts->timer_expires_base = basemono; From patchwork Wed May 24 07:06:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98347 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2654200vqo; Wed, 24 May 2023 00:17:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6xRdjj6GLh8j7qGsAyImdvXT6abupXj4xDcgXJv4WQGQmKiLEGL9yTQL4dxmDbJbXhDdnf X-Received: by 2002:a17:902:e547:b0:1ac:815e:320b with SMTP id n7-20020a170902e54700b001ac815e320bmr18956466plf.17.1684912676677; Wed, 24 May 2023 00:17:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912676; cv=none; d=google.com; s=arc-20160816; b=TWCN0BTgYlmK5Ydut2HVY4TeePLTRILIFn8nPsiSeEUjCBjJTZoeqJ6K3sAFIml9nD KYfLJar5agzbqzGxRKJEacJsDOKQqLwNto0kYMNYGxieFXHmR1AfvFGtGQUzA2XVbQA5 aVaflh4WElmm7CboBf2Fbe0N2QVib5ksBybw1qFbzIBM/K66IN4SUy8GiopAS6dqy8C0 X7DzU/8EVLLxYDSJaOknrZT1ESvSrMb00pncjeI7U/glMbsyVzXI83SD0TQdBxJkNc17 mK3IhMWt27tJGcj3oKwpm8m53jpikYoE7FWWJwyEGIVlM94OqbJy/UwSos/1bN9pTxnd 04SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=bEvFequz/v8RruCId8HNJNPT119D1XkoaDQOT2BX4vs=; b=alJ4V+fUV+9Av05VvKULcGyrAF8RGu46DVGEzXFCq8Bqy/6GU87A4ioRC9PG/svrxv UTYJMGWzpBchZwEKfkoxGo2+CVDadGXGw1XBRL1l6yQH8zRm+w519xOaFNZyrwhvayNt K8v8L5MLxnJyMrvJcplC/eiRzIrDY8zi83j+SlrmwYyrKBgvh2ZCL7YwRbWvN9LrKODp CjJmVccVd/v5yh5Fd+viHcBHTa9jfhstMevyCmcLQPBtrJfpik1eYMxqg+sTxdT0lOdj iwcfFTQchX1zdcHBrcyfVoV8Zi3xWoN/uP7Cxc+o3pqf5FwFsuzWXbfl9wrUER69A+0t gE7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ObzGprqU; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ja13-20020a170902efcd00b001ac6189cac3si517255plb.187.2023.05.24.00.17.44; Wed, 24 May 2023 00:17:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ObzGprqU; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234035AbjEXHIm (ORCPT + 99 others); Wed, 24 May 2023 03:08:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239929AbjEXHHt (ORCPT ); Wed, 24 May 2023 03:07:49 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65B69E45 for ; Wed, 24 May 2023 00:07:08 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bEvFequz/v8RruCId8HNJNPT119D1XkoaDQOT2BX4vs=; b=ObzGprqUayL24Y+3JI8FkJNmBeTslfwnGfr72GM7e2MpSyb08Tcm4uXvYGKHdhihWZ9ckd b4ThYApDTu0NWUBKqcmZYvFaZEbUSVTDqLKNFntC2ubZmrr/eZM3tczGPBXTMvu0eFKITT Ors811Z5TNzGTZgHd9ixJ78+yXx8nVx/LDkmj1qXk64nnktHhfSEyHtjd9tUjn0VUG02Dy PGTGkp+GBkgxn54+rLIaUF5AvBWChfwE/RTy9hgwhsn8ZSQljTb8zJyK/vctCIuHCl4cgA a0MP4bExtMBuzn/sm+gqjIOh+tIW2jY2YbilAkPxBZXuuJ14yqLDuFAJofKx0Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bEvFequz/v8RruCId8HNJNPT119D1XkoaDQOT2BX4vs=; b=yCElpIaC+G3devh5FqHZOTzaQCFPbNXTGV1lutEZr18Zfm9AejaO/pCHe4ZuwiEyLefndm w5YWr0VzF0/sN+AA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 19/21] timer: Implement the hierarchical pull model Date: Wed, 24 May 2023 09:06:27 +0200 Message-Id: <20230524070629.6377-20-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758995046542457?= X-GMAIL-MSGID: =?utf-8?q?1766758995046542457?= Placing timers at enqueue time on a target CPU based on dubious heuristics does not make any sense: 1) Most timer wheel timers are canceled or rearmed before they expire. 2) The heuristics to predict which CPU will be busy when the timer expires are wrong by definition. So placing the timers at enqueue wastes precious cycles. The proper solution to this problem is to always queue the timers on the local CPU and allow the non pinned timers to be pulled onto a busy CPU at expiry time. Therefore split the timer storage into local pinned and global timers: Local pinned timers are always expired on the CPU on which they have been queued. Global timers can be expired on any CPU. As long as a CPU is busy it expires both local and global timers. When a CPU goes idle it arms for the first expiring local timer. If the first expiring pinned (local) timer is before the first expiring movable timer, then no action is required because the CPU will wake up before the first movable timer expires. If the first expiring movable timer is before the first expiring pinned (local) timer, then this timer is queued into a idle timerqueue and eventually expired by some other active CPU. To avoid global locking the timerqueues are implemented as a hierarchy. The lowest level of the hierarchy holds the CPUs. The CPUs are associated to groups of 8, which are separated per node. If more than one CPU group exist, then a second level in the hierarchy collects the groups. Depending on the size of the system more than 2 levels are required. Each group has a "migrator" which checks the timerqueue during the tick for remote expirable timers. If the last CPU in a group goes idle it reports the first expiring event in the group up to the next group(s) in the hierarchy. If the last CPU goes idle it arms its timer for the first system wide expiring timer to ensure that no timer event is missed. Signed-off-by: Anna-Maria Behnsen --- v7: - Review remarks of Frederic and bigeasy: - change logic in tmigr_handle_remote_cpu() - s/kzalloc/kcalloc - move timer_expire_remote() into NO_HZ_COMMON && SMP config section - drop DBG_BUG_ON() makro and use only WARN_ON_ONCE() - remove leftovers from sibling logic during setup - Move timer_expire_remote() into tick-internal.h - Add documentation section about "Required event and timerqueue update after remote expiry" - Fix fallout of kernel test robot v6: - Fix typos - Review remarks of Peter Zijlstra (locking, struct member cleanup, use atomic_try_cmpxchg(), update struct member descriptions) - Fix race in tmigr_handle_remote_cpu() (Frederic Weisbecker) v5: - Review remarks of Frederic - Return nextevt when CPU is marked offline in timer migration hierarchy instead of KTIME_MAX - Fix update of group events issue, after remote expiring v4: - Fold typo fix in comment into proper patch "timer: Split out "get next timer interrupt" functionality" - Update wrong comment for tmigr_state union definition - Fix fallout of kernel test robot --- include/linux/cpuhotplug.h | 1 + kernel/time/Makefile | 3 + kernel/time/tick-internal.h | 1 + kernel/time/timer.c | 73 +- kernel/time/timer_migration.c | 1411 +++++++++++++++++++++++++++++++++ kernel/time/timer_migration.h | 136 ++++ 6 files changed, 1617 insertions(+), 8 deletions(-) create mode 100644 kernel/time/timer_migration.c create mode 100644 kernel/time/timer_migration.h diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 0f1001dca0e0..a32395b128de 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -242,6 +242,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, + CPUHP_AP_TMIGR_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, CPUHP_AP_RANDOM_ONLINE, diff --git a/kernel/time/Makefile b/kernel/time/Makefile index 7e875e63ff3b..4af2a264a160 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -17,6 +17,9 @@ endif obj-$(CONFIG_GENERIC_SCHED_CLOCK) += sched_clock.o obj-$(CONFIG_TICK_ONESHOT) += tick-oneshot.o tick-sched.o obj-$(CONFIG_LEGACY_TIMER_TICK) += tick-legacy.o +ifeq ($(CONFIG_SMP),y) + obj-$(CONFIG_NO_HZ_COMMON) += timer_migration.o +endif obj-$(CONFIG_HAVE_GENERIC_VDSO) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h index 34d44fb50c81..f5020473cc93 100644 --- a/kernel/time/tick-internal.h +++ b/kernel/time/tick-internal.h @@ -166,6 +166,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem, unsigned int cpu); extern void timer_lock_remote_bases(unsigned int cpu); extern void timer_unlock_remote_bases(unsigned int cpu); +extern void timer_expire_remote(unsigned int cpu); # endif #else /* CONFIG_NO_HZ_COMMON */ static inline void timers_update_nohz(void) { } diff --git a/kernel/time/timer.c b/kernel/time/timer.c index c2d66fd4638f..3a8c2c88f845 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -53,6 +53,7 @@ #include #include "tick-internal.h" +#include "timer_migration.h" #define CREATE_TRACE_POINTS #include @@ -2078,6 +2079,21 @@ void timer_lock_remote_bases(unsigned int cpu) raw_spin_lock(&base_local->lock); raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING); } + +static void __run_timer_base(struct timer_base *base); + +/** + * timer_expire_remote - expire global timers of cpu + * @cpu: Remote CPU + * + * Expire timers of global base of remote CPU. + */ +void timer_expire_remote(unsigned int cpu) +{ + struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu); + + __run_timer_base(base); +} # endif /* CONFIG_SMP */ /* @@ -2104,8 +2120,11 @@ static void forward_base_clk(struct timer_base *base, unsigned long nextevt, * idle. Idle marking of timer bases is allowed only to be done by CPU * itself. * - * Returns the tick aligned clock monotonic time of the next pending - * timer or KTIME_MAX if no timer is pending. + * Returns the tick aligned clock monotonic time of the next pending timer + * or KTIME_MAX if no timer is pending. If timer of global base was queued + * into timer migration hierarchy, first global timer is not taken into + * account. If it was the last CPU of timer migration hierarchy going idle, + * first global event is taken into account. */ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) { @@ -2149,6 +2168,40 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ is_idle = time_after(nextevt, basej + 1); + if (is_idle) { + u64 next_tmigr; + + /* + * Enqueue first global timer into timer migration + * hierarchy, afterwards tevt.global is no longer used. + */ + next_tmigr = tmigr_cpu_deactivate(tevt.global); + + /* + * If CPU is the last going idle in timer migration + * hierarchy, make sure CPU will wake up in time to handle + * remote timers. next_tmigr == KTIME_MAX if other CPUs are + * still active. + */ + if (next_tmigr < tevt.local) { + u64 tmp; + + /* If we missed a tick already, force 0 delta */ + if (next_tmigr < basem) + next_tmigr = basem; + + tmp = div_u64(next_tmigr - basem, TICK_NSEC); + + nextevt = basej + (unsigned long)tmp; + tevt.local = next_tmigr; + is_idle = time_after(nextevt, basej + 1); + } + /* + * Update of nextevt is not required in an else path, as it + * is revisited in !is_idle path only. + */ + } + /* We need to mark both bases in sync */ base_local->is_idle = base_global->is_idle = is_idle; @@ -2159,19 +2212,16 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) * If the bases are not marked idle, i.e one of the events is at * max. one tick away, then the CPU can't go into a NOHZ idle * sleep. Use the earlier event of both and store it in the local - * expiry value. The next global event is irrelevant in this case - * and can be left as KTIME_MAX. CPU will wakeup on time. + * expiry value. tevt.global update is superfluous and is + * ignored. CPU will wakeup on time. */ if (!is_idle) { /* If we missed a tick already, force 0 delta */ if (time_before(nextevt, basej)) nextevt = basej; tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC; - tevt.global = KTIME_MAX; } - tevt.local = min_t(u64, tevt.local, tevt.global); - return cmp_next_hrtimer_event(basem, tevt.local); } @@ -2190,6 +2240,9 @@ void timer_clear_idle(void) */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); + + /* Activate without holding the timer_base->lock */ + tmigr_cpu_activate(); } #endif @@ -2255,6 +2308,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h) if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) { run_timer_base(BASE_GLOBAL); run_timer_base(BASE_DEF); + + if (is_timers_nohz_active()) + tmigr_handle_remote(); } } @@ -2269,7 +2325,8 @@ static void run_local_timers(void) for (int i = 0; i < NR_BASES; i++, base++) { /* Raise the softirq only if required. */ - if (time_after_eq(jiffies, base->next_expiry)) { + if (time_after_eq(jiffies, base->next_expiry) || + (i == BASE_DEF && tmigr_requires_handle_remote())) { raise_softirq(TIMER_SOFTIRQ); return; } diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c new file mode 100644 index 000000000000..eab58bc98c7f --- /dev/null +++ b/kernel/time/timer_migration.c @@ -0,0 +1,1411 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Infrastructure for migratable timers + * + * Copyright(C) 2022 linutronix GmbH + */ +#include +#include +#include +#include +#include +#include + +#include "timer_migration.h" +#include "tick-internal.h" + +/* + * The timer migration mechanism is built on a hierarchy of groups. The + * lowest level group contains CPUs, the next level groups of CPU groups + * and so forth. The CPU groups are kept per node so for the normal case + * lock contention won't happen across nodes. Depending on the number of + * CPUs per node even the next level might be kept as groups of CPU groups + * per node and only the levels above cross the node topology. + * + * Example topology for a two node system with 24 CPUs each. + * + * LVL 2 [GRP2:0] + * GRP1:0 = GRP1:M + * + * LVL 1 [GRP1:0] [GRP1:1] + * GRP0:0 - GRP0:2 GRP0:3 - GRP0:5 + * + * LVL 0 [GRP0:0] [GRP0:1] [GRP0:2] [GRP0:3] [GRP0:4] [GRP0:5] + * CPUS 0-7 8-15 16-23 24-31 32-39 40-47 + * + * The groups hold a timer queue of events sorted by expiry time. These + * queues are updated when CPUs go in idle. When they come out of idle + * ignore bit of events is set. + * + * Each group has a designated migrator CPU/group as long as a CPU/group is + * active in the group. This designated role is necessary to avoid that all + * active CPUs in a group try to migrate expired timers from other cpus, + * which would result in massive lock bouncing. + * + * When a CPU is awake, it checks in it's own timer tick the group + * hierarchy up to the point where it is assigned the migrator role or if + * no CPU is active, it also checks the groups where no migrator is set + * (TMIGR_NONE). + * + * If it finds expired timers in one of the group queues it pulls them over + * from the idle CPU and runs the timer function. After that it updates the + * group and the parent groups if required. + * + * CPUs which go idle arm their CPU local timer hardware for the next local + * (pinned) timer event. If the next migratable timer expires after the + * next local timer or the CPU has no migratable timer pending then the + * CPU does not queue an event in the LVL0 group. If the next migratable + * timer expires before the next local timer then the CPU queues that timer + * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0 + * group. + * + * When CPU comes out of idle and when a group has at least a single active + * child, the ignore flag of the tmigr_event is set. This indicates, that + * the event is ignored even if it is still enqueued in the parent groups + * timer queue. It will be removed when touching the timer queue the next + * time. This spares locking in active path as the lock protects (after + * setup) only event information. For more information about locking, + * please read the section "Locking rules". + * + * If the CPU is the migrator of the group then it delegates that role to + * the next active CPU in the group or sets migrator to TMIGR_NONE when + * there is no active CPU in the group. This delegation needs to be + * propagated up the hierarchy so hand over from other leaves can happen at + * all hierarchy levels w/o doing a search. + * + * When the last CPU in the system goes idle, then it drops all migrator + * duties up to the top level of the hierarchy (LVL2 in the example). It + * then has to make sure, that it arms it's own local hardware timer for + * the earliest event in the system. + * + * + * Lifetime rules: + * --------------- + * + * The groups are built up at init time or when CPUs come online. They are + * not destroyed when a group becomes empty due to offlining. The group + * just won't participate in the hierachary management anymore. Destroying + * groups would result in interesting race conditions which would just make + * the whole mechanism slow and complex. + * + * + * Locking rules: + * -------------- + * + * For setting up new groups and handling events it's required to lock both + * child and parent group. The lock odering is always bottom up. This also + * includes the per CPU locks in struct tmigr_cpu. For updating migrator + * and active CPU/group information atomic_try_cmpxchg() is used instead + * and only per CPU tmigr_cpu->lock is held. + * + * During setup of groups tmigr_level_list is required. It is protected by + * tmigr_mutex. + * + * When timer_base->lock as well as tmigr related locks are required, lock + * ordering is: first timer_base->lock, afterwards tmigr related locks. + * + * + * Protection of tmigr group state information: + * -------------------------------------------- + * + * The state information with the list of active children and migrator needs + * to be protected by a sequence counter. It prevents a race when updates + * in child groups are propagated in changed order. The following scenario + * describes what happens without updating the sequence counter: + * + * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well + * as GRP0:1 will not change during the scenario): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU0 migrator = CPU2 + * active = CPU0 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * active idle active idle + * + * + * 1. CPU0 goes idle (changes are updated in GRP0:0; afterwards current + * states of GRP0:0 and GRP1:0 are stored in the data for walking the + * hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = TMIGR_NONE migrator = CPU2 + * --> active = active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * --> idle idle active idle + * + * 2. CPU1 comes out of idle (changes are update in GRP0:0; afterwards + * current states of GRP0:0 and GRP1:0 are stored in the data for + * walking the hierarchy): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:0, GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * --> migrator = CPU1 migrator = CPU2 + * --> active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle --> active active idle + * + * 3. Here comes the change of the order: Propagating the changes of step 2 + * through the hierarchy to GRP1:0 - nothing to be done, because GRP0:0 + * is already up to date. + * + * 4. Propagating the changes of step 1 through the hierarchy to GRP1:0 + * + * LVL 1 [GRP1:0] + * --> migrator = GRP0:1 + * --> active = GRP0:1 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = CPU1 migrator = CPU2 + * active = CPU1 active = CPU2 + * / \ / \ + * CPUs 0 1 2 3 + * idle active active idle + * + * Now there is a inconsistent overall state because GRP0:0 is active, but + * it is marked as idle in the GRP1:0. This is prevented by incrementing + * sequence counter whenever changing the state. + * + * + * Required event and timerqueue update after remote expiry: + * --------------------------------------------------------- + * + * After remote expiry of a CPU, a walk through the hierarchy updating + * events and timerqueues has to be done when there is a 'new' global timer + * of the remote CPU (which is obvious) but also if there is no new global + * timer, but the remote CPU is still idle: + * + * 1. CPU2 is migrator and does the remote expiry in GRP1:0; expiry of + * evt-CPU0 and evt-CPU1 are equal: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = 0 groupevt.ignore = 1 + * groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 2. Remove first event of timerqueue in GRP1:0 and expire timers of CPU0 + * (see evt-GRP0:0->cpu value): + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = 0 groupevt.ignore = 1 + * --> groupevt.cpu = CPU0 groupevt.cpu = + * timerqueue = evt-CPU0, timerqueue = + * evt-CPU1 + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * 3. After remote expiry CPU0 has no global timer that needs to be + * enqueued. When skipping the walk, global timer of CPU1 is not + * handled, as group event of GRP0:0 is not updated and not enqueued + * into GRP1:0. The walk has to be done to update the group events and + * timerqueues: + * + * LVL 1 [GRP1:0] + * migrator = GRP0:1 + * active = GRP0:1 + * --> timerqueue = evt-GRP0:0 + * / \ + * LVL 0 [GRP0:0] [GRP0:1] + * migrator = TMIGR_NONE migrator = CPU2 + * active = active = CPU2 + * groupevt.ignore = 0 groupevt.ignore = 1 + * --> groupevt.cpu = CPU1 groupevt.cpu = + * --> timerqueue = evt-CPU1 timerqueue = + * / \ / \ + * CPUs 0 1 2 3 + * idle idle active idle + * + * Now CPU2 (migrator) is able to handle timer of CPU1 as CPU2 only scans + * the timerqueues of GRP0:1 and GRP1:0. + * + * The update of step 3 is valid to be skipped, when remote CPU went + * offline in the meantime because update was already done during inactive + * path. When CPU became active in the meantime, update isn't required as + * well, because GRP0:0 is now longer idle. + */ + +static DEFINE_MUTEX(tmigr_mutex); +static struct list_head *tmigr_level_list __read_mostly; + +static unsigned int tmigr_hierarchy_levels __read_mostly; +static unsigned int tmigr_crossnode_level __read_mostly; + +static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu); + +#define TMIGR_NONE 0xFF +#define BIT_CNT 8 + +static DEFINE_STATIC_KEY_FALSE(tmigr_enabled); + +static inline bool is_tmigr_enabled(void) +{ + return static_branch_unlikely(&tmigr_enabled); +} + +/* + * Returns true, when @childmask corresponds to group migrator or when group + * is not active - so no migrator is set. + */ +static bool tmigr_check_migrator(struct tmigr_group *group, u8 childmask) +{ + union tmigr_state s; + + s.state = atomic_read(&group->migr_state); + + if ((s.migrator != childmask) && (s.migrator != TMIGR_NONE)) + return false; + + return true; +} + +typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *); + +static void __walk_groups(up_f up, void *data, + struct tmigr_cpu *tmc) +{ + struct tmigr_group *child = NULL, *group = tmc->tmgroup; + + do { + WARN_ON_ONCE(group->level >= tmigr_hierarchy_levels); + + if (up(group, child, data)) + break; + + child = group; + group = group->parent; + } while (group); +} + +static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc) +{ + lockdep_assert_held(&tmc->lock); + + __walk_groups(up, data, tmc); +} + +/** + * struct tmigr_walk - data required for walking the hierarchy + * @evt: Pointer to tmigr_event which needs to be queued (of idle + * child group) + * @childmask: childmask of child group + * @nextexp: Next CPU event expiry information which is handed + * into tmigr code by timer code + * (get_next_timer_interrupt()); it is furthermore + * used for first event which is queued, if timer + * migration hierarchy is completely idle + * @childstate: tmigr_group->migr_state of child - will be only reread + * when cmpxchg in group fails (is required for deactivate + * path and new timer path) + * @groupstate: tmigr_group->migr_state of group - will be only reread + * when cmpxchg in group fails (is required for active, + * deactivate and new timer path) + * @remote: Is set, when new timer path is executed in + * tmigr_handle_remote_cpu() + */ +struct tmigr_walk { + struct tmigr_event *evt; + u8 childmask; + u64 nextexp; + union tmigr_state childstate; + union tmigr_state groupstate; + bool remote; +}; + +/** + * struct tmigr_remote_data - data required for (check) remote expiry + * hierarchy walk + * @basej: timer base in jiffies + * @now: timer base monotonic + * @childmask: childmask of child group + * @check: is set to 1 if there is the need to handle remote timers; + * required in tmigr_check_handle_remote() only + * @wakeup: returns expiry of first timer in idle timer migration hierarchy + * to make sure timer is handled in time; it is stored in per CPU + * tmigr_cpu struct of CPU which expires remote timers + */ +struct tmigr_remote_data { + unsigned long basej; + u64 now; + u8 childmask; + int check; + u64 wakeup; +}; + +/* + * Returns next event of timerqueue @group->events + * + * Removes timers with ignore bits and update next_expiry of group. Values + * of group event are updated in tmigr_update_events() only. + */ +static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group) +{ + struct timerqueue_node *node = NULL; + struct tmigr_event *evt = NULL; + + lockdep_assert_held(&group->lock); + + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + while ((node = timerqueue_getnext(&group->events))) { + evt = container_of(node, struct tmigr_event, nextevt); + + if (!evt->ignore) { + WRITE_ONCE(group->next_expiry, evt->nextevt.expires); + return evt; + } + + /* + * Remove next timers with ignore bits, because group lock + * is held anyway + */ + if (!timerqueue_del(&group->events, node)) + break; + } + + return NULL; +} + +/* + * Return next event which is already expired of group timerqueue + * + * Event is also removed from queue. + */ +static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group, + u64 now) +{ + struct tmigr_event *evt = tmigr_next_groupevt(group); + + if (!evt || now < evt->nextevt.expires) + return NULL; + + /* + * Event is already expired. Remove it. If it's not the last event, + * then update all group event related information. + */ + if (timerqueue_del(&group->events, &evt->nextevt)) + tmigr_next_groupevt(group); + else + WRITE_ONCE(group->next_expiry, KTIME_MAX); + + return evt; +} + +static u64 tmigr_next_groupevt_expires(struct tmigr_group *group) +{ + struct tmigr_event *evt; + + evt = tmigr_next_groupevt(group); + + if (!evt) + return KTIME_MAX; + else + return evt->nextevt.expires; +} + +static bool tmigr_active_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + if (newstate.migrator == TMIGR_NONE) { + newstate.migrator = childmask; + + /* Changes need to be propagated */ + walk_done = false; + } + + newstate.active |= childmask; + + newstate.seq++; + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + goto retry; + } + + if (group->parent && (walk_done == false)) { + data->groupstate.state = atomic_read(&group->parent->migr_state); + data->childmask = group->childmask; + } + + /* + * Group is active, event will be ignored - bit is updated without + * holding the lock. In case bit is set while another CPU already + * handles remote events, nothing happens, because it is clear that + * CPU became active just in this moment, or in worst case event is + * handled remote. Nothing to worry about. + */ + group->groupevt.ignore = 1; + + return walk_done; +} + +static void __tmigr_cpu_activate(struct tmigr_cpu *tmc) +{ + struct tmigr_walk data; + + data.childmask = tmc->childmask; + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + tmc->cpuevt.ignore = 1; + + walk_groups(&tmigr_active_up, &data, tmc); +} + +/** + * tmigr_cpu_activate - set CPU active in timer migration hierarchy + * + * Call site timer_clear_idle() is called with interrupts disabled + */ +void tmigr_cpu_activate(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online || !tmc->idle) + return; + + raw_spin_lock(&tmc->lock); + tmc->idle = 0; + tmc->wakeup = KTIME_MAX; + __tmigr_cpu_activate(tmc); + raw_spin_unlock(&tmc->lock); +} + +/* + * Returns true, if there is nothing to be propagated to the next level + * + * @data->nextexp is reset to KTIME_MAX; it is reused for first global + * event which needs to be handled by migrator (in toplevel group) + * + * This is the only place where group event expiry value is set. + */ +static bool tmigr_update_events(struct tmigr_group *group, + struct tmigr_group *child, + struct tmigr_walk *data) +{ + struct tmigr_event *evt, *first_childevt; + bool walk_done, remote = data->remote; + u64 nextexp; + + if (child) { + if (data->childstate.active) + return true; + + raw_spin_lock(&child->lock); + raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING); + + first_childevt = tmigr_next_groupevt(child); + nextexp = child->next_expiry; + evt = &child->groupevt; + } else { + nextexp = data->nextexp; + + /* + * Set @data->nextexp to KTIME_MAX; it is reused for first + * global event which needs to be handled by migrator (in + * toplevel group) + */ + data->nextexp = KTIME_MAX; + + first_childevt = evt = data->evt; + + /* + * Walking the hierarchy is required in any case, when a + * remote expiry was done before. This ensures to not lost + * already queued events in non active groups (see section + * "Required event and timerqueue update after remote + * expiry" in documentation at the top). + */ + if (evt->ignore && !remote) + return true; + + raw_spin_lock(&group->lock); + } + + if (nextexp == KTIME_MAX) { + evt->ignore = 1; + + /* + * When next child event could be ignored (nextexp is + * KTIME_MAX) and there was no remote timer handling before + * or the group is already active, there is no need to walk + * the hierarchy even if there is a parent group. + * + * The other way round: even if the event could be ignored, + * but if a remote timer handling was executed before and + * the group is not active, walking the hierarchy is + * required to not miss an enqueued timer in the non active + * group. The enqueued timer of the group needs to be + * propagated to a higher level to ensure it is handled. + */ + if (!remote || data->groupstate.active) { + walk_done = true; + goto unlock; + } + } else { + /* + * Update of event cpu and ignore bit is required only when + * @child is set (child is equal or higher than lvl0), but + * it doesn't matter if it is written once more to per cpu + * event; make the update unconditional. + */ + evt->cpu = first_childevt->cpu; + evt->ignore = 0; + } + + walk_done = !group->parent; + + /* + * If child event is already queued in group, remove it from queue + * when expiry time changed only. + */ + if (timerqueue_node_queued(&evt->nextevt)) { + if (evt->nextevt.expires == nextexp) + goto check_toplvl; + else if (!timerqueue_del(&group->events, &evt->nextevt)) + WRITE_ONCE(group->next_expiry, KTIME_MAX); + } + + evt->nextevt.expires = nextexp; + + if (timerqueue_add(&group->events, &evt->nextevt)) + WRITE_ONCE(group->next_expiry, nextexp); + +check_toplvl: + if (walk_done && (data->groupstate.migrator == TMIGR_NONE)) { + /* + * Toplevel group is idle and it has to be ensured global + * timers are handled in time. (This could be optimized by + * keeping track of the last global scheduled event and + * only arming it on CPU if the new event is earlier. Not + * sure if its worth the complexity.) + */ + data->nextexp = tmigr_next_groupevt_expires(group); + } + +unlock: + raw_spin_unlock(&group->lock); + + if (child) + raw_spin_unlock(&child->lock); + + return walk_done; +} + +static bool tmigr_new_timer_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_walk *data = ptr; + bool walk_done; + + walk_done = tmigr_update_events(group, child, data); + + if (!walk_done) { + /* Update state information for next iteration */ + data->childstate.state = atomic_read(&group->migr_state); + if (group->parent) + data->groupstate.state = atomic_read(&group->parent->migr_state); + } + + return walk_done; +} + +/* + * Returns expiry of next timer that needs to be handled. KTIME_MAX is + * returned, when an active CPU will handle all timer migration hierarchy + * timers. + */ +static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .evt = &tmc->cpuevt, + .nextexp = nextexp }; + + lockdep_assert_held(&tmc->lock); + + if (tmc->remote) + return KTIME_MAX; + + tmc->cpuevt.ignore = 0; + + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + data.remote = false; + + walk_groups(&tmigr_new_timer_up, &data, tmc); + + /* If there is a new first global event, make sure it is handled */ + return data.nextexp; +} + +static bool tmigr_inactive_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + union tmigr_state curstate, newstate; + struct tmigr_walk *data = ptr; + bool walk_done; + u8 childmask; + + childmask = data->childmask; + newstate = curstate = data->groupstate; + +retry: + walk_done = true; + + /* Reset active bit when child is no longer active */ + if (!data->childstate.active) + newstate.active &= ~childmask; + + if (newstate.migrator == childmask) { + /* + * Find a new migrator for the group, because child group + * is idle! + */ + if (!data->childstate.active) { + unsigned long new_migr_bit, active = newstate.active; + + new_migr_bit = find_first_bit(&active, BIT_CNT); + + if (new_migr_bit != BIT_CNT) { + newstate.migrator = BIT(new_migr_bit); + } else { + newstate.migrator = TMIGR_NONE; + + /* Changes need to be propagated */ + walk_done = false; + } + } + } + + newstate.seq++; + + WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active)); + + if (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state)) { + newstate.state = curstate.state; + + /* + * Something changed in child/parent group in the meantime, + * reread the state of child and parent; Update of + * data->childstate is required for event handling; + */ + if (child) + data->childstate.state = atomic_read(&child->migr_state); + + goto retry; + } + + data->groupstate = newstate; + data->remote = false; + + /* Event Handling */ + tmigr_update_events(group, child, data); + + if (group->parent && (walk_done == false)) { + data->childmask = group->childmask; + data->childstate = newstate; + data->groupstate.state = atomic_read(&group->parent->migr_state); + } + + /* + * data->nextexp was set by tmigr_update_events() and contains the + * expiry of first global event which needs to be handled + */ + if (data->nextexp != KTIME_MAX) { + WARN_ON_ONCE(group->parent); + /* + * Toplevel path - If this cpu is about going offline wake + * up some random other cpu so it will take over the + * migrator duty and program its timer properly. Ideally + * wake the cpu with the closest expiry time, but that's + * overkill to figure out. + */ + if (!(this_cpu_ptr(&tmigr_cpu)->online)) { + unsigned int cpu = smp_processor_id(); + + cpu = cpumask_any_but(cpu_online_mask, cpu); + smp_send_reschedule(cpu); + } + } + + return walk_done; +} + +static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp) +{ + struct tmigr_walk data = { .childmask = tmc->childmask, + .evt = &tmc->cpuevt, + .nextexp = nextexp, + .childstate.state = 0 }; + + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + + /* + * If nextexp is KTIME_MAX, CPU event will be ignored because, + * local timer expires before global timer, no global timer is set + * or CPU goes offline. + */ + if (nextexp != KTIME_MAX) + tmc->cpuevt.ignore = 0; + + walk_groups(&tmigr_inactive_up, &data, tmc); + return data.nextexp; +} + +/** + * tmigr_cpu_deactivate - Put current CPU into inactive state + * @nextexp: The next timer event expiry set in the current CPU + * + * Must be called with interrupts disabled. + * + * Return: next event of the current CPU or next event from the hierarchy + * if this CPU is the top level migrator or hierarchy is completely idle. + */ +u64 tmigr_cpu_deactivate(u64 nextexp) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + u64 ret; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return nextexp; + + raw_spin_lock(&tmc->lock); + + /* + * CPU is already deactivated in timer migration + * hierarchy. tick_nohz_get_sleep_length() calls + * tick_nohz_next_event() and thereby timer idle path is + * executed once more. tmc->wakeup holds the first timer, when + * timer migration hierarchy is completely idle and remote + * expiry was done. If there is no new next expiry value + * handed in which should be inserted into the timer migration + * hierarchy, wakeup value is returned. + */ + if (tmc->idle) { + ret = tmc->wakeup; + + tmc->wakeup = KTIME_MAX; + + if (nextexp != KTIME_MAX) { + if (nextexp != tmc->cpuevt.nextevt.expires || + tmc->cpuevt.ignore) + ret = tmigr_new_timer(tmc, nextexp); + } + + goto unlock; + } + + ret = __tmigr_cpu_deactivate(tmc, nextexp); + + tmc->idle = 1; + +unlock: + raw_spin_unlock(&tmc->lock); + return ret; +} + +static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, + unsigned long jif) +{ + struct timer_events tevt; + struct tmigr_walk data; + struct tmigr_cpu *tmc; + u64 next = KTIME_MAX; + + tmc = per_cpu_ptr(&tmigr_cpu, cpu); + + raw_spin_lock_irq(&tmc->lock); + /* + * Remote CPU is offline or no longer idle or other cpu handles cpu + * timers already or next event was already expired - return! + */ + if (!tmc->online || tmc->remote || tmc->cpuevt.ignore || + now < tmc->cpuevt.nextevt.expires) { + raw_spin_unlock_irq(&tmc->lock); + return next; + } + + tmc->remote = 1; + + /* Drop the lock to allow the remote CPU to exit idle */ + raw_spin_unlock_irq(&tmc->lock); + + if (cpu != smp_processor_id()) + timer_expire_remote(cpu); + + /* + * Lock ordering needs to be preserved - timer_base locks before + * tmigr related locks (see section "Locking rules" in the + * documentation at the top). During fetching the next timer + * interrupt, also tmc->lock needs to be held. Otherwise there is a + * possible race window against the CPU itself when it comes out of + * idle, updates the first timer in hierarchy and goes back to + * idle. + * + * timer base locks are dropped as fast as possible: After checking + * whether the remote CPU when offline in the meantime and after + * fetching the next remote timer interrupt. Dropping the locks as + * fast as possible keeps the locking region small and prevents + * holding several (unnecessary) locks during walking the hierarchy + * for updating the timerqueue and group events. + */ + local_irq_disable(); + timer_lock_remote_bases(cpu); + raw_spin_lock(&tmc->lock); + + /* + * When CPU went offline in the meantime, no hierarchy walk has to + * be done for updating the queued events, because the walk was + * already done during marking CPU offline in the hierarchy. + * + * When CPU is no longer idle, CPU takes care of the timers and + * also of the timers in the path to the top. + * + * (See also section "Required event and timerqueue update after + * remote expiry" in documentation at the top) + */ + if (!tmc->online || !tmc->idle) { + timer_unlock_remote_bases(cpu); + goto unlock; + } else { + /* next event of cpu */ + fetch_next_timer_interrupt_remote(jif, now, &tevt, cpu); + } + + timer_unlock_remote_bases(cpu); + + data.evt = &tmc->cpuevt; + data.nextexp = tevt.global; + data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); + data.remote = true; + + /* + * Update is done even when there is no 'new' global timer pending + * on remote CPU (see section "Required event and timerqueue update + * after remote expiry" in documentation at the top) + */ + walk_groups(&tmigr_new_timer_up, &data, tmc); + + next = data.nextexp; + +unlock: + tmc->remote = 0; + raw_spin_unlock_irq(&tmc->lock); + + return next; +} + +static bool tmigr_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u64 now, next = KTIME_MAX; + struct tmigr_event *evt; + unsigned long jif; + u8 childmask; + + jif = data->basej; + now = data->now; + + childmask = data->childmask; + +again: + /* + * Handle the group only if @childmask is the migrator or if the + * group has no migrator. Otherwise the group is active and is + * handled by its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + raw_spin_lock_irq(&group->lock); + + evt = tmigr_next_expired_groupevt(group, now); + + if (evt) { + unsigned int remote_cpu = evt->cpu; + + raw_spin_unlock_irq(&group->lock); + + next = tmigr_handle_remote_cpu(remote_cpu, now, jif); + + /* check if there is another event, that needs to be handled */ + goto again; + } else { + raw_spin_unlock_irq(&group->lock); + } + + /* Update of childmask for next level */ + data->childmask = group->childmask; + data->wakeup = next; + + return false; +} + +/** + * tmigr_handle_remote - Handle migratable timers on remote idle CPUs + * + * Called from the timer soft interrupt with interrupts enabled. + */ +void tmigr_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return; + + /* + * NOTE: This is a doubled check because migrator test will be done + * in tmigr_handle_remote_up() anyway. Keep this check to fasten + * the return when nothing has to be done. + */ + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return; + + data.now = get_jiffies_update(&data.basej); + data.childmask = tmc->childmask; + data.wakeup = KTIME_MAX; + + __walk_groups(&tmigr_handle_remote_up, &data, tmc); + + raw_spin_lock_irq(&tmc->lock); + if (tmc->idle) + tmc->wakeup = data.wakeup; + + raw_spin_unlock_irq(&tmc->lock); +} + +static bool tmigr_requires_handle_remote_up(struct tmigr_group *group, + struct tmigr_group *child, + void *ptr) +{ + struct tmigr_remote_data *data = ptr; + u8 childmask; + + childmask = data->childmask; + + /* + * Handle the group only if child is the migrator or if the group + * has no migrator. Otherwise the group is active and is handled by + * its own migrator. + */ + if (!tmigr_check_migrator(group, childmask)) + return true; + + /* + * On 32 bit systems the racy lockless check for next_expiry will + * turn into a random number generator. Therefore do the lockless + * check only on 64 bit systems. + */ + if (IS_ENABLED(CONFIG_64BIT)) { + if (data->now >= READ_ONCE(group->next_expiry)) { + data->check = 1; + return true; + } + } else { + raw_spin_lock(&group->lock); + if (data->now >= group->next_expiry) { + data->check = 1; + raw_spin_unlock(&group->lock); + return true; + } + raw_spin_unlock(&group->lock); + } + + /* Update of childmask for next level */ + data->childmask = group->childmask; + return false; +} + +/** + * tmigr_requires_handle_remote - Check whether remote timer handling is required + * + * Must be called with interrupts disabled. + */ +int tmigr_requires_handle_remote(void) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + struct tmigr_remote_data data; + + if (!is_tmigr_enabled() || !tmc->tmgroup || !tmc->online) + return 0; + + if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask)) + return 0; + + data.now = get_jiffies_update(&data.basej); + data.childmask = tmc->childmask; + + __walk_groups(&tmigr_requires_handle_remote_up, &data, tmc); + + return data.check; +} + +static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl, + int node) +{ + union tmigr_state s; + + raw_spin_lock_init(&group->lock); + + group->level = lvl; + group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE; + + group->num_children = 0; + + s.migrator = TMIGR_NONE; + s.active = 0; + s.seq = 0; + atomic_set(&group->migr_state, s.state); + + timerqueue_init_head(&group->events); + timerqueue_init(&group->groupevt.nextevt); + group->groupevt.nextevt.expires = KTIME_MAX; + WRITE_ONCE(group->next_expiry, KTIME_MAX); + group->groupevt.ignore = 1; +} + +static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, + unsigned int lvl) +{ + struct tmigr_group *tmp, *group = NULL; + + lockdep_assert_held(&tmigr_mutex); + + /* Try to attach to an exisiting group first */ + list_for_each_entry(tmp, &tmigr_level_list[lvl], list) { + /* + * If @lvl is below the cross numa node level, check whether + * this group belongs to the same numa node. + */ + if (lvl < tmigr_crossnode_level && tmp->numa_node != node) + continue; + + /* Capacity left? */ + if (tmp->num_children >= TMIGR_CHILDS_PER_GROUP) + continue; + + /* + * TODO: A possible further improvement: Make sure that all + * CPU siblings end up in the same group of lowest level of + * the hierarchy. Rely on topology sibling mask would be a + * reasonable solution. + */ + + group = tmp; + break; + } + + if (group) + return group; + + /* Allocate and set up a new group */ + group = kzalloc_node(sizeof(*group), GFP_KERNEL, node); + if (!group) + return ERR_PTR(-ENOMEM); + + tmigr_init_group(group, lvl, node); + + /* Setup successful. Add it to the hierarchy */ + list_add(&group->list, &tmigr_level_list[lvl]); + return group; +} + +static void tmigr_connect_child_parent(struct tmigr_group *child, + struct tmigr_group *parent) +{ + union tmigr_state childstate; + + raw_spin_lock_irq(&child->lock); + raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING); + + child->parent = parent; + child->childmask = BIT(parent->num_children++); + + raw_spin_unlock(&parent->lock); + raw_spin_unlock_irq(&child->lock); + + /* + * To prevent inconsistent states, active children need to be + * active in new parent as well. Inactive children are already + * marked inactive in parent group. + */ + childstate.state = atomic_read(&child->migr_state); + if (childstate.migrator != TMIGR_NONE) { + struct tmigr_walk data; + + data.childmask = child->childmask; + data.groupstate.state = atomic_read(&parent->migr_state); + + /* + * There is only one new level per time. When connecting + * child and parent and set child active when parent is + * inactive, parent needs to be the uppermost + * level. Otherwise there went something wrong! + */ + WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent); + } +} + +static int tmigr_setup_groups(unsigned int cpu, unsigned int node) +{ + struct tmigr_group *group, *child, **stack; + int top = 0, err = 0, i = 0; + struct list_head *lvllist; + + stack = kcalloc(tmigr_hierarchy_levels, sizeof(*stack), GFP_KERNEL); + if (!stack) + return -ENOMEM; + + do { + group = tmigr_get_group(cpu, node, i); + if (IS_ERR(group)) { + err = PTR_ERR(group); + break; + } + + top = i; + stack[i++] = group; + + /* + * When booting only less CPUs of a system than CPUs are + * available, not all calculated hierarchy levels are required. + * + * The loop is aborted as soon as the highest level, which might + * be different from tmigr_hierarchy_levels, contains only a + * single group. + */ + if (group->parent || i == tmigr_hierarchy_levels || + (list_empty(&tmigr_level_list[i]) && + list_is_singular(&tmigr_level_list[i - 1]))) + break; + + } while (i < tmigr_hierarchy_levels); + + do { + group = stack[--i]; + + if (err < 0) { + list_del(&group->list); + kfree(group); + continue; + } + + WARN_ON_ONCE(i != group->level); + + /* + * Update tmc -> group / child -> group connection + */ + if (i == 0) { + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&group->lock); + + tmc->tmgroup = group; + tmc->childmask = BIT(group->num_children++); + + raw_spin_unlock_irq(&group->lock); + + /* There are no children that need to be connected */ + continue; + } else { + child = stack[i - 1]; + tmigr_connect_child_parent(child, group); + } + + /* check if uppermost level was newly created */ + if (top != i) + continue; + + WARN_ON_ONCE(top == 0); + + lvllist = &tmigr_level_list[top]; + if (group->num_children == 1 && list_is_singular(lvllist)) { + lvllist = &tmigr_level_list[top - 1]; + list_for_each_entry(child, lvllist, list) { + if (child->parent) + continue; + + tmigr_connect_child_parent(child, group); + } + } + } while (i > 0); + + kfree(stack); + + return err; +} + +static int tmigr_add_cpu(unsigned int cpu) +{ + int node = cpu_to_node(cpu); + int ret; + + mutex_lock(&tmigr_mutex); + ret = tmigr_setup_groups(cpu, node); + mutex_unlock(&tmigr_mutex); + + return ret; +} + +static int tmigr_cpu_online(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + unsigned int ret; + + /* First online attempt? Initialize CPU data */ + if (!tmc->tmgroup) { + raw_spin_lock_init(&tmc->lock); + + ret = tmigr_add_cpu(cpu); + if (ret < 0) + return ret; + + if (tmc->childmask == 0) + return -EINVAL; + + timerqueue_init(&tmc->cpuevt.nextevt); + tmc->cpuevt.nextevt.expires = KTIME_MAX; + tmc->cpuevt.ignore = 1; + tmc->cpuevt.cpu = cpu; + + tmc->remote = 0; + tmc->idle = 0; + tmc->wakeup = KTIME_MAX; + } + raw_spin_lock_irq(&tmc->lock); + __tmigr_cpu_activate(tmc); + tmc->online = 1; + raw_spin_unlock_irq(&tmc->lock); + return 0; +} + +static int tmigr_cpu_offline(unsigned int cpu) +{ + struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu); + + raw_spin_lock_irq(&tmc->lock); + tmc->online = 0; + + /* + * CPU has to handle local events on his own, when on the way to + * offline; Therefore nextevt value is set to KTIME_MAX + */ + __tmigr_cpu_deactivate(tmc, KTIME_MAX); + raw_spin_unlock_irq(&tmc->lock); + + return 0; +} + +static int __init tmigr_init(void) +{ + unsigned int cpulvl, nodelvl, cpus_per_node, i; + unsigned int nnodes = num_possible_nodes(); + unsigned int ncpus = num_possible_cpus(); + int ret = -ENOMEM; + + /* Nothing to do if running on UP */ + if (ncpus == 1) + return 0; + + /* + * Calculate the required hierarchy levels. Unfortunately there is no + * reliable information available, unless all possible CPUs have been + * brought up and all numa nodes are populated. + * + * Estimate the number of levels with the number of possible nodes and + * the number of possible cpus. Assume CPUs are spread evenly across + * nodes. We cannot rely on cpumask_of_node() because there only already + * online CPUs are considered. + */ + cpus_per_node = DIV_ROUND_UP(ncpus, nnodes); + + /* Calc the hierarchy levels required to hold the CPUs of a node */ + cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node), + ilog2(TMIGR_CHILDS_PER_GROUP)); + + /* Calculate the extra levels to connect all nodes */ + nodelvl = DIV_ROUND_UP(order_base_2(nnodes), + ilog2(TMIGR_CHILDS_PER_GROUP)); + + tmigr_hierarchy_levels = cpulvl + nodelvl; + + /* + * If a numa node spawns more than one CPU level group then the next + * level(s) of the hierarchy contains groups which handle all CPU groups + * of the same numa node. The level above goes across numa nodes. Store + * this information for the setup code to decide when node matching is + * not longer required. + */ + tmigr_crossnode_level = cpulvl; + + tmigr_level_list = kcalloc(tmigr_hierarchy_levels, sizeof(struct list_head), GFP_KERNEL); + if (!tmigr_level_list) + goto err; + + for (i = 0; i < tmigr_hierarchy_levels; i++) + INIT_LIST_HEAD(&tmigr_level_list[i]); + + pr_info("Timer migration: %d hierarchy levels; %d children per group;" + " %d crossnode level\n", + tmigr_hierarchy_levels, TMIGR_CHILDS_PER_GROUP, + tmigr_crossnode_level); + + ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online", + tmigr_cpu_online, tmigr_cpu_offline); + if (ret) + goto err; + + static_branch_enable(&tmigr_enabled); + + return 0; + +err: + pr_err("Timer migration setup failed\n"); + return ret; +} +late_initcall(tmigr_init); diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h new file mode 100644 index 000000000000..3a31a16443d2 --- /dev/null +++ b/kernel/time/timer_migration.h @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _KERNEL_TIME_MIGRATION_H +#define _KERNEL_TIME_MIGRATION_H + +/* Per group capacity. Must be a power of 2! */ +#define TMIGR_CHILDS_PER_GROUP 8 + +/** + * struct tmigr_event - a timer event associated to a CPU + * @nextevt: The node to enqueue an event in the parent group queue + * @cpu: The CPU to which this event belongs + * @ignore: Hint whether the event could be ignored; it is set when + * CPU or group is active; + */ +struct tmigr_event { + struct timerqueue_node nextevt; + unsigned int cpu; + int ignore; +}; + +/** + * struct tmigr_group - timer migration hierarchy group + * @lock: Lock protecting the event information and group hierarchy + * information during setup + * @migr_state: State of group (see union tmigr_state) + * @parent: Pointer to parent group + * @groupevt: Next event of group which is only used when group is + * !active. The group event is then queued into parent + * timer queue. + * Ignore bit of groupevt is set when group is active. + * @next_expiry: Base monotonic expiry time of next event of group; + * Used for racy lockless check whether remote expiry is + * required; it is always reliable + * @events: Timer queue for child events queued in the group + * @childmask: childmask of group in parent group; is set during + * setup and will never change; could be read lockless + * @level: Hierarchy level of group; Required during setup and + * when DEBUG is defined + * @list: List head that is added to per level + * tmigr_level_list; is required during setup when a + * new group needs to be connected to the existing + * hierarchy groups + * @numa_node: Is set to numa node when level < tmigr_crossnode_level; + * otherwise it is set to NUMA_NO_NODE; Required for + * setup only to make sure CPUs and groups are per + * numa node as long as level < tmigr_crossnode_level + * @num_children: Counter of group children to make sure group is only filled + * with TMIGR_CHILDS_PER_GROUP; Required for setup only + */ +struct tmigr_group { + raw_spinlock_t lock; + atomic_t migr_state; + struct tmigr_group *parent; + struct tmigr_event groupevt; + u64 next_expiry; + struct timerqueue_head events; + u8 childmask; + unsigned int level; + struct list_head list; + int numa_node; + unsigned int num_children; +}; + +/** + * struct tmigr_cpu - timer migration per CPU group + * @lock: Lock protecting tmigr_cpu group information + * @online: Indicates whether CPU is online; In deactivate path it is + * required to know whether the migrator in top level group is + * on the way to go offline when a timer is pending. Then + * another online CPU needs to be rescheduled to make sure + * timers are handled properly; Furthermore the information is + * required in hotplug path as CPU is able to go idle before + * timer migration hierarchy hotplug AP is reached. During + * this phase CPU has to handle global timers by its own and + * does not act as migrator. + * @idle: Indicates whether CPU is idle in timer migration hierarchy + * @remote: Is set when timers of CPU are expired remote + * @tmgroup: Pointer to parent group + * @childmask: childmask of tmigr_cpu in parent group + * @wakeup: Stores the first timer when the timer migration hierarchy is + * completely idle and remote expiry was done; is returned to + * timer code when tmigr_cpu_deactive() is called and group is + * idle; afterwards a reset to KTIME_MAX is required; + * @cpuevt: CPU event which could be queued into parent group + */ +struct tmigr_cpu { + raw_spinlock_t lock; + int online; + int idle; + int remote; + struct tmigr_group *tmgroup; + u8 childmask; + u64 wakeup; + struct tmigr_event cpuevt; +}; + +/** + * union tmigr_state - state of tmigr_group + * @state: Combined version of the state - only used for atomic + * read/cmpxchg function + * @struct: Split version of the state - only use the struct members to + * update information to stay independent of endianness + */ +union tmigr_state { + u32 state; + /** + * struct - split state of tmigr_group + * @active: Contains each childmask bit of active children + * @migrator: Contains childmask of child which is migrator + * @seq: Sequence counter needs to be increased when update + * to the tmigr_state is done. It prevents a race when + * updates in child groups are propagated in changed + * order. Detailed information about the scenario is + * given in documentation at the begin of + * timer_migration.c. + */ + struct { + u8 active; + u8 migrator; + u16 seq; + } __packed; +}; + +#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) +extern void tmigr_handle_remote(void); +extern int tmigr_requires_handle_remote(void); +extern void tmigr_cpu_activate(void); +extern u64 tmigr_cpu_deactivate(u64 nextevt); +#else +static inline void tmigr_handle_remote(void) { } +static inline int tmigr_requires_handle_remote(void) { return 0; } +static inline void tmigr_cpu_activate(void) { } +static inline u64 tmigr_cpu_deactivate(u64 nextevt) { return KTIME_MAX; } +#endif + +#endif From patchwork Wed May 24 07:06:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98339 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651238vqo; Wed, 24 May 2023 00:11:25 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5LJ4VKdHBbs0gdwK2XIqjylryHA0Es9Woe1eq7HE5aLxpXMvvlQLaG3uVXTEoNov8Ql0Vo X-Received: by 2002:a17:902:c1c4:b0:1af:b678:5168 with SMTP id c4-20020a170902c1c400b001afb6785168mr7787778plc.67.1684912284859; Wed, 24 May 2023 00:11:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912284; cv=none; d=google.com; s=arc-20160816; b=Y0hekoPaD6awWNZXvFqZXOso+DlQD+9TUwT80dsImGibikswlmLejZVIXGT5PmlGQL hmqHATNHQdlMrhrS3q+kLHDDJenW58eTmyQXKUnrr9gRDQbQczutfJXNxXQcOo03tcso 1lW3PDV83e+70EEYF7KX7C0EeQGxtNcil5jwiwwX7u4zMDfj+oracjvJeugpIKbMZCzI v40gSYi4cZZj1mNQae03bxiOnKW/dgHE4RgS6dg5AmUDi9V6+EeCcwAHSmtF6aN4tIwM YL1K6Hhc2Pe0LZVfNNn0f97TLL4+KOoZkVdmGF0evkrMmlm3MzdfEvp81vGgwdJ0DKv9 DAZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=heNPTQfD0cWTbtGRvFw0d1IJtEXxzpmCYrB9DDNKIMc=; b=zzuIKw36/t04/Qpmz6E2E0i/e7Er1/P9EIZ7yAC/zOEUFeUDLJTcg/jxlyA8K6tg9j YW/EZphNwJdNlLc3e4lbm7sxCIQ7j0LvHWkkkbMGisYtwDusGMBMkBmtwcwjFO/UZ5uI Z7U9JYvfGgw2BdSrRZ2pciM7GcoF+AM9tK+iwtEaE3RU/M0ToJmXV1EwMkBm4+aNs4o3 GGaSKhcZZuWS+01UUYzU38gJ/U39xnM5OUIkid+cqJZeZWnM9uRh/V8b1yJfRpxMFoC8 28qUZaK4voEe2BLmhwxdNZd7pDAE+aX7TXvbDZRUrc7nq4VASMDRO6o1b9bn7P268wGp L0vQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=jpf2s3I0; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ja13-20020a170902efcd00b001ac6189cac3si517255plb.187.2023.05.24.00.11.12; Wed, 24 May 2023 00:11:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=jpf2s3I0; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239926AbjEXHIf (ORCPT + 99 others); Wed, 24 May 2023 03:08:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239922AbjEXHHs (ORCPT ); Wed, 24 May 2023 03:07:48 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB0E9E49 for ; Wed, 24 May 2023 00:07:08 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=heNPTQfD0cWTbtGRvFw0d1IJtEXxzpmCYrB9DDNKIMc=; b=jpf2s3I0hJyrGmUQ45RxU/wkv+hs98kmthB0tsGPmNuCY5UncjR2ygOpiWtmSYWxEfvYlG wHfTFlZXSgMAnVmnC9hbdm5f38zJ7xn8Nz64pY229nPUzlnsMk1I2aEbq1TSf5iqcrFdwm Vm+BxphIqk9flrylHyrU8nrmf7CtEo4bairbETcPyLOEUlr/kibXo7fhwQ6UR3PhXovnvG O/iwAR6ZAxcoCKNxvcTxZJ/2xBxYfDxNvby1KsSBpbzFmu7YVBXytsE3tm+hfyxuTWKxpT aGfC6GLJsyT3+bzN+u1bNllOF9vFeM2jQ3j5zs2SQLgO6hATCYtrV61yjDMWdA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=heNPTQfD0cWTbtGRvFw0d1IJtEXxzpmCYrB9DDNKIMc=; b=pHm04MrrK3mCBAucYvgtSdBMzDi2jnRF1hYv5beO7KRfIgveS3HntcoWI6MLzY4ywjt8Sg DKW45k/TxFIDyaCA== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen Subject: [PATCH v7 20/21] timer_migration: Add tracepoints Date: Wed, 24 May 2023 09:06:28 +0200 Message-Id: <20230524070629.6377-21-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758584086660459?= X-GMAIL-MSGID: =?utf-8?q?1766758584086660459?= The timer pull logic needs proper debugging aids. Add tracepoints so the hierarchical idle machinery can be diagnosed. Signed-off-by: Anna-Maria Behnsen --- include/trace/events/timer_migration.h | 277 +++++++++++++++++++++++++ kernel/time/timer_migration.c | 24 +++ 2 files changed, 301 insertions(+) create mode 100644 include/trace/events/timer_migration.h diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h new file mode 100644 index 000000000000..9efbaf5dabe6 --- /dev/null +++ b/include/trace/events/timer_migration.h @@ -0,0 +1,277 @@ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM timer_migration + +#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_TIMER_MIGRATION_H + +#include + +/* Group events */ +TRACE_EVENT(tmigr_group_set, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void *, group ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + ), + + TP_printk("group=%p lvl=%d numa=%d", + __entry->group, __entry->lvl, __entry->numa_node) +); + +TRACE_EVENT(tmigr_connect_child_parent, + + TP_PROTO(struct tmigr_group *child), + + TP_ARGS(child), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->parent = child->parent; + __entry->lvl = child->parent->level; + __entry->numa_node = child->parent->numa_node; + __entry->numa_node = child->parent->num_children; + __entry->childmask = child->childmask; + ), + + TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->child, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +TRACE_EVENT(tmigr_connect_cpu_parent, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent ) + __field( unsigned int, cpu ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( unsigned int, num_children ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->parent = tmc->tmgroup; + __entry->cpu = tmc->cpuevt.cpu; + __entry->lvl = tmc->tmgroup->level; + __entry->numa_node = tmc->tmgroup->numa_node; + __entry->numa_node = tmc->tmgroup->num_children; + __entry->childmask = tmc->childmask; + ), + + TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d", + __entry->cpu, __entry->childmask, __entry->parent, + __entry->lvl, __entry->numa_node, __entry->num_children) +); + +DECLARE_EVENT_CLASS(tmigr_group_and_cpu, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask), + + TP_STRUCT__entry( + __field( void *, group ) + __field( void *, parent ) + __field( unsigned int, lvl ) + __field( unsigned int, numa_node ) + __field( u8, active ) + __field( u8, migrator ) + __field( u32, childmask ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->parent = group->parent; + __entry->lvl = group->level; + __entry->numa_node = group->numa_node; + __entry->active = state.active; + __entry->migrator = state.migrator; + __entry->childmask = childmask; + ), + + TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x " + "parent=%p childmask=%0x", + __entry->group, __entry->lvl, __entry->numa_node, + __entry->active, __entry->migrator, + __entry->parent, __entry->childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active, + + TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask), + + TP_ARGS(group, state, childmask) +); + +/* CPU events*/ +DECLARE_EVENT_CLASS(tmigr_cpugroup, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + ), + + TP_printk("cpu=%d parent=%p", __entry->cpu, __entry->parent) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu, + + TP_PROTO(struct tmigr_cpu *tmc), + + TP_ARGS(tmc) +); + +TRACE_EVENT(tmigr_cpu_idle, + + TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt), + + TP_ARGS(tmc, nextevt), + + TP_STRUCT__entry( + __field( void *, parent) + __field( unsigned int, cpu) + __field( u64, nextevt) + ), + + TP_fast_assign( + __entry->cpu = tmc->cpuevt.cpu; + __entry->parent = tmc->tmgroup; + __entry->nextevt = nextevt; + ), + + TP_printk("cpu=%d parent=%p nextevt=%llu", + __entry->cpu, __entry->parent, __entry->nextevt) +); + +TRACE_EVENT(tmigr_update_events, + + TP_PROTO(struct tmigr_group *child, struct tmigr_group *group, + union tmigr_state childstate, union tmigr_state groupstate, + u64 nextevt), + + TP_ARGS(child, group, childstate, groupstate, nextevt), + + TP_STRUCT__entry( + __field( void *, child ) + __field( void *, group ) + __field( u64, nextevt ) + __field( u64, group_next_expiry ) + __field( unsigned int, group_lvl ) + __field( u8, child_active ) + __field( u8, group_active ) + __field( unsigned int, child_evtcpu ) + __field( u64, child_evt_expiry ) + ), + + TP_fast_assign( + __entry->child = child; + __entry->group = group; + __entry->nextevt = nextevt; + __entry->group_next_expiry = group->next_expiry; + __entry->group_lvl = group->level; + __entry->child_active = childstate.active; + __entry->group_active = groupstate.active; + __entry->child_evtcpu = child ? child->groupevt.cpu : 0; + __entry->child_evt_expiry = child ? child->groupevt.nextevt.expires : 0; + ), + + TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x " + "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d", + __entry->child, __entry->group, __entry->group_lvl, __entry->child_active, + __entry->group_active, + __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry, + __entry->child_evtcpu) +); + +TRACE_EVENT(tmigr_handle_remote, + + TP_PROTO(struct tmigr_group *group), + + TP_ARGS(group), + + TP_STRUCT__entry( + __field( void * , group ) + __field( unsigned int , lvl ) + ), + + TP_fast_assign( + __entry->group = group; + __entry->lvl = group->level; + ), + + TP_printk("group=%p lvl=%d", + __entry->group, __entry->lvl) +); + +#endif /* _TRACE_TIMER_MIGRATION_H */ + +/* This part must be outside protection */ +#include diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index eab58bc98c7f..1526a2d61050 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -14,6 +14,9 @@ #include "timer_migration.h" #include "tick-internal.h" +#define CREATE_TRACE_POINTS +#include + /* * The timer migration mechanism is built on a hierarchy of groups. The * lowest level group contains CPUs, the next level groups of CPU groups @@ -479,6 +482,8 @@ static bool tmigr_active_up(struct tmigr_group *group, */ group->groupevt.ignore = 1; + trace_tmigr_group_set_cpu_active(group, newstate, childmask); + return walk_done; } @@ -509,6 +514,7 @@ void tmigr_cpu_activate(void) raw_spin_lock(&tmc->lock); tmc->idle = 0; tmc->wakeup = KTIME_MAX; + trace_tmigr_cpu_active(tmc); __tmigr_cpu_activate(tmc); raw_spin_unlock(&tmc->lock); } @@ -625,6 +631,9 @@ static bool tmigr_update_events(struct tmigr_group *group, data->nextexp = tmigr_next_groupevt_expires(group); } + trace_tmigr_update_events(child, group, data->childstate, + data->groupstate, nextexp); + unlock: raw_spin_unlock(&group->lock); @@ -668,6 +677,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp) if (tmc->remote) return KTIME_MAX; + trace_tmigr_cpu_new_timer(tmc); + tmc->cpuevt.ignore = 0; data.groupstate.state = atomic_read(&tmc->tmgroup->migr_state); @@ -770,6 +781,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group, } } + trace_tmigr_group_set_cpu_inactive(group, newstate, childmask); + return walk_done; } @@ -842,6 +855,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp) tmc->idle = 1; unlock: + trace_tmigr_cpu_idle(tmc, ret); raw_spin_unlock(&tmc->lock); return ret; } @@ -867,6 +881,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now, return next; } + trace_tmigr_handle_remote_cpu(tmc); + tmc->remote = 1; /* Drop the lock to allow the remote CPU to exit idle */ @@ -952,6 +968,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group, childmask = data->childmask; + trace_tmigr_handle_remote(group); again: /* * Handle the group only if @childmask is the migrator or if the @@ -1152,6 +1169,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node, /* Setup successful. Add it to the hierarchy */ list_add(&group->list, &tmigr_level_list[lvl]); + trace_tmigr_group_set(group); return group; } @@ -1169,6 +1187,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child, raw_spin_unlock(&parent->lock); raw_spin_unlock_irq(&child->lock); + trace_tmigr_connect_child_parent(child); + /* * To prevent inconsistent states, active children need to be * active in new parent as well. Inactive children are already @@ -1250,6 +1270,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node) raw_spin_unlock_irq(&group->lock); + trace_tmigr_connect_cpu_parent(tmc); + /* There are no children that need to be connected */ continue; } else { @@ -1318,6 +1340,7 @@ static int tmigr_cpu_online(unsigned int cpu) tmc->wakeup = KTIME_MAX; } raw_spin_lock_irq(&tmc->lock); + trace_tmigr_cpu_online(tmc); __tmigr_cpu_activate(tmc); tmc->online = 1; raw_spin_unlock_irq(&tmc->lock); @@ -1336,6 +1359,7 @@ static int tmigr_cpu_offline(unsigned int cpu) * offline; Therefore nextevt value is set to KTIME_MAX */ __tmigr_cpu_deactivate(tmc, KTIME_MAX); + trace_tmigr_cpu_offline(tmc); raw_spin_unlock_irq(&tmc->lock); return 0; From patchwork Wed May 24 07:06:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anna-Maria Behnsen X-Patchwork-Id: 98340 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2651416vqo; Wed, 24 May 2023 00:11:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6lCmg5IjcJ5JKjvNJHoZRU7/7RcQSXvbhLmR+ohtHmVEW7iJIsyFp0oMldVg8V6kWdpbHv X-Received: by 2002:a17:902:f809:b0:1ad:e19f:bb66 with SMTP id ix9-20020a170902f80900b001ade19fbb66mr13964154plb.12.1684912304454; Wed, 24 May 2023 00:11:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684912304; cv=none; d=google.com; s=arc-20160816; b=mluFYzCxTUfEUrHLMBv1PgY45dK2qS+MTVD3R1Dz+8Ie7c1Hg+3YuvMzC23aiMAip9 Qr/pe+ZfV2Lc0QPdLQYXne+A988NFAh4477C5ZPvPQwQhDvzfnpZW1wzIlRjCavQf7WA RmIdzPBsbqQNx6OdGqXSC1E/gTLooYB13UT9G9YLscdaLwf+3ilVt2ks2AQ+wheBJTcK pqezbscEDcOEPw9BAXwzVeOP8hgTe0hWfzxGwCs6JZx5WOkek1xhbtXIRmLFTTd7fq6m 3Xha3Rho3yuBPXkuC8I7LAA37dc2p3F5Y3O4hY09EioHAZufjCCV4BOJq6t3feInnTkS Ktdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=PX0GvnZSgqnY7ujdUpceDcnNQERZiJCc/4B3rZc0yHo=; b=Kj4ChQyY98aKiOMcG7RfJLPLuv5sbD8eefi63cYA5BGRtYalMMUPJrO7ZB3jlRvcoK /VNe3h7LzkHkASAyDmiblMchRts7uUtfRC/2FQTIV7dvZW7GDcu/1UwRJlQH7VE8SfXF C8cB/qUghVS6pN++NUFxKaPJrpulGRaFtaz/0al6FsDpumanLrBFZR7/1XJcatPJ523L 1R0JDmmf8tWjNL+lwc1jPRrVo8AsCE+SFp7KLJaYlh61cOCrFCgooDCnBPn1TnVUOlAN 470QaQDqVsylrzABRVT061HbVkAsbe83gBEJ7wKfMIidqZAKJuJlM/pil98VJzg5drur o5eg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=M1lPBF68; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i191-20020a6387c8000000b0051b1b5917f6si5176027pge.160.2023.05.24.00.11.31; Wed, 24 May 2023 00:11:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=M1lPBF68; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239580AbjEXHIh (ORCPT + 99 others); Wed, 24 May 2023 03:08:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236054AbjEXHHs (ORCPT ); Wed, 24 May 2023 03:07:48 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E254E4C for ; Wed, 24 May 2023 00:07:09 -0700 (PDT) From: Anna-Maria Behnsen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684912027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PX0GvnZSgqnY7ujdUpceDcnNQERZiJCc/4B3rZc0yHo=; b=M1lPBF6845MzsyeOBpjLeY1pl8JgglaBl2uHsm3bwxPuuZWmq+QSirNVsls5WMnSiNCFKG 4MqE3IHpGtJP137BrpDVIbWx8i2UxrtQjMlU2xgvHY2O8Au4kB24qHoSXdDKGdd3rma9Su NcFMLMjcAjeQA9hYFNJ8P/L+tHboXwxkIlJDQ2BENHnItXcSPspiaSpcpTbPYvWw9Plgc2 gALlUvfZeN3ocZF01n9aTw9r7OnsXqlILeqXYiwkNJbDfBxELm3mT60g51RmrhrDIFI/L+ VHSG6X03uZOkl3gHU+b8YF0aJWGxxDx9QMkv5DI7ax7bvMk7FvlLyMNMdSiM6Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684912027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PX0GvnZSgqnY7ujdUpceDcnNQERZiJCc/4B3rZc0yHo=; b=+crOMjFeguP3vUg6/IQZCce+puEbkWgVGvZRsY2g8vVwueZ4wcL0A73C7NsiRL4C2JcigC o41MRjH8uIEJcuCg== To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , John Stultz , Thomas Gleixner , Eric Dumazet , "Rafael J . Wysocki" , Arjan van de Ven , "Paul E . McKenney" , Frederic Weisbecker , Rik van Riel , Steven Rostedt , Sebastian Siewior , Giovanni Gherdovich , Lukasz Luba , "Gautham R . Shenoy" , Anna-Maria Behnsen , Richard Cochran Subject: [PATCH v7 21/21] timer: Always queue timers on the local CPU Date: Wed, 24 May 2023 09:06:29 +0200 Message-Id: <20230524070629.6377-22-anna-maria@linutronix.de> In-Reply-To: <20230524070629.6377-1-anna-maria@linutronix.de> References: <20230524070629.6377-1-anna-maria@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766758604439703407?= X-GMAIL-MSGID: =?utf-8?q?1766758604439703407?= The timer pull model is in place so we can remove the heuristics which try to guess the best target CPU at enqueue/modification time. All non pinned timers are queued on the local CPU in the separate storage and eventually pulled at expiry time to a remote CPU. Originally-by: Richard Cochran (linutronix GmbH) Signed-off-by: Anna-Maria Behnsen --- v6: - Update TIMER_PINNED flag description. v5: - Move WARN_ONCE() in add_timer_on() into a previous patch - Fold crystallball magic related hunks into this patch v4: Update comment about TIMER_PINNED flag (heristic is removed) --- include/linux/timer.h | 14 ++++---------- kernel/time/timer.c | 42 ++++++++++++++++++++---------------------- 2 files changed, 24 insertions(+), 32 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 6f96661480dd..c39ae2ad0035 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -50,16 +50,10 @@ struct timer_list { * workqueue locking issues. It's not meant for executing random crap * with interrupts disabled. Abuse is monitored! * - * @TIMER_PINNED: A pinned timer will not be affected by any timer - * placement heuristics (like, NOHZ) and will always expire on the CPU - * on which the timer was enqueued. - * - * Note: Because enqueuing of timers can migrate the timer from one - * CPU to another, pinned timers are not guaranteed to stay on the - * initialy selected CPU. They move to the CPU on which the enqueue - * function is invoked via mod_timer() or add_timer(). If the timer - * should be placed on a particular CPU, then add_timer_on() has to be - * used. + * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the + * timer was enqueued. When a particular CPU is required, add_timer_on() + * has to be used. Enqueue via mod_timer() and add_timer() is always done + * on the local CPU. */ #define TIMER_CPUMASK 0x0003FFFF #define TIMER_MIGRATING 0x00040000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 3a8c2c88f845..5602d411a70e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer) /* * We might have to IPI the remote CPU if the base is idle and the - * timer is not deferrable. If the other CPU is on the way to idle - * then it can't set base->is_idle as we hold the base lock: + * timer is pinned. If it is a non pinned timer, it is only queued + * on the remote CPU, when timer was running during queueing. Then + * everything is handled by remote CPU anyway. If the other CPU is + * on the way to idle then it can't set base->is_idle as we hold + * the base lock: */ - if (base->is_idle) + if (base->is_idle && timer->flags & TIMER_PINNED) wake_up_nohz_cpu(base->cpu); } @@ -941,17 +944,6 @@ static inline struct timer_base *get_timer_base(u32 tflags) return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK); } -static inline struct timer_base * -get_target_base(struct timer_base *base, unsigned tflags) -{ -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON) - if (static_branch_likely(&timers_migration_enabled) && - !(tflags & TIMER_PINNED)) - return get_timer_cpu_base(tflags, get_nohz_timer_target()); -#endif - return get_timer_this_cpu_base(tflags); -} - static inline void forward_timer_base(struct timer_base *base) { unsigned long jnow = READ_ONCE(jiffies); @@ -1103,7 +1095,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option if (!ret && (options & MOD_TIMER_PENDING_ONLY)) goto out_unlock; - new_base = get_target_base(base, timer->flags); + new_base = get_timer_this_cpu_base(timer->flags); if (base != new_base) { /* @@ -2202,8 +2194,14 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) */ } - /* We need to mark both bases in sync */ - base_local->is_idle = base_global->is_idle = is_idle; + /* + * base->is_idle information is required to wakeup a idle CPU when + * a new timer was enqueued. Only pinned timers could be enqueued + * remotely into a idle base. Therefore do maintain only + * base_local->is_idle information and ignore base_global->is_idle + * information. + */ + base_local->is_idle = is_idle; raw_spin_unlock(&base_global->lock); raw_spin_unlock(&base_local->lock); @@ -2233,13 +2231,13 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem) void timer_clear_idle(void) { /* - * We do this unlocked. The worst outcome is a remote enqueue sending - * a pointless IPI, but taking the lock would just make the window for - * sending the IPI a few instructions smaller for the cost of taking - * the lock in the exit from idle path. + * We do this unlocked. The worst outcome is a remote pinned timer + * enqueue sending a pointless IPI, but taking the lock would just + * make the window for sending the IPI a few instructions smaller + * for the cost of taking the lock in the exit from idle + * path. Required for BASE_LOCAL only. */ __this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false); - __this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false); /* Activate without holding the timer_base->lock */ tmigr_cpu_activate();