From patchwork Fri Apr 14 16:30:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 83515 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp511902vqo; Fri, 14 Apr 2023 09:38:07 -0700 (PDT) X-Google-Smtp-Source: AKy350YoqSPjzG/QQGaScHn1MIxT8fMFmzI4Y6KI9piJeCqObf5MNlZx4SnhGR+XO/G2DsipXxF6 X-Received: by 2002:a05:6a20:4927:b0:ea:e535:ce2c with SMTP id ft39-20020a056a20492700b000eae535ce2cmr5787699pzb.0.1681490287161; Fri, 14 Apr 2023 09:38:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681490287; cv=none; d=google.com; s=arc-20160816; b=esOQ3WYIQgEiULvJtgH34gZeSTsXdSdNlaTMLhXm/E2LFnTeaUHToT/g8q+ciBLWUQ ptWbAa1iM6OhNgjihnf1DqL7pV6q+pP5WvBDP4yRXZi1tpX+yDNawMs3qzfuktpGoFFC Mx/Y44p4R9ZMM5yYYw8olkTSOQlfjbVCC7i/TbE+Jb67FvfQo4b0pSEXo8Az8xLFHQW/ U0huXP43guVo7ZabFqr6gM/rkFxZSid8rrW6UDWeoqO3bS0wYc2toA7va3/GRJha3FiK Qcf9u0AsChunQkt4SaIte36KaHoXeYB47qL5svVhVAKBs9dOxeqkuNVZIJvbKxOYLEtU vugg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:date:mime-version:references:subject:cc:to:from :dkim-signature:dkim-signature:message-id; bh=Liy0JL0O99yuZfbRpqXKPbI/GHoH9SgVRO31sSZwo7E=; b=gAcO84SWawNJ7UQ8DrLBPINlb+ULkOfA1qGqGlYqG6Qh1PMb++z5QIZTtNr8y9Hl/P RwX8dRrniGccDUmuqtDQMRWDDwE27ew5oowZiFpRRuw73eDLUUx2ceLmh8z/elgXLsRv IFLM5/IR+H63xqQBXWV2v8HuMm9ZNNOp33stloab4GRwZ0mzM41NvnI5NWgB18nRt0Ab GpOMKJm6v4sa+5duv/sTa0zg22huBzKPQ7bAJaIFT32ND/cBKQH7XJAGGV2UlaZJ6q3e vF1awT+vX8wzFw1MBPTb1A8MUgM2FkHXwd6jnW8S7t/n5lg44Fyihyz9ewVvvwsRbQYt IINQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ijF0ruw1; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e18-20020aa79812000000b0062e23ecb18fsi4708224pfl.374.2023.04.14.09.37.51; Fri, 14 Apr 2023 09:38:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ijF0ruw1; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229996AbjDNQa4 (ORCPT + 99 others); Fri, 14 Apr 2023 12:30:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229841AbjDNQau (ORCPT ); Fri, 14 Apr 2023 12:30:50 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C07C85581 for ; Fri, 14 Apr 2023 09:30:48 -0700 (PDT) Message-ID: <20230414162841.292513270@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1681489847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Liy0JL0O99yuZfbRpqXKPbI/GHoH9SgVRO31sSZwo7E=; b=ijF0ruw1cRMf22DDY8lmd1RpMdgKqhhAKkwlN0FRiBkbd5rpJLGGXc6GIVBW2BbRxz+vrs bZINbc9E23N9wlLYUwjvi3tmXJ1x+IPt4Ms2sgxXE5oSFZGuZ9xJXM6yqt56hOoy5h+e8r IXQV2yLLh0BEcBOUq7fAsL3h3mRk5+BoQvLVLP9DC9TWBqDDBUtrbtyJOHtpVA8WVv2maC I/49Pc6G6I0AGxUSCTvpI+8aUkJHJtFsa0vzxnk8LF60uJJ14iOfxeeQi9R4xPF0/f3xQh qKas2Szjgx59CicyyVV4R+ZlbG1G+0ip+HLYkw+WSJZgNrbys3cRgU0nx79kYQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1681489847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Liy0JL0O99yuZfbRpqXKPbI/GHoH9SgVRO31sSZwo7E=; b=88gs/L+WXECo4w3u1kbaszJZORO8dhFU42usZ2pHN0I3SSSJ1k5U4lRitxe8qPvcWmWzCC 0yekjG5CpzkPrmCw== From: Thomas Gleixner To: LKML Cc: Peter Zijlstra , Valentin Schneider , Dennis Zhou , Tejun Heo , Christoph Lameter , Dave Chinner , Yury Norov , Andy Shevchenko , Rasmus Villemoes , Ye Bin , linux-mm@kvack.org Subject: [patch 3/3] cpu/hotplug: Get rid of cpu_dying_mask References: <20230414162755.281993820@linutronix.de> MIME-Version: 1.0 Date: Fri, 14 Apr 2023 18:30:46 +0200 (CEST) X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763170359607975856?= X-GMAIL-MSGID: =?utf-8?q?1763170359607975856?= The cpu_dying_mask is not only undocumented but also to some extent a misnomer. It's purpose is to capture the last direction of a cpu_up() or cpu_down() operation taking eventual rollback operations into account. The name and the lack of documentation lured already someone to use it in the wrong way. The initial user is the scheduler code which needs to keep the decision correct whether to schedule tasks on a CPU, which is between the CPUHP_ONLINE and the CPUHP_ACTIVE state and has the balance_push() hook installed. cpu_dying mask is not really useful for general consumption. The cpu_dying_mask bits are sticky even after cpu_up() or cpu_down() completes. It might be argued, that the cpu_dying_mask bit could be cleared when cpu_down() completes, but that's not possible under all circumstances. Especially not with partial hotplug operations. In that case the bit must be sticky in order to keep the initial user, i.e. the scheduler correct. Replace the cpumask completely by: - recording the direction internally in the CPU hotplug core state - exposing that state via a documented function to the scheduler After that cpu_dying_mask is not longer in use and removed before the next user trips over it. Signed-off-by: Thomas Gleixner --- include/linux/cpumask.h | 21 --------------------- kernel/cpu.c | 43 +++++++++++++++++++++++++++++++++++++------ kernel/sched/core.c | 4 ++-- kernel/smpboot.h | 2 ++ 4 files changed, 41 insertions(+), 29 deletions(-) --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -126,12 +126,10 @@ extern struct cpumask __cpu_possible_mas extern struct cpumask __cpu_online_mask; extern struct cpumask __cpu_present_mask; extern struct cpumask __cpu_active_mask; -extern struct cpumask __cpu_dying_mask; #define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask) #define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask) #define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask) #define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask) -#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) extern atomic_t __num_online_cpus; @@ -1015,15 +1013,6 @@ set_cpu_active(unsigned int cpu, bool ac cpumask_clear_cpu(cpu, &__cpu_active_mask); } -static inline void -set_cpu_dying(unsigned int cpu, bool dying) -{ - if (dying) - cpumask_set_cpu(cpu, &__cpu_dying_mask); - else - cpumask_clear_cpu(cpu, &__cpu_dying_mask); -} - /** * to_cpumask - convert an NR_CPUS bitmap to a struct cpumask * * @bitmap: the bitmap @@ -1097,11 +1086,6 @@ static inline bool cpu_active(unsigned i return cpumask_test_cpu(cpu, cpu_active_mask); } -static inline bool cpu_dying(unsigned int cpu) -{ - return cpumask_test_cpu(cpu, cpu_dying_mask); -} - #else #define num_online_cpus() 1U @@ -1129,11 +1113,6 @@ static inline bool cpu_active(unsigned i return cpu == 0; } -static inline bool cpu_dying(unsigned int cpu) -{ - return false; -} - #endif /* NR_CPUS > 1 */ #define cpu_is_offline(cpu) unlikely(!cpu_online(cpu)) --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -53,6 +53,9 @@ * @rollback: Perform a rollback * @single: Single callback invocation * @bringup: Single callback bringup or teardown selector + * @goes_down: Indicator for direction of cpu_up()/cpu_down() operations + * including eventual rollbacks. Not affected by state or + * instance add/remove operations. See cpuhp_cpu_goes_down(). * @cpu: CPU number * @node: Remote CPU node; for multi-instance, do a * single entry callback for install/remove @@ -72,6 +75,7 @@ struct cpuhp_cpu_state { bool rollback; bool single; bool bringup; + bool goes_down; struct hlist_node *node; struct hlist_node *last; enum cpuhp_state cb_state; @@ -295,6 +299,37 @@ void cpu_maps_update_done(void) mutex_unlock(&cpu_add_remove_lock); } +/** + * cpuhp_cpu_goes_down - Query the current/last CPU hotplug direction of a CPU + * @cpu: The CPU to query + * + * The direction indicator is modified by the hotplug core on + * cpu_up()/cpu_down() operations including eventual rollback operations. + * The indicator is not affected by state or instance install/remove + * operations. + * + * The indicator is sticky after the hotplug operation completes, whether + * the operation was a full up/down or just a partial bringup/teardown. + * + * goes_down + * cpu_up(target) enter -> False + * rollback on fail -> True + * cpu_up(target) exit Last state + * + * cpu_down(target) enter -> True + * rollback on fail -> False + * cpu_down(target) exit Last state + * + * The return value is a racy snapshot and not protected against concurrent + * CPU hotplug operations which modify the indicator. + * + * Returns: True if cached direction is down, false otherwise + */ +bool cpuhp_cpu_goes_down(unsigned int cpu) +{ + return data_race(per_cpu(cpuhp_state.goes_down, cpu)); +} + /* * If set, cpu_up and cpu_down will return -EBUSY and do nothing. * Should always be manipulated under cpu_add_remove_lock @@ -486,8 +521,7 @@ cpuhp_set_state(int cpu, struct cpuhp_cp st->target = target; st->single = false; st->bringup = bringup; - if (cpu_dying(cpu) != !bringup) - set_cpu_dying(cpu, !bringup); + st->goes_down = !bringup; return prev_state; } @@ -521,8 +555,7 @@ cpuhp_reset_state(int cpu, struct cpuhp_ } st->bringup = bringup; - if (cpu_dying(cpu) != !bringup) - set_cpu_dying(cpu, !bringup); + st->goes_down = !bringup; } /* Regular hotplug invocation of the AP hotplug thread */ @@ -2644,8 +2677,6 @@ EXPORT_SYMBOL(__cpu_present_mask); struct cpumask __cpu_active_mask __read_mostly; -struct cpumask __cpu_dying_mask __read_mostly; - atomic_t __num_online_cpus __read_mostly; EXPORT_SYMBOL(__num_online_cpus); --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2297,7 +2297,7 @@ static inline bool is_cpu_allowed(struct return cpu_online(cpu); /* Regular kernel threads don't get to stay during offline. */ - if (cpu_dying(cpu)) + if (cpuhp_cpu_goes_down(cpu)) return false; /* But are allowed during online. */ @@ -9344,7 +9344,7 @@ static void balance_push(struct rq *rq) * Only active while going offline and when invoked on the outgoing * CPU. */ - if (!cpu_dying(rq->cpu) || rq != this_rq()) + if (!cpuhp_cpu_goes_down(rq->cpu) || rq != this_rq()) return; /* --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -20,4 +20,6 @@ int smpboot_unpark_threads(unsigned int void __init cpuhp_threads_init(void); +bool cpuhp_cpu_goes_down(unsigned int cpu); + #endif