From patchwork Sun Apr 30 17:18:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 88931 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2251726vqo; Sun, 30 Apr 2023 10:19:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5XZy6VhVQmYBm+3KqzZTocLalg+uzpCFoiDzHRbsMa20PptTlsAZSEd/oIz7U97qw0K7dE X-Received: by 2002:a17:902:ea0d:b0:1a6:d2a9:3fab with SMTP id s13-20020a170902ea0d00b001a6d2a93fabmr13278116plg.47.1682875191746; Sun, 30 Apr 2023 10:19:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682875191; cv=none; d=google.com; s=arc-20160816; b=HHTMFdIT2oRv/Ny/MZOiJPCcZdE5ZhBamDFQXcRqH0PwTvMGQX6YR87DD6cAzLQrTq DCIxZ+2bHI4jhSm3Z2kgjXK4+REHieaa38YiQc2pURPeNv4UPIhqca9L2W8TI/AV1EFg tpjv71TBX0A1jCg0K4ycCjqZWAGymofrf+f9TUDtM/+wu3EvZHM/Rh7OHs+IbDt5nqyE BdS1PgVQSvW44yPvThTPpmzjS0MtnDhWqCQKI1ksXtW95zixPCueZ1cakvppH0NXy8iI oY8wt/lD8sqLAsacBHE2umI+JtLuUay4hgerYJ77vCAKBwMm2jtftF8FNR98quw3NWF3 s8VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oognvtBHKp+y57Kqu34zBdtUcBbI+DybK/7lI+VyXho=; b=TK7cdyBOwCsHkLWoIFlpAyTYL9Ir3RvUB8GMhahnlUKLplXkD/ltiEWnYHhqaJ9IR+ Vu0DRqVTBg8/E7Xx3FCVtEFAf5L3USYRMnI8SHYtq/R3juV7jH2T1oRQzVrnIn1AkfrH 3WuTQPjIU+vjM0WWuEqmHzeRIrGy4c3lPU/DR/lYuCliFku3nojAxy0jWgvcSJotp6dS vxuzkFpUttQ+x4rJcxZ0Q3bZxOJjkmOnekU4Z+MD0ssqmKclkNiDH5rpbx3kqkZDsEnC yLtJTbia+NGfchLP0MYTH4zkkTVXCUejsQkGNIhBhVVGQzFlCoYXHqf9U3yscWLUNdel 2KsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VQYUcls8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ik4-20020a170902ab0400b001aaf6228a75si676229plb.88.2023.04.30.10.19.37; Sun, 30 Apr 2023 10:19:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VQYUcls8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231395AbjD3RSd (ORCPT + 99 others); Sun, 30 Apr 2023 13:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231254AbjD3RSS (ORCPT ); Sun, 30 Apr 2023 13:18:18 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EDB62D73; Sun, 30 Apr 2023 10:18:17 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-63f273b219eso1216775b3a.1; Sun, 30 Apr 2023 10:18:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682875096; x=1685467096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oognvtBHKp+y57Kqu34zBdtUcBbI+DybK/7lI+VyXho=; b=VQYUcls8wAkuyKL2R95TT5lGXrflkHTfc53souXigvwJNGmbWpsrcAz4pi4GSaPuur wdDtcgnTPGvqPx+6s7uTkh6gRuYtMZ/EI127bESMIKNKI7i24bA1tyQ1RMe8HDJBgPL1 rCYcAld1i3y6/HG/J3CjVlHxUONZEjcVrJXPsdthyCF02WvUUT4v/Szv/2mluK6Tm3jc dLpTt1jEfV772jz+nT1OwNisbCpmt7iWP+JnlIRmceZzfsgPssiW3my3Obt62bRFA0v4 ZU/z79gda1Kxoijl/KMMb4Z8pnLhA1RPo096Kv3M1P/SVBo9R09suN0KfHmz0HIqZPNJ sgUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682875096; x=1685467096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oognvtBHKp+y57Kqu34zBdtUcBbI+DybK/7lI+VyXho=; b=RAsqQdCeBAnxOy/5gJr/LVqwDJCrMNZauVIBwtCMOSCZJWGgOVo0Ruj5oj9llgQNye kizsL6ZJGwsckODqMdXbAepL8bG5KqPrMkpvIOGVPdF1dVjLwHYAC035+tFoKxMsAO1r aLWCfP/c967OW5opZvgtDqFApXdLk7RuOiqll2aV4JmecTPUU5UBpEPcfpTZ8HfxkaZ+ BuORfNZznRZMxJtY/qsObPswisH+l3SQTHR05n7RoyQOzKHvQJ6uXGCjZ8QPNx/oxmG8 HseK1HvW7Rcmi12Qy3KWztqF6kiPnJsT7YMPXuIA3SNjz02+f4E1NOeQRnr4VizZ5G84 O+tQ== X-Gm-Message-State: AC+VfDzj15/W3xwoArRE6nc7LtyJZqTWf3diif+5TcZXmgTbDGmi8bYN xL5NwMnlmk5pE25bOUQ5U7E= X-Received: by 2002:a05:6a00:c88:b0:63b:1e3b:aa02 with SMTP id a8-20020a056a000c8800b0063b1e3baa02mr16689872pfv.16.1682875096461; Sun, 30 Apr 2023 10:18:16 -0700 (PDT) Received: from localhost ([4.1.102.3]) by smtp.gmail.com with ESMTPSA id f14-20020a056a00238e00b00640e64aa9b7sm11086045pfc.10.2023.04.30.10.18.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Apr 2023 10:18:16 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v3 3/8] sched/topology: introduce sched_numa_find_next_cpu() Date: Sun, 30 Apr 2023 10:18:04 -0700 Message-Id: <20230430171809.124686-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230430171809.124686-1-yury.norov@gmail.com> References: <20230430171809.124686-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764622536778250151?= X-GMAIL-MSGID: =?utf-8?q?1764622536778250151?= The function searches for a next CPU in a given cpumask according to NUMA topology, so that it traverses CPUs per-hop. If the CPU is the last CPU in a given hop, sched_numa_find_next_cpu() switches to the next hop, and picks the first CPU from there, excluding those already traversed. Because only online CPUs are presented in the NUMA topology masks, offline CPUs will be skipped even if presented in the 'cpus' mask provided in the arguments. Signed-off-by: Yury Norov --- include/linux/topology.h | 12 ++++++++++++ kernel/sched/topology.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 52f5850730b3..da92fea38585 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -245,8 +245,13 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) return cpumask_of_node(cpu_to_node(cpu)); } +/* + * sched_numa_find_*_cpu() functions family traverses only accessible CPUs, + * i.e. those listed in cpu_online_mask. + */ #ifdef CONFIG_NUMA int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop); extern const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops); #else static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) @@ -254,6 +259,13 @@ static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, i return cpumask_nth_and(cpu, cpus, cpu_online_mask); } +static __always_inline +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop) +{ + return find_next_and_bit(cpumask_bits(cpus), cpumask_bits(cpu_online_mask), + small_cpumask_bits, cpu); +} + static inline const struct cpumask * sched_numa_hop_mask(unsigned int node, unsigned int hops) { diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 051aaf65c749..fc163e4181e6 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2130,6 +2130,45 @@ int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) } EXPORT_SYMBOL_GPL(sched_numa_find_nth_cpu); +/* + * sched_numa_find_next_cpu() - given the NUMA topology, find the next cpu + * cpumask: cpumask to find a CPU from + * cpu: current CPU + * node: local node + * hop: (in/out) indicates distance order of current CPU to a local node + * + * The function searches for a next CPU at a given NUMA distance, indicated + * by hop, and if nothing found, tries to find CPUs at a greater distance, + * starting from the beginning. + * + * Return: cpu, or >= nr_cpu_ids when nothing found. + */ +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop) +{ + unsigned long *cur, *prev; + struct cpumask ***masks; + unsigned int ret; + + if (*hop >= sched_domains_numa_levels) + return nr_cpu_ids; + + masks = rcu_dereference(sched_domains_numa_masks); + cur = cpumask_bits(masks[*hop][node]); + if (*hop == 0) + ret = find_next_and_bit(cpumask_bits(cpus), cur, nr_cpu_ids, cpu); + else { + prev = cpumask_bits(masks[*hop - 1][node]); + ret = find_next_and_andnot_bit(cpumask_bits(cpus), cur, prev, nr_cpu_ids, cpu); + } + + if (ret < nr_cpu_ids) + return ret; + + *hop += 1; + return sched_numa_find_next_cpu(cpus, 0, node, hop); +} +EXPORT_SYMBOL_GPL(sched_numa_find_next_cpu); + /** * sched_numa_hop_mask() - Get the cpumask of CPUs at most @hops hops away from * @node