From patchwork Sat Jan 21 04:24:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46767 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp566993wrn; Fri, 20 Jan 2023 20:26:14 -0800 (PST) X-Google-Smtp-Source: AMrXdXuuQrt2mpqgJS9FIQuYuKfF8g1w/yebYGf/TY/+isOALXaWPCwEoPn6InUlkmoItkEf08Jp X-Received: by 2002:a05:6402:5110:b0:49d:32d0:126 with SMTP id m16-20020a056402511000b0049d32d00126mr21669156edd.20.1674275174275; Fri, 20 Jan 2023 20:26:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275174; cv=none; d=google.com; s=arc-20160816; b=lG2IdxsehvkUxtP7W+80UegtSsw1C2RCPlDFnYE1LHWfWwPw/kEb9ggtzhy+6cjq6T gDK1p2mw66ZZcmeypIYfDvaRuVVqncLDdDjDWnII76M4ZyX1faKv/UOixdXM3XT8GGOu hG8ClLF80wWw+/0wwkJPFeLavxfJXQHNXYN8IBtdgib4b3iMeAQtLt++ADPGpe4Qu/a5 S5Z/Dc51mge2nLvJW1mbRxXxHA5lP2g32G9vTqtsihuHc/pMX4E/sNz3bVu1MPfZ67JC H3ZmS2CA30K6zFXoqxAWyv5Aes2SlQTITwxjBDYPLw502gOuavJF/Kk++xo8mJoy/m5y W8qQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5rRltMfhDWKXvHDAwG9bDPvvlFe2Ak9PFTADY/ayoZE=; b=ytZXutS+HYV2IdZ4Kld30TE4TFpCFJwiDmAVrZejcKEqGML44g17RJdVHuIUqNWqm1 s7RSvxc06lVDsCPDqGXCYqOp3qUp36/njFv557sCQHnix5yDRY6iqIk996TnXHnzRHo3 caJ+srnGnciSGs1zxnzbuNq3hWgi+kqbk3nPQ8OgICAvsYUNZoPH70di1DYI7RddY3sq GEkzplcbNQNs2By32gDr+Yk2KvIQL3cfvxyFPr2R3K0IDCt0Z7xtbzZ2TW0q5qqKb4Kv gT+GjDLpJO5JW+LgKI7G9iuPlVScPh4WNfzASxZw9uvAhZERDSL6IIkKMfHHykB162IM eMGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Hqx6+XfD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l2-20020a056402230200b004763231785asi20701539eda.33.2023.01.20.20.25.51; Fri, 20 Jan 2023 20:26:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Hqx6+XfD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229861AbjAUEYr (ORCPT + 99 others); Fri, 20 Jan 2023 23:24:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229725AbjAUEYn (ORCPT ); Fri, 20 Jan 2023 23:24:43 -0500 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C7334FCF3; Fri, 20 Jan 2023 20:24:42 -0800 (PST) Received: by mail-qt1-x831.google.com with SMTP id z9so5823690qtv.5; Fri, 20 Jan 2023 20:24:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5rRltMfhDWKXvHDAwG9bDPvvlFe2Ak9PFTADY/ayoZE=; b=Hqx6+XfD4UPfIWr7eVxTnX3tc+uR9jA/tGxrrhZNSmLhVj8hDNmbBW76/zMN5rnNAu jlVoL3E9ZCIfQMCoBEtIxzbiIRxzJ1xPeuJ/o5vX72mFdCHvVzUl6hp1tBSfPDBABDVH X1MDJTc8wGTV7ORNkqW2r6tBW5Y0X6DqnqRoXWWit+x0xAV9YeZhgDA3bQPjrsTjMnv5 689J0OJSL8ORGY6UUnSETPLmokei/9guW90WlLC/puz3qiBKbE3rBPFUQQ+XEgY0lpZw m8SZLZ5Xg338OcayvpRdjjYHpdEiufmJvwad0Seulk6XkSKYi9gdod6KRWS14F+zYANn WymA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5rRltMfhDWKXvHDAwG9bDPvvlFe2Ak9PFTADY/ayoZE=; b=G98wp82AXWtljGtd/Q0OwYkOSQTb/8U7NOLtRQ0HkIz2ld+LQK6hZOBqunUGHAahET fKUt6LaO/RIGHCOHhcapsWnHsGvvx6RTsK2tZ1Kem7xE9EJGcf7HYWa3opgKDpU/+9XI +g9RUqfSyw0syny454CE5/V/Pd45Nup9t8R0RW2hg+Y1Wm7qOdJyNNRHP01RVW/N8y4O 2Uxt1TbRZj17cjJZj/MhSSZguRbGxTO4X5tSsTtvlJiN9u15IBVMP87pZHbnzAxsJyJR r0lo0uB1n5uor0VpxQM1eD1iG5p8WMImnv2k9enda4KVs839QS1qz9lXUZ338OCdrTvi AsiA== X-Gm-Message-State: AFqh2kqXUjFIkbm/SG+dEWcy9JalyUj6FNRwhaF0MOdYsRmb1NMpDZx6 PbN45AQxJSMkRl1jKJYGDJnyIQn64Y8= X-Received: by 2002:ac8:5607:0:b0:3b6:2fcd:6d3e with SMTP id 7-20020ac85607000000b003b62fcd6d3emr31392866qtr.33.1674275080774; Fri, 20 Jan 2023 20:24:40 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id z24-20020ac875d8000000b003b68c7aeebfsm3807547qtq.3.2023.01.20.20.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:40 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 1/9] lib/find: introduce find_nth_and_andnot_bit Date: Fri, 20 Jan 2023 20:24:28 -0800 Message-Id: <20230121042436.2661843-2-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604764786331195?= X-GMAIL-MSGID: =?utf-8?q?1755604764786331195?= In the following patches the function is used to implement in-place bitmaps traversing without storing intermediate result in temporary bitmaps. Signed-off-by: Yury Norov Acked-by: Tariq Toukan Reviewed-by: Jacob Keller Reviewed-by: Peter Lafreniere --- include/linux/find.h | 33 +++++++++++++++++++++++++++++++++ lib/find_bit.c | 9 +++++++++ 2 files changed, 42 insertions(+) diff --git a/include/linux/find.h b/include/linux/find.h index ccaf61a0f5fd..4647864a5ffd 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -22,6 +22,9 @@ unsigned long __find_nth_and_bit(const unsigned long *addr1, const unsigned long unsigned long size, unsigned long n); unsigned long __find_nth_andnot_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long size, unsigned long n); +unsigned long __find_nth_and_andnot_bit(const unsigned long *addr1, const unsigned long *addr2, + const unsigned long *addr3, unsigned long size, + unsigned long n); extern unsigned long _find_first_and_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long size); extern unsigned long _find_first_zero_bit(const unsigned long *addr, unsigned long size); @@ -255,6 +258,36 @@ unsigned long find_nth_andnot_bit(const unsigned long *addr1, const unsigned lon return __find_nth_andnot_bit(addr1, addr2, size, n); } +/** + * find_nth_and_andnot_bit - find N'th set bit in 2 memory regions, + * excluding those set in 3rd region + * @addr1: The 1st address to start the search at + * @addr2: The 2nd address to start the search at + * @addr3: The 3rd address to start the search at + * @size: The maximum number of bits to search + * @n: The number of set bit, which position is needed, counting from 0 + * + * Returns the bit number of the N'th set bit. + * If no such, returns @size. + */ +static __always_inline +unsigned long find_nth_and_andnot_bit(const unsigned long *addr1, + const unsigned long *addr2, + const unsigned long *addr3, + unsigned long size, unsigned long n) +{ + if (n >= size) + return size; + + if (small_const_nbits(size)) { + unsigned long val = *addr1 & *addr2 & (~*addr3) & GENMASK(size - 1, 0); + + return val ? fns(val, n) : size; + } + + return __find_nth_and_andnot_bit(addr1, addr2, addr3, size, n); +} + #ifndef find_first_and_bit /** * find_first_and_bit - find the first set bit in both memory regions diff --git a/lib/find_bit.c b/lib/find_bit.c index 18bc0a7ac8ee..c10920e66788 100644 --- a/lib/find_bit.c +++ b/lib/find_bit.c @@ -155,6 +155,15 @@ unsigned long __find_nth_andnot_bit(const unsigned long *addr1, const unsigned l } EXPORT_SYMBOL(__find_nth_andnot_bit); +unsigned long __find_nth_and_andnot_bit(const unsigned long *addr1, + const unsigned long *addr2, + const unsigned long *addr3, + unsigned long size, unsigned long n) +{ + return FIND_NTH_BIT(addr1[idx] & addr2[idx] & ~addr3[idx], size, n); +} +EXPORT_SYMBOL(__find_nth_and_andnot_bit); + #ifndef find_next_and_bit unsigned long _find_next_and_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, unsigned long start) From patchwork Sat Jan 21 04:24:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46768 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567086wrn; Fri, 20 Jan 2023 20:26:36 -0800 (PST) X-Google-Smtp-Source: AMrXdXt2ofYxPIV2NBZTQXv/IdjSQlTK6aTfBnRgjHSQJUVo5LH+1tnFU+8YOsZWOyzy+DQzIYte X-Received: by 2002:aa7:cd89:0:b0:49d:14cf:5f4a with SMTP id x9-20020aa7cd89000000b0049d14cf5f4amr17606320edv.39.1674275196750; Fri, 20 Jan 2023 20:26:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275196; cv=none; d=google.com; s=arc-20160816; b=PoDxGz5ORsRBktYRueZBtzXrkYaks80MKQL6e6gqRAnkItouqKaf1DSkHFFSc++O0y plAyZQoYQkGBphJ6twuQNDy1sWmAKDjtCo+y6nvjbikyPeB4QDB+FecAW+OUUMrk6Fgu 5hJni6OpkKtjscOTU1iJmU2M7Pl4kmzbbEP/Bl8Vjq9vbYyZAwuR8El4HhguN4EFEzih sfImHBfjzsbeGC15ALtT9EecPNmd2fF7Z/JGEqHARcyNgQPxmPfm4Jb09njjZGryZadj W8sC5GTI1cUD/MpiodT+Bl/OETiI6o97e32NAYDVT/C3yx1FVVI+Al4tZv6bl9RUp/3k llbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WJDoNgHHINrCbuwq1iSTb+uK/rqsM0fqmFaN0vHyvak=; b=Oepw9UC14V6f7kjQJyv/MG9AXoGl6hGoZ9cipaBLU9EjmerLZhKlKIm8pTDbB9rkPF 1j65+p+htSXrnYH/SwN6YwVQuLqaOE1fcpG7m4hlJfwu9qlS5dt+dVPxBzxfmFaQtkkG mbkjr+PNhErd9Ojn7lEqiQWHGLp2JeKOaY7mBYNAD3Q8OVoJo+pVjLnOUAmqNkYFqzAV 9Wze2Vx/yaVmOi0VPgZzbojZ9jjpiezw1MZxWOB55NXZScsX42G5tMC5umco3ElkJsnD sIl4PM5HDn8X+loZHS/X5SSLgK0CDOVfJwFANQ4knlEwxUFNhM7Z/0l0o7FUoHhTz3Cx nGgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Jqhz0cRY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dk10-20020a0564021d8a00b0049e0440d49esi13968628edb.29.2023.01.20.20.26.13; Fri, 20 Jan 2023 20:26:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Jqhz0cRY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229907AbjAUEZL (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229484AbjAUEYq (ORCPT ); Fri, 20 Jan 2023 23:24:46 -0500 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B290D4FCF8; Fri, 20 Jan 2023 20:24:43 -0800 (PST) Received: by mail-qv1-xf2d.google.com with SMTP id j9so5243825qvt.0; Fri, 20 Jan 2023 20:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WJDoNgHHINrCbuwq1iSTb+uK/rqsM0fqmFaN0vHyvak=; b=Jqhz0cRYfCQpS/zjF90KMvOTK02E3o9qpdRZlGwEW4cocE3H5NsKl2zJaBwazrc3XK sPBzcj/azPDRhd6MJnoywMNrFUUwuBBNUI+rnM2CL4aNLGSv0W/mmmuuzOE05/dVpMUS 1m0NfxS72zjlo5IinxUwwd1vpCR/Wj5yp/cyRLpFCP6NwBt2VdtftCn7Nzxa7UBOpM4z Nby+cBRpwvkBTsi0rhpJrcsiEF1izLkf1yDoDwk3gm7ElZSla8KKA+QIGGwKloBSaUBL NxbYs0DTgbqoxFuxM8e+Ut8VsVXH/yzsJjrgb0ytgZ5pqUIEWhO/a7mzI6wSNUDxAJDC tvkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WJDoNgHHINrCbuwq1iSTb+uK/rqsM0fqmFaN0vHyvak=; b=4gxf498YG70mfiEPI90ty1Nm0pB5/iDSF508uVE6EYFnIG3HxtLr5ZPx2hVhd9jCDY UWC+0K4qLAlk0eCYQD/33XWY57aK1CsQUleKkY/xMAxu1GePTqVqKreUluJfCRkamxSH xyujwQWJVD/fUDdqEpXg4ullbIx5Jzfl6KZ2lIvM2jzDtbk0XiFyysD3lsvzI2ME6LRr sVu27LA2GAjwx1Fh6ehsGaj91IKVkRrNkOBi45k1NSftcbGiNTF7dy6uZ2VrJ06UYd3x Zz+5aqaVx/IBxatVkUEe2EmxmW+9xErcYnpOmtwwPR9XikuKwv6IZCec/JC2n+tQL4Tq lHbA== X-Gm-Message-State: AFqh2kpv7w/aIhpJ/5t9ZmZf6++2Pf1VLBS8yX6wzG9N2tiP22UTrTJE BB0+x2erNucI0nHFr0fv073mpSpzh0Q= X-Received: by 2002:a0c:e8c4:0:b0:534:7cf3:44b4 with SMTP id m4-20020a0ce8c4000000b005347cf344b4mr24906622qvo.50.1674275082339; Fri, 20 Jan 2023 20:24:42 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id c5-20020a05620a268500b006e8f8ca8287sm22782385qkp.120.2023.01.20.20.24.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:41 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 2/9] cpumask: introduce cpumask_nth_and_andnot Date: Fri, 20 Jan 2023 20:24:29 -0800 Message-Id: <20230121042436.2661843-3-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604788583229228?= X-GMAIL-MSGID: =?utf-8?q?1755604788583229228?= Introduce cpumask_nth_and_andnot() based on find_nth_and_andnot_bit(). It's used in the following patch to traverse cpumasks without storing intermediate result in temporary cpumask. Signed-off-by: Yury Norov Acked-by: Tariq Toukan Reviewed-by: Jacob Keller Reviewed-by: Peter Lafreniere --- include/linux/cpumask.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index c2aa0aa26b45..7b16aede7ac5 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -391,6 +391,26 @@ unsigned int cpumask_nth_andnot(unsigned int cpu, const struct cpumask *srcp1, nr_cpumask_bits, cpumask_check(cpu)); } +/** + * cpumask_nth_and_andnot - get the Nth cpu set in 1st and 2nd cpumask, and clear in 3rd. + * @srcp1: the cpumask pointer + * @srcp2: the cpumask pointer + * @srcp3: the cpumask pointer + * @cpu: the N'th cpu to find, starting from 0 + * + * Returns >= nr_cpu_ids if such cpu doesn't exist. + */ +static __always_inline +unsigned int cpumask_nth_and_andnot(unsigned int cpu, const struct cpumask *srcp1, + const struct cpumask *srcp2, + const struct cpumask *srcp3) +{ + return find_nth_and_andnot_bit(cpumask_bits(srcp1), + cpumask_bits(srcp2), + cpumask_bits(srcp3), + nr_cpumask_bits, cpumask_check(cpu)); +} + #define CPU_BITS_NONE \ { \ [0 ... BITS_TO_LONGS(NR_CPUS)-1] = 0UL \ From patchwork Sat Jan 21 04:24:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46769 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567096wrn; Fri, 20 Jan 2023 20:26:42 -0800 (PST) X-Google-Smtp-Source: AMrXdXtgLOtaCqkaD7t5DHYQYkw5ieO1J8d/B3zU4X3NGnfnupZYqU7LrPmUTwz33IoH5O+mKjf+ X-Received: by 2002:a05:6402:2419:b0:49d:2a42:b8c8 with SMTP id t25-20020a056402241900b0049d2a42b8c8mr19940149eda.26.1674275201946; Fri, 20 Jan 2023 20:26:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275201; cv=none; d=google.com; s=arc-20160816; b=OKibWcvnLq6A6V7gkvYOAidQrhGCtzFxEgDUq3hcz0dh2MmBJClUO8URuVCEub5KdS CJKr6fCPdx0vWFGifXKIHqVutW+y/x+v6tXIAzDWhIa7zaQ9grO124vZUdj4rQVjoEGI IgUlj9OJQqM+wYQms6HxpWEn7dpWeCd8IDBWieS/+RqHJoxvLLHVoH0/0nFaGlfEz31i bfBlcAxNZGCWlXSyY2Kyj11XGcXM4sdgc6w+iUzSETEuE0FzJHi4XYI1pt0Nctt2xcNt pz7il4cNQZjlNFryfUu7/8WlytiTptj1yvf2wsEuD8pdx73nlfe3DRXR/g1Ol6A9IzWU y90Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DUfnaBP5lJbCL0GNiMEf+7JU9t1M5MfdDFKLQwINrqk=; b=uLDLa1zofcbmlLGcC0WUi2igmXyJJ0nqgCAGqlx8Mkk7ix7YW2V33HhsNojIk6ahgW hy7u2WAcQXdvp7vR48WUpD1Kec3ASwgr4kZ2PJa87sQXg4GmwrPYin0Dc69rylFOdcgg IvEAYjYY6So736gxrCbznZ0XYMBShno3pJq/B+0QNUxigGkS5QgorMzg6oonAYI8B26K Fy0//TTGqo/R5umtP7D6c9d59TwqXp31hZiVWqAOWm5qoThStR6nKNn5T7q7Hf2gk6+g a1ev0qN6evwxUvoUK2czMcT7IiKIdoDXouOywAJBvy3fOXMLA53wOJ5pquLYtnRuaWN3 izZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=JI2Z5SzD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a5-20020aa7d905000000b0049e6a68b509si6590219edr.631.2023.01.20.20.26.18; Fri, 20 Jan 2023 20:26:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=JI2Z5SzD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229604AbjAUEZQ (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229823AbjAUEYq (ORCPT ); Fri, 20 Jan 2023 23:24:46 -0500 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1669150843; Fri, 20 Jan 2023 20:24:45 -0800 (PST) Received: by mail-qt1-x831.google.com with SMTP id g16so3765698qtu.2; Fri, 20 Jan 2023 20:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DUfnaBP5lJbCL0GNiMEf+7JU9t1M5MfdDFKLQwINrqk=; b=JI2Z5SzD2aaTHlQx9e0J54AC8wdh7+DH+cR5sUi/bLrniSWMzxwFF7NL5awZ/lhqaF W5jiQGZPn58mj5hutiwKOHq+LBNCi3NWVqvfhK+3rTeu5C5/1Sebepnnbx7jhWShj2L8 zzY9N1e4mskl2NwiPSoUBlqA7gfWm/TlWIwTfsJtizaEtNyHPHhkm8A81dYUiOsGmTVA aJnEodZO/UHF55Aaa8Ya6ORBgPtXdVloQUbalUekikBiJJMmpUtLNPrT2zxW5KqTuEjD fue8rw06THQRNzmYOYEOO8Z0xtSgNrkxC98hJC5KZKNjpcsI8iJw8FdndTEjx9FiRTit G/oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DUfnaBP5lJbCL0GNiMEf+7JU9t1M5MfdDFKLQwINrqk=; b=jCrYIenIYx430tbWCM7t/J9vZUl2+XZkLayYMZkDv9y5nejMWwrFgSxTtJ/i8nOwDG Y6TsjIelcVo7jKEx4jT3uI6eDrNbJ0tyrEfOwvrnsUQYHAiB/KVyYAl9dcluiaBblDve efD9pQYiW5TgumNKsUFkbKI+bGGLfUMyBo2oVFNc/SBUmehuGD7gaich50+KUtZYL8EW M4fczhOf+QriQ6EzZp30Jse26+txG5MzcbVodGzWZj9b6ItN4hhjvTvdUvoXBcB3d88S 0oP194nKQlza9aydUk41N9BxwNFeMF1uL16Jg0MDHmGPB1sp0jBd7jy7gFIDvZ2m4a+7 j3Ow== X-Gm-Message-State: AFqh2kpzi8dRu5pjz0p8aJpps/rnVzepcviFAUGPhDXA8k/esbLbxpky ZsOnAp6zVCVJXSLW2Bu+k6zexCu+lck= X-Received: by 2002:ac8:6bd2:0:b0:3b2:2195:e2a2 with SMTP id b18-20020ac86bd2000000b003b22195e2a2mr23673991qtt.45.1674275083683; Fri, 20 Jan 2023 20:24:43 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id fe13-20020a05622a4d4d00b003a580cd979asm21689174qtb.58.2023.01.20.20.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:43 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 3/9] sched: add sched_numa_find_nth_cpu() Date: Fri, 20 Jan 2023 20:24:30 -0800 Message-Id: <20230121042436.2661843-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604794190910539?= X-GMAIL-MSGID: =?utf-8?q?1755604794190910539?= The function finds Nth set CPU in a given cpumask starting from a given node. Leveraging the fact that each hop in sched_domains_numa_masks includes the same or greater number of CPUs than the previous one, we can use binary search on hops instead of linear walk, which makes the overall complexity of O(log n) in terms of number of cpumask_weight() calls. Signed-off-by: Yury Norov Acked-by: Tariq Toukan Reviewed-by: Jacob Keller Reviewed-by: Peter Lafreniere Reported-by: Bruno Goncalves Signed-off-by: Yury Norov --- include/linux/topology.h | 8 ++++++ kernel/sched/topology.c | 57 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 4564faafd0e1..72f264575698 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -245,5 +245,13 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) return cpumask_of_node(cpu_to_node(cpu)); } +#ifdef CONFIG_NUMA +int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); +#else +static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) +{ + return cpumask_nth(cpu, cpus); +} +#endif /* CONFIG_NUMA */ #endif /* _LINUX_TOPOLOGY_H */ diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 8739c2a5a54e..2bf89186a10f 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -3,6 +3,8 @@ * Scheduler topology setup/handling methods */ +#include + DEFINE_MUTEX(sched_domains_mutex); /* Protected by sched_domains_mutex: */ @@ -2067,6 +2069,61 @@ int sched_numa_find_closest(const struct cpumask *cpus, int cpu) return found; } +struct __cmp_key { + const struct cpumask *cpus; + struct cpumask ***masks; + int node; + int cpu; + int w; +}; + +static int hop_cmp(const void *a, const void *b) +{ + struct cpumask **prev_hop = *((struct cpumask ***)b - 1); + struct cpumask **cur_hop = *(struct cpumask ***)b; + struct __cmp_key *k = (struct __cmp_key *)a; + + if (cpumask_weight_and(k->cpus, cur_hop[k->node]) <= k->cpu) + return 1; + + k->w = (b == k->masks) ? 0 : cpumask_weight_and(k->cpus, prev_hop[k->node]); + if (k->w <= k->cpu) + return 0; + + return -1; +} + +/* + * sched_numa_find_nth_cpu() - given the NUMA topology, find the Nth next cpu + * closest to @cpu from @cpumask. + * cpumask: cpumask to find a cpu from + * cpu: Nth cpu to find + * + * returns: cpu, or nr_cpu_ids when nothing found. + */ +int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) +{ + struct __cmp_key k = { .cpus = cpus, .node = node, .cpu = cpu }; + struct cpumask ***hop_masks; + int hop, ret = nr_cpu_ids; + + rcu_read_lock(); + + k.masks = rcu_dereference(sched_domains_numa_masks); + if (!k.masks) + goto unlock; + + hop_masks = bsearch(&k, k.masks, sched_domains_numa_levels, sizeof(k.masks[0]), hop_cmp); + hop = hop_masks - k.masks; + + ret = hop ? + cpumask_nth_and_andnot(cpu - k.w, cpus, k.masks[hop][node], k.masks[hop-1][node]) : + cpumask_nth_and(cpu, cpus, k.masks[0][node]); +unlock: + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL_GPL(sched_numa_find_nth_cpu); #endif /* CONFIG_NUMA */ static int __sdt_alloc(const struct cpumask *cpu_map) From patchwork Sat Jan 21 04:24:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46775 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567533wrn; Fri, 20 Jan 2023 20:28:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXuSBWQK6toiZli9KWKo9Q9PxdS1pdB+13znyBargCpEkGmnjUW1Mx/hYwwvZxk3YP1GLBcU X-Received: by 2002:a17:906:60d0:b0:877:612e:516e with SMTP id f16-20020a17090660d000b00877612e516emr38542551ejk.61.1674275306973; Fri, 20 Jan 2023 20:28:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275306; cv=none; d=google.com; s=arc-20160816; b=AwfX5BDPW/GlPYgODDN3MldK2C48RptC4zo8wLAI69aqw2xLv2uHC/k+mmqpoVMXCa USM81YlOpoeXXWjnfoeMwGghgIkMCrATaJXpF7KtZgWzNUqOozYK5QUZc142+LL1Zc4C KIrDNKetG0yVMydX+dhBZHClihqSLkmbWofQ14HyAam96LACgtHzM3n523HujcqrwN3t N4/8enKd8U5WoYrokYjtNV0e5UiZeXJnE9BHbcYL2ukEZElpbxygaz3dS+BttgCQtC2G +1Re1DIiGygvKW7U4OKSLMeyOKgroQgZ7UVVEftcQszxQbf497EGS1GbH6g3Mqg0+3zU 3FBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PJerbQOhdlxy484c709V9ypFLveYN3mWM3Ilqj2sA+c=; b=ix+Vtb7sU8OsLJywOCg3o14oxYM3TrRSDPFcR6cnVGE8YXMIez49rjxoM1KrskCWhw +S/SmUUTQK7hkv/OEuiC6JHXmRNe55mZLg22qVyh81iV+gXfTffWrGDi0JsKMzFqVY+d A+3BIina7aS6p2cNNDa11Y+qZj5l66YHMi3FOHQ4ON5rH63Uanmt4XYNKrl2ZK3THcbq GavunKt8gNB5R/JmF4LkGhPO+lMMx8ttYNNneWBrfl+G6Rk+jNzV1KfGg88qQ4rZeehm jOuuz1DADJCMgwF9xfjSQBaWtCJFoKWSvLFlNMuetK7hdKSWmbUywUo3W1dOypoazo7i fXKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=h1Zx6Bhj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xe6-20020a170907318600b00780488c11bbsi47860432ejb.388.2023.01.20.20.28.03; Fri, 20 Jan 2023 20:28:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=h1Zx6Bhj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229690AbjAUEZW (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229853AbjAUEYr (ORCPT ); Fri, 20 Jan 2023 23:24:47 -0500 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 665E64FCFF; Fri, 20 Jan 2023 20:24:46 -0800 (PST) Received: by mail-qv1-xf2a.google.com with SMTP id i12so5213778qvs.2; Fri, 20 Jan 2023 20:24:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PJerbQOhdlxy484c709V9ypFLveYN3mWM3Ilqj2sA+c=; b=h1Zx6Bhj5dIG43PbAkkOc/sx/Fw4MABjfDO4NdVNETxZjrABFVi6CcIhSBHShTZ1ey 7xxCZ7kV52SFvmCDe24u2JxXV88gSDlATevMNCebe9/fGgVnlqVu2wvZxvcf6pkIxFCP Yg07TUTUpDY70BypZSjPbfsQfvOYVWOipY5iIChJ9za68UvhDPPrlG5eYi0Czepch6JV uAl7svrTw2xooaVcTHmVUrgnseoJyn7NbHCekN0aw0H0QteMyTj2fyS/fVVo1U2ce1iC QTrinnJz35pOkMDhXv0YTmwS41BJGrL2TTpQo1T/++7hFZhbVhNO7VYLsxMXjNx2aNYL Ilbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PJerbQOhdlxy484c709V9ypFLveYN3mWM3Ilqj2sA+c=; b=WHMjiHKMjkx0Dm+w1AdWr7egZKmQCDcoehK/7bhrxDyHI5LmE+sJn0LEJRhO0TUB2c Em+aBH0BazK9hilETidP4QthwMok+9HY8cmdzyTSD20R1zA/9eiTn0KjWy+ggDf23FtH 0oEMmgOWPgVyAlR6GUfkcD7uJErrL+fj0sJGxhPDZbDIrTLREx8BNNZVMOu1mUoe4+ki oI6Wq1WQ+zj+BSk+4c2gqNCrT5WVT/3E99ea0AfJZbO9BSpyJbdBySIMmTzrfunqfLxr 2s/KwbNT9nz0qc+u/E5f8I76Wlmy88xBft31AqzTtY6/1tZrk38gsbypq4pv+zzlV9Cr oYjg== X-Gm-Message-State: AFqh2kq+ceSsP7Bb6N/kSkOL0GD9h0bzajoZqH3km7r3xZyZE+oid70x ORm+ouQ4exC5jCwV9shaLtbplSNPaSc= X-Received: by 2002:a05:6214:57d1:b0:532:1f17:2ce7 with SMTP id lw17-20020a05621457d100b005321f172ce7mr26333198qvb.16.1674275085080; Fri, 20 Jan 2023 20:24:45 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id h10-20020a05620a284a00b006feea093006sm27597979qkp.124.2023.01.20.20.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:44 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 4/9] cpumask: improve on cpumask_local_spread() locality Date: Fri, 20 Jan 2023 20:24:31 -0800 Message-Id: <20230121042436.2661843-5-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604904288254631?= X-GMAIL-MSGID: =?utf-8?q?1755604904288254631?= Switch cpumask_local_spread() to use newly added sched_numa_find_nth_cpu(), which takes into account distances to each node in the system. For the following NUMA configuration: root@debian:~# numactl -H available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 node 0 size: 3869 MB node 0 free: 3740 MB node 1 cpus: 4 5 node 1 size: 1969 MB node 1 free: 1937 MB node 2 cpus: 6 7 node 2 size: 1967 MB node 2 free: 1873 MB node 3 cpus: 8 9 10 11 12 13 14 15 node 3 size: 7842 MB node 3 free: 7723 MB node distances: node 0 1 2 3 0: 10 50 30 70 1: 50 10 70 30 2: 30 70 10 50 3: 70 30 50 10 The new cpumask_local_spread() traverses cpus for each node like this: node 0: 0 1 2 3 6 7 4 5 8 9 10 11 12 13 14 15 node 1: 4 5 8 9 10 11 12 13 14 15 0 1 2 3 6 7 node 2: 6 7 0 1 2 3 8 9 10 11 12 13 14 15 4 5 node 3: 8 9 10 11 12 13 14 15 4 5 6 7 0 1 2 3 Signed-off-by: Yury Norov Acked-by: Tariq Toukan Reviewed-by: Jacob Keller Reviewed-by: Peter Lafreniere --- lib/cpumask.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/lib/cpumask.c b/lib/cpumask.c index c7c392514fd3..255974cd6734 100644 --- a/lib/cpumask.c +++ b/lib/cpumask.c @@ -110,7 +110,7 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask) #endif /** - * cpumask_local_spread - select the i'th cpu with local numa cpu's first + * cpumask_local_spread - select the i'th cpu based on NUMA distances * @i: index number * @node: local numa_node * @@ -132,15 +132,7 @@ unsigned int cpumask_local_spread(unsigned int i, int node) if (cpu < nr_cpu_ids) return cpu; } else { - /* NUMA first. */ - cpu = cpumask_nth_and(i, cpu_online_mask, cpumask_of_node(node)); - if (cpu < nr_cpu_ids) - return cpu; - - i -= cpumask_weight_and(cpu_online_mask, cpumask_of_node(node)); - - /* Skip NUMA nodes, done above. */ - cpu = cpumask_nth_andnot(i, cpu_online_mask, cpumask_of_node(node)); + cpu = sched_numa_find_nth_cpu(cpu_online_mask, i, node); if (cpu < nr_cpu_ids) return cpu; } From patchwork Sat Jan 21 04:24:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46770 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567294wrn; Fri, 20 Jan 2023 20:27:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXvKCzOf+fuSeIVe8sBgddE8avIdHVNDO5DSV6Uq8FYNEUMIVUA4C8+w+Wft/X2ycg0uGLYf X-Received: by 2002:a17:907:2bdd:b0:857:b916:94c2 with SMTP id gv29-20020a1709072bdd00b00857b91694c2mr12801210ejc.35.1674275248061; Fri, 20 Jan 2023 20:27:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275248; cv=none; d=google.com; s=arc-20160816; b=Pmc1BpNL2YpafkxbXqX8KqO2lpOTSmrd4lGI+25D39pr7+OBRmsRSpopg6zFNXhIXg 774NeTSOyfWngEcwgk0G9Yn8FWRECkDXcL9YD1aTg1AVo/FHr0C+qebyuI7VXv7BWR/g PauCOv1Iw4VsrReENhV7Q6Dbtma1MXTj8gt/5S4bxTpxgQjgeEQuUkcw7ZeroGWXViyc 4XOn9BJcktZX8A6cEj/b754MRHQBJggxy8+P6sVUCG/UWtqX4SswDRs2sK8Bi8PVRsnl 9XmJGHFUSpH5NogxFmPfBKBhDirWFRMd+PX0C52/t/UL/PZjBmpc+9EENZseqOevnW4N SOfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GwdL6i5RDIaOTjKHDRQpPnyPsA8RatFMsU5WrNy3SA0=; b=VzGQB3yFkwrXvvQU61S0lED/RL+LrdhtjX7sxQzgWI56U7QIKBhMve7wbrroC+tVyf mIuYLWoOO64V8+TvGE7WDGXfex/AlHUyavPqDiZX5T/rKbQL8hUzXhHn7rcG+1WC6dI6 /1V+X6kE7Br14BEEpudBhYuMmCVPTeKETkYnWYIwW0g+7LLgh2qcIAVbSPhRwI6BTRJr O1c2uuvK6Dx0eKPrrnWo/qv30xVRB63yn9CVdQ5kC7voEgyRwb+E5qWLXsZz8A4Qer5d lSVd4Di6xiFruVqWwT+qjSFFKBXvBk8hTTObi9Z8skry1DvYvlMgj51nj67fWxCK4FPf J5Xw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=lOKIeoFZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b60-20020a509f42000000b0049eeeff0949si2555276edf.234.2023.01.20.20.27.04; Fri, 20 Jan 2023 20:27:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=lOKIeoFZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229782AbjAUEZ1 (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229867AbjAUEYs (ORCPT ); Fri, 20 Jan 2023 23:24:48 -0500 Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E1CB518D6; Fri, 20 Jan 2023 20:24:47 -0800 (PST) Received: by mail-qt1-x82b.google.com with SMTP id o5so5796684qtr.11; Fri, 20 Jan 2023 20:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GwdL6i5RDIaOTjKHDRQpPnyPsA8RatFMsU5WrNy3SA0=; b=lOKIeoFZ2XmWttrm+5mhAr+EoKXQrXKDkyPM0i9zjpJ7oEZ7xf20CcBTFzpC7sbxpP HYGr+UfCS7Y6l0DcfL0vfyfshN+LQYevRg6KS5hazTyVvRIfZ1zOP8onArDxsjjx8szU 1Hi8MUAbBTPnxKBkZ/ri41lTBruU6fn35huesBk939D+g/+bJxbXsov3j2O3od+KvYPW qkJjJmYKbWoMLYP/WnEu3Yl+5uWM/ixSGDHJ28Wj9E/laWGUAySoIIq+Ug+0ONRY4Dlq Gdi129W9RHo8gvEFNm3Bye67FtlnI1xoGS9Zd2kKnJsTpOEJG17fDNPvCNJSSRlyiOW1 BqeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GwdL6i5RDIaOTjKHDRQpPnyPsA8RatFMsU5WrNy3SA0=; b=si/SsffpTv9+J57PbXFX82yptkZ0wmcSpiS2O8SaBzyqNbV7UiCgb1tlDjVZxrfuM7 ossxus4MNqDWDAceX5hitbZuvVg+bG/g7N4fgIloYGJ+dl4CA3LTZGUxiWgRZ5uVLpWV AAeliZ504Cf9Q6LMwxjFB0F1bGFMvk24I2F8TLEi5KjyN5hfVajWn9za5TIj57DAXGP7 AcjEQPhRAJRRFiYIc25W+03SkQNIjVXuj+eh4bp8fx0yRChxZkhWxrp+Fwm19qFmh3hN RYlJVqd2xlKhswzrJ4fgnktchm02IPzq19ER4pcxDjGJGj8CDM/ABNfgHJZ7XgMlu6sW MD6A== X-Gm-Message-State: AFqh2krMWxWG+SnLiNOyeinOB9KeHNpVM9EVUtk10oQNNe8XbTK/LqXw woK4YwCL1NW68e4XUXIiV2gOaovAgOs= X-Received: by 2002:ac8:604d:0:b0:39c:da20:f7ac with SMTP id k13-20020ac8604d000000b0039cda20f7acmr23825028qtm.9.1674275086283; Fri, 20 Jan 2023 20:24:46 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id jt14-20020a05622aa00e00b003adc7f652a0sm17772522qtb.66.2023.01.20.20.24.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:45 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 5/9] lib/cpumask: reorganize cpumask_local_spread() logic Date: Fri, 20 Jan 2023 20:24:32 -0800 Message-Id: <20230121042436.2661843-6-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604842574915677?= X-GMAIL-MSGID: =?utf-8?q?1755604842574915677?= Now after moving all NUMA logic into sched_numa_find_nth_cpu(), else-branch of cpumask_local_spread() is just a function call, and we can simplify logic by using ternary operator. While here, replace BUG() with WARN_ON(). Signed-off-by: Yury Norov Acked-by: Tariq Toukan Reviewed-by: Jacob Keller Reviewed-by: Peter Lafreniere --- lib/cpumask.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/lib/cpumask.c b/lib/cpumask.c index 255974cd6734..10aa15715c0d 100644 --- a/lib/cpumask.c +++ b/lib/cpumask.c @@ -127,16 +127,12 @@ unsigned int cpumask_local_spread(unsigned int i, int node) /* Wrap: we always want a cpu. */ i %= num_online_cpus(); - if (node == NUMA_NO_NODE) { - cpu = cpumask_nth(i, cpu_online_mask); - if (cpu < nr_cpu_ids) - return cpu; - } else { - cpu = sched_numa_find_nth_cpu(cpu_online_mask, i, node); - if (cpu < nr_cpu_ids) - return cpu; - } - BUG(); + cpu = (node == NUMA_NO_NODE) ? + cpumask_nth(i, cpu_online_mask) : + sched_numa_find_nth_cpu(cpu_online_mask, i, node); + + WARN_ON(cpu >= nr_cpu_ids); + return cpu; } EXPORT_SYMBOL(cpumask_local_spread); From patchwork Sat Jan 21 04:24:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46773 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567324wrn; Fri, 20 Jan 2023 20:27:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXsynTm4HqViFst+pPtScke/BjcwXVPo13CL4P0fYQK2J5Enr4C3Srv1pwzz2mLU+ec5UqpX X-Received: by 2002:a17:906:6814:b0:870:ef77:e186 with SMTP id k20-20020a170906681400b00870ef77e186mr16924389ejr.49.1674275258193; Fri, 20 Jan 2023 20:27:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275258; cv=none; d=google.com; s=arc-20160816; b=DwTYbjuDLkqUHUxHNw3gm8Ol0OvA1C+D4PrYmLDeYjalUKkdefzVZHKFpcUfHRF8Z5 7eA2YInFq5LUEFzk8mlQslVFHI5+GLX7Ld0H8BjX10IWgFXlcJNRHsAUbljJF4Piic5K H6uUImQyPdUPg4NoY5RWwCV0AhlKdofEww1SIZ48IHSiR12rSv3FS70RPGuP6pecn/Bf fyuJtUqqPZeOvDUh0gH6mT5yiG7hBqfKNgG0arJfpvzi2J5UK0Fsa8U55cDpWcRHQEj7 lbpE2klVyScD23QAGpTlUgxMOtghok0Am7zsoHptRL+H2o5/ndCsFjTDiID2EOYXeSB4 xKig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HV7BA/e9O5RDYHc0GpSLhXnbGhSUVych30GpzNnayWI=; b=TMvEYZoktz4dsd3RjEGQd+ttW3KszWPGjNbMnwZY03jp8yjG5zEvvZBxWF1WVOvq+4 wofkGeBOan1sa0C0v/U0Nu5Wi14/13fmEyU6r0ZxfVU9kfZ8idhMeixc9Gmv8irHiGna QJNfJhFRNVr0ARLKb7N6Jig8GhUI6ax3TkNUCFubnRZoh2NpLLZX+sGYwhYUFYLpGvTw qlngYAY0CXBe2Ius/wui6VQDF7YJ1F+EAIiVBZanxWwvrvfag0fgpOpMLDymUL5OjHED BD5mgFgXdwvSWuvxvmp0bkeVSLL2N5IU6H+5bL4jj7JzuIMDdNs6KfiVvU9eKeoeaB8H ytOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=p1ffXzls; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qf42-20020a1709077f2a00b0087766a5d70esi9723980ejc.588.2023.01.20.20.27.14; Fri, 20 Jan 2023 20:27:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=p1ffXzls; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229711AbjAUEZm (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbjAUEZJ (ORCPT ); Fri, 20 Jan 2023 23:25:09 -0500 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E6B5518DD; Fri, 20 Jan 2023 20:24:48 -0800 (PST) Received: by mail-qt1-x831.google.com with SMTP id g16so3765754qtu.2; Fri, 20 Jan 2023 20:24:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HV7BA/e9O5RDYHc0GpSLhXnbGhSUVych30GpzNnayWI=; b=p1ffXzlsXkgwWfnkIN5dHKAoPB+qji3zKzAGLiA4NxXTWvd6Z5peqTNBzwpVXCjgWv X2KKCxq2paGC1/zLxjdoEhzEWNI2LNxkGahKE2Db+Ym896BEyDDW5Qm0XZfSf31bZzo4 72HnXJPhg8EvFePscmT7w3TWAXiYUav9sZfCYD6u5hdSpAGdi3hYJea4S9WrfmxTusgK kdX2DgvAeI25XaoojL/FWH5uJ4NwJym5E8VvzXANuNP9chp+0WWyAetY4hDJTl2nNaGr p6j0dB11+ap2OQmYo4f/7ESulz4ABsI0p+JBqEOIMBbq+7qP0BIHpb5lbQ2neX3XRYo1 Y0Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HV7BA/e9O5RDYHc0GpSLhXnbGhSUVych30GpzNnayWI=; b=iYYJ8KwaY1vyyDOsQxu6nBdhTL3vINiPF7HMfmrvYhEa08fpfgn+nHpM7jMvTlnEy2 Kh/zIuIYc6MUhKd9wN56BlXwHNOnGOGfPLj+cSj7vq6Vz5hOzFiASHFUl3iyOBBnsySO qTWOUhDuBZexF+TiK9yJEjV21AHeOaRy84VSk+tc9K0re4UJhCRE6JbTqxFWZk3njQW9 Qp2JOLklHEg+zzQ8nhlVjt+EpAyn6E3s/5ISYkEcmxKmyDO93Oc5FHMj0GgzYAbanGk9 B/1PSpE1+0/opHANMnCf51KcpeoMd1/nMlB63z0k7rbnTByzCjVQ54jvaQ0zGcY+kbFZ PF3Q== X-Gm-Message-State: AFqh2kqNXMHZ7WUYJ1I11q0OQXYAk4AwPmv5aAiyo8UnGMIDd0TDxcn1 SnaC3xsiWewDrEUlpJYCn2OLJAqt0M0= X-Received: by 2002:ac8:760b:0:b0:3ac:7f6b:ed11 with SMTP id t11-20020ac8760b000000b003ac7f6bed11mr23372537qtq.15.1674275087742; Fri, 20 Jan 2023 20:24:47 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id e13-20020ac84b4d000000b003b62cd6e60esm9329010qts.43.2023.01.20.20.24.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:47 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 6/9] sched/topology: Introduce sched_numa_hop_mask() Date: Fri, 20 Jan 2023 20:24:33 -0800 Message-Id: <20230121042436.2661843-7-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604853079740734?= X-GMAIL-MSGID: =?utf-8?q?1755604853079740734?= From: Valentin Schneider Tariq has pointed out that drivers allocating IRQ vectors would benefit from having smarter NUMA-awareness - cpumask_local_spread() only knows about the local node and everything outside is in the same bucket. sched_domains_numa_masks is pretty much what we want to hand out (a cpumask of CPUs reachable within a given distance budget), introduce sched_numa_hop_mask() to export those cpumasks. Link: http://lore.kernel.org/r/20220728191203.4055-1-tariqt@nvidia.com Signed-off-by: Valentin Schneider Reviewed-by: Yury Norov Signed-off-by: Yury Norov --- include/linux/topology.h | 7 +++++++ kernel/sched/topology.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 72f264575698..344c2362755a 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -247,11 +247,18 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) #ifdef CONFIG_NUMA int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); +extern const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops); #else static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) { return cpumask_nth(cpu, cpus); } + +static inline const struct cpumask * +sched_numa_hop_mask(unsigned int node, unsigned int hops) +{ + return ERR_PTR(-EOPNOTSUPP); +} #endif /* CONFIG_NUMA */ #endif /* _LINUX_TOPOLOGY_H */ diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 2bf89186a10f..1233affc106c 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2124,6 +2124,39 @@ int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) return ret; } EXPORT_SYMBOL_GPL(sched_numa_find_nth_cpu); + +/** + * sched_numa_hop_mask() - Get the cpumask of CPUs at most @hops hops away from + * @node + * @node: The node to count hops from. + * @hops: Include CPUs up to that many hops away. 0 means local node. + * + * Return: On success, a pointer to a cpumask of CPUs at most @hops away from + * @node, an error value otherwise. + * + * Requires rcu_lock to be held. Returned cpumask is only valid within that + * read-side section, copy it if required beyond that. + * + * Note that not all hops are equal in distance; see sched_init_numa() for how + * distances and masks are handled. + * Also note that this is a reflection of sched_domains_numa_masks, which may change + * during the lifetime of the system (offline nodes are taken out of the masks). + */ +const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops) +{ + struct cpumask ***masks; + + if (node >= nr_node_ids || hops >= sched_domains_numa_levels) + return ERR_PTR(-EINVAL); + + masks = rcu_dereference(sched_domains_numa_masks); + if (!masks) + return ERR_PTR(-EBUSY); + + return masks[hops][node]; +} +EXPORT_SYMBOL_GPL(sched_numa_hop_mask); + #endif /* CONFIG_NUMA */ static int __sdt_alloc(const struct cpumask *cpu_map) From patchwork Sat Jan 21 04:24:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46772 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567319wrn; Fri, 20 Jan 2023 20:27:36 -0800 (PST) X-Google-Smtp-Source: AMrXdXtgYxF5Oup0FmKGeQKHE1NN/6U5a7eHUKDM0b4Muwuh5xpOz7l3uqcspg0sQi4N29uaM4I9 X-Received: by 2002:a17:906:df49:b0:84c:f9b0:b54a with SMTP id if9-20020a170906df4900b0084cf9b0b54amr14933388ejc.58.1674275256449; Fri, 20 Jan 2023 20:27:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275256; cv=none; d=google.com; s=arc-20160816; b=jTCGPmV9I/+oXjr1j9E20ZDIyqgta/fn9YhMMluybev95NMyiRgIbTCN6MAbkt1PJy W7cr4hPGMCM6k0jey076v2OAhTYOHlNOLTzmY96BJ3TfIcGtBXxWmDos5aF3jOFq4djJ EMkIKMBSOTckA/4as+nAfsEDWMgptg7L22Z2cob5oF1rzJgOp3V+fZA1TqRM3aIA+wB6 wqfEgpYyyKeDX8krXuy3KNJmyJcMlYp28iMP6i3xmgr8Ifa5UXAC9WZPX6MuIWUZfwFP kx0+IAo7jVgbRv9TgvC+hsOEuntp/IUvB34LgsFjwygJmxByCLz4ogqMfSxo09LNB+Uy Y1ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+S0aKvtNtHji2MwhEJCerU5PDiGf8bD52pNGNrh5jes=; b=nbIoYrgdxV8NoA4ZubZ6XJCyKKz14fDvjYw5/YNnwle2oosb+nCvL2AaetK71qsIO8 t+MSZcNCJ635Y8r5z9MTtq/MknPAggK2ZE9N/VsAjMKQnf7Rug4AqikOv2PntCnCwUnB RQkxNulbm+EbVY43nLQOC2XUc1qGHor2tRol9EMnTb6wfK48ayUmOObE3sOSxQYvOhAN 43hYv8qVUx0QJRMYF9fDnJYFtnFkkfgTB0G4XdaHk2QzkhZmaH9Fh5fazvFZtIV7Yo/T TOoMRn6k7fkkSAVUP2DUcP9Ha/yvPBxbz12UxBz91JVeUvh5sTPjHfuoRXCZJu6pitU6 uFmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=aMj8AEtU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g3-20020a50d5c3000000b0049e9777ccbcsi5785650edj.215.2023.01.20.20.27.13; Fri, 20 Jan 2023 20:27:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=aMj8AEtU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229645AbjAUEZg (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229707AbjAUEZJ (ORCPT ); Fri, 20 Jan 2023 23:25:09 -0500 Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D83E54119; Fri, 20 Jan 2023 20:24:50 -0800 (PST) Received: by mail-qt1-x834.google.com with SMTP id jr19so4581282qtb.7; Fri, 20 Jan 2023 20:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+S0aKvtNtHji2MwhEJCerU5PDiGf8bD52pNGNrh5jes=; b=aMj8AEtUF7+dyhnay+QHtRZynp8reOr1WKixbCoxjKPgzn4RbIoBzC6ObzXVtimmCl gxeLz4Pf+NcsqI6ANdrOSokQ5PV2rFWyi8IuNdK/+325yIgghnbm2i35b/kBDRSzgU1E hHoJ9jiV//+RqlxtX18tN+85qBpVl9OTclfHDu7NSijv3lrX24mdNc0XJFldC/9B8tca zBY7y4R7i6JKiTNnPDjHxQrjyvvc8ob2w7rputsx8wrzgIZBWuhD3g764GOTseVURSl6 jdS+kq++ChxJmCNbWH1lTil1mePsY0mHnURRwnCoJy8tBXKgFpWOPgYjOghP27CjLDfs erxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+S0aKvtNtHji2MwhEJCerU5PDiGf8bD52pNGNrh5jes=; b=DxEqhksqJBmykFmrTzAO9S/MGMNsFhlcA/beSpojMxfg40U1D9udDMQWqHYQccFT0X uBivr6yXtB2dmu4xG5z5ZVV4UvBQ+mJs3qWqi06EoOXfOYedUjFCQn3UPTAMW4ygRE35 pHOStVX5ALLCExuaE9QrU3uLncm1k/EN86G1w9zxiRCkks+P/7rUfDxZaLCFHrN8CkS5 /wsV1TEVgWbn4ltHXitN6pgjoxvQ4t8s9rtNjB3SuiM+PJxkd91PzOP7J/HMgUu+nbVb +KxA8RTx96CHoTt/gmIDWrvKbVDX/lCVGppUvBZgWdmlVEDzcV5bmt/54s2H2KvIa0fU pZQw== X-Gm-Message-State: AFqh2kqRyITJn3BUYG3elXBx0DN/Hc4N2xhP3kV4UV/3+wgaOKKVNYbY XSBO3wcflJdacZ+HxSuwPIsCtA39sfo= X-Received: by 2002:ac8:6681:0:b0:3b6:3b1a:d028 with SMTP id d1-20020ac86681000000b003b63b1ad028mr22452427qtp.19.1674275089170; Fri, 20 Jan 2023 20:24:49 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id dt26-20020a05620a479a00b00705c8cce5dcsm16652425qkb.111.2023.01.20.20.24.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:48 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 7/9] sched/topology: Introduce for_each_numa_hop_mask() Date: Fri, 20 Jan 2023 20:24:34 -0800 Message-Id: <20230121042436.2661843-8-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604851626706601?= X-GMAIL-MSGID: =?utf-8?q?1755604851626706601?= From: Valentin Schneider The recently introduced sched_numa_hop_mask() exposes cpumasks of CPUs reachable within a given distance budget, wrap the logic for iterating over all (distance, mask) values inside an iterator macro. Signed-off-by: Valentin Schneider Reviewed-by: Yury Norov Signed-off-by: Yury Norov --- include/linux/topology.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 344c2362755a..fea32377f7c7 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -261,4 +261,22 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops) } #endif /* CONFIG_NUMA */ +/** + * for_each_numa_hop_mask - iterate over cpumasks of increasing NUMA distance + * from a given node. + * @mask: the iteration variable. + * @node: the NUMA node to start the search from. + * + * Requires rcu_lock to be held. + * + * Yields cpu_online_mask for @node == NUMA_NO_NODE. + */ +#define for_each_numa_hop_mask(mask, node) \ + for (unsigned int __hops = 0; \ + mask = (node != NUMA_NO_NODE || __hops) ? \ + sched_numa_hop_mask(node, __hops) : \ + cpu_online_mask, \ + !IS_ERR_OR_NULL(mask); \ + __hops++) + #endif /* _LINUX_TOPOLOGY_H */ From patchwork Sat Jan 21 04:24:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46774 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567358wrn; Fri, 20 Jan 2023 20:27:46 -0800 (PST) X-Google-Smtp-Source: AMrXdXvVTkzGlOhxnoLTi2W5CVrW+Cl3xT1sHQWCFFojqyhz/Csc7+LAPyfc2eZO7A7mFszO0KSr X-Received: by 2002:a05:6402:3214:b0:496:37c9:b8e6 with SMTP id g20-20020a056402321400b0049637c9b8e6mr19917127eda.8.1674275266450; Fri, 20 Jan 2023 20:27:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275266; cv=none; d=google.com; s=arc-20160816; b=gHZCrUBkPtjBcNDJ21nJ0TXyzNDChjtsEFnn32m2T9V6gylxUIeRCZEuFlB2SSslWt qV6QQLmK1JNS2PdibOAey3nYDae7fc9VVgwFzM+PYsOWRCZmlQdIVq+SqJiIX9XRbZtb AGS99jJIKSC06Y7n+6QjuCZwKmfuYYEt3tMZwfv+nHU9xzsPDF4pkWMVzbifziBqotMM VM65NOxhEOlF946OzlcjChaciLKSdgYcNz3bnzmAnymLWG9bT2wt90VK/pLbEUipxVeB Eq4gemKWD04e+FiksLLai7xYbqVWTXh3mTCtFl/sxz+sFXIC63lt3PC2Lgh1rK69wQkl ar9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8i7hEQKy3Tv22Tf+BYE3TPaMMnMVMCijj+nmvEv3jfY=; b=DS4ogeoi44vahIwVkoYi4hUf2sQLfJZ7/SkOlkms+DBTosf9wRsuD0yLXQPCnJu19c LgnFhvwx+UmvcQYUleQMGLX2rSxyArv5HPs8XO+08WBZUaGp8QduHaEPmwrDcdFJ1ez2 sm0j3eJ8bRa6cVQVd/dUeFZ4fkZlRTrIDmcB6aMVclBVyQbaZO3MGO9F49ln+y7Uk81n e1pd8wHvMXWG5jdWZVqFiukyNJwIRvxefpvJXjiDVQOPODu/K4k8fuJ7FaBM+XJP0Nq0 UxtGrfVdw3jt8xMcy/QiUtIZXhvPZFRuCgqr6GJScrX81J3b8THt9obHr/j7O1HE5BiH sjGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hxLoDXhk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g10-20020aa7c58a000000b0046fe2f565e0si43535805edq.315.2023.01.20.20.27.23; Fri, 20 Jan 2023 20:27:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hxLoDXhk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbjAUEZq (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229725AbjAUEZK (ORCPT ); Fri, 20 Jan 2023 23:25:10 -0500 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D655B56184; Fri, 20 Jan 2023 20:24:51 -0800 (PST) Received: by mail-qt1-x831.google.com with SMTP id g16so3765826qtu.2; Fri, 20 Jan 2023 20:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8i7hEQKy3Tv22Tf+BYE3TPaMMnMVMCijj+nmvEv3jfY=; b=hxLoDXhk7C2v5VFNwnQT5dh6rtaO4xLTCoG0IavPdT3JHzjHXd9b8swC9gYXB4e8iU hfUdQD+foziEYn0XHCAUe2w7frKlSpKJpNysPxnz2oHecjVcw9DFsFsFgqODY9GZZuSu 9jrkdiDSe8BQDEqgwFmKv+7Oud7krKv6f6HegsifvYDUBavEIb3FUgerGZJfrsil8kQI QoE38VtE70Q3w/HYSSKRkyNYwE+S4cp8Y9nxE9NH00xltda8Kf2W+17EpmodEBmXj1+X N3KhofWfZsX/aGsbJvbqS5kS61x3Be3m/cz427NqWgw4RDoV88qW35l04CVL1WjMqj1L 9eYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8i7hEQKy3Tv22Tf+BYE3TPaMMnMVMCijj+nmvEv3jfY=; b=5yCtpZRiLkv3w+ZofJsx3tYY/qlktmY6yRqhSH4kNy6nq3nMAjn4cX+toWqyy0WRqK 5oiWEZ1BIfPhP+rPnGFdKSuWh6/Hut5JlIwkWz1vYvROHVPOageLJ5cSYNgB1elrAaAT +dnoRQbpqYnEOlfy47cNX6Kwsa0mIOOm7VRMZSKZ7mLY6V+p02O60FI3nFEhmmOaliDL HDLy7vDIJQ47gB2dygz/gJhjWqLPPxxDGGNbWqTtzdnmBgV2ifmvVPr+AfPBTaWYcPi0 g9bYm0ye7op5y0iiJZqmQZXDpsbTdD9MT5jjuF6TyF/bmSRM7lOa8EPbxtevhTuXq6J3 qV1w== X-Gm-Message-State: AFqh2krzVLBK1puIKKwrFyANwmgh8+btUt4ieKYhAGfnd0z9zuwaDNRr 8dWh3zlhlQF5koSzXWg2477uC7jOOJE= X-Received: by 2002:ac8:7352:0:b0:3b2:e2e3:df2c with SMTP id q18-20020ac87352000000b003b2e2e3df2cmr20816675qtp.57.1674275090779; Fri, 20 Jan 2023 20:24:50 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id x7-20020ac81207000000b003b68ea3d5c8sm3428081qti.41.2023.01.20.20.24.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:50 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 8/9] net/mlx5e: Improve remote NUMA preferences used for the IRQ affinity hints Date: Fri, 20 Jan 2023 20:24:35 -0800 Message-Id: <20230121042436.2661843-9-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604861872731590?= X-GMAIL-MSGID: =?utf-8?q?1755604861872731590?= From: Tariq Toukan In the IRQ affinity hints, replace the binary NUMA preference (local / remote) with the improved for_each_numa_hop_cpu() API that minds the actual distances, so that remote NUMAs with short distance are preferred over farther ones. This has significant performance implications when using NUMA-aware allocated memory (follow [1] and derivatives for example). [1] drivers/net/ethernet/mellanox/mlx5/core/en_main.c :: mlx5e_open_channel() int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix)); Performance tests: TCP multi-stream, using 16 iperf3 instances pinned to 16 cores (with aRFS on). Active cores: 64,65,72,73,80,81,88,89,96,97,104,105,112,113,120,121 +-------------------------+-----------+------------------+------------------+ | | BW (Gbps) | TX side CPU util | RX side CPU util | +-------------------------+-----------+------------------+------------------+ | Baseline | 52.3 | 6.4 % | 17.9 % | +-------------------------+-----------+------------------+------------------+ | Applied on TX side only | 52.6 | 5.2 % | 18.5 % | +-------------------------+-----------+------------------+------------------+ | Applied on RX side only | 94.9 | 11.9 % | 27.2 % | +-------------------------+-----------+------------------+------------------+ | Applied on both sides | 95.1 | 8.4 % | 27.3 % | +-------------------------+-----------+------------------+------------------+ Bottleneck in RX side is released, reached linerate (~1.8x speedup). ~30% less cpu util on TX. * CPU util on active cores only. Setups details (similar for both sides): NIC: ConnectX6-DX dual port, 100 Gbps each. Single port used in the tests. $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 16 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7763 64-Core Processor Stepping: 1 CPU MHz: 2594.804 BogoMIPS: 4890.73 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 32768K NUMA node0 CPU(s): 0-7,128-135 NUMA node1 CPU(s): 8-15,136-143 NUMA node2 CPU(s): 16-23,144-151 NUMA node3 CPU(s): 24-31,152-159 NUMA node4 CPU(s): 32-39,160-167 NUMA node5 CPU(s): 40-47,168-175 NUMA node6 CPU(s): 48-55,176-183 NUMA node7 CPU(s): 56-63,184-191 NUMA node8 CPU(s): 64-71,192-199 NUMA node9 CPU(s): 72-79,200-207 NUMA node10 CPU(s): 80-87,208-215 NUMA node11 CPU(s): 88-95,216-223 NUMA node12 CPU(s): 96-103,224-231 NUMA node13 CPU(s): 104-111,232-239 NUMA node14 CPU(s): 112-119,240-247 NUMA node15 CPU(s): 120-127,248-255 .. $ numactl -H .. node distances: node 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0: 10 11 11 11 12 12 12 12 32 32 32 32 32 32 32 32 1: 11 10 11 11 12 12 12 12 32 32 32 32 32 32 32 32 2: 11 11 10 11 12 12 12 12 32 32 32 32 32 32 32 32 3: 11 11 11 10 12 12 12 12 32 32 32 32 32 32 32 32 4: 12 12 12 12 10 11 11 11 32 32 32 32 32 32 32 32 5: 12 12 12 12 11 10 11 11 32 32 32 32 32 32 32 32 6: 12 12 12 12 11 11 10 11 32 32 32 32 32 32 32 32 7: 12 12 12 12 11 11 11 10 32 32 32 32 32 32 32 32 8: 32 32 32 32 32 32 32 32 10 11 11 11 12 12 12 12 9: 32 32 32 32 32 32 32 32 11 10 11 11 12 12 12 12 10: 32 32 32 32 32 32 32 32 11 11 10 11 12 12 12 12 11: 32 32 32 32 32 32 32 32 11 11 11 10 12 12 12 12 12: 32 32 32 32 32 32 32 32 12 12 12 12 10 11 11 11 13: 32 32 32 32 32 32 32 32 12 12 12 12 11 10 11 11 14: 32 32 32 32 32 32 32 32 12 12 12 12 11 11 10 11 15: 32 32 32 32 32 32 32 32 12 12 12 12 11 11 11 10 $ cat /sys/class/net/ens5f0/device/numa_node 14 Affinity hints (127 IRQs): Before: 331: 00000000,00000000,00000000,00000000,00010000,00000000,00000000,00000000 332: 00000000,00000000,00000000,00000000,00020000,00000000,00000000,00000000 333: 00000000,00000000,00000000,00000000,00040000,00000000,00000000,00000000 334: 00000000,00000000,00000000,00000000,00080000,00000000,00000000,00000000 335: 00000000,00000000,00000000,00000000,00100000,00000000,00000000,00000000 336: 00000000,00000000,00000000,00000000,00200000,00000000,00000000,00000000 337: 00000000,00000000,00000000,00000000,00400000,00000000,00000000,00000000 338: 00000000,00000000,00000000,00000000,00800000,00000000,00000000,00000000 339: 00010000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 340: 00020000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 341: 00040000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 342: 00080000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 343: 00100000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 344: 00200000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 345: 00400000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 346: 00800000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 347: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001 348: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002 349: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004 350: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008 351: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000010 352: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000020 353: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000040 354: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000080 355: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000100 356: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000200 357: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000400 358: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000800 359: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00001000 360: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00002000 361: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00004000 362: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00008000 363: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00010000 364: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00020000 365: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00040000 366: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00080000 367: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00100000 368: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00200000 369: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00400000 370: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00800000 371: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,01000000 372: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,02000000 373: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,04000000 374: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,08000000 375: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,10000000 376: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,20000000 377: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,40000000 378: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,80000000 379: 00000000,00000000,00000000,00000000,00000000,00000000,00000001,00000000 380: 00000000,00000000,00000000,00000000,00000000,00000000,00000002,00000000 381: 00000000,00000000,00000000,00000000,00000000,00000000,00000004,00000000 382: 00000000,00000000,00000000,00000000,00000000,00000000,00000008,00000000 383: 00000000,00000000,00000000,00000000,00000000,00000000,00000010,00000000 384: 00000000,00000000,00000000,00000000,00000000,00000000,00000020,00000000 385: 00000000,00000000,00000000,00000000,00000000,00000000,00000040,00000000 386: 00000000,00000000,00000000,00000000,00000000,00000000,00000080,00000000 387: 00000000,00000000,00000000,00000000,00000000,00000000,00000100,00000000 388: 00000000,00000000,00000000,00000000,00000000,00000000,00000200,00000000 389: 00000000,00000000,00000000,00000000,00000000,00000000,00000400,00000000 390: 00000000,00000000,00000000,00000000,00000000,00000000,00000800,00000000 391: 00000000,00000000,00000000,00000000,00000000,00000000,00001000,00000000 392: 00000000,00000000,00000000,00000000,00000000,00000000,00002000,00000000 393: 00000000,00000000,00000000,00000000,00000000,00000000,00004000,00000000 394: 00000000,00000000,00000000,00000000,00000000,00000000,00008000,00000000 395: 00000000,00000000,00000000,00000000,00000000,00000000,00010000,00000000 396: 00000000,00000000,00000000,00000000,00000000,00000000,00020000,00000000 397: 00000000,00000000,00000000,00000000,00000000,00000000,00040000,00000000 398: 00000000,00000000,00000000,00000000,00000000,00000000,00080000,00000000 399: 00000000,00000000,00000000,00000000,00000000,00000000,00100000,00000000 400: 00000000,00000000,00000000,00000000,00000000,00000000,00200000,00000000 401: 00000000,00000000,00000000,00000000,00000000,00000000,00400000,00000000 402: 00000000,00000000,00000000,00000000,00000000,00000000,00800000,00000000 403: 00000000,00000000,00000000,00000000,00000000,00000000,01000000,00000000 404: 00000000,00000000,00000000,00000000,00000000,00000000,02000000,00000000 405: 00000000,00000000,00000000,00000000,00000000,00000000,04000000,00000000 406: 00000000,00000000,00000000,00000000,00000000,00000000,08000000,00000000 407: 00000000,00000000,00000000,00000000,00000000,00000000,10000000,00000000 408: 00000000,00000000,00000000,00000000,00000000,00000000,20000000,00000000 409: 00000000,00000000,00000000,00000000,00000000,00000000,40000000,00000000 410: 00000000,00000000,00000000,00000000,00000000,00000000,80000000,00000000 411: 00000000,00000000,00000000,00000000,00000000,00000001,00000000,00000000 412: 00000000,00000000,00000000,00000000,00000000,00000002,00000000,00000000 413: 00000000,00000000,00000000,00000000,00000000,00000004,00000000,00000000 414: 00000000,00000000,00000000,00000000,00000000,00000008,00000000,00000000 415: 00000000,00000000,00000000,00000000,00000000,00000010,00000000,00000000 416: 00000000,00000000,00000000,00000000,00000000,00000020,00000000,00000000 417: 00000000,00000000,00000000,00000000,00000000,00000040,00000000,00000000 418: 00000000,00000000,00000000,00000000,00000000,00000080,00000000,00000000 419: 00000000,00000000,00000000,00000000,00000000,00000100,00000000,00000000 420: 00000000,00000000,00000000,00000000,00000000,00000200,00000000,00000000 421: 00000000,00000000,00000000,00000000,00000000,00000400,00000000,00000000 422: 00000000,00000000,00000000,00000000,00000000,00000800,00000000,00000000 423: 00000000,00000000,00000000,00000000,00000000,00001000,00000000,00000000 424: 00000000,00000000,00000000,00000000,00000000,00002000,00000000,00000000 425: 00000000,00000000,00000000,00000000,00000000,00004000,00000000,00000000 426: 00000000,00000000,00000000,00000000,00000000,00008000,00000000,00000000 427: 00000000,00000000,00000000,00000000,00000000,00010000,00000000,00000000 428: 00000000,00000000,00000000,00000000,00000000,00020000,00000000,00000000 429: 00000000,00000000,00000000,00000000,00000000,00040000,00000000,00000000 430: 00000000,00000000,00000000,00000000,00000000,00080000,00000000,00000000 431: 00000000,00000000,00000000,00000000,00000000,00100000,00000000,00000000 432: 00000000,00000000,00000000,00000000,00000000,00200000,00000000,00000000 433: 00000000,00000000,00000000,00000000,00000000,00400000,00000000,00000000 434: 00000000,00000000,00000000,00000000,00000000,00800000,00000000,00000000 435: 00000000,00000000,00000000,00000000,00000000,01000000,00000000,00000000 436: 00000000,00000000,00000000,00000000,00000000,02000000,00000000,00000000 437: 00000000,00000000,00000000,00000000,00000000,04000000,00000000,00000000 438: 00000000,00000000,00000000,00000000,00000000,08000000,00000000,00000000 439: 00000000,00000000,00000000,00000000,00000000,10000000,00000000,00000000 440: 00000000,00000000,00000000,00000000,00000000,20000000,00000000,00000000 441: 00000000,00000000,00000000,00000000,00000000,40000000,00000000,00000000 442: 00000000,00000000,00000000,00000000,00000000,80000000,00000000,00000000 443: 00000000,00000000,00000000,00000000,00000001,00000000,00000000,00000000 444: 00000000,00000000,00000000,00000000,00000002,00000000,00000000,00000000 445: 00000000,00000000,00000000,00000000,00000004,00000000,00000000,00000000 446: 00000000,00000000,00000000,00000000,00000008,00000000,00000000,00000000 447: 00000000,00000000,00000000,00000000,00000010,00000000,00000000,00000000 448: 00000000,00000000,00000000,00000000,00000020,00000000,00000000,00000000 449: 00000000,00000000,00000000,00000000,00000040,00000000,00000000,00000000 450: 00000000,00000000,00000000,00000000,00000080,00000000,00000000,00000000 451: 00000000,00000000,00000000,00000000,00000100,00000000,00000000,00000000 452: 00000000,00000000,00000000,00000000,00000200,00000000,00000000,00000000 453: 00000000,00000000,00000000,00000000,00000400,00000000,00000000,00000000 454: 00000000,00000000,00000000,00000000,00000800,00000000,00000000,00000000 455: 00000000,00000000,00000000,00000000,00001000,00000000,00000000,00000000 456: 00000000,00000000,00000000,00000000,00002000,00000000,00000000,00000000 457: 00000000,00000000,00000000,00000000,00004000,00000000,00000000,00000000 After: 331: 00000000,00000000,00000000,00000000,00010000,00000000,00000000,00000000 332: 00000000,00000000,00000000,00000000,00020000,00000000,00000000,00000000 333: 00000000,00000000,00000000,00000000,00040000,00000000,00000000,00000000 334: 00000000,00000000,00000000,00000000,00080000,00000000,00000000,00000000 335: 00000000,00000000,00000000,00000000,00100000,00000000,00000000,00000000 336: 00000000,00000000,00000000,00000000,00200000,00000000,00000000,00000000 337: 00000000,00000000,00000000,00000000,00400000,00000000,00000000,00000000 338: 00000000,00000000,00000000,00000000,00800000,00000000,00000000,00000000 339: 00010000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 340: 00020000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 341: 00040000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 342: 00080000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 343: 00100000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 344: 00200000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 345: 00400000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 346: 00800000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 347: 00000000,00000000,00000000,00000000,00000001,00000000,00000000,00000000 348: 00000000,00000000,00000000,00000000,00000002,00000000,00000000,00000000 349: 00000000,00000000,00000000,00000000,00000004,00000000,00000000,00000000 350: 00000000,00000000,00000000,00000000,00000008,00000000,00000000,00000000 351: 00000000,00000000,00000000,00000000,00000010,00000000,00000000,00000000 352: 00000000,00000000,00000000,00000000,00000020,00000000,00000000,00000000 353: 00000000,00000000,00000000,00000000,00000040,00000000,00000000,00000000 354: 00000000,00000000,00000000,00000000,00000080,00000000,00000000,00000000 355: 00000000,00000000,00000000,00000000,00000100,00000000,00000000,00000000 356: 00000000,00000000,00000000,00000000,00000200,00000000,00000000,00000000 357: 00000000,00000000,00000000,00000000,00000400,00000000,00000000,00000000 358: 00000000,00000000,00000000,00000000,00000800,00000000,00000000,00000000 359: 00000000,00000000,00000000,00000000,00001000,00000000,00000000,00000000 360: 00000000,00000000,00000000,00000000,00002000,00000000,00000000,00000000 361: 00000000,00000000,00000000,00000000,00004000,00000000,00000000,00000000 362: 00000000,00000000,00000000,00000000,00008000,00000000,00000000,00000000 363: 00000000,00000000,00000000,00000000,01000000,00000000,00000000,00000000 364: 00000000,00000000,00000000,00000000,02000000,00000000,00000000,00000000 365: 00000000,00000000,00000000,00000000,04000000,00000000,00000000,00000000 366: 00000000,00000000,00000000,00000000,08000000,00000000,00000000,00000000 367: 00000000,00000000,00000000,00000000,10000000,00000000,00000000,00000000 368: 00000000,00000000,00000000,00000000,20000000,00000000,00000000,00000000 369: 00000000,00000000,00000000,00000000,40000000,00000000,00000000,00000000 370: 00000000,00000000,00000000,00000000,80000000,00000000,00000000,00000000 371: 00000001,00000000,00000000,00000000,00000000,00000000,00000000,00000000 372: 00000002,00000000,00000000,00000000,00000000,00000000,00000000,00000000 373: 00000004,00000000,00000000,00000000,00000000,00000000,00000000,00000000 374: 00000008,00000000,00000000,00000000,00000000,00000000,00000000,00000000 375: 00000010,00000000,00000000,00000000,00000000,00000000,00000000,00000000 376: 00000020,00000000,00000000,00000000,00000000,00000000,00000000,00000000 377: 00000040,00000000,00000000,00000000,00000000,00000000,00000000,00000000 378: 00000080,00000000,00000000,00000000,00000000,00000000,00000000,00000000 379: 00000100,00000000,00000000,00000000,00000000,00000000,00000000,00000000 380: 00000200,00000000,00000000,00000000,00000000,00000000,00000000,00000000 381: 00000400,00000000,00000000,00000000,00000000,00000000,00000000,00000000 382: 00000800,00000000,00000000,00000000,00000000,00000000,00000000,00000000 383: 00001000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 384: 00002000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 385: 00004000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 386: 00008000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 387: 01000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 388: 02000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 389: 04000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 390: 08000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 391: 10000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 392: 20000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 393: 40000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 394: 80000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000 395: 00000000,00000000,00000000,00000000,00000000,00000001,00000000,00000000 396: 00000000,00000000,00000000,00000000,00000000,00000002,00000000,00000000 397: 00000000,00000000,00000000,00000000,00000000,00000004,00000000,00000000 398: 00000000,00000000,00000000,00000000,00000000,00000008,00000000,00000000 399: 00000000,00000000,00000000,00000000,00000000,00000010,00000000,00000000 400: 00000000,00000000,00000000,00000000,00000000,00000020,00000000,00000000 401: 00000000,00000000,00000000,00000000,00000000,00000040,00000000,00000000 402: 00000000,00000000,00000000,00000000,00000000,00000080,00000000,00000000 403: 00000000,00000000,00000000,00000000,00000000,00000100,00000000,00000000 404: 00000000,00000000,00000000,00000000,00000000,00000200,00000000,00000000 405: 00000000,00000000,00000000,00000000,00000000,00000400,00000000,00000000 406: 00000000,00000000,00000000,00000000,00000000,00000800,00000000,00000000 407: 00000000,00000000,00000000,00000000,00000000,00001000,00000000,00000000 408: 00000000,00000000,00000000,00000000,00000000,00002000,00000000,00000000 409: 00000000,00000000,00000000,00000000,00000000,00004000,00000000,00000000 410: 00000000,00000000,00000000,00000000,00000000,00008000,00000000,00000000 411: 00000000,00000000,00000000,00000000,00000000,00010000,00000000,00000000 412: 00000000,00000000,00000000,00000000,00000000,00020000,00000000,00000000 413: 00000000,00000000,00000000,00000000,00000000,00040000,00000000,00000000 414: 00000000,00000000,00000000,00000000,00000000,00080000,00000000,00000000 415: 00000000,00000000,00000000,00000000,00000000,00100000,00000000,00000000 416: 00000000,00000000,00000000,00000000,00000000,00200000,00000000,00000000 417: 00000000,00000000,00000000,00000000,00000000,00400000,00000000,00000000 418: 00000000,00000000,00000000,00000000,00000000,00800000,00000000,00000000 419: 00000000,00000000,00000000,00000000,00000000,01000000,00000000,00000000 420: 00000000,00000000,00000000,00000000,00000000,02000000,00000000,00000000 421: 00000000,00000000,00000000,00000000,00000000,04000000,00000000,00000000 422: 00000000,00000000,00000000,00000000,00000000,08000000,00000000,00000000 423: 00000000,00000000,00000000,00000000,00000000,10000000,00000000,00000000 424: 00000000,00000000,00000000,00000000,00000000,20000000,00000000,00000000 425: 00000000,00000000,00000000,00000000,00000000,40000000,00000000,00000000 426: 00000000,00000000,00000000,00000000,00000000,80000000,00000000,00000000 427: 00000000,00000001,00000000,00000000,00000000,00000000,00000000,00000000 428: 00000000,00000002,00000000,00000000,00000000,00000000,00000000,00000000 429: 00000000,00000004,00000000,00000000,00000000,00000000,00000000,00000000 430: 00000000,00000008,00000000,00000000,00000000,00000000,00000000,00000000 431: 00000000,00000010,00000000,00000000,00000000,00000000,00000000,00000000 432: 00000000,00000020,00000000,00000000,00000000,00000000,00000000,00000000 433: 00000000,00000040,00000000,00000000,00000000,00000000,00000000,00000000 434: 00000000,00000080,00000000,00000000,00000000,00000000,00000000,00000000 435: 00000000,00000100,00000000,00000000,00000000,00000000,00000000,00000000 436: 00000000,00000200,00000000,00000000,00000000,00000000,00000000,00000000 437: 00000000,00000400,00000000,00000000,00000000,00000000,00000000,00000000 438: 00000000,00000800,00000000,00000000,00000000,00000000,00000000,00000000 439: 00000000,00001000,00000000,00000000,00000000,00000000,00000000,00000000 440: 00000000,00002000,00000000,00000000,00000000,00000000,00000000,00000000 441: 00000000,00004000,00000000,00000000,00000000,00000000,00000000,00000000 442: 00000000,00008000,00000000,00000000,00000000,00000000,00000000,00000000 443: 00000000,00010000,00000000,00000000,00000000,00000000,00000000,00000000 444: 00000000,00020000,00000000,00000000,00000000,00000000,00000000,00000000 445: 00000000,00040000,00000000,00000000,00000000,00000000,00000000,00000000 446: 00000000,00080000,00000000,00000000,00000000,00000000,00000000,00000000 447: 00000000,00100000,00000000,00000000,00000000,00000000,00000000,00000000 448: 00000000,00200000,00000000,00000000,00000000,00000000,00000000,00000000 449: 00000000,00400000,00000000,00000000,00000000,00000000,00000000,00000000 450: 00000000,00800000,00000000,00000000,00000000,00000000,00000000,00000000 451: 00000000,01000000,00000000,00000000,00000000,00000000,00000000,00000000 452: 00000000,02000000,00000000,00000000,00000000,00000000,00000000,00000000 453: 00000000,04000000,00000000,00000000,00000000,00000000,00000000,00000000 454: 00000000,08000000,00000000,00000000,00000000,00000000,00000000,00000000 455: 00000000,10000000,00000000,00000000,00000000,00000000,00000000,00000000 456: 00000000,20000000,00000000,00000000,00000000,00000000,00000000,00000000 457: 00000000,40000000,00000000,00000000,00000000,00000000,00000000,00000000 Signed-off-by: Tariq Toukan [Tweaked API use] Suggested-by: Yury Norov Signed-off-by: Valentin Schneider Reviewed-by: Yury Norov Signed-off-by: Yury Norov --- drivers/net/ethernet/mellanox/mlx5/core/eq.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c index 8f7580fec193..c111e8657d54 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c @@ -817,9 +817,12 @@ static void comp_irqs_release(struct mlx5_core_dev *dev) static int comp_irqs_request(struct mlx5_core_dev *dev) { struct mlx5_eq_table *table = dev->priv.eq_table; + const struct cpumask *prev = cpu_none_mask; + const struct cpumask *mask; int ncomp_eqs = table->num_comp_eqs; u16 *cpus; int ret; + int cpu; int i; ncomp_eqs = table->num_comp_eqs; @@ -838,8 +841,19 @@ static int comp_irqs_request(struct mlx5_core_dev *dev) ret = -ENOMEM; goto free_irqs; } - for (i = 0; i < ncomp_eqs; i++) - cpus[i] = cpumask_local_spread(i, dev->priv.numa_node); + + i = 0; + rcu_read_lock(); + for_each_numa_hop_mask(mask, dev->priv.numa_node) { + for_each_cpu_andnot(cpu, mask, prev) { + cpus[i] = cpu; + if (++i == ncomp_eqs) + goto spread_done; + } + prev = mask; + } +spread_done: + rcu_read_unlock(); ret = mlx5_irqs_request_vectors(dev, cpus, ncomp_eqs, table->comp_irqs); kfree(cpus); if (ret < 0) From patchwork Sat Jan 21 04:24:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 46771 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp567305wrn; Fri, 20 Jan 2023 20:27:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXti4ySKPirNytW+hvdw0WQECHjlLPVnH8Pv1bPxNg0MkfEnItagpUGEEL+Ybs3ixLU5XyQD X-Received: by 2002:a17:906:4e54:b0:871:9359:3763 with SMTP id g20-20020a1709064e5400b0087193593763mr16728287ejw.40.1674275252646; Fri, 20 Jan 2023 20:27:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674275252; cv=none; d=google.com; s=arc-20160816; b=OOwh2IyMd18l1D1atI7YBP3tRqoOulSarjVwOcXS0BPATSkIUlnFg/JU2ZmQ9PIePs /HCRvnBuabwAq7gjbHjwjSucvQag4w0q5BEPVVLd0OL877+IOThXgYpZaBTecxkGS6e6 rDbWFqLvYW7lLVslZstfFa2zGJDv7TXo6q62zOorg2hZ8cLEqsJ9WhHDI3a8/Re70HLi AltPuRTuX5Q2PXKlGHEovOjer94qMm344lmUqt3521Uo8HLWBNgQfo79mm9pz/cvl57+ gs1P+xMKKQ/7M81SulJCA4hPIM+kSTPluiiT+GNdYQQfC737R38BLq7UW8/z8V//2FyL cq1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qwRl6AriAxtLsQshG0WLT8xXpSJi+NGIVxpWZYX/6II=; b=Z1LZNIRfLfWuZZ3/cH4CzkWJrwddTFQf32q7jNzWLm/a+MS6hBBtr+XUg7/s7vL1Ko 213bssNn4+kWG63srqz0AlEOJIa+1bO8VAlqM9S0OtfGp6WXMt8Me+oRjsNXXhskVeq6 b5lum8g75rJ9ghST+A/0Kx3ad/RcF+Z3QnbMYErT/ko09P0oAz1hUvFnQNpKYiRGGl4X ftsq/+WZEKOZBilq7pLgVaWlDkD4GSqWhCzlJjemyuQB7eeITHQ/CXqf1zjb8xU4cqYX 8NOnKKoA5S5lHu+jfo5T1Ittpz+du+8L1jjGcAuXYBUUwAi57fn49FfIBeT0PUldLrkS uQ8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=oYxzuFmr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hu8-20020a170907a08800b0087787cec039si505247ejc.311.2023.01.20.20.27.09; Fri, 20 Jan 2023 20:27:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=oYxzuFmr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229643AbjAUEZb (ORCPT + 99 others); Fri, 20 Jan 2023 23:25:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229568AbjAUEZJ (ORCPT ); Fri, 20 Jan 2023 23:25:09 -0500 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70723530EC; Fri, 20 Jan 2023 20:24:53 -0800 (PST) Received: by mail-qt1-x829.google.com with SMTP id g16so3765838qtu.2; Fri, 20 Jan 2023 20:24:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qwRl6AriAxtLsQshG0WLT8xXpSJi+NGIVxpWZYX/6II=; b=oYxzuFmr5pVt01gJibPdD5ns77LWVJeQmmVcsoPATQ4AhkNE8Da5wsu9Mc1wJLeNaB tehkdeWQw8aXAeGragVTrfuwq647RKKSq23gNAuq9fjHG5+mwpwbdicJcnlrSaljEfQr Eb4bFX6NvVp0r5sOxhE2t/7kgma9WzMeWKopGbGj4cFeEf3SghA9EFKoNP2F8ewXDmpg 1GJMDoD0tN/M476NAiSuwmkMZjRnvs29wvpAoqtHYdA4dhfCJDtFiNT4jD3KDxwhJWJy 7aVDQmCmdhwT0P+5kyStCHbeHIKIm0dZSkrbfczaE/6i5VUbDGixfB8lbev7Cga34Dh4 WNOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qwRl6AriAxtLsQshG0WLT8xXpSJi+NGIVxpWZYX/6II=; b=wVG6Gd/rAGHQc2rlv9wYu0+LBdmUgtcntNe1rssNUWfZw5j2OSNzHu/4eEiqJ2uNWE 6RIGnjH147H+VLLT2/YiWJTrT8+wWgGb2Ki+n8IvUpIPxZdDGaou6jyOpuoDK5X2cDhk zfJYB53CgMlRqEAp1ggjbDvtI3HvJDCX9vegzal9dTvROZAPt+82jS9AFHZ4bX8grCH9 jLliBqUQT/UpADEPHWtxJBfiBZ3OaRhrTlgPPN4CP08xqKzjtJOD9LEdo5At0Ap1jkVd W+CZLY3VXAG9EY6IQtNaCEPXpVPajuiLcYgDyWZCNMBczHXd+m9+Q1kHsCZ3B4Ib2I8f E3Rg== X-Gm-Message-State: AFqh2kolEj+i1bDZ8BECtUVoBZpNOYxNwwyWHOGyDjE2h+Xl4U2XD8M+ Z/btl2BBENcohzdoB5giy0VUGH9b2tE= X-Received: by 2002:ac8:6a0e:0:b0:3ab:7928:526c with SMTP id t14-20020ac86a0e000000b003ab7928526cmr23006426qtr.17.1674275092061; Fri, 20 Jan 2023 20:24:52 -0800 (PST) Received: from localhost (50-242-44-45-static.hfc.comcastbusiness.net. [50.242.44.45]) by smtp.gmail.com with ESMTPSA id ga15-20020a05622a590f00b003b6325dfc4esm8255304qtb.67.2023.01.20.20.24.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Jan 2023 20:24:51 -0800 (PST) From: Yury Norov To: linux-kernel@vger.kernel.org, "David S. Miller" , Andy Shevchenko , Barry Song , Ben Segall , Dietmar Eggemann , Gal Pressman , Greg Kroah-Hartman , Haniel Bristot de Oliveira , Heiko Carstens , Ingo Molnar , Jacob Keller , Jakub Kicinski , Jason Gunthorpe , Jesse Brandeburg , Jonathan Cameron , Juri Lelli , Leon Romanovsky , Linus Torvalds , Mel Gorman , Peter Lafreniere , Peter Zijlstra , Rasmus Villemoes , Saeed Mahameed , Steven Rostedt , Tariq Toukan , Tariq Toukan , Tony Luck , Valentin Schneider , Vincent Guittot Cc: Yury Norov , linux-crypto@vger.kernel.org, netdev@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 9/9] lib/cpumask: update comment for cpumask_local_spread() Date: Fri, 20 Jan 2023 20:24:36 -0800 Message-Id: <20230121042436.2661843-10-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230121042436.2661843-1-yury.norov@gmail.com> References: <20230121042436.2661843-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755604847036934501?= X-GMAIL-MSGID: =?utf-8?q?1755604847036934501?= Now that we have an iterator-based alternative for a very common case of using cpumask_local_spread for all cpus in a row, it's worth to mention that in comment to cpumask_local_spread(). Signed-off-by: Yury Norov Reviewed-by: Valentin Schneider Reviewed-by: Tariq Toukan --- lib/cpumask.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/lib/cpumask.c b/lib/cpumask.c index 10aa15715c0d..98291b07c756 100644 --- a/lib/cpumask.c +++ b/lib/cpumask.c @@ -114,11 +114,29 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask) * @i: index number * @node: local numa_node * - * This function selects an online CPU according to a numa aware policy; - * local cpus are returned first, followed by non-local ones, then it - * wraps around. + * Returns online CPU according to a numa aware policy; local cpus are returned + * first, followed by non-local ones, then it wraps around. * - * It's not very efficient, but useful for setup. + * For those who wants to enumerate all CPUs based on their NUMA distances, + * i.e. call this function in a loop, like: + * + * for (i = 0; i < num_online_cpus(); i++) { + * cpu = cpumask_local_spread(i, node); + * do_something(cpu); + * } + * + * There's a better alternative based on for_each()-like iterators: + * + * for_each_numa_hop_mask(mask, node) { + * for_each_cpu_andnot(cpu, mask, prev) + * do_something(cpu); + * prev = mask; + * } + * + * It's simpler and more verbose than above. Complexity of iterator-based + * enumeration is O(sched_domains_numa_levels * nr_cpu_ids), while + * cpumask_local_spread() when called for each cpu is + * O(sched_domains_numa_levels * nr_cpu_ids * log(nr_cpu_ids)). */ unsigned int cpumask_local_spread(unsigned int i, int node) {