From patchwork Wed Mar 15 08:49:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 70073 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2211101wrd; Wed, 15 Mar 2023 01:50:57 -0700 (PDT) X-Google-Smtp-Source: AK7set+STZzYlfC1YIVyw6rYeMA9JPXgdPowk9pKhjx6M9lJuLH+/Ey6ardVKFRMblb1q0+VpaoK X-Received: by 2002:a05:6a21:6d85:b0:d4:c605:4512 with SMTP id wl5-20020a056a216d8500b000d4c6054512mr9635794pzb.30.1678870257379; Wed, 15 Mar 2023 01:50:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678870257; cv=none; d=google.com; s=arc-20160816; b=QYBHKs+h05wi+8eNlpvjOim+hsjdw7iWtm/HqffRVsXGEUkJfBVPYtrcC0gqdwdyHp 8o+kKmQPRqa96KXkEqngc1carJXtkTJmNH5+pJARgljuSpzeB7xd3OvoEPXtquRqe0jr yaecTHM9vpsBM0BLKPHbGI6Degs9XTH3trcehTPN5Zpv8JDDEK13zkzxq6Wyq21Td7zU jeHYFvBK3C5QVZ69hRS+G4ylM9gHgXD2ilQQRs9vCzNZXTphFBvN6DstR5VyC6POtTva 7g4rQ4pP2oXoRFteiAsvi3tcOMtCrOHRCtCovzDbxGKqOn8oq3U+X3zMldGGqk4hE9Kg Sn+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YkxChdLDa4UFijk1Jheg8QY/La3zPetPy6md4h64POE=; b=SBvgIMs8THytI8Lk1XNikc0sG1v66i0CzKk4n+tLnmxpSHRWPKw4ZfwUp8W0dE0oRB soJyXc/D0P+wbCyN+Hjx2eSRju/m2zr3xy7r1D3JWLDABGLSjZ+I/BmVaHcGnF6Dkyp7 s4m3FOKVO328x3U21A2uWBI8v3RQI0tJer9f6StIncXZaTHQn9b2BuI4oxyJHNPRqAr4 9jljxHlnXujBiHtCgAJW8S6WG5AS4Aw14VEH3rry+HO++0l4hj0TsBHu23DDrk1UdsH4 bQLSKc7tU+/kE5FAh+XP+VjxXbAkYCoJE87eSlNvhQGZjEmKQotNek+y7aMA42UACJrn T7BQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=QjRMzeMP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d9-20020a654249000000b0050726756ee2si4413258pgq.96.2023.03.15.01.50.40; Wed, 15 Mar 2023 01:50:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=QjRMzeMP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231751AbjCOIuC (ORCPT + 99 others); Wed, 15 Mar 2023 04:50:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229487AbjCOItz (ORCPT ); Wed, 15 Mar 2023 04:49:55 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CA0B5D468 for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id c4so11279628pfl.0 for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20210112.gappssmtp.com; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YkxChdLDa4UFijk1Jheg8QY/La3zPetPy6md4h64POE=; b=QjRMzeMPM7bEXlB6ydwS837/NFUyQ8FTlay7FyNeoNs0VDwU6tSXowOh5PEijcS3y8 sD6nxAC6h4K1cRCQD9qu0IFALoN8EolNk7Bid2nRlhUXSKUTeDvUpAvWMnwt+HPHUQx/ KyyxJcLdPcRikCzCFdYvFjT+Mqx43i8pHeTDS+TQTsjpdDOaSGXHR1WhZ00ymwBzz/6L oUBDFOeHVrVNwCUzNaMl/ntSSHv31y6BC2N0XzgLSBSKpat6+edLAyQmzHqhhsxZ7a1M lj1RUSkyW5Ia0Sp/kwGQHE1me5OM6WQIgiCDHeaI+wyfJ3J//4uGOBLUlhHOul1FoJei V5fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YkxChdLDa4UFijk1Jheg8QY/La3zPetPy6md4h64POE=; b=uWZxx9xya5r0b9RfPfAt+WfiC1V1+4/nKB84LI4sVuS1csB/mVdWzWe4MiWGYJKTG0 JCtFrC6wRneIi2u0P1b3S27AtKknC4WJ2t595C4PjWJLGiaw3A7SL8tNsdBfh++8XftO 9Ehi1TTL2/UpU7fHAMNzHgtkYD+PcnDKe1XyofKyHD81ODeLYv0JXG4VkYzxnIWE+iqX vj39/qGDmN2XxYFrv/1VK3CVui6HEN4SJAqPTCPJnLzWkiq+fFvk9hJfJWN/qbxyQ1EY zCRSUKZDB7vFwfe1e9Ocx4WiuacJjGTCF5XUwp3OeYmLOmxoCcUN0WLM3DWOGsDjubo9 paCw== X-Gm-Message-State: AO0yUKV2X7pxHUUv/miXAE7PCM715sm6uqsGiFD5prWZudrqoxIL30JY /bVTl4fFyV6Na3PxP7tpNmVu1LKk2QSta1EjwWU= X-Received: by 2002:aa7:8ec1:0:b0:625:8217:15b9 with SMTP id b1-20020aa78ec1000000b00625821715b9mr3081112pfr.2.1678870191951; Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from dread.disaster.area (pa49-186-4-237.pa.vic.optusnet.com.au. [49.186.4.237]) by smtp.gmail.com with ESMTPSA id i17-20020aa787d1000000b0058d9058fe8asm2966024pfo.103.2023.03.15.01.49.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from [192.168.253.23] (helo=devoid.disaster.area) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1pcMpQ-008zeS-2N; Wed, 15 Mar 2023 19:49:48 +1100 Received: from dave by devoid.disaster.area with local (Exim 4.96) (envelope-from ) id 1pcMpQ-00Ag6L-0D; Wed, 15 Mar 2023 19:49:48 +1100 From: Dave Chinner To: linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org Cc: linux-mm@vger.kernel.org, linux-fsdevel@vger.kernel.org, yebin10@huawei.com Subject: [PATCH 1/4] cpumask: introduce for_each_cpu_or Date: Wed, 15 Mar 2023 19:49:35 +1100 Message-Id: <20230315084938.2544737-2-david@fromorbit.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230315084938.2544737-1-david@fromorbit.com> References: <20230315084938.2544737-1-david@fromorbit.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760423059210035205?= X-GMAIL-MSGID: =?utf-8?q?1760423059210035205?= From: Dave Chinner Equivalent of for_each_cpu_and, except it ORs the two masks together so it iterates all the CPUs present in either mask. Signed-off-by: Dave Chinner --- include/linux/cpumask.h | 17 +++++++++++++++++ include/linux/find.h | 37 +++++++++++++++++++++++++++++++++++++ lib/find_bit.c | 9 +++++++++ 3 files changed, 63 insertions(+) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index d4901ca8883c..ca736b05ec7b 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -350,6 +350,23 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta #define for_each_cpu_andnot(cpu, mask1, mask2) \ for_each_andnot_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits) +/** + * for_each_cpu_or - iterate over every cpu present in either mask + * @cpu: the (optionally unsigned) integer iterator + * @mask1: the first cpumask pointer + * @mask2: the second cpumask pointer + * + * This saves a temporary CPU mask in many places. It is equivalent to: + * struct cpumask tmp; + * cpumask_or(&tmp, &mask1, &mask2); + * for_each_cpu(cpu, &tmp) + * ... + * + * After the loop, cpu is >= nr_cpu_ids. + */ +#define for_each_cpu_or(cpu, mask1, mask2) \ + for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits) + /** * cpumask_any_but - return a "random" in a cpumask, but not this one. * @mask: the cpumask to search diff --git a/include/linux/find.h b/include/linux/find.h index 4647864a5ffd..5e4f39ef2e72 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -14,6 +14,8 @@ unsigned long _find_next_and_bit(const unsigned long *addr1, const unsigned long unsigned long nbits, unsigned long start); unsigned long _find_next_andnot_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, unsigned long start); +unsigned long _find_next_or_bit(const unsigned long *addr1, const unsigned long *addr2, + unsigned long nbits, unsigned long start); unsigned long _find_next_zero_bit(const unsigned long *addr, unsigned long nbits, unsigned long start); extern unsigned long _find_first_bit(const unsigned long *addr, unsigned long size); @@ -127,6 +129,36 @@ unsigned long find_next_andnot_bit(const unsigned long *addr1, } #endif +#ifndef find_next_or_bit +/** + * find_next_or_bit - find the next set bit in either memory regions + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @size: The bitmap size in bits + * @offset: The bitnumber to start searching at + * + * Returns the bit number for the next set bit + * If no bits are set, returns @size. + */ +static inline +unsigned long find_next_or_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (unlikely(offset >= size)) + return size; + + val = (*addr1 | *addr2) & GENMASK(size - 1, offset); + return val ? __ffs(val) : size; + } + + return _find_next_or_bit(addr1, addr2, size, offset); +} +#endif + #ifndef find_next_zero_bit /** * find_next_zero_bit - find the next cleared bit in a memory region @@ -536,6 +568,11 @@ unsigned long find_next_bit_le(const void *addr, unsigned (bit) = find_next_andnot_bit((addr1), (addr2), (size), (bit)), (bit) < (size);\ (bit)++) +#define for_each_or_bit(bit, addr1, addr2, size) \ + for ((bit) = 0; \ + (bit) = find_next_or_bit((addr1), (addr2), (size), (bit)), (bit) < (size);\ + (bit)++) + /* same as for_each_set_bit() but use bit as value to start with */ #define for_each_set_bit_from(bit, addr, size) \ for (; (bit) = find_next_bit((addr), (size), (bit)), (bit) < (size); (bit)++) diff --git a/lib/find_bit.c b/lib/find_bit.c index c10920e66788..32f99e9a670e 100644 --- a/lib/find_bit.c +++ b/lib/find_bit.c @@ -182,6 +182,15 @@ unsigned long _find_next_andnot_bit(const unsigned long *addr1, const unsigned l EXPORT_SYMBOL(_find_next_andnot_bit); #endif +#ifndef find_next_or_bit +unsigned long _find_next_or_bit(const unsigned long *addr1, const unsigned long *addr2, + unsigned long nbits, unsigned long start) +{ + return FIND_NEXT_BIT(addr1[idx] | addr2[idx], /* nop */, nbits, start); +} +EXPORT_SYMBOL(_find_next_or_bit); +#endif + #ifndef find_next_zero_bit unsigned long _find_next_zero_bit(const unsigned long *addr, unsigned long nbits, unsigned long start) From patchwork Wed Mar 15 08:49:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 70075 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2214422wrd; Wed, 15 Mar 2023 02:00:50 -0700 (PDT) X-Google-Smtp-Source: AK7set8+8ZlLg3l7tYClrQApKf1YTlcUfsZB06lgKO2cUmJiJrjlQqkc0pX24eP46vbFcTRpJJ6M X-Received: by 2002:a05:6a20:a5a8:b0:cc:c925:1e7a with SMTP id bc40-20020a056a20a5a800b000ccc9251e7amr42493275pzb.59.1678870850302; Wed, 15 Mar 2023 02:00:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678870850; cv=none; d=google.com; s=arc-20160816; b=HAA2B4vR52d68ksbmwnzoYKrf3wOLVM5pO7r0JLCfjft1WVBlrleIun4Utmj5UzFzj CNhQ1bHaEbcaEyUHnfn6clGLHV0r5A0CR+v6jy7hFLDN9mPWl2c4HACG5R9BJ40XrlQz S8Rnp2s9ZrgbbfmgwWc7NNcp6h+aesDUpiXu1xM+hEwrlw5qGdH7FyX1q0tlu4GfcZ94 tMMRwaQWJ1gTyOgCEIg7gSgJGMafPu7Ykg51rJVO0BFGMyVYCxdbbWMlhzkNWRCXI/Mo 1tT29PH2R5D1WL6btKEFnUsmtuUe6HBpI99mZQkCoyBYwwcr5Lwd1bT+J+9zzxhwXRgJ BMZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tkFj6yYLyA565/QhOvX6PgrnENIUFczCaPlTlQYQGRk=; b=DSPqPoBdxcUd5LDBfekq1g1qogQrDzpAN2H73bq6VneNBjY6dC2B79lz/DfXQojJvY 4QZFg79CgFpBJVEipFLw0nLp6LskR8jelPrijBM+qRzn3LdjzfARlfVS2JLguOLo5p1Q KTH1S17Wzq/YuldFDEfh0GqcnVedRI3qEdoOyi7k/9h4axzAITC4E3/2Uzq607eG2JE4 agLy6Jk44YFqKeZDzqmqEllX6mFP5ZTy3DOsZkLpE9Yp/PTE6ejfAQOtmzpvLtbsO7ZD zLN2sdsosCxtqI9uxK2dTSbZTWWVTF+EERiL4bUtwnPN1pgFd11fni2EXqZiUzqsGNIJ 77xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=P8lBfpG7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t21-20020a635f15000000b004f1a97a4c9dsi4478078pgb.799.2023.03.15.02.00.34; Wed, 15 Mar 2023 02:00:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=P8lBfpG7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231831AbjCOIuI (ORCPT + 99 others); Wed, 15 Mar 2023 04:50:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231465AbjCOIt4 (ORCPT ); Wed, 15 Mar 2023 04:49:56 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EE4658B40 for ; Wed, 15 Mar 2023 01:49:53 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id n16so5097773pfa.12 for ; Wed, 15 Mar 2023 01:49:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20210112.gappssmtp.com; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tkFj6yYLyA565/QhOvX6PgrnENIUFczCaPlTlQYQGRk=; b=P8lBfpG7IED6FPgo2ZB7apKwy3nUll9jUKZ8XzlrGpjDtYG8AbCcwo7zvIxIedMAi2 H6VZUAxroyRe59ECAAvynvkq8QSQkYykDwiNYazLkZA+lH3wCju9gPC+hT8a43f2BmoA j9RpQWC41i0mNupI+oXacSSvjKAGJ4+vgvsFXxD4G/haDpYnbt0PZ0RSkuEWQjljHrLG hGZcIKjas3JKkHHKlgv2t0oVs/hWgaL7pp9XDcpTbRs3hXgx0O3YC0Xiy7NF2n3Ej7tU RslGq9jpYn6NsD498EG4rurP0SM0c4G4pWSHhohSvWR1DWUZXDA7LXMdnh2+YmirVJdi 9A6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tkFj6yYLyA565/QhOvX6PgrnENIUFczCaPlTlQYQGRk=; b=diglQXSkc3wFwncCbAr2Jt1LV+daWJsEAlBr+83J5sVHOsInM+H+PFIgjYvyB8PjZs tC0fw6mynlt7XY0WXM2uET/lAletqRHxktNwD1HnKPNnojISIbTWhnqowhVdfJeQyUbq ccTvNDxLcCxqDbPeasiiblu8eY+qufQnAOq702MA9x8nfFXiKuRTLTv370BD0jeWV+45 7vM0XMzdWERgKWNnlA5+yHqvXt8Ol8CHlrh7JgAPXO3H+z5SKltiVyBU5wp4N5f96ocR Q+oZSrnDc+6knoIWJt6HpwO7gT3TcE1A56701RSTbHQa5DEyi7/V3X8fLB0lxRZuG+Fp SNXw== X-Gm-Message-State: AO0yUKXBnc7nTGeXrhrRTUA8RPN33JcBb7bm1Rf788H/4PCV5lt2u/jh nK1hfuKs6vaGr7IAQ5KXeOm23sibUdNSi3BYK7c= X-Received: by 2002:a05:6a00:41:b0:625:5560:696e with SMTP id i1-20020a056a00004100b006255560696emr5343794pfk.16.1678870192212; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) Received: from dread.disaster.area (pa49-186-4-237.pa.vic.optusnet.com.au. [49.186.4.237]) by smtp.gmail.com with ESMTPSA id h11-20020a62b40b000000b0062505afff9fsm2971980pfn.126.2023.03.15.01.49.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from [192.168.253.23] (helo=devoid.disaster.area) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1pcMpQ-008zeT-3D; Wed, 15 Mar 2023 19:49:48 +1100 Received: from dave by devoid.disaster.area with local (Exim 4.96) (envelope-from ) id 1pcMpQ-00Ag6P-0I; Wed, 15 Mar 2023 19:49:48 +1100 From: Dave Chinner To: linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org Cc: linux-mm@vger.kernel.org, linux-fsdevel@vger.kernel.org, yebin10@huawei.com Subject: [PATCH 2/4] pcpcntrs: fix dying cpu summation race Date: Wed, 15 Mar 2023 19:49:36 +1100 Message-Id: <20230315084938.2544737-3-david@fromorbit.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230315084938.2544737-1-david@fromorbit.com> References: <20230315084938.2544737-1-david@fromorbit.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760423680872137444?= X-GMAIL-MSGID: =?utf-8?q?1760423680872137444?= From: Dave Chinner In commit f689054aace2 ("percpu_counter: add percpu_counter_sum_all interface") a race condition between a cpu dying and percpu_counter_sum() iterating online CPUs was identified. The solution was to iterate all possible CPUs for summation via percpu_counter_sum_all(). We recently had a percpu_counter_sum() call in XFS trip over this same race condition and it fired a debug assert because the filesystem was unmounting and the counter *should* be zero just before we destroy it. That was reported here: https://lore.kernel.org/linux-kernel/20230314090649.326642-1-yebin@huaweicloud.com/ likely as a result of running generic/648 which exercises filesystems in the presence of CPU online/offline events. The solution to use percpu_counter_sum_all() is an awful one. We use percpu counters and percpu_counter_sum() for accurate and reliable threshold detection for space management, so a summation race condition during these operations can result in overcommit of available space and that may result in filesystem shutdowns. As percpu_counter_sum_all() iterates all possible CPUs rather than just those online or even those present, the mask can include CPUs that aren't even installed in the machine, or in the case of machines that can hot-plug CPU capable nodes, even have physical sockets present in the machine. Fundamentally, this race condition is caused by the CPU being offlined being removed from the cpu_online_mask before the notifier that cleans up per-cpu state is run. Hence percpu_counter_sum() will not sum the count for a cpu currently being taken offline, regardless of whether the notifier has run or not. This is the root cause of the bug. The percpu counter notifier iterates all the registered counters, locks the counter and moves the percpu count to the global sum. This is serialised against other operations that move the percpu counter to the global sum as well as percpu_counter_sum() operations that sum the percpu counts while holding the counter lock. Hence the notifier is safe to run concurrently with sum operations, and the only thing we actually need to care about is that percpu_counter_sum() iterates dying CPUs. That's trivial to do, and when there are no CPUs dying, it has no addition overhead except for a cpumask_or() operation. This change makes percpu_counter_sum() always do the right thing in the presence of CPU hot unplug events and makes percpu_counter_sum_all() unnecessary. This, in turn, means that filesystems like XFS, ext4, and btrfs don't have to work out when they should use percpu_counter_sum() vs percpu_counter_sum_all() in their space accounting algorithms Signed-off-by: Dave Chinner --- lib/percpu_counter.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index dba56c5c1837..0e096311e0c0 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -131,7 +131,7 @@ static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc, raw_spin_lock_irqsave(&fbc->lock, flags); ret = fbc->count; - for_each_cpu(cpu, cpu_mask) { + for_each_cpu_or(cpu, cpu_online_mask, cpu_mask) { s32 *pcount = per_cpu_ptr(fbc->counters, cpu); ret += *pcount; } @@ -141,11 +141,20 @@ static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc, /* * Add up all the per-cpu counts, return the result. This is a more accurate - * but much slower version of percpu_counter_read_positive() + * but much slower version of percpu_counter_read_positive(). + * + * We use the cpu mask of (cpu_online_mask | cpu_dying_mask) to capture sums + * from CPUs that are in the process of being taken offline. Dying cpus have + * been removed from the online mask, but may not have had the hotplug dead + * notifier called to fold the percpu count back into the global counter sum. + * By including dying CPUs in the iteration mask, we avoid this race condition + * so __percpu_counter_sum() just does the right thing when CPUs are being taken + * offline. */ s64 __percpu_counter_sum(struct percpu_counter *fbc) { - return __percpu_counter_sum_mask(fbc, cpu_online_mask); + + return __percpu_counter_sum_mask(fbc, cpu_dying_mask); } EXPORT_SYMBOL(__percpu_counter_sum); From patchwork Wed Mar 15 08:49:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 70072 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2211005wrd; Wed, 15 Mar 2023 01:50:40 -0700 (PDT) X-Google-Smtp-Source: AK7set+zthaQTnO1Xl1MrMDRsh0PreO384lm6Dr9hG9nT60efQ3Y1JXLFUyt6qkuqzGF6/W44bN1 X-Received: by 2002:a17:90b:3ec3:b0:237:ae7c:1594 with SMTP id rm3-20020a17090b3ec300b00237ae7c1594mr42501705pjb.8.1678870240692; Wed, 15 Mar 2023 01:50:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678870240; cv=none; d=google.com; s=arc-20160816; b=rJ0sSGoTxB6B0JAV2AAaCxfVvS00AobHW7BAWcy5nO7/zG3TcYLaCIocZ73d+OINfC yz9kAmtawiNg3D3okbXL47/zxRYeHi+7LfZTyeQlNleH9IP/WeHtWJ186ALID9VwOGiG RP+DEVIB0Y0IVIb44AfQjVi6ikOs1mEZxgQ74+LFAGtDi892MN6ckYscaZS2seHVmLY8 cfP52LeDqvvai7+LayGzRd/cvjUpEzrdm0ziprVel+W+sNb0AfwEldncshymB86tjgOI BBdnw5ZzLx/B+5uBzInW+8g1ZjJVZ5SIKlQfYPYF7gx2xRXYpxJQuR9zloFWqOajBKhJ QTFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3PiH3qSD87Gy2R0sEveTfXmyJns2RRHbdlIcrLyx7vY=; b=RPLHmwgI0YPVgjEVRqYyPCcIyVtXBkCsOxRxUbmq4aUAEelq7MFHPxdOakErtN2MTF CzdhK3I0JNMqdFojCMDKa2YyY1NYi8cFblEfcmNnODXnKKXcDkHhTx1grf5HQxaQxlYT jn+CQeVCweVIYy0op6vUJkA9aw2xOPpHpGRVn+d/gNhVJx6eSg6QsZ31zC/cM8IksZGD mzo1Co9KG/ly2My6lj6dNuil70Gp/DZ7gkJsl6+pNjJufuHVDYWelho+wuOpLHQJvnEX omYz4dyx1AbzYH9xU5fhk5pttqiGRIaW4E6pUPGOqjXZNEempxw7tzJdJiCHc6m4QTHM PNtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=dGXEAZWG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p15-20020a170902ebcf00b0019c93e0dce0si4472598plg.254.2023.03.15.01.50.25; Wed, 15 Mar 2023 01:50:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=dGXEAZWG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231776AbjCOIt6 (ORCPT + 99 others); Wed, 15 Mar 2023 04:49:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231789AbjCOIty (ORCPT ); Wed, 15 Mar 2023 04:49:54 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 810C75BD9D for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id x11so19240915pln.12 for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20210112.gappssmtp.com; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3PiH3qSD87Gy2R0sEveTfXmyJns2RRHbdlIcrLyx7vY=; b=dGXEAZWGtw3hjf4evG9X6AHe1yTsnbPEjFBkPoP86wCPr8QN6TVnbPOULBMveWDVoT jiIftWXhqKYSzp0qmhThWBeyCKsD4Tm6Xp+7PMte7cJF7wgzVLSiYssGL15uSOr3lvfV rQd66fF+t9UQq3jhdoFYBw6GCsPIQuVOxn5UolXc0160hrbGsy9zlV4+B/qhV9yCDS7F H/CMxjUw1Z+tGw+1u0CeYT7yhOpK5oiWVK6GvR3bPJpgJzz75iWIvVxMm4iFmXRulZ4+ jvTJP5PpwGUCDBuh+DdFDKIgLX1jwAvPcxNGWCCVoZa7qwZqgDG/cKCvCt3mmTsN4n4r PO9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678870192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3PiH3qSD87Gy2R0sEveTfXmyJns2RRHbdlIcrLyx7vY=; b=NP3I6nxvDXaOVRBo+YZChppC+Nlw1O3H5vyIOxgt7aRMwWkGxt//PwDIPzJM6YCdMZ vo0IzsMOCRsrt+FYCuzlBMkGr2GxT2NjsRRWKyD3y2gQeBqMu6Ri5VTaDy9b+N352rAU SilaIbs8i/H7b4jPZtyTzlyKwqPznJGACuxWSlZ5EZl8HJZQ69lPmysDiSygPttQfW6f GLG1BWyCn2LpBPDxLa4sZs1gZvGA7WOcDVFdZAtLV6Am40yY27JdxmjQ5ByChi2UTccV z//pCx4R6AMzuh3lY0N08U7Qpx6efqx1oddsZ/4x+FRG1jIop+BxsAL9PKcrsMACgeCw AJcg== X-Gm-Message-State: AO0yUKUVOklWGN3GAjqmyqHRTh2DwmZh4tYm9zKhrM34yIG7sqHwIvm3 jWlTz6pz7oJY7saeDHc3QEtI6JOviemjq2S+vNY= X-Received: by 2002:a17:902:e541:b0:19a:70f9:affb with SMTP id n1-20020a170902e54100b0019a70f9affbmr2478057plf.2.1678870191825; Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from dread.disaster.area (pa49-186-4-237.pa.vic.optusnet.com.au. [49.186.4.237]) by smtp.gmail.com with ESMTPSA id kk5-20020a170903070500b0019a928a8982sm3096849plb.118.2023.03.15.01.49.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from [192.168.253.23] (helo=devoid.disaster.area) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1pcMpQ-008zeU-45; Wed, 15 Mar 2023 19:49:48 +1100 Received: from dave by devoid.disaster.area with local (Exim 4.96) (envelope-from ) id 1pcMpQ-00Ag6T-0N; Wed, 15 Mar 2023 19:49:48 +1100 From: Dave Chinner To: linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org Cc: linux-mm@vger.kernel.org, linux-fsdevel@vger.kernel.org, yebin10@huawei.com Subject: [PATCH 3/4] fork: remove use of percpu_counter_sum_all Date: Wed, 15 Mar 2023 19:49:37 +1100 Message-Id: <20230315084938.2544737-4-david@fromorbit.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230315084938.2544737-1-david@fromorbit.com> References: <20230315084938.2544737-1-david@fromorbit.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760423041255700828?= X-GMAIL-MSGID: =?utf-8?q?1760423041255700828?= From: Dave Chinner This effectively reverts the change made in commit f689054aace2 ("percpu_counter: add percpu_counter_sum_all interface") as the race condition percpu_counter_sum_all() was invented to avoid is now handled directly in percpu_counter_sum() and nobody needs to care about summing racing with cpu unplug anymore. Signed-off-by: Dave Chinner --- kernel/fork.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index d8cda4c6de6c..c0257cbee093 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -755,11 +755,6 @@ static void check_mm(struct mm_struct *mm) for (i = 0; i < NR_MM_COUNTERS; i++) { long x = percpu_counter_sum(&mm->rss_stat[i]); - if (likely(!x)) - continue; - - /* Making sure this is not due to race with CPU offlining. */ - x = percpu_counter_sum_all(&mm->rss_stat[i]); if (unlikely(x)) pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", mm, resident_page_types[i], x); From patchwork Wed Mar 15 08:49:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 70071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2210971wrd; Wed, 15 Mar 2023 01:50:34 -0700 (PDT) X-Google-Smtp-Source: AK7set869eenbbq2tSibve01t65aRLrxvG51prJEfjQnCKdrbyCjTSAVDt7iT4EVAPiLXJKYkbrS X-Received: by 2002:a05:6a20:244d:b0:bf:58d1:ce86 with SMTP id t13-20020a056a20244d00b000bf58d1ce86mr16299470pzc.5.1678870233836; Wed, 15 Mar 2023 01:50:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678870233; cv=none; d=google.com; s=arc-20160816; b=T26ba+I7McVzoM52UJNpETBaJZG7PrGVP0fufMJbg+MztVmFcnBfjCtc0GR2iLOfQ5 qRCpZiC/5k6N5caO1plRqT/SgSiyQbOWyRxqcazlXOpEBBR+VyGyINMJDoclR1gQ9nWF 2U/CQVYDmLZbkqIeniQQlav57tCQex86ZCoE8djt5pqwK9SFAKe0WKogzrWz+43puTff qqPjUPtnR/17X+UhFbU9E87VmXbxH2fATKVIQoZCezSz7CxLnNx/4xIPUVCaeJzlSWF/ 5biiZGIGrjTt7AnSANofbJ3zot8jetKkELEaihkFGwKA+F4gLSPhqhani4D2lQHhyzM2 WtZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DnnMb4Et7WhpEe76RPSGsI6qWidW0Ymzw0QUU6G4yao=; b=WxFIC3/zvjO7Zs25DkNC/cNDNUUWhYX1stOSj8tNZgto/tFQ+0PPZxbftFSgatHGJi FHcVgvtHSN7/N8VZLgenVR1BfpaGHrM7uTaHvZGIDPb3EjuFNzCGvLHgbQAOMCz8Q1If jHg1IlmOEwwVhJ97F98KdZ7oin+cTlpw2fkWttd2cqcydAR1V/u/Q63B98Nk3VVbM15+ nA4Lah9CfEaEy4YpzjgDSznfRo/JMrJiAJZCaFKr+OdJU+RC9pDKB+UcnmPhn+42ug9r aSJ6cw5AYSL3JA+rCwM5FXIKcmpPX1z6atJQwYreqm0Bc8tgUcRnYed5gbIW0gcD/gR1 nvkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=MldnmFi+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t184-20020a6381c1000000b0050be676d6casi25484pgd.689.2023.03.15.01.50.16; Wed, 15 Mar 2023 01:50:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@fromorbit-com.20210112.gappssmtp.com header.s=20210112 header.b=MldnmFi+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=fromorbit.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231562AbjCOIt4 (ORCPT + 99 others); Wed, 15 Mar 2023 04:49:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231770AbjCOIty (ORCPT ); Wed, 15 Mar 2023 04:49:54 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30DA45BCA1 for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id h12-20020a17090aea8c00b0023d1311fab3so1150053pjz.1 for ; Wed, 15 Mar 2023 01:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20210112.gappssmtp.com; s=20210112; t=1678870191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DnnMb4Et7WhpEe76RPSGsI6qWidW0Ymzw0QUU6G4yao=; b=MldnmFi+QRjEcfycD9KY0bZvSKUHkvSL3UtA0U/PWvEzo8AB2GYhsOMJ9Yna93phsr 3pmBXGrj79AYyoeIfCO7JsUKNVP8tmqlS8mY/FTf5EtAxhPN1xd1Fm9/pzGq/ZdxfhQz eU2tuyfglF9o3N9mm4hhAzPHlUfBjcvnSq9Nb2s84Jjh61ufX2fH3XB7ShgyRL4j80Ax QyS9p698XnwvyGRgs0e4BrNF4ilCCsUmxrRDQcMuw618JIheaRe7R9Qmg/xhef7MwYkb GmW1c4Xe7Xs5vN1n3/NKD+0FvXsYvic5EsCMGBEC3z5N+9iBqt2u48HdxYZQHcZIPME0 idug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678870191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DnnMb4Et7WhpEe76RPSGsI6qWidW0Ymzw0QUU6G4yao=; b=TOCfPVzWCZst/cKGnOi7IDqEWlI3YVQcbrEmcmo4kE3UDFLVpK32+H90PI4l3/8Wif mtGdA9CKA0IjXFaK3MqDuSSF12lWZDrOk8BTIBG/9ZfRU+ZJrCY8cvkToJDFPtP6p7Dc LdXEyXUii5N9ctvEOIiaTFKSxbjzsJA65o+WrqAO2IpKMtW6ngIRmpxCwWquZH7qfDka FuKmg2i8NH76Ni5XF/w6ZVn7G5SkpI7Br/QlVS97Oruvyn2zDsyNYBHMmJluGb1tVTDG ilqmJNTSOALxY1tyQ6c5f6LXUs3TH8HH2ffUlBUICut90mDFupJUABbzbDJe/KzQ/oUo +Sug== X-Gm-Message-State: AO0yUKWVsgE78ScfeIuwHSFGDzzE/TG3yZsNu5DPfsqxRBVG2PBkeR4F FDONjth5NueT6tvgaUhdeWtQzUWUcjQ+T7TeqOQ= X-Received: by 2002:a05:6a20:7d88:b0:cc:32a8:323d with SMTP id v8-20020a056a207d8800b000cc32a8323dmr15446982pzj.28.1678870191517; Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from dread.disaster.area (pa49-186-4-237.pa.vic.optusnet.com.au. [49.186.4.237]) by smtp.gmail.com with ESMTPSA id e29-20020a63501d000000b0050be183459bsm593527pgb.34.2023.03.15.01.49.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 01:49:51 -0700 (PDT) Received: from [192.168.253.23] (helo=devoid.disaster.area) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1pcMpQ-008zeV-50; Wed, 15 Mar 2023 19:49:48 +1100 Received: from dave by devoid.disaster.area with local (Exim 4.96) (envelope-from ) id 1pcMpQ-00Ag6X-0S; Wed, 15 Mar 2023 19:49:48 +1100 From: Dave Chinner To: linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org Cc: linux-mm@vger.kernel.org, linux-fsdevel@vger.kernel.org, yebin10@huawei.com Subject: [PATCH 4/4] pcpcntr: remove percpu_counter_sum_all() Date: Wed, 15 Mar 2023 19:49:38 +1100 Message-Id: <20230315084938.2544737-5-david@fromorbit.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230315084938.2544737-1-david@fromorbit.com> References: <20230315084938.2544737-1-david@fromorbit.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760423034099019501?= X-GMAIL-MSGID: =?utf-8?q?1760423034099019501?= From: Dave Chinner percpu_counter_sum_all() is now redundant as the race condition it was invented to handle is now dealt with by percpu_counter_sum() directly and all users of percpu_counter_sum_all() have been removed. Remove it. This effectively reverts the changes made in f689054aace2 ("percpu_counter: add percpu_counter_sum_all interface") except for the cpumask iteration that fixes percpu_counter_sum() made earlier in this series. Signed-off-by: Dave Chinner --- include/linux/percpu_counter.h | 6 ----- lib/percpu_counter.c | 40 ++++++++++------------------------ 2 files changed, 11 insertions(+), 35 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 521a733e21a9..75b73c83bc9d 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -45,7 +45,6 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount); void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch); s64 __percpu_counter_sum(struct percpu_counter *fbc); -s64 percpu_counter_sum_all(struct percpu_counter *fbc); int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch); void percpu_counter_sync(struct percpu_counter *fbc); @@ -196,11 +195,6 @@ static inline s64 percpu_counter_sum(struct percpu_counter *fbc) return percpu_counter_read(fbc); } -static inline s64 percpu_counter_sum_all(struct percpu_counter *fbc) -{ - return percpu_counter_read(fbc); -} - static inline bool percpu_counter_initialized(struct percpu_counter *fbc) { return true; diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 0e096311e0c0..5004463c4f9f 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -122,23 +122,6 @@ void percpu_counter_sync(struct percpu_counter *fbc) } EXPORT_SYMBOL(percpu_counter_sync); -static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc, - const struct cpumask *cpu_mask) -{ - s64 ret; - int cpu; - unsigned long flags; - - raw_spin_lock_irqsave(&fbc->lock, flags); - ret = fbc->count; - for_each_cpu_or(cpu, cpu_online_mask, cpu_mask) { - s32 *pcount = per_cpu_ptr(fbc->counters, cpu); - ret += *pcount; - } - raw_spin_unlock_irqrestore(&fbc->lock, flags); - return ret; -} - /* * Add up all the per-cpu counts, return the result. This is a more accurate * but much slower version of percpu_counter_read_positive(). @@ -153,22 +136,21 @@ static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc, */ s64 __percpu_counter_sum(struct percpu_counter *fbc) { + s64 ret; + int cpu; + unsigned long flags; - return __percpu_counter_sum_mask(fbc, cpu_dying_mask); + raw_spin_lock_irqsave(&fbc->lock, flags); + ret = fbc->count; + for_each_cpu_or(cpu, cpu_online_mask, cpu_dying_mask) { + s32 *pcount = per_cpu_ptr(fbc->counters, cpu); + ret += *pcount; + } + raw_spin_unlock_irqrestore(&fbc->lock, flags); + return ret; } EXPORT_SYMBOL(__percpu_counter_sum); -/* - * This is slower version of percpu_counter_sum as it traverses all possible - * cpus. Use this only in the cases where accurate data is needed in the - * presense of CPUs getting offlined. - */ -s64 percpu_counter_sum_all(struct percpu_counter *fbc) -{ - return __percpu_counter_sum_mask(fbc, cpu_possible_mask); -} -EXPORT_SYMBOL(percpu_counter_sum_all); - int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, struct lock_class_key *key) {