From patchwork Thu Apr 6 01:56:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ye Bin X-Patchwork-Id: 7792 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp711793vqo; Wed, 5 Apr 2023 18:59:00 -0700 (PDT) X-Google-Smtp-Source: AKy350bPp2cvm54SLowOwDJQLJH/VDysGBIwB3EuA08N6jmL/rfOIoS0Dlvp+SgrWg5nTT6A91/9 X-Received: by 2002:a05:6402:607:b0:4fb:9fd4:6ac5 with SMTP id n7-20020a056402060700b004fb9fd46ac5mr4091449edv.14.1680746340746; Wed, 05 Apr 2023 18:59:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680746340; cv=none; d=google.com; s=arc-20160816; b=qcqzDrcEh6tAzwItBhplWsanAbG2YgcQVa9Iq9wI+bmYCHt3VFrvr7Ubd7i03YrzCq uDlKbj6m9ZfASgv22s6rQsIntNhUxpywJyrLcnCiury8lZGp4LOLeOqeecW+Rzu4Bxvg 7EfZRiebR+V8FNngGxfXXwJLcD7hdMgC+VBHXxM72lDhuZW1TSaQmRx1eQ2ntclE5J6E KJs5bC1kcj9Mgvj9JGUqJiwy27Oe6J4rwFs6UVKnzQgMrSyw9cKd8dZ8aW/aaAo3RrHf di0uEUxgtZqVRUmjzaxVE3WoRcKJvWvvvqF/yyl2OJ5VnS+ak+PfEotVIbyQgENQihJC iWVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=y0mMIsyoHIFKhUmiSbuuW0Oh7xs6wDECo6XujLpsONI=; b=shtiYwKSueHN7xjyxLdYTCTA8RfHOe2M/xF9Y7dTzpkq77h6HFWKMjfWxLS+qaidNS bGmho85oUl1VRRWVMnn+CzyJ31sJuBrLvKTOb+K4ZE2uT2AyBmbblCE4fQofkb+sGGRC 0rYGPDrJbO0YCD6Je5R1JoWjwxdZT9h8b95X9OgP+sFD/gG/IJ6xEWigrs9fgL3obyx6 k89MC37ULuaVPnrKCI40nWKIDuoWZRdp60UiBDYosurKuST+FqadPX+PYDtKclsxUbq7 CcJgiQQL7ypnEfFIx6I4bYzVlvhNhY+DwgO3Hdx7VWqqAPomhXckHiiSS/C859iN50Kk AjuQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o16-20020aa7d3d0000000b005021cce645asi156102edr.617.2023.04.05.18.58.36; Wed, 05 Apr 2023 18:59:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233370AbjDFB5H (ORCPT + 99 others); Wed, 5 Apr 2023 21:57:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233074AbjDFB5D (ORCPT ); Wed, 5 Apr 2023 21:57:03 -0400 Received: from dggsgout12.his.huawei.com (unknown [45.249.212.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6D02769E for ; Wed, 5 Apr 2023 18:57:01 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.30.67.169]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4PsPkd0RQsz4f3nqD for ; Thu, 6 Apr 2023 09:56:57 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP3 (Coremail) with SMTP id _Ch0CgCnUyHoJi5kX936GA--.31319S4; Thu, 06 Apr 2023 09:56:58 +0800 (CST) From: Ye Bin To: dennis@kernel.org, tj@kernel.org, cl@linux.com, linux-mm@kvack.org, yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk Cc: linux-kernel@vger.kernel.org, dchinner@redhat.com, yebin10@huawei.com, yebin@huaweicloud.com Subject: [PATCH v2 0/2] fix dying cpu compare race Date: Thu, 6 Apr 2023 09:56:27 +0800 Message-Id: <20230406015629.1804722-1-yebin@huaweicloud.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgCnUyHoJi5kX936GA--.31319S4 X-Coremail-Antispam: 1UD129KBjDUn29KB7ZKAUJUUUUU529EdanIXcx71UUUUU7v73 VFW2AGmfu7bjvjm3AaLaJ3UjIYCTnIWjp_UUUYK7AC8VAFwI0_Gr0_Xr1l1xkIjI8I6I8E 6xAIw20EY4v20xvaj40_Wr0E3s1l1IIY67AEw4v_Jr0_Jr4l8cAvFVAK0II2c7xJM28Cjx kF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8I cVCY1x0267AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87 Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21le4C262xC7I0v67AE wI8IwI1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I 8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lF7I2 1c0EjII2zVCS5cI20VAGYxC7M4IIrI8v6xkF7I0E8cxan2IY04v7M4kE6I8I3I0E14AKx2 xKxVC2ax8xMxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8C rVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_Jr0_Jr4lx4CE17CEb7AF67AKxVWUtVW8Zw CIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x02 67AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_WFyUJVCq3wCI42IY6I8E87Iv67AKxV WUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjfU oOJ5UUUUU X-CM-SenderInfo: p1hex046kxt4xhlfz01xgou0bp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=2.1 required=5.0 tests=KHOP_HELO_FCRDNS,MAY_BE_FORGED, SPF_HELO_NONE,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762390274671489145?= X-GMAIL-MSGID: =?utf-8?q?1762390274671489145?= From: Ye Bin This patch set solve race between '__percpu_counter_compare()' and cpu offline. Before commit 5825bea05265("xfs: __percpu_counter_compare() inode count debug too expensive"). I got issue as follows when do cpu online/offline test: smpboot: CPU 1 is now offline XFS: Assertion failed: percpu_counter_compare(&mp->m_ifree, 0) >= 0, file: fs/xfs/xfs_trans.c, line: 622 ------------[ cut here ]------------ kernel BUG at fs/xfs/xfs_message.c:110! invalid opcode: 0000 [#1] SMP KASAN PTI CPU: 3 PID: 25512 Comm: fsstress Not tainted 5.10.0-04288-gcb31bdc8c65d #8 RIP: 0010:assfail+0x77/0x8b fs/xfs/xfs_message.c:110 RSP: 0018:ffff88810a5df5c0 EFLAGS: 00010293 RAX: ffff88810f3a8000 RBX: 0000000000000201 RCX: ffffffffaa8bd7c0 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 RBP: 0000000000000000 R08: ffff88810f3a8000 R09: ffffed103edf71cd R10: ffff8881f6fb8e67 R11: ffffed103edf71cc R12: ffffffffab0108c0 R13: ffffffffab010220 R14: ffffffffffffffff R15: 0000000000000000 FS: 00007f8536e16b80(0000) GS:ffff8881f6f80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005617e1115f44 CR3: 000000015873a005 CR4: 0000000000370ee0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: xfs_trans_unreserve_and_mod_sb+0x833/0xca0 fs/xfs/xfs_trans.c:622 xlog_cil_commit+0x1169/0x29b0 fs/xfs/xfs_log_cil.c:1325 __xfs_trans_commit+0x2c0/0xe20 fs/xfs/xfs_trans.c:889 xfs_create_tmpfile+0x6a6/0x9a0 fs/xfs/xfs_inode.c:1320 xfs_rename_alloc_whiteout fs/xfs/xfs_inode.c:3193 [inline] xfs_rename+0x58a/0x1e00 fs/xfs/xfs_inode.c:3245 xfs_vn_rename+0x28e/0x410 fs/xfs/xfs_iops.c:436 vfs_rename+0x10b5/0x1dd0 fs/namei.c:4329 do_renameat2+0xa19/0xb10 fs/namei.c:4474 __do_sys_renameat2 fs/namei.c:4512 [inline] __se_sys_renameat2 fs/namei.c:4509 [inline] __x64_sys_renameat2+0xe4/0x120 fs/namei.c:4509 do_syscall_64+0x33/0x40 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x61/0xc6 RIP: 0033:0x7f853623d91d I can reproduce above issue by injecting kernel latency to invalidate the quick judgment of “__percpu_counter_compare()”. For quick judgment logic, the number of CPUs may have decreased before calling percpu_counter_cpu_dead() when concurrent with CPU offline. That leads to calculation errors. For example: Assumption: (1) batch = 32 (2) The final count is 2 (3) The number of CPUs is 4 If the number of percpu variables on each CPU is as follows when CPU3 is offline: cpu0 cpu1 cpu2 cpu3 31 31 31 31 fbc->count = -122 -> 'percpu_counter_cpu_dead()' isn't called. So at this point, check if percpu counter is greater than 0. abs(count - rhs) = -122 batch * num_ online_ cpus() = 32 * 3 = 96 -> Online CPUs number become 3 That is: abs (count rhs) > batch * num_online_cpus() condition met. The actual value is 2, but the fact that count<0 returns -1 is the opposite. Ye Bin (2): cpu/hotplug: introduce 'num_dying_cpus' to get dying CPUs count lib/percpu_counter: fix dying cpu compare race include/linux/cpumask.h | 20 ++++++++++++++++---- kernel/cpu.c | 2 ++ lib/percpu_counter.c | 11 ++++++++++- 3 files changed, 28 insertions(+), 5 deletions(-)