From patchwork Mon Dec 11 11:25:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ye Bin X-Patchwork-Id: 176605 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6972567vqy; Mon, 11 Dec 2023 03:22:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IGS7f4sGvqiHl3XtyKVCr/VKc1q1nygzWSMPp4BrsNOm2Qc1KvLlHb+KhMORUR3FZXCsNsZ X-Received: by 2002:a17:903:2b04:b0:1d0:c28e:2ec with SMTP id mc4-20020a1709032b0400b001d0c28e02ecmr6094561plb.32.1702293742044; Mon, 11 Dec 2023 03:22:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702293742; cv=none; d=google.com; s=arc-20160816; b=p1v/9whrlAL8DkaAbqu8ErRg61+wHXzI2TJ8fqgYT0t2hAOTaDocdyJMUX1ygdi50X 1SJ3vws40RCvgvcevi21ePIJWqSrFLyRGTtgujcYhr2263yL9RkB8zTThMHOJQ9qgtdX CYJAywo9SmStTP3YG34ntzk1UPhgS6aSXOerm2OPnPTXmZrGDvRuUf16MSxf6Z8iEkrJ 0rG15LAuCYAmyqZSsb75uerVkRWVxW29jNsSnJmMrTvscHHbtZNkLsLhYaaGlx6FVnZA Z2cis20MnUWZNAQethWjgIjBD3yJKq/RqS6Zfz80Q3Elx15VhnNjiVQAOylaMn7c/+iG CDzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=DtENEzl3N8gVB1VpaISmVgQEBuHF4EOX97eYXmUi1yo=; fh=0YRSAtnrSRvhtzUzSfhpD62rtNa0OOsDCBO4xMqK++A=; b=0D8ck6PcmcuGVXzbh6Rp10/rFLp5TeW/YozwJWbAqNT2ZCZ+8ulMnJdvdhryRE9q+A 2ZmhzqtDsfH0tNW3O2YMCCKQ8NWwaxM+qInH73tEMCPer+qnR/T0+X7HFZhjp65YmsYJ YLdSDMi4DpnA5mSBKN5Ia/gIYwFEuDxoPB/2d8BBShrnCmrlKgnBCQYwzxETaM3b60zS l/iiBqguoZ4poN19FAMNIccyBtp2H0bujlbFEO0+h2AABoB20NzSf7TnDX/E7g/nL+W+ 2XIY8dOE8PA9bJJ2yU4XRAxRo/ff+XpGzeCZ/suv3ZnIKKrhcktpOMRVbP/eUKR303aI leFA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id h15-20020a170902f7cf00b001d006d3b84csi5805914plw.1.2023.12.11.03.22.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 03:22:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A50F180A30AD; Mon, 11 Dec 2023 03:22:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234700AbjLKLWI (ORCPT + 99 others); Mon, 11 Dec 2023 06:22:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231734AbjLKLWG (ORCPT ); Mon, 11 Dec 2023 06:22:06 -0500 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5802FAC; Mon, 11 Dec 2023 03:22:12 -0800 (PST) Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4SpfTm691yzsRjx; Mon, 11 Dec 2023 19:22:04 +0800 (CST) Received: from canpemm500010.china.huawei.com (unknown [7.192.105.118]) by mail.maildlp.com (Postfix) with ESMTPS id 2995918005A; Mon, 11 Dec 2023 19:22:10 +0800 (CST) Received: from huawei.com (10.175.127.227) by canpemm500010.china.huawei.com (7.192.105.118) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Mon, 11 Dec 2023 19:22:09 +0800 From: Ye Bin To: , , CC: , , Ye Bin Subject: [PATCH] jbd2: fix soft lockup in journal_finish_inode_data_buffers() Date: Mon, 11 Dec 2023 19:25:44 +0800 Message-ID: <20231211112544.3879780-1-yebin10@huawei.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Originating-IP: [10.175.127.227] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To canpemm500010.china.huawei.com (7.192.105.118) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 03:22:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784984362782787266 X-GMAIL-MSGID: 1784984362782787266 There's issue when do io test: WARN: soft lockup - CPU#45 stuck for 11s! [jbd2/dm-2-8:4170] CPU: 45 PID: 4170 Comm: jbd2/dm-2-8 Kdump: loaded Tainted: G OE Call trace: dump_backtrace+0x0/0x1a0 show_stack+0x24/0x30 dump_stack+0xb0/0x100 watchdog_timer_fn+0x254/0x3f8 __hrtimer_run_queues+0x11c/0x380 hrtimer_interrupt+0xfc/0x2f8 arch_timer_handler_phys+0x38/0x58 handle_percpu_devid_irq+0x90/0x248 generic_handle_irq+0x3c/0x58 __handle_domain_irq+0x68/0xc0 gic_handle_irq+0x90/0x320 el1_irq+0xcc/0x180 queued_spin_lock_slowpath+0x1d8/0x320 jbd2_journal_commit_transaction+0x10f4/0x1c78 [jbd2] kjournald2+0xec/0x2f0 [jbd2] kthread+0x134/0x138 ret_from_fork+0x10/0x18 Analyzed informations from vmcore as follows: (1) There are about 5k+ jbd2_inode in 'commit_transaction->t_inode_list'; (2) Now is processing the 855th jbd2_inode; (3) JBD2 task has TIF_NEED_RESCHED flag; (4) There's no pags in address_space around the 855th jbd2_inode; (5) There are some process is doing drop caches; (6) Mounted with 'nodioread_nolock' option; (7) 128 CPUs; According to informations from vmcore we know 'journal->j_list_lock' spin lock competition is fierce. So journal_finish_inode_data_buffers() maybe process slowly. Theoretically, there is scheduling point in the filemap_fdatawait_range_keep_errors(). However, if inode's address_space has no pages which taged with PAGECACHE_TAG_WRITEBACK, will not call cond_resched(). So may lead to soft lockup. journal_finish_inode_data_buffers filemap_fdatawait_range_keep_errors __filemap_fdatawait_range while (index <= end) nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end, PAGECACHE_TAG_WRITEBACK); if (!nr_pages) break; --> If 'nr_pages' is equal zero will break, then will not call cond_resched() for (i = 0; i < nr_pages; i++) wait_on_page_writeback(page); cond_resched(); To solve above issue, add scheduling point in the journal_finish_inode_data_buffers(); Signed-off-by: Ye Bin Reviewed-by: Jan Kara --- fs/jbd2/commit.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c index 9bdb377a348f..5e122586e06e 100644 --- a/fs/jbd2/commit.c +++ b/fs/jbd2/commit.c @@ -270,6 +270,7 @@ static int journal_finish_inode_data_buffers(journal_t *journal, if (!ret) ret = err; } + cond_resched(); spin_lock(&journal->j_list_lock); jinode->i_flags &= ~JI_COMMIT_RUNNING; smp_mb();