From patchwork Wed Feb 22 04:09:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Li Nan X-Patchwork-Id: 60335 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp370800wrd; Tue, 21 Feb 2023 20:03:49 -0800 (PST) X-Google-Smtp-Source: AK7set+h20hYm49Wa6MVkA4LAUVP7uiTVlCU2wlS0BAgFXCne/5lX96l6w8IPcMy8WCGOlTHrb9i X-Received: by 2002:a17:906:ce38:b0:8b2:8876:6a3c with SMTP id sd24-20020a170906ce3800b008b288766a3cmr17080602ejb.29.1677038629276; Tue, 21 Feb 2023 20:03:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677038629; cv=none; d=google.com; s=arc-20160816; b=odbbluwTJeRYw9PwZoOYh11V5n1W6ytNLpP16zoY+aJlgTfYjkrgfo9yI1/H4vdZr6 c0PptdUZh38++WCBocjI+f3dmVmrNBWgwbxDa4Bwwe9T6To5FTROXf31QPi8DnaaAKHs 4Vqhp2MYsR+3gvfXDotGLrYKNa/Nb5DGISsuzZny1IgqFbPdhsU4piIvutM96td8QKYA 23yuh+cScd1SH8YbBw8p5SltBin0Upuo1Si/yxs9KaEBMvvy3izXd61gCQxkmVNQV55d dOc8IvWHRjvPn863mmO68ga2WJwVG6AimJgi3mQPKZvA+RKQf8Hui/+9LKZcqVbc3H9L Ubag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=4p6Xw8/dmD8OWJDgOBteFk6peW+2G64XYsrym1wE2IM=; b=fEoBD66roR6YQLqMN8tlqcaZdbPust3wkwpiR/pzUoWzhgz+m1EQxE2KaGnidApAoc +yunfPEP9oKXGOJaW+3dQtGVs56cjjtFTk+0TbCkrgZOwGjz7nzqjYx2/EgyUUnxBv/9 +snRGhLVaf6umcaW+nN0lsQFzHu3Tt0Bbvr0xvE9LgG609t+hV9HmklzN6MVD9SjBj4l MRfK/hnnSDqPTeKCXYJfkkYQ+5hzTq136xi8M6phBBg7d1AbBCyHrTxS8EmGFeXFIXy9 /j5O67F05MX7u8YFOFqoBMqcDszjL2toD+Tc1cq07NrMunRdhFDH8L3vy/k+ro5vCAHT 4u0Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z25-20020aa7c659000000b004ace655ef51si21046108edr.192.2023.02.21.20.03.23; Tue, 21 Feb 2023 20:03:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231418AbjBVDqo (ORCPT + 99 others); Tue, 21 Feb 2023 22:46:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231196AbjBVDqi (ORCPT ); Tue, 21 Feb 2023 22:46:38 -0500 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0D8F2CC46; Tue, 21 Feb 2023 19:46:35 -0800 (PST) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4PM2Bs3YJHz4f3wYF; Wed, 22 Feb 2023 11:46:29 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP3 (Coremail) with SMTP id _Ch0CgA35CEVkPVjUibtDg--.64292S5; Wed, 22 Feb 2023 11:46:31 +0800 (CST) From: linan666@huaweicloud.com To: song@kernel.org, ncroxon@redhat.com, vmayatskikh@digitalocean.com Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com, axboe@kernel.dk, linan122@huawei.com, yukuai3@huawei.com, yi.zhang@huawei.com Subject: [PATCH 1/2] md/raid10: fix taks hung in raid10d Date: Wed, 22 Feb 2023 12:09:59 +0800 Message-Id: <20230222041000.3341651-2-linan666@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230222041000.3341651-1-linan666@huaweicloud.com> References: <20230222041000.3341651-1-linan666@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgA35CEVkPVjUibtDg--.64292S5 X-Coremail-Antispam: 1UD129KBjvJXoWxZr1UtF1xurWrCr1xJw1UWrg_yoW5Aw4fp3 yfGrn3Ary5Ga4ayF1DtF1DC34F93yfJFW3CFZ3W34xZ3WDZFZ3JFWDXFWY9ryDZr95Zay5 XFZFy3yYkF47tFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPEb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1lnxkEFVAIw20F6cxK64vIFxWle2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64 xvF2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j 6r4UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2 kIc2xKxwAKzVCY07xG64k0F24l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_ Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17 CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0 I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I 8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73 UjIFyTuYvjxUc6wZUUUUU X-CM-SenderInfo: polqt0awwwqx5xdzvxpfor3voofrz/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758502457667690357?= X-GMAIL-MSGID: =?utf-8?q?1758502457667690357?= From: Li Nan commit fe630de009d0 ("md/raid10: avoid deadlock on recovery.") allowed normal io and sync io to exist at the same time. Task hung will occur as below: T1 T2 T3 T4 raid10d handle_read_error allow_barrier conf->nr_pending-- -> 0 //submit sync io raid10_sync_request raise_barrier ->will not be blocked ... //submit to drivers raid10_read_request wait_barrier conf->nr_pending++ -> 1 //retry read fail raid10_end_read_request reschedule_retry add to retry_list conf->nr_queued++ -> 1 //sync io fail end_sync_read __end_sync_read reschedule_retry add to retry_list conf->nr_queued++ -> 2 ... handle_read_error get form retry_list conf->nr_queued-- freeze_array wait nr_pending == nr_queued+1 ->1 ->2 //task hung retry read and sync io will be added to retry_list(nr_queued->2) if they fails. raid10d() called handle_read_error() and hung in freeze_array(). nr_queued will not decrease because raid10d is blocked, nr_pending will not increase because conf->barrier is not released. Fix it by moving allow_barrier() after raid10_read_request(). raise_barrier() will wait for nr_waiting to become 0. Therefore, sync io and regular io will not be issued at the same time. We also removed the check of nr_queued. It can be 0 but don't need to be blocked. MD_RECOVERY_RUNNING always is set after this patch, because all sync io is waitting in raise_barrier(), remove it, too. Fixes: fe630de009d0 ("md/raid10: avoid deadlock on recovery.") Signed-off-by: Li Nan --- drivers/md/raid10.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 6c66357f92f5..db9ee3b637d6 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -995,11 +995,15 @@ static bool stop_waiting_barrier(struct r10conf *conf) (!bio_list_empty(&bio_list[0]) || !bio_list_empty(&bio_list[1]))) return true; - /* move on if recovery thread is blocked by us */ - if (conf->mddev->thread->tsk == current && - test_bit(MD_RECOVERY_RUNNING, &conf->mddev->recovery) && - conf->nr_queued > 0) + /* + * move on if io is issued from raid10d(), nr_pending is not released + * from original io(see handle_read_error()). All raise barrier is + * blocked until this io is done. + */ + if (conf->mddev->thread->tsk == current) { + WARN_ON_ONCE(atomic_read(&conf->nr_pending) == 0); return true; + } return false; } @@ -2978,9 +2982,13 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) md_error(mddev, rdev); rdev_dec_pending(rdev, mddev); - allow_barrier(conf); r10_bio->state = 0; raid10_read_request(mddev, r10_bio->master_bio, r10_bio); + /* + * allow_barrier after re-submit to ensure no sync io + * can be issued while regular io pending. + */ + allow_barrier(conf); } static void handle_write_completed(struct r10conf *conf, struct r10bio *r10_bio) From patchwork Wed Feb 22 04:10:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Li Nan X-Patchwork-Id: 60336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp371361wrd; Tue, 21 Feb 2023 20:05:17 -0800 (PST) X-Google-Smtp-Source: AK7set/qHv3MgJtZOgRTq0wGhZIf6KnwUF3LdAQLGhu1rhn1xukYOkycqm3Bjn/tV4p786ekIFcd X-Received: by 2002:a17:906:f8c6:b0:882:cdd4:14d9 with SMTP id lh6-20020a170906f8c600b00882cdd414d9mr11861105ejb.46.1677038717322; Tue, 21 Feb 2023 20:05:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677038717; cv=none; d=google.com; s=arc-20160816; b=V0vnjLVZdNRMN1DRo3cUc/wuPL1WiL/Mx/qGYBYVoS71T2cJUjTqOeonplaUU39yUD K4bZWIARWf15PyXtLK52J+xNH5m++zsESsksDV0G+Aegp3u48ueqEan4sDpgIBesEvjJ JWfKm16LYCAm+K6Ua6iYAO4xltZAOi7X5NnbxEHjU9sIikOFSjCkcPBe32XysmZYCHRj b0IWGDqCPctVIYtqp6zAAbdQ7DWmzKqGC7oyeMglxOi6SpQGmRlP4c2NhCHihk1Sblxl RWHARFmGExgvxGWIFq9Il2XKHwCDvBYIJPxdGXMi3d43nraKvc+X2ubufRLet08msD6U K1cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=JrxPrlIkZUm6sPuG5jMRPek7PzFAVKm35rmI0L/FYuk=; b=rEVHD+vCm5ESZ32VG8HaalXqQfWi+ClXcEtP4fMS+/nySirsirt1vauCqoXdaK566P FQWf3UKEowtd4nPK76hT8i6leu5UWw9uqh2gb4zDBrnWb/vqD4AMxLGugEileY7lpJM/ vm7QUNnAC7nsGsWfWq/LLWP/RkCgvUdCdjoGEJApVxo3Y0lo2Cwkr3kv+b7VBzT0RMg+ KT/Fi8bIPOIKO+o8u3lcV/P9AARJVj5gw3o+zmrVJZ+CIf/6uqpmNpvdl5OlhcYvLVqo 1tMBF17+La7JihgzhIjJc2fAGu2hKCBPhJF3jKZ5EL+iDYF7F9SFk32x10hpVuFJn5OL ExpQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e9-20020a170906748900b008d074999798si9666697ejl.96.2023.02.21.20.04.51; Tue, 21 Feb 2023 20:05:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231382AbjBVDql (ORCPT + 99 others); Tue, 21 Feb 2023 22:46:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229930AbjBVDqh (ORCPT ); Tue, 21 Feb 2023 22:46:37 -0500 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0B8A1167D; Tue, 21 Feb 2023 19:46:35 -0800 (PST) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4PM2Bs63VDz4f3l88; Wed, 22 Feb 2023 11:46:29 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP3 (Coremail) with SMTP id _Ch0CgA35CEVkPVjUibtDg--.64292S6; Wed, 22 Feb 2023 11:46:31 +0800 (CST) From: linan666@huaweicloud.com To: song@kernel.org, ncroxon@redhat.com, vmayatskikh@digitalocean.com Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com, axboe@kernel.dk, linan122@huawei.com, yukuai3@huawei.com, yi.zhang@huawei.com Subject: [PATCH 2/2] md/raid10: fix null-ptr-deref in raid10_sync_request Date: Wed, 22 Feb 2023 12:10:00 +0800 Message-Id: <20230222041000.3341651-3-linan666@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230222041000.3341651-1-linan666@huaweicloud.com> References: <20230222041000.3341651-1-linan666@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgA35CEVkPVjUibtDg--.64292S6 X-Coremail-Antispam: 1UD129KBjvJXoW7CFyfGr4rKF15JF4ktFyDGFg_yoW8ZFWfpa nrXwnxtrW8W39Yya1kJw17WFyF934xJ3y5tr4fu3s3CFn5WFW7ArW5Kay2qFyUXryrtFWU X3yUJrW5CFn8AaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmjb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2vYz4IE04k24VAvwVAKI4IrM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrV ACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWU JVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2 ka0xkIwI1lw4CEc2x0rVAKj4xxMxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j 6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7 AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE 2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcV C2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2Kfnx nUUI43ZEXa7IU1H7K7UUUUU== X-CM-SenderInfo: polqt0awwwqx5xdzvxpfor3voofrz/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758502549778879489?= X-GMAIL-MSGID: =?utf-8?q?1758502549778879489?= From: Li Nan init_resync() init mempool and set conf->have_replacemnt at the begaining of sync, close_sync() free the mempool when sync is completed. After commit 7e83ccbecd60 ("md/raid10: Allow skipping recovery when clean arrays are assembled"), recovery might skipped and init_resync() is called but close_sync() is not. null-ptr-deref occurs as below: 1) creat a array, wait for resync to complete, mddev->recovery_cp is set to MaxSector. 2) recovery is woken and it is skipped. conf->have_replacement is set to 0 in init_resync(). close_sync() not called. 3) some io errors and rdev A is set to WantReplacement. 4) a new device is added and set to A's replacement. 5) recovery is woken, A have replacement, but conf->have_replacemnt is 0. r10bio->dev[i].repl_bio will not be alloced and null-ptr-deref occurs. Fix it by not init_resync() if recovery skipped. Fixes: 7e83ccbecd60 md/raid10: Allow skipping recovery when clean arrays are assembled") Signed-off-by: Li Nan --- drivers/md/raid10.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index db9ee3b637d6..9e0e7bf524aa 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -3297,10 +3297,6 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr, sector_t chunk_mask = conf->geo.chunk_mask; int page_idx = 0; - if (!mempool_initialized(&conf->r10buf_pool)) - if (init_resync(conf)) - return 0; - /* * Allow skipping a full rebuild for incremental assembly * of a clean array, like RAID1 does. @@ -3316,6 +3312,10 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr, return mddev->dev_sectors - sector_nr; } + if (!mempool_initialized(&conf->r10buf_pool)) + if (init_resync(conf)) + return 0; + skipped: max_sector = mddev->dev_sectors; if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery) ||