From patchwork Tue Jul 25 13:00:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 125627 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp2482575vqg; Tue, 25 Jul 2023 06:49:07 -0700 (PDT) X-Google-Smtp-Source: APBJJlGu+YCeXKWSWGzX+dik5ToHfLZ73DZrmScAlobO1c3WttKSif8gZ65I/MViedjJzF720L4C X-Received: by 2002:a17:907:7846:b0:994:8e9:67fe with SMTP id lb6-20020a170907784600b0099408e967femr11410108ejc.35.1690292947490; Tue, 25 Jul 2023 06:49:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690292947; cv=none; d=google.com; s=arc-20160816; b=tcwsfXTOcxAB/CqP1mR+Dxu85IH79tN1M9Quw8+eNZN6n9oA5T14bqsNcgqT5XeyZB pJgWlcy6rCAXqlSG0RUxBHlCdY9vf0+bvOR/RrHzkQylnHbmbilDXIK+N4m31cRRXogv i2AKuPiuCyal37F5EhyeXW3ha8xfwpMLZo1oe0B2YAXarBwSSoISH0eQER6o/7LVtw90 xYt5ZS2hgGKCvOgo0JrgxlGVcG0Sq1TUbdl1xX6KUMX6eB256m6sh2JxHaiB0Vb9BjqQ icF2Gvve7pXgWbnbUN/Vz5GPQIyEHVUHJk2YZDofbsn/6nGERQVZTcNtm2Rz6gb4l1it XGCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/QeLRD3PFm/UhjbcDO63KGqnRYCXFOsqKzmt3mk4Vfs=; fh=4fzGWvE2LW7K10hgePOrl39slR3lbqz4nggQweyfPdc=; b=eOciJsvxAPQGZwalPgnheicIE9oHfLGVUXhqZYRNCSbAnXTHhzBFiwRZ4k6gzH1hcy YIIOwueohXdnEJTUgJZRU3NXIAY69zjGoORZd8sNAUzdpPtxOqlQv4sa4vD+xpsJ9k1L Sz3SmyPE//iFEQ3367y+jakfz+gIkmaXKf2Mfcjrbj7olFmD53yY2sBoCc/NljPkFCci PIqHDo+au90RWlm+0dAqS9wPkiTCPu9lky9rhAmnFzacuohmYzQaYojA9C+zBVVSc65b nDwMf0WdA+2hClrCL1QDNp5yYO1zhs7E1tBIDTfqRVF5AO28UuVHnbH40H+PNtGL/hK5 BA4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dYoyHX5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v23-20020a1709060b5700b009888b517932si8026881ejg.333.2023.07.25.06.48.43; Tue, 25 Jul 2023 06:49:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dYoyHX5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229891AbjGYNTG (ORCPT + 99 others); Tue, 25 Jul 2023 09:19:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229830AbjGYNTF (ORCPT ); Tue, 25 Jul 2023 09:19:05 -0400 Received: from out-27.mta0.migadu.com (out-27.mta0.migadu.com [91.218.175.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5825F19BA for ; Tue, 25 Jul 2023 06:18:59 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1690291137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/QeLRD3PFm/UhjbcDO63KGqnRYCXFOsqKzmt3mk4Vfs=; b=dYoyHX5VNzJ1aicvzztBVYudt9ptphJkTO29hYSGPPfG0Z0U7+n46WMtbxZ4J7Q35KtYcO VI3RNCkVOFCy/kp957wNpvXCnsfHnxP/W8pdkhTwK+I9mj16ZUuVEqp1sFFDxTF5iSA752 y0QkHtWqGy1/KK6cPZOi18C8u5+TCYc= From: chengming.zhou@linux.dev To: axboe@kernel.dk, hch@lst.de, ming.lei@redhat.com Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH v2 1/4] blk-flush: flush_rq should inherit first_rq's cmd_flags Date: Tue, 25 Jul 2023 21:00:59 +0800 Message-ID: <20230725130102.3030032-2-chengming.zhou@linux.dev> In-Reply-To: <20230725130102.3030032-1-chengming.zhou@linux.dev> References: <20230725130102.3030032-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772400617812067378 X-GMAIL-MSGID: 1772400617812067378 From: Chengming Zhou The cmd_flags in blk_kick_flush() should inherit the original request's cmd_flags, but the current code looks buggy to me: flush_end_io() blk_flush_complete_seq() // requests on flush running list blk_kick_flush() So the request passed to blk_flush_complete_seq() may will be ended before blk_kick_flush(). On the other hand, flush_rq will inherit first_rq's tag, it should use first_rq's cmd_flags too. This patch is just preparation for the following patches, no bugfix intended. Signed-off-by: Chengming Zhou --- block/blk-flush.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index e73dc22d05c1..fc25228f7bb1 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -92,7 +92,7 @@ enum { }; static void blk_kick_flush(struct request_queue *q, - struct blk_flush_queue *fq, blk_opf_t flags); + struct blk_flush_queue *fq); static inline struct blk_flush_queue * blk_get_flush_queue(struct request_queue *q, struct blk_mq_ctx *ctx) @@ -166,11 +166,9 @@ static void blk_flush_complete_seq(struct request *rq, { struct request_queue *q = rq->q; struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; - blk_opf_t cmd_flags; BUG_ON(rq->flush.seq & seq); rq->flush.seq |= seq; - cmd_flags = rq->cmd_flags; if (likely(!error)) seq = blk_flush_cur_seq(rq); @@ -210,7 +208,7 @@ static void blk_flush_complete_seq(struct request *rq, BUG(); } - blk_kick_flush(q, fq, cmd_flags); + blk_kick_flush(q, fq); } static enum rq_end_io_ret flush_end_io(struct request *flush_rq, @@ -277,7 +275,6 @@ bool is_flush_rq(struct request *rq) * blk_kick_flush - consider issuing flush request * @q: request_queue being kicked * @fq: flush queue - * @flags: cmd_flags of the original request * * Flush related states of @q have changed, consider issuing flush request. * Please read the comment at the top of this file for more info. @@ -286,8 +283,7 @@ bool is_flush_rq(struct request *rq) * spin_lock_irq(fq->mq_flush_lock) * */ -static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, - blk_opf_t flags) +static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) { struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; struct request *first_rq = @@ -336,7 +332,8 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, flush_rq->internal_tag = first_rq->internal_tag; flush_rq->cmd_flags = REQ_OP_FLUSH | REQ_PREFLUSH; - flush_rq->cmd_flags |= (flags & REQ_DRV) | (flags & REQ_FAILFAST_MASK); + flush_rq->cmd_flags |= (first_rq->cmd_flags & REQ_DRV) | + (first_rq->cmd_flags & REQ_FAILFAST_MASK); flush_rq->rq_flags |= RQF_FLUSH_SEQ; flush_rq->end_io = flush_end_io; /* From patchwork Tue Jul 25 13:01:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 125648 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp2508883vqg; Tue, 25 Jul 2023 07:26:05 -0700 (PDT) X-Google-Smtp-Source: APBJJlG89sa6H0cw7A4KGR4zEXhM6Gc4FIvvozxiuWfMJgOz5LWi6OHE7YLPdobYKUlfN9cCcFoV X-Received: by 2002:a17:902:694c:b0:1b8:6cac:ffe8 with SMTP id k12-20020a170902694c00b001b86cacffe8mr12856774plt.51.1690295164965; Tue, 25 Jul 2023 07:26:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690295164; cv=none; d=google.com; s=arc-20160816; b=sDsBaWEqUjtf7RTxRufKEygw8oOxdlgAJVgV0rr0Ts8hFi0wyf0roD5z7v5C0hKv55 RJE0pzHk6diAsRAG2qnT0mUcnmQjReJ+G6ObFL3vfRKo/zMzlkJZWDP+aHfmhK5gvLno otMpOxcEwDcwDanD/Zou8XrmHe2CfdrF6M6htonkkWg6Zg6/EhxFCqNaKiOZkMsob/cB yXS8no4Ikxukl3yS0mBJV13fwVgwUWRB0snDuwRT+CoCCzh37kXx71HVnL49Iaz7Q5gh Yacq/ChWzlxdYL/hmiFRX+FSLogTdYHt7CDmfBb+VKiuqPsarL5l0Z0jyxivPzwVdUbG Cclg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fMRjU4cz+GWWgAQcR8jyEbZWLfFxQN9dbiOOgRzcy7Q=; fh=4fzGWvE2LW7K10hgePOrl39slR3lbqz4nggQweyfPdc=; b=iu3TAf/oKVNAg8/BNzldVb3q3QVdD9REo7dPF2MSpJHdlRtpl3lSROlq6xSOx8G08O y72jmLzq4mJaxH3C46S85PAiVEhwzdl9fU/8UVqHpMn61fZbKFzwkiwofiayApczvbtW h40Le4AiS+C0ZyXlwpgYKW9XCN6a0wMCDcO/PR2+pdIkd84uuXF2TBHHLQpnp60jsiLE 92W3aj/xrJ7BUlcP77bpB+7GBQ0rWGYP3ZzeXXWfb072WCEcwmqUetRlqL8+/eNQ5Cwn k0YjXl+pm2KnXOPIOR+zuiGnvGAU5ieCwQUhHWgD5GLEstYqFraZl/T3TjNejlShpPmx 3G4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=o0QLFYhh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u8-20020a1709026e0800b001b873b80747si10951077plk.39.2023.07.25.07.25.50; Tue, 25 Jul 2023 07:26:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=o0QLFYhh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229945AbjGYNTO (ORCPT + 99 others); Tue, 25 Jul 2023 09:19:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229926AbjGYNTM (ORCPT ); Tue, 25 Jul 2023 09:19:12 -0400 Received: from out-51.mta0.migadu.com (out-51.mta0.migadu.com [91.218.175.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75BE0173B for ; Tue, 25 Jul 2023 06:19:06 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1690291143; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fMRjU4cz+GWWgAQcR8jyEbZWLfFxQN9dbiOOgRzcy7Q=; b=o0QLFYhhnAKVPgEuwksv90YzlrU3SZDj1gKO8UX5Zp2fr5dHEDB+NhvBCQhO80EseFtGx1 ojhQ2JJTZEnpq2r6H5gTEZqAL5C4GGXMbHGD5Txk/k00Xy7sjh/tYn7YdFvkD9w1vamZY8 2cCxukur8gFBFNgRa4BZ/ZoB7DB29Eo= From: chengming.zhou@linux.dev To: axboe@kernel.dk, hch@lst.de, ming.lei@redhat.com Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH v2 2/4] blk-flush: split queues for preflush and postflush requests Date: Tue, 25 Jul 2023 21:01:00 +0800 Message-ID: <20230725130102.3030032-3-chengming.zhou@linux.dev> In-Reply-To: <20230725130102.3030032-1-chengming.zhou@linux.dev> References: <20230725130102.3030032-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772402942859747534 X-GMAIL-MSGID: 1772402942859747534 From: Chengming Zhou We don't need to replace rq->end_io to make it return to the flush state machine if it doesn't need post-flush. The previous approach [1] we take is to move blk_rq_init_flush() to REQ_FSEQ_DATA stage and only replace rq->end_io if it needs post-flush. Otherwise, it can end like normal request and doesn't need to return back to the flush state machine. But this way add more magic to the already way too magic flush sequence. Christoph suggested that we can kill the flush sequence entirely, and just split the flush_queue into a preflush and a postflush queue. The reason we need separate queues for preflush and postflush requests is that in flush_end_io(), we need to handle differently: end request for postflush requests, but requeue dispatch for preflush requests. This patch is just in preparation for the following patches, no functional changes intended. [1] https://lore.kernel.org/lkml/20230710133308.GB23157@lst.de/ Signed-off-by: Chengming Zhou --- block/blk-flush.c | 50 +++++++++++++++++++++++++++++++++++------------ block/blk.h | 3 ++- 2 files changed, 40 insertions(+), 13 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index fc25228f7bb1..4993c3c3b502 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -165,7 +165,7 @@ static void blk_flush_complete_seq(struct request *rq, unsigned int seq, blk_status_t error) { struct request_queue *q = rq->q; - struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; + struct list_head *pending; BUG_ON(rq->flush.seq & seq); rq->flush.seq |= seq; @@ -177,9 +177,9 @@ static void blk_flush_complete_seq(struct request *rq, switch (seq) { case REQ_FSEQ_PREFLUSH: - case REQ_FSEQ_POSTFLUSH: + pending = &fq->preflush_queue[fq->flush_pending_idx]; /* queue for flush */ - if (list_empty(pending)) + if (!fq->flush_pending_since) fq->flush_pending_since = jiffies; list_move_tail(&rq->queuelist, pending); break; @@ -192,6 +192,14 @@ static void blk_flush_complete_seq(struct request *rq, blk_mq_kick_requeue_list(q); break; + case REQ_FSEQ_POSTFLUSH: + pending = &fq->postflush_queue[fq->flush_pending_idx]; + /* queue for flush */ + if (!fq->flush_pending_since) + fq->flush_pending_since = jiffies; + list_move_tail(&rq->queuelist, pending); + break; + case REQ_FSEQ_DONE: /* * @rq was previously adjusted by blk_insert_flush() for @@ -215,7 +223,7 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq, blk_status_t error) { struct request_queue *q = flush_rq->q; - struct list_head *running; + struct list_head *preflush_running, *postflush_running; struct request *rq, *n; unsigned long flags = 0; struct blk_flush_queue *fq = blk_get_flush_queue(q, flush_rq->mq_ctx); @@ -248,14 +256,22 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq, flush_rq->internal_tag = BLK_MQ_NO_TAG; } - running = &fq->flush_queue[fq->flush_running_idx]; + preflush_running = &fq->preflush_queue[fq->flush_running_idx]; + postflush_running = &fq->postflush_queue[fq->flush_running_idx]; BUG_ON(fq->flush_pending_idx == fq->flush_running_idx); /* account completion of the flush request */ fq->flush_running_idx ^= 1; /* and push the waiting requests to the next stage */ - list_for_each_entry_safe(rq, n, running, queuelist) { + list_for_each_entry_safe(rq, n, preflush_running, queuelist) { + unsigned int seq = blk_flush_cur_seq(rq); + + BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH); + blk_flush_complete_seq(rq, fq, seq, error); + } + + list_for_each_entry_safe(rq, n, postflush_running, queuelist) { unsigned int seq = blk_flush_cur_seq(rq); BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH); @@ -285,13 +301,20 @@ bool is_flush_rq(struct request *rq) */ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) { - struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; - struct request *first_rq = - list_first_entry(pending, struct request, queuelist); + struct list_head *preflush_pending = &fq->preflush_queue[fq->flush_pending_idx]; + struct list_head *postflush_pending = &fq->postflush_queue[fq->flush_pending_idx]; + struct request *first_rq = NULL; struct request *flush_rq = fq->flush_rq; /* C1 described at the top of this file */ - if (fq->flush_pending_idx != fq->flush_running_idx || list_empty(pending)) + if (fq->flush_pending_idx != fq->flush_running_idx) + return; + + if (!list_empty(preflush_pending)) + first_rq = list_first_entry(preflush_pending, struct request, queuelist); + else if (!list_empty(postflush_pending)) + first_rq = list_first_entry(postflush_pending, struct request, queuelist); + else return; /* C2 and C3 */ @@ -305,6 +328,7 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) * different from running_idx, which means flush is in flight. */ fq->flush_pending_idx ^= 1; + fq->flush_pending_since = 0; blk_rq_init(q, flush_rq); @@ -496,8 +520,10 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size, if (!fq->flush_rq) goto fail_rq; - INIT_LIST_HEAD(&fq->flush_queue[0]); - INIT_LIST_HEAD(&fq->flush_queue[1]); + INIT_LIST_HEAD(&fq->preflush_queue[0]); + INIT_LIST_HEAD(&fq->preflush_queue[1]); + INIT_LIST_HEAD(&fq->postflush_queue[0]); + INIT_LIST_HEAD(&fq->postflush_queue[1]); return fq; diff --git a/block/blk.h b/block/blk.h index 686712e13835..1a11675152ac 100644 --- a/block/blk.h +++ b/block/blk.h @@ -20,7 +20,8 @@ struct blk_flush_queue { unsigned int flush_running_idx:1; blk_status_t rq_status; unsigned long flush_pending_since; - struct list_head flush_queue[2]; + struct list_head preflush_queue[2]; + struct list_head postflush_queue[2]; unsigned long flush_data_in_flight; struct request *flush_rq; }; From patchwork Tue Jul 25 13:01:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 125637 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp2505860vqg; Tue, 25 Jul 2023 07:21:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlEafk+vQ3zUFuEvFdZmOwSsktmQEOusz8i8Cn0riA1pkOVlB4z/SCXe5ObkapA6nY+zSRai X-Received: by 2002:a17:906:cc49:b0:991:f427:2fd8 with SMTP id mm9-20020a170906cc4900b00991f4272fd8mr12067333ejb.74.1690294878179; Tue, 25 Jul 2023 07:21:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690294878; cv=none; d=google.com; s=arc-20160816; b=Bn6boIG/PghnkAqGG9zQOcv7id8dFbS+jFu2lpYM3OlyKrjuptlH1y968/2Aieek4b pF9Yqnd9h+ha4TB8AsHgGHc5Pp877ziEQUWDGO0OrUQvbw1SlMh9QBWIe1jmtZFmJnrc usy46WpJmM7CYA6Kytz0EZbTPSKmMLG3JI/Tn4y6vw6H7Y8mXeaYOkp/EFggYfExA7bN 5dInNfn/fvHtOgIZUCcNk4gmhqV4mrU/USg5lDVYbNyyLDme7hQxVgXicjRqGibe5t6j m7gcSYsoVkl00jtojlcVrtj5A70uFoUCMX0B2q0/7qyuL+nrHpSlRwNCufRfVuOzq3hB /PEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XANlsJMqfDtLDth7AmpKRAOz7/FxobYtfojym2xLXV0=; fh=4fzGWvE2LW7K10hgePOrl39slR3lbqz4nggQweyfPdc=; b=xGLAIhbyIg6IgEtGbQBhCqWlhGwCPcq5Pr8i0PVwFCMJaEsSHp3QzinHR2UmC8XMqO ObONT4tsOo9egSr/V/YyPMT9k6Uu6PrR7Ev0iHkS3yY40XTUY540HDjc8ynauuCulCNG GZS4c1bRzUnp1fWtodBRF17df/t7gl1lyqt3xOtn/NoU7cazJl4gHku6aV925vttCBPO dhmFcFyYUvNZ6OqhvJbpuxezBulQrKvLuDnR16xTEYZDo5ObP5sJM2Jh4/u1TkjQgwHv wMXvtlygrNdZ+3t/1aJLz8fh1f/GINZT8yKvsF47wc11R0L+mZSO/cV1uwzbLXdCuirv eapA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=b0plt3Bt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k12-20020a1709061c0c00b00991f9e2a81asi8306272ejg.238.2023.07.25.07.20.53; Tue, 25 Jul 2023 07:21:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=b0plt3Bt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229972AbjGYNTQ (ORCPT + 99 others); Tue, 25 Jul 2023 09:19:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229903AbjGYNTN (ORCPT ); Tue, 25 Jul 2023 09:19:13 -0400 Received: from out-13.mta0.migadu.com (out-13.mta0.migadu.com [IPv6:2001:41d0:1004:224b::d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB1281BF8 for ; Tue, 25 Jul 2023 06:19:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1690291146; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XANlsJMqfDtLDth7AmpKRAOz7/FxobYtfojym2xLXV0=; b=b0plt3BtcFnwzAyKOqNYzpGkxcY9ezH+pdE4wwMVaZlJVquMLw8y9nCJvDM0EfG+FlXAEM AqmMF4GWh4ESt29PaRcDXbcDwePYOkfQ3cVxKJcqlqBf4T/cCGXktU/ETZw9Kqja5f7Cqj rhcEj39Ek3nr8MOtTdFcpFsOtbU4Pok= From: chengming.zhou@linux.dev To: axboe@kernel.dk, hch@lst.de, ming.lei@redhat.com Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH v2 3/4] blk-flush: kill the flush state machine Date: Tue, 25 Jul 2023 21:01:01 +0800 Message-ID: <20230725130102.3030032-4-chengming.zhou@linux.dev> In-Reply-To: <20230725130102.3030032-1-chengming.zhou@linux.dev> References: <20230725130102.3030032-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772402642041539196 X-GMAIL-MSGID: 1772402642041539196 From: Chengming Zhou Since now we put preflush and postflush requests in separate queues, we don't need the flush sequence to record anymore. REQ_FSEQ_PREFLUSH: blk_enqueue_preflush() REQ_FSEQ_POSTFLUSH: blk_enqueue_postflush() REQ_FSEQ_DONE: blk_end_flush() In blk_flush_complete(), we have two list to handle: preflush_running and postflush_running. We just blk_end_flush() directly for postflush requests, but need to move preflush requests to requeue_list to dispatch. This patch just kill the flush state machine and directly call these functions, in preparation for the next patch. Signed-off-by: Chengming Zhou --- block/blk-flush.c | 158 ++++++++++++++++++----------------------- include/linux/blk-mq.h | 1 - 2 files changed, 70 insertions(+), 89 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index 4993c3c3b502..ed195c760617 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -117,11 +117,6 @@ static unsigned int blk_flush_policy(unsigned long fflags, struct request *rq) return policy; } -static unsigned int blk_flush_cur_seq(struct request *rq) -{ - return 1 << ffz(rq->flush.seq); -} - static void blk_flush_restore_request(struct request *rq) { /* @@ -147,75 +142,81 @@ static void blk_account_io_flush(struct request *rq) part_stat_unlock(); } -/** - * blk_flush_complete_seq - complete flush sequence - * @rq: PREFLUSH/FUA request being sequenced - * @fq: flush queue - * @seq: sequences to complete (mask of %REQ_FSEQ_*, can be zero) - * @error: whether an error occurred - * - * @rq just completed @seq part of its flush sequence, record the - * completion and trigger the next step. - * - * CONTEXT: - * spin_lock_irq(fq->mq_flush_lock) - */ -static void blk_flush_complete_seq(struct request *rq, - struct blk_flush_queue *fq, - unsigned int seq, blk_status_t error) +static void blk_enqueue_preflush(struct request *rq, struct blk_flush_queue *fq) { struct request_queue *q = rq->q; - struct list_head *pending; + struct list_head *pending = &fq->preflush_queue[fq->flush_pending_idx]; - BUG_ON(rq->flush.seq & seq); - rq->flush.seq |= seq; + if (!fq->flush_pending_since) + fq->flush_pending_since = jiffies; + list_move_tail(&rq->queuelist, pending); - if (likely(!error)) - seq = blk_flush_cur_seq(rq); - else - seq = REQ_FSEQ_DONE; + blk_kick_flush(q, fq); +} - switch (seq) { - case REQ_FSEQ_PREFLUSH: - pending = &fq->preflush_queue[fq->flush_pending_idx]; - /* queue for flush */ - if (!fq->flush_pending_since) - fq->flush_pending_since = jiffies; - list_move_tail(&rq->queuelist, pending); - break; +static void blk_enqueue_postflush(struct request *rq, struct blk_flush_queue *fq) +{ + struct request_queue *q = rq->q; + struct list_head *pending = &fq->postflush_queue[fq->flush_pending_idx]; - case REQ_FSEQ_DATA: - fq->flush_data_in_flight++; - spin_lock(&q->requeue_lock); - list_move(&rq->queuelist, &q->requeue_list); - spin_unlock(&q->requeue_lock); - blk_mq_kick_requeue_list(q); - break; + if (!fq->flush_pending_since) + fq->flush_pending_since = jiffies; + list_move_tail(&rq->queuelist, pending); - case REQ_FSEQ_POSTFLUSH: - pending = &fq->postflush_queue[fq->flush_pending_idx]; - /* queue for flush */ - if (!fq->flush_pending_since) - fq->flush_pending_since = jiffies; - list_move_tail(&rq->queuelist, pending); - break; + blk_kick_flush(q, fq); +} - case REQ_FSEQ_DONE: - /* - * @rq was previously adjusted by blk_insert_flush() for - * flush sequencing and may already have gone through the - * flush data request completion path. Restore @rq for - * normal completion and end it. - */ - list_del_init(&rq->queuelist); - blk_flush_restore_request(rq); - blk_mq_end_request(rq, error); - break; +static void blk_end_flush(struct request *rq, struct blk_flush_queue *fq, + blk_status_t error) +{ + struct request_queue *q = rq->q; - default: - BUG(); + /* + * @rq was previously adjusted by blk_insert_flush() for + * flush sequencing and may already have gone through the + * flush data request completion path. Restore @rq for + * normal completion and end it. + */ + list_del_init(&rq->queuelist); + blk_flush_restore_request(rq); + blk_mq_end_request(rq, error); + + blk_kick_flush(q, fq); +} + +static void blk_flush_complete(struct request_queue *q, + struct blk_flush_queue *fq, + blk_status_t error) +{ + unsigned int nr_requeue = 0; + struct list_head *preflush_running; + struct list_head *postflush_running; + struct request *rq, *n; + + preflush_running = &fq->preflush_queue[fq->flush_running_idx]; + postflush_running = &fq->postflush_queue[fq->flush_running_idx]; + + list_for_each_entry_safe(rq, n, postflush_running, queuelist) { + blk_end_flush(rq, fq, error); } + list_for_each_entry_safe(rq, n, preflush_running, queuelist) { + if (unlikely(error || !blk_rq_sectors(rq))) + blk_end_flush(rq, fq, error); + else + nr_requeue++; + } + + if (nr_requeue) { + fq->flush_data_in_flight += nr_requeue; + spin_lock(&q->requeue_lock); + list_splice_init(preflush_running, &q->requeue_list); + spin_unlock(&q->requeue_lock); + blk_mq_kick_requeue_list(q); + } + + /* account completion of the flush request */ + fq->flush_running_idx ^= 1; blk_kick_flush(q, fq); } @@ -223,8 +224,6 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq, blk_status_t error) { struct request_queue *q = flush_rq->q; - struct list_head *preflush_running, *postflush_running; - struct request *rq, *n; unsigned long flags = 0; struct blk_flush_queue *fq = blk_get_flush_queue(q, flush_rq->mq_ctx); @@ -256,27 +255,9 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq, flush_rq->internal_tag = BLK_MQ_NO_TAG; } - preflush_running = &fq->preflush_queue[fq->flush_running_idx]; - postflush_running = &fq->postflush_queue[fq->flush_running_idx]; BUG_ON(fq->flush_pending_idx == fq->flush_running_idx); - /* account completion of the flush request */ - fq->flush_running_idx ^= 1; - - /* and push the waiting requests to the next stage */ - list_for_each_entry_safe(rq, n, preflush_running, queuelist) { - unsigned int seq = blk_flush_cur_seq(rq); - - BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH); - blk_flush_complete_seq(rq, fq, seq, error); - } - - list_for_each_entry_safe(rq, n, postflush_running, queuelist) { - unsigned int seq = blk_flush_cur_seq(rq); - - BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH); - blk_flush_complete_seq(rq, fq, seq, error); - } + blk_flush_complete(q, fq, error); spin_unlock_irqrestore(&fq->mq_flush_lock, flags); return RQ_END_IO_NONE; @@ -401,7 +382,10 @@ static enum rq_end_io_ret mq_flush_data_end_io(struct request *rq, * re-initialize rq->queuelist before reusing it here. */ INIT_LIST_HEAD(&rq->queuelist); - blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error); + if (likely(!error)) + blk_enqueue_postflush(rq, fq); + else + blk_end_flush(rq, fq, error); spin_unlock_irqrestore(&fq->mq_flush_lock, flags); blk_mq_sched_restart(hctx); @@ -410,7 +394,6 @@ static enum rq_end_io_ret mq_flush_data_end_io(struct request *rq, static void blk_rq_init_flush(struct request *rq) { - rq->flush.seq = 0; rq->rq_flags |= RQF_FLUSH_SEQ; rq->flush.saved_end_io = rq->end_io; /* Usually NULL */ rq->end_io = mq_flush_data_end_io; @@ -469,7 +452,6 @@ bool blk_insert_flush(struct request *rq) * the post flush, and then just pass the command on. */ blk_rq_init_flush(rq); - rq->flush.seq |= REQ_FSEQ_PREFLUSH; spin_lock_irq(&fq->mq_flush_lock); fq->flush_data_in_flight++; spin_unlock_irq(&fq->mq_flush_lock); @@ -481,7 +463,7 @@ bool blk_insert_flush(struct request *rq) */ blk_rq_init_flush(rq); spin_lock_irq(&fq->mq_flush_lock); - blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0); + blk_enqueue_preflush(rq, fq); spin_unlock_irq(&fq->mq_flush_lock); return true; } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 01e8c31db665..d46fefdacea8 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -177,7 +177,6 @@ struct request { } elv; struct { - unsigned int seq; rq_end_io_fn *saved_end_io; } flush; From patchwork Tue Jul 25 13:01:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 125625 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp2480389vqg; Tue, 25 Jul 2023 06:45:27 -0700 (PDT) X-Google-Smtp-Source: APBJJlE5O3plyzMtAVKEn2pBxP5Lra0es3ChectI/Im3Ugyeid3VzNrtH2dkAF/CA8KYPqXyO7pe X-Received: by 2002:a17:907:78d7:b0:993:e752:1a71 with SMTP id kv23-20020a17090778d700b00993e7521a71mr12684706ejc.9.1690292726697; Tue, 25 Jul 2023 06:45:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690292726; cv=none; d=google.com; s=arc-20160816; b=fWTYPgCcRWVtTo+P6KM1jA9FfEpf7lnTR6hu9zABljxgA0psQdZkDyU7ZqKfRZcoIz rV5HcjiPMnyoI3+3NI3wllyCBCBJF7PFkPe5qdg7VKiTtBNTXyIF2N93BC1d0v6XGYwm CmUwtmYYZxr+Epr0qzNDD246gcyooy/K2NqwgF7HcqlPh+SQKq2mXYeicwOmah/CwNxS xzl3WXZqLDdFEReSrAeBS6sjQQHxf5lhvVVJG9ATVu+xHHQZOyAo4AxtNfZFqHomMahQ VEvJd44PYKBrL1on1fwKxFSR9CbX+hQ/KUvc3YpIu4xEPCE2SjTjB5b6FUj7v6jkXZGs 6kTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUddR5wiZ9cxmW/YJtgRqWXnFRr905MF2Tu61yP3IU0=; fh=4fzGWvE2LW7K10hgePOrl39slR3lbqz4nggQweyfPdc=; b=DdYAk5aB9Qae4y9td9ZHTHv/hsR/jAFLH2p4+Y2ZV/VYB0+af0sRSvFhyL4xlQl54e Xs9+9YTf0ih7lNu62Ay8pxt+vWcm70xcoLTS0MnJlmt3glKJ8rHi0bZVk6h4u7uKjLtg eKZ/+Ruugoted/2oPGTkSaW/MwUPvri1Yy9MerEHtcxPm4kQgsTa88kKHMRmRGzz9usA IIMBK2tDpCZfFbHvscbxDAa4EOmUu8q9TlUmDaoc2GnndqQLMP/6+z2WFcOr84usSxDJ mLPYLb0NANHbLruTPVOxrUPY8A8c8l7bSunuW4iIEp+AKqLNn42bgu7DXSvyBCyRq2n+ VbkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rSKMhlIF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w8-20020a1709062f8800b00993398bcae2si7891151eji.122.2023.07.25.06.45.02; Tue, 25 Jul 2023 06:45:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rSKMhlIF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230040AbjGYNTU (ORCPT + 99 others); Tue, 25 Jul 2023 09:19:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbjGYNTP (ORCPT ); Tue, 25 Jul 2023 09:19:15 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [91.218.175.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1295B1FF3 for ; Tue, 25 Jul 2023 06:19:10 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1690291149; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUddR5wiZ9cxmW/YJtgRqWXnFRr905MF2Tu61yP3IU0=; b=rSKMhlIFuqqMn3IHOfN8AP7mdsm25+x6DSm5KPAxRxGD47IVz6ycLH4I4AV4ZRiRNOhVvm 5XftPsf+x4V36bWCTlw5rKI3vavReqP7U0tbxcNaohBHS6d4IdyAyPa0WlQknaKP8VRD4n dw/7MPuxIL123WPEpdesOORx3TrJHig= From: chengming.zhou@linux.dev To: axboe@kernel.dk, hch@lst.de, ming.lei@redhat.com Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH v2 4/4] blk-flush: don't need to end rq twice for non postflush Date: Tue, 25 Jul 2023 21:01:02 +0800 Message-ID: <20230725130102.3030032-5-chengming.zhou@linux.dev> In-Reply-To: <20230725130102.3030032-1-chengming.zhou@linux.dev> References: <20230725130102.3030032-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772400386056572253 X-GMAIL-MSGID: 1772400386056572253 From: Chengming Zhou Now we unconditionally blk_rq_init_flush() to replace rq->end_io to make rq return twice back to the flush state machine for post-flush. Obviously, non post-flush requests don't need it, they don't need to end request twice, so they don't need to replace rq->end_io callback. And the same for requests with the FUA bit on hardware with FUA support. There are also some other good points: 1. all requests on hardware with FUA support won't have post-flush, so all of them don't need to end twice. 2. non post-flush requests won't have RQF_FLUSH_SEQ rq_flags set, so they can merge like normal requests. 3. we don't account non post-flush requests in flush_data_in_flight, since there is no point to defer pending flush for these requests. Signed-off-by: Chengming Zhou Reviewed-by: Christoph Hellwig --- block/blk-flush.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index ed195c760617..a299dae65350 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -178,7 +178,8 @@ static void blk_end_flush(struct request *rq, struct blk_flush_queue *fq, * normal completion and end it. */ list_del_init(&rq->queuelist); - blk_flush_restore_request(rq); + if (rq->rq_flags & RQF_FLUSH_SEQ) + blk_flush_restore_request(rq); blk_mq_end_request(rq, error); blk_kick_flush(q, fq); @@ -461,7 +462,8 @@ bool blk_insert_flush(struct request *rq) * Mark the request as part of a flush sequence and submit it * for further processing to the flush state machine. */ - blk_rq_init_flush(rq); + if (policy & REQ_FSEQ_POSTFLUSH) + blk_rq_init_flush(rq); spin_lock_irq(&fq->mq_flush_lock); blk_enqueue_preflush(rq, fq); spin_unlock_irq(&fq->mq_flush_lock);