From patchwork Thu Nov 3 16:26:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 14980 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp634548wru; Thu, 3 Nov 2022 09:29:00 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4vAD/wK7QhFjaVaj2a7DuZzQ+ObkhC50PNP1GBDIJLVfboPisSy0k7mZQzrta0T4Vc/PMs X-Received: by 2002:a17:906:cc10:b0:7ad:d776:8b7a with SMTP id ml16-20020a170906cc1000b007add7768b7amr20691712ejb.508.1667492940715; Thu, 03 Nov 2022 09:29:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667492940; cv=none; d=google.com; s=arc-20160816; b=uW+98lC1CT44bHxGuKMf5VMrmMN3MdT4FcLhz0yCly9bOM/zBhW8m9/TFNwoePJNgo aomRIh9+U+EjhdiLklj0kstxIuF+3kWnOwQhzZhXu62fAPCP4IfD6YYIeiUJ9JakhFh4 2nrHZJXde0yqHO1Wpo7WUdiM+IX26pKzw7wmp6jpX1SkNLIv2pCYLEzuAJaxO2cgXTEF j3jZswX6CgkJU6Waaqa+I6TxeCfqsc68rO9KEDVZuLKewnwUKTwGjKVeGJbMIbBsPfl/ PrXml6mgJqf/xfKmaUldjrT77k/pr+l0Th4DkkdYMCaNkiXFYfJMu6B2XChYeBlQX1MM 6GWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=D9JQ0TAWVkUj+MsWn6O89MpNEj0ssd0Wu+90axtvmFs=; b=SelZ9T4hOoyspK1u0wJCF2FThgfjXEAqD+2d1UKZypcDB3BGwaZ7qKGDsymuXtFgPG vbRDzIV6kZVCTeeHnkh7eOwAWuzQIQGJEDtdOM1ybb7ue/9dYdW+jKR13xlA0HeHywsO MtqQn1U9JgtQGaMkykQfkmCRwVQZH/p0Vbm+vwVGgKAQlTBEQlDN8x7iabh7KxliN2AQ /BhNWay6mhVConryZQv2iVlbwH2IXwGqzXirWwvAA36UTB2wW8034f43301GTzJd0JMF M+nrKcjAMBOGleLR/kf9erVhvW4N9e7WLwR1YTNW3Emo6Eycf5ZJih2x6pE8Tl6yIfkJ tRKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KBnSAZl3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sc6-20020a1709078a0600b0077ea290986fsi1761409ejc.584.2022.11.03.09.28.34; Thu, 03 Nov 2022 09:29:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KBnSAZl3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231596AbiKCQ05 (ORCPT + 99 others); Thu, 3 Nov 2022 12:26:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231421AbiKCQ0o (ORCPT ); Thu, 3 Nov 2022 12:26:44 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D7921B1F8 for ; Thu, 3 Nov 2022 09:26:43 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id bj12so6628685ejb.13 for ; Thu, 03 Nov 2022 09:26:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D9JQ0TAWVkUj+MsWn6O89MpNEj0ssd0Wu+90axtvmFs=; b=KBnSAZl3fHHbwZJ0peE8u6BPasYhPI4egpnUWzwq7u7QYRBxhaGUvZDDn1/oPVvVwH HtkRvGlSZE+4gHpcGOMd4KSAL0YEInPgngqYH4ndTufLp5+74hUHcWmRZKJfN8wUlPmZ L88P5WbqrqLlDGIFQN8Zk/rjEi6Q+/gfnKirG/TLFZmphtW0OXQbnxNeLV0ATPyr+T30 vsjrX6XVUAAaOo606gzC8OyNcFT1W39GTV33UptCKekRbYDCLARlcjo0QEgJXQDVj5bS uDaa03XLIo6OCoIMAXs37ojPcy0YhHMJlMuzopOaNm40olg5lW1G/q2plOZ3HqS1lPlX IHwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D9JQ0TAWVkUj+MsWn6O89MpNEj0ssd0Wu+90axtvmFs=; b=W1uW1gjLW5x07wUbaLSIJyZQNvo3kqZqyP9d+1Ppgl1gszS0gKj9bZiyWfPNQNLl6l 7d+o8wbGtPh3f1kiGV/FjQ7QbOWgRxdk8DJhs7f8lkf5PqiXynVTgH+rJYLVHP+LRfW9 LNjrRDxNK6OcjFJlrgTqIOA+KpqK5uVL0zsMWHYB75XROGj3Y6z2XjfSEhsFfTNdHwjJ LAqNpaJjo0lakeQAbSpXFbMjPaQbVOgZTLJBE/kbv/bSjELxddqPePx3aEBGnn4mq5C3 HuRlLkOUbPZry3XlkixaWWa9knXoCBx8FTLnxyEHqSZrcaZQflOH/swbty6Bo9ZH5+LE D/1Q== X-Gm-Message-State: ACrzQf1O6Dn/AzyV5brzpPXO/mMKrhcOtfMVUPJ+rC/mYmTetYQTSa9e FNiF2AdtlmWCVX//8nNJa4DfuA== X-Received: by 2002:a17:907:a647:b0:7ae:c63:b9e8 with SMTP id vu7-20020a170907a64700b007ae0c63b9e8mr7719895ejc.446.1667492801646; Thu, 03 Nov 2022 09:26:41 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id kx9-20020a170907774900b0078116c361d9sm702507ejc.10.2022.11.03.09.26.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Nov 2022 09:26:41 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, Paolo Valente , Gabriele Felici , Gianmarco Lusvardi , Giulio Barabino , Emiliano Maccaferri Subject: [PATCH V6 4/8] block, bfq: turn bfqq_data into an array in bfq_io_cq Date: Thu, 3 Nov 2022 17:26:19 +0100 Message-Id: <20221103162623.10286-5-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221103162623.10286-1-paolo.valente@linaro.org> References: <20221103162623.10286-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748493077642030778?= X-GMAIL-MSGID: =?utf-8?q?1748493077642030778?= When a bfq_queue Q is merged with another queue, several pieces of information are saved about Q. These pieces are stored in the bfqq_data field in the bfq_io_cq data structure of the process associated with Q. Yet, with a multi-actuator drive, a process may get associated with multiple bfq_queues: one queue for each of the N actuators. Each of these queues may undergo a merge. So, the bfq_io_cq data structure must be able to accommodate the above information for N queues. This commit solves this problem by turning the bfqq_data scalar field into an array of N elements (and by changing code so as to handle this array). This solution is written under the assumption that bfq_queues associated with different actuators cannot be cross-merged. This assumption holds naturally with basic queue merging: the latter is triggered by spatial locality, and sectors for different actuators are not close to each other. As for stable cross-merging, the assumption here is that it is disabled. Signed-off-by: Gabriele Felici Signed-off-by: Gianmarco Lusvardi Signed-off-by: Giulio Barabino Signed-off-by: Emiliano Maccaferri Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 72 ++++++++++++++++++++++++--------------------- block/bfq-iosched.h | 12 +++++--- 2 files changed, 47 insertions(+), 37 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 01528182c0c5..f44bac054aaf 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -404,7 +404,7 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[actuator_idx]; bic->bfqq[is_sync][actuator_idx] = bfqq; if (bfqq && bfqq_data->stable_merge_bfqq == bfqq) { @@ -1175,9 +1175,10 @@ static void bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, struct bfq_io_cq *bic, bool bfq_already_existing) { - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; unsigned int old_wr_coeff = 1; bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; if (bfqq_data->saved_has_short_ttime) bfq_mark_bfqq_has_short_ttime(bfqq); @@ -1827,6 +1828,16 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, return bfqq_weight > in_serv_weight; } +/* get the index of the actuator that will serve bio */ +static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) +{ + /* + * Multi-actuator support not complete yet, so always return 0 + * for the moment. + */ + return 0; +} + static bool bfq_better_to_idle(struct bfq_queue *bfqq); static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, @@ -1881,7 +1892,9 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || (bfq_bfqq_sync(bfqq) && - (bfqq->bic || RQ_BIC(rq)->bfqq_data.stably_merged) && + (bfqq->bic || + RQ_BIC(rq)->bfqq_data[bfq_actuator_index(bfqd, rq->bio)] + .stably_merged) && (*interactive || soft_rt))); /* @@ -2469,16 +2482,6 @@ static void bfq_remove_request(struct request_queue *q, } -/* get the index of the actuator that will serve bio */ -static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) -{ - /* - * Multi-actuator support not complete yet, so always return 0 - * for the moment. - */ - return 0; -} - static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { @@ -2905,7 +2908,8 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, void *io_struct, bool request, struct bfq_io_cq *bic) { struct bfq_queue *in_service_bfqq, *new_bfqq; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* if a merge has already been setup, then proceed with that first */ if (bfqq->new_bfqq) @@ -2952,8 +2956,9 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, if (new_bfqq) { bfqq_data->stably_merged = true; if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = - true; + new_bfqq->bic->bfqq_data + [new_bfqq->actuator_idx] + .stably_merged = true; } return new_bfqq; } else @@ -3052,7 +3057,9 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, static void bfq_bfqq_save_state(struct bfq_queue *bfqq) { struct bfq_io_cq *bic = bfqq->bic; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + /* State must be saved for the right queue index. */ + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* * If !bfqq->bic, the queue is already shared or its requests @@ -3063,7 +3070,7 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) return; bfqq_data->saved_last_serv_time_ns = bfqq->last_serv_time_ns; - bfqq_data->saved_inject_limit = bfqq->inject_limit; + bfqq_data->saved_inject_limit = bfqq->inject_limit; bfqq_data->saved_decrease_time_jif = bfqq->decrease_time_jif; bfqq_data->saved_weight = bfqq->entity.orig_weight; @@ -5425,7 +5432,7 @@ static void bfq_exit_icq(struct io_cq *icq) unsigned long flags; unsigned int act_idx; unsigned int num_actuators; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = bic->bfqq_data; /* * bfqd is NULL if scheduler already exited, and in that case @@ -5445,10 +5452,10 @@ static void bfq_exit_icq(struct io_cq *icq) num_actuators = BFQ_MAX_ACTUATORS; } - if (bfqq_data->stable_merge_bfqq) - bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); - for (act_idx = 0; act_idx < num_actuators; act_idx++) { + if (bfqq_data[act_idx].stable_merge_bfqq) + bfq_put_stable_ref(bfqq_data[act_idx].stable_merge_bfqq); + bfq_exit_icq_bfqq(bic, true, act_idx); bfq_exit_icq_bfqq(bic, false, act_idx); } @@ -5635,16 +5642,16 @@ bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, struct bfq_io_cq *bic, struct bfq_queue *last_bfqq_created) { + unsigned int a_idx = last_bfqq_created->actuator_idx; struct bfq_queue *new_bfqq = bfq_setup_merge(bfqq, last_bfqq_created); - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; if (!new_bfqq) return bfqq; if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = true; - bfqq_data->stably_merged = true; + new_bfqq->bic->bfqq_data[a_idx].stably_merged = true; + bic->bfqq_data[a_idx].stably_merged = true; /* * Reusing merge functions. This implies that @@ -5713,7 +5720,6 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, &bfqd->last_bfqq_created; struct bfq_queue *last_bfqq_created = *source_bfqq; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* * If last_bfqq_created has not been set yet, then init it. If @@ -5775,7 +5781,8 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, /* * Record the bfqq to merge to. */ - bfqq_data->stable_merge_bfqq = last_bfqq_created; + bic->bfqq_data[last_bfqq_created->actuator_idx].stable_merge_bfqq = + last_bfqq_created; } } @@ -6696,7 +6703,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, { unsigned int act_idx = bfq_actuator_index(bfqd, bio); struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[act_idx]; if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6804,7 +6811,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) struct bfq_queue *bfqq; bool new_queue = false; bool bfqq_already_existing = false, split = false; - struct bfq_iocq_bfqq_data *bfqq_data; + unsigned int a_idx = bfq_actuator_index(bfqd, bio); if (unlikely(!rq->elv.icq)) return NULL; @@ -6828,17 +6835,16 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, &new_queue); - bfqq_data = &bic->bfqq_data; - if (likely(!new_queue)) { /* If the queue was seeky for too long, break it apart. */ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq) && - !bfqq_data->stably_merged) { + !bic->bfqq_data[a_idx].stably_merged) { struct bfq_queue *old_bfqq = bfqq; /* Update bic before losing reference to bfqq */ if (bfq_bfqq_in_large_burst(bfqq)) - bfqq_data->saved_in_large_burst = true; + bic->bfqq_data[a_idx].saved_in_large_burst = + true; bfqq = bfq_split_bfqq(bic, bfqq); split = true; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index f2e8ab91951c..e27897d66a0f 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -416,7 +416,7 @@ struct bfq_queue { struct bfq_iocq_bfqq_data { /* * Snapshot of the has_short_time flag before merging; taken - * to remember its value while the queue is merged, so as to + * to remember its values while the queue is merged, so as to * be able to restore it in case of split. */ bool saved_has_short_ttime; @@ -430,7 +430,7 @@ struct bfq_iocq_bfqq_data { u64 saved_tot_idle_time; /* - * Same purpose as the previous fields for the value of the + * Same purpose as the previous fields for the values of the * field keeping the queue's belonging to a large burst */ bool saved_in_large_burst; @@ -493,8 +493,12 @@ struct bfq_io_cq { uint64_t blkcg_serial_nr; /* the current blkcg serial */ #endif - /* persistent data for associated synchronous process queue */ - struct bfq_iocq_bfqq_data bfqq_data; + /* + * Persistent data for associated synchronous process queues + * (one queue per actuator, see field bfqq above). In + * particular, each of these queues may undergo a merge. + */ + struct bfq_iocq_bfqq_data bfqq_data[BFQ_MAX_ACTUATORS]; unsigned int requests; /* Number of requests this process has in flight */ };