From patchwork Sun Oct 30 10:02:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12994 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726415wru; Sun, 30 Oct 2022 03:05:45 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7ZIz+Ji0hBe8vzl+l+Lz3r0MaTh4ZDKCDNl9bHUB/v8WFfkhyT1BP/Y4vSlFlVagP5vKbN X-Received: by 2002:a17:90b:153:b0:213:b853:5db1 with SMTP id em19-20020a17090b015300b00213b8535db1mr5807802pjb.168.1667124345162; Sun, 30 Oct 2022 03:05:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124345; cv=none; d=google.com; s=arc-20160816; b=uurMLN9WqKWk5P6xpRLKVV+htUXVNnSiXVZ4ssSEw/KDjn6sI7sxweQ/kvFOwE4tgH Y9jgTMXQjOWx4b7F0+zx8GekdiLkL8u1YrSYMfmg+z5NtPxZcgzvcGPba93Y6/uCyM22 ng8DapaFFDWtZDpLt/Gkk++jz0K0MSLn+pIxVk8NhdFpl4oJM8Pr5ux56KJ86EIh1ki6 I4Pb9gfGJyrsZM5RA3QMgZZ26bq2zQ2Z5D9m/KwjLaAZJp+CmFSZnyNy+pc4uWF/0ZQ9 P0CeNt6jgwSbt3CmY5WAs57tgXwJxgbVemi7G1IUb2CZxuQecvxbAwKe5gN3GSTb0jOu DGMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YuQet8vAfXYjtJovtWvfi564eTZoov1XJZh14ZhEs4I=; b=eU4/5vPxb6rNsfHhQwLGuGInC0tBqylMvV54f8QZuAwRYANzvj2QGowf6CdRT+F2tV UrmSgT9iGsGzStoQESecA2O2Xb5nlcDVUji4y3+xwaBA4GfuRDtO/JHVRjvtNIJAGkD9 4+3pHohiHalpMqC7hjMoRRL41iGFWabiro0OvHdWmFiUHa1WmwEdefxbppBNpt9aK1P5 1ckOfO8UW2g0hPgOpIRaY8X5/4ZyMbxMlwOngL/WjYnoY0dF584x3oCeM0oTmfPABkSG Ga5k+RixywKrvUqF5g26wHqhJgzLVPJu2MifGgD+2iS/zXAnue3EhoGqGOau/CrI7QF2 +gJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MoCV288N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w8-20020a170902904800b0017532e01e57si4413699plz.200.2022.10.30.03.05.33; Sun, 30 Oct 2022 03:05:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MoCV288N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230046AbiJ3KDn (ORCPT + 99 others); Sun, 30 Oct 2022 06:03:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230017AbiJ3KDk (ORCPT ); Sun, 30 Oct 2022 06:03:40 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A40B5194 for ; Sun, 30 Oct 2022 03:03:36 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id kt23so22773955ejc.7 for ; Sun, 30 Oct 2022 03:03:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YuQet8vAfXYjtJovtWvfi564eTZoov1XJZh14ZhEs4I=; b=MoCV288NV6piyppcD8QDHQZaPRlAEEeT7Yj6ZPnFAwtE+/2oyx/V8GC3xznHoMvVMc DPWQScX9P/d5C29d2uKSpKixlVykUHTF5KYz4NPVrh7RE97A7wMXZWKbKfuFcFZ/ei+5 hgGz1SlDtlaUBhdDKwrHRyFOGWRcw0JPdPUQlC/gHTdIg1JQUFV+om8aDgR135H7vhCN ND+LfQZOMmyForrXMkYc7o0nzxFzdat+6qogQlDI0kJRwUfC6ofGs6+x9qLmHJLu5Hmi GOJGFOffIc8GiWEwdnHjTIekdNwedmt9R+YrRAp90+ur8pInvLPZwmReXvJeU5yx56jd s8rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YuQet8vAfXYjtJovtWvfi564eTZoov1XJZh14ZhEs4I=; b=kztr0CBm5zkl6YyCOPC77YC3tGS6zWRSMk4xhkzZo9ouR0rLR1Tx/fecVMjcVtDigR DhkHZHb0nceOdzDJg+g73VN2x/H41YwTpAF00m4mSM/EMQTokv5Hed0om1FHxHRGtWg9 Bj+fal4tg7qlHlM7xf7hhHcwqQgX20iTbkqZ5Dbgt/R4x3elRPEluEfYwSvSYA5VmIhI R3xuLMy/HISEnkaGVXCWOnHdnnO7P5YtS0HvZVstDL7CSY80C/VFYLyOWcLJWUy/mX2R /FgS9Q3BPYvmz2Y5tAkNZaropllRfruRPddWp5N7llwew/4jDHJlkwFWO0Qa4SBcc0n5 hHvA== X-Gm-Message-State: ACrzQf0EQg3potHIWB0keskIzg21Wx0VNUaIOHznYP2b4gRxat/eUrNG TsJ3a9oJNCpJQvSk8/+B1YIK7w== X-Received: by 2002:a17:907:3f85:b0:733:3f0e:2f28 with SMTP id hr5-20020a1709073f8500b007333f0e2f28mr7174254ejc.376.1667124215019; Sun, 30 Oct 2022 03:03:35 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:34 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Gabriele Felici , Carmine Zaccagnino Subject: [PATCH V5 1/8] block, bfq: split sync bfq_queues on a per-actuator basis Date: Sun, 30 Oct 2022 11:02:53 +0100 Message-Id: <20221030100300.3085-2-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106577224540713?= X-GMAIL-MSGID: =?utf-8?q?1748106577224540713?= Single-LUN multi-actuator SCSI drives, as well as all multi-actuator SATA drives appear as a single device to the I/O subsystem [1]. Yet they address commands to different actuators internally, as a function of Logical Block Addressing (LBAs). A given sector is reachable by only one of the actuators. For example, Seagate’s Serial Advanced Technology Attachment (SATA) version contains two actuators and maps the lower half of the SATA LBA space to the lower actuator and the upper half to the upper actuator. Evidently, to fully utilize actuators, no actuator must be left idle or underutilized while there is pending I/O for it. The block layer must somehow control the load of each actuator individually. This commit lays the ground for allowing BFQ to provide such a per-actuator control. BFQ associates an I/O-request sync bfq_queue with each process doing synchronous I/O, or with a group of processes, in case of queue merging. Then BFQ serves one bfq_queue at a time. While in service, a bfq_queue is emptied in request-position order. Yet the same process, or group of processes, may generate I/O for different actuators. In this case, different streams of I/O (each for a different actuator) get all inserted into the same sync bfq_queue. So there is basically no individual control on when each stream is served, i.e., on when the I/O requests of the stream are picked from the bfq_queue and dispatched to the drive. This commit enables BFQ to control the service of each actuator individually for synchronous I/O, by simply splitting each sync bfq_queue into N queues, one for each actuator. In other words, a sync bfq_queue is now associated to a pair (process, actuator). As a consequence of this split, the per-queue proportional-share policy implemented by BFQ will guarantee that the sync I/O generated for each actuator, by each process, receives its fair share of service. This is just a preparatory patch. If the I/O of the same process happens to be sent to different queues, then each of these queues may undergo queue merging. To handle this event, the bfq_io_cq data structure must be properly extended. In addition, stable merging must be disabled to avoid loss of control on individual actuators. Finally, also async queues must be split. These issues are described in detail and addressed in next commits. As for this commit, although multiple per-process bfq_queues are provided, the I/O of each process or group of processes is still sent to only one queue, regardless of the actuator the I/O is for. The forwarding to distinct bfq_queues will be enabled after addressing the above issues. [1] https://www.linaro.org/blog/budget-fair-queueing-bfq-linux-io-scheduler-optimizations-for-multi-actuator-sata-hard-drives/ Signed-off-by: Gabriele Felici Signed-off-by: Carmine Zaccagnino Signed-off-by: Paolo Valente --- block/bfq-cgroup.c | 95 ++++++++++++++++------------ block/bfq-iosched.c | 151 +++++++++++++++++++++++++++++--------------- block/bfq-iosched.h | 51 ++++++++++++--- 3 files changed, 194 insertions(+), 103 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 144bca006463..d243c429d9c0 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -700,6 +700,48 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_put_queue(bfqq); } +static void bfq_sync_bfqq_move(struct bfq_data *bfqd, + struct bfq_queue *sync_bfqq, + struct bfq_io_cq *bic, + struct bfq_group *bfqg, + unsigned int act_idx) +{ + if (!sync_bfqq->new_bfqq && !bfq_bfqq_coop(sync_bfqq)) { + /* We are the only user of this bfqq, just move it */ + if (sync_bfqq->entity.sched_data != &bfqg->sched_data) + bfq_bfqq_move(bfqd, sync_bfqq, bfqg); + } else { + struct bfq_queue *bfqq; + + /* + * The queue was merged to a different queue. Check + * that the merge chain still belongs to the same + * cgroup. + */ + for (bfqq = sync_bfqq; bfqq; bfqq = bfqq->new_bfqq) + if (bfqq->entity.sched_data != + &bfqg->sched_data) + break; + if (bfqq) { + /* + * Some queue changed cgroup so the merge is + * not valid anymore. We cannot easily just + * cancel the merge (by clearing new_bfqq) as + * there may be other processes using this + * queue and holding refs to all queues below + * sync_bfqq->new_bfqq. Similarly if the merge + * already happened, we need to detach from + * bfqq now so that we cannot merge bio to a + * request from the old cgroup. + */ + bfq_put_cooperator(sync_bfqq); + bfq_release_process_ref(bfqd, sync_bfqq); + bic_set_bfqq(bic, NULL, 1, act_idx); + } + } +} + + /** * __bfq_bic_change_cgroup - move @bic to @bfqg. * @bfqd: the queue descriptor. @@ -714,53 +756,24 @@ static void *__bfq_bic_change_cgroup(struct bfq_data *bfqd, struct bfq_io_cq *bic, struct bfq_group *bfqg) { - struct bfq_queue *async_bfqq = bic_to_bfqq(bic, 0); - struct bfq_queue *sync_bfqq = bic_to_bfqq(bic, 1); struct bfq_entity *entity; + unsigned int act_idx; - if (async_bfqq) { - entity = &async_bfqq->entity; - - if (entity->sched_data != &bfqg->sched_data) { - bic_set_bfqq(bic, NULL, 0); - bfq_release_process_ref(bfqd, async_bfqq); - } - } + for (act_idx = 0; act_idx < bfqd->num_actuators; act_idx++) { + struct bfq_queue *async_bfqq = bic_to_bfqq(bic, 0, act_idx); + struct bfq_queue *sync_bfqq = bic_to_bfqq(bic, 1, act_idx); - if (sync_bfqq) { - if (!sync_bfqq->new_bfqq && !bfq_bfqq_coop(sync_bfqq)) { - /* We are the only user of this bfqq, just move it */ - if (sync_bfqq->entity.sched_data != &bfqg->sched_data) - bfq_bfqq_move(bfqd, sync_bfqq, bfqg); - } else { - struct bfq_queue *bfqq; + if (async_bfqq) { + entity = &async_bfqq->entity; - /* - * The queue was merged to a different queue. Check - * that the merge chain still belongs to the same - * cgroup. - */ - for (bfqq = sync_bfqq; bfqq; bfqq = bfqq->new_bfqq) - if (bfqq->entity.sched_data != - &bfqg->sched_data) - break; - if (bfqq) { - /* - * Some queue changed cgroup so the merge is - * not valid anymore. We cannot easily just - * cancel the merge (by clearing new_bfqq) as - * there may be other processes using this - * queue and holding refs to all queues below - * sync_bfqq->new_bfqq. Similarly if the merge - * already happened, we need to detach from - * bfqq now so that we cannot merge bio to a - * request from the old cgroup. - */ - bfq_put_cooperator(sync_bfqq); - bfq_release_process_ref(bfqd, sync_bfqq); - bic_set_bfqq(bic, NULL, 1); + if (entity->sched_data != &bfqg->sched_data) { + bic_set_bfqq(bic, NULL, 0, act_idx); + bfq_release_process_ref(bfqd, async_bfqq); } } + + if (sync_bfqq) + bfq_sync_bfqq_move(bfqd, sync_bfqq, bic, bfqg, act_idx); } return bfqg; diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 7ea427817f7f..5c69394bbb65 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -377,14 +377,19 @@ static const unsigned long bfq_late_stable_merging = 600; #define RQ_BIC(rq) ((struct bfq_io_cq *)((rq)->elv.priv[0])) #define RQ_BFQQ(rq) ((rq)->elv.priv[1]) -struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync) +struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, + bool is_sync, + unsigned int actuator_idx) { - return bic->bfqq[is_sync]; + return bic->bfqq[is_sync][actuator_idx]; } static void bfq_put_stable_ref(struct bfq_queue *bfqq); -void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync) +void bic_set_bfqq(struct bfq_io_cq *bic, + struct bfq_queue *bfqq, + bool is_sync, + unsigned int actuator_idx) { /* * If bfqq != NULL, then a non-stable queue merge between @@ -399,7 +404,7 @@ void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync) * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ - bic->bfqq[is_sync] = bfqq; + bic->bfqq[is_sync][actuator_idx] = bfqq; if (bfqq && bic->stable_merge_bfqq == bfqq) { /* @@ -672,9 +677,9 @@ static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data) { struct bfq_data *bfqd = data->q->elevator->elevator_data; struct bfq_io_cq *bic = bfq_bic_lookup(data->q); - struct bfq_queue *bfqq = bic ? bic_to_bfqq(bic, op_is_sync(opf)) : NULL; int depth; unsigned limit = data->q->nr_requests; + unsigned int act_idx; /* Sync reads have full depth available */ if (op_is_sync(opf) && !op_is_write(opf)) { @@ -684,14 +689,21 @@ static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data) limit = (limit * depth) >> bfqd->full_depth_shift; } - /* - * Does queue (or any parent entity) exceed number of requests that - * should be available to it? Heavily limit depth so that it cannot - * consume more available requests and thus starve other entities. - */ - if (bfqq && bfqq_request_over_limit(bfqq, limit)) - depth = 1; + for (act_idx = 0; act_idx < bfqd->num_actuators; act_idx++) { + struct bfq_queue *bfqq = + bic ? bic_to_bfqq(bic, op_is_sync(opf), act_idx) : NULL; + /* + * Does queue (or any parent entity) exceed number of + * requests that should be available to it? Heavily + * limit depth so that it cannot consume more + * available requests and thus starve other entities. + */ + if (bfqq && bfqq_request_over_limit(bfqq, limit)) { + depth = 1; + break; + } + } bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u", __func__, bfqd->wr_busy_queues, op_is_sync(opf), depth); if (depth) @@ -2142,7 +2154,7 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, * We reset waker detection logic also if too much time has passed * since the first detection. If wakeups are rare, pointless idling * doesn't hurt throughput that much. The condition below makes sure - * we do not uselessly idle blocking waker in more than 1/64 cases. + * we do not uselessly idle blocking waker in more than 1/64 cases. */ if (bfqd->last_completed_rq_bfqq != bfqq->tentative_waker_bfqq || @@ -2454,6 +2466,16 @@ static void bfq_remove_request(struct request_queue *q, } +/* get the index of the actuator that will serve bio */ +static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) +{ + /* + * Multi-actuator support not complete yet, so always return 0 + * for the moment. + */ + return 0; +} + static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { @@ -2478,7 +2500,8 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, */ bfq_bic_update_cgroup(bic, bio); - bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf)); + bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf), + bfq_actuator_index(bfqd, bio)); } else { bfqd->bio_bfqq = NULL; } @@ -3174,7 +3197,7 @@ bfq_merge_bfqqs(struct bfq_data *bfqd, struct bfq_io_cq *bic, /* * Merge queues (that is, let bic redirect its requests to new_bfqq) */ - bic_set_bfqq(bic, new_bfqq, 1); + bic_set_bfqq(bic, new_bfqq, 1, bfqq->actuator_idx); bfq_mark_bfqq_coop(new_bfqq); /* * new_bfqq now belongs to at least two bics (it is a shared queue): @@ -4808,11 +4831,12 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) */ if (bfq_bfqq_wait_request(bfqq) || (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) { + unsigned int act_idx = bfqq->actuator_idx; struct bfq_queue *async_bfqq = - bfqq->bic && bfqq->bic->bfqq[0] && - bfq_bfqq_busy(bfqq->bic->bfqq[0]) && - bfqq->bic->bfqq[0]->next_rq ? - bfqq->bic->bfqq[0] : NULL; + bfqq->bic && bfqq->bic->bfqq[0][act_idx] && + bfq_bfqq_busy(bfqq->bic->bfqq[0][act_idx]) && + bfqq->bic->bfqq[0][act_idx]->next_rq ? + bfqq->bic->bfqq[0][act_idx] : NULL; struct bfq_queue *blocked_bfqq = !hlist_empty(&bfqq->woken_list) ? container_of(bfqq->woken_list.first, @@ -4904,7 +4928,7 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic && bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= bfq_bfqq_budget_left(async_bfqq)) - bfqq = bfqq->bic->bfqq[0]; + bfqq = bfqq->bic->bfqq[0][act_idx]; else if (bfqq->waker_bfqq && bfq_bfqq_busy(bfqq->waker_bfqq) && bfqq->waker_bfqq->next_rq && @@ -5365,49 +5389,59 @@ static void bfq_exit_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq) bfq_release_process_ref(bfqd, bfqq); } -static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync) +static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, + bool is_sync, + unsigned int actuator_idx) { - struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync); + struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, actuator_idx); struct bfq_data *bfqd; if (bfqq) bfqd = bfqq->bfqd; /* NULL if scheduler already exited */ if (bfqq && bfqd) { - unsigned long flags; - - spin_lock_irqsave(&bfqd->lock, flags); bfqq->bic = NULL; bfq_exit_bfqq(bfqd, bfqq); - bic_set_bfqq(bic, NULL, is_sync); - spin_unlock_irqrestore(&bfqd->lock, flags); + bic_set_bfqq(bic, NULL, is_sync, actuator_idx); } } static void bfq_exit_icq(struct io_cq *icq) { struct bfq_io_cq *bic = icq_to_bic(icq); + struct bfq_data *bfqd = bic_to_bfqd(bic); + unsigned long flags; + unsigned int act_idx; + unsigned int num_actuators; - if (bic->stable_merge_bfqq) { - struct bfq_data *bfqd = bic->stable_merge_bfqq->bfqd; - + /* + * bfqd is NULL if scheduler already exited, and in that case + * this is the last time these queues are accessed. + */ + if (bfqd) { + spin_lock_irqsave(&bfqd->lock, flags); + num_actuators = bfqd->num_actuators; + } else { /* - * bfqd is NULL if scheduler already exited, and in - * that case this is the last time bfqq is accessed. + * bfqd->num_actuators not available any longer, cycle + * over all possible per-actuator bfqqs in next + * loop. We rely on bic being zeroed on creation, and + * therefore on its unused per-actuator fields being + * NULL. */ - if (bfqd) { - unsigned long flags; + num_actuators = BFQ_MAX_ACTUATORS; + } - spin_lock_irqsave(&bfqd->lock, flags); - bfq_put_stable_ref(bic->stable_merge_bfqq); - spin_unlock_irqrestore(&bfqd->lock, flags); - } else { - bfq_put_stable_ref(bic->stable_merge_bfqq); - } + if (bic->stable_merge_bfqq) + bfq_put_stable_ref(bic->stable_merge_bfqq); + + for (act_idx = 0; act_idx < num_actuators; act_idx++) { + bfq_exit_icq_bfqq(bic, true, act_idx); + bfq_exit_icq_bfqq(bic, false, act_idx); } - bfq_exit_icq_bfqq(bic, true); - bfq_exit_icq_bfqq(bic, false); + if (bfqd) + spin_unlock_irqrestore(&bfqd->lock, flags); } /* @@ -5484,23 +5518,25 @@ static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio) bic->ioprio = ioprio; - bfqq = bic_to_bfqq(bic, false); + bfqq = bic_to_bfqq(bic, false, bfq_actuator_index(bfqd, bio)); if (bfqq) { bfq_release_process_ref(bfqd, bfqq); bfqq = bfq_get_queue(bfqd, bio, false, bic, true); - bic_set_bfqq(bic, bfqq, false); + bic_set_bfqq(bic, bfqq, false, bfq_actuator_index(bfqd, bio)); } - bfqq = bic_to_bfqq(bic, true); + bfqq = bic_to_bfqq(bic, true, bfq_actuator_index(bfqd, bio)); if (bfqq) bfq_set_next_ioprio_data(bfqq, bic); } static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct bfq_io_cq *bic, pid_t pid, int is_sync) + struct bfq_io_cq *bic, pid_t pid, int is_sync, + unsigned int act_idx) { u64 now_ns = ktime_get_ns(); + bfqq->actuator_idx = act_idx; RB_CLEAR_NODE(&bfqq->entity.rb_node); INIT_LIST_HEAD(&bfqq->fifo); INIT_HLIST_NODE(&bfqq->burst_list_node); @@ -5739,6 +5775,7 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, struct bfq_group *bfqg; bfqg = bfq_bio_bfqg(bfqd, bio); + if (!is_sync) { async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class, ioprio); @@ -5753,7 +5790,7 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, if (bfqq) { bfq_init_bfqq(bfqd, bfqq, bic, current->pid, - is_sync); + is_sync, bfq_actuator_index(bfqd, bio)); bfq_init_entity(&bfqq->entity, bfqg); bfq_log_bfqq(bfqd, bfqq, "allocated"); } else { @@ -6068,7 +6105,8 @@ static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) * then complete the merge and redirect it to * new_bfqq. */ - if (bic_to_bfqq(RQ_BIC(rq), 1) == bfqq) + if (bic_to_bfqq(RQ_BIC(rq), 1, + bfq_actuator_index(bfqd, rq->bio)) == bfqq) bfq_merge_bfqqs(bfqd, RQ_BIC(rq), bfqq, new_bfqq); @@ -6622,7 +6660,7 @@ bfq_split_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq) return bfqq; } - bic_set_bfqq(bic, NULL, 1); + bic_set_bfqq(bic, NULL, 1, bfqq->actuator_idx); bfq_put_cooperator(bfqq); @@ -6636,7 +6674,8 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bool split, bool is_sync, bool *new_queue) { - struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync); + unsigned int act_idx = bfq_actuator_index(bfqd, bio); + struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6648,7 +6687,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bfq_put_queue(bfqq); bfqq = bfq_get_queue(bfqd, bio, is_sync, bic, split); - bic_set_bfqq(bic, bfqq, is_sync); + bic_set_bfqq(bic, bfqq, is_sync, act_idx); if (split && is_sync) { if ((bic->was_in_burst_list && bfqd->large_burst) || bic->saved_in_large_burst) @@ -7090,8 +7129,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. * Grab a permanent reference to it, so that the normal code flow * will not attempt to free it. + * Set zero as actuator index: we will pretend that + * all I/O requests are for the same actuator. */ - bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0); + bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0, 0); bfqd->oom_bfqq.ref++; bfqd->oom_bfqq.new_ioprio = BFQ_DEFAULT_QUEUE_IOPRIO; bfqd->oom_bfqq.new_ioprio_class = IOPRIO_CLASS_BE; @@ -7110,6 +7151,12 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue = q; + /* + * Multi-actuator support not complete yet, default to single + * actuator for the moment. + */ + bfqd->num_actuators = 1; + INIT_LIST_HEAD(&bfqd->dispatch); hrtimer_init(&bfqd->idle_slice_timer, CLOCK_MONOTONIC, diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 71f721670ab6..bfcbd8ea9000 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -33,6 +33,14 @@ */ #define BFQ_SOFTRT_WEIGHT_FACTOR 100 +/* + * Maximum number of actuators supported. This constant is used simply + * to define the size of the static array that will contain + * per-actuator data. The current value is hopefully a good upper + * bound to the possible number of actuators of any actual drive. + */ +#define BFQ_MAX_ACTUATORS 32 + struct bfq_entity; /** @@ -225,12 +233,14 @@ struct bfq_ttime { * struct bfq_queue - leaf schedulable entity. * * A bfq_queue is a leaf request queue; it can be associated with an - * io_context or more, if it is async or shared between cooperating - * processes. @cgroup holds a reference to the cgroup, to be sure that it - * does not disappear while a bfqq still references it (mostly to avoid - * races between request issuing and task migration followed by cgroup - * destruction). - * All the fields are protected by the queue lock of the containing bfqd. + * io_context or more, if it is async or shared between cooperating + * processes. Besides, it contains I/O requests for only one actuator + * (an io_context is associated with a different bfq_queue for each + * actuator it generates I/O for). @cgroup holds a reference to the + * cgroup, to be sure that it does not disappear while a bfqq still + * references it (mostly to avoid races between request issuing and + * task migration followed by cgroup destruction). All the fields are + * protected by the queue lock of the containing bfqd. */ struct bfq_queue { /* reference counter */ @@ -395,6 +405,9 @@ struct bfq_queue { * the woken queues when this queue exits. */ struct hlist_head woken_list; + + /* index of the actuator this queue is associated with */ + unsigned int actuator_idx; }; /** @@ -403,8 +416,17 @@ struct bfq_queue { struct bfq_io_cq { /* associated io_cq structure */ struct io_cq icq; /* must be the first member */ - /* array of two process queues, the sync and the async */ - struct bfq_queue *bfqq[2]; + /* + * Matrix of associated process queues: first row for async + * queues, second row sync queues. Each row contains one + * column for each actuator. An I/O request generated by the + * process is inserted into the queue pointed by bfqq[i][j] if + * the request is to be served by the j-th actuator of the + * drive, where i==0 or i==1, depending on whether the request + * is async or sync. So there is a distinct queue for each + * actuator. + */ + struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; /* per (request_queue, blkcg) ioprio */ int ioprio; #ifdef CONFIG_BFQ_GROUP_IOSCHED @@ -768,6 +790,13 @@ struct bfq_data { */ unsigned int word_depths[2][2]; unsigned int full_depth_shift; + + /* + * Number of independent actuators. This is equal to 1 in + * case of single-actuator drives. + */ + unsigned int num_actuators; + }; enum bfqq_state_flags { @@ -964,8 +993,10 @@ struct bfq_group { extern const int bfq_timeout; -struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync); -void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync); +struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync, + unsigned int actuator_idx); +void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync, + unsigned int actuator_idx); struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic); void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq); void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, From patchwork Sun Oct 30 10:02:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12990 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726256wru; Sun, 30 Oct 2022 03:05:15 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7VQ4X+G4saW9cFskKsGMdt6zEDJZpDhq0ZCUxUmnM052HXjawNUQOfK5tAvxrZAlyRtOcd X-Received: by 2002:a17:903:41c5:b0:186:ceff:f805 with SMTP id u5-20020a17090341c500b00186cefff805mr8572349ple.31.1667124315535; Sun, 30 Oct 2022 03:05:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124315; cv=none; d=google.com; s=arc-20160816; b=xsGH/qM2uQecQP8zVcsUQH4/6cR4EJ7wESgPrzFRbdQNYfaSK5QdmZh4WwOlJo2k0v ZdV4LPVkdL01oB1oiRz+uP0+1Py4IMmWK9erxfsy8FB/9bnioJbhEfs2l8JLWUOhLIZq UuN//T6s3AR6guoMJ3NxLx9mQ3fUHDI0uZZUKvoGaynKgu1kDwJKZ1m3yX9ZRUYIv5Nh 3Y3hWmfhAy/BjytRXVJDODisCrNbOjHnGIE5c9qfsnfFXz1crd8g+8OUWGKMZwYxIwBE jD2NUMAauuIG32U26o3DJlY9Na/h7mlxwHwhmX8/7EiP79aZjJ+j0Dlwaetl0Ay6fmlq S8hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Dagqyailu11jgL84XY9XZ7ZO8PNuYsgn+9sY89yrA6U=; b=qSnob59SECE/8UNLmgULxe/OH5rIX9lNUqtGZfP4k1kxOvH9PGbdoPwOO32Y1hvMJ2 SKjxJoLYmtsp7c2WeRybqcWPxtyVuBvHMxGmT5XxSSoxO/cz4c3zMS4B4YhlFf4LuN+l fAtwHnddNq7zPYvGE7cCGCz7vBX4OID0sKo11tAFSsoA8K8eFJeFDsIYx27NCdNEL6VZ r1a7zOetRYpfrKQunHvKZ6SG+QBt2sF9YuZ/Ku6D0bAlKerpJyZ8X7c30siRktWkvJ4r sGaST45s+0GQrS8Zth16Y9MMygTVBOovYu4ZhppWG5m66oCV99OE5Tevu3Q8hnIvm9Wy ellg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=T9Fs+8e8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m5-20020a656a05000000b0046086f8f5d0si6227489pgu.537.2022.10.30.03.05.03; Sun, 30 Oct 2022 03:05:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=T9Fs+8e8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230089AbiJ3KDx (ORCPT + 99 others); Sun, 30 Oct 2022 06:03:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbiJ3KDk (ORCPT ); Sun, 30 Oct 2022 06:03:40 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B763EBC7 for ; Sun, 30 Oct 2022 03:03:37 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id sc25so22763130ejc.12 for ; Sun, 30 Oct 2022 03:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dagqyailu11jgL84XY9XZ7ZO8PNuYsgn+9sY89yrA6U=; b=T9Fs+8e83L0bqfPmtM639a+YtCRBr5w4o65ZRTQeHdi6bOvQ2M+n9GYOuQdbiXrPx2 AM10J7h1PtP7OamKIIoXg2/E8f+KqVpVdjkQ4ksPCyM0UH6cLHxTsLuK0nwWhzJUBzQ4 m8r8BAzySbvyPVPufA8mmGAeTT688CWVKfHTy0lv43BgcLCQx7CkvFqULSHisNVXhcDZ gwDsKyBIXwXlEpf58IYHkos+NLInCv+yXoRsLyi3EwosNDDxFxPbnV+eyxja4CMvhyof XHAgfzscqrBXqmg28LYSe3ic1EgEYgadVLmeeKvS5/deKAD/pXnaLy5w0p0B2X64Mndh 7eYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dagqyailu11jgL84XY9XZ7ZO8PNuYsgn+9sY89yrA6U=; b=noPx1GEY9o6z86I7OL834OpYCXL0mRVYod2xt2pJ0k2zApPrsuvjvy9RfTB1MIzRLM 4RtrAtRmAQ8H8hnbFhRAz+nyHi7yMLfzMvRdJVRlwTmkhhjqBb1GZi+smQ2Sbqt1S2LT QNVveyeJLcMFW1Gx+ZuG8MTNFYqkaFj5F9agLnh035F8x+sObqcPdgqECHdu5v+XGmQo vt6mdCkjcge/lC5gcClNgkzodf7zWriiCWU3T3Jee5jrcxLTziFNtb84Ryq4h9HvGOQ7 YQbh9KHHwxMpwdy8GS6RetYgmQTvhzR/GMOIAnA2fSi3Fenny2dzRQb1LACP6WXAtFtj 8FFw== X-Gm-Message-State: ACrzQf1TTzIl/FKb7qTQ/InUcLoYvRoYiT1zWnnRUCEGZyGfXP0u4Lr9 xO0nE5HY0/9P2GDmebLNwsiqww== X-Received: by 2002:a17:906:7949:b0:7ac:9917:90b9 with SMTP id l9-20020a170906794900b007ac991790b9mr7337741ejo.536.1667124216223; Sun, 30 Oct 2022 03:03:36 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:35 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente Subject: [PATCH V5 2/8] block, bfq: forbid stable merging of queues associated with different actuators Date: Sun, 30 Oct 2022 11:02:54 +0100 Message-Id: <20221030100300.3085-3-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106546228528291?= X-GMAIL-MSGID: =?utf-8?q?1748106546228528291?= If queues associated with different actuators are merged, then control is lost on each actuator. Therefore some actuator may be underutilized, and throughput may decrease. This problem cannot occur with basic queue merging, because the latter is triggered by spatial locality, and sectors for different actuators are not close to each other. Yet it may happen with stable merging. To address this issue, this commit prevents stable merging from occurring among queues associated with different actuators. Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 5c69394bbb65..ec4b0e70265f 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -5705,9 +5705,13 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, * it has been set already, but too long ago, then move it * forward to bfqq. Finally, move also if bfqq belongs to a * different group than last_bfqq_created, or if bfqq has a - * different ioprio or ioprio_class. If none of these - * conditions holds true, then try an early stable merge or - * schedule a delayed stable merge. + * different ioprio, ioprio_class or actuator_idx. If none of + * these conditions holds true, then try an early stable merge + * or schedule a delayed stable merge. As for the condition on + * actuator_idx, the reason is that, if queues associated with + * different actuators are merged, then control is lost on + * each actuator. Therefore some actuator may be + * underutilized, and throughput may decrease. * * A delayed merge is scheduled (instead of performing an * early merge), in case bfqq might soon prove to be more @@ -5725,7 +5729,8 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, bfqq->creation_time) || bfqq->entity.parent != last_bfqq_created->entity.parent || bfqq->ioprio != last_bfqq_created->ioprio || - bfqq->ioprio_class != last_bfqq_created->ioprio_class) + bfqq->ioprio_class != last_bfqq_created->ioprio_class || + bfqq->actuator_idx != last_bfqq_created->actuator_idx) *source_bfqq = bfqq; else if (time_after_eq(last_bfqq_created->creation_time + bfqd->bfq_burst_interval, From patchwork Sun Oct 30 10:02:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12991 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726248wru; Sun, 30 Oct 2022 03:05:15 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6nP03u1LxwSYVH+W3jwVS0yMSgKw1Oq8R32EyVhUU5hDv9LmWKJZaJSy109QD3K4nQSbZe X-Received: by 2002:a17:903:2cb:b0:171:4f0d:beb6 with SMTP id s11-20020a17090302cb00b001714f0dbeb6mr8392658plk.53.1667124315013; Sun, 30 Oct 2022 03:05:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124315; cv=none; d=google.com; s=arc-20160816; b=cEcRqzuUpljTDl4hvp7VijoRfL94096lLH3789YBVMc7M6oZ/k7CLpJuNmsTW2fpi8 bSmkDlxh2zDXoRvtWK+OL6NVrM3JpetaJjdzw87o6cKvdCwVtu7OsO+R9a34KjFRR0gf 3lS/0/qYnmxrtbbDmMrGAxlYkSquncHe54nAFxq7JVjg/AS5kof6xVxhZViwEfGXaS52 7RX7e33T+LRPy/jPskRpmvAgocG3vSF638YlL/aGlQKPwOp8EgFpP0qwq9dpemwH/Cab uNHaNX15St8s0RcKONt3O8xyXbqCPbvrhuTtyS6Ch6eC6w0hTUJrK0OvRAyojGhlo79E HowA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9tB9eObh3Vne5izy9wSWj1GaOcjX6R5NF7vt6yMovsw=; b=a9GlupQAAWbJdUdWOgeQojehLdY0V90lZDSwobbuP4ndb7UX/n4x9f6ZK1jnnYEczs KWTOFez4pGXeq6uu9gkeKWhOZewRDj1nZujOB8PNT04Rr8Ni0dtN2s+EBiVfZMcOPI3Z M3QSkt+vW6dvCzgqngHOm0a/Uxu/xigU/seDw6eiKaK4igRdXib06pDGNSrLYMIXN7fL XYJGR0tnRG9DTeuDliY9K77cqUnO2CGCltqdyY1nGLgCarLFwYm8gw4xsJyGJhP7Y6xW qmk2jNB7x+r4pBtUVp/1xcVI2cfJ6OTywzbBQ27S7+Onp1crgaIYTnu15LnY6v81sGwn m47w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="X+Wfh5H/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h23-20020a17090aa89700b00212cae10683si4282916pjq.69.2022.10.30.03.05.02; Sun, 30 Oct 2022 03:05:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="X+Wfh5H/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229744AbiJ3KDr (ORCPT + 99 others); Sun, 30 Oct 2022 06:03:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229933AbiJ3KDk (ORCPT ); Sun, 30 Oct 2022 06:03:40 -0400 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5C6A1A0 for ; Sun, 30 Oct 2022 03:03:38 -0700 (PDT) Received: by mail-ed1-x536.google.com with SMTP id x2so13733176edd.2 for ; Sun, 30 Oct 2022 03:03:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9tB9eObh3Vne5izy9wSWj1GaOcjX6R5NF7vt6yMovsw=; b=X+Wfh5H/RZeSi8Kt7fTrGwLqbc/NnXXmiagL5glKrMG0MZ0sGc4b8HpO/zrq//6pbb pNrzaLbzr8iudfQLhFBJd0uWvk/C1mHLx6ODggUzUnocmHQzWIEZHfoXyXI/imePL+vD cnYLO6C//y6Gy3f+AgXlrX6fj8X5JRPSdNqAKpdFs9UWCIX36fLbR5OydeAsw/Y0/uVj 3kBVgLjOUUL/eAzI9dre0itnsARyJa9a6tiGQmDPYfPegq5qNUSMF/2O6qo2rIF2OSsM FmgHEbXRqsQh5Vz9z2VdHtYHNCiD+JZj7TpU1vt2yFiMnamHaaqH37fh/5eAzD8vC6J9 rmOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9tB9eObh3Vne5izy9wSWj1GaOcjX6R5NF7vt6yMovsw=; b=OjQYOlYhLa6NqjmLUfHUr1TmS1PHppk0L986aovWJDLpb7Sr+i8qmCWz4mnYCJP07K 0znvwB5X3kWdghx2puEWNFFlyB+VsmVzrdYGQNWSrbkG+vwfKtu82csMC/6GgeA8hrTy 0XAjp4ZVb1FTkr2NQvWYPh8FzzOYn3WTtafAQxW/1YJgzrflOX8ga486hRIrm7JKkehN Z6fFEjSbk1w+/EyGuYciHO8sO38gIoLC81te7w5mPljRd6of762SZZEydOHtz3gPqdqy 9liTGDuJvMVtGPmg6FgZvtB8f2TxaYti95gTEGN6VqMv47usHCiejx4zhzKOmJh22Xmx 24zg== X-Gm-Message-State: ACrzQf1fJC8rLSnLpMUIBKGYVP41xpAqLL86j3/ZgFKgJqVrN3o/MnWG IOmL1DiOb1DyrtKp0LejjgyaCQ== X-Received: by 2002:a05:6402:440d:b0:450:de54:3fcf with SMTP id y13-20020a056402440d00b00450de543fcfmr7914149eda.312.1667124217377; Sun, 30 Oct 2022 03:03:37 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:36 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Damien Le Moal , Gianmarco Lusvardi , Giulio Barabino , Emiliano Maccaferri Subject: [PATCH V5 3/8] block, bfq: move io_cq-persistent bfqq data into a dedicated struct Date: Sun, 30 Oct 2022 11:02:55 +0100 Message-Id: <20221030100300.3085-4-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106545588992073?= X-GMAIL-MSGID: =?utf-8?q?1748106545588992073?= With a multi-actuator drive, a process may get associated with multiple bfq_queues: one queue for each of the N actuators. So, the bfq_io_cq data structure must be able to accommodate its per-queue persistent information for N queues. Currently it stores this information for just one queue, in several scalar fields. This is a preparatory commit for moving to accommodating persistent information for N queues. In particular, this commit packs all the above scalar fields into a single data structure. Then there is now only one fieldi, in bfq_io_cq, that stores all the above information. This scalar field will then be turned into an array by a following commit. Suggested-by: Damien Le Moal Signed-off-by: Gianmarco Lusvardi Signed-off-by: Giulio Barabino Signed-off-by: Emiliano Maccaferri Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 129 +++++++++++++++++++++++++------------------- block/bfq-iosched.h | 52 ++++++++++-------- 2 files changed, 105 insertions(+), 76 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index ec4b0e70265f..139b8f1ba439 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -404,9 +404,10 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; bic->bfqq[is_sync][actuator_idx] = bfqq; - if (bfqq && bic->stable_merge_bfqq == bfqq) { + if (bfqq && bfqq_data.stable_merge_bfqq == bfqq) { /* * Actually, these same instructions are executed also * in bfq_setup_cooperator, in case of abort or actual @@ -415,9 +416,9 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * did so, we would nest even more complexity in this * function. */ - bfq_put_stable_ref(bic->stable_merge_bfqq); + bfq_put_stable_ref(bfqq_data.stable_merge_bfqq); - bic->stable_merge_bfqq = NULL; + bfqq_data.stable_merge_bfqq = NULL; } } @@ -1174,38 +1175,40 @@ static void bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, struct bfq_io_cq *bic, bool bfq_already_existing) { + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; unsigned int old_wr_coeff = 1; bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); - if (bic->saved_has_short_ttime) + if (bfqq_data.saved_has_short_ttime) bfq_mark_bfqq_has_short_ttime(bfqq); else bfq_clear_bfqq_has_short_ttime(bfqq); - if (bic->saved_IO_bound) + if (bfqq_data.saved_IO_bound) bfq_mark_bfqq_IO_bound(bfqq); else bfq_clear_bfqq_IO_bound(bfqq); - bfqq->last_serv_time_ns = bic->saved_last_serv_time_ns; - bfqq->inject_limit = bic->saved_inject_limit; - bfqq->decrease_time_jif = bic->saved_decrease_time_jif; + bfqq->last_serv_time_ns = bfqq_data.saved_last_serv_time_ns; + bfqq->inject_limit = bfqq_data.saved_inject_limit; + bfqq->decrease_time_jif = bfqq_data.saved_decrease_time_jif; - bfqq->entity.new_weight = bic->saved_weight; - bfqq->ttime = bic->saved_ttime; - bfqq->io_start_time = bic->saved_io_start_time; - bfqq->tot_idle_time = bic->saved_tot_idle_time; + bfqq->entity.new_weight = bfqq_data.saved_weight; + bfqq->ttime = bfqq_data.saved_ttime; + bfqq->io_start_time = bfqq_data.saved_io_start_time; + bfqq->tot_idle_time = bfqq_data.saved_tot_idle_time; /* * Restore weight coefficient only if low_latency is on */ if (bfqd->low_latency) { old_wr_coeff = bfqq->wr_coeff; - bfqq->wr_coeff = bic->saved_wr_coeff; + bfqq->wr_coeff = bfqq_data.saved_wr_coeff; } - bfqq->service_from_wr = bic->saved_service_from_wr; - bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; - bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish; - bfqq->wr_cur_max_time = bic->saved_wr_cur_max_time; + bfqq->service_from_wr = bfqq_data.saved_service_from_wr; + bfqq->wr_start_at_switch_to_srt = + bfqq_data.saved_wr_start_at_switch_to_srt; + bfqq->last_wr_start_finish = bfqq_data.saved_last_wr_start_finish; + bfqq->wr_cur_max_time = bfqq_data.saved_wr_cur_max_time; if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) || time_is_before_jiffies(bfqq->last_wr_start_finish + @@ -1878,7 +1881,7 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || (bfq_bfqq_sync(bfqq) && - (bfqq->bic || RQ_BIC(rq)->stably_merged) && + (bfqq->bic || RQ_BIC(rq)->bfqq_data.stably_merged) && (*interactive || soft_rt))); /* @@ -2902,6 +2905,7 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, void *io_struct, bool request, struct bfq_io_cq *bic) { struct bfq_queue *in_service_bfqq, *new_bfqq; + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; /* if a merge has already been setup, then proceed with that first */ if (bfqq->new_bfqq) @@ -2923,21 +2927,21 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, * stable merging) also if bic is associated with a * sync queue, but this bfqq is async */ - if (bfq_bfqq_sync(bfqq) && bic->stable_merge_bfqq && + if (bfq_bfqq_sync(bfqq) && bfqq_data.stable_merge_bfqq && !bfq_bfqq_just_created(bfqq) && time_is_before_jiffies(bfqq->split_time + msecs_to_jiffies(bfq_late_stable_merging)) && time_is_before_jiffies(bfqq->creation_time + msecs_to_jiffies(bfq_late_stable_merging))) { struct bfq_queue *stable_merge_bfqq = - bic->stable_merge_bfqq; + bfqq_data.stable_merge_bfqq; int proc_ref = min(bfqq_process_refs(bfqq), bfqq_process_refs(stable_merge_bfqq)); /* deschedule stable merge, because done or aborted here */ bfq_put_stable_ref(stable_merge_bfqq); - bic->stable_merge_bfqq = NULL; + bfqq_data.stable_merge_bfqq = NULL; if (!idling_boosts_thr_without_issues(bfqd, bfqq) && proc_ref > 0) { @@ -2946,10 +2950,10 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_setup_merge(bfqq, stable_merge_bfqq); if (new_bfqq) { - bic->stably_merged = true; + bfqq_data.stably_merged = true; if (new_bfqq->bic) - new_bfqq->bic->stably_merged = - true; + new_bfqq->bic->bfqq_data.stably_merged = + true; } return new_bfqq; } else @@ -3048,6 +3052,7 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, static void bfq_bfqq_save_state(struct bfq_queue *bfqq) { struct bfq_io_cq *bic = bfqq->bic; + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; /* * If !bfqq->bic, the queue is already shared or its requests @@ -3057,18 +3062,21 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) if (!bic) return; - bic->saved_last_serv_time_ns = bfqq->last_serv_time_ns; - bic->saved_inject_limit = bfqq->inject_limit; - bic->saved_decrease_time_jif = bfqq->decrease_time_jif; - - bic->saved_weight = bfqq->entity.orig_weight; - bic->saved_ttime = bfqq->ttime; - bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); - bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); - bic->saved_io_start_time = bfqq->io_start_time; - bic->saved_tot_idle_time = bfqq->tot_idle_time; - bic->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); - bic->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node); + bfqq_data.saved_last_serv_time_ns = bfqq->last_serv_time_ns; + bfqq_data.saved_inject_limit = bfqq->inject_limit; + bfqq_data.saved_decrease_time_jif = bfqq->decrease_time_jif; + + bfqq_data.saved_weight = bfqq->entity.orig_weight; + bfqq_data.saved_ttime = bfqq->ttime; + bfqq_data.saved_has_short_ttime = + bfq_bfqq_has_short_ttime(bfqq); + bfqq_data.saved_IO_bound = bfq_bfqq_IO_bound(bfqq); + bfqq_data.saved_io_start_time = bfqq->io_start_time; + bfqq_data.saved_tot_idle_time = bfqq->tot_idle_time; + bfqq_data.saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); + bfqq_data.was_in_burst_list = + !hlist_unhashed(&bfqq->burst_list_node); + if (unlikely(bfq_bfqq_just_created(bfqq) && !bfq_bfqq_in_large_burst(bfqq) && bfqq->bfqd->low_latency)) { @@ -3081,17 +3089,21 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) * to bfqq, so that to avoid that bfqq unjustly fails * to enjoy weight raising if split soon. */ - bic->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; - bic->saved_wr_start_at_switch_to_srt = bfq_smallest_from_now(); - bic->saved_wr_cur_max_time = bfq_wr_duration(bfqq->bfqd); - bic->saved_last_wr_start_finish = jiffies; + bfqq_data.saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; + bfqq_data.saved_wr_start_at_switch_to_srt = + bfq_smallest_from_now(); + bfqq_data.saved_wr_cur_max_time = + bfq_wr_duration(bfqq->bfqd); + bfqq_data.saved_last_wr_start_finish = jiffies; } else { - bic->saved_wr_coeff = bfqq->wr_coeff; - bic->saved_wr_start_at_switch_to_srt = + bfqq_data.saved_wr_coeff = bfqq->wr_coeff; + bfqq_data.saved_wr_start_at_switch_to_srt = bfqq->wr_start_at_switch_to_srt; - bic->saved_service_from_wr = bfqq->service_from_wr; - bic->saved_last_wr_start_finish = bfqq->last_wr_start_finish; - bic->saved_wr_cur_max_time = bfqq->wr_cur_max_time; + bfqq_data.saved_service_from_wr = + bfqq->service_from_wr; + bfqq_data.saved_last_wr_start_finish = + bfqq->last_wr_start_finish; + bfqq_data.saved_wr_cur_max_time = bfqq->wr_cur_max_time; } } @@ -5413,6 +5425,7 @@ static void bfq_exit_icq(struct io_cq *icq) unsigned long flags; unsigned int act_idx; unsigned int num_actuators; + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; /* * bfqd is NULL if scheduler already exited, and in that case @@ -5432,8 +5445,8 @@ static void bfq_exit_icq(struct io_cq *icq) num_actuators = BFQ_MAX_ACTUATORS; } - if (bic->stable_merge_bfqq) - bfq_put_stable_ref(bic->stable_merge_bfqq); + if (bfqq_data.stable_merge_bfqq) + bfq_put_stable_ref(bfqq_data.stable_merge_bfqq); for (act_idx = 0; act_idx < num_actuators; act_idx++) { bfq_exit_icq_bfqq(bic, true, act_idx); @@ -5624,13 +5637,14 @@ bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, { struct bfq_queue *new_bfqq = bfq_setup_merge(bfqq, last_bfqq_created); + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; if (!new_bfqq) return bfqq; if (new_bfqq->bic) - new_bfqq->bic->stably_merged = true; - bic->stably_merged = true; + new_bfqq->bic->bfqq_data.stably_merged = true; + bfqq_data.stably_merged = true; /* * Reusing merge functions. This implies that @@ -5699,6 +5713,7 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, &bfqd->last_bfqq_created; struct bfq_queue *last_bfqq_created = *source_bfqq; + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; /* * If last_bfqq_created has not been set yet, then init it. If @@ -5760,7 +5775,7 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, /* * Record the bfqq to merge to. */ - bic->stable_merge_bfqq = last_bfqq_created; + bfqq_data.stable_merge_bfqq = last_bfqq_created; } } @@ -6681,6 +6696,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, { unsigned int act_idx = bfq_actuator_index(bfqd, bio); struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); + struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6694,12 +6710,12 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bic_set_bfqq(bic, bfqq, is_sync, act_idx); if (split && is_sync) { - if ((bic->was_in_burst_list && bfqd->large_burst) || - bic->saved_in_large_burst) + if ((bfqq_data.was_in_burst_list && bfqd->large_burst) || + bfqq_data.saved_in_large_burst) bfq_mark_bfqq_in_large_burst(bfqq); else { bfq_clear_bfqq_in_large_burst(bfqq); - if (bic->was_in_burst_list) + if (bfqq_data.was_in_burst_list) /* * If bfqq was in the current * burst list before being @@ -6788,6 +6804,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) struct bfq_queue *bfqq; bool new_queue = false; bool bfqq_already_existing = false, split = false; + struct bfq_iocq_bfqq_data bfqq_data; if (unlikely(!rq->elv.icq)) return NULL; @@ -6811,15 +6828,17 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, &new_queue); + bfqq_data = bic->bfqq_data; + if (likely(!new_queue)) { /* If the queue was seeky for too long, break it apart. */ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq) && - !bic->stably_merged) { + !bfqq_data.stably_merged) { struct bfq_queue *old_bfqq = bfqq; /* Update bic before losing reference to bfqq */ if (bfq_bfqq_in_large_burst(bfqq)) - bic->saved_in_large_burst = true; + bfqq_data.saved_in_large_burst = true; bfqq = bfq_split_bfqq(bic, bfqq); split = true; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index bfcbd8ea9000..f2e8ab91951c 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -411,27 +411,9 @@ struct bfq_queue { }; /** - * struct bfq_io_cq - per (request_queue, io_context) structure. - */ -struct bfq_io_cq { - /* associated io_cq structure */ - struct io_cq icq; /* must be the first member */ - /* - * Matrix of associated process queues: first row for async - * queues, second row sync queues. Each row contains one - * column for each actuator. An I/O request generated by the - * process is inserted into the queue pointed by bfqq[i][j] if - * the request is to be served by the j-th actuator of the - * drive, where i==0 or i==1, depending on whether the request - * is async or sync. So there is a distinct queue for each - * actuator. - */ - struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; - /* per (request_queue, blkcg) ioprio */ - int ioprio; -#ifdef CONFIG_BFQ_GROUP_IOSCHED - uint64_t blkcg_serial_nr; /* the current blkcg serial */ -#endif +* struct bfq_data - bfqq data unique and persistent for associated bfq_io_cq +*/ +struct bfq_iocq_bfqq_data { /* * Snapshot of the has_short_time flag before merging; taken * to remember its value while the queue is merged, so as to @@ -486,6 +468,34 @@ struct bfq_io_cq { struct bfq_queue *stable_merge_bfqq; bool stably_merged; /* non splittable if true */ +}; + +/** + * struct bfq_io_cq - per (request_queue, io_context) structure. + */ +struct bfq_io_cq { + /* associated io_cq structure */ + struct io_cq icq; /* must be the first member */ + /* + * Matrix of associated process queues: first row for async + * queues, second row sync queues. Each row contains one + * column for each actuator. An I/O request generated by the + * process is inserted into the queue pointed by bfqq[i][j] if + * the request is to be served by the j-th actuator of the + * drive, where i==0 or i==1, depending on whether the request + * is async or sync. So there is a distinct queue for each + * actuator. + */ + struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; + /* per (request_queue, blkcg) ioprio */ + int ioprio; +#ifdef CONFIG_BFQ_GROUP_IOSCHED + uint64_t blkcg_serial_nr; /* the current blkcg serial */ +#endif + + /* persistent data for associated synchronous process queue */ + struct bfq_iocq_bfqq_data bfqq_data; + unsigned int requests; /* Number of requests this process has in flight */ }; From patchwork Sun Oct 30 10:02:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12992 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726281wru; Sun, 30 Oct 2022 03:05:21 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4X8A2P9EeRQFLj08L3QfrDwFyjUSQlrPaaIQl0rCg7FhfnvFZQ/0bYOKdLKNCUCE5cGeKC X-Received: by 2002:a17:902:e752:b0:186:9efb:71f3 with SMTP id p18-20020a170902e75200b001869efb71f3mr8429636plf.153.1667124321140; Sun, 30 Oct 2022 03:05:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124321; cv=none; d=google.com; s=arc-20160816; b=NpFS5Hu8C/ZcLepoG6hXGX6O6TgO5e3TB4oiN8duYqfFI9GJwcmn/mZpZ5bHzQu4k4 lmdl2LJsZojd7jNu9bfUE8iYWq44qGXJPIDDzFGGH9NWEYCNv1F42qc96uPLpK7J+YJr eda4S1QvJXnce8iCceHUUe2J4gVz/DIopPqKP6jDCva5bWhpRX4KF5q0Xvh93fmMvNEh 0eeP3rVXgRYKEy92h5rSAXArqVDJmxPIrSkAboQaH/isVQYmYaf9yERKCIxL7J0xTc8h apbFZV1uczxrBfnC77qESgH7kERxJw0qA59hsEfKOYK22hS5mhY8gojB+zSwWTrh6mla 4f5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5a6As9UVgwj6yXvVVDplGTtGvsy9ujSwGGjDgxJRrts=; b=Ai9nSi4FCV0EmUCqo3SdJbfI2GbHB3/sAxcNVpFMnrPQqB4NL/nOWogxtq1ufhc4Ta XfNr27ijGdq93rBb/a9vJzwed83N6puaLpW8TKoAyx4jDIeT0iRFcDOQLHYicFEnMlVi veGtnKfzKXQoDwCp5cCgjraH+6UtkX/vAwgqRsAOpe0JO6WCs3GrPwCnI873W3DOPq1y G2ZWi8DggV9Q+OZYmQhTqiEK1GNhApxF3NhVAaQ/QEJYuli7OCVeGEYMFPAlxgVIYZRa GA8Kc2V7ugfIFVy5G2kLl8ul86piWG9EK8x/1g2srSGihSmwJ4zxcMLy+hbQ6fDNICRN i6Qw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eawdJckT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i8-20020a17090a718800b00213a61cca5csi4186131pjk.145.2022.10.30.03.05.08; Sun, 30 Oct 2022 03:05:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eawdJckT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230071AbiJ3KEA (ORCPT + 99 others); Sun, 30 Oct 2022 06:04:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229995AbiJ3KDm (ORCPT ); Sun, 30 Oct 2022 06:03:42 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D7321A1 for ; Sun, 30 Oct 2022 03:03:40 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id kt23so22774177ejc.7 for ; Sun, 30 Oct 2022 03:03:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5a6As9UVgwj6yXvVVDplGTtGvsy9ujSwGGjDgxJRrts=; b=eawdJckTVbzkCIQZ9+YQ2zSimtcKcyQ9pF8VtJQM5MQeCtJ8AU6l5ZV7fKJBs+Lnua Y/kn6HGxkU2gbPRz9PnUS9y7CUJJja+Y8MtP6C1gaYzXmSOKd4VIxSPRgw7CX/qADpZE oASeB/K71mfwwFI4zkqFL9xej8aAgJYn8ILBJbIcEKkU2c1lwPbVB+/u9giAPlm0Qx0o woUSqechGR57X/GeWGQQZpXuobjoXaV+paaRZBKj5I9w9UMq1b2dTu+tvFLcHBUiMtWo jVdyY2GEYUu+aP7WNBzjKvYSOEn3iEZDMZsdDggu5Ka6Ki0BaIgQ0sqJPT55zK3iEZLf 56/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5a6As9UVgwj6yXvVVDplGTtGvsy9ujSwGGjDgxJRrts=; b=HVxMJ+eRAqQhk0lMWWjK9ATpJh6wlu0QIBiSq/QPR4rCnfD6cb1UUtzQj5cM3x28qG 7/QursY+60ZsoxxvmD/UTqA7R/z04ZQxlxyeKsWXQ8Yt4Pb+mH43j2+hyT3HyMolukbo QlZfXesAcWk8luuBSBoi0cM+uSNuPfZcNVXSBEbxBpdWD81jdY9kPTgvhB22vp5P+trK EFIgYrjBi8m5lkFYrFPXWx/ORxL4scI6IRmMGxa+XI4mpwhN25UyRSheXkZ4/pDb5i1d fA1rXJW2KUv7dVZ5iAdtuVAah3BY0rOdM+0eX6yEIhZhiMucviLFLK6Z5edixIwj1oD+ g82w== X-Gm-Message-State: ACrzQf0hdo2LbWxz9kPJqf/f0XOg6QjXapiiXZykX5C1fbHTAqbbzO0v eL8pZoQS4O/CL2tGtOJVmSLF7g== X-Received: by 2002:a17:907:7621:b0:741:6656:bd14 with SMTP id jy1-20020a170907762100b007416656bd14mr7416763ejc.298.1667124218755; Sun, 30 Oct 2022 03:03:38 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:38 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Paolo Valente , Gabriele Felici , Gianmarco Lusvardi , Giulio Barabino , Emiliano Maccaferri Subject: [PATCH V5 4/8] block, bfq: turn bfqq_data into an array in bfq_io_cq Date: Sun, 30 Oct 2022 11:02:56 +0100 Message-Id: <20221030100300.3085-5-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106552149017145?= X-GMAIL-MSGID: =?utf-8?q?1748106552149017145?= When a bfq_queue Q is merged with another queue, several pieces of information are saved about Q. These pieces are stored in the bfqq_data field in the bfq_io_cq data structure of the process associated with Q. Yet, with a multi-actuator drive, a process may get associated with multiple bfq_queues: one queue for each of the N actuators. Each of these queues may undergo a merge. So, the bfq_io_cq data structure must be able to accommodate the above information for N queues. This commit solves this problem by turning the bfqq_data scalar field into an array of N elements (and by changing code so as to handle this array). This solution is written under the assumption that bfq_queues associated with different actuators cannot be cross-merged. This assumption holds naturally with basic queue merging: the latter is triggered by spatial locality, and sectors for different actuators are not close to each other. As for stable cross-merging, the assumption here is that it is disabled. Signed-off-by: Gabriele Felici Signed-off-by: Gianmarco Lusvardi Signed-off-by: Giulio Barabino Signed-off-by: Emiliano Maccaferri Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 170 ++++++++++++++++++++++++-------------------- block/bfq-iosched.h | 12 ++-- 2 files changed, 101 insertions(+), 81 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 139b8f1ba439..bfdf954da5b7 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -404,10 +404,10 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[actuator_idx]; bic->bfqq[is_sync][actuator_idx] = bfqq; - if (bfqq && bfqq_data.stable_merge_bfqq == bfqq) { + if (bfqq && bfqq_data->stable_merge_bfqq == bfqq) { /* * Actually, these same instructions are executed also * in bfq_setup_cooperator, in case of abort or actual @@ -416,9 +416,9 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * did so, we would nest even more complexity in this * function. */ - bfq_put_stable_ref(bfqq_data.stable_merge_bfqq); + bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); - bfqq_data.stable_merge_bfqq = NULL; + bfqq_data->stable_merge_bfqq = NULL; } } @@ -1175,40 +1175,43 @@ static void bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, struct bfq_io_cq *bic, bool bfq_already_existing) { - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; unsigned int old_wr_coeff = 1; bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; - if (bfqq_data.saved_has_short_ttime) + if (bfqq_data->saved_has_short_ttime) bfq_mark_bfqq_has_short_ttime(bfqq); else bfq_clear_bfqq_has_short_ttime(bfqq); - if (bfqq_data.saved_IO_bound) + if (bfqq_data->saved_IO_bound) bfq_mark_bfqq_IO_bound(bfqq); else bfq_clear_bfqq_IO_bound(bfqq); - bfqq->last_serv_time_ns = bfqq_data.saved_last_serv_time_ns; - bfqq->inject_limit = bfqq_data.saved_inject_limit; - bfqq->decrease_time_jif = bfqq_data.saved_decrease_time_jif; - - bfqq->entity.new_weight = bfqq_data.saved_weight; - bfqq->ttime = bfqq_data.saved_ttime; - bfqq->io_start_time = bfqq_data.saved_io_start_time; - bfqq->tot_idle_time = bfqq_data.saved_tot_idle_time; + bfqq->last_serv_time_ns = + bfqq_data->saved_last_serv_time_ns; + bfqq->inject_limit = bfqq_data->saved_inject_limit; + bfqq->decrease_time_jif = + bfqq_data->saved_decrease_time_jif; + bfqq->entity.new_weight = bfqq_data->saved_weight; + bfqq->ttime = bfqq_data->saved_ttime; + bfqq->io_start_time = bfqq_data->saved_io_start_time; + bfqq->tot_idle_time = bfqq_data->saved_tot_idle_time; /* * Restore weight coefficient only if low_latency is on */ if (bfqd->low_latency) { old_wr_coeff = bfqq->wr_coeff; - bfqq->wr_coeff = bfqq_data.saved_wr_coeff; + bfqq->wr_coeff = bfqq_data->saved_wr_coeff; } - bfqq->service_from_wr = bfqq_data.saved_service_from_wr; + bfqq->service_from_wr = bfqq_data->saved_service_from_wr; bfqq->wr_start_at_switch_to_srt = - bfqq_data.saved_wr_start_at_switch_to_srt; - bfqq->last_wr_start_finish = bfqq_data.saved_last_wr_start_finish; - bfqq->wr_cur_max_time = bfqq_data.saved_wr_cur_max_time; + bfqq_data->saved_wr_start_at_switch_to_srt; + bfqq->last_wr_start_finish = + bfqq_data->saved_last_wr_start_finish; + bfqq->wr_cur_max_time = bfqq_data->saved_wr_cur_max_time; if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) || time_is_before_jiffies(bfqq->last_wr_start_finish + @@ -1827,6 +1830,16 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, return bfqq_weight > in_serv_weight; } +/* get the index of the actuator that will serve bio */ +static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) +{ + /* + * Multi-actuator support not complete yet, so always return 0 + * for the moment. + */ + return 0; +} + static bool bfq_better_to_idle(struct bfq_queue *bfqq); static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, @@ -1881,7 +1894,9 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || (bfq_bfqq_sync(bfqq) && - (bfqq->bic || RQ_BIC(rq)->bfqq_data.stably_merged) && + (bfqq->bic || + RQ_BIC(rq)->bfqq_data[bfq_actuator_index(bfqd, rq->bio)] + .stably_merged) && (*interactive || soft_rt))); /* @@ -2469,16 +2484,6 @@ static void bfq_remove_request(struct request_queue *q, } -/* get the index of the actuator that will serve bio */ -static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) -{ - /* - * Multi-actuator support not complete yet, so always return 0 - * for the moment. - */ - return 0; -} - static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { @@ -2905,7 +2910,8 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, void *io_struct, bool request, struct bfq_io_cq *bic) { struct bfq_queue *in_service_bfqq, *new_bfqq; - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* if a merge has already been setup, then proceed with that first */ if (bfqq->new_bfqq) @@ -2927,21 +2933,22 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, * stable merging) also if bic is associated with a * sync queue, but this bfqq is async */ - if (bfq_bfqq_sync(bfqq) && bfqq_data.stable_merge_bfqq && + if (bfq_bfqq_sync(bfqq) && + bfqq_data->stable_merge_bfqq && !bfq_bfqq_just_created(bfqq) && time_is_before_jiffies(bfqq->split_time + msecs_to_jiffies(bfq_late_stable_merging)) && time_is_before_jiffies(bfqq->creation_time + msecs_to_jiffies(bfq_late_stable_merging))) { struct bfq_queue *stable_merge_bfqq = - bfqq_data.stable_merge_bfqq; + bfqq_data->stable_merge_bfqq; int proc_ref = min(bfqq_process_refs(bfqq), bfqq_process_refs(stable_merge_bfqq)); /* deschedule stable merge, because done or aborted here */ bfq_put_stable_ref(stable_merge_bfqq); - bfqq_data.stable_merge_bfqq = NULL; + bfqq_data->stable_merge_bfqq = NULL; if (!idling_boosts_thr_without_issues(bfqd, bfqq) && proc_ref > 0) { @@ -2950,10 +2957,11 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_setup_merge(bfqq, stable_merge_bfqq); if (new_bfqq) { - bfqq_data.stably_merged = true; + bfqq_data->stably_merged = true; if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = - true; + new_bfqq->bic->bfqq_data + [new_bfqq->actuator_idx] + .stably_merged = true; } return new_bfqq; } else @@ -3052,7 +3060,9 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, static void bfq_bfqq_save_state(struct bfq_queue *bfqq) { struct bfq_io_cq *bic = bfqq->bic; - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; + /* State must be saved for the right queue index. */ + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* * If !bfqq->bic, the queue is already shared or its requests @@ -3062,19 +3072,23 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) if (!bic) return; - bfqq_data.saved_last_serv_time_ns = bfqq->last_serv_time_ns; - bfqq_data.saved_inject_limit = bfqq->inject_limit; - bfqq_data.saved_decrease_time_jif = bfqq->decrease_time_jif; + bfqq_data->saved_last_serv_time_ns = + bfqq->last_serv_time_ns; + bfqq_data->saved_inject_limit = + bfqq->inject_limit; + bfqq_data->saved_decrease_time_jif = + bfqq->decrease_time_jif; - bfqq_data.saved_weight = bfqq->entity.orig_weight; - bfqq_data.saved_ttime = bfqq->ttime; - bfqq_data.saved_has_short_ttime = + bfqq_data->saved_weight = bfqq->entity.orig_weight; + bfqq_data->saved_ttime = bfqq->ttime; + bfqq_data->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); - bfqq_data.saved_IO_bound = bfq_bfqq_IO_bound(bfqq); - bfqq_data.saved_io_start_time = bfqq->io_start_time; - bfqq_data.saved_tot_idle_time = bfqq->tot_idle_time; - bfqq_data.saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); - bfqq_data.was_in_burst_list = + bfqq_data->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); + bfqq_data->saved_io_start_time = bfqq->io_start_time; + bfqq_data->saved_tot_idle_time = bfqq->tot_idle_time; + bfqq_data->saved_in_large_burst = + bfq_bfqq_in_large_burst(bfqq); + bfqq_data->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node); if (unlikely(bfq_bfqq_just_created(bfqq) && @@ -3089,21 +3103,23 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) * to bfqq, so that to avoid that bfqq unjustly fails * to enjoy weight raising if split soon. */ - bfqq_data.saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; - bfqq_data.saved_wr_start_at_switch_to_srt = + bfqq_data->saved_wr_coeff = + bfqq->bfqd->bfq_wr_coeff; + bfqq_data->saved_wr_start_at_switch_to_srt = bfq_smallest_from_now(); - bfqq_data.saved_wr_cur_max_time = + bfqq_data->saved_wr_cur_max_time = bfq_wr_duration(bfqq->bfqd); - bfqq_data.saved_last_wr_start_finish = jiffies; + bfqq_data->saved_last_wr_start_finish = jiffies; } else { - bfqq_data.saved_wr_coeff = bfqq->wr_coeff; - bfqq_data.saved_wr_start_at_switch_to_srt = + bfqq_data->saved_wr_coeff = bfqq->wr_coeff; + bfqq_data->saved_wr_start_at_switch_to_srt = bfqq->wr_start_at_switch_to_srt; - bfqq_data.saved_service_from_wr = + bfqq_data->saved_service_from_wr = bfqq->service_from_wr; - bfqq_data.saved_last_wr_start_finish = + bfqq_data->saved_last_wr_start_finish = bfqq->last_wr_start_finish; - bfqq_data.saved_wr_cur_max_time = bfqq->wr_cur_max_time; + bfqq_data->saved_wr_cur_max_time = + bfqq->wr_cur_max_time; } } @@ -5425,7 +5441,7 @@ static void bfq_exit_icq(struct io_cq *icq) unsigned long flags; unsigned int act_idx; unsigned int num_actuators; - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = bic->bfqq_data; /* * bfqd is NULL if scheduler already exited, and in that case @@ -5445,10 +5461,10 @@ static void bfq_exit_icq(struct io_cq *icq) num_actuators = BFQ_MAX_ACTUATORS; } - if (bfqq_data.stable_merge_bfqq) - bfq_put_stable_ref(bfqq_data.stable_merge_bfqq); - for (act_idx = 0; act_idx < num_actuators; act_idx++) { + if (bfqq_data[act_idx].stable_merge_bfqq) + bfq_put_stable_ref(bfqq_data[act_idx].stable_merge_bfqq); + bfq_exit_icq_bfqq(bic, true, act_idx); bfq_exit_icq_bfqq(bic, false, act_idx); } @@ -5635,16 +5651,16 @@ bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, struct bfq_io_cq *bic, struct bfq_queue *last_bfqq_created) { + unsigned int a_idx = last_bfqq_created->actuator_idx; struct bfq_queue *new_bfqq = bfq_setup_merge(bfqq, last_bfqq_created); - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; if (!new_bfqq) return bfqq; if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = true; - bfqq_data.stably_merged = true; + new_bfqq->bic->bfqq_data[a_idx].stably_merged = true; + bic->bfqq_data[a_idx].stably_merged = true; /* * Reusing merge functions. This implies that @@ -5713,7 +5729,6 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, &bfqd->last_bfqq_created; struct bfq_queue *last_bfqq_created = *source_bfqq; - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; /* * If last_bfqq_created has not been set yet, then init it. If @@ -5775,7 +5790,8 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, /* * Record the bfqq to merge to. */ - bfqq_data.stable_merge_bfqq = last_bfqq_created; + bic->bfqq_data[last_bfqq_created->actuator_idx].stable_merge_bfqq = + last_bfqq_created; } } @@ -6696,7 +6712,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, { unsigned int act_idx = bfq_actuator_index(bfqd, bio); struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); - struct bfq_iocq_bfqq_data bfqq_data = bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[act_idx]; if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6710,12 +6726,13 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bic_set_bfqq(bic, bfqq, is_sync, act_idx); if (split && is_sync) { - if ((bfqq_data.was_in_burst_list && bfqd->large_burst) || - bfqq_data.saved_in_large_burst) + if ((bfqq_data->was_in_burst_list && + bfqd->large_burst) || + bfqq_data->saved_in_large_burst) bfq_mark_bfqq_in_large_burst(bfqq); else { bfq_clear_bfqq_in_large_burst(bfqq); - if (bfqq_data.was_in_burst_list) + if (bfqq_data->was_in_burst_list) /* * If bfqq was in the current * burst list before being @@ -6804,7 +6821,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) struct bfq_queue *bfqq; bool new_queue = false; bool bfqq_already_existing = false, split = false; - struct bfq_iocq_bfqq_data bfqq_data; + unsigned int a_idx = bfq_actuator_index(bfqd, bio); if (unlikely(!rq->elv.icq)) return NULL; @@ -6828,17 +6845,16 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, &new_queue); - bfqq_data = bic->bfqq_data; - if (likely(!new_queue)) { /* If the queue was seeky for too long, break it apart. */ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq) && - !bfqq_data.stably_merged) { + !bic->bfqq_data[a_idx].stably_merged) { struct bfq_queue *old_bfqq = bfqq; /* Update bic before losing reference to bfqq */ if (bfq_bfqq_in_large_burst(bfqq)) - bfqq_data.saved_in_large_burst = true; + bic->bfqq_data[a_idx].saved_in_large_burst = + true; bfqq = bfq_split_bfqq(bic, bfqq); split = true; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index f2e8ab91951c..e27897d66a0f 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -416,7 +416,7 @@ struct bfq_queue { struct bfq_iocq_bfqq_data { /* * Snapshot of the has_short_time flag before merging; taken - * to remember its value while the queue is merged, so as to + * to remember its values while the queue is merged, so as to * be able to restore it in case of split. */ bool saved_has_short_ttime; @@ -430,7 +430,7 @@ struct bfq_iocq_bfqq_data { u64 saved_tot_idle_time; /* - * Same purpose as the previous fields for the value of the + * Same purpose as the previous fields for the values of the * field keeping the queue's belonging to a large burst */ bool saved_in_large_burst; @@ -493,8 +493,12 @@ struct bfq_io_cq { uint64_t blkcg_serial_nr; /* the current blkcg serial */ #endif - /* persistent data for associated synchronous process queue */ - struct bfq_iocq_bfqq_data bfqq_data; + /* + * Persistent data for associated synchronous process queues + * (one queue per actuator, see field bfqq above). In + * particular, each of these queues may undergo a merge. + */ + struct bfq_iocq_bfqq_data bfqq_data[BFQ_MAX_ACTUATORS]; unsigned int requests; /* Number of requests this process has in flight */ }; From patchwork Sun Oct 30 10:02:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12993 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726344wru; Sun, 30 Oct 2022 03:05:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6SJpIPy3I3VzCfpgQUuof10ILIk+8Ri3rK4qk3xOgHeh1UBxNirF7lfzYXB1NsNh1W4b/y X-Received: by 2002:a63:1861:0:b0:462:4961:9a8f with SMTP id 33-20020a631861000000b0046249619a8fmr7662934pgy.372.1667124332251; Sun, 30 Oct 2022 03:05:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124332; cv=none; d=google.com; s=arc-20160816; b=C8Dq9on68vP4Q8rW9decSV9ec2MHJkA1LihuZ64OXmjCR1O93o0moUhbaOtG6jJWiv 2gtXPrNiOCQ0ZJgbXi5qVC1vX34V/6/BZAhnnW3eIYbmb8guoCXFACW5ZfEdIki2eOQ7 riEXsg3kDvTWm+xjO8/2FNrvpA4gaEt24JB2qywQ5p8vaueQfohKx6o9XiMx4D4XGvvi ZmmvUKL1wPY7Wi2nIsidi5wi/1tS26fqxwsZ03y4IrAr9GoB43uRwXmPUrG+e2MdT8rw nKUT5C9KC20ebJph4/r3EDrBZrZno9YFC/zqRrZfTeC2iiCSp+ykCmnYKBV9HIaNqGx3 ccNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=krwG5u3iMnmu3ZPyM320ApdNLoJb3oY0Z8Dtz0r35Fs=; b=j68PoWsDs+9Qx1xUP5sl/n+4hz6hsczTSeMqY83AMsGRXZeGOFhbhGEc9Wy2wb46xn Z5Mg/NgGDy3lsArloNNN+IP9Q4tu+sbOvTmgP2kiTLdswIv2ea/1cG/GoqucDWcYbk4a HhK4EkhEYVIm9Kl986RKzejVoa/uhNooM7epKc5ke6dqAONGR2a9ykZn2O2CENhuVFlI 5brLQHhVuXwLK6ZDcApUORvhtCNC0r3d6AglxA7F+PlziREmWtjXcWMbBLoZGxTJyCmt czuxNvYyygzeP8hSBRwdf6xl5JXOiIJFdcnQYciZch7/nYfmc7bUGFTA4ws9o75WNXIf dRHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jNOSzmpH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w64-20020a638243000000b00453f9620c4dsi4665863pgd.503.2022.10.30.03.05.19; Sun, 30 Oct 2022 03:05:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jNOSzmpH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230064AbiJ3KEF (ORCPT + 99 others); Sun, 30 Oct 2022 06:04:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbiJ3KDm (ORCPT ); Sun, 30 Oct 2022 06:03:42 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 547231F2 for ; Sun, 30 Oct 2022 03:03:41 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id a67so13662628edf.12 for ; Sun, 30 Oct 2022 03:03:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=krwG5u3iMnmu3ZPyM320ApdNLoJb3oY0Z8Dtz0r35Fs=; b=jNOSzmpHJBbkT5huF8E1SaX7WqMvDB9fnJk8W0LOrGS3g1NReJYWh0CfxzZP6B8bEj H53Z44w1dPVbv7h4NXh7OELX2CXFUPjDOieVp8BQ6a4LsfOL53LOPIdOS+TrN6bi3yQi YBwiA6nrFVunL51D8WHcgOuKbdoGp6MCEQ3zUcrO60xqZ1Rr145kG57RHRZPwGmi87nu x23zHyot3FHmvhOUzOqYXYADqUCGiyaIqogjHLGOrOzT6GCrlXpHOpVrWDlsX0G5e6aM tw9v+EPKCUd9VewrDVtDzTB6W3bLWTm6MDvdK7YftEMta16k4YzGwLuIVX57kbXsHRLT YUYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=krwG5u3iMnmu3ZPyM320ApdNLoJb3oY0Z8Dtz0r35Fs=; b=e5ZBpIb8qz75jy8It94uDiazcAuUqDbEdKviJuaCh2NJXlWwd1NmVytuFXCctBwbUN ogEmAmeRCe/jdIeQP6lYd+dV3ThKA4xx73kI2xdVeJr1MFQAR8kF44GIlb40F4RuvM2b LxSYacmulqdDzmSZ+Eoxv4EAZ28Cw0oalnJ38ZdsixmZ4UdE8UwUjAYj5+0DmtbDpTIb 3kIOmUwsGk3McGPEsTTOFgevEVBgkHHrk0cqoX8EEydWmVGOcSRVdQ+BwS0yD5m6zzYW AABtWN8R1r1tNY9nyRRVvBpkgNtcQb7mGqRKI1teq52ys7fc+BqXKZhkbvAFjUWP1pDO U+qg== X-Gm-Message-State: ACrzQf2S0wBJ48lfFadcmv+VGUP3w9z+7DvijKsBrSkbR249WC0hQheb jI7Y+cR20ijNMxGX2K+SoEpc2A== X-Received: by 2002:a05:6402:5187:b0:461:ac01:7512 with SMTP id q7-20020a056402518700b00461ac017512mr8011937edd.327.1667124219916; Sun, 30 Oct 2022 03:03:39 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:39 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Davide Zini , Paolo Valente Subject: [PATCH V5 5/8] block, bfq: split also async bfq_queues on a per-actuator basis Date: Sun, 30 Oct 2022 11:02:57 +0100 Message-Id: <20221030100300.3085-6-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106563857002610?= X-GMAIL-MSGID: =?utf-8?q?1748106563857002610?= From: Davide Zini Similarly to sync bfq_queues, also async bfq_queues need to be split on a per-actuator basis. Signed-off-by: Paolo Valente Signed-off-by: Davide Zini --- block/bfq-iosched.c | 41 +++++++++++++++++++++++------------------ block/bfq-iosched.h | 8 ++++---- 2 files changed, 27 insertions(+), 22 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index bfdf954da5b7..f1ea24775d90 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2675,14 +2675,16 @@ static void bfq_bfqq_end_wr(struct bfq_queue *bfqq) void bfq_end_wr_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) { - int i, j; - - for (i = 0; i < 2; i++) - for (j = 0; j < IOPRIO_NR_LEVELS; j++) - if (bfqg->async_bfqq[i][j]) - bfq_bfqq_end_wr(bfqg->async_bfqq[i][j]); - if (bfqg->async_idle_bfqq) - bfq_bfqq_end_wr(bfqg->async_idle_bfqq); + int i, j, k; + + for (k = 0; k < bfqd->num_actuators; k++) { + for (i = 0; i < 2; i++) + for (j = 0; j < IOPRIO_NR_LEVELS; j++) + if (bfqg->async_bfqq[i][j][k]) + bfq_bfqq_end_wr(bfqg->async_bfqq[i][j][k]); + if (bfqg->async_idle_bfqq[k]) + bfq_bfqq_end_wr(bfqg->async_idle_bfqq[k]); + } } static void bfq_end_wr(struct bfq_data *bfqd) @@ -5629,18 +5631,18 @@ static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, static struct bfq_queue **bfq_async_queue_prio(struct bfq_data *bfqd, struct bfq_group *bfqg, - int ioprio_class, int ioprio) + int ioprio_class, int ioprio, int act_idx) { switch (ioprio_class) { case IOPRIO_CLASS_RT: - return &bfqg->async_bfqq[0][ioprio]; + return &bfqg->async_bfqq[0][ioprio][act_idx]; case IOPRIO_CLASS_NONE: ioprio = IOPRIO_BE_NORM; fallthrough; case IOPRIO_CLASS_BE: - return &bfqg->async_bfqq[1][ioprio]; + return &bfqg->async_bfqq[1][ioprio][act_idx]; case IOPRIO_CLASS_IDLE: - return &bfqg->async_idle_bfqq; + return &bfqg->async_idle_bfqq[act_idx]; default: return NULL; } @@ -5814,7 +5816,8 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, if (!is_sync) { async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class, - ioprio); + ioprio, + bfq_actuator_index(bfqd, bio)); bfqq = *async_bfqq; if (bfqq) goto out; @@ -7032,13 +7035,15 @@ static void __bfq_put_async_bfqq(struct bfq_data *bfqd, */ void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) { - int i, j; + int i, j, k; - for (i = 0; i < 2; i++) - for (j = 0; j < IOPRIO_NR_LEVELS; j++) - __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j]); + for (k = 0; k < bfqd->num_actuators; k++) { + for (i = 0; i < 2; i++) + for (j = 0; j < IOPRIO_NR_LEVELS; j++) + __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j][k]); - __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq); + __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq[k]); + } } /* diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index e27897d66a0f..f1c2e77cbf9a 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -976,8 +976,8 @@ struct bfq_group { void *bfqd; - struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS]; - struct bfq_queue *async_idle_bfqq; + struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS][BFQ_MAX_ACTUATORS]; + struct bfq_queue *async_idle_bfqq[BFQ_MAX_ACTUATORS]; struct bfq_entity *my_entity; @@ -993,8 +993,8 @@ struct bfq_group { struct bfq_entity entity; struct bfq_sched_data sched_data; - struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS]; - struct bfq_queue *async_idle_bfqq; + struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS][BFQ_MAX_ACTUATORS]; + struct bfq_queue *async_idle_bfqq[BFQ_MAX_ACTUATORS]; struct rb_root rq_pos_tree; }; From patchwork Sun Oct 30 10:02:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726497wru; Sun, 30 Oct 2022 03:05:58 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6DQ+TitkqY075D36Avkrt8DJ5LVYP8gQToczPOcPEi/cls9bCvWTmWdgUOg4KVrVvxg4Cd X-Received: by 2002:a17:90a:a4f:b0:213:8283:ac53 with SMTP id o73-20020a17090a0a4f00b002138283ac53mr16858369pjo.114.1667124358345; Sun, 30 Oct 2022 03:05:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124358; cv=none; d=google.com; s=arc-20160816; b=EJJho7CZ84uJF0GVWMdK+0VihDsJ9ZndWoJpCQ+rDc9Xx6iqTen7eo2X612Au35/Mn 8iIdgnGw9XjLG3arIAbp6Q20LJlsVFvdGCubE5hhu28r0KNikP4mZp/Ljg7+iNqvnX21 4jSCMk9z1hZPsVY7A/s209fbfhbs5UttIFJr/RsC+lSxqV5NXyjZF++IYXjJDZMg4Edo vXz5HBnyJEJZu+XggGWoVBKIPrlA/lWieNPLml1t3oz43MCdCoCcpbqqyayRQjqJ0psU /7EnIMawFpssgJiDzLAxLkiVHUkFpKH2HO31JfYFljeLoWBtdSma5RL0Ww+MYLqPIlTU 9+RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=w0jpEUiIStstn+cz9/9ycvh0etWYQc/7Lgp1ggYo6Sw=; b=v8mSt4GtBSnLmPFvbEee5nNa6CeG2s4rbyQbCXoLGoMfF1qLPxgxg9hjbGe5lVaBie VaCUfTQ/HTCiKZ/oAkDLf6XX4jXd+iSTAllIaDLN9RjcGnnOqFxpDTJIBMkUoNx35X8U 6IKpZVURR9H2JCM5jJDKSp3N9E6o29h1abkw6li8RQ/5BTacnV5Qm9+MLulmSj+wglfy 6lrllpwBRreRe8HOIWIBUemGlLk0v5t9r2soLi48fnqSBbzRUD3eEOx9OiC5mhZ+ObDg fM445I/j8iyZtYf1ncIElZSKnBj2RrOG1gVEFg5Rve2iLjtvmlCTzLC54t1lmVBG8vRb 1NuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Pkc1XWVi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m11-20020aa78a0b000000b0056cac533dfdsi4333552pfa.261.2022.10.30.03.05.45; Sun, 30 Oct 2022 03:05:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Pkc1XWVi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230158AbiJ3KEK (ORCPT + 99 others); Sun, 30 Oct 2022 06:04:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229552AbiJ3KDn (ORCPT ); Sun, 30 Oct 2022 06:03:43 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A44401A1 for ; Sun, 30 Oct 2022 03:03:42 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id b2so22853787eja.6 for ; Sun, 30 Oct 2022 03:03:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w0jpEUiIStstn+cz9/9ycvh0etWYQc/7Lgp1ggYo6Sw=; b=Pkc1XWViIueLswGPwN/I3OqzS7h/cpEacd6eJo4Ox3Jaj4auWsLxY8H+eHhQbfZwRD XOHTF0roKn+zZoEmsuyC6vkLOz8mEgN0JQshN6I1VxjapsOrLnNL7mm1/Y4OmCxXj+Qk L7gm1cthaMVcLQlHjVdf6lqZvSjYpp63pN0nuvpM1Ifck3HVrDqDucK7+J3rZCUbN1S7 nvpo5hagUfc+/DgMLJtVHcs/jdIaushxUGwrjwDK4hLX0f5HJGnnhgUMIpQ1ZpcUXxR1 0ElKUXESHZyiyad1V7csYj3tfFXyORTkbtVstwkjCMAP6z7H2qiVkH40GCPbqMG9BaO8 Hnow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w0jpEUiIStstn+cz9/9ycvh0etWYQc/7Lgp1ggYo6Sw=; b=vqF+KwswzbZqNpJwljrK60cjYygNMVUtMI8eIjed81SizmobJfLo/jNVtAu88F8Nwa WVIeNi64zpwkjYASF1qoEK5qXbMlzLkFS0aWN+pOOt8QYRMbNpYN4AmV4ijx/V9Rc5+x NWpylTDPbFuqkmulktE7b2rJvOPzfA3KqFLUCjFSXfXvk4LyyBJpjWDz9QFYFElO7ZAj BCvcnE8metczxRfocpj5xv5NyAu4CiUA4EbI8xM8mnLof7kPNq4ICoSRqs2yiztUWwyQ 8i22qRUtBpli984PD0UFBvSltqIQ0GZoCmv5lpdlqSEzuT/EyN4tPvDsWPmNmO/8+m8K VUQw== X-Gm-Message-State: ACrzQf1aeXLD/dpMBHBwtc2/hLHT3E3zYg9u7ocWgwh7YtGVsJPII5e0 v5rTd8ReKgij1RGDnEiG5UlB1Q== X-Received: by 2002:a17:907:6e03:b0:78e:1c82:1f2a with SMTP id sd3-20020a1709076e0300b0078e1c821f2amr7190243ejc.611.1667124221217; Sun, 30 Oct 2022 03:03:41 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:40 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Federico Gavioli , Rory Chen , Paolo Valente Subject: [PATCH V5 6/8] block, bfq: retrieve independent access ranges from request queue Date: Sun, 30 Oct 2022 11:02:58 +0100 Message-Id: <20221030100300.3085-7-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106591333460431?= X-GMAIL-MSGID: =?utf-8?q?1748106591333460431?= From: Federico Gavioli This patch implements the code to gather the content of the independent_access_ranges structure from the request_queue and copy it into the queue's bfq_data. This copy is done at queue initialization. We copy the access ranges into the bfq_data to avoid taking the queue lock each time we access the ranges. This implementation, however, puts a limit to the maximum independent ranges supported by the scheduler. Such a limit is equal to the constant BFQ_MAX_ACTUATORS. This limit was placed to avoid the allocation of dynamic memory. Co-developed-by: Rory Chen Signed-off-by: Rory Chen Signed-off-by: Federico Gavioli Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 54 ++++++++++++++++++++++++++++++++++++++------- block/bfq-iosched.h | 5 +++++ 2 files changed, 51 insertions(+), 8 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index f1ea24775d90..6f464d422098 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1833,10 +1833,26 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, /* get the index of the actuator that will serve bio */ static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) { - /* - * Multi-actuator support not complete yet, so always return 0 - * for the moment. - */ + struct blk_independent_access_range *iar; + unsigned int i; + sector_t end; + + /* no search needed if one or zero ranges present */ + if (bfqd->num_actuators < 2) + return 0; + + /* bio_end_sector(bio) gives the sector after the last one */ + end = bio_end_sector(bio) - 1; + + for (i = 0; i < bfqd->num_actuators; i++) { + iar = &(bfqd->ia_ranges[i]); + if (end >= iar->sector && end < iar->sector + iar->nr_sectors) + return i; + } + + WARN_ONCE(true, + "bfq_actuator_index: bio sector out of ranges: end=%llu\n", + end); return 0; } @@ -2481,7 +2497,6 @@ static void bfq_remove_request(struct request_queue *q, if (rq->cmd_flags & REQ_META) bfqq->meta_pending--; - } static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, @@ -7154,6 +7169,8 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) { struct bfq_data *bfqd; struct elevator_queue *eq; + unsigned int i; + struct blk_independent_access_ranges *ia_ranges = q->disk->ia_ranges; eq = elevator_alloc(q, e); if (!eq) @@ -7197,10 +7214,31 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue = q; /* - * Multi-actuator support not complete yet, default to single - * actuator for the moment. + * If the disk supports multiple actuators, we copy the independent + * access ranges from the request queue structure. */ - bfqd->num_actuators = 1; + spin_lock_irq(&q->queue_lock); + if (ia_ranges) { + /* + * Check if the disk ia_ranges size exceeds the current bfq + * actuator limit. + */ + if (ia_ranges->nr_ia_ranges > BFQ_MAX_ACTUATORS) { + pr_crit("nr_ia_ranges higher than act limit: iars=%d, max=%d.\n", + ia_ranges->nr_ia_ranges, BFQ_MAX_ACTUATORS); + pr_crit("Falling back to single actuator mode.\n"); + bfqd->num_actuators = 0; + } else { + bfqd->num_actuators = ia_ranges->nr_ia_ranges; + + for (i = 0; i < bfqd->num_actuators; i++) + bfqd->ia_ranges[i] = ia_ranges->ia_range[i]; + } + } else { + bfqd->num_actuators = 0; + } + + spin_unlock_irq(&q->queue_lock); INIT_LIST_HEAD(&bfqd->dispatch); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index f1c2e77cbf9a..90130a893c8f 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -811,6 +811,11 @@ struct bfq_data { */ unsigned int num_actuators; + /* + * Disk independent access ranges for each actuator + * in this device. + */ + struct blk_independent_access_range ia_ranges[BFQ_MAX_ACTUATORS]; }; enum bfqq_state_flags { From patchwork Sun Oct 30 10:02:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12997 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726589wru; Sun, 30 Oct 2022 03:06:17 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7vUFDn4WorhNT/A9zdDiK0Zkx1RZdqQ2ysxjdd4VS32ogC5LG5gZ/5P8yfmqzXqrzgxh4f X-Received: by 2002:a17:902:d4cc:b0:186:f57d:ba61 with SMTP id o12-20020a170902d4cc00b00186f57dba61mr8536695plg.97.1667124377391; Sun, 30 Oct 2022 03:06:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124377; cv=none; d=google.com; s=arc-20160816; b=pdqA6maFNxyU2oDSQJHI74PgrrpN/2Qx+5Wj4AxA5RrhyejR7OYGYQfYHtJJNK38go o9TIYzEgSO6OB4tsLw0S/0YI4fxHTLR3uFPOMG0nLx0T8iTNnkv21+KUrQkY2onOO9+g O9rewfAgSKuXUYUH8yqkItmUlx4LWxuUsJUi4bcfUa7XoqU7z88q5k5Agn7Btgh/c2m1 0CVuR/EshILu/iGrilql3Sz0mTeQpr2pETCEc2xZI1EPZ785tHYlJ8TJYt82nhULx7t8 CJx5DEszzsxxgK5FR7lHc2vocvlMKSrIJSsPu1R3HDDjJjk5N3NgTYi7m+V7J3fa925k RGyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mNQDX/3BgflPUItKLMTHQsd/7uDvCbPOZxZcndPgzvw=; b=NH3JWjeHCvbugYJ0/aF1cfD/zPNMT7ZHahguEk6opjmMyrbARPCmxJ6mOLZOA8cgX5 q/Vk/fzqptibCbWHpHuP6GF1/T+TEq5k30mhYC5BVj+zgx1czxE85CDpx5Qruth7S5fJ arsVSpHmjYZLRWn6rRyLWyCaE/Bs3DbTYf5vl566Vf52FXBHnYUzl98jcjamO6w7k7Y6 oxIrtCGBMWMzjm730iLqDN7Iu3IvKnNmg8yeHL3uasUrJH94poZRNdtiYepithDEh+uo stuxu8+GkFxOo/cEZMJzsX0rgz+TPJVIHIY+qzzo+pwEvD3lT8q3M/UcVgK1IfjlF69Z 9ETg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=l9tnIuhr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id li16-20020a17090b48d000b002131407c208si4962855pjb.101.2022.10.30.03.06.05; Sun, 30 Oct 2022 03:06:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=l9tnIuhr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230060AbiJ3KER (ORCPT + 99 others); Sun, 30 Oct 2022 06:04:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230063AbiJ3KDq (ORCPT ); Sun, 30 Oct 2022 06:03:46 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFDC4DEC for ; Sun, 30 Oct 2022 03:03:43 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id r14so13704166edc.7 for ; Sun, 30 Oct 2022 03:03:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mNQDX/3BgflPUItKLMTHQsd/7uDvCbPOZxZcndPgzvw=; b=l9tnIuhrO2XBhtPnHth7rTlh9rNNG+b9pCqDVZJV1jO8CDfCBbimoVhZ35guCGE8pa m5EE5zndcKtq5KmsGSSOdV1e1uEtTdgPIqTEmf38q9jwS7DhoU0NUp4Hb9Y0SpDl0urY OVN4613EwLpbPolpRYw4T8G5H3PD3czjsK7ebHQuk1LKNZtLHQ30sFHusNrRb6nnE7wa SnGLQ9OqsrLP7R4HGgwnV0ZocBxoOYDUkUv/WzCGaBVD/ZnP+O+7HRMAhZjLyru51Cwi QAUfJ1qmxvGQMwaQQ5BiceLDErcoGjkSth/8QO2vvjxsE1yy8FFWzAqngEZIiKoN4lR/ f2cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mNQDX/3BgflPUItKLMTHQsd/7uDvCbPOZxZcndPgzvw=; b=lzPbxYPFt/qaaLtjDsltPVxdbxIBk8CfzMhnPybcxocMQsbnJkO0Z4Fs6sfbMtp1CI Gxezrb2joWR31LuwQ5MF2GFhNNAEPUzk3f7GBD2MPNLQABSoAa9wq1fg7h7l+Qz4pNBF 0Sm6QSZvgl7tgFoH99iUztbIHQcao2DB1iWzXkFEyX5i1NovBcVHsJLzfmfLQNoOCh4e WQx5CFOVFeN7lD19yaep6ilyEVpwLRfjEphpyrxHdHlnZXpu9rD4HNSn+x2pcaw7OI0i yXU0QtJ18iRLVFQhVQdRy9rCdP6dIkgj2jUzG16cLSseyBcY0RthFefEIisiGQbqulyf x93Q== X-Gm-Message-State: ACrzQf1gHXgDWXy2FjQnMt4gLLv4qlaw4qqN/BOvCV4pFP0K7dpXtC6w wBRLlTVXuJRjiylYfC3dOPqUYg== X-Received: by 2002:a50:fe85:0:b0:458:5562:bf1e with SMTP id d5-20020a50fe85000000b004585562bf1emr7843570edt.167.1667124222229; Sun, 30 Oct 2022 03:03:42 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:41 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Davide Zini , Paolo Valente Subject: [PATCH V5 7/8] block, bfq: inject I/O to underutilized actuators Date: Sun, 30 Oct 2022 11:02:59 +0100 Message-Id: <20221030100300.3085-8-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106611118238966?= X-GMAIL-MSGID: =?utf-8?q?1748106611118238966?= From: Davide Zini The main service scheme of BFQ for sync I/O is serving one sync bfq_queue at a time, for a while. In particular, BFQ enforces this scheme when it deems the latter necessary to boost throughput or to preserve service guarantees. Unfortunately, when BFQ enforces this policy, only one actuator at a time gets served for a while, because each bfq_queue contains I/O only for one actuator. The other actuators may remain underutilized. Actually, BFQ may serve (inject) extra I/O, taken from other bfq_queues, in parallel with that of the in-service queue. This injection mechanism may provide the ground for dealing also with the above actuator-underutilization problem. Yet BFQ does not take the actuator load into account when choosing which queue to pick extra I/O from. In addition, BFQ may happen to inject extra I/O only when the in-service queue is temporarily empty. In view of these facts, this commit extends the injection mechanism in such a way that the latter: (1) takes into account also the actuator load; (2) checks such a load on each dispatch, and injects I/O for an underutilized actuator, if there is one and there is I/O for it. To perform the check in (2), this commit introduces a load threshold, currently set to 4. A linear scan of each actuator is performed, until an actuator is found for which the following two conditions hold: the load of the actuator is below the threshold, and there is at least one non-in-service queue that contains I/O for that actuator. If such a pair (actuator, queue) is found, then the head request of that queue is returned for dispatch, instead of the head request of the in-service queue. We have set the threshold, empirically, to the minimum possible value for which an actuator is fully utilized, or close to be fully utilized. By doing so, injected I/O 'steals' as few drive-queue slots as possibile to the in-service queue. This reduces as much as possible the probability that the service of I/O from the in-service bfq_queue gets delayed because of slot exhaustion, i.e., because all the slots of the drive queue are filled with I/O injected from other queues (NCQ provides for 32 slots). This new mechanism also counters actuator underutilization in the case of asymmetric configurations of bfq_queues. Namely if there are few bfq_queues containing I/O for some actuators and many bfq_queues containing I/O for other actuators. Or if the bfq_queues containing I/O for some actuators have lower weights than the other bfq_queues. Signed-off-by: Paolo Valente Signed-off-by: Davide Zini --- block/bfq-cgroup.c | 2 +- block/bfq-iosched.c | 139 +++++++++++++++++++++++++++++++++----------- block/bfq-iosched.h | 39 ++++++++++++- block/bfq-wf2q.c | 2 +- 4 files changed, 143 insertions(+), 39 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index d243c429d9c0..38ccfe55ad46 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -694,7 +694,7 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_activate_bfqq(bfqd, bfqq); } - if (!bfqd->in_service_queue && !bfqd->rq_in_driver) + if (!bfqd->in_service_queue && !bfqd->tot_rq_in_driver) bfq_schedule_dispatch(bfqd); /* release extra ref taken above, bfqq may happen to be freed now */ bfq_put_queue(bfqq); diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 6f464d422098..c9af17a36219 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2254,6 +2254,7 @@ static void bfq_add_request(struct request *rq) bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); bfqq->queued[rq_is_sync(rq)]++; + /* * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it * may be read without holding the lock in bfq_has_work(). @@ -2299,9 +2300,9 @@ static void bfq_add_request(struct request *rq) * elapsed. */ if (bfqq == bfqd->in_service_queue && - (bfqd->rq_in_driver == 0 || + (bfqd->tot_rq_in_driver == 0 || (bfqq->last_serv_time_ns > 0 && - bfqd->rqs_injected && bfqd->rq_in_driver > 0)) && + bfqd->rqs_injected && bfqd->tot_rq_in_driver > 0)) && time_is_before_eq_jiffies(bfqq->decrease_time_jif + msecs_to_jiffies(10))) { bfqd->last_empty_occupied_ns = ktime_get_ns(); @@ -2325,7 +2326,7 @@ static void bfq_add_request(struct request *rq) * will be set in case injection is performed * on bfqq before rq is completed). */ - if (bfqd->rq_in_driver == 0) + if (bfqd->tot_rq_in_driver == 0) bfqd->rqs_injected = false; } } @@ -2423,15 +2424,18 @@ static sector_t get_sdist(sector_t last_pos, struct request *rq) static void bfq_activate_request(struct request_queue *q, struct request *rq) { struct bfq_data *bfqd = q->elevator->elevator_data; + unsigned int act_idx = bfq_actuator_index(bfqd, rq->bio); - bfqd->rq_in_driver++; + bfqd->tot_rq_in_driver++; + bfqd->rq_in_driver[act_idx]++; } static void bfq_deactivate_request(struct request_queue *q, struct request *rq) { struct bfq_data *bfqd = q->elevator->elevator_data; - bfqd->rq_in_driver--; + bfqd->tot_rq_in_driver--; + bfqd->rq_in_driver[bfq_actuator_index(bfqd, rq->bio)]--; } #endif @@ -2705,11 +2709,14 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd, static void bfq_end_wr(struct bfq_data *bfqd) { struct bfq_queue *bfqq; + int i; spin_lock_irq(&bfqd->lock); - list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) - bfq_bfqq_end_wr(bfqq); + for (i = 0; i < bfqd->num_actuators; i++) { + list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) + bfq_bfqq_end_wr(bfqq); + } list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list) bfq_bfqq_end_wr(bfqq); bfq_end_wr_async(bfqd); @@ -3660,13 +3667,13 @@ static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq) * - start a new observation interval with this dispatch */ if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC && - bfqd->rq_in_driver == 0) + bfqd->tot_rq_in_driver == 0) goto update_rate_and_reset; /* Update sampling information */ bfqd->peak_rate_samples++; - if ((bfqd->rq_in_driver > 0 || + if ((bfqd->tot_rq_in_driver > 0 || now_ns - bfqd->last_completion < BFQ_MIN_TT) && !BFQ_RQ_SEEKY(bfqd, bfqd->last_position, rq)) bfqd->sequential_samples++; @@ -3933,7 +3940,7 @@ static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd, return (bfqq->wr_coeff > 1 && (bfqd->wr_busy_queues < tot_busy_queues || - bfqd->rq_in_driver >= + bfqd->tot_rq_in_driver >= bfqq->dispatched + 4)) || bfq_asymmetric_scenario(bfqd, bfqq) || tot_busy_queues == 1; @@ -4705,6 +4712,7 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) { struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue; unsigned int limit = in_serv_bfqq->inject_limit; + int i; /* * If * - bfqq is not weight-raised and therefore does not carry @@ -4736,7 +4744,7 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) ) limit = 1; - if (bfqd->rq_in_driver >= limit) + if (bfqd->tot_rq_in_driver >= limit) return NULL; /* @@ -4751,11 +4759,12 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) * (and re-added only if it gets new requests, but then it * is assigned again enough budget for its new backlog). */ - list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) - if (!RB_EMPTY_ROOT(&bfqq->sort_list) && - (in_serv_always_inject || bfqq->wr_coeff > 1) && - bfq_serv_to_charge(bfqq->next_rq, bfqq) <= - bfq_bfqq_budget_left(bfqq)) { + for (i = 0; i < bfqd->num_actuators; i++) { + list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) + if (!RB_EMPTY_ROOT(&bfqq->sort_list) && + (in_serv_always_inject || bfqq->wr_coeff > 1) && + bfq_serv_to_charge(bfqq->next_rq, bfqq) <= + bfq_bfqq_budget_left(bfqq)) { /* * Allow for only one large in-flight request * on non-rotational devices, for the @@ -4780,22 +4789,69 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) else limit = in_serv_bfqq->inject_limit; - if (bfqd->rq_in_driver < limit) { + if (bfqd->tot_rq_in_driver < limit) { bfqd->rqs_injected = true; return bfqq; } } + } + + return NULL; +} + +static struct bfq_queue * +bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, + int idx) +{ + struct bfq_queue *bfqq = NULL; + + if (bfqd->in_service_queue && + bfqd->in_service_queue->actuator_idx == idx) + return bfqd->in_service_queue; + + list_for_each_entry(bfqq, &bfqd->active_list[idx], bfqq_list) { + if (!RB_EMPTY_ROOT(&bfqq->sort_list) && + bfq_serv_to_charge(bfqq->next_rq, bfqq) <= + bfq_bfqq_budget_left(bfqq)) { + return bfqq; + } + } return NULL; } +/* + * Perform a linear scan of each actuator, until an actuator is found + * for which the following two conditions hold: the load of the + * actuator is below the threshold (see comments on actuator_load_threshold + * for details), and there is a queue that contains I/O for that + * actuator. On success, return that queue. + */ +static struct bfq_queue * +bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) +{ + int i; + + for (i = 0 ; i < bfqd->num_actuators; i++) + if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold) { + struct bfq_queue *bfqq = + bfq_find_active_bfqq_for_actuator(bfqd, i); + + if (bfqq) + return bfqq; + } + + return NULL; +} + + /* * Select a queue for service. If we have a current queue in service, * check whether to continue servicing it, or retrieve and set a new one. */ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) { - struct bfq_queue *bfqq; + struct bfq_queue *bfqq, *inject_bfqq; struct request *next_rq; enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT; @@ -4817,6 +4873,15 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) goto expire; check_queue: + /* + * If some actuator is underutilized, but the in-service + * queue does not contain I/O for that actuator, then try to + * inject I/O for that actuator. + */ + inject_bfqq = bfq_find_bfqq_for_underused_actuator(bfqd); + if (inject_bfqq && inject_bfqq != bfqq) + return inject_bfqq; + /* * This loop is rarely executed more than once. Even when it * happens, it is much more convenient to re-execute this loop @@ -5172,11 +5237,11 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) /* * We exploit the bfq_finish_requeue_request hook to - * decrement rq_in_driver, but + * decrement tot_rq_in_driver, but * bfq_finish_requeue_request will not be invoked on * this request. So, to avoid unbalance, just start - * this request, without incrementing rq_in_driver. As - * a negative consequence, rq_in_driver is deceptively + * this request, without incrementing tot_rq_in_driver. As + * a negative consequence, tot_rq_in_driver is deceptively * lower than it should be while this request is in * service. This may cause bfq_schedule_dispatch to be * invoked uselessly. @@ -5185,7 +5250,7 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) * bfq_finish_requeue_request hook, if defined, is * probably invoked also on this request. So, by * exploiting this hook, we could 1) increment - * rq_in_driver here, and 2) decrement it in + * tot_rq_in_driver here, and 2) decrement it in * bfq_finish_requeue_request. Such a solution would * let the value of the counter be always accurate, * but it would entail using an extra interface @@ -5214,7 +5279,7 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) * Of course, serving one request at a time may cause loss of * throughput. */ - if (bfqd->strict_guarantees && bfqd->rq_in_driver > 0) + if (bfqd->strict_guarantees && bfqd->tot_rq_in_driver > 0) goto exit; bfqq = bfq_select_queue(bfqd); @@ -5225,7 +5290,8 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) if (rq) { inc_in_driver_start_rq: - bfqd->rq_in_driver++; + bfqd->rq_in_driver[bfqq->actuator_idx]++; + bfqd->tot_rq_in_driver++; start_rq: rq->rq_flags |= RQF_STARTED; } @@ -6298,7 +6364,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) struct bfq_queue *bfqq = bfqd->in_service_queue; bfqd->max_rq_in_driver = max_t(int, bfqd->max_rq_in_driver, - bfqd->rq_in_driver); + bfqd->tot_rq_in_driver); if (bfqd->hw_tag == 1) return; @@ -6309,7 +6375,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) * sum is not exact, as it's not taking into account deactivated * requests. */ - if (bfqd->rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD) + if (bfqd->tot_rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD) return; /* @@ -6320,7 +6386,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) if (bfqq && bfq_bfqq_has_short_ttime(bfqq) && bfqq->dispatched + bfqq->queued[0] + bfqq->queued[1] < BFQ_HW_QUEUE_THRESHOLD && - bfqd->rq_in_driver < BFQ_HW_QUEUE_THRESHOLD) + bfqd->tot_rq_in_driver < BFQ_HW_QUEUE_THRESHOLD) return; if (bfqd->hw_tag_samples++ < BFQ_HW_QUEUE_SAMPLES) @@ -6341,7 +6407,8 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) bfq_update_hw_tag(bfqd); - bfqd->rq_in_driver--; + bfqd->rq_in_driver[bfqq->actuator_idx]--; + bfqd->tot_rq_in_driver--; bfqq->dispatched--; if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) { @@ -6460,7 +6527,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) BFQQE_NO_MORE_REQUESTS); } - if (!bfqd->rq_in_driver) + if (!bfqd->tot_rq_in_driver) bfq_schedule_dispatch(bfqd); } @@ -6591,13 +6658,13 @@ static void bfq_update_inject_limit(struct bfq_data *bfqd, * conditions to do it, or we can lower the last base value * computed. * - * NOTE: (bfqd->rq_in_driver == 1) means that there is no I/O + * NOTE: (bfqd->tot_rq_in_driver == 1) means that there is no I/O * request in flight, because this function is in the code * path that handles the completion of a request of bfqq, and, * in particular, this function is executed before - * bfqd->rq_in_driver is decremented in such a code path. + * bfqd->tot_rq_in_driver is decremented in such a code path. */ - if ((bfqq->last_serv_time_ns == 0 && bfqd->rq_in_driver == 1) || + if ((bfqq->last_serv_time_ns == 0 && bfqd->tot_rq_in_driver == 1) || tot_time_ns < bfqq->last_serv_time_ns) { if (bfqq->last_serv_time_ns == 0) { /* @@ -6607,7 +6674,7 @@ static void bfq_update_inject_limit(struct bfq_data *bfqd, bfqq->inject_limit = max_t(unsigned int, 1, old_limit); } bfqq->last_serv_time_ns = tot_time_ns; - } else if (!bfqd->rqs_injected && bfqd->rq_in_driver == 1) + } else if (!bfqd->rqs_injected && bfqd->tot_rq_in_driver == 1) /* * No I/O injected and no request still in service in * the drive: these are the exact conditions for @@ -7249,7 +7316,8 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue_weights_tree = RB_ROOT_CACHED; bfqd->num_groups_with_pending_reqs = 0; - INIT_LIST_HEAD(&bfqd->active_list); + INIT_LIST_HEAD(&bfqd->active_list[0]); + INIT_LIST_HEAD(&bfqd->active_list[1]); INIT_LIST_HEAD(&bfqd->idle_list); INIT_HLIST_HEAD(&bfqd->burst_list); @@ -7294,6 +7362,9 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) ref_wr_duration[blk_queue_nonrot(bfqd->queue)]; bfqd->peak_rate = ref_rate[blk_queue_nonrot(bfqd->queue)] * 2 / 3; + /* see comments on the definition of next field inside bfq_data */ + bfqd->actuator_load_threshold = 4; + spin_lock_init(&bfqd->lock); /* diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 90130a893c8f..adb3ba6a9d90 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -586,7 +586,12 @@ struct bfq_data { /* number of queued requests */ int queued; /* number of requests dispatched and waiting for completion */ - int rq_in_driver; + int tot_rq_in_driver; + /* + * number of requests dispatched and waiting for completion + * for each actuator + */ + int rq_in_driver[BFQ_MAX_ACTUATORS]; /* true if the device is non rotational and performs queueing */ bool nonrot_with_queueing; @@ -680,8 +685,13 @@ struct bfq_data { /* maximum budget allotted to a bfq_queue before rescheduling */ int bfq_max_budget; - /* list of all the bfq_queues active on the device */ - struct list_head active_list; + /* + * List of all the bfq_queues active for a specific actuator + * on the device. Keeping active queues separate on a + * per-actuator basis helps implementing per-actuator + * injection more efficiently. + */ + struct list_head active_list[BFQ_MAX_ACTUATORS]; /* list of all the bfq_queues idle on the device */ struct list_head idle_list; @@ -816,6 +826,29 @@ struct bfq_data { * in this device. */ struct blk_independent_access_range ia_ranges[BFQ_MAX_ACTUATORS]; + + /* + * If the number of I/O requests queued in the device for a + * given actuator is below next threshold, then the actuator + * is deemed as underutilized. If this condition is found to + * hold for some actuator upon a dispatch, but (i) the + * in-service queue does not contain I/O for that actuator, + * while (ii) some other queue does contain I/O for that + * actuator, then the head I/O request of the latter queue is + * returned (injected), instead of the head request of the + * currently in-service queue. + * + * We set the threshold, empirically, to the minimum possible + * value for which an actuator is fully utilized, or close to + * be fully utilized. By doing so, injected I/O 'steals' as + * few drive-queue slots as possibile to the in-service + * queue. This reduces as much as possible the probability + * that the service of I/O from the in-service bfq_queue gets + * delayed because of slot exhaustion, i.e., because all the + * slots of the drive queue are filled with I/O injected from + * other queues (NCQ provides for 32 slots). + */ + unsigned int actuator_load_threshold; }; enum bfqq_state_flags { diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c index 8fc3da4c23bb..ec0273e2cd07 100644 --- a/block/bfq-wf2q.c +++ b/block/bfq-wf2q.c @@ -477,7 +477,7 @@ static void bfq_active_insert(struct bfq_service_tree *st, bfqd = (struct bfq_data *)bfqg->bfqd; #endif if (bfqq) - list_add(&bfqq->bfqq_list, &bfqq->bfqd->active_list); + list_add(&bfqq->bfqq_list, &bfqq->bfqd->active_list[bfqq->actuator_idx]); #ifdef CONFIG_BFQ_GROUP_IOSCHED if (bfqg != bfqd->root_group) bfqg->active_entities++; From patchwork Sun Oct 30 10:03:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 12996 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1726540wru; Sun, 30 Oct 2022 03:06:06 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6/e4H6JimsHCm2QGZOXmrJys4nFgAUW4KhAZ04YylYq3xAh27CEgj56ATar1tDzSmcVMgw X-Received: by 2002:a17:902:f68d:b0:185:43a1:555a with SMTP id l13-20020a170902f68d00b0018543a1555amr8468474plg.2.1667124366147; Sun, 30 Oct 2022 03:06:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667124366; cv=none; d=google.com; s=arc-20160816; b=SkoUcd2hsnIINhHTAstWr7gvM2ikNViE5oKgc9hUtuAeMTgIKiCLmkEy2AyYk06krj lR4Ce8Gb4UX+gmyUKfQxOjHubqa9y6+KAjabQZKXQBYPJCUWpWHtBWMfQ3Nbw5zCqt8E sbnykz50kNyi8OrdAiW4DefJ8Fgh8xQHKOYzmLY4J3hr0Xwkp15wuk3+2qFFNuf6NVAN y3JIF87fOk1K8y9/DMqITn+1/eQfhY5Wn9+oQNKdJVtPU5Ft7f19c6wN6aCFFIGay0Yf QV9xN29vC6HNdG804qORpASXEqvMNW3dbbOs8vRoBqc9aYSlC42NJLV8aqgB7t0ghAmw 9gIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jM3QRj1EVjuG2HTGTPAAvacJgPBGfkkz821afQrSz+o=; b=jyicEBM+rIqyIt70Y0BG8rpAo7VMwOWa0o2GGeIL8cXOdpKjfRC88TH47Dw03kiCzt Nh2z2cxE/zGHZiAHHyOY2WoKQMyLf5LkSF+gD3daaTX3wT/Av30HQR/ziFGx2P7aCG1e 9Qy1Y1poNm+eKHTtGImeTYWXGjiqwS0amgzncNOQYSGc0+HSoXdSvhtA13ZNiz/s3aT8 8AV60FlcUy7Cu/WO1rYyMwaQP+9HwSXJd3q2Hm7HqmwD0E3sSYWug0VFwns3tbzW6hbx WXFsqhTkI898V4mFBHGAHT6P+QI56rN5r2a07arQt+3Q/Du2OsVRMx9BFQe3LUG2wA8l srHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PbgYJjGD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c17-20020a170902f31100b00186b3464c9esi4211837ple.251.2022.10.30.03.05.53; Sun, 30 Oct 2022 03:06:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PbgYJjGD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230107AbiJ3KEM (ORCPT + 99 others); Sun, 30 Oct 2022 06:04:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230060AbiJ3KDo (ORCPT ); Sun, 30 Oct 2022 06:03:44 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F98F1F2 for ; Sun, 30 Oct 2022 03:03:43 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id kt23so22774426ejc.7 for ; Sun, 30 Oct 2022 03:03:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jM3QRj1EVjuG2HTGTPAAvacJgPBGfkkz821afQrSz+o=; b=PbgYJjGDH30zX4rdDzg7LxIyS9IvehyrFKlkCIMb7d38wicDPUuQlJrs0rlAyP5U5e gF2SvDn3urpigL1xA5RxlXjyWPOb8UoVzphOzXCUHrUdDoFVoc/hmGnszls6M1uxXFgr Beu4AjvpK5w1rgouIZ/uj4IBPDJkajcSKV90jRKzTRKOEgshyukaP6xnaFU20bw5DUnh lQnkNUR2CkKgR/utbea9eMQDriAxG5DFHGumBuVHbJfAXR3yzws9cA+NZgkib7aIOfOy I+7kNFs35qV3dXazGpwls6/22scL8c5Bwqmypqnvt5o5KODldohefOy5FkcF8RSNnunE OBTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jM3QRj1EVjuG2HTGTPAAvacJgPBGfkkz821afQrSz+o=; b=3uYPFuOoYsOZmjySCWQI98B1b9bWK/pctMGMnrsO68/a3V+8TFnLZyLeJcegxVy+Rw moK5Ew3qrk+QOx5q0FqqRRx8A3L55oQWYyFffBWJp0yHYmTKxVayJkwX1BYW4Ht15gPR seBCumD2MBXXsiA9SUWVixP14RiI1xDUBrmgbvYU3Xi7NYwN7TyZm0LqC3BoDwrnuvkx TG39QGfAJumaPHrhrIgufJZddfShGpEU58XLdUupGRtCdIlM0DTbIyZPceMObz8EcrRc VrS11LmQcXOk3Z3S9Wq1ZRKv3JqvLSqKKn4gGVgCG+SGvUMuZQkdHFZI5F3xqUiJOtvs ujog== X-Gm-Message-State: ACrzQf1okYI9eepY+rqoAgNK9BXAtc0nSmoIq8StRiZSehJAu5cBeAfQ MBZXRTeXPqlMqak/5dz2eJkUFA== X-Received: by 2002:a17:907:761b:b0:7a3:86dd:d330 with SMTP id jx27-20020a170907761b00b007a386ddd330mr7454455ejc.34.1667124223196; Sun, 30 Oct 2022 03:03:43 -0700 (PDT) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id d27-20020a170906305b00b0073d71792c8dsm1666088ejd.180.2022.10.30.03.03.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Oct 2022 03:03:42 -0700 (PDT) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Davide Zini , Paolo Valente Subject: [PATCH V5 8/8] block, bfq: balance I/O injection among underutilized actuators Date: Sun, 30 Oct 2022 11:03:00 +0100 Message-Id: <20221030100300.3085-9-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221030100300.3085-1-paolo.valente@linaro.org> References: <20221030100300.3085-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748106599349558845?= X-GMAIL-MSGID: =?utf-8?q?1748106599349558845?= From: Davide Zini Upon the invocation of its dispatch function, BFQ returns the next I/O request of the in-service bfq_queue, unless some exception holds. One such exception is that there is some underutilized actuator, different from the actuator for which the in-service queue contains I/O, and that some other bfq_queue happens to contain I/O for such an actuator. In this case, the next I/O request of the latter bfq_queue, and not of the in-service bfq_queue, is returned (I/O is injected from that bfq_queue). To find such an actuator, a linear scan, in increasing index order, is performed among actuators. Performing a linear scan entails a prioritization among actuators: an underutilized actuator may be considered for injection only if all actuators with a lower index are currently fully utilized, or if there is no pending I/O for any lower-index actuator that happens to be underutilized. This commits breaks this prioritization and tends to distribute injection uniformly across actuators. This is obtained by adding the following condition to the linear scan: even if an actuator A is underutilized, A is however skipped if its load is higher than that of the next actuator. Signed-off-by: Paolo Valente Signed-off-by: Davide Zini --- block/bfq-iosched.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index c9af17a36219..77002ebcab39 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -4822,10 +4822,16 @@ bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, /* * Perform a linear scan of each actuator, until an actuator is found - * for which the following two conditions hold: the load of the - * actuator is below the threshold (see comments on actuator_load_threshold - * for details), and there is a queue that contains I/O for that - * actuator. On success, return that queue. + * for which the following three conditions hold: the load of the + * actuator is below the threshold (see comments on + * actuator_load_threshold for details) and lower than that of the + * next actuator (comments on this extra condition below), and there + * is a queue that contains I/O for that actuator. On success, return + * that queue. + * + * Performing a plain linear scan entails a prioritization among + * actuators. The extra condition above breaks this prioritization and + * tends to distribute injection uniformly across actuators. */ static struct bfq_queue * bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) @@ -4833,7 +4839,9 @@ bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) int i; for (i = 0 ; i < bfqd->num_actuators; i++) - if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold) { + if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold && + (i == bfqd->num_actuators - 1 || + bfqd->rq_in_driver[i] < bfqd->rq_in_driver[i+1])) { struct bfq_queue *bfqq = bfq_find_active_bfqq_for_actuator(bfqd, i);