From patchwork Thu Dec 8 10:43:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31284 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp126044wrr; Thu, 8 Dec 2022 03:01:10 -0800 (PST) X-Google-Smtp-Source: AA0mqf4aDWYQdkQY61/S9Rybsv1lyRJ1u3HI7tDcMJTBa2oIUZJD8kFQj6UHdtmsIujLsC2X8FXX X-Received: by 2002:a50:fb16:0:b0:46a:fd15:3129 with SMTP id d22-20020a50fb16000000b0046afd153129mr44769198edq.165.1670497270024; Thu, 08 Dec 2022 03:01:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670497270; cv=none; d=google.com; s=arc-20160816; b=lZPEt51KzaGVmB9aOsdctk5dXya0bPvNhuVp44rEeEHtzdOtL3XPt5XvtWXXM8HuYQ A4UWpeAa4sI48SdO/h/Sd5bYWk8PLxm1iFhMZ7G1SWIaSR2PTOzWY/3DnK7r8NVjjH6J UJrJITsIxd6oLWUf1OrEwQkGsHUlYG0Ui3IxK69jvOQvxz9XGVgPcCc5dltyFkQ57t5M GhiQR01QMWGVief8IDIVZ71lsdVQj2fAM1S2e6nc6CgWSTWhNXX8SanBjvgwFULsFqZk xM7wcQPAZYtPpMaRbvWCRtVSD3/s8G5han/HaTanhFiPlNv1+ASuYfP8UAU+fXy0YpIl XOUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MHNOxudxM6McPM4QNqz211aWfFtZ/qdgv2gbbQO9ibA=; b=WJIQLZkSrPtMGln/PLRqgc9eS33bWbdH/dIDIe7U1xDy8BMfX8WVGnqb2yWdq53bgJ SpM/eeJQ94TUUQylGEjHbm/XzOuh95BeLlozvnC25K0Ez6OyqxNo1z9SLwkCODPE3inQ Q+ITCA39KHWF9HFWEyF9P8iVrp85NxShtK4d6wRjsaM5F3w0CzqWlc/egO/H503TBmw4 G/CMfECi6l+4FpT+xRLG26g0OiJFwLu+vE/ly4Z1Mgm6VQNFR7Zls2DXmKDclVWCOxx+ NhFNlcAlkJOwYC9APecBMSj9AIPYdJwoooNJwLxyOP6KxaENlhhOSKwIlB5W2VADaKxy 7jmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qybt3oJB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v13-20020a170906b00d00b0077fc66b581esi13022970ejy.688.2022.12.08.03.00.43; Thu, 08 Dec 2022 03:01:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qybt3oJB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230345AbiLHKuO (ORCPT + 99 others); Thu, 8 Dec 2022 05:50:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbiLHKs6 (ORCPT ); Thu, 8 Dec 2022 05:48:58 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95CDE5C0CE for ; Thu, 8 Dec 2022 02:44:00 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id gh17so2877973ejb.6 for ; Thu, 08 Dec 2022 02:44:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MHNOxudxM6McPM4QNqz211aWfFtZ/qdgv2gbbQO9ibA=; b=Qybt3oJB1aA4r80UQUqSZya5nZkEzaYeSdp95K36VW5FPlzlvQZiXrSF6UGrLVe55a AkqouBPSy7XMAf38hv12Kuzo95S9LPiW56OD/U/zMVI5YgYoyQm0QUAEkr+E7rjruDnH GXZf9o/gklT0uI25Rxpteeb5byz/P/sIYRnR5+I1Jz07NE3qSuuwqj2tL68DL0+AmjZq r1mSIqnSTcypcfwY0xVZRdwpcQUZrvBnz2qiPX3ZQWHl0r/TC7FQShEQYmtNRpM/YmMa YVt9eY+RRPoYyiiIsdRUXkvA5rUFy6pEcKJTMhdDFPD9yR2P7HU3iN9Lt0QZBZybt2vm SzNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MHNOxudxM6McPM4QNqz211aWfFtZ/qdgv2gbbQO9ibA=; b=qZvo+MKAWT5rz58uLRBGY+tPm9Ws674doiYf9xnDHkuIEUWxrAdt/YhcAb4REd834m NQ27e2LNqDv2zA66mgEJtM/4eNhyIwTfS0pW1Ad6YJ/TPd5xOl+rvOuZCIG6x8tj6wpo dHfPPkbOyEYZaFZnAwvNcm9cBpJEJC614k1hgOn7mXerIrD43QMKYOCg6YIi4DOLtA9X TUwf1eEnmpfFrUsw4iVesMYLL4p7eFlUKHNjtEzOmsMFMUS3qx2dPDN2S14sW6gYHZ2v 6/+C3LNXDdVxZ6/EfZUKGatQhRRgC69YveCHxrPWKHz4zTJTOGQSG2lmrC90WsMSyXme H3bw== X-Gm-Message-State: ANoB5pnsbyszr5qmtwM8szngAwIkxor2HmW+s41n48jHfYJ2bg2p/jCw aCLQtpsAe9TVXgjg3VqxTyRiSw== X-Received: by 2002:a17:906:b108:b0:7b5:dd3e:69c1 with SMTP id u8-20020a170906b10800b007b5dd3e69c1mr62996950ejy.381.1670496239182; Thu, 08 Dec 2022 02:43:59 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.43.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:43:58 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Paolo Valente , Gabriele Felici , Carmine Zaccagnino Subject: [PATCH V9 1/8] block, bfq: split sync bfq_queues on a per-actuator basis Date: Thu, 8 Dec 2022 11:43:44 +0100 Message-Id: <20221208104351.35038-2-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751643345181574438?= X-GMAIL-MSGID: =?utf-8?q?1751643345181574438?= Single-LUN multi-actuator SCSI drives, as well as all multi-actuator SATA drives appear as a single device to the I/O subsystem [1]. Yet they address commands to different actuators internally, as a function of Logical Block Addressing (LBAs). A given sector is reachable by only one of the actuators. For example, Seagate’s Serial Advanced Technology Attachment (SATA) version contains two actuators and maps the lower half of the SATA LBA space to the lower actuator and the upper half to the upper actuator. Evidently, to fully utilize actuators, no actuator must be left idle or underutilized while there is pending I/O for it. The block layer must somehow control the load of each actuator individually. This commit lays the ground for allowing BFQ to provide such a per-actuator control. BFQ associates an I/O-request sync bfq_queue with each process doing synchronous I/O, or with a group of processes, in case of queue merging. Then BFQ serves one bfq_queue at a time. While in service, a bfq_queue is emptied in request-position order. Yet the same process, or group of processes, may generate I/O for different actuators. In this case, different streams of I/O (each for a different actuator) get all inserted into the same sync bfq_queue. So there is basically no individual control on when each stream is served, i.e., on when the I/O requests of the stream are picked from the bfq_queue and dispatched to the drive. This commit enables BFQ to control the service of each actuator individually for synchronous I/O, by simply splitting each sync bfq_queue into N queues, one for each actuator. In other words, a sync bfq_queue is now associated to a pair (process, actuator). As a consequence of this split, the per-queue proportional-share policy implemented by BFQ will guarantee that the sync I/O generated for each actuator, by each process, receives its fair share of service. This is just a preparatory patch. If the I/O of the same process happens to be sent to different queues, then each of these queues may undergo queue merging. To handle this event, the bfq_io_cq data structure must be properly extended. In addition, stable merging must be disabled to avoid loss of control on individual actuators. Finally, also async queues must be split. These issues are described in detail and addressed in next commits. As for this commit, although multiple per-process bfq_queues are provided, the I/O of each process or group of processes is still sent to only one queue, regardless of the actuator the I/O is for. The forwarding to distinct bfq_queues will be enabled after addressing the above issues. [1] https://www.linaro.org/blog/budget-fair-queueing-bfq-linux-io-scheduler-optimizations-for-multi-actuator-sata-hard-drives/ Signed-off-by: Gabriele Felici Signed-off-by: Carmine Zaccagnino Signed-off-by: Paolo Valente --- block/bfq-cgroup.c | 92 ++++++++++++++------------ block/bfq-iosched.c | 158 +++++++++++++++++++++++++++++--------------- block/bfq-iosched.h | 51 +++++++++++--- 3 files changed, 195 insertions(+), 106 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 7d624a3a3f0f..7ea77f994424 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -704,6 +704,47 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_put_queue(bfqq); } +static void bfq_sync_bfqq_move(struct bfq_data *bfqd, + struct bfq_queue *sync_bfqq, + struct bfq_io_cq *bic, + struct bfq_group *bfqg, + unsigned int act_idx) +{ + struct bfq_queue *bfqq; + + if (!sync_bfqq->new_bfqq && !bfq_bfqq_coop(sync_bfqq)) { + /* We are the only user of this bfqq, just move it */ + if (sync_bfqq->entity.sched_data != &bfqg->sched_data) + bfq_bfqq_move(bfqd, sync_bfqq, bfqg); + return; + } + + /* + * The queue was merged to a different queue. Check + * that the merge chain still belongs to the same + * cgroup. + */ + for (bfqq = sync_bfqq; bfqq; bfqq = bfqq->new_bfqq) + if (bfqq->entity.sched_data != + &bfqg->sched_data) + break; + if (bfqq) { + /* + * Some queue changed cgroup so the merge is not valid + * anymore. We cannot easily just cancel the merge (by + * clearing new_bfqq) as there may be other processes + * using this queue and holding refs to all queues + * below sync_bfqq->new_bfqq. Similarly if the merge + * already happened, we need to detach from bfqq now + * so that we cannot merge bio to a request from the + * old cgroup. + */ + bfq_put_cooperator(sync_bfqq); + bfq_release_process_ref(bfqd, sync_bfqq); + bic_set_bfqq(bic, NULL, true, act_idx); + } +} + /** * __bfq_bic_change_cgroup - move @bic to @bfqg. * @bfqd: the queue descriptor. @@ -718,53 +759,20 @@ static void *__bfq_bic_change_cgroup(struct bfq_data *bfqd, struct bfq_io_cq *bic, struct bfq_group *bfqg) { - struct bfq_queue *async_bfqq = bic_to_bfqq(bic, 0); - struct bfq_queue *sync_bfqq = bic_to_bfqq(bic, 1); - struct bfq_entity *entity; + unsigned int act_idx; - if (async_bfqq) { - entity = &async_bfqq->entity; + for (act_idx = 0; act_idx < bfqd->num_actuators; act_idx++) { + struct bfq_queue *async_bfqq = bic_to_bfqq(bic, false, act_idx); + struct bfq_queue *sync_bfqq = bic_to_bfqq(bic, true, act_idx); - if (entity->sched_data != &bfqg->sched_data) { - bic_set_bfqq(bic, NULL, 0); + if (async_bfqq && + async_bfqq->entity.sched_data != &bfqg->sched_data) { + bic_set_bfqq(bic, NULL, false, act_idx); bfq_release_process_ref(bfqd, async_bfqq); } - } - if (sync_bfqq) { - if (!sync_bfqq->new_bfqq && !bfq_bfqq_coop(sync_bfqq)) { - /* We are the only user of this bfqq, just move it */ - if (sync_bfqq->entity.sched_data != &bfqg->sched_data) - bfq_bfqq_move(bfqd, sync_bfqq, bfqg); - } else { - struct bfq_queue *bfqq; - - /* - * The queue was merged to a different queue. Check - * that the merge chain still belongs to the same - * cgroup. - */ - for (bfqq = sync_bfqq; bfqq; bfqq = bfqq->new_bfqq) - if (bfqq->entity.sched_data != - &bfqg->sched_data) - break; - if (bfqq) { - /* - * Some queue changed cgroup so the merge is - * not valid anymore. We cannot easily just - * cancel the merge (by clearing new_bfqq) as - * there may be other processes using this - * queue and holding refs to all queues below - * sync_bfqq->new_bfqq. Similarly if the merge - * already happened, we need to detach from - * bfqq now so that we cannot merge bio to a - * request from the old cgroup. - */ - bfq_put_cooperator(sync_bfqq); - bfq_release_process_ref(bfqd, sync_bfqq); - bic_set_bfqq(bic, NULL, 1); - } - } + if (sync_bfqq) + bfq_sync_bfqq_move(bfqd, sync_bfqq, bic, bfqg, act_idx); } return bfqg; diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 7ea427817f7f..16dbf52ce74b 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -377,14 +377,21 @@ static const unsigned long bfq_late_stable_merging = 600; #define RQ_BIC(rq) ((struct bfq_io_cq *)((rq)->elv.priv[0])) #define RQ_BFQQ(rq) ((rq)->elv.priv[1]) -struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync) +struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync, + unsigned int actuator_idx) { - return bic->bfqq[is_sync]; + if (is_sync) + return bic->bfqq[1][actuator_idx]; + + return bic->bfqq[0][actuator_idx]; } static void bfq_put_stable_ref(struct bfq_queue *bfqq); -void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync) +void bic_set_bfqq(struct bfq_io_cq *bic, + struct bfq_queue *bfqq, + bool is_sync, + unsigned int actuator_idx) { /* * If bfqq != NULL, then a non-stable queue merge between @@ -399,7 +406,10 @@ void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync) * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ - bic->bfqq[is_sync] = bfqq; + if (is_sync) + bic->bfqq[1][actuator_idx] = bfqq; + else + bic->bfqq[0][actuator_idx] = bfqq; if (bfqq && bic->stable_merge_bfqq == bfqq) { /* @@ -672,9 +682,9 @@ static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data) { struct bfq_data *bfqd = data->q->elevator->elevator_data; struct bfq_io_cq *bic = bfq_bic_lookup(data->q); - struct bfq_queue *bfqq = bic ? bic_to_bfqq(bic, op_is_sync(opf)) : NULL; int depth; unsigned limit = data->q->nr_requests; + unsigned int act_idx; /* Sync reads have full depth available */ if (op_is_sync(opf) && !op_is_write(opf)) { @@ -684,14 +694,21 @@ static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data) limit = (limit * depth) >> bfqd->full_depth_shift; } - /* - * Does queue (or any parent entity) exceed number of requests that - * should be available to it? Heavily limit depth so that it cannot - * consume more available requests and thus starve other entities. - */ - if (bfqq && bfqq_request_over_limit(bfqq, limit)) - depth = 1; + for (act_idx = 0; act_idx < bfqd->num_actuators; act_idx++) { + struct bfq_queue *bfqq = + bic ? bic_to_bfqq(bic, op_is_sync(opf), act_idx) : NULL; + /* + * Does queue (or any parent entity) exceed number of + * requests that should be available to it? Heavily + * limit depth so that it cannot consume more + * available requests and thus starve other entities. + */ + if (bfqq && bfqq_request_over_limit(bfqq, limit)) { + depth = 1; + break; + } + } bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u", __func__, bfqd->wr_busy_queues, op_is_sync(opf), depth); if (depth) @@ -1812,6 +1829,18 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, return bfqq_weight > in_serv_weight; } +/* + * Get the index of the actuator that will serve bio. + */ +static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) +{ + /* + * Multi-actuator support not complete yet, so always return 0 + * for the moment (to keep incomplete mechanisms off). + */ + return 0; +} + static bool bfq_better_to_idle(struct bfq_queue *bfqq); static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, @@ -2142,7 +2171,7 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, * We reset waker detection logic also if too much time has passed * since the first detection. If wakeups are rare, pointless idling * doesn't hurt throughput that much. The condition below makes sure - * we do not uselessly idle blocking waker in more than 1/64 cases. + * we do not uselessly idle blocking waker in more than 1/64 cases. */ if (bfqd->last_completed_rq_bfqq != bfqq->tentative_waker_bfqq || @@ -2478,7 +2507,8 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, */ bfq_bic_update_cgroup(bic, bio); - bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf)); + bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf), + bfq_actuator_index(bfqd, bio)); } else { bfqd->bio_bfqq = NULL; } @@ -3174,7 +3204,7 @@ bfq_merge_bfqqs(struct bfq_data *bfqd, struct bfq_io_cq *bic, /* * Merge queues (that is, let bic redirect its requests to new_bfqq) */ - bic_set_bfqq(bic, new_bfqq, 1); + bic_set_bfqq(bic, new_bfqq, true, bfqq->actuator_idx); bfq_mark_bfqq_coop(new_bfqq); /* * new_bfqq now belongs to at least two bics (it is a shared queue): @@ -4808,11 +4838,8 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) */ if (bfq_bfqq_wait_request(bfqq) || (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) { - struct bfq_queue *async_bfqq = - bfqq->bic && bfqq->bic->bfqq[0] && - bfq_bfqq_busy(bfqq->bic->bfqq[0]) && - bfqq->bic->bfqq[0]->next_rq ? - bfqq->bic->bfqq[0] : NULL; + unsigned int act_idx = bfqq->actuator_idx; + struct bfq_queue *async_bfqq = NULL; struct bfq_queue *blocked_bfqq = !hlist_empty(&bfqq->woken_list) ? container_of(bfqq->woken_list.first, @@ -4820,6 +4847,10 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) woken_list_node) : NULL; + if (bfqq->bic && bfqq->bic->bfqq[0][act_idx] && + bfq_bfqq_busy(bfqq->bic->bfqq[0][act_idx]) && + bfqq->bic->bfqq[0][act_idx]->next_rq) + async_bfqq = bfqq->bic->bfqq[0][act_idx]; /* * The next four mutually-exclusive ifs decide * whether to try injection, and choose the queue to @@ -4904,7 +4935,7 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic && bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= bfq_bfqq_budget_left(async_bfqq)) - bfqq = bfqq->bic->bfqq[0]; + bfqq = bfqq->bic->bfqq[0][act_idx]; else if (bfqq->waker_bfqq && bfq_bfqq_busy(bfqq->waker_bfqq) && bfqq->waker_bfqq->next_rq && @@ -5365,49 +5396,55 @@ static void bfq_exit_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq) bfq_release_process_ref(bfqd, bfqq); } -static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync) +static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync, + unsigned int actuator_idx) { - struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync); + struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, actuator_idx); struct bfq_data *bfqd; if (bfqq) bfqd = bfqq->bfqd; /* NULL if scheduler already exited */ if (bfqq && bfqd) { - unsigned long flags; - - spin_lock_irqsave(&bfqd->lock, flags); bfqq->bic = NULL; bfq_exit_bfqq(bfqd, bfqq); - bic_set_bfqq(bic, NULL, is_sync); - spin_unlock_irqrestore(&bfqd->lock, flags); + bic_set_bfqq(bic, NULL, is_sync, actuator_idx); } } static void bfq_exit_icq(struct io_cq *icq) { struct bfq_io_cq *bic = icq_to_bic(icq); + struct bfq_data *bfqd = bic_to_bfqd(bic); + unsigned long flags; + unsigned int act_idx; + /* + * If bfqd and thus bfqd->num_actuators is not available any + * longer, then cycle over all possible per-actuator bfqqs in + * next loop. We rely on bic being zeroed on creation, and + * therefore on its unused per-actuator fields being NULL. + */ + unsigned int num_actuators = BFQ_MAX_ACTUATORS; - if (bic->stable_merge_bfqq) { - struct bfq_data *bfqd = bic->stable_merge_bfqq->bfqd; + /* + * bfqd is NULL if scheduler already exited, and in that case + * this is the last time these queues are accessed. + */ + if (bfqd) { + spin_lock_irqsave(&bfqd->lock, flags); + num_actuators = bfqd->num_actuators; + } - /* - * bfqd is NULL if scheduler already exited, and in - * that case this is the last time bfqq is accessed. - */ - if (bfqd) { - unsigned long flags; + if (bic->stable_merge_bfqq) + bfq_put_stable_ref(bic->stable_merge_bfqq); - spin_lock_irqsave(&bfqd->lock, flags); - bfq_put_stable_ref(bic->stable_merge_bfqq); - spin_unlock_irqrestore(&bfqd->lock, flags); - } else { - bfq_put_stable_ref(bic->stable_merge_bfqq); - } + for (act_idx = 0; act_idx < num_actuators; act_idx++) { + bfq_exit_icq_bfqq(bic, true, act_idx); + bfq_exit_icq_bfqq(bic, false, act_idx); } - bfq_exit_icq_bfqq(bic, true); - bfq_exit_icq_bfqq(bic, false); + if (bfqd) + spin_unlock_irqrestore(&bfqd->lock, flags); } /* @@ -5484,23 +5521,25 @@ static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio) bic->ioprio = ioprio; - bfqq = bic_to_bfqq(bic, false); + bfqq = bic_to_bfqq(bic, false, bfq_actuator_index(bfqd, bio)); if (bfqq) { bfq_release_process_ref(bfqd, bfqq); bfqq = bfq_get_queue(bfqd, bio, false, bic, true); - bic_set_bfqq(bic, bfqq, false); + bic_set_bfqq(bic, bfqq, false, bfq_actuator_index(bfqd, bio)); } - bfqq = bic_to_bfqq(bic, true); + bfqq = bic_to_bfqq(bic, true, bfq_actuator_index(bfqd, bio)); if (bfqq) bfq_set_next_ioprio_data(bfqq, bic); } static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct bfq_io_cq *bic, pid_t pid, int is_sync) + struct bfq_io_cq *bic, pid_t pid, int is_sync, + unsigned int act_idx) { u64 now_ns = ktime_get_ns(); + bfqq->actuator_idx = act_idx; RB_CLEAR_NODE(&bfqq->entity.rb_node); INIT_LIST_HEAD(&bfqq->fifo); INIT_HLIST_NODE(&bfqq->burst_list_node); @@ -5753,7 +5792,7 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, if (bfqq) { bfq_init_bfqq(bfqd, bfqq, bic, current->pid, - is_sync); + is_sync, bfq_actuator_index(bfqd, bio)); bfq_init_entity(&bfqq->entity, bfqg); bfq_log_bfqq(bfqd, bfqq, "allocated"); } else { @@ -6068,7 +6107,8 @@ static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) * then complete the merge and redirect it to * new_bfqq. */ - if (bic_to_bfqq(RQ_BIC(rq), 1) == bfqq) + if (bic_to_bfqq(RQ_BIC(rq), true, + bfq_actuator_index(bfqd, rq->bio)) == bfqq) bfq_merge_bfqqs(bfqd, RQ_BIC(rq), bfqq, new_bfqq); @@ -6622,7 +6662,7 @@ bfq_split_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq) return bfqq; } - bic_set_bfqq(bic, NULL, 1); + bic_set_bfqq(bic, NULL, true, bfqq->actuator_idx); bfq_put_cooperator(bfqq); @@ -6636,7 +6676,8 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bool split, bool is_sync, bool *new_queue) { - struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync); + unsigned int act_idx = bfq_actuator_index(bfqd, bio); + struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6648,7 +6689,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bfq_put_queue(bfqq); bfqq = bfq_get_queue(bfqd, bio, is_sync, bic, split); - bic_set_bfqq(bic, bfqq, is_sync); + bic_set_bfqq(bic, bfqq, is_sync, act_idx); if (split && is_sync) { if ((bic->was_in_burst_list && bfqd->large_burst) || bic->saved_in_large_burst) @@ -7090,8 +7131,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. * Grab a permanent reference to it, so that the normal code flow * will not attempt to free it. + * Set zero as actuator index: we will pretend that + * all I/O requests are for the same actuator. */ - bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0); + bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0, 0); bfqd->oom_bfqq.ref++; bfqd->oom_bfqq.new_ioprio = BFQ_DEFAULT_QUEUE_IOPRIO; bfqd->oom_bfqq.new_ioprio_class = IOPRIO_CLASS_BE; @@ -7110,6 +7153,13 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue = q; + /* + * Multi-actuator support not complete yet, unconditionally + * set to only one actuator for the moment (to keep incomplete + * mechanisms off). + */ + bfqd->num_actuators = 1; + INIT_LIST_HEAD(&bfqd->dispatch); hrtimer_init(&bfqd->idle_slice_timer, CLOCK_MONOTONIC, diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 71f721670ab6..2b413ddffbb9 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -33,6 +33,14 @@ */ #define BFQ_SOFTRT_WEIGHT_FACTOR 100 +/* + * Maximum number of actuators supported. This constant is used simply + * to define the size of the static array that will contain + * per-actuator data. The current value is hopefully a good upper + * bound to the possible number of actuators of any actual drive. + */ +#define BFQ_MAX_ACTUATORS 8 + struct bfq_entity; /** @@ -225,12 +233,14 @@ struct bfq_ttime { * struct bfq_queue - leaf schedulable entity. * * A bfq_queue is a leaf request queue; it can be associated with an - * io_context or more, if it is async or shared between cooperating - * processes. @cgroup holds a reference to the cgroup, to be sure that it - * does not disappear while a bfqq still references it (mostly to avoid - * races between request issuing and task migration followed by cgroup - * destruction). - * All the fields are protected by the queue lock of the containing bfqd. + * io_context or more, if it is async or shared between cooperating + * processes. Besides, it contains I/O requests for only one actuator + * (an io_context is associated with a different bfq_queue for each + * actuator it generates I/O for). @cgroup holds a reference to the + * cgroup, to be sure that it does not disappear while a bfqq still + * references it (mostly to avoid races between request issuing and + * task migration followed by cgroup destruction). All the fields are + * protected by the queue lock of the containing bfqd. */ struct bfq_queue { /* reference counter */ @@ -395,6 +405,9 @@ struct bfq_queue { * the woken queues when this queue exits. */ struct hlist_head woken_list; + + /* index of the actuator this queue is associated with */ + unsigned int actuator_idx; }; /** @@ -403,8 +416,17 @@ struct bfq_queue { struct bfq_io_cq { /* associated io_cq structure */ struct io_cq icq; /* must be the first member */ - /* array of two process queues, the sync and the async */ - struct bfq_queue *bfqq[2]; + /* + * Matrix of associated process queues: first row for async + * queues, second row sync queues. Each row contains one + * column for each actuator. An I/O request generated by the + * process is inserted into the queue pointed by bfqq[i][j] if + * the request is to be served by the j-th actuator of the + * drive, where i==0 or i==1, depending on whether the request + * is async or sync. So there is a distinct queue for each + * actuator. + */ + struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; /* per (request_queue, blkcg) ioprio */ int ioprio; #ifdef CONFIG_BFQ_GROUP_IOSCHED @@ -768,6 +790,13 @@ struct bfq_data { */ unsigned int word_depths[2][2]; unsigned int full_depth_shift; + + /* + * Number of independent actuators. This is equal to 1 in + * case of single-actuator drives. + */ + unsigned int num_actuators; + }; enum bfqq_state_flags { @@ -964,8 +993,10 @@ struct bfq_group { extern const int bfq_timeout; -struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync); -void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync); +struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync, + unsigned int actuator_idx); +void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync, + unsigned int actuator_idx); struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic); void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq); void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, From patchwork Thu Dec 8 10:43:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31280 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp122440wrr; Thu, 8 Dec 2022 02:50:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Ow6b6mnLG2+KxYi4ANPEJf0bDHk2NkODBcUzJ9H8p7DXQO8srO3wnNFnkUFIgDLj6sfsV X-Received: by 2002:a05:6a00:4210:b0:56c:b8c2:ee89 with SMTP id cd16-20020a056a00421000b0056cb8c2ee89mr76008251pfb.21.1670496618785; Thu, 08 Dec 2022 02:50:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670496618; cv=none; d=google.com; s=arc-20160816; b=eNosjA9D1XsbYI4kAf7xPGgv5ssAuDkRKR79aGTluaL6grrlzgSMCIPtFlnT462mxv 0S0GnTdG276VqIYcMWpIOzTp38NwDHzOd+AGMiSXYu8GHaFUpEckzk6v0Xr90yIMlNzM fInLv/mfUcFFghuta01B1A/96Nf7JuaHj+VYxI0wctU+JGq/O2QIhYjyCnnmShXzhLZc ++2NPYQKWvvoMPEbZcP5AOrEz+UKRCK2Gp06yzac+A+Hq5oU4ocglqpk/FeWvGLCQwwK DJ/yOt+kAO5aqhTN3C3R27zkmiyMa7usgs9e4fFhu6nhHippbzy5VpethzIySeDSgvw0 gDeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=H138aqobhHjmcHH6RFqFD59W1hrmC7/KCJez2I0jafo=; b=QqsfwfBjH+R3W8PNMHl6QsaImhv6P7+9o/wx0KXFdxY9URDBYQgbH+6ML3vVTRqfzK cYO7oIlEf0SlAVyomNk/5mHNy8nECzr82XEzWwzbGFWCpqCFWG87+b2H4D0c5tgGxfV2 iVLiBpB62Amec/N+LZ9y+ZDTgr7vXHHr4QIk3Sy1jCngwrm1J6sfUEhCLRVK6b1ZMAAr egmxK8uEXPGAGccpiO6ZLQaEqheDhrzVRlLFmkQn9ZFANf/SbkorGtZcu9+6kSrZU5+o s3bai4cXQ0Yoxi795Ar/+/ThA8nNqBWh8W1uwyCblDCMfiK/uftqpPgPqQQtpiKM/Pct Ym3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CzlENSXI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h3-20020a170902f54300b001898c6b8ec4si10718492plf.317.2022.12.08.02.50.05; Thu, 08 Dec 2022 02:50:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CzlENSXI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229746AbiLHKto (ORCPT + 99 others); Thu, 8 Dec 2022 05:49:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230139AbiLHKsq (ORCPT ); Thu, 8 Dec 2022 05:48:46 -0500 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95C8D5BD7C for ; Thu, 8 Dec 2022 02:44:01 -0800 (PST) Received: by mail-ej1-x62a.google.com with SMTP id vp12so2861032ejc.8 for ; Thu, 08 Dec 2022 02:44:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H138aqobhHjmcHH6RFqFD59W1hrmC7/KCJez2I0jafo=; b=CzlENSXIlHjwUOyd67Oro8J9zGfNfaRzYFG93Ulb9hv/6zS6G+b9XaV+dCu+ZWNhED +HU0nltoTBvmExw2O29qJ+TsptddL3PAmYZQ62XqZCQ7QUhqcDChUUEvOTiA9h34wooq sUqN1lZtAMUiBz0QW9pgWcObvCpJNtHdQf9A1Wb0QOaEvZFn1AsAn3NoFOUGvuh5hfJq uEZNaaLFLSoeOZMIvXYLgLrACvltaevVvTBp814KWYSsJj9RXoyQQcXqPCZKN3RXRHwT pnjBA9M8a7Zt0PLWb49IGqdZ/TJO+pq8S5tdzZ7AO9cS9Hw2JB4ERmAcrNW33XNk1oHD cGfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H138aqobhHjmcHH6RFqFD59W1hrmC7/KCJez2I0jafo=; b=O9tw0I751yxdT69u+z7D3XIJvf5hEq7XjSvEUwmawOT/SbTO6ZcwBm+3K6asX46NV+ N/VUMmLJXK5eQE3woQg7NifN9ehEGDWw6e2Nlx8zlsiFs5/94+WI7O64dL1QpxyeFP5g OT5Ol5UYCNZC9vR/6MOrt8WWxPShsz0uP3qBqHuyFjp+GIuZP03uscQW2hPF5Cihk7QF jW0ttd1jQn1ZMKOqmdGOS5Seg3L4bqRRI668ZYCFW7nE/2tqzHpnTyqctydl9Rw8GchM mNHfUNEIPPEvrkbPv700jTTGphhZXxeMf5ewaycVq/+Ii4gtXKMefvh4UU+2H7mS7N3k SwXQ== X-Gm-Message-State: ANoB5pl9ypjAINsOw7HAe/izKVIFChm2Smj5edHUaR3AP4HYSj7yTe1z 6K0pVJX1EixsOkhYCx9nUMANMg== X-Received: by 2002:a17:906:3a85:b0:7ab:afd4:d7ed with SMTP id y5-20020a1709063a8500b007abafd4d7edmr11333715ejd.228.1670496240066; Thu, 08 Dec 2022 02:44:00 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.43.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:43:59 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Paolo Valente , Damien Le Moal Subject: [PATCH V9 2/8] block, bfq: forbid stable merging of queues associated with different actuators Date: Thu, 8 Dec 2022 11:43:45 +0100 Message-Id: <20221208104351.35038-3-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751642662485017462?= X-GMAIL-MSGID: =?utf-8?q?1751642662485017462?= If queues associated with different actuators are merged, then control is lost on each actuator. Therefore some actuator may be underutilized, and throughput may decrease. This problem cannot occur with basic queue merging, because the latter is triggered by spatial locality, and sectors for different actuators are not close to each other. Yet it may happen with stable merging. To address this issue, this commit prevents stable merging from occurring among queues associated with different actuators. Reviewed-by: Damien Le Moal Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 16dbf52ce74b..456b5f5c24fd 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -5708,9 +5708,13 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, * it has been set already, but too long ago, then move it * forward to bfqq. Finally, move also if bfqq belongs to a * different group than last_bfqq_created, or if bfqq has a - * different ioprio or ioprio_class. If none of these - * conditions holds true, then try an early stable merge or - * schedule a delayed stable merge. + * different ioprio, ioprio_class or actuator_idx. If none of + * these conditions holds true, then try an early stable merge + * or schedule a delayed stable merge. As for the condition on + * actuator_idx, the reason is that, if queues associated with + * different actuators are merged, then control is lost on + * each actuator. Therefore some actuator may be + * underutilized, and throughput may decrease. * * A delayed merge is scheduled (instead of performing an * early merge), in case bfqq might soon prove to be more @@ -5728,7 +5732,8 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, bfqq->creation_time) || bfqq->entity.parent != last_bfqq_created->entity.parent || bfqq->ioprio != last_bfqq_created->ioprio || - bfqq->ioprio_class != last_bfqq_created->ioprio_class) + bfqq->ioprio_class != last_bfqq_created->ioprio_class || + bfqq->actuator_idx != last_bfqq_created->actuator_idx) *source_bfqq = bfqq; else if (time_after_eq(last_bfqq_created->creation_time + bfqd->bfq_burst_interval, From patchwork Thu Dec 8 10:43:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31281 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp123609wrr; Thu, 8 Dec 2022 02:54:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf6Fjb7QHmE4vcmVUKZXrmik0/2HFYEzfmWuwmYwnQBnh30Do2Grw9pKxCHRd3KPljuNsnSd X-Received: by 2002:a05:6a00:2444:b0:558:991a:6671 with SMTP id d4-20020a056a00244400b00558991a6671mr78155846pfj.58.1670496848636; Thu, 08 Dec 2022 02:54:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670496848; cv=none; d=google.com; s=arc-20160816; b=lvEcNbqpSLYbxcq8R0oDigQkuMlbsga3ruLJMhPWJ4V8NP80CZ3lvjrCGNboG/0zc8 lcMol8xJ6HU9szHy0xsCoT00Cvbtma2lCfLe70o3PMUszJu0sEBTH8aG6YTIjsuq5JvE EkC9ByL7Ka/dwf218eOabFsYaJj4dCcnPLnd4lBoQJeJW2noxn/VG/JmqjWVEZG89lXm gJ2JqtvJkIYiMvDwbqNrVZ8Fug5YGqVfBuiec6WCAPrDTQVTOZz7VMEFr8q7rHInjxQF E101glk23QhDP2jPrj1hSnigtoiYQWwA6y6NTrltN3fIh0cncA4BkVGpl36Z0VqEDK5l MQXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XTwJnYwXqmbEqiuqtSDQykovTGEftLU9E09zUz3ipHY=; b=KsFp+NyLi9UY9vczgMOU9d1NfittFFhCX1qWXqDnEekphPlaohcz11JJ28LbF+z1Xt dLE5EegcA3xvZKMg4uKyisiqBBlcyAZRRCpy6wI34MsYtIzT7/Kp1c9jKt1ckySWZlwN stiNkSAQFzl6IdTjnBOkj7OfKkT8R/W2ecCnhcc03w8D7N+vjZPP+nKQ7Phw928lbNpg yPqOHxszIIw6P7pdrG2h7rxiNcbf66UqpW8z/RdFJJw/J6R6XugVAQUOPF4kizQH9OST HTWMFkcBrrsRFW8YL+OwxjulyVqXVx9fYAmOVzRp4YQcUgUnmluI8bwRwKOfIdndKQzE I6kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="O0C6/fNW"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h1-20020a654681000000b0046f57b135c9si22877221pgr.182.2022.12.08.02.53.52; Thu, 08 Dec 2022 02:54:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="O0C6/fNW"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230335AbiLHKuI (ORCPT + 99 others); Thu, 8 Dec 2022 05:50:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229757AbiLHKsz (ORCPT ); Thu, 8 Dec 2022 05:48:55 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9445E26AE8 for ; Thu, 8 Dec 2022 02:44:02 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id r26so1501634edc.10 for ; Thu, 08 Dec 2022 02:44:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XTwJnYwXqmbEqiuqtSDQykovTGEftLU9E09zUz3ipHY=; b=O0C6/fNWHXuwVueMIxC9QMUOmkZZsvbY39LNXGFIFUDOphVPAKwirLAjiGsIylgpny nICDzhjtYaDNOlEqscG/pKSD/nxS4clfol/NqnnahSMd4WmZnwA8nhulJTnea7yXxlGP VFSDZIpRCGwlWdfZ2QfOkQjIerJibqHb4RUOUDssjhMibFlCaYkmG94ikHTEt+40+AZM GaYAmTRYJZbtEMkySl2WqEWQ0HB4gVNDOvofkzQQXDhGse10DGrlLlhuGyA1SdHejxzL oDlZa53vBW1H9No6kw5yuTHURXDvlDwyuMvzr8XVlOBWt14D+q27dIpVnHy6aq4w2b4J pqtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XTwJnYwXqmbEqiuqtSDQykovTGEftLU9E09zUz3ipHY=; b=zWOB8sIArp5r+qXp76kUhXBBf75hJBbHcc0Hh6wIKzd3d4iL/LNL8+BHBodCcTqAk2 U0ccfW5YbXmJl//J4/HWkGXe5j1sA6R9g8B8hTllryJ1yfggjnynv7hcN9kMrACmgX+s DEQoiegdJl4szRMxNjwdDRgRALlDY3fmKmzfw8oFJWZYGWjUNaoSV2XIbbzGn/LbWLLW 42i6qTmchaTdynJCFL0BIKHAYwYlnHiarZmZfCLGX6nDW14mDOGdDlpIubEmvBcWjtHa Y3aL2zeitayl6ExIJaZdsjWILZPVMYEBKhH6tXtj7TU+7200/2GppOr2iO8oXJlTwRoz 16uw== X-Gm-Message-State: ANoB5plkIQ34un+DFRT31uaLWbHWWzXfIfg1bMBFzkMpXqPs/HJAPLuI Qed2ZurRAxMULLlAZr7Svtfn3w== X-Received: by 2002:a05:6402:e04:b0:469:e6ef:9164 with SMTP id h4-20020a0564020e0400b00469e6ef9164mr35727484edh.185.1670496241029; Thu, 08 Dec 2022 02:44:01 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:00 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Paolo Valente , Damien Le Moal , Gianmarco Lusvardi , Giulio Barabino , Emiliano Maccaferri Subject: [PATCH V9 3/8] block, bfq: move io_cq-persistent bfqq data into a dedicated struct Date: Thu, 8 Dec 2022 11:43:46 +0100 Message-Id: <20221208104351.35038-4-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751642903692305087?= X-GMAIL-MSGID: =?utf-8?q?1751642903692305087?= With a multi-actuator drive, a process may get associated with multiple bfq_queues: one queue for each of the N actuators. So, the bfq_io_cq data structure must be able to accommodate its per-queue persistent information for N queues. Currently it stores this information for just one queue, in several scalar fields. This is a preparatory commit for moving to accommodating persistent information for N queues. In particular, this commit packs all the above scalar fields into a single data structure. Then there is now only one field, in bfq_io_cq, that stores all the above information. This scalar field will then be turned into an array by a following commit. Suggested-by: Damien Le Moal Reviewed-by: Damien Le Moal Signed-off-by: Gianmarco Lusvardi Signed-off-by: Giulio Barabino Signed-off-by: Emiliano Maccaferri Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 130 +++++++++++++++++++++++++------------------- block/bfq-iosched.h | 52 +++++++++++------- 2 files changed, 106 insertions(+), 76 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 456b5f5c24fd..0d6b35ef3d3f 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -406,12 +406,14 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + if (is_sync) bic->bfqq[1][actuator_idx] = bfqq; else bic->bfqq[0][actuator_idx] = bfqq; - if (bfqq && bic->stable_merge_bfqq == bfqq) { + if (bfqq && bfqq_data->stable_merge_bfqq == bfqq) { /* * Actually, these same instructions are executed also * in bfq_setup_cooperator, in case of abort or actual @@ -420,9 +422,9 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * did so, we would nest even more complexity in this * function. */ - bfq_put_stable_ref(bic->stable_merge_bfqq); + bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); - bic->stable_merge_bfqq = NULL; + bfqq_data->stable_merge_bfqq = NULL; } } @@ -1179,38 +1181,40 @@ static void bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, struct bfq_io_cq *bic, bool bfq_already_existing) { + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; unsigned int old_wr_coeff = 1; bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); - if (bic->saved_has_short_ttime) + if (bfqq_data->saved_has_short_ttime) bfq_mark_bfqq_has_short_ttime(bfqq); else bfq_clear_bfqq_has_short_ttime(bfqq); - if (bic->saved_IO_bound) + if (bfqq_data->saved_IO_bound) bfq_mark_bfqq_IO_bound(bfqq); else bfq_clear_bfqq_IO_bound(bfqq); - bfqq->last_serv_time_ns = bic->saved_last_serv_time_ns; - bfqq->inject_limit = bic->saved_inject_limit; - bfqq->decrease_time_jif = bic->saved_decrease_time_jif; + bfqq->last_serv_time_ns = bfqq_data->saved_last_serv_time_ns; + bfqq->inject_limit = bfqq_data->saved_inject_limit; + bfqq->decrease_time_jif = bfqq_data->saved_decrease_time_jif; - bfqq->entity.new_weight = bic->saved_weight; - bfqq->ttime = bic->saved_ttime; - bfqq->io_start_time = bic->saved_io_start_time; - bfqq->tot_idle_time = bic->saved_tot_idle_time; + bfqq->entity.new_weight = bfqq_data->saved_weight; + bfqq->ttime = bfqq_data->saved_ttime; + bfqq->io_start_time = bfqq_data->saved_io_start_time; + bfqq->tot_idle_time = bfqq_data->saved_tot_idle_time; /* * Restore weight coefficient only if low_latency is on */ if (bfqd->low_latency) { old_wr_coeff = bfqq->wr_coeff; - bfqq->wr_coeff = bic->saved_wr_coeff; + bfqq->wr_coeff = bfqq_data->saved_wr_coeff; } - bfqq->service_from_wr = bic->saved_service_from_wr; - bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; - bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish; - bfqq->wr_cur_max_time = bic->saved_wr_cur_max_time; + bfqq->service_from_wr = bfqq_data->saved_service_from_wr; + bfqq->wr_start_at_switch_to_srt = + bfqq_data->saved_wr_start_at_switch_to_srt; + bfqq->last_wr_start_finish = bfqq_data->saved_last_wr_start_finish; + bfqq->wr_cur_max_time = bfqq_data->saved_wr_cur_max_time; if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) || time_is_before_jiffies(bfqq->last_wr_start_finish + @@ -1895,7 +1899,7 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || (bfq_bfqq_sync(bfqq) && - (bfqq->bic || RQ_BIC(rq)->stably_merged) && + (bfqq->bic || RQ_BIC(rq)->bfqq_data.stably_merged) && (*interactive || soft_rt))); /* @@ -2909,6 +2913,7 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, void *io_struct, bool request, struct bfq_io_cq *bic) { struct bfq_queue *in_service_bfqq, *new_bfqq; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* if a merge has already been setup, then proceed with that first */ if (bfqq->new_bfqq) @@ -2930,21 +2935,21 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, * stable merging) also if bic is associated with a * sync queue, but this bfqq is async */ - if (bfq_bfqq_sync(bfqq) && bic->stable_merge_bfqq && + if (bfq_bfqq_sync(bfqq) && bfqq_data->stable_merge_bfqq && !bfq_bfqq_just_created(bfqq) && time_is_before_jiffies(bfqq->split_time + msecs_to_jiffies(bfq_late_stable_merging)) && time_is_before_jiffies(bfqq->creation_time + msecs_to_jiffies(bfq_late_stable_merging))) { struct bfq_queue *stable_merge_bfqq = - bic->stable_merge_bfqq; + bfqq_data->stable_merge_bfqq; int proc_ref = min(bfqq_process_refs(bfqq), bfqq_process_refs(stable_merge_bfqq)); /* deschedule stable merge, because done or aborted here */ bfq_put_stable_ref(stable_merge_bfqq); - bic->stable_merge_bfqq = NULL; + bfqq_data->stable_merge_bfqq = NULL; if (!idling_boosts_thr_without_issues(bfqd, bfqq) && proc_ref > 0) { @@ -2953,10 +2958,10 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_setup_merge(bfqq, stable_merge_bfqq); if (new_bfqq) { - bic->stably_merged = true; + bfqq_data->stably_merged = true; if (new_bfqq->bic) - new_bfqq->bic->stably_merged = - true; + new_bfqq->bic->bfqq_data.stably_merged = + true; } return new_bfqq; } else @@ -3055,6 +3060,7 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, static void bfq_bfqq_save_state(struct bfq_queue *bfqq) { struct bfq_io_cq *bic = bfqq->bic; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* * If !bfqq->bic, the queue is already shared or its requests @@ -3064,18 +3070,21 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) if (!bic) return; - bic->saved_last_serv_time_ns = bfqq->last_serv_time_ns; - bic->saved_inject_limit = bfqq->inject_limit; - bic->saved_decrease_time_jif = bfqq->decrease_time_jif; - - bic->saved_weight = bfqq->entity.orig_weight; - bic->saved_ttime = bfqq->ttime; - bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); - bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); - bic->saved_io_start_time = bfqq->io_start_time; - bic->saved_tot_idle_time = bfqq->tot_idle_time; - bic->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); - bic->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node); + bfqq_data->saved_last_serv_time_ns = bfqq->last_serv_time_ns; + bfqq_data->saved_inject_limit = bfqq->inject_limit; + bfqq_data->saved_decrease_time_jif = bfqq->decrease_time_jif; + + bfqq_data->saved_weight = bfqq->entity.orig_weight; + bfqq_data->saved_ttime = bfqq->ttime; + bfqq_data->saved_has_short_ttime = + bfq_bfqq_has_short_ttime(bfqq); + bfqq_data->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); + bfqq_data->saved_io_start_time = bfqq->io_start_time; + bfqq_data->saved_tot_idle_time = bfqq->tot_idle_time; + bfqq_data->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); + bfqq_data->was_in_burst_list = + !hlist_unhashed(&bfqq->burst_list_node); + if (unlikely(bfq_bfqq_just_created(bfqq) && !bfq_bfqq_in_large_burst(bfqq) && bfqq->bfqd->low_latency)) { @@ -3088,17 +3097,21 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) * to bfqq, so that to avoid that bfqq unjustly fails * to enjoy weight raising if split soon. */ - bic->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; - bic->saved_wr_start_at_switch_to_srt = bfq_smallest_from_now(); - bic->saved_wr_cur_max_time = bfq_wr_duration(bfqq->bfqd); - bic->saved_last_wr_start_finish = jiffies; + bfqq_data->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; + bfqq_data->saved_wr_start_at_switch_to_srt = + bfq_smallest_from_now(); + bfqq_data->saved_wr_cur_max_time = + bfq_wr_duration(bfqq->bfqd); + bfqq_data->saved_last_wr_start_finish = jiffies; } else { - bic->saved_wr_coeff = bfqq->wr_coeff; - bic->saved_wr_start_at_switch_to_srt = + bfqq_data->saved_wr_coeff = bfqq->wr_coeff; + bfqq_data->saved_wr_start_at_switch_to_srt = bfqq->wr_start_at_switch_to_srt; - bic->saved_service_from_wr = bfqq->service_from_wr; - bic->saved_last_wr_start_finish = bfqq->last_wr_start_finish; - bic->saved_wr_cur_max_time = bfqq->wr_cur_max_time; + bfqq_data->saved_service_from_wr = + bfqq->service_from_wr; + bfqq_data->saved_last_wr_start_finish = + bfqq->last_wr_start_finish; + bfqq_data->saved_wr_cur_max_time = bfqq->wr_cur_max_time; } } @@ -5425,6 +5438,7 @@ static void bfq_exit_icq(struct io_cq *icq) * therefore on its unused per-actuator fields being NULL. */ unsigned int num_actuators = BFQ_MAX_ACTUATORS; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* * bfqd is NULL if scheduler already exited, and in that case @@ -5435,8 +5449,8 @@ static void bfq_exit_icq(struct io_cq *icq) num_actuators = bfqd->num_actuators; } - if (bic->stable_merge_bfqq) - bfq_put_stable_ref(bic->stable_merge_bfqq); + if (bfqq_data->stable_merge_bfqq) + bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); for (act_idx = 0; act_idx < num_actuators; act_idx++) { bfq_exit_icq_bfqq(bic, true, act_idx); @@ -5627,13 +5641,14 @@ bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, { struct bfq_queue *new_bfqq = bfq_setup_merge(bfqq, last_bfqq_created); + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; if (!new_bfqq) return bfqq; if (new_bfqq->bic) - new_bfqq->bic->stably_merged = true; - bic->stably_merged = true; + new_bfqq->bic->bfqq_data.stably_merged = true; + bfqq_data->stably_merged = true; /* * Reusing merge functions. This implies that @@ -5702,6 +5717,7 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, &bfqd->last_bfqq_created; struct bfq_queue *last_bfqq_created = *source_bfqq; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* * If last_bfqq_created has not been set yet, then init it. If @@ -5763,7 +5779,7 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, /* * Record the bfqq to merge to. */ - bic->stable_merge_bfqq = last_bfqq_created; + bfqq_data->stable_merge_bfqq = last_bfqq_created; } } @@ -6683,6 +6699,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, { unsigned int act_idx = bfq_actuator_index(bfqd, bio); struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6696,12 +6713,12 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, bic_set_bfqq(bic, bfqq, is_sync, act_idx); if (split && is_sync) { - if ((bic->was_in_burst_list && bfqd->large_burst) || - bic->saved_in_large_burst) + if ((bfqq_data->was_in_burst_list && bfqd->large_burst) || + bfqq_data->saved_in_large_burst) bfq_mark_bfqq_in_large_burst(bfqq); else { bfq_clear_bfqq_in_large_burst(bfqq); - if (bic->was_in_burst_list) + if (bfqq_data->was_in_burst_list) /* * If bfqq was in the current * burst list before being @@ -6790,6 +6807,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) struct bfq_queue *bfqq; bool new_queue = false; bool bfqq_already_existing = false, split = false; + struct bfq_iocq_bfqq_data *bfqq_data; if (unlikely(!rq->elv.icq)) return NULL; @@ -6813,15 +6831,17 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, &new_queue); + bfqq_data = &bic->bfqq_data; + if (likely(!new_queue)) { /* If the queue was seeky for too long, break it apart. */ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq) && - !bic->stably_merged) { + !bfqq_data->stably_merged) { struct bfq_queue *old_bfqq = bfqq; /* Update bic before losing reference to bfqq */ if (bfq_bfqq_in_large_burst(bfqq)) - bic->saved_in_large_burst = true; + bfqq_data->saved_in_large_burst = true; bfqq = bfq_split_bfqq(bic, bfqq); split = true; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 2b413ddffbb9..9d77d964620f 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -411,27 +411,9 @@ struct bfq_queue { }; /** - * struct bfq_io_cq - per (request_queue, io_context) structure. - */ -struct bfq_io_cq { - /* associated io_cq structure */ - struct io_cq icq; /* must be the first member */ - /* - * Matrix of associated process queues: first row for async - * queues, second row sync queues. Each row contains one - * column for each actuator. An I/O request generated by the - * process is inserted into the queue pointed by bfqq[i][j] if - * the request is to be served by the j-th actuator of the - * drive, where i==0 or i==1, depending on whether the request - * is async or sync. So there is a distinct queue for each - * actuator. - */ - struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; - /* per (request_queue, blkcg) ioprio */ - int ioprio; -#ifdef CONFIG_BFQ_GROUP_IOSCHED - uint64_t blkcg_serial_nr; /* the current blkcg serial */ -#endif +* struct bfq_data - bfqq data unique and persistent for associated bfq_io_cq +*/ +struct bfq_iocq_bfqq_data { /* * Snapshot of the has_short_time flag before merging; taken * to remember its value while the queue is merged, so as to @@ -486,6 +468,34 @@ struct bfq_io_cq { struct bfq_queue *stable_merge_bfqq; bool stably_merged; /* non splittable if true */ +}; + +/** + * struct bfq_io_cq - per (request_queue, io_context) structure. + */ +struct bfq_io_cq { + /* associated io_cq structure */ + struct io_cq icq; /* must be the first member */ + /* + * Matrix of associated process queues: first row for async + * queues, second row sync queues. Each row contains one + * column for each actuator. An I/O request generated by the + * process is inserted into the queue pointed by bfqq[i][j] if + * the request is to be served by the j-th actuator of the + * drive, where i==0 or i==1, depending on whether the request + * is async or sync. So there is a distinct queue for each + * actuator. + */ + struct bfq_queue *bfqq[2][BFQ_MAX_ACTUATORS]; + /* per (request_queue, blkcg) ioprio */ + int ioprio; +#ifdef CONFIG_BFQ_GROUP_IOSCHED + uint64_t blkcg_serial_nr; /* the current blkcg serial */ +#endif + + /* persistent data for associated synchronous process queue */ + struct bfq_iocq_bfqq_data bfqq_data; + unsigned int requests; /* Number of requests this process has in flight */ }; From patchwork Thu Dec 8 10:43:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp127077wrr; Thu, 8 Dec 2022 03:02:58 -0800 (PST) X-Google-Smtp-Source: AA0mqf5cR23XcWWm+6BayFe/Z+14bz77+aCEj90Ks3MiQzdlyaDOwttsqMTvKNBtUeaGQ5AFo3TC X-Received: by 2002:a17:906:fc0d:b0:7c0:d4c7:c00d with SMTP id ov13-20020a170906fc0d00b007c0d4c7c00dmr18199258ejb.345.1670497378440; Thu, 08 Dec 2022 03:02:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670497378; cv=none; d=google.com; s=arc-20160816; b=g2ks0AWQbnbHgDlVV1M4JGbesvQpypxA7EhmLx4PdBufoHcld3TXI0Blm80bj7PJd6 tiPLm0Nc951xX6pgmIEEUI60XUWTt5/II1iHtGSGKk/WuOT6Js6aDqCpjCc2oZAyAx5O 8XMKKAedXcn1mFTZbJ9otvk5DtgpFxhB0W5TVS/hvuVGWlkea/AFKV66AHEbAVdXeE43 IDiEog47vE3rZSe16xzvcoFB0jzlHJIiwXga0nuzeEIlxKQDl5wA4TZ//3J8X017jwjx Rianv5ZeONW45KFRCriHsEH7V0U75536qnVtXozbqqWU3Y491aOfgtRTTgow+dz4wasi /w6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8LXubQ3FrSF6IYeikSKGn7KB26hbqba/FTsLwOsytAM=; b=EaUsocq3BWJUPgVm/frZtnRAO9LGnVBsk+cImO1J5jlrQHv+HnzZ4vV6PbD2x/GBh5 QHpCqggnw+MZToKQm9isWhRbx5bP6+Q2dppCuZbIjrJKy23AnA63CFu+K5S49SkyVUhj ShrLfVxYOucJ6A0cjcXyBd5zh1G6adt2QtFAFAetVqY1ktzHd3wcaHJCN3VUwz+dtiJZ YVnZSyy0r/U1HeomPFVQSbhpQffUWvbBXzQhKszuXocyUpLxZ4zIIUa63n3ge7e65QVl L0l+Uu8pPNB/TK5hJCKJZIDp2s148dOmaQDKctvN89EpcRM42+ACClBpSzpzygx/AfL2 WYxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pwXUxZQd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sh12-20020a1709076e8c00b007b5ce4a436csi18694088ejc.227.2022.12.08.03.02.33; Thu, 08 Dec 2022 03:02:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pwXUxZQd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230324AbiLHKuB (ORCPT + 99 others); Thu, 8 Dec 2022 05:50:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229750AbiLHKsz (ORCPT ); Thu, 8 Dec 2022 05:48:55 -0500 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96BA38AADE for ; Thu, 8 Dec 2022 02:44:03 -0800 (PST) Received: by mail-ed1-x52f.google.com with SMTP id s5so1501547edc.12 for ; Thu, 08 Dec 2022 02:44:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8LXubQ3FrSF6IYeikSKGn7KB26hbqba/FTsLwOsytAM=; b=pwXUxZQdpNog61Jgd1cVHAQGDN53XeteCA/9vZ8DDz3qwy2Q9GBve9r8EQLXiAiwVa CggG1/R0/v9IWDlL8Ocucb7DwTfQpqPMWSp9e9o0FI1sTvWIiiaPhLszWBsxdpbH1fFs rl3VDuBvzK1E9dheDrneuN7/jgDWmrxl6fedg7TuXpbXZFeq9fgAXGwVnJaU45s+8J4x QkRyAL4d2wNP/pkn3Xd6Ih7AsgosrzLgtPkGdMJYYKJ/Chy/d9lcll6ICSo45izAmPU5 sbryGP0mK5dK1tZF5335EUsOWHzLAy+/gX1VrHwVrDZ1Gx8JAmNiToc6KpOd3uNvaZdg 6Ycg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8LXubQ3FrSF6IYeikSKGn7KB26hbqba/FTsLwOsytAM=; b=TaTxqwtqaTaXFQ3tHlu6zyo8ex48HrjYKmBg+R/bLXKt5xRvGHPYt3u9sk46NbfLak ak5DK/zLBuGdFZXgQvrLJDqQnHxgbQ/YWHM5Skn778KmGgr03MTN7yWdrKoXW41k/J39 /TaU0KuiPoK5BLLpJSJ89wRLh344hiP8Ol2m65ZJYOBaTrBRyisK0+OXaQl5BfnDV9/Y NtmhfvzUdFmyODG+uiAVSSmBAUY1gYFNsf//EI32fYMP537wYekxVP1Nwa4MiBwTzFEL eq/jbOmai5+v0fi8GGb0rVO2Hh7IWMffwvnlyZfobMkIElPXq0LXtuM3rBrhm5sxZBIR q/cg== X-Gm-Message-State: ANoB5pmgeIWqj+W/RAdaCzWbRKbwLW7XvJP4nqnD1ecCjEvWxiYQ/dvv 4yheIj9tiw05agAw++iPmRE9Ml15JEPGUJm5 X-Received: by 2002:a05:6402:f:b0:468:56c3:7c8 with SMTP id d15-20020a056402000f00b0046856c307c8mr11079753edu.109.1670496241964; Thu, 08 Dec 2022 02:44:01 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:01 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Paolo Valente , Gabriele Felici , Gianmarco Lusvardi , Giulio Barabino , Emiliano Maccaferri Subject: [PATCH V9 4/8] block, bfq: turn bfqq_data into an array in bfq_io_cq Date: Thu, 8 Dec 2022 11:43:47 +0100 Message-Id: <20221208104351.35038-5-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751643459181937478?= X-GMAIL-MSGID: =?utf-8?q?1751643459181937478?= When a bfq_queue Q is merged with another queue, several pieces of information are saved about Q. These pieces are stored in the bfqq_data field in the bfq_io_cq data structure of the process associated with Q. Yet, with a multi-actuator drive, a process may get associated with multiple bfq_queues: one queue for each of the N actuators. Each of these queues may undergo a merge. So, the bfq_io_cq data structure must be able to accommodate the above information for N queues. This commit solves this problem by turning the bfqq_data scalar field into an array of N elements (and by changing code so as to handle this array). This solution is written under the assumption that bfq_queues associated with different actuators cannot be cross-merged. This assumption holds naturally with basic queue merging: the latter is triggered by spatial locality, and sectors for different actuators are not close to each other (apart from the corner case of the last sectors served by a given actuator and the first sectors served by the next actuator). As for stable cross-merging, the assumption here is that it is disabled. Signed-off-by: Gabriele Felici Signed-off-by: Gianmarco Lusvardi Signed-off-by: Giulio Barabino Signed-off-by: Emiliano Maccaferri Signed-off-by: Paolo Valente Reviewed-by: Damien Le Moal --- block/bfq-iosched.c | 95 +++++++++++++++++++++++++++------------------ block/bfq-iosched.h | 12 ++++-- 2 files changed, 65 insertions(+), 42 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 0d6b35ef3d3f..18e2b8f75435 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -406,7 +406,7 @@ void bic_set_bfqq(struct bfq_io_cq *bic, * we cancel the stable merge if * bic->stable_merge_bfqq == bfqq. */ - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[actuator_idx]; if (is_sync) bic->bfqq[1][actuator_idx] = bfqq; @@ -1181,9 +1181,10 @@ static void bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, struct bfq_io_cq *bic, bool bfq_already_existing) { - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; unsigned int old_wr_coeff = 1; bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; if (bfqq_data->saved_has_short_ttime) bfq_mark_bfqq_has_short_ttime(bfqq); @@ -1899,7 +1900,9 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || (bfq_bfqq_sync(bfqq) && - (bfqq->bic || RQ_BIC(rq)->bfqq_data.stably_merged) && + (bfqq->bic || + RQ_BIC(rq)->bfqq_data[bfq_actuator_index(bfqd, rq->bio)] + .stably_merged) && (*interactive || soft_rt))); /* @@ -2888,6 +2891,35 @@ static bool bfq_may_be_close_cooperator(struct bfq_queue *bfqq, static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, struct bfq_queue *bfqq); +static struct bfq_queue * +bfq_setup_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, + struct bfq_queue *stable_merge_bfqq, + struct bfq_iocq_bfqq_data *bfqq_data) +{ + int proc_ref = min(bfqq_process_refs(bfqq), + bfqq_process_refs(stable_merge_bfqq)); + + if (!idling_boosts_thr_without_issues(bfqd, bfqq) && + proc_ref > 0) { + /* next function will take at least one ref */ + struct bfq_queue *new_bfqq = + bfq_setup_merge(bfqq, stable_merge_bfqq); + + if (new_bfqq) { + bfqq_data->stably_merged = true; + if (new_bfqq->bic) { + unsigned int new_a_idx = new_bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *new_bfqq_data = + &new_bfqq->bic->bfqq_data[new_a_idx]; + + new_bfqq_data->stably_merged = true; + } + } + return new_bfqq; + } else + return NULL; +} + /* * Attempt to schedule a merge of bfqq with the currently in-service * queue or with a close queue among the scheduled queues. Return @@ -2913,7 +2945,8 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, void *io_struct, bool request, struct bfq_io_cq *bic) { struct bfq_queue *in_service_bfqq, *new_bfqq; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* if a merge has already been setup, then proceed with that first */ if (bfqq->new_bfqq) @@ -2943,29 +2976,15 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, msecs_to_jiffies(bfq_late_stable_merging))) { struct bfq_queue *stable_merge_bfqq = bfqq_data->stable_merge_bfqq; - int proc_ref = min(bfqq_process_refs(bfqq), - bfqq_process_refs(stable_merge_bfqq)); /* deschedule stable merge, because done or aborted here */ bfq_put_stable_ref(stable_merge_bfqq); bfqq_data->stable_merge_bfqq = NULL; - if (!idling_boosts_thr_without_issues(bfqd, bfqq) && - proc_ref > 0) { - /* next function will take at least one ref */ - struct bfq_queue *new_bfqq = - bfq_setup_merge(bfqq, stable_merge_bfqq); - - if (new_bfqq) { - bfqq_data->stably_merged = true; - if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = - true; - } - return new_bfqq; - } else - return NULL; + return bfq_setup_stable_merge(bfqd, bfqq, + stable_merge_bfqq, + bfqq_data); } } @@ -3060,7 +3079,8 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, static void bfq_bfqq_save_state(struct bfq_queue *bfqq) { struct bfq_io_cq *bic = bfqq->bic; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + unsigned int a_idx = bfqq->actuator_idx; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; /* * If !bfqq->bic, the queue is already shared or its requests @@ -3071,7 +3091,7 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) return; bfqq_data->saved_last_serv_time_ns = bfqq->last_serv_time_ns; - bfqq_data->saved_inject_limit = bfqq->inject_limit; + bfqq_data->saved_inject_limit = bfqq->inject_limit; bfqq_data->saved_decrease_time_jif = bfqq->decrease_time_jif; bfqq_data->saved_weight = bfqq->entity.orig_weight; @@ -5438,7 +5458,7 @@ static void bfq_exit_icq(struct io_cq *icq) * therefore on its unused per-actuator fields being NULL. */ unsigned int num_actuators = BFQ_MAX_ACTUATORS; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = bic->bfqq_data; /* * bfqd is NULL if scheduler already exited, and in that case @@ -5449,10 +5469,10 @@ static void bfq_exit_icq(struct io_cq *icq) num_actuators = bfqd->num_actuators; } - if (bfqq_data->stable_merge_bfqq) - bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); - for (act_idx = 0; act_idx < num_actuators; act_idx++) { + if (bfqq_data[act_idx].stable_merge_bfqq) + bfq_put_stable_ref(bfqq_data[act_idx].stable_merge_bfqq); + bfq_exit_icq_bfqq(bic, true, act_idx); bfq_exit_icq_bfqq(bic, false, act_idx); } @@ -5639,16 +5659,16 @@ bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, struct bfq_io_cq *bic, struct bfq_queue *last_bfqq_created) { + unsigned int a_idx = last_bfqq_created->actuator_idx; struct bfq_queue *new_bfqq = bfq_setup_merge(bfqq, last_bfqq_created); - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; if (!new_bfqq) return bfqq; if (new_bfqq->bic) - new_bfqq->bic->bfqq_data.stably_merged = true; - bfqq_data->stably_merged = true; + new_bfqq->bic->bfqq_data[a_idx].stably_merged = true; + bic->bfqq_data[a_idx].stably_merged = true; /* * Reusing merge functions. This implies that @@ -5717,7 +5737,6 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, &bfqd->last_bfqq_created; struct bfq_queue *last_bfqq_created = *source_bfqq; - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; /* * If last_bfqq_created has not been set yet, then init it. If @@ -5779,7 +5798,8 @@ static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, /* * Record the bfqq to merge to. */ - bfqq_data->stable_merge_bfqq = last_bfqq_created; + bic->bfqq_data[last_bfqq_created->actuator_idx].stable_merge_bfqq = + last_bfqq_created; } } @@ -6699,7 +6719,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, { unsigned int act_idx = bfq_actuator_index(bfqd, bio); struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); - struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data; + struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[act_idx]; if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) return bfqq; @@ -6807,7 +6827,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) struct bfq_queue *bfqq; bool new_queue = false; bool bfqq_already_existing = false, split = false; - struct bfq_iocq_bfqq_data *bfqq_data; + unsigned int a_idx = bfq_actuator_index(bfqd, bio); if (unlikely(!rq->elv.icq)) return NULL; @@ -6831,17 +6851,16 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, &new_queue); - bfqq_data = &bic->bfqq_data; - if (likely(!new_queue)) { /* If the queue was seeky for too long, break it apart. */ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq) && - !bfqq_data->stably_merged) { + !bic->bfqq_data[a_idx].stably_merged) { struct bfq_queue *old_bfqq = bfqq; /* Update bic before losing reference to bfqq */ if (bfq_bfqq_in_large_burst(bfqq)) - bfqq_data->saved_in_large_burst = true; + bic->bfqq_data[a_idx].saved_in_large_burst = + true; bfqq = bfq_split_bfqq(bic, bfqq); split = true; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 9d77d964620f..a685aa32b037 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -416,7 +416,7 @@ struct bfq_queue { struct bfq_iocq_bfqq_data { /* * Snapshot of the has_short_time flag before merging; taken - * to remember its value while the queue is merged, so as to + * to remember its values while the queue is merged, so as to * be able to restore it in case of split. */ bool saved_has_short_ttime; @@ -430,7 +430,7 @@ struct bfq_iocq_bfqq_data { u64 saved_tot_idle_time; /* - * Same purpose as the previous fields for the value of the + * Same purpose as the previous fields for the values of the * field keeping the queue's belonging to a large burst */ bool saved_in_large_burst; @@ -493,8 +493,12 @@ struct bfq_io_cq { uint64_t blkcg_serial_nr; /* the current blkcg serial */ #endif - /* persistent data for associated synchronous process queue */ - struct bfq_iocq_bfqq_data bfqq_data; + /* + * Persistent data for associated synchronous process queues + * (one queue per actuator, see field bfqq above). In + * particular, each of these queues may undergo a merge. + */ + struct bfq_iocq_bfqq_data bfqq_data[BFQ_MAX_ACTUATORS]; unsigned int requests; /* Number of requests this process has in flight */ }; From patchwork Thu Dec 8 10:43:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp122377wrr; Thu, 8 Dec 2022 02:50:06 -0800 (PST) X-Google-Smtp-Source: AA0mqf4bBaNyK1RtaNZGFxynpz1M8nhqQENavJeIwz5T+UFjQb15rPgmCiG0btoPgtfv5tOSiRGE X-Received: by 2002:a63:38b:0:b0:478:b931:9388 with SMTP id 133-20020a63038b000000b00478b9319388mr16037077pgd.400.1670496606418; Thu, 08 Dec 2022 02:50:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670496606; cv=none; d=google.com; s=arc-20160816; b=FPQ1CddAoO9YjZJwunB6yfUtZVwKGgVijGIRxld3Z+Ukq0cFT+zBF9AJONPG+AI8Bw POo+/f1o/Q3K1LxvISxPnXWDk1/9Svq2jr2+lA1G0xGN95JTmfty6oUZDj4qTJnCOV22 jNAfwHnrfyatT6juXIf3kf+rvWYLvSQRR33vukLLUlaPk+e+S9zzndYK7thh4sbov+0A pdkHKRxkvovhLeQ7+BUieJR4PmHnjkYyc4PcxwDm5Ge3o0LyJJZdlRcaeD5mIoK3UeKI bgf8NVzUICF8fRJdKdbU0kV3DZZmVlXlKA7NX0oQRGe6lrn326ldxYaNHu34RJDHNYjB lELg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fn/nDo9qIIYXqMkVd3lLzuszzQkwXilXWu3OxWIyjxw=; b=W7IX1tzD6uHEIM0XSSg1+3KxvQm+K52z8dm3sAvyciTKvhvHQVI1nOSUDsyZ5X+8QY fMHVjhi60WtJ9gowe8cAKGy5toztg5cc/7qP053VtCkU0yKlRu1FfsSDLVOD7dbdfDqe 7yOomTLuLMcmGHXP/vkTOxquAGpBWsJc524OHyUv4JhpN9pDooKqSJTr592dnTBIbetG 4VKjYA8jxeBgoMwVDU8ZBX8V+YEwc/c22/MmXFvJUMQV7kQy4iCT1Qyg58YAtqS81qV4 Jjcxws9AB8ZRt4nvsBHdIU0dEhPBEWYHJGRkoty6Gd2sx/YdaB80VTvjUiRsCIGNLpDd 0S5A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Z5yQrD88; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n3-20020a1709026a8300b0018890a7e9c5si20897827plk.509.2022.12.08.02.49.53; Thu, 08 Dec 2022 02:50:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Z5yQrD88; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230086AbiLHKtV (ORCPT + 99 others); Thu, 8 Dec 2022 05:49:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230088AbiLHKsl (ORCPT ); Thu, 8 Dec 2022 05:48:41 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 965638AAC3 for ; Thu, 8 Dec 2022 02:44:04 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id l11so1523572edb.4 for ; Thu, 08 Dec 2022 02:44:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fn/nDo9qIIYXqMkVd3lLzuszzQkwXilXWu3OxWIyjxw=; b=Z5yQrD88ndJqQ+6XBnBPfnx2LoG7MtjsYpwqOu0KsyEebRAqQCmfUOLL72hPoV8ZbJ 5kR8jtWCjTqJOfCXZ9jciP+FYbBqtPtGBNilKgs9wh7s6wYSE2LRMgaley4/ISoEK7WP zahq97ZasErz6TIh2wA/UJKdF/krGepZTfiQa7xAbQU7+NPRxkPeGgzbiGIPzpZ8TCmp b1gdikjLvS+rkMRb/0Je5gYecFrduL0Y2Qvk2C0egs7tgNOvIhm8NoPDj4X4lSHZSHuX IJ8QyhNy5uj0E+WfZ9mJYqmURDivAbnwWnjP1WAi3MHeuk40K3ZZspACoYmR1N+KfxYu ISXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fn/nDo9qIIYXqMkVd3lLzuszzQkwXilXWu3OxWIyjxw=; b=Gy/QYVNGVrg4iTXvbGse4mUgD4kHgV+uVJ0B6HLURCSLwaAUPkhUxtBGdl5/LD4W6P 5fkPN9UdkvgB3mLa859xieOmQfePhkjI+dSrkiHXGob0KhNRdKpmZH3zWwWOXTJZU/+R Teea7tuPN/5TkW8YV2IAeGbJc5nuR8bwHtJGF6n+yp1Lpl7GjBejE6i2WNVIwwISU6R3 iVymgp29w3232F7eu8Oj25klofpVO+1K8J0KWcE+EdKR5BgYVSjlBHD95BFxlDqrELL6 cUKDSzEi0CWy5zo5V8xEViBw9Q4kUPmTYe6qJCAUP3Ns9ENxkLLHVmj2gqXhoJMf7k9d ybaQ== X-Gm-Message-State: ANoB5pk5JdqVW1Gkx0EKZ6+wBYZuiIX7+UGaPKO3CReObMBifxDN3ueG ZOII1NFJl97Oida001Mko+1hKQ== X-Received: by 2002:a05:6402:3644:b0:45f:c7f2:297d with SMTP id em4-20020a056402364400b0045fc7f2297dmr85303225edb.266.1670496242819; Thu, 08 Dec 2022 02:44:02 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:02 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Davide Zini , Damien Le Moal , Paolo Valente Subject: [PATCH V9 5/8] block, bfq: split also async bfq_queues on a per-actuator basis Date: Thu, 8 Dec 2022 11:43:48 +0100 Message-Id: <20221208104351.35038-6-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751642649382194200?= X-GMAIL-MSGID: =?utf-8?q?1751642649382194200?= From: Davide Zini Similarly to sync bfq_queues, also async bfq_queues need to be split on a per-actuator basis. Reviewed-by: Damien Le Moal Signed-off-by: Paolo Valente Signed-off-by: Davide Zini --- block/bfq-iosched.c | 41 +++++++++++++++++++++++------------------ block/bfq-iosched.h | 8 ++++---- 2 files changed, 27 insertions(+), 22 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 18e2b8f75435..dcecba3c6e23 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2681,14 +2681,16 @@ static void bfq_bfqq_end_wr(struct bfq_queue *bfqq) void bfq_end_wr_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) { - int i, j; - - for (i = 0; i < 2; i++) - for (j = 0; j < IOPRIO_NR_LEVELS; j++) - if (bfqg->async_bfqq[i][j]) - bfq_bfqq_end_wr(bfqg->async_bfqq[i][j]); - if (bfqg->async_idle_bfqq) - bfq_bfqq_end_wr(bfqg->async_idle_bfqq); + int i, j, k; + + for (k = 0; k < bfqd->num_actuators; k++) { + for (i = 0; i < 2; i++) + for (j = 0; j < IOPRIO_NR_LEVELS; j++) + if (bfqg->async_bfqq[i][j][k]) + bfq_bfqq_end_wr(bfqg->async_bfqq[i][j][k]); + if (bfqg->async_idle_bfqq[k]) + bfq_bfqq_end_wr(bfqg->async_idle_bfqq[k]); + } } static void bfq_end_wr(struct bfq_data *bfqd) @@ -5637,18 +5639,18 @@ static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, static struct bfq_queue **bfq_async_queue_prio(struct bfq_data *bfqd, struct bfq_group *bfqg, - int ioprio_class, int ioprio) + int ioprio_class, int ioprio, int act_idx) { switch (ioprio_class) { case IOPRIO_CLASS_RT: - return &bfqg->async_bfqq[0][ioprio]; + return &bfqg->async_bfqq[0][ioprio][act_idx]; case IOPRIO_CLASS_NONE: ioprio = IOPRIO_BE_NORM; fallthrough; case IOPRIO_CLASS_BE: - return &bfqg->async_bfqq[1][ioprio]; + return &bfqg->async_bfqq[1][ioprio][act_idx]; case IOPRIO_CLASS_IDLE: - return &bfqg->async_idle_bfqq; + return &bfqg->async_idle_bfqq[act_idx]; default: return NULL; } @@ -5821,7 +5823,8 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, bfqg = bfq_bio_bfqg(bfqd, bio); if (!is_sync) { async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class, - ioprio); + ioprio, + bfq_actuator_index(bfqd, bio)); bfqq = *async_bfqq; if (bfqq) goto out; @@ -7038,13 +7041,15 @@ static void __bfq_put_async_bfqq(struct bfq_data *bfqd, */ void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) { - int i, j; + int i, j, k; - for (i = 0; i < 2; i++) - for (j = 0; j < IOPRIO_NR_LEVELS; j++) - __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j]); + for (k = 0; k < bfqd->num_actuators; k++) { + for (i = 0; i < 2; i++) + for (j = 0; j < IOPRIO_NR_LEVELS; j++) + __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j][k]); - __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq); + __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq[k]); + } } /* diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index a685aa32b037..1450990dba32 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -976,8 +976,8 @@ struct bfq_group { void *bfqd; - struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS]; - struct bfq_queue *async_idle_bfqq; + struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS][BFQ_MAX_ACTUATORS]; + struct bfq_queue *async_idle_bfqq[BFQ_MAX_ACTUATORS]; struct bfq_entity *my_entity; @@ -993,8 +993,8 @@ struct bfq_group { struct bfq_entity entity; struct bfq_sched_data sched_data; - struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS]; - struct bfq_queue *async_idle_bfqq; + struct bfq_queue *async_bfqq[2][IOPRIO_NR_LEVELS][BFQ_MAX_ACTUATORS]; + struct bfq_queue *async_idle_bfqq[BFQ_MAX_ACTUATORS]; struct rb_root rq_pos_tree; }; From patchwork Thu Dec 8 10:43:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31286 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp127007wrr; Thu, 8 Dec 2022 03:02:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf4zwlEkZLbPL91SeBW39IzERrjpAAgjE5N1vteuM2gnoqnEtMscbgkXJvxc9bdRS8y9wgSv X-Received: by 2002:a17:906:b1c6:b0:7ad:e82b:b66b with SMTP id bv6-20020a170906b1c600b007ade82bb66bmr15929684ejb.453.1670497370160; Thu, 08 Dec 2022 03:02:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670497370; cv=none; d=google.com; s=arc-20160816; b=Cz1Bc50hSeH6UEkZb+hbNXKFOifF/3e7U40rTYvSu0UnnuWE002xbEPlpgy+yS4u3k E4SXBZgZ8zMsXYiBNkN0AJxcHERO9ZMSq83s5GxRPYRB9KNAAq1Y2YfmHZtM7fEFQAQV f6IM70Kc/qZuyfCNVUY/ABcKJYn+CM2W1c5AVrKtP7d+YsRySErIIqoh2kSbdGKAhhxB mz8xtZJQrG/Joj4XSPjVVNJ2f3EB+wIlXTNe0OolFHRhHebozpOS57s5fz+G6AlsFc9Y AzSvHDAyyApuXdJTUPixwDBjmcVGQLGMHkQ7uiPbvFw3SSkyGh4mpjt5cZcmanjl1B6H rOsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=K7xEq2tmiQO2KsEHqRclviFrnGV/8d8j2wl23Ajh0XU=; b=Hnd+LurCjMyR69AZ0hdoprt8el4TnlurRSVaYUGkXaP+A0sZzL4hoi6wqVIDFyN8XT 7Gh0Ktn/Edce8JR9Kf9/bOWSj8+H+JnkCpigkAqPnMx+XmQy9QyTYDKgut0AEJvB61/g 9Zm0hOMlt6xKgRv/B9aIoxwkP67oDBo2oGZJH9ekJGA1pTodVd5yeEnnQOid91Y21I0P g6GVToKrrW6+xsMyeRE5eT3JjkjTZhLdYDPe9Ngh1CjM7cCTtEoqiYwUogDa7GNJJ2RE P8NYDzPAbJb2YO77W4lHyjiOPmXO52XEQX4m/Jbcoja+H8Mvv9FwCIB+RXnxt7it3W8v BUwA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vDa4LXRW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y12-20020aa7cccc000000b004619acbc70fsi5719017edt.505.2022.12.08.03.02.26; Thu, 08 Dec 2022 03:02:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=vDa4LXRW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229892AbiLHKt5 (ORCPT + 99 others); Thu, 8 Dec 2022 05:49:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230263AbiLHKss (ORCPT ); Thu, 8 Dec 2022 05:48:48 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94412178A4 for ; Thu, 8 Dec 2022 02:44:05 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id c17so1492864edj.13 for ; Thu, 08 Dec 2022 02:44:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K7xEq2tmiQO2KsEHqRclviFrnGV/8d8j2wl23Ajh0XU=; b=vDa4LXRWKgw4W2u9WKOecDvbeqsF9ooSi/mdfU9eLI18SHW8WE45FLgdDSGl603Rhx 4mzQdt0J6sAmNEIPUpOZk/sfi/gh22x9VJJT9DeWCZYF+mNwQQA4FG4BjNFwj2aMxAQt Fnmv3nNzeEimIMEhK5XVtM/hiBbOiTmhVtgF33VIYoZ/3BYisWwxRPbhBw54+FH2XYET 1u0Mkp88JuR3rKvQb9S3wbRyEX4W2WNtNXdmIipx8FterzkopVBR3m3AKLFPPHkEymnQ 6k3A2SW+OW9uZ75B7YF0NZAOgj0AYJg2SqdlPomq9KWRfnqlOcCYsf/7t+l2zhv8CdsY 21jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K7xEq2tmiQO2KsEHqRclviFrnGV/8d8j2wl23Ajh0XU=; b=RXJDpt8YmYikT7CrWYEdTxV3H340qKT/yIWZypSKVXlYmgj2omo7ivEjOJCJpNR0PU 8SV5dnwLe9A9j1/LzKI8i2/TzKRuSDzW9Ov8VNdmYS/nU4qlb4LCc/jl7YwwAm5bapki TiFIl2QraSpgUna6OGUyEgOJCOKN2GrkLi+zjlMeANs+0UlbISqagI7eheWGQqiKAZnF PA5ceadIEYM+YmtFflFzhKrTizUFDlaGluFT6gqayMtm6IVc1hziSvFhUXTah0r5a/+c dJZ9n/slq+tS9xO8Ihb/9SPd6LMXLU4fGJI7//PMt0ihtvY5CeTTc8QMKXa7AX31YuRS Q/ng== X-Gm-Message-State: ANoB5pmxE2X1nxCkM36DZmC6v18HGXJmt4RbQNrqt4D5RzIEhk9sZdDU jdoir3TDK2pBZyFPW/lpmojHig== X-Received: by 2002:a05:6402:2949:b0:468:fb6b:3a79 with SMTP id ed9-20020a056402294900b00468fb6b3a79mr26310140edb.63.1670496243600; Thu, 08 Dec 2022 02:44:03 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:03 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Federico Gavioli , Paolo Valente Subject: [PATCH V9 6/8] block, bfq: retrieve independent access ranges from request queue Date: Thu, 8 Dec 2022 11:43:49 +0100 Message-Id: <20221208104351.35038-7-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751643450786179545?= X-GMAIL-MSGID: =?utf-8?q?1751643450786179545?= From: Federico Gavioli This patch implements the code to gather the content of the independent_access_ranges structure from the request_queue and copy it into the queue's bfq_data. This copy is done at queue initialization. We copy the access ranges into the bfq_data to avoid taking the queue lock each time we access the ranges. This implementation, however, puts a limit to the maximum independent ranges supported by the scheduler. Such a limit is equal to the constant BFQ_MAX_ACTUATORS. This limit was placed to avoid the allocation of dynamic memory. Co-developed-by: Rory Chen Signed-off-by: Rory Chen Signed-off-by: Federico Gavioli Signed-off-by: Paolo Valente Reviewed-by: Damien Le Moal --- block/bfq-iosched.c | 60 +++++++++++++++++++++++++++++++++++++++------ block/bfq-iosched.h | 8 +++++- 2 files changed, 59 insertions(+), 9 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index dcecba3c6e23..957ce61faaf2 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1839,10 +1839,25 @@ static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, */ static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) { - /* - * Multi-actuator support not complete yet, so always return 0 - * for the moment (to keep incomplete mechanisms off). - */ + unsigned int i; + sector_t end; + + /* no search needed if one or zero ranges present */ + if (bfqd->num_actuators == 1) + return 0; + + /* bio_end_sector(bio) gives the sector after the last one */ + end = bio_end_sector(bio) - 1; + + for (i = 0; i < bfqd->num_actuators; i++) { + if (end >= bfqd->sector[i] && + end < bfqd->sector[i] + bfqd->nr_sectors[i]) + return i; + } + + WARN_ONCE(true, + "bfq_actuator_index: bio sector out of ranges: end=%llu\n", + end); return 0; } @@ -7160,6 +7175,8 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) { struct bfq_data *bfqd; struct elevator_queue *eq; + unsigned int i; + struct blk_independent_access_ranges *ia_ranges = q->disk->ia_ranges; eq = elevator_alloc(q, e); if (!eq) @@ -7202,12 +7219,39 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue = q; + bfqd->num_actuators = 1; /* - * Multi-actuator support not complete yet, unconditionally - * set to only one actuator for the moment (to keep incomplete - * mechanisms off). + * If the disk supports multiple actuators, copy independent + * access ranges from the request queue structure. */ - bfqd->num_actuators = 1; + spin_lock_irq(&q->queue_lock); + if (ia_ranges) { + /* + * Check if the disk ia_ranges size exceeds the current bfq + * actuator limit. + */ + if (ia_ranges->nr_ia_ranges > BFQ_MAX_ACTUATORS) { + pr_crit("nr_ia_ranges higher than act limit: iars=%d, max=%d.\n", + ia_ranges->nr_ia_ranges, BFQ_MAX_ACTUATORS); + pr_crit("Falling back to single actuator mode.\n"); + } else { + bfqd->num_actuators = ia_ranges->nr_ia_ranges; + + for (i = 0; i < bfqd->num_actuators; i++) { + bfqd->sector[i] = ia_ranges->ia_range[i].sector; + bfqd->nr_sectors[i] = + ia_ranges->ia_range[i].nr_sectors; + } + } + } + + /* Otherwise use single-actuator dev info */ + if (bfqd->num_actuators == 1) { + bfqd->sector[0] = 0; + bfqd->nr_sectors[0] = + bdev_nr_sectors(dev_to_bdev(disk_to_dev(q->disk))); + } + spin_unlock_irq(&q->queue_lock); INIT_LIST_HEAD(&bfqd->dispatch); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 1450990dba32..953980de6b4b 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -810,7 +810,13 @@ struct bfq_data { * case of single-actuator drives. */ unsigned int num_actuators; - + /* + * Disk independent access ranges for each actuator + * in this device. + */ + sector_t sector[BFQ_MAX_ACTUATORS]; + sector_t nr_sectors[BFQ_MAX_ACTUATORS]; + struct blk_independent_access_range ia_ranges[BFQ_MAX_ACTUATORS]; }; enum bfqq_state_flags { From patchwork Thu Dec 8 10:43:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31285 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp126510wrr; Thu, 8 Dec 2022 03:01:54 -0800 (PST) X-Google-Smtp-Source: AA0mqf4KazZXTj9sqlYkdNbGOUDFsBisrlZ/PBmE5K2hqrhgTOffE8BIcPflEmpjSYTZ3UfmMCmG X-Received: by 2002:a50:fc0b:0:b0:46b:5812:a03a with SMTP id i11-20020a50fc0b000000b0046b5812a03amr37566723edr.91.1670497314664; Thu, 08 Dec 2022 03:01:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670497314; cv=none; d=google.com; s=arc-20160816; b=uyP9TyYRqrtHJ/lquEtKP+/PgScYoBqGUH1/fEXCt1uml0QGu7DG/UOK5tbaSPyLOp XgTjKEiCQqbDeQd4I1FKyz0q2RXo/bfc+L1SC9RHnsxZOruyI2iO6gxSRZ0ypw9X/4cQ g1hlobhnNxKDNZCnDkBWaZ5DplDlYlmK0lKve3Ruq/HmYYuqxVfXX8NN/9Ug28q5VBrn CK6cNTuCWUTJ1/PwmOaEvW4TuN2tMPE5SMjl3Yh4ctH5P/28bSudDjKsEKbQORr8sedi hmk7D5+R9S14jY1VKlivEYCov7dA5Low/TBDr4QqdyjbG+tXdeeYsClYDUbc52PGFqUn DHyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HF+jHBemdMEbA7CJGD5K+MOMwDPKDHUqbroB/FrxP0A=; b=DiUaLi3lSXOhfw/dmwtExwhDRoBEQlXIWv4CnwsHn+Lqjr34ifigpZX/yjICBQqR0+ EeEfYo2Noa7CZl/dbMuYd9ImWo5D3Ein3luI/nAoOeBDl+gsEpTIQZZQuCZO7W+Vlcam y8iTzQEJkZw0J5G3l9wO+s2cJZaeUgWR5bg0A3juu5vH4mojxpcr5NN7iq6IYZDpY5zW U5UcGVmlCulRiiuCtUkjQMWokXUf559sZln6ciJS1JkYsgAVyxxgHtLHKNeh02fCdwig S0EZIGxITmAac4MFHmlfsP4Zc6sKMJ7yAPm1bku38q/CB+NHQzZ1qKWlXRASbQM+zTrN Hrew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YfUHikUB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id un2-20020a170907cb8200b007c0e0d7abe5si8512318ejc.782.2022.12.08.03.01.29; Thu, 08 Dec 2022 03:01:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YfUHikUB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230332AbiLHKuF (ORCPT + 99 others); Thu, 8 Dec 2022 05:50:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229714AbiLHKs5 (ORCPT ); Thu, 8 Dec 2022 05:48:57 -0500 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9552A56D78 for ; Thu, 8 Dec 2022 02:44:06 -0800 (PST) Received: by mail-ed1-x529.google.com with SMTP id a16so1512525edb.9 for ; Thu, 08 Dec 2022 02:44:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HF+jHBemdMEbA7CJGD5K+MOMwDPKDHUqbroB/FrxP0A=; b=YfUHikUBep8tR8BjFy3p/9hj2HDRr7exZdIaZFabQoAN4L8jHxEYX1OlegIeBmH07A VaYrHwAELC2nun9P5VYFmNnjfXZliopTDEcLMJelPT6fmyjcJhoply4+z0icIAUnzlC7 IMFuo8T6UH1enoE5HjrAaXTvTN37EnnusFgVV1AuHjVOtbNXNe0WeR24IAMCZmbvZdDc 4yqVwKwKKRm1qp+cBJgsHG/TMC0CWTP34hlXjXE+TLDfNKx937amQO7JzfL5vAIt9Dw+ HeT3pHRp/4z+icOV8XxQd8GCC7zSroz5WEZTbADq6gUr28XLiu4Ny9ZED262WiJxcu/5 d8sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HF+jHBemdMEbA7CJGD5K+MOMwDPKDHUqbroB/FrxP0A=; b=oqK+MZr2yfNh13z1AvO93H1S0Rcf4VwMq/QbficzqOyL+rbgoUn2Re2Qj38oMTwcZy mkWrMkBZVOptdOK78p+ULCIp4c28f0y8yKixVZF4+ICkufNwkXfcr2zGhSiDY6Gr4WCU V3agUYJ1y4GMF0V31qOJAu7A6oS+NPhNQkxE0vl0BgAqsDIlYJowkMF1MY9VCosajEcP d8e81v+18yRsT/uHUcOAPEpSBM9sOeyVqVw6Rb1Zvm/aVPwiXHkt/3NwfNplzwgblcQn kpfTR8REJ3rol2IG4Io0QcQmgbfBISRTIcxKib1if2FaqbyXACMFFt79+fWhJGct1PL6 +Y2Q== X-Gm-Message-State: ANoB5pl7JJszKxUmaSyR1Ie37gppsNaKKqAq2G64+DJWoUX0jrvJ2/qF wcgNMzo6EJb8pL32kLQXCd01Kw== X-Received: by 2002:a05:6402:6c6:b0:46c:1326:e186 with SMTP id n6-20020a05640206c600b0046c1326e186mr23743794edy.228.1670496244712; Thu, 08 Dec 2022 02:44:04 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:04 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Davide Zini , Damien Le Moal , Paolo Valente Subject: [PATCH V9 7/8] block, bfq: inject I/O to underutilized actuators Date: Thu, 8 Dec 2022 11:43:50 +0100 Message-Id: <20221208104351.35038-8-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751643392315945447?= X-GMAIL-MSGID: =?utf-8?q?1751643392315945447?= From: Davide Zini The main service scheme of BFQ for sync I/O is serving one sync bfq_queue at a time, for a while. In particular, BFQ enforces this scheme when it deems the latter necessary to boost throughput or to preserve service guarantees. Unfortunately, when BFQ enforces this policy, only one actuator at a time gets served for a while, because each bfq_queue contains I/O only for one actuator. The other actuators may remain underutilized. Actually, BFQ may serve (inject) extra I/O, taken from other bfq_queues, in parallel with that of the in-service queue. This injection mechanism may provide the ground for dealing also with the above actuator-underutilization problem. Yet BFQ does not take the actuator load into account when choosing which queue to pick extra I/O from. In addition, BFQ may happen to inject extra I/O only when the in-service queue is temporarily empty. In view of these facts, this commit extends the injection mechanism in such a way that the latter: (1) takes into account also the actuator load; (2) checks such a load on each dispatch, and injects I/O for an underutilized actuator, if there is one and there is I/O for it. To perform the check in (2), this commit introduces a load threshold, currently set to 4. A linear scan of each actuator is performed, until an actuator is found for which the following two conditions hold: the load of the actuator is below the threshold, and there is at least one non-in-service queue that contains I/O for that actuator. If such a pair (actuator, queue) is found, then the head request of that queue is returned for dispatch, instead of the head request of the in-service queue. We have set the threshold, empirically, to the minimum possible value for which an actuator is fully utilized, or close to be fully utilized. By doing so, injected I/O 'steals' as few drive-queue slots as possibile to the in-service queue. This reduces as much as possible the probability that the service of I/O from the in-service bfq_queue gets delayed because of slot exhaustion, i.e., because all the slots of the drive queue are filled with I/O injected from other queues (NCQ provides for 32 slots). This new mechanism also counters actuator underutilization in the case of asymmetric configurations of bfq_queues. Namely if there are few bfq_queues containing I/O for some actuators and many bfq_queues containing I/O for other actuators. Or if the bfq_queues containing I/O for some actuators have lower weights than the other bfq_queues. Reviewed-by: Damien Le Moal Signed-off-by: Paolo Valente Signed-off-by: Davide Zini Reviewed-by: Damien Le Moal --- block/bfq-cgroup.c | 2 +- block/bfq-iosched.c | 143 ++++++++++++++++++++++++++++++++------------ block/bfq-iosched.h | 39 +++++++++++- block/bfq-wf2q.c | 2 +- 4 files changed, 144 insertions(+), 42 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 7ea77f994424..5783601f3d7a 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -698,7 +698,7 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_activate_bfqq(bfqd, bfqq); } - if (!bfqd->in_service_queue && !bfqd->rq_in_driver) + if (!bfqd->in_service_queue && !bfqd->tot_rq_in_driver) bfq_schedule_dispatch(bfqd); /* release extra ref taken above, bfqq may happen to be freed now */ bfq_put_queue(bfqq); diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 957ce61faaf2..16230a9cf794 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2304,9 +2304,9 @@ static void bfq_add_request(struct request *rq) * elapsed. */ if (bfqq == bfqd->in_service_queue && - (bfqd->rq_in_driver == 0 || + (bfqd->tot_rq_in_driver == 0 || (bfqq->last_serv_time_ns > 0 && - bfqd->rqs_injected && bfqd->rq_in_driver > 0)) && + bfqd->rqs_injected && bfqd->tot_rq_in_driver > 0)) && time_is_before_eq_jiffies(bfqq->decrease_time_jif + msecs_to_jiffies(10))) { bfqd->last_empty_occupied_ns = ktime_get_ns(); @@ -2330,7 +2330,7 @@ static void bfq_add_request(struct request *rq) * will be set in case injection is performed * on bfqq before rq is completed). */ - if (bfqd->rq_in_driver == 0) + if (bfqd->tot_rq_in_driver == 0) bfqd->rqs_injected = false; } } @@ -2428,15 +2428,18 @@ static sector_t get_sdist(sector_t last_pos, struct request *rq) static void bfq_activate_request(struct request_queue *q, struct request *rq) { struct bfq_data *bfqd = q->elevator->elevator_data; + unsigned int act_idx = bfq_actuator_index(bfqd, rq->bio); - bfqd->rq_in_driver++; + bfqd->tot_rq_in_driver++; + bfqd->rq_in_driver[act_idx]++; } static void bfq_deactivate_request(struct request_queue *q, struct request *rq) { struct bfq_data *bfqd = q->elevator->elevator_data; - bfqd->rq_in_driver--; + bfqd->tot_rq_in_driver--; + bfqd->rq_in_driver[bfq_actuator_index(bfqd, rq->bio)]--; } #endif @@ -2711,11 +2714,14 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd, static void bfq_end_wr(struct bfq_data *bfqd) { struct bfq_queue *bfqq; + int i; spin_lock_irq(&bfqd->lock); - list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) - bfq_bfqq_end_wr(bfqq); + for (i = 0; i < bfqd->num_actuators; i++) { + list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) + bfq_bfqq_end_wr(bfqq); + } list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list) bfq_bfqq_end_wr(bfqq); bfq_end_wr_async(bfqd); @@ -3672,13 +3678,13 @@ static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq) * - start a new observation interval with this dispatch */ if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC && - bfqd->rq_in_driver == 0) + bfqd->tot_rq_in_driver == 0) goto update_rate_and_reset; /* Update sampling information */ bfqd->peak_rate_samples++; - if ((bfqd->rq_in_driver > 0 || + if ((bfqd->tot_rq_in_driver > 0 || now_ns - bfqd->last_completion < BFQ_MIN_TT) && !BFQ_RQ_SEEKY(bfqd, bfqd->last_position, rq)) bfqd->sequential_samples++; @@ -3943,10 +3949,8 @@ static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd, return false; return (bfqq->wr_coeff > 1 && - (bfqd->wr_busy_queues < - tot_busy_queues || - bfqd->rq_in_driver >= - bfqq->dispatched + 4)) || + (bfqd->wr_busy_queues < tot_busy_queues || + bfqd->tot_rq_in_driver >= bfqq->dispatched + 4)) || bfq_asymmetric_scenario(bfqd, bfqq) || tot_busy_queues == 1; } @@ -4717,6 +4721,8 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) { struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue; unsigned int limit = in_serv_bfqq->inject_limit; + int i; + /* * If * - bfqq is not weight-raised and therefore does not carry @@ -4748,7 +4754,7 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) ) limit = 1; - if (bfqd->rq_in_driver >= limit) + if (bfqd->tot_rq_in_driver >= limit) return NULL; /* @@ -4763,11 +4769,12 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) * (and re-added only if it gets new requests, but then it * is assigned again enough budget for its new backlog). */ - list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) - if (!RB_EMPTY_ROOT(&bfqq->sort_list) && - (in_serv_always_inject || bfqq->wr_coeff > 1) && - bfq_serv_to_charge(bfqq->next_rq, bfqq) <= - bfq_bfqq_budget_left(bfqq)) { + for (i = 0; i < bfqd->num_actuators; i++) { + list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) + if (!RB_EMPTY_ROOT(&bfqq->sort_list) && + (in_serv_always_inject || bfqq->wr_coeff > 1) && + bfq_serv_to_charge(bfqq->next_rq, bfqq) <= + bfq_bfqq_budget_left(bfqq)) { /* * Allow for only one large in-flight request * on non-rotational devices, for the @@ -4792,22 +4799,69 @@ bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) else limit = in_serv_bfqq->inject_limit; - if (bfqd->rq_in_driver < limit) { + if (bfqd->tot_rq_in_driver < limit) { bfqd->rqs_injected = true; return bfqq; } } + } + + return NULL; +} + +static struct bfq_queue * +bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, int idx) +{ + struct bfq_queue *bfqq = NULL; + + if (bfqd->in_service_queue && + bfqd->in_service_queue->actuator_idx == idx) + return bfqd->in_service_queue; + + list_for_each_entry(bfqq, &bfqd->active_list[idx], bfqq_list) { + if (!RB_EMPTY_ROOT(&bfqq->sort_list) && + bfq_serv_to_charge(bfqq->next_rq, bfqq) <= + bfq_bfqq_budget_left(bfqq)) { + return bfqq; + } + } return NULL; } +/* + * Perform a linear scan of each actuator, until an actuator is found + * for which the following two conditions hold: the load of the + * actuator is below the threshold (see comments on actuator_load_threshold + * for details), and there is a queue that contains I/O for that + * actuator. On success, return that queue. + */ +static struct bfq_queue * +bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) +{ + int i; + + for (i = 0 ; i < bfqd->num_actuators; i++) { + if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold) { + struct bfq_queue *bfqq = + bfq_find_active_bfqq_for_actuator(bfqd, i); + + if (bfqq) + return bfqq; + } + } + + return NULL; +} + + /* * Select a queue for service. If we have a current queue in service, * check whether to continue servicing it, or retrieve and set a new one. */ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) { - struct bfq_queue *bfqq; + struct bfq_queue *bfqq, *inject_bfqq; struct request *next_rq; enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT; @@ -4829,6 +4883,15 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) goto expire; check_queue: + /* + * If some actuator is underutilized, but the in-service + * queue does not contain I/O for that actuator, then try to + * inject I/O for that actuator. + */ + inject_bfqq = bfq_find_bfqq_for_underused_actuator(bfqd); + if (inject_bfqq && inject_bfqq != bfqq) + return inject_bfqq; + /* * This loop is rarely executed more than once. Even when it * happens, it is much more convenient to re-execute this loop @@ -5184,11 +5247,11 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) /* * We exploit the bfq_finish_requeue_request hook to - * decrement rq_in_driver, but + * decrement tot_rq_in_driver, but * bfq_finish_requeue_request will not be invoked on * this request. So, to avoid unbalance, just start - * this request, without incrementing rq_in_driver. As - * a negative consequence, rq_in_driver is deceptively + * this request, without incrementing tot_rq_in_driver. As + * a negative consequence, tot_rq_in_driver is deceptively * lower than it should be while this request is in * service. This may cause bfq_schedule_dispatch to be * invoked uselessly. @@ -5197,7 +5260,7 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) * bfq_finish_requeue_request hook, if defined, is * probably invoked also on this request. So, by * exploiting this hook, we could 1) increment - * rq_in_driver here, and 2) decrement it in + * tot_rq_in_driver here, and 2) decrement it in * bfq_finish_requeue_request. Such a solution would * let the value of the counter be always accurate, * but it would entail using an extra interface @@ -5226,7 +5289,7 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) * Of course, serving one request at a time may cause loss of * throughput. */ - if (bfqd->strict_guarantees && bfqd->rq_in_driver > 0) + if (bfqd->strict_guarantees && bfqd->tot_rq_in_driver > 0) goto exit; bfqq = bfq_select_queue(bfqd); @@ -5237,7 +5300,8 @@ static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) if (rq) { inc_in_driver_start_rq: - bfqd->rq_in_driver++; + bfqd->rq_in_driver[bfqq->actuator_idx]++; + bfqd->tot_rq_in_driver++; start_rq: rq->rq_flags |= RQF_STARTED; } @@ -6305,7 +6369,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) struct bfq_queue *bfqq = bfqd->in_service_queue; bfqd->max_rq_in_driver = max_t(int, bfqd->max_rq_in_driver, - bfqd->rq_in_driver); + bfqd->tot_rq_in_driver); if (bfqd->hw_tag == 1) return; @@ -6316,7 +6380,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) * sum is not exact, as it's not taking into account deactivated * requests. */ - if (bfqd->rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD) + if (bfqd->tot_rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD) return; /* @@ -6327,7 +6391,7 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) if (bfqq && bfq_bfqq_has_short_ttime(bfqq) && bfqq->dispatched + bfqq->queued[0] + bfqq->queued[1] < BFQ_HW_QUEUE_THRESHOLD && - bfqd->rq_in_driver < BFQ_HW_QUEUE_THRESHOLD) + bfqd->tot_rq_in_driver < BFQ_HW_QUEUE_THRESHOLD) return; if (bfqd->hw_tag_samples++ < BFQ_HW_QUEUE_SAMPLES) @@ -6348,7 +6412,8 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) bfq_update_hw_tag(bfqd); - bfqd->rq_in_driver--; + bfqd->rq_in_driver[bfqq->actuator_idx]--; + bfqd->tot_rq_in_driver--; bfqq->dispatched--; if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) { @@ -6467,7 +6532,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) BFQQE_NO_MORE_REQUESTS); } - if (!bfqd->rq_in_driver) + if (!bfqd->tot_rq_in_driver) bfq_schedule_dispatch(bfqd); } @@ -6598,13 +6663,13 @@ static void bfq_update_inject_limit(struct bfq_data *bfqd, * conditions to do it, or we can lower the last base value * computed. * - * NOTE: (bfqd->rq_in_driver == 1) means that there is no I/O + * NOTE: (bfqd->tot_rq_in_driver == 1) means that there is no I/O * request in flight, because this function is in the code * path that handles the completion of a request of bfqq, and, * in particular, this function is executed before - * bfqd->rq_in_driver is decremented in such a code path. + * bfqd->tot_rq_in_driver is decremented in such a code path. */ - if ((bfqq->last_serv_time_ns == 0 && bfqd->rq_in_driver == 1) || + if ((bfqq->last_serv_time_ns == 0 && bfqd->tot_rq_in_driver == 1) || tot_time_ns < bfqq->last_serv_time_ns) { if (bfqq->last_serv_time_ns == 0) { /* @@ -6614,7 +6679,7 @@ static void bfq_update_inject_limit(struct bfq_data *bfqd, bfqq->inject_limit = max_t(unsigned int, 1, old_limit); } bfqq->last_serv_time_ns = tot_time_ns; - } else if (!bfqd->rqs_injected && bfqd->rq_in_driver == 1) + } else if (!bfqd->rqs_injected && bfqd->tot_rq_in_driver == 1) /* * No I/O injected and no request still in service in * the drive: these are the exact conditions for @@ -7262,7 +7327,8 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->queue_weights_tree = RB_ROOT_CACHED; bfqd->num_groups_with_pending_reqs = 0; - INIT_LIST_HEAD(&bfqd->active_list); + INIT_LIST_HEAD(&bfqd->active_list[0]); + INIT_LIST_HEAD(&bfqd->active_list[1]); INIT_LIST_HEAD(&bfqd->idle_list); INIT_HLIST_HEAD(&bfqd->burst_list); @@ -7307,6 +7373,9 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) ref_wr_duration[blk_queue_nonrot(bfqd->queue)]; bfqd->peak_rate = ref_rate[blk_queue_nonrot(bfqd->queue)] * 2 / 3; + /* see comments on the definition of next field inside bfq_data */ + bfqd->actuator_load_threshold = 4; + spin_lock_init(&bfqd->lock); /* diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 953980de6b4b..be725d9143fa 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -586,7 +586,12 @@ struct bfq_data { /* number of queued requests */ int queued; /* number of requests dispatched and waiting for completion */ - int rq_in_driver; + int tot_rq_in_driver; + /* + * number of requests dispatched and waiting for completion + * for each actuator + */ + int rq_in_driver[BFQ_MAX_ACTUATORS]; /* true if the device is non rotational and performs queueing */ bool nonrot_with_queueing; @@ -680,8 +685,13 @@ struct bfq_data { /* maximum budget allotted to a bfq_queue before rescheduling */ int bfq_max_budget; - /* list of all the bfq_queues active on the device */ - struct list_head active_list; + /* + * List of all the bfq_queues active for a specific actuator + * on the device. Keeping active queues separate on a + * per-actuator basis helps implementing per-actuator + * injection more efficiently. + */ + struct list_head active_list[BFQ_MAX_ACTUATORS]; /* list of all the bfq_queues idle on the device */ struct list_head idle_list; @@ -817,6 +827,29 @@ struct bfq_data { sector_t sector[BFQ_MAX_ACTUATORS]; sector_t nr_sectors[BFQ_MAX_ACTUATORS]; struct blk_independent_access_range ia_ranges[BFQ_MAX_ACTUATORS]; + + /* + * If the number of I/O requests queued in the device for a + * given actuator is below next threshold, then the actuator + * is deemed as underutilized. If this condition is found to + * hold for some actuator upon a dispatch, but (i) the + * in-service queue does not contain I/O for that actuator, + * while (ii) some other queue does contain I/O for that + * actuator, then the head I/O request of the latter queue is + * returned (injected), instead of the head request of the + * currently in-service queue. + * + * We set the threshold, empirically, to the minimum possible + * value for which an actuator is fully utilized, or close to + * be fully utilized. By doing so, injected I/O 'steals' as + * few drive-queue slots as possibile to the in-service + * queue. This reduces as much as possible the probability + * that the service of I/O from the in-service bfq_queue gets + * delayed because of slot exhaustion, i.e., because all the + * slots of the drive queue are filled with I/O injected from + * other queues (NCQ provides for 32 slots). + */ + unsigned int actuator_load_threshold; }; enum bfqq_state_flags { diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c index 8fc3da4c23bb..ec0273e2cd07 100644 --- a/block/bfq-wf2q.c +++ b/block/bfq-wf2q.c @@ -477,7 +477,7 @@ static void bfq_active_insert(struct bfq_service_tree *st, bfqd = (struct bfq_data *)bfqg->bfqd; #endif if (bfqq) - list_add(&bfqq->bfqq_list, &bfqq->bfqd->active_list); + list_add(&bfqq->bfqq_list, &bfqq->bfqd->active_list[bfqq->actuator_idx]); #ifdef CONFIG_BFQ_GROUP_IOSCHED if (bfqg != bfqd->root_group) bfqg->active_entities++; From patchwork Thu Dec 8 10:43:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 31279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp122408wrr; Thu, 8 Dec 2022 02:50:12 -0800 (PST) X-Google-Smtp-Source: AA0mqf69pwGZR6hWQ076gup2BwYS+IDTDw0ajrpTLZjaOatzYqkRaH13dTFsCaJ3Cqv9dqcP67zm X-Received: by 2002:a17:90a:ab84:b0:213:343:9873 with SMTP id n4-20020a17090aab8400b0021303439873mr107933089pjq.102.1670496612012; Thu, 08 Dec 2022 02:50:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670496612; cv=none; d=google.com; s=arc-20160816; b=CZYhHT0pKKzZWxbPvKBtvDxU84ECbWw6Wr9ntd7tcM9aUQhTNhJ1MoeAniTsIj0f/X R6MZYigCV3IoN4LlXhkv6BIuCu84+9Nk8sczes0M6iyGJth/SC9m7epmMS0B3eSpXv7C b5/Q+6IyjSpX529yxubas1woHZUhkCQOe/PCvMKCFDkzo/wgxlyD/JmmMOwkzGcMHKZZ HRLkONAv8zehNIjnB4T2eUYiqtrz8xFRNFEGELKIrrTx2YQ/I06MMPRqe/EjKEU/Rjgl sWDIq+zBmhLBu3kn1ZsZlN9mJJASWCB+px86hJli5XhCmzC+1U17HOq/X5RV4aotpY4o kEkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TOY4Ejzg/9nAKzPYo9tETGRaOoSPvAWifxplSzBGXYw=; b=Lsu/3qzdDq3zk8uBW8Gv5qjP5rAA/KJvRh7bf6cbRRjP6NIfXhYP8EnOn9qw2ztvq9 hSqID7ZHW8VW+NS48CpDrn+1T3G6QUXtdlCoSaZQwjaO/bvp9+uP2jAz5/igT3WJgCfF +06tbmeMUhd3kBinJrLR9e1NMsCKimOi2GXvFenvMScNupE0G0o9d0Kx4WRbtr/GbInB T0gPoDjnHiNE3+Ot81J/Vk/sLBjfOYcbviDdgD1ZLwCvFq0rBpE23t7NQmYC0J4gIkRn vORZ/iLjfOAd/UxCm3XwaZofpZm77+wSOX6w+nlIsmOr5xofGyAAogPJAwqh47DP4fPt OIVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cDvtWPvf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ix17-20020a170902f81100b00174b83659fasi21376605plb.502.2022.12.08.02.49.58; Thu, 08 Dec 2022 02:50:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cDvtWPvf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230292AbiLHKtj (ORCPT + 99 others); Thu, 8 Dec 2022 05:49:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230125AbiLHKsq (ORCPT ); Thu, 8 Dec 2022 05:48:46 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9530D554FA for ; Thu, 8 Dec 2022 02:44:07 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id c17so1492959edj.13 for ; Thu, 08 Dec 2022 02:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TOY4Ejzg/9nAKzPYo9tETGRaOoSPvAWifxplSzBGXYw=; b=cDvtWPvf3S8NQ0fSkfns/Z6fkpvUbHGNiMlgP5rHJmAIJJ3gdhSwr+Ct+fE6bKpMyH Kyowaj7BdY6BUilN+ffaZ4FZr/895aMTTysEHE517g0mCsydJxDpfHxD9/gN9fpwmHHY o+vfejQrND/Vc2WPHgebfl+3y0sLQHLoW+kaR+PRnkEL2cnIpjpneq9jEDLcvYLcl4X9 kwiMJAQI1/WsxSUi1sGBOgRdwADaoQQpKB6Lm8f7Lp8iCMrTd24/rsv04FdEM30l/Gba QZCZtWtShZLblG4OfO9JO2qGGQc7en+24D3ol/7+Fme1C+wxJqzYNEDswr2uO/J8WMZz 2w+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TOY4Ejzg/9nAKzPYo9tETGRaOoSPvAWifxplSzBGXYw=; b=40viGIg4JAIT/BBSKMaZcKnVzHwvclpUHUlL3FjYtmIyKSOhQuUA5XgPRHHlBnkaW3 1xiynBGIivQQdNvT93IAGRwI77pWXwguj8YhMkrD6px+8Jzu7KNIZt/ZkDpRf3EEMIki 8E2gMWCDMkqUP+iurdM1EbbeaDiI6H/Pco4/6k+rhyyPfkSpQ4+SkfvJHYSUT1PJ7JqV g9fWo5xs+2JP65lpVdxhZlNAxDY8gJGQhLSeEE9soJMsln81XRQwCiZG1qIjzy71y7Ha zsUry/Y1IxQEoloEvAauXlc8F5PX1uvO/Xn/9bavM95br1gmsW8U20DEW7VqL+0LY/R1 1HMQ== X-Gm-Message-State: ANoB5pkJoKMAWj366bDIWztRjOFQFV71vEdUBrTMYaEGBVZgTwWarEaW MlF8PKs5tejoZDMsxb20lIOulw== X-Received: by 2002:a05:6402:2946:b0:468:febe:ebab with SMTP id ed6-20020a056402294600b00468febeebabmr37209892edb.337.1670496245540; Thu, 08 Dec 2022 02:44:05 -0800 (PST) Received: from MBP-di-Paolo.station (net-2-35-55-161.cust.vodafonedsl.it. [2.35.55.161]) by smtp.gmail.com with ESMTPSA id fe17-20020a1709072a5100b0077a8fa8ba55sm9544193ejc.210.2022.12.08.02.44.04 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2022 02:44:05 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, arie.vanderhoeven@seagate.com, rory.c.chen@seagate.com, glen.valante@linaro.org, Davide Zini , Paolo Valente Subject: [PATCH V9 8/8] block, bfq: balance I/O injection among underutilized actuators Date: Thu, 8 Dec 2022 11:43:51 +0100 Message-Id: <20221208104351.35038-9-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221208104351.35038-1-paolo.valente@linaro.org> References: <20221208104351.35038-1-paolo.valente@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751642655328161364?= X-GMAIL-MSGID: =?utf-8?q?1751642655328161364?= From: Davide Zini Upon the invocation of its dispatch function, BFQ returns the next I/O request of the in-service bfq_queue, unless some exception holds. One such exception is that there is some underutilized actuator, different from the actuator for which the in-service queue contains I/O, and that some other bfq_queue happens to contain I/O for such an actuator. In this case, the next I/O request of the latter bfq_queue, and not of the in-service bfq_queue, is returned (I/O is injected from that bfq_queue). To find such an actuator, a linear scan, in increasing index order, is performed among actuators. Performing a linear scan entails a prioritization among actuators: an underutilized actuator may be considered for injection only if all actuators with a lower index are currently fully utilized, or if there is no pending I/O for any lower-index actuator that happens to be underutilized. This commits breaks this prioritization and tends to distribute injection uniformly across actuators. This is obtained by adding the following condition to the linear scan: even if an actuator A is underutilized, A is however skipped if its load is higher than that of the next actuator. Signed-off-by: Paolo Valente Signed-off-by: Davide Zini --- block/bfq-iosched.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 16230a9cf794..016b1b73aeb2 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -4831,10 +4831,16 @@ bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, int idx) /* * Perform a linear scan of each actuator, until an actuator is found - * for which the following two conditions hold: the load of the - * actuator is below the threshold (see comments on actuator_load_threshold - * for details), and there is a queue that contains I/O for that - * actuator. On success, return that queue. + * for which the following three conditions hold: the load of the + * actuator is below the threshold (see comments on + * actuator_load_threshold for details) and lower than that of the + * next actuator (comments on this extra condition below), and there + * is a queue that contains I/O for that actuator. On success, return + * that queue. + * + * Performing a plain linear scan entails a prioritization among + * actuators. The extra condition above breaks this prioritization and + * tends to distribute injection uniformly across actuators. */ static struct bfq_queue * bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) @@ -4842,7 +4848,9 @@ bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) int i; for (i = 0 ; i < bfqd->num_actuators; i++) { - if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold) { + if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold && + (i == bfqd->num_actuators - 1 || + bfqd->rq_in_driver[i] < bfqd->rq_in_driver[i+1])) { struct bfq_queue *bfqq = bfq_find_active_bfqq_for_actuator(bfqd, i);