From patchwork Tue Jan 17 17:54:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 44773 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1911095wrn; Tue, 17 Jan 2023 10:20:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXtnSFZOJEN2XaXYVfs1jAn+Al9lN6X3Z0ItcT1Y4NL47GORqTsrJN+KMxOpaXImemxDqzpt X-Received: by 2002:a17:90a:dac5:b0:229:502f:3910 with SMTP id g5-20020a17090adac500b00229502f3910mr4134793pjx.11.1673979650921; Tue, 17 Jan 2023 10:20:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673979650; cv=none; d=google.com; s=arc-20160816; b=J8XErfMQwsYWMp36rkKkPIhMrqn1qhTRFNxFI34Dui5lYhHT2Q8xr9P4zOCPHOzYSp eNs6vkVuIMFIBTbPUR3gpnjIwyk94LBF6S4hTxnK1XKVMApJTetKWG8VdISMvOUpKNeA D+CBVVT5NBIQTYS3qOBY08io+clENT5FRB3bQPAFSDcc11WOVq7xNVOdvPEZDPkp6ws0 SJoPtXeFQXiKBCGsu3yF/ex9nf7KHK3sDOgl5TU9FmDZyigpDkgyTurxpRip8/U7s66i +l9Y+ezLkAV6aok09fWtZx5oBMSEylTpoln/j/3O/cFLdQhYRZfQDNg3UxzYSAZDFrV1 LyXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=PQcgs/gZzBl+B7TsSGFyPNXbOKfIcXW+ewk0g5LuBAs=; b=VSE7a70F9Zd3gM2BCJQxQB67FuF/Urro9sRm45du9tS2FkBdA2kvQ3nVXTeeVGAd6w rUtWsw1ZBIFFok8pynByisYnMulsrwKYL5+ch05EhxTKrq6pVoe277tmMHru4P2WSHAf pxxeRk8w1ZstZPjdJT0LjCrV+P3orTWGhxuffG3NM48t23RjKaXmS+bOh+RsZCrf6dUY PpnyaReNqvgzg2/YYdjwUIiLZSuQbhIqgj4MR1h/x2ZTtGq+PabeQ1IepOjhfWW+ZNy0 GEXZoiLhbvL8Dv5lRPOwizuHgqzPZusSGNEghkXwlb9m32/vfo7yF42SgvTdprWhBoIK N0DQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FebkwlXy; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Pe4ejAGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qi2-20020a17090b274200b0022915c0c7e2si14972440pjb.64.2023.01.17.10.20.30; Tue, 17 Jan 2023 10:20:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FebkwlXy; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Pe4ejAGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229841AbjAQSQX (ORCPT + 99 others); Tue, 17 Jan 2023 13:16:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233148AbjAQSOE (ORCPT ); Tue, 17 Jan 2023 13:14:04 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3105959E42; Tue, 17 Jan 2023 09:55:00 -0800 (PST) Date: Tue, 17 Jan 2023 17:54:56 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1673978097; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQcgs/gZzBl+B7TsSGFyPNXbOKfIcXW+ewk0g5LuBAs=; b=FebkwlXyNt/QvcRlqP2e2ODF801LwQvy8pLHm8+5rYd+GUBTR8EpwhgYXkk6i1gkoj20U2 JwmSwDQW8hJgeFtgOSmbrcRWmKc4f9Tscn/RQpPjL0YzK0j4aM5LvSwtVGG0Dlweq8Sj3g eCNtLWKzpZJsxichtENTxeJAmEIcayVjnfaBON5o6ysUiVb/x2eUrQayS2YOUHoNxNntPi f3JS7d7/7+TmzujPzCXVvwd/cEWyQmLJiYn6/3tDGF//2bzVrJP6PkDKOLOCD7LuWab3uS 0cDjwRmxQqQpd+uPpCHz6IdwdZk2sqbYUTfRtYZnRJaaLuLYqB8vc5+nbKSuYw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1673978097; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQcgs/gZzBl+B7TsSGFyPNXbOKfIcXW+ewk0g5LuBAs=; b=Pe4ejAGe7ctTLneORQMPSIfvAlhqga7TqaZ+dLqhAp/T8W2t3po691JVAXtYHUUrtJsWuy 8gO1XNYGsO9YZ9Cw== From: "tip-bot2 for Ming Lei" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: irq/core] blk-mq: Build default queue map via group_cpus_evenly() Cc: Ming Lei , Thomas Gleixner , Christoph Hellwig , John Garry , Jens Axboe , x86@kernel.org, linux-kernel@vger.kernel.org, maz@kernel.org In-Reply-To: <20221227022905.352674-7-ming.lei@redhat.com> References: <20221227022905.352674-7-ming.lei@redhat.com> MIME-Version: 1.0 Message-ID: <167397809674.4906.6016765295947842293.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753332607303615231?= X-GMAIL-MSGID: =?utf-8?q?1755294886160012803?= The following commit has been merged into the irq/core branch of tip: Commit-ID: 6a6dcae8f486c3f3298d0767d34505121c7b0b81 Gitweb: https://git.kernel.org/tip/6a6dcae8f486c3f3298d0767d34505121c7b0b81 Author: Ming Lei AuthorDate: Tue, 27 Dec 2022 10:29:05 +08:00 Committer: Thomas Gleixner CommitterDate: Tue, 17 Jan 2023 18:50:06 +01:00 blk-mq: Build default queue map via group_cpus_evenly() The default queue mapping builder of blk_mq_map_queues doesn't take NUMA topo into account, so the built mapping is pretty bad, since CPUs belonging to different NUMA node are assigned to same queue. It is observed that IOPS drops by ~30% when running two jobs on same hctx of null_blk from two CPUs belonging to two NUMA nodes compared with from same NUMA node. Address the issue by reusing group_cpus_evenly() for building queue mapping since group_cpus_evenly() does group cpus according to CPU/NUMA locality. Also performance data becomes more stable with this given correct queue mapping is applied wrt. numa locality viewpoint, for example, on one two nodes arm64 machine with 160 cpus, node 0(cpu 0~79), node 1(cpu 80~159): 1) modprobe null_blk nr_devices=1 submit_queues=2 2) run 'fio(t/io_uring -p 0 -n 4 -r 20 /dev/nullb0)', and observe that IOPS becomes much stable on multiple tests: - unpatched: IOPS is 2.5M ~ 4.5M - patched: IOPS is 4.3M ~ 5.0M Lots of drivers may benefit from the change, such as nvme pci poll, nvme tcp, ... Signed-off-by: Ming Lei Signed-off-by: Thomas Gleixner Reviewed-by: Christoph Hellwig Reviewed-by: John Garry Reviewed-by: Jens Axboe Link: https://lore.kernel.org/r/20221227022905.352674-7-ming.lei@redhat.com --- block/blk-mq-cpumap.c | 63 ++++++++---------------------------------- 1 file changed, 13 insertions(+), 50 deletions(-) diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c index 9c2fce1..0c612c1 100644 --- a/block/blk-mq-cpumap.c +++ b/block/blk-mq-cpumap.c @@ -10,66 +10,29 @@ #include #include #include +#include #include #include "blk.h" #include "blk-mq.h" -static int queue_index(struct blk_mq_queue_map *qmap, - unsigned int nr_queues, const int q) -{ - return qmap->queue_offset + (q % nr_queues); -} - -static int get_first_sibling(unsigned int cpu) -{ - unsigned int ret; - - ret = cpumask_first(topology_sibling_cpumask(cpu)); - if (ret < nr_cpu_ids) - return ret; - - return cpu; -} - void blk_mq_map_queues(struct blk_mq_queue_map *qmap) { - unsigned int *map = qmap->mq_map; - unsigned int nr_queues = qmap->nr_queues; - unsigned int cpu, first_sibling, q = 0; - - for_each_possible_cpu(cpu) - map[cpu] = -1; - - /* - * Spread queues among present CPUs first for minimizing - * count of dead queues which are mapped by all un-present CPUs - */ - for_each_present_cpu(cpu) { - if (q >= nr_queues) - break; - map[cpu] = queue_index(qmap, nr_queues, q++); + const struct cpumask *masks; + unsigned int queue, cpu; + + masks = group_cpus_evenly(qmap->nr_queues); + if (!masks) { + for_each_possible_cpu(cpu) + qmap->mq_map[cpu] = qmap->queue_offset; + return; } - for_each_possible_cpu(cpu) { - if (map[cpu] != -1) - continue; - /* - * First do sequential mapping between CPUs and queues. - * In case we still have CPUs to map, and we have some number of - * threads per cores then map sibling threads to the same queue - * for performance optimizations. - */ - if (q < nr_queues) { - map[cpu] = queue_index(qmap, nr_queues, q++); - } else { - first_sibling = get_first_sibling(cpu); - if (first_sibling == cpu) - map[cpu] = queue_index(qmap, nr_queues, q++); - else - map[cpu] = map[first_sibling]; - } + for (queue = 0; queue < qmap->nr_queues; queue++) { + for_each_cpu(cpu, &masks[queue]) + qmap->mq_map[cpu] = qmap->queue_offset + queue; } + kfree(masks); } EXPORT_SYMBOL_GPL(blk_mq_map_queues);