Message ID | 20230503080258.14525-10-dwagner@suse.de |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1153111vqo; Wed, 3 May 2023 01:05:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ53kXs0bLzLF83FF8kGNyuYNMYRb2CPlucNyw2LUesB+ceyNlntxj7PG6TF+Am50bg7GlaP X-Received: by 2002:a05:6a20:3c8b:b0:ef:6e5a:8b1e with SMTP id b11-20020a056a203c8b00b000ef6e5a8b1emr24437590pzj.24.1683101131141; Wed, 03 May 2023 01:05:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683101131; cv=none; d=google.com; s=arc-20160816; b=vV1i0rOGUsta8gA9MNL2+rX7IjE6JhN93xclQ9ZKji56bAA3EiUUeX1KV6h8G5/wfs oMm1w7vwDd3djWoYbyMIWWsypFVjYIMFMY73dD4e4Oco3AWeFAKN1t9qNl7lTLMIOE6+ JmriId9DuO4kkfD8QDF+QA+gztvBk6PV7GenjHRbwxRdy0hVyLLjcALXuY4dg0cuhIIi J2bJa32VajuCr/+v6YkDKuigwVbx5QbzbNrKdl/gVNRdWnN1v4I/5hOQSndI33+C/5IZ UviYkRuCkX2FEhsz1Uiz+WuSnjBWJ/6p4JMMh90enTfD+YBi7pht6yGZYX6dQOqoLn0g jlag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=e2pq1wjnhWpzpAUzywg7yjYCKpUJJXlcUeZCAZuKjeQ=; b=BRcHLpLYcJQHreF9eJQShiPmuf37UQ8VpwPZfthWq1iQ+kG3UrAF80WAIXFxKTFp+6 r2ve8pesvx4iIMYhjIc3U8tFgM2W1qpBkYIG0e1kpvGHK+tRqEev/9vYh5h9taZdTAxU hM/6XrGeUGcWGNgPtk+GFdUk975QaF2CzQnFzVhaR9FOBnOAWN+CF/JpaOpRQZ254bGw vcDxrzNz8Q2jWx4PPJ5UZsiQIubQhwIL4s8WQ8ek7DcBDmbeEvGTLRnbAm/GxBq5r+sY LjXvoh09W0/YQHsFmuFnR53BUOkhhuT7y42ssPmHDjoyGK0Z2CPX5fK4EXKXRlBt2Hp+ szYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=ajMzP1vc; dkim=neutral (no key) header.i=@suse.de header.b=Fdy1nUpx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r7-20020a63e507000000b00518580cb7b8si17530193pgh.269.2023.05.03.01.05.18; Wed, 03 May 2023 01:05:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=ajMzP1vc; dkim=neutral (no key) header.i=@suse.de header.b=Fdy1nUpx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229522AbjECIEE (ORCPT <rfc822;heyuhang3455@gmail.com> + 99 others); Wed, 3 May 2023 04:04:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229730AbjECIDT (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 3 May 2023 04:03:19 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A733646AE; Wed, 3 May 2023 01:03:16 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 3D6B4223FE; Wed, 3 May 2023 08:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683100995; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e2pq1wjnhWpzpAUzywg7yjYCKpUJJXlcUeZCAZuKjeQ=; b=ajMzP1vczLO7G7jIFOc9NiidYl3WFlbFEYBsoWk5mT4U81vaYTbjZr9pcOOAfYA5/Wnv8n gF9mfH3QzZ7eqmT/3/XLXcCx9W5GVaars2uGz/GGATDQq6hjZmjYuzdhQ3Je48S7Eq3jCT uDZ5XBFtdofwoW/HsWrSRNr3T6fr1CE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683100995; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e2pq1wjnhWpzpAUzywg7yjYCKpUJJXlcUeZCAZuKjeQ=; b=Fdy1nUpx1DqPaJ5pDiwa7wy5yJ4yRGpqmlpU+rtEVOjvLbIo3wBU/xMLMqZWOjUkcxeUOP 6NKxwyuyAGhxFrBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2F66A1331F; Wed, 3 May 2023 08:03:15 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id bwWbC0MVUmRDYgAAMHmgww (envelope-from <dwagner@suse.de>); Wed, 03 May 2023 08:03:15 +0000 From: Daniel Wagner <dwagner@suse.de> To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, Chaitanya Kulkarni <kch@nvidia.com>, Shin'ichiro Kawasaki <shinichiro@fastmail.com>, Hannes Reinecke <hare@suse.de>, Daniel Wagner <dwagner@suse.de> Subject: [PATCH blktests v3 09/12] common/fio: Limit number of random jobs Date: Wed, 3 May 2023 10:02:55 +0200 Message-Id: <20230503080258.14525-10-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230503080258.14525-1-dwagner@suse.de> References: <20230503080258.14525-1-dwagner@suse.de> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764859451634141357?= X-GMAIL-MSGID: =?utf-8?q?1764859451634141357?= |
Series |
nvme testsuite runtime optimization
|
|
Commit Message
Daniel Wagner
May 3, 2023, 8:02 a.m. UTC
Limit the number of random threads to 32 for big machines. This still
gives enough randomness but limits the resource usage.
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
common/fio | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Comments
On 5/3/23 01:02, Daniel Wagner wrote: > Limit the number of random threads to 32 for big machines. This still > gives enough randomness but limits the resource usage. > > Signed-off-by: Daniel Wagner <dwagner@suse.de> > --- I don't think we should change this, the point of all the tests is to not limit the resources but use threads at least equal to $(nproc), see recent patches from lenovo they have 448 cores, limiting 32 is < 10% CPUs and that is really small number for a large machine if we decide to run tests on that machine ... -ck
On Wed, May 03, 2023 at 09:41:37AM +0000, Chaitanya Kulkarni wrote: > On 5/3/23 01:02, Daniel Wagner wrote: > > Limit the number of random threads to 32 for big machines. This still > > gives enough randomness but limits the resource usage. > > > > Signed-off-by: Daniel Wagner <dwagner@suse.de> > > --- > > I don't think we should change this, the point of all the tests is > to not limit the resources but use threads at least equal to > $(nproc), see recent patches from lenovo they have 448 cores, > limiting 32 is < 10% CPUs and that is really small number for > a large machine if we decide to run tests on that machine ... I just wonder how handle the limits for the job size. Hannes asked to limit it to 32 CPUs so that the job size doesn't get small, e.g. nvme_img_size=16M job size per job with 448 CPUs is roughly 36kB. Is this good, bad or does it even make sense? I don't know. My question is what should the policy be? Should we reject configuration which try to run too small jobs sizes? Reject anything below 1M for example? Or is there a metric which we could as base for a limit calculation (disk geometry)?
On 5/3/23 04:01, Daniel Wagner wrote: > On Wed, May 03, 2023 at 09:41:37AM +0000, Chaitanya Kulkarni wrote: >> On 5/3/23 01:02, Daniel Wagner wrote: >>> Limit the number of random threads to 32 for big machines. This still >>> gives enough randomness but limits the resource usage. >>> >>> Signed-off-by: Daniel Wagner <dwagner@suse.de> >>> --- >> I don't think we should change this, the point of all the tests is >> to not limit the resources but use threads at least equal to >> $(nproc), see recent patches from lenovo they have 448 cores, >> limiting 32 is < 10% CPUs and that is really small number for >> a large machine if we decide to run tests on that machine ... > I just wonder how handle the limits for the job size. Hannes asked to limit it > to 32 CPUs so that the job size doesn't get small, e.g. nvme_img_size=16M job > size per job with 448 CPUs is roughly 36kB. Is this good, bad or does it even > make sense? I don't know. 16M is very small number .. from my experience with smaller I/O sizes we don't see the lokdeps that we see with the large I/O sizes hence it is a bad idea to use small I/O sizes and limiting the jobs to hard coded 32 number ... > My question is what should the policy be? Should we reject configuration which > try to run too small jobs sizes? Reject anything below 1M for example? Or is > there a metric which we could as base for a limit calculation (disk geometry)? the basic requirement here is we need to run the I/O from every processor, so let's keep --numjobs=($nproc) constant now and let the user set job size.. in this particular case for NVMe we set the size 1G and that is sufficient since numbjobs are set to nproc and with this series user can set the size based on a particular arch ... See [1] if you are interested in how to quantify small or large job size. For this series to merge let's keep is simple and not worry about erroring out on a particular job size but just keeping the nproc as it is ... -ck Ideally in past what I've done is :- 1. Accept the % of the CPU cores that we want to keep it busy. 2. Accept the % of the disk space we want to exercise test. 3. Use the combination of the #1 and #2 to spread out the job size across the number of jobs. with above design one doesn't have to assume what is small or what it large job size and system gets tested according to user's expectations such as 50% CPUs are busy on 80% disk size or 100% CPUs are busy with 50% of disk size.
On Thu, May 04, 2023 at 05:16:34AM +0000, Chaitanya Kulkarni wrote: > For this series to merge let's keep is simple and not worry about erroring > out on a particular job size but just keeping the nproc as it is ... Works for me and as you pointed out, it avoids regressions with the default values. Anyone who is tinkering with nvme_img_size should be aware of the implications. I'll add a note about this to the documentation.
diff --git a/common/fio b/common/fio index 1db6128be632..06659d7d1d84 100644 --- a/common/fio +++ b/common/fio @@ -189,7 +189,8 @@ _run_fio() { # Wrapper around _run_fio used if you need some I/O but don't really care much # about the details _run_fio_rand_io() { - _run_fio --bs=4k --rw=randread --norandommap --numjobs="$(nproc)" \ + _run_fio --bs=4k --rw=randread --norandommap \ + --numjobs="$(OMP_THREAD_LIMIT=32 nproc)" \ --name=reads --direct=1 "$@" }