From patchwork Tue Feb 27 21:06:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 207434 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2968467dyb; Tue, 27 Feb 2024 13:13:24 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXJx+JGaL5mfm0rQzdB7SzSR6rOqJpRULki7oqWZGDmKH5hTJ1mIuTC5ELmwWK4lZKR+E9k/QY633S0haL+lr3aq8E3mw== X-Google-Smtp-Source: AGHT+IGjsRp0Cc3bp0wHZHGXGNpex9sxfK+rwLwg2nEWK2J1IcqMwEAN/qCOq9AvR2Wli5MeVDih X-Received: by 2002:a05:6e02:de4:b0:365:1a88:f654 with SMTP id m4-20020a056e020de400b003651a88f654mr11279774ilj.28.1709068404118; Tue, 27 Feb 2024 13:13:24 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709068404; cv=pass; d=google.com; s=arc-20160816; b=fPb/JhAjTvux/jb5aHpzT+kr8KbTZkGB8ErKALF3peHpAlC3++FALwbuu/E4m60CAI Bv6XWXxZ92+5O+kmgMLWUgNiDL/KUdyTYZcGemxC+Ur/RROmISCyFOGtuYW3yI5ExCrZ 4SaplzslTzxOmFbRISz6ys+E8ZavVDCKXJbZbYVBi0vs0cl7/I3Wi7lhEdLPJ5Jae4KN 1Cdh1IvucZVpkZjwMym26Gt9nKRZZ0hmZr5ZK17aalBihU3zBy1RSyqjfah07LPQEdeF fFbYpxWGo2oBysQglnU0A2bNlK7YnoRp7hxFcZFew04EiIJa0/1jeLTAbFTS+VezZSZU cpqQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=/RAeKNhAgZmKkiugFg+etSPi3yLpie0L6GcI/HOOxDk=; fh=QLeERbPuyyXG4VUTEwvqfDOsG6ebgMl25C7foFloUoY=; b=aZtnUTnEHBXegPojO/tSEoVE0qJGJ3o45uA7rrTOPUxLzyZ+zZlsQSB9LD4QpLePZk TZ21aHuXfbbCymxkcaxmKzA5tqLXVKuIEjga3Kg0WcoxGmuS4lMSZRf4bPIHYtRhfNIr mkUyLBu8BDGCrQFD2azIyiEzkvFdbSiJOLtdINdRT4cc3egQRrkSyJqnjWFLhOM8aJn0 OXmSG0jheV454+pu8M2mjpAcjMvrxWccE/2cCOfURWsrQaK7fZFA98ogcZhw2rP0qAoC XJQqaG3m7FEO2FueXo94wA0cCgZWoksvFWeaOYDxipq297IcmdajnExaluHaEjVOzG07 MCXQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=rvqQUTMi; arc=pass (i=1 spf=pass spfdomain=kernel.dk dkim=pass dkdomain=kernel-dk.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-84017-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84017-ouuuleilei=gmail.com@vger.kernel.org" Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id ln17-20020a056a003cd100b006e468fd088bsi6152703pfb.7.2024.02.27.13.13.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 13:13:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-84017-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=rvqQUTMi; arc=pass (i=1 spf=pass spfdomain=kernel.dk dkim=pass dkdomain=kernel-dk.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-84017-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84017-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3473A285B04 for ; Tue, 27 Feb 2024 21:12:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 607F9150982; Tue, 27 Feb 2024 21:12:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="rvqQUTMi" Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5801F14EFE6 for ; Tue, 27 Feb 2024 21:11:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709068322; cv=none; b=UEGsqxh5iEAryo6517qujxo03njjQw3oDVDWpI5LMbgZjzkuy46xv5ghEE6zQX3HvYeYTUL5C06YEFnq6x8ywfdKiqX+eHD49lYYuqqExRA41B2ia6B75IF6z+DT/NxlWlC4OhhwZFJIc8TCBNgZ9sdFGgK2Fvg8Bhm5wiADaZ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709068322; c=relaxed/simple; bh=JKIl3QSWViM2CannFhQlXB4MJVpGyv7AV+fHXS552Io=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V3WxZ5e0hM7E2C99jPx71QGfIBt6y+VDKM5ktFRqCfNRxMBeZRFh8aKyJ4Dsmz2RmVXgrYghG3Si9LZMHMxPAf1Db3aXLqytgYmqHZ/w2ZLIQNKU05qiyrA7u53vOVe7nanu2RLNT7+jNjm/993fGGH8Y6e9iGnXpMPdpO/TZFY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=rvqQUTMi; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-55b5a37acb6so1830146a12.0 for ; Tue, 27 Feb 2024 13:11:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709068318; x=1709673118; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/RAeKNhAgZmKkiugFg+etSPi3yLpie0L6GcI/HOOxDk=; b=rvqQUTMiw48RQbW03QNOppu9UIG/0lvLX56qorKc8JiCKnYThga7ToxPjcgfwjtUQq xXYoQ4jiFWULU+sozAKGjBc9ta1IVj3l9EEZe4Y7godGuDeQfnc0/J2RnaMIL0Jpfnzg 61pIF7feXGppHS0WFpTu0zx3OrZNFDjL7Tj7ie/EoMlXYZkcg7HUp0JPBV6KRL95BW5t qxDfIBAmnWeYzOKo6MKAiJkUrNUDnMKCve6BQMhZgDQuUrzpcS1W7gZRT3S8u6jrFQKC E1zFW7PBSPJJsvQ9VedEp6ReB6tCp0fToHDdWfAHzD7Of7/neuiRLVB+vCCv+UbmfR0Q ViIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709068318; x=1709673118; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/RAeKNhAgZmKkiugFg+etSPi3yLpie0L6GcI/HOOxDk=; b=h8O62VTLzuycL3iHrYi5d7B2i2cwU0J6ylRsDnm9ofIjeJuyrLsUhunsSSguqahj/q OmJ9bxxbHUXM3sWaXxOhcuO+f+yr1fTlyu4tRlE0hn8VE0WRSb1MIPQGHj5HwQqr6WrM Ze1rv+e2MUZwFNqgBvjsGDl3xxL5LP6zuhxR1pCfh4GTG0tE3JRHTxfAfK+dnjOdfqgb /Pe8yB3R2rySjgzwS3x85tQCa42LCSdJDA3CRQTngrCfKCNxkP87cpPecPsZ7KA48Dyr XyYrM6Z+Ex474aqBcehvAe3gTtOSQQeVTFsyAzQ1R3UfaVwr15Mpa33FHiIdqBc7BjMF 5jbA== X-Gm-Message-State: AOJu0Yz9Pm56VmU4fBavKHd9YAygnXnv6/PXtmunkw8RPpBZW9HxYcpx Iejz85OzBBe0/EkbQmRFtAzWnxcQBmdyLSi/QABDSS7uxf/R2cSZXNoJGhOBE8ivGEZfDdpF4JP U X-Received: by 2002:a05:6a00:1384:b0:6e5:4abe:fd4c with SMTP id t4-20020a056a00138400b006e54abefd4cmr2833506pfg.3.1709068318162; Tue, 27 Feb 2024 13:11:58 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u8-20020a056a00098800b006e5557128efsm735390pfg.133.2024.02.27.13.11.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 13:11:57 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, mingo@redhat.com, Jens Axboe Subject: [PATCH 1/2] sched/core: switch struct rq->nr_iowait to an unsigned int Date: Tue, 27 Feb 2024 14:06:02 -0700 Message-ID: <20240227211152.1099534-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227211152.1099534-1-axboe@kernel.dk> References: <20240227211152.1099534-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792088111011746735 X-GMAIL-MSGID: 1792088111011746735 In 3 of the 4 spots where we modify rq->nr_iowait we already hold the rq lock, and in the 4th one we can just grab it. This avoids an atomic in the scheduler fast path if we're in iowait, with the tradeoff being that we'll grab the rq lock for the case where a task is scheduled out in iowait mode on one CPU, and scheduled in on another CPU. This obviously leaves the reading side as potentially racy, but that should be OK. iowait states change all of the time, and can change while it's being read as well, or summed up. Signed-off-by: Jens Axboe --- kernel/sched/core.c | 15 ++++++++++----- kernel/sched/cputime.c | 2 +- kernel/sched/sched.h | 2 +- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9116bcc90346..ecc6c26096e5 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3789,7 +3789,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, #endif if (p->in_iowait) { delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); + task_rq(p)->nr_iowait--; } activate_task(rq, p, en_flags); @@ -4354,8 +4354,13 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU); if (task_cpu(p) != cpu) { if (p->in_iowait) { + struct rq *rq = task_rq(p); + struct rq_flags rf; + + rq_lock(rq, &rf); + task_rq(p)->nr_iowait--; + rq_unlock(rq, &rf); delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); } wake_flags |= WF_MIGRATED; @@ -5463,7 +5468,7 @@ unsigned long long nr_context_switches(void) unsigned int nr_iowait_cpu(int cpu) { - return atomic_read(&cpu_rq(cpu)->nr_iowait); + return cpu_rq(cpu)->nr_iowait; } /* @@ -6681,7 +6686,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); if (prev->in_iowait) { - atomic_inc(&rq->nr_iowait); + rq->nr_iowait++; delayacct_blkio_start(); } } @@ -10029,7 +10034,7 @@ void __init sched_init(void) #endif #endif /* CONFIG_SMP */ hrtick_rq_init(rq); - atomic_set(&rq->nr_iowait, 0); + rq->nr_iowait = 0; #ifdef CONFIG_SCHED_CORE rq->core = rq; diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index af7952f12e6c..b970b6c6151e 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -224,7 +224,7 @@ void account_idle_time(u64 cputime) u64 *cpustat = kcpustat_this_cpu->cpustat; struct rq *rq = this_rq(); - if (atomic_read(&rq->nr_iowait) > 0) + if (rq->nr_iowait) cpustat[CPUTIME_IOWAIT] += cputime; else cpustat[CPUTIME_IDLE] += cputime; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 001fe047bd5d..a1222a4bdc7b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1049,7 +1049,7 @@ struct rq { u64 clock_idle_copy; #endif - atomic_t nr_iowait; + unsigned int nr_iowait; #ifdef CONFIG_SCHED_DEBUG u64 last_seen_need_resched_ns;