From patchwork Sat Feb 17 01:27:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202554 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93992dyc; Fri, 16 Feb 2024 17:29:29 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVzd4J4Z2HYkBLq4bWwb+87dSpkU+dKln1XRPxC6r/5U8+2gFwUiZXPi/YLCjvCnhcSTIkxl9hQtwcp7Jf5DLyHE0D2FA== X-Google-Smtp-Source: AGHT+IFKOSDalHMqbVP7piSgP1u3p+FBbz1CWxsBtfmFPbpSqelX9Tx7lH3yC7pxK8+96BoNPgRl X-Received: by 2002:a17:902:d48e:b0:1db:403c:68b5 with SMTP id c14-20020a170902d48e00b001db403c68b5mr7731714plg.12.1708133369800; Fri, 16 Feb 2024 17:29:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133369; cv=pass; d=google.com; s=arc-20160816; b=IMKrbRuPOZl5MUf2ETcDUsP2X7hUUGf33eqcR38pBtipfcn8Wr9YbzmsQTafH6uOEs SbvvHtre6/NMYGGMECkHxSuGykNfW09t4uEtMLJC+qdBcwp9sJ+ZLuSO8iBceTNI6S8B rxX9MiusiM+IO4wiRHnz92cV8XeR/EUksTEzr+PHOnXQkXTAClVkO8PJMRV4uLuM7DOL TLCpRaR3cG2p7E6MUwM769ASVDPz5GOfUy2qEPFX2G1Bb4V+g8vXNMMNFMoCs8PyF4u1 ICBJt+1p22fpeCuMdFp4lGDKUKQsSGdxzndN+9z2tBjJcdhzkisEch3paUORtJwCu0IQ 9mJg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; fh=9Qg1YJOEWMF7QEtQ5YYQgQW0vfRLI3VVtBnP5xjwM30=; b=Q5QWN/bmM/zaeL6XFn/qcyJeT+gLDQh9Fcu45AJzxUMqQh4r5LzaMiFpTVAIFpXrC2 lJrGIwFSya/NKFhvlXGs4m+QwfG6MQh7bHI/TJ54gs4t4b5MjK5ALMOlVTP0Mood9Yg3 3bT3vc9rzJPaz0S5+b+bJ+DDYLqNgmfuR7Qm3BxOfhRqp+Wud/FCu7JAxWKxnTVznQ6Z hMawuMiQFoxIqutkyDsk2ZJoAiqYHqYil8RvgaHPwc3gyI0fEpgIU2DpPaqHWRTY1ce1 jxzz6lWpcrzSw9uy2MugsPpagYltGbMxxct5SxYUPdMJISKrmk+8xlVoRfSaMRWM+V+A b/8w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=T+stnbu8; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id n8-20020a170902f60800b001db535f1530si683872plg.419.2024.02.16.17.29.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:29:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=T+stnbu8; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 8E897284605 for ; Sat, 17 Feb 2024 01:29:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 50E211D55F; Sat, 17 Feb 2024 01:28:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T+stnbu8" Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 930521CD1F; Sat, 17 Feb 2024 01:28:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133291; cv=none; b=cb3qYWaFTahpJxShmiELn/k1Q50AEN+QgH8aywRU+cZ2WyXG/yjJXU8KZJEJ6gyjZpzVisPtAVg6lmLfq14mLp3zWqlPxgymHAB/Pia5FMLhcjCEgi7hlviz13M1mhvViR8okgGSq5X9aUoE7LTyjehklsbx6uGM7AV9WLfBXko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133291; c=relaxed/simple; bh=KLY1+2mGq3H9diWHchRwxDC5ACzJHAgMPqkliWw0HYE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FeFsxln59LthlOtK5mn2FHdwSCGtg/1zVmajFkWNnB376sNRq/zij2BlgJk93hpVAyNrHb7+vMbF28sQPdfewp5MvlCH3A0HbaV22kk1t5ZPFqQ3bd2WSGpJMLINheVv7a3lPkIBK74kq5hHVjjT4kEck+qfOOe9Sqc0S28LLW8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=T+stnbu8; arc=none smtp.client-ip=209.85.219.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-dcc80d6006aso1353037276.0; Fri, 16 Feb 2024 17:28:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133288; x=1708738088; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; b=T+stnbu8W7xsXBaJXXb5IZCo+ZKzT0/hgW2KW/n0IXmzvJTlvXAt4tbeItEsPJ5lQl CnwoLlKGDIt4tY3nXQW4TeZAl17otodsn8hZpe99QJVNcBScvghsVPk86kClrMX7/Lh1 rzFVrnHjZViROBcdKs53jVj8h31pX1mhM29IfBC5LSgYQryCBFy1TZGop/NOcdplQD1u jNl5jB9n0VtFZNAyQGJE4owG5XTO4IsVMBKgSJQaAKQCdqH/RTqEAbzGuU4s4H70llyK ZDt581xJ9k0Vy+ZecbMkOzmI9Pjh55bLwb6TfSG/Y5yQk2m1JNk3ys9XbFNZ+7Vz7ytk NGmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133288; x=1708738088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; b=H2/eyaXay2yfiCOf+fKt66L3wwUlQl/1S9lF47ZPVEkDs2amuOCSEAyRnOCjsRQ1hX KDmnbOcAfQghDqWAdzE098ZtnLh4oQLuRl2GlgqWsk8sqxGWjauTUO+mfZ+KWzoCVQ4R L/6u6gM4HOtcDKcep8JSl0wKcfgt0ryHw5INqvQ+HaG3ci/eZHCaUN5uvHf9AEtJnfF8 v3SMyLdxkjuuStvuiSX1xDT/UDBHrakDNxUFflhbXgApKDB57xEbkg184/VggcVxVM/h DTwd15Q9owOhinp5VsCuW/peuwmN0KQCRdIrYIxEKYEUeLCCtp9yj3kyT5W+Wdk2l3u3 VTSQ== X-Forwarded-Encrypted: i=1; AJvYcCXWOrXkpdP2sFHkBKnIPnst0rCIGJ4B81h4D9roQ7bkkvnPzb3gGggAQ2N9dxqVuvLZSBHY1LWGhiQpw5/BBrGq7IyR X-Gm-Message-State: AOJu0Yy9x9esI6FGLGIL58KYTcpM3R4RAm/6DChjHw75JKB2VI5DMvSQ 5L0z4Ss6mdpfl6GIHnfKA5nqs9OrIyjpgJoDxwGUQCVSh4vZl1o/ X-Received: by 2002:a5b:708:0:b0:dc6:cc35:35e9 with SMTP id g8-20020a5b0708000000b00dc6cc3535e9mr6286134ybq.35.1708133288319; Fri, 16 Feb 2024 17:28:08 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id y23-20020a37e317000000b00785e016ec39sm442118qki.42.2024.02.16.17.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:07 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfauth.nyi.internal (Postfix) with ESMTP id 849BE1200043; Fri, 16 Feb 2024 20:28:07 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 16 Feb 2024 20:28:07 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeg gefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsg hoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieeg qddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigi hmvgdrnhgrmhgv X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:28:07 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Thomas Gleixner , Sebastian Siewior , Anna-Maria Behnsen , Steven Rostedt , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 6/6] rcu-tasks: Maintain real-time response in rcu_tasks_postscan() Date: Fri, 16 Feb 2024 17:27:41 -0800 Message-ID: <20240217012745.3446231-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107655970418374 X-GMAIL-MSGID: 1791107655970418374 From: "Paul E. McKenney" The current code will scan the entirety of each per-CPU list of exiting tasks in ->rtp_exit_list with interrupts disabled. This is normally just fine, because each CPU typically won't have very many tasks in this state. However, if a large number of tasks block late in do_exit(), these lists could be arbitrarily long. Low probability, perhaps, but it really could happen. This commit therefore occasionally re-enables interrupts while traversing these lists, inserting a dummy element to hold the current place in the list. In kernels built with CONFIG_PREEMPT_RT=y, this re-enabling happens after each list element is processed, otherwise every one-to-two jiffies. Signed-off-by: Paul E. McKenney Cc: Thomas Gleixner Cc: Sebastian Siewior Cc: Anna-Maria Behnsen Cc: Steven Rostedt Signed-off-by: Boqun Feng --- kernel/rcu/tasks.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4dc355b2ac22..866743e0796f 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -971,13 +971,32 @@ static void rcu_tasks_postscan(struct list_head *hop) */ for_each_possible_cpu(cpu) { + unsigned long j = jiffies + 1; struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); struct task_struct *t; + struct task_struct *t1; + struct list_head tmp; raw_spin_lock_irq_rcu_node(rtpcp); - list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) + list_for_each_entry_safe(t, t1, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) { if (list_empty(&t->rcu_tasks_holdout_list)) rcu_tasks_pertask(t, hop); + + // RT kernels need frequent pauses, otherwise + // pause at least once per pair of jiffies. + if (!IS_ENABLED(CONFIG_PREEMPT_RT) && time_before(jiffies, j)) + continue; + + // Keep our place in the list while pausing. + // Nothing else traverses this list, so adding a + // bare list_head is OK. + list_add(&tmp, &t->rcu_tasks_exit_list); + raw_spin_unlock_irq_rcu_node(rtpcp); + cond_resched(); // For CONFIG_PREEMPT=n kernels + raw_spin_lock_irq_rcu_node(rtpcp); + list_del(&tmp); + j = jiffies + 1; + } raw_spin_unlock_irq_rcu_node(rtpcp); }