From patchwork Sat Feb 17 01:27:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202556 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp96345dyc; Fri, 16 Feb 2024 17:37:17 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUzTp8mZuY9IRSZy2ILXBqamnDbv82AjSfRWa9H2kJ2pwOSlYeU+68BEw08DTx4MSzh1wTNnDUCsGwOgR8hqBE5HC+RVQ== X-Google-Smtp-Source: AGHT+IF+wAUfSGg0MHrYIGRCq2N2i0Nn9tKU4piwYcbX8jEmZ+jNlpMnNq+1BKYTbuae7huS98to X-Received: by 2002:a05:6870:7b4f:b0:21a:c3c0:40c7 with SMTP id ji15-20020a0568707b4f00b0021ac3c040c7mr6786475oab.48.1708133836893; Fri, 16 Feb 2024 17:37:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133836; cv=pass; d=google.com; s=arc-20160816; b=XACJrtcZl6ch0rRh9FutAC2NYYJ7t7I7XlawE4kffKoMXoR9NcRwokAJ1DBx9Kf2sq ExNE4L/mKg9chJQf+Y6rJ1exY4aF/AXvjlchZYxInwQF/SeXXCd3YfzxyBdaoO3WiZZc 6fF6pWfzczWzaVaRfYF1fDNDI8hi63tN7yRuoJJhhLlxH38++kOeD+IhiVIW2IuHgE0y 8s2XAooUXgVTxi2CzybHXqPP8wfcGmkKqUjVeeU6x7GUKBh3/5zkaSfkUdBDCHE7k6Zx BH6cVQwFrYZJzNf+ilXfQoCrzPuc5w9nZBhF4+wNb0uV8/89zRoSe81ll1pVJ2PUL9m1 qPwQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=eSS+leYgx/1CnoRDNP0u4WvLSnhQI+J7A/LTWeqZnSM=; fh=9oilMSntsrFYP7dgQAt69A8yD3LklPyGE6zHxU54lRs=; b=UrLu/ywh+LlxiJr2InOBfqgo1ddCCDvRV4XW8uDvy5hdSdniSd9NKxHx86UrnLxkfC hEPe9QqQAS4pYB3L8Pt9MT6hLxwWH/2wP2+NUFp98zHBl8r6G2nNjZ+4W6iWhvIRqTY1 v5PsHHHN1GnUTeUjB8Cy2Q0rQzkN7eAPdNsPcyxNW20NkqI6GPAGjUpAsF0U0IFi9cSf nh9wGGEbzOLTnQEyWrudrVKq6pIEKRYkAW9aEnvnWA8Aieus/KhnexL5ca/pn77vgEUz c3l5dKnTbNzzo4xlYbBR6xjWZu20XwTPmeR2z+tYm/4Xsy4wWZIHepy27RjmXgOI7NrR tZ0A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=XT9C+nFm; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id bz9-20020a056a02060900b005ce016cce21si759505pgb.191.2024.02.16.17.37.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:37:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=XT9C+nFm; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 6ABB9B23508 for ; Sat, 17 Feb 2024 01:28:18 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4A0481B7F4; Sat, 17 Feb 2024 01:28:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XT9C+nFm" Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB5D74C8F; Sat, 17 Feb 2024 01:27:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133280; cv=none; b=Hae704mcMC4M5q9G/2OpbadNvL7QbnqZ+tPBqkMkLdx5QSjFL8SBxXkbmg2MlzQIVxONMsDTbhwpvNkVfrk55srH9pij4DPzWuD4dTVGNOEiRc359ZB1v2glCLklbvU1rWMXOZv/rwyqc2u38wpwRDgLGR6QikhhSxSAsYJrgrg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133280; c=relaxed/simple; bh=IPhTwhd+sG/j2XT8YGeqy+Bq/Ymg/GX/lulfajzIek0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rJwFe0Qao+Xk4nNOYrsRgb74kSdgOBoim74lFi2OvTGZgeobEJFUdBZyGXWACYgNvJL9ikLG6KlhqEQVqu/CI3AqfNd40iXcL3xvqm+FE87I+qEHQC8nkShNWGLOkxThIy7QuEPhS0yATaNxK53675MFqJg7rHpA7U+C3aWZGoc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XT9C+nFm; arc=none smtp.client-ip=209.85.160.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-429d7896d35so15653621cf.3; Fri, 16 Feb 2024 17:27:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133277; x=1708738077; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=eSS+leYgx/1CnoRDNP0u4WvLSnhQI+J7A/LTWeqZnSM=; b=XT9C+nFmGsjUZelHvCh7UDqtDD42//xwV/56k6cCZLDvmYUkD9sDbV6YYbFcPS0dpZ eDCWIvQcrse746JM2AtyG4xmO4C9A0+1zAy9dmO/R1RD+a40FYxABwdyaxrz36s/ZDSq Y8PkUXjyD27O3h7ezsatIZAju3AktWYznlFTpELllRQk4BSDQDmaxAUk1XtuaR90dcEM hDYzQvzGlIZ7Ugu1JwUUlpVddz8N//CNVquXmwyqIJDGdXPM4cc4go93yUWBVl0FgRj1 oUzayr8jEUsTOLDs9/gPoJqn0sSTiy6tbeMfLGqQgD8JTK5n+gnMqyc4L746qzXfeY1r HYHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133277; x=1708738077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eSS+leYgx/1CnoRDNP0u4WvLSnhQI+J7A/LTWeqZnSM=; b=Gmz1sDU+L0U/Mj1hsLwIgnqGPdOSI/UJcVQzch17WL72NeIlhOSH/XOVKaDLCmBhn7 QBx+YH/DjkjMrGykfIrNDjhdCfWbhRc1Q5JvO7rb2V7vacOh/Z20IdYwb4mvCyCV5ra+ BCc/kPNQg0TH1cIZchrNBUks7l4zvwTFSrwQ9NEXMGJ2hopa380pTkmkgQLvoTa0xt1f jQqkO9G0I7tTUuHq4eXSSTBHwPKi4pcF8glmkQWWzOvNrPlmwMTSZVXAmUevESLGXXFG LrPoDyKl5HSdWdqBRVmoja8hnmxn1diP6Q4fDVbpmnxU+VyUT+BKGvTdNlKuycnSikPr +8GQ== X-Forwarded-Encrypted: i=1; AJvYcCUU1Mb5O1oyfnRfKGK1mSbAf687DbAcfSxHqzrwM3SxG1yjE6D28x7e9+TcvVAZZXL/aX5mO4Mi3c0DQvBLBWjH8EXb X-Gm-Message-State: AOJu0YxhVuAjXoQOxCuI0IO4Mib4j6Wqv4Xm35HQUc5gGHUbE62+R9Cm PuXRsnIn1cou8HO4oikDoQFlGVw5LDgnY2Q18YOffZFaVL7oAn/W X-Received: by 2002:a05:622a:316:b0:42a:9d03:1c1d with SMTP id q22-20020a05622a031600b0042a9d031c1dmr7243965qtw.63.1708133277533; Fri, 16 Feb 2024 17:27:57 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 2-20020ac85742000000b0042d8bf105c7sm432772qtx.2.2024.02.16.17.27.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:27:56 -0800 (PST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfauth.nyi.internal (Postfix) with ESMTP id 77F1D1200064; Fri, 16 Feb 2024 20:27:56 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 16 Feb 2024 20:27:56 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeg gefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsg hoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieeg qddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigi hmvgdrnhgrmhgv X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:27:55 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Steven Rostedt , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 1/6] rcu-tasks: Repair RCU Tasks Trace quiescence check Date: Fri, 16 Feb 2024 17:27:36 -0800 Message-ID: <20240217012745.3446231-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791108146260545577 X-GMAIL-MSGID: 1791108146260545577 From: "Paul E. McKenney" The context-switch-time check for RCU Tasks Trace quiescence expects current->trc_reader_special.b.need_qs to be zero, and if so, updates it to TRC_NEED_QS_CHECKED. This is backwards, because if this value is zero, there is no RCU Tasks Trace grace period in flight, an thus no need for a quiescent state. Instead, when a grace period starts, this field is set to TRC_NEED_QS. This commit therefore changes the check from zero to TRC_NEED_QS. Reported-by: Steven Rostedt Signed-off-by: Paul E. McKenney Tested-by: Steven Rostedt (Google) Signed-off-by: Boqun Feng --- include/linux/rcupdate.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 0746b1b0b663..16f519914415 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -184,9 +184,9 @@ void rcu_tasks_trace_qs_blkd(struct task_struct *t); do { \ int ___rttq_nesting = READ_ONCE((t)->trc_reader_nesting); \ \ - if (likely(!READ_ONCE((t)->trc_reader_special.b.need_qs)) && \ + if (unlikely(READ_ONCE((t)->trc_reader_special.b.need_qs) == TRC_NEED_QS) && \ likely(!___rttq_nesting)) { \ - rcu_trc_cmpxchg_need_qs((t), 0, TRC_NEED_QS_CHECKED); \ + rcu_trc_cmpxchg_need_qs((t), TRC_NEED_QS, TRC_NEED_QS_CHECKED); \ } else if (___rttq_nesting && ___rttq_nesting != INT_MIN && \ !READ_ONCE((t)->trc_reader_special.b.blocked)) { \ rcu_tasks_trace_qs_blkd(t); \ From patchwork Sat Feb 17 01:27:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202550 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93700dyc; Fri, 16 Feb 2024 17:28:38 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW9PbUWcjP/6TQehi8IVpvOMHMbOxyxgUvM7YZE+kSXMpG9iDKlnxdRic8DH5oYu58MGNFPajgq07VActCtGdsPNmdlYA== X-Google-Smtp-Source: AGHT+IHxF+ah1YvTsY+3/L2EXKC2wbWoH9FyRiV1vsXM/1ZVEvVAasToaj8gMKvR5PWoPWiNmCF3 X-Received: by 2002:a05:622a:50e:b0:42c:5c56:bbd7 with SMTP id l14-20020a05622a050e00b0042c5c56bbd7mr7979744qtx.58.1708133318730; Fri, 16 Feb 2024 17:28:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133318; cv=pass; d=google.com; s=arc-20160816; b=Dm08oQSLXFmdFCPFYhtPy34guawGH0Y3PV2sq3Hoxb9/AqiHTZ/0Enej9WI0vx4Qn1 J32qHL3qKmO81YVXcmIXjcvSGqCwxiZnZIhmLppOAu5lfxuaVjvP2u6t8ONk0zj2qVA6 WR7S2cNc5g+GV7SGRVCo+Usjb1NA1AvWWVEmXQYVZ7OP6OqDCTFUXZVzMna1Z7uwAOL4 7B0yTvy+/JyfscIGwp/ixYSE7ILRAhFqyNkdRDTjHjDpkbSBWRsUVkY6ALCLG6LPfo5V E9XvRPQ9BQWv8wspE5H46Atymfs7w7IPGAQU5mgUfuQXXK6kX9SPxYlI6YNmFyktgleq GuYg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=iO0FGo+JHvvXIFJUwYJqm/v2xV/l0hzo3O16zP5UEz0=; fh=pUJXpYKZJA5KPqlBJjhsw0gTxnegXo4HmgydtDHY5kU=; b=hRRRwCuG0Bamk3FkCFdzyi+yTcvrC7ToVblcBWOHpitcvu1z2DQ3cIKRFznTi4jqtQ 7oBMHmd+vVHK6K245Wkx9/eMJ1TeYkHMXZ7l8bpm7bnpawHZ7oEk34LQeM7EG+sJpNgU Zh6UMlkwuLcd7FRjBg6kuI/oQI659kifEtUziDeV8fgdudiHsTVhElT0WZ45i3pijnWy PySFhrl2NG06DN3mYkXnlvnIi9EBo5CO/QiDt1H3/He0WGERpj0XP/fhs7HrcNjPFvrr MlqxDn7joCITZUWZ9JhmnZh4Tu/KlQ9rv3xVcSaY9mhAIPkrXxk7oTUYeCHf/4qmpWHM DCaQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=VoWXBY72; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id h5-20020ac85485000000b0042c66cff88asi1068062qtq.407.2024.02.16.17.28.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=VoWXBY72; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 75E0D1C21A59 for ; Sat, 17 Feb 2024 01:28:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6F85C1CA9A; Sat, 17 Feb 2024 01:28:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VoWXBY72" Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F93217745; Sat, 17 Feb 2024 01:28:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133283; cv=none; b=J0pScchgqqM5G3gxZGBIJk6GrtCXzrq4dk+t/fthw5eR8oNWWblZBpIb4PdmmdnqghLY6epm5Z1hFLd/2a5585R+tJQxjpal+WHtK5UjszOGWULIG5RQ+LYpLo8nuiQQPRbcZtMmgv+rNr/d6nnJ0sfEcaZibS1RBHATqqxMnCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133283; c=relaxed/simple; bh=T+CUzhbVEdJ9yuhSl92mWeUi9ua4EdFF8tCRqQ8HMkg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r0Bied3p/OZfueqLIBUVd+KWHpkDUKPcMnHxKs6BD+cu930RY3f8WFsEgTJgmWzKWZkecOzrvEoHnHPof5rBuTTwwz2m6PAAxSbS5TD0ZdgO+KfsDZT8LWMgGrc0fPB2U9n3Yemld3aiNlKV/7rtjbVYtZZ55UPqFAMfeG0a5z8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VoWXBY72; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6868823be58so5007756d6.0; Fri, 16 Feb 2024 17:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133280; x=1708738080; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=iO0FGo+JHvvXIFJUwYJqm/v2xV/l0hzo3O16zP5UEz0=; b=VoWXBY72iTUwW5E9/gDaellDfrbbDTP0Wu8Upy2H3OPRSzdrpKMTjTZkVUcI8o8wXu dfrhnmtdllVNzP2UVTkrTfA45+DBxNJS/DAzByREtFSHCBtRY0ToCJabjr5OrmCCSIrL 953lDekAyucvqe4SJ+IRh+GEjxFKbsfu3EkulYGa/QjHL2JTKf23gFdl6lm6Yc3WG8DS UiDe+nPGTqjJBaqjyE9Yo55GDCN4EotUV+XNUHmnCBsQiHv84PCN4rEVB83qjnw6R0NC Le9vc+P2kNLTn95/T3/4+dKSR/UK2+qbRkdGDuLz3x6z7IY2eu8NnrxFNB64I8ShilJF TvaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133280; x=1708738080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=iO0FGo+JHvvXIFJUwYJqm/v2xV/l0hzo3O16zP5UEz0=; b=AxnUMla2alhS1FF+ubw9CYZlsY/cyaODm4jFdiEW65SrZGoqTkAkph9Tie7rMOkVga nrZwsmRZP3xmj4BOugA3orONCKmQ6xYaAD10cBajOAoVYXa8b/ei5yCwQvPf68WS0kg/ r/QiYE3a6uZ1ezjabv0M4YWTEc3UyFLJ7IspzGg6nNikMfAKuIWm1zknEEn/SOirlM9v glhMRspBv0PQC/2NxzNa1LmNH0fKJFXm2zYm+81GFQ4ZOU9mjbOQMlWU9Mjh5eN57L++ eFjA+uQfjGwfRLwQBQPhMT8YtoLvyrUtNs0hJhAodrNGFDnMY/02y5PbLV25rwKwH9wz XWPg== X-Forwarded-Encrypted: i=1; AJvYcCU9N30kzAOX2QNNNJ0Uu/ThywrocX6uGrVXXrvxjxjOI+gFl9FApHhSka0aeQD8C/N+Q1rTPgJ/tLYMjWXjdCzeVPmg X-Gm-Message-State: AOJu0Yw0yL4eCm3VvWBvRhSms0ySzNsSPyXRA9Uvf7llU+E5w61T3yAI zpQEO+Nb5+UnynnQM4axZ/nX/HwXNmZNWwmxE7VAU+z0O6onUBYI X-Received: by 2002:a0c:e24b:0:b0:68c:a4a8:b134 with SMTP id x11-20020a0ce24b000000b0068ca4a8b134mr5789764qvl.45.1708133280142; Fri, 16 Feb 2024 17:28:00 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id er19-20020a056214191300b0068f304336f5sm487340qvb.36.2024.02.16.17.27.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:27:59 -0800 (PST) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfauth.nyi.internal (Postfix) with ESMTP id AC5FC1200064; Fri, 16 Feb 2024 20:27:58 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Fri, 16 Feb 2024 20:27:58 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefg heeinecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhp vghrshhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrd hfvghngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:27:57 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Chen Zhongjin , Yang Jihong , Boqun Feng , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 2/6] rcu-tasks: Add data to eliminate RCU-tasks/do_exit() deadlocks Date: Fri, 16 Feb 2024 17:27:37 -0800 Message-ID: <20240217012745.3446231-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107603034672891 X-GMAIL-MSGID: 1791107603034672891 From: "Paul E. McKenney" Holding a mutex across synchronize_rcu_tasks() and acquiring that same mutex in code called from do_exit() after its call to exit_tasks_rcu_start() but before its call to exit_tasks_rcu_stop() results in deadlock. This is by design, because tasks that are far enough into do_exit() are no longer present on the tasks list, making it a bit difficult for RCU Tasks to find them, let alone wait on them to do a voluntary context switch. However, such deadlocks are becoming more frequent. In addition, lockdep currently does not detect such deadlocks and they can be difficult to reproduce. In addition, if a task voluntarily context switches during that time (for example, if it blocks acquiring a mutex), then this task is in an RCU Tasks quiescent state. And with some adjustments, RCU Tasks could just as well take advantage of that fact. This commit therefore adds the data structures that will be needed to rely on these quiescent states and to eliminate these deadlocks. Link: https://lore.kernel.org/all/20240118021842.290665-1-chenzhongjin@huawei.com/ Reported-by: Chen Zhongjin Reported-by: Yang Jihong Signed-off-by: Paul E. McKenney Tested-by: Yang Jihong Tested-by: Chen Zhongjin Signed-off-by: Boqun Feng Reviewed-by: Frederic Weisbecker --- include/linux/sched.h | 2 ++ kernel/rcu/tasks.h | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index ffe8f618ab86..5eeebed2dd9b 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -858,6 +858,8 @@ struct task_struct { u8 rcu_tasks_idx; int rcu_tasks_idle_cpu; struct list_head rcu_tasks_holdout_list; + int rcu_tasks_exit_cpu; + struct list_head rcu_tasks_exit_list; #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_TRACE_RCU diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 732ad5b39946..b7d5f2757053 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -32,6 +32,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp); * @rtp_irq_work: IRQ work queue for deferred wakeups. * @barrier_q_head: RCU callback for barrier operation. * @rtp_blkd_tasks: List of tasks blocked as readers. + * @rtp_exit_list: List of tasks in the latter portion of do_exit(). * @cpu: CPU number corresponding to this entry. * @rtpp: Pointer to the rcu_tasks structure. */ @@ -46,6 +47,7 @@ struct rcu_tasks_percpu { struct irq_work rtp_irq_work; struct rcu_head barrier_q_head; struct list_head rtp_blkd_tasks; + struct list_head rtp_exit_list; int cpu; struct rcu_tasks *rtpp; }; From patchwork Sat Feb 17 01:27:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202551 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93808dyc; Fri, 16 Feb 2024 17:28:58 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWpkgCxC1MbTPARRPYTO5TKrFbI7XrMYWDbErfUUIaSwD/vrXzVDJE8MGPh/mEG9Lg26DTzDubISNw5LINKZjzBDcVpAQ== X-Google-Smtp-Source: AGHT+IGou7ZDo0BQc9zUAlu3df2lz3SWnWq17ey9gEsxq4ueLt7PpORv1W66y4HvceP5VKEktCDs X-Received: by 2002:a05:620a:2043:b0:785:96a6:f6ce with SMTP id d3-20020a05620a204300b0078596a6f6cemr6611022qka.43.1708133337970; Fri, 16 Feb 2024 17:28:57 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133337; cv=pass; d=google.com; s=arc-20160816; b=jbvlEqBL2uG2wXbly/oL57VCgCGgtYA4VE9aUN0VPPo72maP8vusX+05L7d7JQQvtS fxIU6KnKtUI9noq+qnBrJ57jjkqfoA5mgiSAA8/tR+cBnXzp/OPnMM2aljNWccunuFc5 jx+CcqRc0LU5z6GovJTbzk6eC0kqBI1kndUhgNajEsic3h2/UyqAXD6kvVs8BrzwHGbv Z1VchiNx/7yqBZjWdzLRtmdWmt27Q2J3fpU2sq0FGYh7eGTcy4C3GKkANi6NN+Hnuhl1 u3vEsMD7Y8slqsKi5M6L5noiaskiYoA9Qq52fLQ4v4jOtbu/tZNgPrw+yKXB/67ZqAU7 88qQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=EqWJLzhbN2hPYk8CsSuDuGTVd75xnF2oMWP55z7qdX0=; fh=IiUbLwkbFIh/44BWntCRyQXtVRSaZb2zc/Z+4fDlNG4=; b=IBtQ0HwbPB0s+N7JBRd+pdSU5ZiTSsdQbCYFLUrRm5eSi6ga8znshCl6LxLIpN4jcL vhGDUkF9C3rzIYoYU6Z34JNdDIslaz95mNF+dqd0gXMcJcGiLM6916hylUGm2JvgJId8 f0E2yEamhesp2y2oUCd29t5l77v8nWXEday5a7dp3DSx0xSg7da5j6RyHuyvFJJ/eOd/ a4lHSHbdkxit6Nl9AUmpwRvVEiibwW0L67k20gJd4Q7DlUGvmsJ9TWiv080OdmhpJ2e8 LL80g73/NcqR1yuY9BjcC8Whmw3Q397QEdNBrPTXCC3GS1Z3FIW/G7FmyU4j/QptonvR pbkA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=hl2soOzW; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id m4-20020a05620a24c400b007872569fc53si1413757qkn.650.2024.02.16.17.28.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=hl2soOzW; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B0B8E1C21AD7 for ; Sat, 17 Feb 2024 01:28:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C9DD41CD27; Sat, 17 Feb 2024 01:28:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hl2soOzW" Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7EA11C2B2; Sat, 17 Feb 2024 01:28:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133286; cv=none; b=bSs05HGZRnjViVIaoIkkl5AoEoJguRbI18nm1hdcSLuhZwjr8Lz3YZwePgAq00PQ11x0Qb14uTdj9/vow7Z9X8oo/LJedT0Mikxzm0Q8dCWX316ptXVKbgUwpa/joViqOtuJ0deQoM7iJ7gfx5sLBiCb6PvSCP4lnXQ+ZGJpIew= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133286; c=relaxed/simple; bh=2d1ogzLrDb9BJTG1ntCaJltOgc+YNgTsi3wE71ajevA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G1FAmyXhihioosK35fp2Fvcp3IlNKTzzj/qU5M75CKK0cNbPFBRUB1AS4HmyfwfcgS3NFJICrz3wm8w9T/H15K5aIF4Yjo16p6pWmLVeDul4eqMwwI9g2aNMm06RG/U7odu9IBj5n/tPpYTD9xnmhRI/uN10sD0Pz6AuFh2NO0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hl2soOzW; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3c04535b706so1493106b6e.0; Fri, 16 Feb 2024 17:28:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133284; x=1708738084; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=EqWJLzhbN2hPYk8CsSuDuGTVd75xnF2oMWP55z7qdX0=; b=hl2soOzWS2ODFCHneH2gN0rdc3gRIrQTduxh6JzAfH8xo8ZNAFtfdWJH6RvvejT2zN IZQFnh+wgvbS+F5NUo9bRb2RI9uuxcfxbXG3xENhoHUCb5J5ob/eJjEL/DjReJoSBhdz bSJWlvgipgR9fYQjYHU7hVIHTeYTmnn/apvK7ueYkOfPbpC49F738M8Ms/BzatYDa3uR odfllwqXbhkGcnKQcc0IQzOtCJi7Q2tSA+U/A5AIsKkwvnfGU5ohbLXV1KryeKtdIU1V IkZPsr7T8KTLCdDXdPadYwAiRcyjiXIuQJ/w+gHKwqBqQgESLd1+0qBr9DOYQgB5umqp L8OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133284; x=1708738084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EqWJLzhbN2hPYk8CsSuDuGTVd75xnF2oMWP55z7qdX0=; b=Vyjuu5NN7im8NcnGuP5TGNYcgJuQItf0Cfg5L4dUt3DCo6wpxClwWUGKvEU5oZaaNq KnWq2CapsYAqOoXbY2wgxNWEhqHrgTgjt6IN3EQGHc5NWdrrkPrWUKE6cLkz1aUEPWv2 9TDY13vwmQYmp3yWNZQUySHXuqMmIsuFgfGwgCxDagNgKYFWE3VjvgU9Zt13CTYs+dpS JXgFemOfZ+eQo75ruxzHsD6m/R2tvfzVmx/CW7WFzO1UZbOC7tms2gsUR8Q0VOt6DtOF jtbwjWG0tys8fyky3Gsl3qauBDR8vL1Qht5cg1oTKFl13R24z7a+nfHgoeypoHRtA3n9 bAdQ== X-Forwarded-Encrypted: i=1; AJvYcCXbF6IUXh0YupEUbneFkwB+dz3cVObJ5VQjhU3KEAa4E02mhnMAxs4/6OhtCvsZoTi3MIBHYxsW7HZqzSw77kxLxWb8 X-Gm-Message-State: AOJu0Yz5URBLvI8D2wThcsPZ9J/JyEN8Lka606AfEjCAUNnYfZP4QdQz yGnf3cjcsrW7HG8iabUMPgSex0JrrTH1upFKNPf6xTDy57c2Vjt2 X-Received: by 2002:aca:1a0c:0:b0:3bf:f452:651a with SMTP id a12-20020aca1a0c000000b003bff452651amr5780521oia.52.1708133283981; Fri, 16 Feb 2024 17:28:03 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id bk36-20020a05620a1a2400b007873d8734cfsm423907qkb.132.2024.02.16.17.28.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:03 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfauth.nyi.internal (Postfix) with ESMTP id A571F1200043; Fri, 16 Feb 2024 20:28:02 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 16 Feb 2024 20:28:02 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefg heeinecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhp vghrshhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrd hfvghngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:28:01 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Chen Zhongjin , Yang Jihong , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Kent Overstreet , Oleg Nesterov , Heiko Carstens , Christian Brauner , Suren Baghdasaryan , "Michael S. Tsirkin" , Mike Christie , Mateusz Guzik , Nicholas Piggin , Peng Zhang Subject: [PATCH v2 3/6] rcu-tasks: Initialize data to eliminate RCU-tasks/do_exit() deadlocks Date: Fri, 16 Feb 2024 17:27:38 -0800 Message-ID: <20240217012745.3446231-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107622829453121 X-GMAIL-MSGID: 1791107622829453121 From: "Paul E. McKenney" Holding a mutex across synchronize_rcu_tasks() and acquiring that same mutex in code called from do_exit() after its call to exit_tasks_rcu_start() but before its call to exit_tasks_rcu_stop() results in deadlock. This is by design, because tasks that are far enough into do_exit() are no longer present on the tasks list, making it a bit difficult for RCU Tasks to find them, let alone wait on them to do a voluntary context switch. However, such deadlocks are becoming more frequent. In addition, lockdep currently does not detect such deadlocks and they can be difficult to reproduce. In addition, if a task voluntarily context switches during that time (for example, if it blocks acquiring a mutex), then this task is in an RCU Tasks quiescent state. And with some adjustments, RCU Tasks could just as well take advantage of that fact. This commit therefore initializes the data structures that will be needed to rely on these quiescent states and to eliminate these deadlocks. Link: https://lore.kernel.org/all/20240118021842.290665-1-chenzhongjin@huawei.com/ Reported-by: Chen Zhongjin Reported-by: Yang Jihong Signed-off-by: Paul E. McKenney Tested-by: Yang Jihong Tested-by: Chen Zhongjin Signed-off-by: Boqun Feng Reviewed-by: Frederic Weisbecker --- init/init_task.c | 1 + kernel/fork.c | 1 + kernel/rcu/tasks.h | 2 ++ 3 files changed, 4 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index 7ecb458eb3da..4daee6d761c8 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -147,6 +147,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .rcu_tasks_holdout = false, .rcu_tasks_holdout_list = LIST_HEAD_INIT(init_task.rcu_tasks_holdout_list), .rcu_tasks_idle_cpu = -1, + .rcu_tasks_exit_list = LIST_HEAD_INIT(init_task.rcu_tasks_exit_list), #endif #ifdef CONFIG_TASKS_TRACE_RCU .trc_reader_nesting = 0, diff --git a/kernel/fork.c b/kernel/fork.c index 0d944e92a43f..af7203be1d2d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1976,6 +1976,7 @@ static inline void rcu_copy_process(struct task_struct *p) p->rcu_tasks_holdout = false; INIT_LIST_HEAD(&p->rcu_tasks_holdout_list); p->rcu_tasks_idle_cpu = -1; + INIT_LIST_HEAD(&p->rcu_tasks_exit_list); #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_TRACE_RCU p->trc_reader_nesting = 0; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index b7d5f2757053..4a5d562e3189 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -277,6 +277,8 @@ static void cblist_init_generic(struct rcu_tasks *rtp) rtpcp->rtpp = rtp; if (!rtpcp->rtp_blkd_tasks.next) INIT_LIST_HEAD(&rtpcp->rtp_blkd_tasks); + if (!rtpcp->rtp_exit_list.next) + INIT_LIST_HEAD(&rtpcp->rtp_exit_list); } pr_info("%s: Setting shift to %d and lim to %d rcu_task_cb_adjust=%d.\n", rtp->name, From patchwork Sat Feb 17 01:27:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202552 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93880dyc; Fri, 16 Feb 2024 17:29:10 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVpPtRYSa7Y/f9BcRPlEwv1OdZYmB95KfC2XoD9bJOZEj/BCnm7/lBYWsvPaI6JE58xf6IT7Kve89RBX+iV+/mqZA+WEA== X-Google-Smtp-Source: AGHT+IEqm3FNqUJPCQPTRlEiHaSHVus/vJ7hsBg4tSU90SFA6EqCB8hYa5Ov1/JJz31vdTY2eiDU X-Received: by 2002:ac8:5b81:0:b0:42d:f2d4:9158 with SMTP id a1-20020ac85b81000000b0042df2d49158mr496719qta.41.1708133349920; Fri, 16 Feb 2024 17:29:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133349; cv=pass; d=google.com; s=arc-20160816; b=Tj80uuk0vFoOj2ZcLfIz687r3uD0Qo8Yoqlrm1A5VwDBEkMUxTTwH/0EEXxYeXoCdg yLYWLShkmJ8itAqZSo1FZNB/EhL32W5vcUBI2ik0eSItOyDyjyinESMeIWOMjXIgkMU7 SStYigCmBeelg+jsMoagxmsaouI2k8wECqcrVum4LJk2fZxAA9+gKawxnOsfcu5/x/yD 0RDwwUDVCDGrVL/pF7MDH8wzYXSrWdADSTa/GbD1f+gDLGPH0P9xJj2sU6eT8T5he4VO B6iFmPKKhINW/GLVq5+xSUk2E47Gjhrsre0eQrodpf4SJB0NUzAKBOjHWlj5Cvwe0wxp S0mg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=hviMEBDsjs4mPn8ckWbDhMJvstiAXO/xVzEmAIrjOf0=; fh=NnPehC6FTgCewrlY7Hxnfdpmaj9TQEbqxjLp6tItBEw=; b=zIXp9LVLRHO6s9buKZI80TgN2oe8l3pXGaPaWBF+fqdfNPN+70wN+AU0KTnK/sgPbG yQM5vVAGz0hxe+W2EOTwIwU6tv03oOZjSQf2ncCmGvahrpp3yI5kd/WYCEiKqvS0scuC kCHoun3cC1ff0Nptu+Qaj5LiJ1KIXXxXtu4R8OWhGUeeP8sXpNyXVc85FO0dhSjmqdxl srLret0RFMSShK7gmZoIT9COAyq7ACvFoYVpBb+aBiWoCpZgUe4MMwVOQd6k2cgnvxY8 VPJlk+Eq4nCjrvPUi4WsLY03/flz1VlkS9wYlLdASupTJxmBJgRZef6HcRdunCNeftvZ qOXA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Nq3WjxKR; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id x5-20020a05620a0b4500b00787329174desi1170460qkg.490.2024.02.16.17.29.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:29:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Nq3WjxKR; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id A4AB71C217A0 for ; Sat, 17 Feb 2024 01:29:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 49F5A1CF94; Sat, 17 Feb 2024 01:28:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Nq3WjxKR" Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FE0D1CA98; Sat, 17 Feb 2024 01:28:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133288; cv=none; b=IdKVK64tK94YhYiA7HuibPttaLdipcbrbubGrERrQwsxgPkFTZSBd+Hg9WWmvwTxzko2kOzCdu4axaQVxIKN8DNF0sSDEW4XzNNOXDpDDFyAYEA9DPbu+HKAWCk11XkLrqEnfzrrySBIXaheT5HJF5/Z2Gn/wH7my3VOePSri8M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133288; c=relaxed/simple; bh=guwEaF/rDbBs7UTfGuQDT1thFEEvaDXW74LeCy5xw24=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Th2l7A9GAGNKcisD+5G/3ztPHxXz4Fv38xwNxYDlbLpXk8Jp/jhka0HcBeJ9WVkFYD7Rg8Bb+Hqg9HL3+zv8dCznJbSBKzn8Epw9H9PfycLUAxjZ2dQeF8GzSjYaXOJFXsh75GjxH5tFF0oryQYuGe28Q0/UNETBQebJoQ564eQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Nq3WjxKR; arc=none smtp.client-ip=209.85.219.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6869233d472so6436016d6.2; Fri, 16 Feb 2024 17:28:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133285; x=1708738085; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=hviMEBDsjs4mPn8ckWbDhMJvstiAXO/xVzEmAIrjOf0=; b=Nq3WjxKR6mU7gcOf1RMVTeXpEGP2bL1Uguoao7B4FA1xQpubnhz8BepxH4+Up2OM9D 1UCLLVz4JVgSV2mhR7xKJDN5T2ViXDJ6nZrY2FfZ+0QEu57ILCqnZma27ME7VdqsqIUg VGg5FrxqSrdlZO+tOKZN0KRpIxuYZaw59lXnASvxhCZZKs8xQiXU6EpaiTwYxud4VXJy nOurVsaIE0n0u7vixkYW9dLtbdJcHQzMT5kG4nTknqKgZMz8gtl8UxKIC08GgTmw0xIb KbwRBDD74S7AcLmrxver2KZoIEt51ma28cshOr6OmSdBzLWQEtmkJy1LXlPD4KMe5RGn Ij8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133285; x=1708738085; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=hviMEBDsjs4mPn8ckWbDhMJvstiAXO/xVzEmAIrjOf0=; b=YY6ZpRPZa9LIvMVO/jEONU6yrO/cHx9mjHfkzjuyFq7wYddT//LqoKyy0LCiLeY3KI O5B/Oxtw1zw/pYWfri3bOtjEP4YaUn4LZ4Ez7YrQIxJlSyx2xJ148q9U5Dm+ZwT0SYs7 iwUsLton7/SXG4XJdFnDxg8k3Q2s02j8/43M7/EsHV2MQfOd2JLgKd6YngQ5CzIwTpr4 dziUliFnWZyJSqm//zlT3I2WK4Zop1ytPVrCCAHRcAXL1+wZ4ygIF9p/m3xWcpK4yQAz xOxhEKADcAOX1/D7TvjWkzGCUWl2JGMfP/KdktePN7EDfBTJLB0daJEb7WIoSAIH0JUT OAUw== X-Forwarded-Encrypted: i=1; AJvYcCWIdoIzal5a7NVUe0LzlxWuvNPnggD7e/N8lyuwJK8KcakVJRBpq6MX7I87kOKhYxN1LG9Wf2GQQ/bstiUwmr02fhDw X-Gm-Message-State: AOJu0Yx5o8IyLiD0eI5DELcgFUCMxmCC2Lpew91TmBR0ZberCfUbLaTT UDVPpbJLciw1J0KrW3NbPkUtwsFTSwtALJ2T4Z7EEs6X25YVywh2 X-Received: by 2002:a0c:ab17:0:b0:68f:ff3:299c with SMTP id h23-20020a0cab17000000b0068f0ff3299cmr6889855qvb.26.1708133285236; Fri, 16 Feb 2024 17:28:05 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id lg21-20020a056214549500b0068f1bc6c57asm481938qvb.69.2024.02.16.17.28.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:04 -0800 (PST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfauth.nyi.internal (Postfix) with ESMTP id 6EDA31200043; Fri, 16 Feb 2024 20:28:04 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Fri, 16 Feb 2024 20:28:04 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefg heeinecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhp vghrshhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrd hfvghngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:28:03 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Chen Zhongjin , Yang Jihong , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 4/6] rcu-tasks: Maintain lists to eliminate RCU-tasks/do_exit() deadlocks Date: Fri, 16 Feb 2024 17:27:39 -0800 Message-ID: <20240217012745.3446231-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107635551090734 X-GMAIL-MSGID: 1791107635551090734 From: "Paul E. McKenney" This commit continues the elimination of deadlocks involving do_exit() and RCU tasks by causing exit_tasks_rcu_start() to add the current task to a per-CPU list and causing exit_tasks_rcu_stop() to remove the current task from whatever list it is on. These lists will be used to track tasks that are exiting, while still accounting for any RCU-tasks quiescent states that these tasks pass though. [ paulmck: Apply Frederic Weisbecker feedback. ] Link: https://lore.kernel.org/all/20240118021842.290665-1-chenzhongjin@huawei.com/ Reported-by: Chen Zhongjin Reported-by: Yang Jihong Signed-off-by: Paul E. McKenney Tested-by: Yang Jihong Tested-by: Chen Zhongjin Signed-off-by: Boqun Feng Reviewed-by: Frederic Weisbecker --- kernel/rcu/tasks.h | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4a5d562e3189..68a8adf7de8e 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1151,25 +1151,48 @@ struct task_struct *get_rcu_tasks_gp_kthread(void) EXPORT_SYMBOL_GPL(get_rcu_tasks_gp_kthread); /* - * Contribute to protect against tasklist scan blind spot while the - * task is exiting and may be removed from the tasklist. See - * corresponding synchronize_srcu() for further details. + * Protect against tasklist scan blind spot while the task is exiting and + * may be removed from the tasklist. Do this by adding the task to yet + * another list. + * + * Note that the task will remove itself from this list, so there is no + * need for get_task_struct(), except in the case where rcu_tasks_pertask() + * adds it to the holdout list, in which case rcu_tasks_pertask() supplies + * the needed get_task_struct(). */ -void exit_tasks_rcu_start(void) __acquires(&tasks_rcu_exit_srcu) +void exit_tasks_rcu_start(void) { - current->rcu_tasks_idx = __srcu_read_lock(&tasks_rcu_exit_srcu); + unsigned long flags; + struct rcu_tasks_percpu *rtpcp; + struct task_struct *t = current; + + WARN_ON_ONCE(!list_empty(&t->rcu_tasks_exit_list)); + preempt_disable(); + rtpcp = this_cpu_ptr(rcu_tasks.rtpcpu); + t->rcu_tasks_exit_cpu = smp_processor_id(); + raw_spin_lock_irqsave_rcu_node(rtpcp, flags); + if (!rtpcp->rtp_exit_list.next) + INIT_LIST_HEAD(&rtpcp->rtp_exit_list); + list_add(&t->rcu_tasks_exit_list, &rtpcp->rtp_exit_list); + raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); + preempt_enable(); } /* - * Contribute to protect against tasklist scan blind spot while the - * task is exiting and may be removed from the tasklist. See - * corresponding synchronize_srcu() for further details. + * Remove the task from the "yet another list" because do_exit() is now + * non-preemptible, allowing synchronize_rcu() to wait beyond this point. */ -void exit_tasks_rcu_stop(void) __releases(&tasks_rcu_exit_srcu) +void exit_tasks_rcu_stop(void) { + unsigned long flags; + struct rcu_tasks_percpu *rtpcp; struct task_struct *t = current; - __srcu_read_unlock(&tasks_rcu_exit_srcu, t->rcu_tasks_idx); + WARN_ON_ONCE(list_empty(&t->rcu_tasks_exit_list)); + rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, t->rcu_tasks_exit_cpu); + raw_spin_lock_irqsave_rcu_node(rtpcp, flags); + list_del_init(&t->rcu_tasks_exit_list); + raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); } /* From patchwork Sat Feb 17 01:27:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202553 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93963dyc; Fri, 16 Feb 2024 17:29:26 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXTnP0x0qPuKIkLa91u9EmCiiJ0KAXIgQLwH1GdeEu8hSlWuzhdjZDERypCqkyLJgkEKUFULbWs1+TjI2MwsGk9cVG5vA== X-Google-Smtp-Source: AGHT+IErzYbF3DKXWbXaC5MpnFHhaOLMXM4ngF9czntiwy7TjfmtFama+BCZIKu9a9QzpZ0oZ0SF X-Received: by 2002:a05:6a00:cca:b0:6de:1d0b:ae29 with SMTP id b10-20020a056a000cca00b006de1d0bae29mr8498400pfv.0.1708133366152; Fri, 16 Feb 2024 17:29:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133366; cv=pass; d=google.com; s=arc-20160816; b=oqQT+VdKHnuXxdLJwXPGgCT5TWIuFC6xyy8Kx7rGIUGa5YhYa7cXTzshoP36H0lECo WIpYd7QeTQbCTY6LhbRBJbcirQ4Jbv5+tE3/xQUc5tGhGaHGY19Kse8NF62OSOBXyqPf 63dnp4qXZS82lm+zc0TmLimcVx1ItEMVJiRQSMeqw1ym8pupMGgZELwQ2chjE6GbY0iQ Mhb5UBlRZhy9rwbfEp4L5ERAiht5QtiawZxvtL+X/VjCASgYk5pEz7oMe+PvyqX4O0fh YW/68aU1D4h6u1sWkVVUtJ2gdKEcYHad9kd9qU8XQJm/PFBD/8NfpZw5ev7XK2ghctf5 j9NQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=VPm3Mqubx73iHgEJtMpZHgeN5M0pBWSh7mWOguIogNY=; fh=qzla3mBTLVEyNSkpyqg0focAFiSWCvwXq4Owh87zfDc=; b=qF78yczpNIoxjq7hztW3Ayb+06SD6scHODe/y8u0RfD9n0tiDfqj4CS7806Czb0Rpw /90WRXzEE8l9hS+qZYpuX0lLbtxiD9loRYz6a3MIewOUlqnKUm1RR8Y4d8ez4zNYZCUh iMtYn14aJkX8jt2JkNjifhgXkjB3nMkchhG+new/auIr5BEg6heHFdjZOUxUtOH5gbo7 jVTrlLy+8YYR2enNbEbO1SbE/ZsTPqrYgt3pKKALxyrrKcRqmvwQnVU+V2SzIvBLolVZ qj+vAMpfRry5SuGDEFVfa2l6JWowK2rXh7nTnhu191E7jweyC1hbLOklb+ozqIGIxASv RgrA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Eky6hQco; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id y8-20020a056a00190800b006e0f90ac3b8si761374pfi.141.2024.02.16.17.29.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:29:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Eky6hQco; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id EEBC728451D for ; Sat, 17 Feb 2024 01:29:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9799D1D54A; Sat, 17 Feb 2024 01:28:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eky6hQco" Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F05291CAB8; Sat, 17 Feb 2024 01:28:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133289; cv=none; b=ekGITg09yM8beTdFxCg9T/CQqHyyr40dcoQtdewj5/C7yI42u4ZmuLQUe0J8bRqzqfZV9yt50Ub1cNl7/1SG93oSxffgEnbPb+DSGUxvfLczn/Hm2auwvYBn4s7WDIUIc+EhcSrQOo/LiWAedTqKvRzOjDkcYQQ4bFVMfPskTvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133289; c=relaxed/simple; bh=Op/qg2WbQ4DaBSM3dzivfqtp+3FWxvhNMZYkDYGtYgg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SilSHy8OamJrKyxuM+YDbNrxjYAN2bxrZmmZPDEvI4FxB1K3oNVHVzIKrtWoqSaiVrrqwevr89ym/MROLWjobhshR12Y1deIz2jU2fLuPK9xMyje22OVYWAnhSMotPj09JMI9ShVu1fEmGeJik/wDiRqoLBlLFRZa9qeH1El9G0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Eky6hQco; arc=none smtp.client-ip=209.85.222.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7873eaa0ce7so65050285a.1; Fri, 16 Feb 2024 17:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133287; x=1708738087; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=VPm3Mqubx73iHgEJtMpZHgeN5M0pBWSh7mWOguIogNY=; b=Eky6hQcoitD3z6ylBouU9PDsoe+/dq6mV4Krtc8Kppe+rwvXcqV7HcZBdcfiCR0Ywx PTZVBq2UG1PGzyeNHN689UCnk/KKBwnmRc+TnDrvAnOs6xEAVVx7TIBhGs4dQ6I2WdtN im0Wcy0/s3djrJXfhkNHQvUN68v82osjbC7vj2r0AwHNBJeKb0O+2296gjcovl7ZZT45 Jv/NvTY8YXt7ykiOPm2nWfHdsUPB5xkHxRAFl5J919ehbSIsu3atJFUrTFXWTLN6B5jE CBRvth1myy/eG8G5YSxTg5zuIDHY5zB0Oj1DFoIHlWPd4IwvFzbOlWXFRI2tHRNw0YIa ESUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133287; x=1708738087; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=VPm3Mqubx73iHgEJtMpZHgeN5M0pBWSh7mWOguIogNY=; b=of/vfc4vDVlRD5SlVbDI518Qq3Eai6vtc7jQWENvq6kFGsKANXoiHMb83Xn1IqKayf 1uoHFKDfLVHRYMhJCETe0aAfSRWD70HXez4ng7K6C/OM/BdbPtFFj6zKwBTxBpc0J4rO IdntNgcElVsdjRQ5+N8B1d9DHKi5ZmdW/Q3eDpysC2RuyKZaWj5kVFF+08IKRHQfqk/L nWMYqk330eRQ2GaGol8wV4XKTNG13Z4TAuNlBPjGJCWH5Y7WL1K+552KzHVB6iOVcO08 55AVmN/5ZYHUIYAvIAyASgkx5fDOIR4urekkQT2+q1XXHAp2jzZ4C5l1i2LPeVg3kcQG WtsQ== X-Forwarded-Encrypted: i=1; AJvYcCULE+jmnomnyUDauU+j/27mKOeMcxZVb9BGYl9qvhpMGl9o4xP/QcpXsa+VBrNtRiKx/2/WhuARY2TWGoW31Dut6KkL X-Gm-Message-State: AOJu0YzSwiK/ios3eJBJglbqRDZeWUqD/qGv9lwzVUZKDvGwznHGLsPX s9UwR2SzAq9s4joQY61c2uohpIb8+dHXPTty2gD1t2hqZOlx6Qzt X-Received: by 2002:a05:620a:1a0e:b0:785:d97d:a4ed with SMTP id bk14-20020a05620a1a0e00b00785d97da4edmr8335910qkb.37.1708133286872; Fri, 16 Feb 2024 17:28:06 -0800 (PST) Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com. [66.111.4.228]) by smtp.gmail.com with ESMTPSA id i9-20020ae9ee09000000b0078726073228sm440440qkg.26.2024.02.16.17.28.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:06 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailauth.nyi.internal (Postfix) with ESMTP id 08EAB27C005B; Fri, 16 Feb 2024 20:28:06 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 16 Feb 2024 20:28:06 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefg heeinecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhp vghrshhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrd hfvghngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:28:05 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Chen Zhongjin , Yang Jihong , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 5/6] rcu-tasks: Eliminate deadlocks involving do_exit() and RCU tasks Date: Fri, 16 Feb 2024 17:27:40 -0800 Message-ID: <20240217012745.3446231-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107652453652309 X-GMAIL-MSGID: 1791107652453652309 From: "Paul E. McKenney" Holding a mutex across synchronize_rcu_tasks() and acquiring that same mutex in code called from do_exit() after its call to exit_tasks_rcu_start() but before its call to exit_tasks_rcu_stop() results in deadlock. This is by design, because tasks that are far enough into do_exit() are no longer present on the tasks list, making it a bit difficult for RCU Tasks to find them, let alone wait on them to do a voluntary context switch. However, such deadlocks are becoming more frequent. In addition, lockdep currently does not detect such deadlocks and they can be difficult to reproduce. In addition, if a task voluntarily context switches during that time (for example, if it blocks acquiring a mutex), then this task is in an RCU Tasks quiescent state. And with some adjustments, RCU Tasks could just as well take advantage of that fact. This commit therefore eliminates these deadlock by replacing the SRCU-based wait for do_exit() completion with per-CPU lists of tasks currently exiting. A given task will be on one of these per-CPU lists for the same period of time that this task would previously have been in the previous SRCU read-side critical section. These lists enable RCU Tasks to find the tasks that have already been removed from the tasks list, but that must nevertheless be waited upon. The RCU Tasks grace period gathers any of these do_exit() tasks that it must wait on, and adds them to the list of holdouts. Per-CPU locking and get_task_struct() are used to synchronize addition to and removal from these lists. Link: https://lore.kernel.org/all/20240118021842.290665-1-chenzhongjin@huawei.com/ Reported-by: Chen Zhongjin Reported-by: Yang Jihong Signed-off-by: Paul E. McKenney Tested-by: Yang Jihong Tested-by: Chen Zhongjin Signed-off-by: Boqun Feng Reviewed-by: Frederic Weisbecker --- kernel/rcu/tasks.h | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 68a8adf7de8e..4dc355b2ac22 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -146,8 +146,6 @@ static struct rcu_tasks rt_name = \ } #ifdef CONFIG_TASKS_RCU -/* Track exiting tasks in order to allow them to be waited for. */ -DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu); /* Report delay in synchronize_srcu() completion in rcu_tasks_postscan(). */ static void tasks_rcu_exit_srcu_stall(struct timer_list *unused); @@ -855,10 +853,12 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) // number of voluntary context switches, and add that task to the // holdout list. // rcu_tasks_postscan(): -// Invoke synchronize_srcu() to ensure that all tasks that were -// in the process of exiting (and which thus might not know to -// synchronize with this RCU Tasks grace period) have completed -// exiting. +// Gather per-CPU lists of tasks in do_exit() to ensure that all +// tasks that were in the process of exiting (and which thus might +// not know to synchronize with this RCU Tasks grace period) have +// completed exiting. The synchronize_rcu() in rcu_tasks_postgp() +// will take care of any tasks stuck in the non-preemptible region +// of do_exit() following its call to exit_tasks_rcu_stop(). // check_all_holdout_tasks(), repeatedly until holdout list is empty: // Scans the holdout list, attempting to identify a quiescent state // for each task on the list. If there is a quiescent state, the @@ -871,8 +871,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) // with interrupts disabled. // // For each exiting task, the exit_tasks_rcu_start() and -// exit_tasks_rcu_finish() functions begin and end, respectively, the SRCU -// read-side critical sections waited for by rcu_tasks_postscan(). +// exit_tasks_rcu_finish() functions add and remove, respectively, the +// current task to a per-CPU list of tasks that rcu_tasks_postscan() must +// wait on. This is necessary because rcu_tasks_postscan() must wait on +// tasks that have already been removed from the global list of tasks. // // Pre-grace-period update-side code is ordered before the grace // via the raw_spin_lock.*rcu_node(). Pre-grace-period read-side code @@ -936,9 +938,13 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) } } +void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); +DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); + /* Processing between scanning taskslist and draining the holdout list. */ static void rcu_tasks_postscan(struct list_head *hop) { + int cpu; int rtsi = READ_ONCE(rcu_task_stall_info); if (!IS_ENABLED(CONFIG_TINY_RCU)) { @@ -952,9 +958,9 @@ static void rcu_tasks_postscan(struct list_head *hop) * this, divide the fragile exit path part in two intersecting * read side critical sections: * - * 1) An _SRCU_ read side starting before calling exit_notify(), - * which may remove the task from the tasklist, and ending after - * the final preempt_disable() call in do_exit(). + * 1) A task_struct list addition before calling exit_notify(), + * which may remove the task from the tasklist, with the + * removal after the final preempt_disable() call in do_exit(). * * 2) An _RCU_ read side starting with the final preempt_disable() * call in do_exit() and ending with the final call to schedule() @@ -963,7 +969,17 @@ static void rcu_tasks_postscan(struct list_head *hop) * This handles the part 1). And postgp will handle part 2) with a * call to synchronize_rcu(). */ - synchronize_srcu(&tasks_rcu_exit_srcu); + + for_each_possible_cpu(cpu) { + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); + struct task_struct *t; + + raw_spin_lock_irq_rcu_node(rtpcp); + list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) + if (list_empty(&t->rcu_tasks_holdout_list)) + rcu_tasks_pertask(t, hop); + raw_spin_unlock_irq_rcu_node(rtpcp); + } if (!IS_ENABLED(CONFIG_TINY_RCU)) del_timer_sync(&tasks_rcu_exit_srcu_stall_timer); @@ -1031,7 +1047,6 @@ static void rcu_tasks_postgp(struct rcu_tasks *rtp) * * In addition, this synchronize_rcu() waits for exiting tasks * to complete their final preempt_disable() region of execution, - * cleaning up after synchronize_srcu(&tasks_rcu_exit_srcu), * enforcing the whole region before tasklist removal until * the final schedule() with TASK_DEAD state to be an RCU TASKS * read side critical section. @@ -1039,9 +1054,6 @@ static void rcu_tasks_postgp(struct rcu_tasks *rtp) synchronize_rcu(); } -void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); -DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); - static void tasks_rcu_exit_srcu_stall(struct timer_list *unused) { #ifndef CONFIG_TINY_RCU From patchwork Sat Feb 17 01:27:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 202554 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp93992dyc; Fri, 16 Feb 2024 17:29:29 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVzd4J4Z2HYkBLq4bWwb+87dSpkU+dKln1XRPxC6r/5U8+2gFwUiZXPi/YLCjvCnhcSTIkxl9hQtwcp7Jf5DLyHE0D2FA== X-Google-Smtp-Source: AGHT+IFKOSDalHMqbVP7piSgP1u3p+FBbz1CWxsBtfmFPbpSqelX9Tx7lH3yC7pxK8+96BoNPgRl X-Received: by 2002:a17:902:d48e:b0:1db:403c:68b5 with SMTP id c14-20020a170902d48e00b001db403c68b5mr7731714plg.12.1708133369800; Fri, 16 Feb 2024 17:29:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708133369; cv=pass; d=google.com; s=arc-20160816; b=IMKrbRuPOZl5MUf2ETcDUsP2X7hUUGf33eqcR38pBtipfcn8Wr9YbzmsQTafH6uOEs SbvvHtre6/NMYGGMECkHxSuGykNfW09t4uEtMLJC+qdBcwp9sJ+ZLuSO8iBceTNI6S8B rxX9MiusiM+IO4wiRHnz92cV8XeR/EUksTEzr+PHOnXQkXTAClVkO8PJMRV4uLuM7DOL TLCpRaR3cG2p7E6MUwM769ASVDPz5GOfUy2qEPFX2G1Bb4V+g8vXNMMNFMoCs8PyF4u1 ICBJt+1p22fpeCuMdFp4lGDKUKQsSGdxzndN+9z2tBjJcdhzkisEch3paUORtJwCu0IQ 9mJg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; fh=9Qg1YJOEWMF7QEtQ5YYQgQW0vfRLI3VVtBnP5xjwM30=; b=Q5QWN/bmM/zaeL6XFn/qcyJeT+gLDQh9Fcu45AJzxUMqQh4r5LzaMiFpTVAIFpXrC2 lJrGIwFSya/NKFhvlXGs4m+QwfG6MQh7bHI/TJ54gs4t4b5MjK5ALMOlVTP0Mood9Yg3 3bT3vc9rzJPaz0S5+b+bJ+DDYLqNgmfuR7Qm3BxOfhRqp+Wud/FCu7JAxWKxnTVznQ6Z hMawuMiQFoxIqutkyDsk2ZJoAiqYHqYil8RvgaHPwc3gyI0fEpgIU2DpPaqHWRTY1ce1 jxzz6lWpcrzSw9uy2MugsPpagYltGbMxxct5SxYUPdMJISKrmk+8xlVoRfSaMRWM+V+A b/8w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=T+stnbu8; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id n8-20020a170902f60800b001db535f1530si683872plg.419.2024.02.16.17.29.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:29:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=T+stnbu8; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-69646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 8E897284605 for ; Sat, 17 Feb 2024 01:29:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 50E211D55F; Sat, 17 Feb 2024 01:28:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T+stnbu8" Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 930521CD1F; Sat, 17 Feb 2024 01:28:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133291; cv=none; b=cb3qYWaFTahpJxShmiELn/k1Q50AEN+QgH8aywRU+cZ2WyXG/yjJXU8KZJEJ6gyjZpzVisPtAVg6lmLfq14mLp3zWqlPxgymHAB/Pia5FMLhcjCEgi7hlviz13M1mhvViR8okgGSq5X9aUoE7LTyjehklsbx6uGM7AV9WLfBXko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708133291; c=relaxed/simple; bh=KLY1+2mGq3H9diWHchRwxDC5ACzJHAgMPqkliWw0HYE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FeFsxln59LthlOtK5mn2FHdwSCGtg/1zVmajFkWNnB376sNRq/zij2BlgJk93hpVAyNrHb7+vMbF28sQPdfewp5MvlCH3A0HbaV22kk1t5ZPFqQ3bd2WSGpJMLINheVv7a3lPkIBK74kq5hHVjjT4kEck+qfOOe9Sqc0S28LLW8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=T+stnbu8; arc=none smtp.client-ip=209.85.219.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-dcc80d6006aso1353037276.0; Fri, 16 Feb 2024 17:28:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708133288; x=1708738088; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; b=T+stnbu8W7xsXBaJXXb5IZCo+ZKzT0/hgW2KW/n0IXmzvJTlvXAt4tbeItEsPJ5lQl CnwoLlKGDIt4tY3nXQW4TeZAl17otodsn8hZpe99QJVNcBScvghsVPk86kClrMX7/Lh1 rzFVrnHjZViROBcdKs53jVj8h31pX1mhM29IfBC5LSgYQryCBFy1TZGop/NOcdplQD1u jNl5jB9n0VtFZNAyQGJE4owG5XTO4IsVMBKgSJQaAKQCdqH/RTqEAbzGuU4s4H70llyK ZDt581xJ9k0Vy+ZecbMkOzmI9Pjh55bLwb6TfSG/Y5yQk2m1JNk3ys9XbFNZ+7Vz7ytk NGmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708133288; x=1708738088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=T5Wf+uCUWSLEUPm4v8/DfTpOrC7IR1g418bIMiGKo4M=; b=H2/eyaXay2yfiCOf+fKt66L3wwUlQl/1S9lF47ZPVEkDs2amuOCSEAyRnOCjsRQ1hX KDmnbOcAfQghDqWAdzE098ZtnLh4oQLuRl2GlgqWsk8sqxGWjauTUO+mfZ+KWzoCVQ4R L/6u6gM4HOtcDKcep8JSl0wKcfgt0ryHw5INqvQ+HaG3ci/eZHCaUN5uvHf9AEtJnfF8 v3SMyLdxkjuuStvuiSX1xDT/UDBHrakDNxUFflhbXgApKDB57xEbkg184/VggcVxVM/h DTwd15Q9owOhinp5VsCuW/peuwmN0KQCRdIrYIxEKYEUeLCCtp9yj3kyT5W+Wdk2l3u3 VTSQ== X-Forwarded-Encrypted: i=1; AJvYcCXWOrXkpdP2sFHkBKnIPnst0rCIGJ4B81h4D9roQ7bkkvnPzb3gGggAQ2N9dxqVuvLZSBHY1LWGhiQpw5/BBrGq7IyR X-Gm-Message-State: AOJu0Yy9x9esI6FGLGIL58KYTcpM3R4RAm/6DChjHw75JKB2VI5DMvSQ 5L0z4Ss6mdpfl6GIHnfKA5nqs9OrIyjpgJoDxwGUQCVSh4vZl1o/ X-Received: by 2002:a5b:708:0:b0:dc6:cc35:35e9 with SMTP id g8-20020a5b0708000000b00dc6cc3535e9mr6286134ybq.35.1708133288319; Fri, 16 Feb 2024 17:28:08 -0800 (PST) Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id y23-20020a37e317000000b00785e016ec39sm442118qki.42.2024.02.16.17.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Feb 2024 17:28:07 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfauth.nyi.internal (Postfix) with ESMTP id 849BE1200043; Fri, 16 Feb 2024 20:28:07 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 16 Feb 2024 20:28:07 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeg gefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsg hoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieeg qddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigi hmvgdrnhgrmhgv X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 20:28:07 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Neeraj Upadhyay , "Paul E. McKenney" , Thomas Gleixner , Sebastian Siewior , Anna-Maria Behnsen , Steven Rostedt , Boqun Feng , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 6/6] rcu-tasks: Maintain real-time response in rcu_tasks_postscan() Date: Fri, 16 Feb 2024 17:27:41 -0800 Message-ID: <20240217012745.3446231-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217012745.3446231-1-boqun.feng@gmail.com> References: <20240217012745.3446231-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791107655970418374 X-GMAIL-MSGID: 1791107655970418374 From: "Paul E. McKenney" The current code will scan the entirety of each per-CPU list of exiting tasks in ->rtp_exit_list with interrupts disabled. This is normally just fine, because each CPU typically won't have very many tasks in this state. However, if a large number of tasks block late in do_exit(), these lists could be arbitrarily long. Low probability, perhaps, but it really could happen. This commit therefore occasionally re-enables interrupts while traversing these lists, inserting a dummy element to hold the current place in the list. In kernels built with CONFIG_PREEMPT_RT=y, this re-enabling happens after each list element is processed, otherwise every one-to-two jiffies. Signed-off-by: Paul E. McKenney Cc: Thomas Gleixner Cc: Sebastian Siewior Cc: Anna-Maria Behnsen Cc: Steven Rostedt Signed-off-by: Boqun Feng --- kernel/rcu/tasks.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4dc355b2ac22..866743e0796f 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -971,13 +971,32 @@ static void rcu_tasks_postscan(struct list_head *hop) */ for_each_possible_cpu(cpu) { + unsigned long j = jiffies + 1; struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu); struct task_struct *t; + struct task_struct *t1; + struct list_head tmp; raw_spin_lock_irq_rcu_node(rtpcp); - list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) + list_for_each_entry_safe(t, t1, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) { if (list_empty(&t->rcu_tasks_holdout_list)) rcu_tasks_pertask(t, hop); + + // RT kernels need frequent pauses, otherwise + // pause at least once per pair of jiffies. + if (!IS_ENABLED(CONFIG_PREEMPT_RT) && time_before(jiffies, j)) + continue; + + // Keep our place in the list while pausing. + // Nothing else traverses this list, so adding a + // bare list_head is OK. + list_add(&tmp, &t->rcu_tasks_exit_list); + raw_spin_unlock_irq_rcu_node(rtpcp); + cond_resched(); // For CONFIG_PREEMPT=n kernels + raw_spin_lock_irq_rcu_node(rtpcp); + list_del(&tmp); + j = jiffies + 1; + } raw_spin_unlock_irq_rcu_node(rtpcp); }