From patchwork Thu Feb 1 01:40:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195121 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp144702dyc; Wed, 31 Jan 2024 17:42:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IF5muy2e86w8WlMnrEMGRYIyJNZIWVUU6D1FYYpk4PH4RwZTKwDhbXkPn3w8XFIVraPgaMD X-Received: by 2002:a05:651c:893:b0:2cd:3731:9c39 with SMTP id d19-20020a05651c089300b002cd37319c39mr465730ljq.33.1706751770103; Wed, 31 Jan 2024 17:42:50 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706751770; cv=pass; d=google.com; s=arc-20160816; b=HiARBmS8fhmAjJOpe0a7OUetG7zPd5s1ADgJDKqQIMS4EKmXfj4Qhhx/3Z+bXbMghy UsKM4+5zw3nPw38wINZSo4cMGQxT+6sudnHLK2TAvqAjvaG0HhDoHf+Fk7NVFvW4Zy5u /5Wp6ZDJ25A7qPP2x1AqCvzw4kFZ2nfzEP4OVEt9Dp87wb8kJ1jEimllNDARcdQVOlhy P0yFujsybr3yq293D7njSS02RDgye9/G7K0S+og11y4EBxxfYfZNM4iDweM6jfLwHUHT X9RYPkfuYlPyJL0MiCHob288fwJ/0N9jniNfinDSjUk63rtz0nOYM6jhXZLjM7e+3FBO xuLA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=zLJWc1KOEuIrLOS4qhIHcNcluYpoORRWU7wyfjkuepk=; fh=rrnuCDxbc+zYZ4VKxX5rBGE+q8B8uaG8Y9UhPUIFaZc=; b=NVL64Y8gCBt/lhkKwOvUaaotCb/ZlA+SG6rbwoE5WlU6LC6vDctjvp1+RPzUhLfJc9 GZ09vWModcTSFtnSPNrvaqZeeKVE5WLjtAFka9sCg2R3483NozwcrU1CV29hRsH0Sbh3 bTfWebGB/4UOEztMR1lF7aaSwKd5p+NEIWUrnEGKvWfZs1Grhp7eyGJs0PfR99jbH2cn cnC8EOWiuuON1Rnwq48GnMbXa306899u8O0ExeABWuoKa9TO30F3IyfiSqoN93UiXfsy txJ8qKnk4N4SEAENp9oVCOsN90MLHCbwGmJOlIlmgT9m56vBZ3rrxVXocyhOyXN+VcV/ oQAw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="VoUBmB5/"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47507-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47507-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCX0w/frxZ23QLmYJepsU3oJBxgsTZgDU6gF+bvuhJbBrXjEfR06Wj1YSRJ+f5ZvjbiujGfi+hLBAg2J+RHgBQeMzOp90w== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id m25-20020a509999000000b0055efe9d3fb7si3627767edb.364.2024.01.31.17.42.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47507-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="VoUBmB5/"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47507-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47507-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 7F32D1F2E32B for ; Thu, 1 Feb 2024 01:42:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 98EABDF6D; Thu, 1 Feb 2024 01:42:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VoUBmB5/" Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D06AA443E; Thu, 1 Feb 2024 01:42:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751734; cv=none; b=q/Vgdb62OOhhRPXen2cwqyMSwA33hNHO6dDT2ttt/izVBfMMJTHonFkx7rSTbFh5YqM7Dk35goRjryN98M6VBBJ36LhqHbyI/hs9E+rNtOm0IhMmucpqbd3+Xyw/xl0a1pMB5kRTXEf11CFXNWaG6wbmVFB2eS6q5VKI0xyJ0eE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751734; c=relaxed/simple; bh=JYMQcq8E1zGYMJ0i7nVikl7w2oNjNCpEj0nSc8NMMoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dm0FWI6r0KGjYGa1deE/wSgOfEWEMUM+po3Pg5tyE9CmVAf2ITqPeV4xPRMjTNDMv1DnX8tFsNkbOYqBumhwUChgNvna1v+wTkyNGFu5YSgMS//m9grFiQfGAtY0D37mOMJqwx4AlBSA0CEJc7UdIOZKwT/hmLrHeaZrwHJgBGI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VoUBmB5/; arc=none smtp.client-ip=209.85.219.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6818a9fe380so2470446d6.2; Wed, 31 Jan 2024 17:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751731; x=1707356531; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=zLJWc1KOEuIrLOS4qhIHcNcluYpoORRWU7wyfjkuepk=; b=VoUBmB5/NTIxGqwdwplE+GmmEsEDc2b4PLVocFyysIXCoTcVyuidpPnpUmJLuDS8rY KkEZpeM47JbigKSk1Z95LmgV/8xsbSI0GlfV9DGD2Gc66+LQwlo/8au/T6KepqOO56Lv vzlPzNo66u2vVMdCBXoQAqMwGuD1jBIlheBzchEjOJ9M2eOW7IZ9YotCAVrN0KpHrHmD koryuz7qn+g/OhMG/WOZfBi8XLmCiDkgWQhtIsnBT1PTwItossmRtbECoPNWZMHvsVFM ovT9ccqZvECnw73ughgvFQz47memrI0uUgaSroUKcCxMwcfoIURILB55/M/7MxllgnTg K6eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751731; x=1707356531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zLJWc1KOEuIrLOS4qhIHcNcluYpoORRWU7wyfjkuepk=; b=awoPAK5pSLUx81swMxemKimGPD3rWEo5xEf4n6vkCPc9eqm4lCf06a2nOAaH3FJ9XU eRNOy12SlcbOhimMD7DP11MeDNRE+owhupQAPeijg5gbMDvp37MBFU46ujIjRbLce2u9 ugXq7Nmq7FSl06GTmjIkkhOWQT+oGngmMTDUhSYu8v8CE4vlfgE8dPurBY159aKapRmF nyKmOstW2PYY6hSwDUp9wOGTQvdVHJZzsXSPuORy1gmaHt6XMTEuAhOHyCQ340SFMfGB wdCpr5rljxU+VMNsAqOrXD/3s7JxCGepMAxHJtEANFciWCBEDp/uVos2nxvcKyhue/or ZlTg== X-Gm-Message-State: AOJu0YwKS77FCzXQTFvI2aYDFnU+URRZ/esyLkAVEMV/eQKHdRfCwpp2 AEBfK2P3Dsj8l06vPgHyfB5U44B+bfaoc7FoJFDvwQF6vBfXciBr X-Received: by 2002:a0c:9a91:0:b0:68c:7a2f:1c0b with SMTP id y17-20020a0c9a91000000b0068c7a2f1c0bmr461906qvd.62.1706751731594; Wed, 31 Jan 2024 17:42:11 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUptm6AUOJltqfcS2AtCtlsJsb1yHlur08z+RnNtjG+RLixsXaPqKBfcIAfMkLUGEQRDMiOE+x5UtJKVII8pPqVKwzgwpxE4HpTJVl9f65F3Kvh+5puy/8pobwzkXOpTU+mE2IEu2YDSmhK2TSr4l3KcFwZ90Pj6nF/a50lL5X53tYcx3hCGl07UGR4dkvb22ZarKUiQaxOp/0civ0rzoFHo2hC+BCeL2K0w4aRuZhGkJV9fgvIzZxyqYLGeZweWkqkJSubiIJFhSfNWLOM6h8trZFPpTIoBRx7KzaigUXdXukNE3g3p8PYSeY96ZNz7hdrJqVkY75zerOko9dhV0GxE1iJr5Iza/r1fg11lxK2gtOqTnr+BNmTSjbtTP79 Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id pc19-20020a056214489300b0068c6e9fa24asm1018586qvb.10.2024.01.31.17.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:11 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailauth.nyi.internal (Postfix) with ESMTP id D58CA27C005B; Wed, 31 Jan 2024 20:42:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 31 Jan 2024 20:42:10 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:10 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 1/6] rcu/nocb: Remove needless LOAD-ACQUIRE Date: Wed, 31 Jan 2024 17:40:53 -0800 Message-ID: <20240201014100.3204532-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789658944020318173 X-GMAIL-MSGID: 1789658944020318173 From: Frederic Weisbecker The LOAD-ACQUIRE access performed on rdp->nocb_cb_sleep advertizes ordering callback execution against grace period completion. However this is contradicted by the following: * This LOAD-ACQUIRE doesn't pair with anything. The only counterpart barrier that can be found is the smp_mb() placed after callbacks advancing in nocb_gp_wait(). However the barrier is placed _after_ ->nocb_cb_sleep write. * Callbacks can be concurrently advanced between the LOAD-ACQUIRE on ->nocb_cb_sleep and the call to rcu_segcblist_extract_done_cbs() in rcu_do_batch(), making any ordering based on ->nocb_cb_sleep broken. * Both rcu_segcblist_extract_done_cbs() and rcu_advance_cbs() are called under the nocb_lock, the latter hereby providing already the desired ACQUIRE semantics. Therefore it is safe to access ->nocb_cb_sleep with a simple compiler barrier. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 4efbf7333d4e..785946834c6b 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -933,8 +933,7 @@ static void nocb_cb_wait(struct rcu_data *rdp) swait_event_interruptible_exclusive(rdp->nocb_cb_wq, nocb_cb_wait_cond(rdp)); - // VVV Ensure CB invocation follows _sleep test. - if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^ + if (READ_ONCE(rdp->nocb_cb_sleep)) { WARN_ON(signal_pending(current)); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty")); } From patchwork Thu Feb 1 01:40:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195126 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp146166dyc; Wed, 31 Jan 2024 17:47:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IGRyojkXbLkPNoiOY6v8nybKP2MJmQhfdK/HZmaU+PRVxjK5yftJT0Scs/C1SAMfHoL0i3Z X-Received: by 2002:a05:6870:5aa7:b0:214:876a:a517 with SMTP id dt39-20020a0568705aa700b00214876aa517mr910002oab.35.1706752022222; Wed, 31 Jan 2024 17:47:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706752022; cv=pass; d=google.com; s=arc-20160816; b=O6lOSpEo4gKLbKTDhwbdJ7xW0XS5qxG0t4xn4vK09ZQFT6VZlXVBFKdUoAX/UthnnX BXyJdRAjLA1EXD8pQCJdDD+uDwICkuSxAW5/SBgTNLYZLwkwAJBzkIpKhr6LAmviXmz4 d/UoCNxUHzHWhTvWrSLJrXDSFBf+2hd28W3yAZKOq230gywTsy0TC4fbep+fJybs7YdO 2p7BG8dkndeakdofdpQ1ZNldl+wOVIHC0lnq56wOBfjxWbvAsYQYa+qwRq30H7/gu/fJ /GqtxAAtkBsHIP1RCN3k45jRjK9bVGQrRLABRMe9ozkoPis3GaW5r8fer25gCMcgL4MA u53g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=p4HlLJA6ewOwBM3r/aagQRFX9+hzF0ZwosHf3q2R4gw=; fh=qFVmf7YyooynadLtDXQHJxG4NCATav9Mkr65ZNKxd/A=; b=K7WkyLCVFCSoUOWAjU32oI/hrGUqLhrYr7Hz/w+xtj8S/UFT0vcP7ZSr/54xXtQ+sB OivI2evWQ+7833g5uBWE0duF1+udJxkFDVL6WKxipNWQzckzzvS15vhcNrLLwtvM8EsS TREj1VoJpkgXrWelsXYOlbf51Xs2nHFwnHD7t40aT0m/aIJKRTdhUqZ+70ghzoBycR+J hm3/GVHurYR/6nFUIj+L81YVaCsxm1YYYUF35rUQFSyhqWjtymRDEwpjQ18cHalF3neI 7LNTvy1Fv724AqoVGCa1UOtnsPEPrXtpRv+stiK8187BoqMFW2/TBA8j9wvSEEI7K4Iy QsyQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=hv+ft3aY; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47508-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47508-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCW58prrG2U1Jmzl5A+zYqRPe0NiVCy0GLdmvi0rJkNYx39oWVfa9wphSMV7l4Z/pPy8O97w/u43CmEqaUpsbaDuU4G5ZQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id r4-20020a632b04000000b005d9b919b948si4269408pgr.379.2024.01.31.17.47.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:47:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47508-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=hv+ft3aY; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47508-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47508-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 1CD6B28EBC4 for ; Thu, 1 Feb 2024 01:43:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2494E18C2A; Thu, 1 Feb 2024 01:42:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hv+ft3aY" Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD5195255; Thu, 1 Feb 2024 01:42:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751736; cv=none; b=HrVu/gbVvO74T15ThLIowdFrCIQ8pKGl0NSfYHWSqq1JvVLG2IBF1w4mauCO5yVy//MUhq4hB2cBexA95/a/Mp0rOUGpsImz8gU+rHfgTsdU+892zvtRHm9ek/2jbCUFavDxNFLDndGTgyOmECQa0+urUMFihiazbY1ONlRIBoc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751736; c=relaxed/simple; bh=pSSj7EvIHKtcO8CNxnXkeBO80f/T+be+cHyUt2QsZkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GNbExlc3auSM7UhfiIfLHe/Wt93ez3p3MwiTz+jcGVtbTUzcyPeS+PyJHJiuBADwJi0pf+oledVSvph08oHeFgoRTqnPHg+jF+2sYLRZdENNMYAowHfFP1gDuAxoAL6e0ff/mDfhovN2KhL3FkmjxYY3raHWSWEM9nhl8IcJwOc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hv+ft3aY; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-42a99f34cc8so2367231cf.2; Wed, 31 Jan 2024 17:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751733; x=1707356533; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=p4HlLJA6ewOwBM3r/aagQRFX9+hzF0ZwosHf3q2R4gw=; b=hv+ft3aYUzIpq6tWxxvhor34MCO+ATd6ORApMYmwOMfN6IHrGPz9g+QGKQ0ibUUmWf aEnUvZYs4fkKFrsicTlHd6WW8nAuXVXbEdJ+qv0iAXj9+8i9K87jTMkpvGtH8YCCj7hk 66byqc6tYj72H9mz9wYTz1+cWnuqWVK+mPVGOoT5JTB1RoKC3V+tPeQObG6Z8j49leIr Ir+II3ndh3TDfYY+4y5MafXPIhzA0nGT/kdFk9kiPHeiBPxfocNgJjsQ4xG8KBDmiDkI DP0qWUUiwceLl02pPB2LWG2Y63QmSZcyA6t9d734tr0++Xv1Jhx9Zw2wYjAahx586ji9 kHiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751733; x=1707356533; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=p4HlLJA6ewOwBM3r/aagQRFX9+hzF0ZwosHf3q2R4gw=; b=EHsreN1XVT87/jeKPPqs4eRihJKB6tN+X8ewsJes/Qr5tuuwxB2QKvzhAiAbu3qsje fOMzHQNcpzwZj/7zuWCsC9ny5YyCpPEcHmzx/MXsVu/KRzO05TDvwtVFOI5jqaio90Rv +sNIAFBSVjahp9mA0dBONpOk5xPlAE0UevGwc3yGxOG52NziLjgYRX19aokKgDeclNkM ZiBgQKR15+iIU0ir63TrOHFvFSBQWIn9NF0bJbFgizI57V7qXLpvG9DW9TwVYqSxdC17 kO4BDNurLHUfXeCYNMRVxTcsi22EdaG2nVXN6T1ynwaykl/1gVBefdhqpgGuYacCUWf6 9mGw== X-Gm-Message-State: AOJu0YwlHYab+2tivuV12uOILRNmLijDm8/u2cApb4TB8LOadAEruMFF HkPV64cHwAcWrNWczjYQKpoN5VIg0WIJgD3ELSQszwVkVrHxFkxA X-Received: by 2002:ac8:5795:0:b0:42a:a934:1f4c with SMTP id v21-20020ac85795000000b0042aa9341f4cmr1433192qta.2.1706751733703; Wed, 31 Jan 2024 17:42:13 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXD5i1//U9u8ubT2vzi0K2o+/GwzeSTAQDt4PPCY2iGVlZvaeJzwvA34bwkEtmcQoMOulFFSQdo9AD4Y/ZpmEHbfvf7mb9DFbXm5iGTLa+ZqJze9TZ87NASpoXm26/h30XAyDauZldtN8u51C0sdQe4lkp5HFIyiYP07OpR7boPbMxcPqxXbTORJ0tnPUgwZkZZ3WIl5SCBwjLCM0NEA8mKiKQZKcAiF+J6o3sX9SxmZ0Br0zS2qnpXJ/dt23lJtos2fRlTDlJDclZrhbr9PA7E+LmSYtBwJSyNV3wZYBV91/r/cYwcLJ1E4rIIpSl/YR24UWwstvBtpL6ZRdBAAqwE6a7QhL6qjAl+kadWlIlJdpZlXufQiME1Tg09ektM Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id fv6-20020a05622a4a0600b00429be14d3bbsm3638748qtb.13.2024.01.31.17.42.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:13 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailauth.nyi.internal (Postfix) with ESMTP id E295F27C005B; Wed, 31 Jan 2024 20:42:12 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 31 Jan 2024 20:42:12 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:11 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 2/6] rcu/nocb: Remove needless full barrier after callback advancing Date: Wed, 31 Jan 2024 17:40:54 -0800 Message-ID: <20240201014100.3204532-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789659208624483011 X-GMAIL-MSGID: 1789659208624483011 From: Frederic Weisbecker A full barrier is issued from nocb_gp_wait() upon callbacks advancing to order grace period completion with callbacks execution. However these two events are already ordered by the smp_mb__after_unlock_lock() barrier within the call to raw_spin_lock_rcu_node() that is necessary for callbacks advancing to happen. The following litmus test shows the kind of guarantee that this barrier provides: C smp_mb__after_unlock_lock {} // rcu_gp_cleanup() P0(spinlock_t *rnp_lock, int *gpnum) { // Grace period cleanup increase gp sequence number spin_lock(rnp_lock); WRITE_ONCE(*gpnum, 1); spin_unlock(rnp_lock); } // nocb_gp_wait() P1(spinlock_t *rnp_lock, spinlock_t *nocb_lock, int *gpnum, int *cb_ready) { int r1; // Call rcu_advance_cbs() from nocb_gp_wait() spin_lock(nocb_lock); spin_lock(rnp_lock); smp_mb__after_unlock_lock(); r1 = READ_ONCE(*gpnum); WRITE_ONCE(*cb_ready, 1); spin_unlock(rnp_lock); spin_unlock(nocb_lock); } // nocb_cb_wait() P2(spinlock_t *nocb_lock, int *cb_ready, int *cb_executed) { int r2; // rcu_do_batch() -> rcu_segcblist_extract_done_cbs() spin_lock(nocb_lock); r2 = READ_ONCE(*cb_ready); spin_unlock(nocb_lock); // Actual callback execution WRITE_ONCE(*cb_executed, 1); } P3(int *cb_executed, int *gpnum) { int r3; WRITE_ONCE(*cb_executed, 2); smp_mb(); r3 = READ_ONCE(*gpnum); } exists (1:r1=1 /\ 2:r2=1 /\ cb_executed=2 /\ 3:r3=0) (* Bad outcome. *) Here the bad outcome only occurs if the smp_mb__after_unlock_lock() is removed. This barrier orders the grace period completion against callbacks advancing and even later callbacks invocation, thanks to the opportunistic propagation via the ->nocb_lock to nocb_cb_wait(). Therefore the smp_mb() placed after callbacks advancing can be safely removed. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_nocb.h | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b2bccfd37c38..d540d210e5c7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2145,6 +2145,12 @@ static void rcu_do_batch(struct rcu_data *rdp) * Extract the list of ready callbacks, disabling IRQs to prevent * races with call_rcu() from interrupt handlers. Leave the * callback counts, as rcu_barrier() needs to be conservative. + * + * Callbacks execution is fully ordered against preceding grace period + * completion (materialized by rnp->gp_seq update) thanks to the + * smp_mb__after_unlock_lock() upon node locking required for callbacks + * advancing. In NOCB mode this ordering is then further relayed through + * the nocb locking that protects both callbacks advancing and extraction. */ rcu_nocb_lock_irqsave(rdp, flags); WARN_ON_ONCE(cpu_is_offline(smp_processor_id())); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 785946834c6b..b2c3145c4c13 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -779,7 +779,6 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) if (rcu_segcblist_ready_cbs(&rdp->cblist)) { needwake = rdp->nocb_cb_sleep; WRITE_ONCE(rdp->nocb_cb_sleep, false); - smp_mb(); /* CB invocation -after- GP end. */ } else { needwake = false; } From patchwork Thu Feb 1 01:40:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195127 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp146257dyc; Wed, 31 Jan 2024 17:47:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHoiJgiaXdsGy3EamtqJ++9q7A0pbEKynNl6KknavI8Ezu0dFWEpw30905CQPDkDKvESlP X-Received: by 2002:a17:90b:18e:b0:290:16e6:deeb with SMTP id t14-20020a17090b018e00b0029016e6deebmr3599783pjs.47.1706752036440; Wed, 31 Jan 2024 17:47:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706752036; cv=pass; d=google.com; s=arc-20160816; b=ws9HAhSw05ywRzAYeGOizlIevSQnFRLzXRhtXQbRTDTL8r46diTU3CpZMrGzGtBOjb osCtFnWrvGFUpuj1QR8Nd6LILNnGWb9fQm/555Ha5pMCEWtsq0a8KTm28W6GooyMEPoR yzS5/y482tDISnxUgHWeqbu8N40cHkGDcW60J+Ufy1s3TLhWgQS4Fs/ikqsRAPzcnKqG g9ef5nP04YEyPX+UExQbUyd6VQ+VQdGhTAdpHrCceZSfRQovWvDnU231wpCf+qxvUIIH aKsnMJpyiQfXHlUPuinShhk24gyr1R9uKLyBryOYSSf/CTjUuOwPmiGeWv6vqLBZTDN+ Ndlw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=98uYtdjZqt8mPN8u771GmaA9N12/7R+F5RUUjJgYUPQ=; fh=t/n2FvjA2au6hEZ/pj/X4tWoqBu1jBYorsXwa95LCYc=; b=WLVm8dQnIIRkad+d3A3Mw/smvxkQJfmkvJssv4rACji+0ix4029QDS8Qc958xob/z5 tm4Qbqm614S0iYUBqmVDg08WuPib1wON97EKOlq/wjLlBITd/ztJINlTyG5vhqxk8fht AGu05Cc3qY/qrsB/z/QXh/LU3W91J8puY6888z6D75q8tDBxVwgHm9eJBDFm8iEdzamE 9If6j2wkSrs/OynV2f5ColgUkwmnAUozunwWj/WvMbNoh8lqXSLnOEG6P1s5YF7CpLf4 qKdFkRvhGEFdWhJPDlOY6ty5IX9ui+KMQmtV3aOqPM80RQhpq/7HBdooTHhtp27WVzwc MoRw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LZQUa7Mh; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47509-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47509-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCUGiuDqWvFvGzOno5CWXLl8uyEe8ycmvHC4cm/P4McrB0Ts2lqUDtjrVjS5GoaVuR4LNQrjg7fT5rSWJ9TXF5TVd0VGtA== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id a12-20020a17090ad80c00b00295b65faa82si2570339pjv.80.2024.01.31.17.47.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:47:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47509-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=LZQUa7Mh; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47509-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47509-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 18A2728FA5F for ; Thu, 1 Feb 2024 01:43:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4165A27447; Thu, 1 Feb 2024 01:42:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LZQUa7Mh" Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8247463A0; Thu, 1 Feb 2024 01:42:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751738; cv=none; b=WTTEMcTbp2/WHaBh3yj9NKgmuqdF1grpd9AoljSbUSnNXlCy6dwuoga4twi+rEHGRlvxUL+WiDYtLabD4M/hYUeWfpbTU4b4JeZesBR1JfwN6ta2Y5b2M/pMJod4D8G/aHy11e1MA0aFvKwiFGj50mglnMU9R0sswdrcRLgjyEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751738; c=relaxed/simple; bh=URsVmEp6Jr873vr5jenxJaWo72Ku3rabQQ+gAmu0f1E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gXGpFfytrlHZ4wevifCyEUNggv6h7755Jp9lQpPszX3R7zB26QBv7/9/J1xeqFgsh3azPkHe3ypVsxDlT+vJPeyw8RwoCN3PAeJmEa3r8RCYXNbGAEEmO9Ebd//95M2e+os9sMnnWjChiBUFhWCaIlEkIsGB173lGOBvDddf/t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LZQUa7Mh; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-68c44100327so2017606d6.2; Wed, 31 Jan 2024 17:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751735; x=1707356535; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=98uYtdjZqt8mPN8u771GmaA9N12/7R+F5RUUjJgYUPQ=; b=LZQUa7MhUF0BNfXRKevudcd1zOzldfHlWvlQbbyG94tvN4k+wYl1veVoII+0e/Yo8g HMYRv0PEFk6NEyPDcp1XWYHRIjggnCGXunk1E8Sc5xAMy2zZgVjnA21BHVHWFcQQXmhK dTDxePnOo1zHb982AgdeqK2zga4Ixc33cSGAKXspn6gz5tntGnqfLshVYxnYg1HAL2SJ 6jw9kWK2y/638TNb+IA8Qy3arkGXLf8L0ekD3l7QfjHhKkTWx6uPUO0KyMI/0eNT+fox Qwc+Bhe1tpd7LMURQMoKOjeXuKPoLtXUyIIWzbcIhfIqhT6Bdxj6eVNeExqqiijXNc8z TgqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751735; x=1707356535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=98uYtdjZqt8mPN8u771GmaA9N12/7R+F5RUUjJgYUPQ=; b=lOpcmiQgWDgf2rQ1fU0jywC3TFB/0CnokAN3nN1L2P2rnI9lIjItS/eQOJU9y5sU+8 KDQbuIScdXrNYppvZyTsUWReP+7LOSkqblhHaofaXdL0vFYu913RpY8CqJkd4lUYgkZy /oiaUszXWTch5u4sKFZUh2ZP+4tKscyM5fURFVpD/pilqOjY2n+lYzGSr3SDIDYfr7x+ QR2AbegCpMcqpF6zg6Z74cLYYXkOPa/ZPTSv8henDp0Nk3geFVOtgXFhJXPOmgCxLTWb QAPLHQQLO9yRcpPjYILrXrgzSNTtJtyuY1yPq/CBXW+x4az/EMegpY7aGOG0IOpKrT9g 1QdQ== X-Gm-Message-State: AOJu0YyxIrOgtmw5bVghEZLpt4rkJbEDwsi++/S3woE2qcL/f1ncUSV2 hkYMR/5L04X1zUFaouEeyhwImNby1QaiJC/YrLmZgk0coMjQKKFo X-Received: by 2002:a05:6214:20e6:b0:68c:4caa:dd17 with SMTP id 6-20020a05621420e600b0068c4caadd17mr4015452qvk.6.1706751735429; Wed, 31 Jan 2024 17:42:15 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCV5H58O5j3sQdOL+vJNGvEAGH3Se5j3EHzm0jbl5Emvk7HuvTTSdqJiI8P12gEB7VHZI0tMOi+oHbFCB8tcfbHWoqSr8eVDiLAz+Nd1kH7UNxPM7BHiMTLWibDtND+WY1s2/dkmpcbkiSfKmfPxhMAehNYm3ygbG6ahESqeKE4zHyPaWiAuEUaNs/jnIT5MUD16IcTUJIpCgFIkaRQzsXPy7rqnG/kNrzZv0WiwgmNvEyS4fcO0d+4c5AUReRm4cgeacFQueuIrzyC/+iBOzlMV4Gubck2GXuZ44mIvqVIujzNMIJdyoFJQChJh0gpSVfn+E3Kvcv5Gyn3BCxOBeg0uJ9O73QE/G27CDzlpDLfnFTMQcJIEvV5WwYdV5Zl9f+J/jaG9DSbD+g/+o4xRXKXUMxr6moZVUw== Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id on6-20020a056214448600b00685ad9090basm2453293qvb.97.2024.01.31.17.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:15 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id A91B027C005B; Wed, 31 Jan 2024 20:42:14 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 31 Jan 2024 20:42:14 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:14 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , Neeraj Upadhyay , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 3/6] rcu/nocb: Make IRQs disablement symmetric Date: Wed, 31 Jan 2024 17:40:55 -0800 Message-ID: <20240201014100.3204532-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789659223551326365 X-GMAIL-MSGID: 1789659223551326365 From: Frederic Weisbecker Currently IRQs are disabled on call_rcu() and then depending on the context: * If the CPU is in nocb mode: - If the callback is enqueued in the bypass list, IRQs are re-enabled implictly by rcu_nocb_try_bypass() - If the callback is enqueued in the normal list, IRQs are re-enabled implicitly by __call_rcu_nocb_wake() * If the CPU is NOT in nocb mode, IRQs are reenabled explicitly from call_rcu() This makes the code a bit hard to follow, especially as it interleaves with nocb locking. To make the IRQ flags coverage clearer and also in order to prepare for moving all the nocb enqueue code to its own function, always re-enable the IRQ flags explicitly from call_rcu(). Reviewed-by: Neeraj Upadhyay (AMD) Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 9 ++++++--- kernel/rcu/tree_nocb.h | 20 +++++++++----------- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d540d210e5c7..a402dc4e9a9c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2735,8 +2735,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + local_irq_restore(flags); return; // Enqueued onto ->nocb_bypass, so just leave. + } // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); if (__is_kvfree_rcu_offset((unsigned long)func)) @@ -2754,8 +2756,8 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ } else { __call_rcu_core(rdp, head, flags); - local_irq_restore(flags); } + local_irq_restore(flags); } #ifdef CONFIG_RCU_LAZY @@ -4646,8 +4648,9 @@ void rcutree_migrate_callbacks(int cpu) __call_rcu_nocb_wake(my_rdp, true, flags); } else { rcu_nocb_unlock(my_rdp); /* irqs remain disabled. */ - raw_spin_unlock_irqrestore_rcu_node(my_rnp, flags); + raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */ } + local_irq_restore(flags); if (needwake) rcu_gp_kthread_wake(); lockdep_assert_irqs_enabled(); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b2c3145c4c13..1d5c03c5c702 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -532,9 +532,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // 2. Both of these conditions are met: // a. The bypass list previously had only lazy CBs, and: // b. The new CB is non-lazy. - if (ncbs && (!bypass_is_lazy || lazy)) { - local_irq_restore(flags); - } else { + if (!ncbs || (bypass_is_lazy && !lazy)) { // No-CBs GP kthread might be indefinitely asleep, if so, wake. rcu_nocb_lock(rdp); // Rare during call_rcu() flood. if (!rcu_segcblist_pend_cbs(&rdp->cblist)) { @@ -544,7 +542,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, } else { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQnoWake")); - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); } } return true; // Callback already enqueued. @@ -570,7 +568,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, // If we are being polled or there is no kthread, just leave. t = READ_ONCE(rdp->nocb_gp_kthread); if (rcu_nocb_poll || !t) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNotPoll")); return; @@ -583,17 +581,17 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, rdp->qlen_last_fqs_check = len; // Only lazy CBs in bypass list if (lazy_len && bypass_len == lazy_len) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, TPS("WakeLazy")); } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp(rdp, false); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeEmpty")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE, TPS("WakeEmptyIsDeferred")); } @@ -611,15 +609,15 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, if ((rdp->nocb_cb_sleep || !rcu_segcblist_ready_cbs(&rdp->cblist)) && !timer_pending(&rdp->nocb_timer)) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } From patchwork Thu Feb 1 01:40:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195122 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp145080dyc; Wed, 31 Jan 2024 17:43:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IHHUz1PeT+FcoD7Pan49ow+GtCDDN9UWugiwCXYaVDo43vdg4v0I71MUzSAD5c0UUtSfEqd X-Received: by 2002:a05:6402:35c2:b0:55f:adc6:47e8 with SMTP id z2-20020a05640235c200b0055fadc647e8mr559261edc.7.1706751832170; Wed, 31 Jan 2024 17:43:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706751832; cv=pass; d=google.com; s=arc-20160816; b=T/K1JkH4yfDTLB82wnGa/XUpjJTBra/pwByrAiXe0WdYohfm0y+rPArIeanfVoi37z 5AWl+a+YT27+f4nHX6A2CzpxL/TmIvsmL+vfhZOyQ7FH9eddKT8MrC0+GP75gs8i/9U8 jQHHOLv1C+6oMVngPAVGNbS+OkHtF7rhjVfAIfxvEyWUbaGqC6o3QcAn6Cj1HXsoW+Ma maqNIWYVLbO9hl/1eWMGp1SMKj6OPAb8lCzp0ZuFfvC+CvEHPCv9ZjpefXg9boxJQoI/ JTKLws12dMNA/QLDb7XBH3fY58TctmMXXBwXw0oY2t1mXiLgz8IsSpmmnfyHgpYoUTrc WIiQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=HkR3gzB832b3FW96NlBLB1xXylKJv2Axeh5KafyCU8E=; fh=1no/tT3ISkEhj+j7IkUhS53m7NQq8/sYUDhH3QHsLBg=; b=owrO0MqU4ENbr/4aX5GboWi+EpI6ls8QfjVjtlvkbSR4nhcIp/V1xHwDuabnii6A6D sk7QrNSn6KkVbZBYWv0lVelXFBeJ0TDqMLTphnOol2YvuwDA4UUCRgd7gktFrEBuLLkh 13d1kTwBES/dUvVR8D/B/P1H/PW5gV9zi+vnC+qkDcVGkm0z9i8HLEWfzbK5awml0HCR 6fiWqQw6Omni6oGHOjo2pJzKM19E5fWUMCJdG3MHE8Jdfzb/lg3uQrZFiYwVqecXFiDq qE0G5xvZS0rg/oLt4n61GXRR4rPXIzokLvT3DLKp/Zz7ysqpv3pF+/vrhfawHUfhATLu 63Nw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=DUvovgep; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47510-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47510-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCW+tEOisfkaRc53xM9PMsWhirtVNfc6KwjW8uA+DNFkCqm+AWNp4L1tGYgs5qM+08k3lISxt5g2A6M77phrVTz0DEmbWQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id e5-20020a50fb85000000b0055f325ca620si2593778edq.638.2024.01.31.17.43.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:43:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47510-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=DUvovgep; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47510-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47510-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 98DAA1F2E38F for ; Thu, 1 Feb 2024 01:43:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6C63636B04; Thu, 1 Feb 2024 01:42:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DUvovgep" Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8794FBE7; Thu, 1 Feb 2024 01:42:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751740; cv=none; b=QPqyCbRroOYylcHDPSjL4+F6i7FiyHy6ZzMybEg6fsRVMeHmYpZsFN+8mfjpEnZeECdOh+qHdaazz8S4x+yss9p41Wp35EWXMQ1ZGNOK16UtKLBEcozYOOuy1Zmnhn2nBXqzxHBhtJd8XfbFJ4jiqFEIxtrmH19+3R+AHQrue1U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751740; c=relaxed/simple; bh=vN+b/ifMf2GpJznfNVNFzi7NjJGu81cMQy2PppVHW9I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JBV1ECOF8yKYSFv0/BxpOMbQRzxy46hUUMt/b1SrH9XkRKg5DZTVV4gSQr/Rlz/s1NLlN00iMa7Rvma43piwlda2ODSko/hPujz1Oa4o/UK82RpuPMGK+n7s41QEyrj7sVa8jltoeEiZgd1mTzEyP4E1M2PfnTzvu78Z/QVRZto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DUvovgep; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-783f27300b1so25369385a.1; Wed, 31 Jan 2024 17:42:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751737; x=1707356537; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=HkR3gzB832b3FW96NlBLB1xXylKJv2Axeh5KafyCU8E=; b=DUvovgep9E9pgBwh78jqlcgwESoSlQmrkW6esP7dXciXNMwMH+7oNhoL5dH6OoKm8O dNwG1rnje4peYgTqMtn/h3k2U4G8tQOykEoMSrZ4Km/GLWrvEd9KQS83GjYEa3cmS4/T VJEhvAETw5i6IFPkKosKSgvp1gsKC5s8d5bJHjIz3/4t4lw7qpzjA4agkcz2+9RnL8Uv G+xCZtiYWlQnlw4U9D+xXzwKWBaMCza/LeWYK5xEHfasBbTE8VTNWvHa5WlKJSOkWaBd ocCrntCV8pkZ/g5tsq/rF4RvKCvyZT2qqLVoYl/fRMXDa2gwUfvpf57wGrInI8W5iyZ7 7ykA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751737; x=1707356537; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HkR3gzB832b3FW96NlBLB1xXylKJv2Axeh5KafyCU8E=; b=Qrhzb4D7IGXlIPhWAG2K3FSEGn5zRY0mXyUeQr14odkhLPTWnqrDeBX+yhGauLF1Ij Kb3CSvGf7IVaWshj/JdJnYdnbdeSs1VTCvnFkeb87WKtpgjsctxMudJQNmNbcaKkh1Qz 91PrZ5Ts2fGQR6QaxFSGu8lkjYmySe3amdeWonxRas9PF30JnR6mCN7ttjHDAA4qm9yf 55R2UaOpZVXbsbkgU5CJLBquSzQbnE0SMfNc93bZ2kanQr3O4+pcm9yvS1kjBjvKI5sJ lPxtvcmOOdxKxZVJFqFVcBjmpjyr4iExisngzDNxABX3UjA0JqvyAhq+maHeZq4hKarQ yefQ== X-Gm-Message-State: AOJu0YyLndnOdGRbwref6tAcFBVtAOmglcLCVtyY4+RlJsu3aUhvuXKL jPvjyXyZY+rvJjmo/Xouy3N5+xN23hi68ddLII3gO991vDSiLSP/ X-Received: by 2002:a05:6214:20ea:b0:68c:7aab:696f with SMTP id 10-20020a05621420ea00b0068c7aab696fmr637624qvk.56.1706751737591; Wed, 31 Jan 2024 17:42:17 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCX3uAeKx2NmzGlq4BnurQEwL6ItQoP/F3SbwkDACvEGgNeBwQxAo/QKhyoFI2JF9X77wB3dAdO9A3VSv83pA7F1yVKeMe0wXdAYcd2PLRm+Qr+PCLG3AQHrT5d6AigUqPpX3UyTI6OvKEkqaSS5H9YoTcES8XCDZqcwop9PID5rNq9NKzIUPOII3c0vnnIWDtMATMFe4ymughYaqyIpqi9LAJbeV289cLo21MZuCnDvpvWVGiE5ICi3b8pkicsrBHAAQNw3tdK8+Q/S7B9rn2jpdmRf79mnUPCqoWzEfZCdaYBl4uzx/+LNgSWuS2IolvrynmESZvtL5ER0ZrpyudDO6rPBYcxz8vcU6KYrcZeopiahm+XO6V4z2p86dT5Q Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id eo10-20020ad4594a000000b0068179afede6sm3263621qvb.100.2024.01.31.17.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:17 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfauth.nyi.internal (Postfix) with ESMTP id 84C091200043; Wed, 31 Jan 2024 20:42:16 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 31 Jan 2024 20:42:16 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeeuohhq uhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrf grthhtvghrnhepheffieehhefhveehhfelfeejtdeitdejhfevkeefjeeiudegkeekueeu hfeiveeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedt ieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfh higihmvgdrnhgrmhgv X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:15 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 4/6] rcu/nocb: Re-arrange call_rcu() NOCB specific code Date: Wed, 31 Jan 2024 17:40:56 -0800 Message-ID: <20240201014100.3204532-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789659009096695286 X-GMAIL-MSGID: 1789659009096695286 From: Frederic Weisbecker Currently the call_rcu() function interleaves NOCB and !NOCB enqueue code in a complicated way such that: * The bypass enqueue code may or may not have enqueued and may or may not have locked the ->nocb_lock. Everything that follows is in a Schrödinger locking state for the unwary reviewer's eyes. * The was_alldone is always set but only used in NOCB related code. * The NOCB wake up is distantly related to the locking hopefully performed by the bypass enqueue code that did not enqueue on the bypass list. Unconfuse the whole and gather NOCB and !NOCB specific enqueue code to their own functions. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 44 +++++++++++++++++++----------------------- kernel/rcu/tree.h | 9 ++++----- kernel/rcu/tree_nocb.h | 18 ++++++++++++++--- 3 files changed, 39 insertions(+), 32 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a402dc4e9a9c..cc0e169e299a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2597,12 +2597,26 @@ static int __init rcu_spawn_core_kthreads(void) return 0; } +static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func) +{ + rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) + trace_rcu_kvfree_callback(rcu_state.name, head, + (unsigned long)func, + rcu_segcblist_n_cbs(&rdp->cblist)); + else + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); +} + /* * Handle any core-RCU processing required by a call_rcu() invocation. */ -static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, - unsigned long flags) +static void call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags) { + rcutree_enqueue(rdp, head, func); /* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness. @@ -2698,7 +2712,6 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) unsigned long flags; bool lazy; struct rcu_data *rdp; - bool was_alldone; /* Misaligned rcu_head! */ WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); @@ -2735,28 +2748,11 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { - local_irq_restore(flags); - return; // Enqueued onto ->nocb_bypass, so just leave. - } - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); - else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); - - trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); - /* Go handle any RCU core processing required. */ - if (unlikely(rcu_rdp_is_offloaded(rdp))) { - __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ - } else { - __call_rcu_core(rdp, head, flags); - } + if (unlikely(rcu_rdp_is_offloaded(rdp))) + call_rcu_nocb(rdp, head, func, flags, lazy); + else + call_rcu_core(rdp, head, func, flags); local_irq_restore(flags); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e9821a8422db..bf478da89a8f 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -467,11 +467,10 @@ static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j, bool lazy); -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, - bool lazy); -static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, - unsigned long flags); +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy); +static void __maybe_unused __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, + unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); static bool do_nocb_deferred_wakeup(struct rcu_data *rdp); static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1d5c03c5c702..9e8052ba14b9 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -622,6 +622,18 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } } +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) +{ + bool was_alldone; + + if (!rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + /* Not enqueued on bypass but locked, do regular enqueue */ + rcutree_enqueue(rdp, head, func); + __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ + } +} + static int nocb_gp_toggle_rdp(struct rcu_data *rdp, bool *wake_state) { @@ -1764,10 +1776,10 @@ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, return true; } -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, bool lazy) +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) { - return false; + WARN_ON_ONCE(1); /* Should be dead code! */ } static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, From patchwork Thu Feb 1 01:40:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195123 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp145131dyc; Wed, 31 Jan 2024 17:44:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IGgcN+eSb0LbdUBR1NWpgUqUbrEFaxD2OuPcHzVD7Me9NXSJTy10gRMSUmYrSmYEqopu78u X-Received: by 2002:a17:906:f243:b0:a36:33ac:3086 with SMTP id gy3-20020a170906f24300b00a3633ac3086mr585350ejb.72.1706751840345; Wed, 31 Jan 2024 17:44:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706751840; cv=pass; d=google.com; s=arc-20160816; b=QpfmtA1m6AbUOVPx9boKGIxvOgAA9uyDrOX96Jk5hC1kSdViDC8Sw9CAc+fC4ZAvM4 pdGqSrkgL5Xbzdn4UgJFFW6pE4Wft5KgiHuBUYTenFickfs6qvHxKpGZBIGb4j22AkST 70jTff4hVQ369xVtCSbZiD4+EDnG1Q/4zWxdfb7J7PARGQ32frPyKPnQOynL3nayd2uJ PBFE576Zsym1XnxC5quqrbnWnRc6Xs8GOxLb13Som57zgHzAGAG2JJv0YFHtvxoM8c95 x46iltSp+CxzjjHH/SofWahHLZBkZ8Usqcl4BhYj+SxYpJTKKCW+rxUr0276yTy4kclb W15w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=kITZv0k9fxTTu9rhEhSjW/uQbfGz/wypbxDOcIf6Ytk=; fh=iAomVH4581Ec5BAuWPIinsHuFyY90GroYGxv4de8Ufg=; b=a+ZacSSjayhWJpJ/I4//uKyCGTfqjkD1ybHLmvAx8HNY+0cSbHk4p7ykhNDT//60Aw 4/yjghqr1rUhU5NNxzU+7u2XTLhRDXUpjj54+15nL3wjMCQwATSR682OjPpM3Cp4CrsF q21BnPdxvo5r62aGS4UAxQhiO3OxZA3eRowKeq7zqTtbvZNVczA0++UpnPt6wyaFPROz r1IR08ajGSF9FMw2gVqFotV0+og5CRwEQxKBgCtmsIDTiqwgAzX5sTp7IojMB8QMBwqD Gu+NRKk1cD2bmrjzNnrxGpVkTRZ71X5PT5qc3yFkUkbP9V72Lg1ZdggmcaAuk7YPL8qJ 6W2A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ODXak1IS; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47511-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47511-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCV6eZJUMnu1ajCSYrZH1Ir8h3T4E9btuVSCDgCe+Dxjk/J9t43TZ9msbdm3JOwsPU4GPJOlRJFRsfFdAGhwdeHqDLFDEw== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id cb19-20020a170906a45300b00a314870b103si6098356ejb.879.2024.01.31.17.44.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:44:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47511-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ODXak1IS; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47511-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47511-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C33601F2E335 for ; Thu, 1 Feb 2024 01:43:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2D9BD39848; Thu, 1 Feb 2024 01:42:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ODXak1IS" Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3764419BDC; Thu, 1 Feb 2024 01:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751741; cv=none; b=USTIHFdH+04gRHNsr1+Hd5fc5MAeycDaN/WycXibE3ScqHd2PwLwOQaVjMoZq+NCzevuxuV1sSXbNPsqZJLDmK2RHL6aGPOX6r/IuE2Cw60n/9fdcEoGnhgcY5vtJoW53S31loikxXlgxPhE/wPZT1f0d7BxXlo4t8Cw8MXiDNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751741; c=relaxed/simple; bh=O8nU5bEULJUElIjpZZSaHK6+7EZs2Vxmkgn27DuOtNg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OH/KiaTktl71BvdPb7EE/CU4gubHves2PX9f998FgnmxIib1t4dspGrBqGMO4u/qFe3nhkhAW1jtZQ9DHnhQGYl5yuasAprx+C33afpl993jsiQAHgF4i8XEokB8UvSpMA2dtZdPcJgaAEQzy3AkrGO9lY2EB3fUMVh69lrC96w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ODXak1IS; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-42993124fa1so2500961cf.3; Wed, 31 Jan 2024 17:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751739; x=1707356539; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=kITZv0k9fxTTu9rhEhSjW/uQbfGz/wypbxDOcIf6Ytk=; b=ODXak1ISq3RHSWZkzysRoiT0/3TxPFnaRFie//clqDP80Q3g3grT/GdOK449OdLs5f 3183wCIfRVA3/vwTXRo3lyfN4PnrJyYX8oq70q/Gr6Z+3BA14wrbL2+y4p4ZLW/0eHPo u9IhFKG4B9c4UgtuQE3d01QSre1RcaKv50OANTWxFK++w5319eXt8TgmUrZJhv2lk41u XAldT2rZKMNEWlTKoisLuxBLbL4Bjxi7RsaVhfdw2ZwJZRHwzmU2Zps6HFsaZCENr8BJ BgJaNshSNXhQnbYw6a+7q5k8gCDqwBNnYVOfG77OqACKl5peEIoqIA9hVfxLjgqp1Yoa ZTqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751739; x=1707356539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kITZv0k9fxTTu9rhEhSjW/uQbfGz/wypbxDOcIf6Ytk=; b=wlRkShvR7xaeJgzZMzgpOdJTMEJjMDLs8F1w/pNf3xAWfIpw+e177ixKp0Xdhm+KCv f8BrlER90VeRXXZHAnkxusLbze6hV2P1X5YGQNyAJleIPv7knar4WU0AJfaFsr4MaBdK uh81BVj4wcGGsEMjaFd9hgMwekPrmmUzSgwiIRSFguU4yLzPkbquAJEgIUw3eGoi47hL RhLoT4gnvl7jc52307KEvBAhzsKubdHn9MEwDJFzXYPdpg9FoQb+GeubXiAR/A4H1wg2 nBXSTp6/7vxqxrzLEqiOAYKDOJQ/cJtpJ8FjHH+pL7DzX5IEcrXlO9iSMqNfOH6jAo6M /acw== X-Gm-Message-State: AOJu0YxK0fdgJNfCfktbGmu7Hd0IrFKIDqFzEwoEMlqnrNoiwKkDZ6R7 1RQttLCfxuC+3jAEQ8vP3/TfU/N+oSUsx91JjgvQ7UNSdugM+rvc X-Received: by 2002:ac8:5947:0:b0:42b:ec11:1a6b with SMTP id 7-20020ac85947000000b0042bec111a6bmr3654721qtz.67.1706751739040; Wed, 31 Jan 2024 17:42:19 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWyQ/sB2p8wMYMTrAO5UrG4Qn303OKDFKXkbM494Xo3znQVI0CPuSatGuIIl7tjl5Ipx9gD9xmEvBfHDcdi3f6XqxCMfJ3CgzxfZmt5xR1sZOrcOvv9+qWeZ0fkIovuhKs8au6MeHk2kCqz/mFa3SEBbv/JtYn2/jtB5WkIVJBb91c5R5pWwwd9G2zZSI3L0N61ZMZI1PikHXWuRYBTyMy2qLEvB56XnqscgUK3ifJo2aKdzRfMmiqUogxqeWSyqVaUeM8umARUCyZsJR8TAkLBTzDKSKYcmBVlqhWfZ5j0++XMWMrho1MHAIl+whO0Eajq+zcxdYWHEeeJsf/+1TWeUK/HnJkih5BCWXdS7MJMmywmtZ15nZFg Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id gd13-20020a05622a5c0d00b0042992b06012sm1644885qtb.2.2024.01.31.17.42.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:18 -0800 (PST) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfauth.nyi.internal (Postfix) with ESMTP id 438251200043; Wed, 31 Jan 2024 20:42:18 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 31 Jan 2024 20:42:18 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:17 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Zqiang , Joel Fernandes , Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan Subject: [PATCH 5/6] rcu/nocb: Fix WARN_ON_ONCE() in the rcu_nocb_bypass_lock() Date: Wed, 31 Jan 2024 17:40:57 -0800 Message-ID: <20240201014100.3204532-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789659018050728728 X-GMAIL-MSGID: 1789659018050728728 From: Zqiang For the kernels built with CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y and CONFIG_RCU_LAZY=y, the following scenarios will trigger WARN_ON_ONCE() in the rcu_nocb_bypass_lock() and rcu_nocb_wait_contended() functions: CPU2 CPU11 kthread rcu_nocb_cb_kthread ksys_write rcu_do_batch vfs_write rcu_torture_timer_cb proc_sys_write __kmem_cache_free proc_sys_call_handler kmemleak_free drop_caches_sysctl_handler delete_object_full drop_slab __delete_object shrink_slab put_object lazy_rcu_shrink_scan call_rcu rcu_nocb_flush_bypass __call_rcu_commn rcu_nocb_bypass_lock raw_spin_trylock(&rdp->nocb_bypass_lock) fail atomic_inc(&rdp->nocb_lock_contended); rcu_nocb_wait_contended WARN_ON_ONCE(smp_processor_id() != rdp->cpu); WARN_ON_ONCE(atomic_read(&rdp->nocb_lock_contended)) | |_ _ _ _ _ _ _ _ _ _same rdp and rdp->cpu != 11_ _ _ _ _ _ _ _ _ __| Reproduce this bug with "echo 3 > /proc/sys/vm/drop_caches". This commit therefore uses rcu_nocb_try_flush_bypass() instead of rcu_nocb_flush_bypass() in lazy_rcu_shrink_scan(). If the nocb_bypass queue is being flushed, then rcu_nocb_try_flush_bypass will return directly. Signed-off-by: Zqiang Reviewed-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 9e8052ba14b9..ffa69a5e18f4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1391,7 +1391,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) rcu_nocb_unlock_irqrestore(rdp, flags); continue; } - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); + rcu_nocb_try_flush_bypass(rdp, jiffies); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; From patchwork Thu Feb 1 01:40:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 195124 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:106:209c:c626 with SMTP id mn5csp145188dyc; Wed, 31 Jan 2024 17:44:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTe/WATsvEDaS8js7QYjYGRFoERhur2HhixjpbpOEFvWHbYoK0OrsxUf2K67J8HTMJrdpZ X-Received: by 2002:ad4:5fc8:0:b0:68c:4167:fc5f with SMTP id jq8-20020ad45fc8000000b0068c4167fc5fmr4891052qvb.52.1706751849317; Wed, 31 Jan 2024 17:44:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706751849; cv=pass; d=google.com; s=arc-20160816; b=MmxDokx64pW6ZmP0WE1PYePHFj6tREDlFhUIdicLFcTpGPEOKWVjrhN0bu0saU6GY7 MuoMLKANSGp3IwiHeR2tyz/qoL4P2EnscIYt1ffGntm0p1BAwPZ+lJAKAm8Drf1SuhDc NiJgmKxtRRE/PqZTwInQOnQbqTb1F9/dAuz43Gr9du2tN6YfbfP9IAVJjjZM7nRlEQbG 49nE0WQcmFfthaD940xJu72sBxK89jkxQW+7LWI1RO7ABu5gGg31mpypT0JwRFjOnqU/ 03tdmxSDMNEcAREe22w1Ow7Ca1BOYA6Jr2h3m46Ian0dVPJKZ2xOH3mH5NxmGrpXHImO 1Gwg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:feedback-id:dkim-signature; bh=4WRKA+8oB7qFwtR1n75bKmIl1ybu1sGsFv65Z7DNMuk=; fh=OxshDux2db+Xx0x5jdCqe1f/qCHc1TEyKrz9wUd4iio=; b=NcvK1k7mTb9rcyYK00pqSEtuK7tqyYsFjipyOYiu6jYukY7w80bcAWUfxcuB0rXVe+ q/VVHfC73awtSABI1ZYUXSnZLvbAxtNDXBHNi9qeF/Cu6FDTycrbm11CarxnD0Rogmul 7lALTR0qg96s9P6VQv9QqBS5Lk979/dDS8ZFvcQOVANtd7RuQGZGOLnzoG+5rr2Iwduc 3mpeb+RUa/fjnNnaCb2QXhYYXi2RUttYt3slLYcQ1oGI7lITiIhm0AyHEhaAcbpqlwg6 9i4S0nYgq+LTNBT7hsc9WrmTW7XBsIJZurXNc/WoNWaftBWcr3g3yF3sxBtzU2BlUOza sTPQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ZuLWh71B; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47512-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com X-Forwarded-Encrypted: i=1; AJvYcCXPHySjeaGTfHAIEs9DN/XGjGjooPRcpdx8J4tJWnVc2iVDuazVT2DcBiQgaTvhBuZOFsUbw+tPyr2ueniUl6r72XrJsA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id jf14-20020a0562142a4e00b00684771ba580si13483935qvb.454.2024.01.31.17.44.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:44:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-47512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ZuLWh71B; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-47512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-47512-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 110F51C215E6 for ; Thu, 1 Feb 2024 01:44:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9FE8E39FF9; Thu, 1 Feb 2024 01:42:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZuLWh71B" Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B3F4C7B; Thu, 1 Feb 2024 01:42:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751743; cv=none; b=NVJGuMpcfFDwA6F7NnNYay3crQGsqyTHpDXLi3JkZW/EBFYrT7eO1aHCi5jULXv9zaDEZq5Pr80qeOJMmQfA9I92BGGHQxIvePfVjpX59vcyIYu5Jxm2QhPxHK8iKPDH+PkYu37BciMNYD47EfrMTOdWPjnbElKrJLgk+WOylRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751743; c=relaxed/simple; bh=6oMVGx/JwGgsizKVfPO1lHIPkL1xgjV9NuMQhjvciWc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tudURNa+bwSd/RYoc8m1Aer1fBOD6WtfD9FARAGJ32yVWeR2HYTbEG/RTACr+yteeqP7ZpC8ZgXD860HTyn5mpQaOf43jifIw0o3utbzPpKMQUb3Lnzf+wi6Ve2LGrcndr9tFPi9tRBvFrpVgZlmXuZtsD7mr+3jnKrBV0vqff8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZuLWh71B; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-783ced12f9bso19792585a.3; Wed, 31 Jan 2024 17:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751740; x=1707356540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=4WRKA+8oB7qFwtR1n75bKmIl1ybu1sGsFv65Z7DNMuk=; b=ZuLWh71BL1EJNOFSk/gid87c2vx7R3bxoiGdyL8PffH0ma/Utl7wwCbs0PoXeed9X5 +wE0iMkAyMv2A7yysekrh88dGErbyN7rlV9/UZc868cRe74aYPxKswDl+LsvMU4FY62+ MzcIlBLHbwuK0KcWOpWqbGle/qniXSoNZfQNX+8Rxw8yRADYaLNxbSWjN9yi38euSb0E MTgqxvu5Y9GTu3y31KssoK3YBqQLnSLVZtKTdC2jH2ORUl7Bs0wfAD6IjACaOUh8G7bo TsoUZjC50SHX66/JE82hoZZA4UKSiWJCOOKwt29gqdECLI0KQDh/7sWNEaEXkR7RlTHT R34A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751740; x=1707356540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4WRKA+8oB7qFwtR1n75bKmIl1ybu1sGsFv65Z7DNMuk=; b=U8fb9UNf1hCOxXDd5X2KDpAoZJT2WaRcY8QJYIKyhaUQLJshAzgSraDz+86URiMvF4 ffZRCsA+thjJUr+n0si6AIOsy0pbk5haI6x/2koXaYSdB8f8z8Pgh70K14YRGXb6LBtl 6JWiiMlHakn03rV9CHbnJY8bgqTndXeApNmF6A9o1zUrYkqw3GdRdV0To4Pm/dZEEZgy gRh6FykbyNvPzJhwcntBl/fXrI3Tn8rMFEzG9c+jdm2DWGScd0U19eCf3l56tYMs7JEF kvBZJJL7BJYIgh4M91U0WUgVhUM8z/3tMKTC9oB6OX0+ddriUEC+O/xpanaqhLHP4vVj 9HXg== X-Gm-Message-State: AOJu0Yzltku3mK5WOrUL2olSJmBK50Ed1ah8qEFzxEKbdAnccuVF4NQM HxLp2p9cRRrYGMCVtsdgmPEq2TrizdsoAGrSYcegbYfCBOr2JVPuy2qOXRpn X-Received: by 2002:a05:620a:124c:b0:783:4a89:f1d9 with SMTP id a12-20020a05620a124c00b007834a89f1d9mr1233836qkl.59.1706751740697; Wed, 31 Jan 2024 17:42:20 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWHG9KQtwYecuVAw3zxj3CBVPr63QVPiot25Ti3U39+yLxoXmX2tiyTEzAMYm2wos+U1uG/a36SWyhSXx0T+NMz0XuoixOz4zXYn5QMYtMo8DQ4xhy1OYv1U2EyWVgWItmvawjkFZMDd822hmOW/ZBcZ/Jho9XnKLij7ig8pGhCnA0BVGlqKr9bF/LAEq7cffAh1q4siV6BVIttYaBaO3yGpyLnd92oFQ1z3rwwN5OXA5erZ8afTzxG+DS9KoWAplGRveeVz2MY4/p1XQ7w6RHbfPMZKlkRwpS1ktIiWZeMThX0t10Mp4k3K8fC/AMFVHncaHXI6k+ApnZkS7nXhRyYiccemfoIxRslq0PxKIwc5kJvDFNf8L4q Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id qp15-20020a05620a388f00b007840843f3b2sm2079154qkn.18.2024.01.31.17.42.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:20 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id E1BC627C005B; Wed, 31 Jan 2024 20:42:19 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 31 Jan 2024 20:42:19 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:19 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Zqiang , Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan Subject: [PATCH 6/6] rcu/nocb: Check rdp_gp->nocb_timer in __call_rcu_nocb_wake() Date: Wed, 31 Jan 2024 17:40:58 -0800 Message-ID: <20240201014100.3204532-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789659026893817417 X-GMAIL-MSGID: 1789659026893817417 From: Zqiang Currently, only rdp_gp->nocb_timer is used, for nocb_timer of no-rdp_gp structure, the timer_pending() is always return false, this commit therefore need to check rdp_gp->nocb_timer in __call_rcu_nocb_wake(). Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ffa69a5e18f4..f124d4d45ce6 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -564,6 +564,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, long lazy_len; long len; struct task_struct *t; + struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; // If we are being polled or there is no kthread, just leave. t = READ_ONCE(rdp->nocb_gp_kthread); @@ -608,7 +609,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, smp_mb(); /* Enqueue before timer_pending(). */ if ((rdp->nocb_cb_sleep || !rcu_segcblist_ready_cbs(&rdp->cblist)) && - !timer_pending(&rdp->nocb_timer)) { + !timer_pending(&rdp_gp->nocb_timer)) { rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred"));