Message ID | 20240228100354.3285-2-wsa+renesas@sang-engineering.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp3246476dyb; Wed, 28 Feb 2024 02:04:35 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUsgm6S/lu33sxqxzr7TrJU/gUw8crRGGVc6VoI5KR0jhUgIqD98TB+YK3PkjWUHULVyBA3+6bMLWx4ZUPZAOcBzjdUug== X-Google-Smtp-Source: AGHT+IGt8J1a5UIz1/x/6x8jJ7AyaElVtPj8bxQP6BuPb08GV3UxNDgvKoLn1lQ06J26GmQVxmWm X-Received: by 2002:a17:907:930:b0:a3f:a562:ded2 with SMTP id au16-20020a170907093000b00a3fa562ded2mr8171473ejc.73.1709114675065; Wed, 28 Feb 2024 02:04:35 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709114675; cv=pass; d=google.com; s=arc-20160816; b=0KzW3fzC01dubZv+0+4k8dxnZGtrOWyv/HH7Uu/gOb/20zW0uN9W4IvQTTE/xJo9sn 1ooi+uXbn4Ey1bitXj7wpwrnBb4+h5LemU/s4D7AlMnpjCmSF5lke+c7uiVAMSEecMEL S4vxmdKVYgJQfDiYzd9ce18krp0l29rrfxosFdKor1H7ZrHiRMND2Wt6lQ91HIRr8J3i hoz7cNtteriOziOfawI/SUjDAGRyZ9K0n7q3tVynAUdM6La1YYNr9w9yOv6HOJHum3GB jtMZ3ufj7ebKFcP4HfJ42kbOHU7nMLGqJUPhEl7I518DpSj5n+bf1ebGHgLChSiJN/na IKJA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=rDq91xwkdeBq+poVwEnDTvivychr45vA3bZxWuSbk7g=; fh=TxzlyfPjoNzZ3gH2FNqg89sqlQrfZoexDuM7GS8b4xU=; b=F2r6KT8uj8oZNdFn42G4X5xfpbBhi7Hi/8zIMhvhq17U/HByOqPnTIJSnEaihumCpB zdD1tj3bGXpKnnVhyoZcxpK3jRHjihsa/ocXj9bpqFYwqXWBHxEXfS89DrIOL3gKFTYV sn7Oh2zFNFVOHSzZMxY/kDSfdqljbV30chHGRDzcmCEkvcCynkfeIJIKedtTj3GFBH6U /HOYv7MpsSBRBfAPC0W/PJHnHs+bKVdo3G28wEOAzO0nAlGdJZ/R7QsQ3KrGm+b2Cfap JoYWpJaBDDF0blFJAbTqcEoX4MLdyIzRX15TWmmPuGH6MAdu0qcst7gixjCsKdN2sGIC ILsQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sang-engineering.com header.s=k1 header.b=N7XJfSGv; arc=pass (i=1 spf=pass spfdomain=sang-engineering.com dkim=pass dkdomain=sang-engineering.com); spf=pass (google.com: domain of linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id ka17-20020a170907991100b00a3fc7f1c1d5si1593896ejc.968.2024.02.28.02.04.34 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Feb 2024 02:04:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@sang-engineering.com header.s=k1 header.b=N7XJfSGv; arc=pass (i=1 spf=pass spfdomain=sang-engineering.com dkim=pass dkdomain=sang-engineering.com); spf=pass (google.com: domain of linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84807-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id AE9571F283B3 for <ouuuleilei@gmail.com>; Wed, 28 Feb 2024 10:04:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A7288376E0; Wed, 28 Feb 2024 10:04:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sang-engineering.com header.i=@sang-engineering.com header.b="N7XJfSGv" Received: from mail.zeus03.de (www.zeus03.de [194.117.254.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AAF5364C0 for <linux-kernel@vger.kernel.org>; Wed, 28 Feb 2024 10:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=194.117.254.33 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709114648; cv=none; b=fuEk5RJf0sujxtbDCd8QGCZrobIBo0yK5fswBtTzbzSN0hdadBSdUPJ4aImIT5bjDQmS2mUxqJyB5hSvocJoHOMlxM/cscrGXgoV5Dht4GM1NNRStnjJW4sH2NSCrLSqYQ01XopvApbQX2cix1EE+8lkSl8Zv/C8zRAG01Ls2IU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709114648; c=relaxed/simple; bh=KQzzUhLT/q9KbVtq8OAn3e60tMLMhyaDIKLmaNMeNN4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=bwqVF+Vp7wHyIahQgsKKi/P8VrAYd6JHNXM9vJLvqdnkNYRKP1KwnMk3ojl/uXKfCcpC+FLgvl8SFd2pywyo/BrgBQfAzQqEQXHIViHB7ZoWipYCP3dO7x4yzg9WCCD8r3nZKTEUY5O3VHhQRVReUkQoeniZHyaZUmjLmNVNgkQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sang-engineering.com; spf=pass smtp.mailfrom=sang-engineering.com; dkim=pass (2048-bit key) header.d=sang-engineering.com header.i=@sang-engineering.com header.b=N7XJfSGv; arc=none smtp.client-ip=194.117.254.33 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sang-engineering.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sang-engineering.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= sang-engineering.com; h=from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; s=k1; bh=rDq91xwkdeBq+p oVwEnDTvivychr45vA3bZxWuSbk7g=; b=N7XJfSGvkmJmWbKxaWscPZOpHc4+QO 6D3dbNNONjAyvRDPFam8H2yUdSC4/afPOW0KWMmYfX9wEc+E7wnIvyNRlrWzWxZb wytxOaen9+3EeuLvKwkhGfgzivPanwwU0IKmZVbwkvWrBo/+tmPTuQ6671MxFn8S mbC3nPZqN+HBy/vW+9EfSmn1+y2sgoDwSSnfJ8WPTQ2CotaHbbWuO14LWRGRk5cQ Pdd3y5Qpqlvk5LgJYrVsl5tvApAkhjDYyJrysnDn+GowpscN+Y6Ts2VpY/jgwqKl M5xdYJw747B/9HN/Vx3M8l97pUBPfslRtdgCitq/CWI8olUTGqHKQ9zQ== Received: (qmail 1906275 invoked from network); 28 Feb 2024 11:04:02 +0100 Received: by mail.zeus03.de with ESMTPSA (TLS_AES_256_GCM_SHA384 encrypted, authenticated); 28 Feb 2024 11:04:02 +0100 X-UD-Smtp-Session: l3s3148p1@nlEfQ24SbIMgAwDPXyvUACsbJ+4rxOVH From: Wolfram Sang <wsa+renesas@sang-engineering.com> To: linux-renesas-soc@vger.kernel.org Cc: Wolfram Sang <wsa+renesas@sang-engineering.com>, Dirk Behme <dirk.behme@de.bosch.com>, Ulf Hansson <ulf.hansson@linaro.org>, linux-mmc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH RFT] mmc: tmio: avoid concurrent runs of mmc_request_done() Date: Wed, 28 Feb 2024 11:03:45 +0100 Message-ID: <20240228100354.3285-2-wsa+renesas@sang-engineering.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: <linux-kernel.vger.kernel.org> List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org> List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792136629228815412 X-GMAIL-MSGID: 1792136629228815412 |
Series |
[RFT] mmc: tmio: avoid concurrent runs of mmc_request_done()
|
|
Commit Message
Wolfram Sang
Feb. 28, 2024, 10:03 a.m. UTC
With the to-be-fixed commit, the reset_work handler cleared 'host->mrq'
outside of the spinlock protected critical section. That leaves a small
race window during execution of 'tmio_mmc_reset()' where the done_work
handler could grab a pointer to the now invalid 'host->mrq'. Both would
use it to call mmc_request_done() causing problems (see Link).
However, 'host->mrq' cannot simply be cleared earlier inside the
critical section. That would allow new mrqs to come in asynchronously
while the actual reset of the controller still needs to be done. So,
like 'tmio_mmc_set_ios()', an ERR_PTR is used to prevent new mrqs from
coming in but still avoiding concurrency between work handlers.
Reported-by: Dirk Behme <dirk.behme@de.bosch.com>
Closes: https://lore.kernel.org/all/20240220061356.3001761-1-dirk.behme@de.bosch.com/
Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com>
Fixes: df3ef2d3c92c ("mmc: protect the tmio_mmc driver against a theoretical race")
---
Dirk: could you get this tested on your affected setups? I am somewhat
optimistic that this is already enough. For sure, it is a needed first
step.
drivers/mmc/host/tmio_mmc_core.c | 2 ++
1 file changed, 2 insertions(+)
Comments
Hi Wolfram, On 28.02.2024 11:03, Wolfram Sang wrote: > With the to-be-fixed commit, the reset_work handler cleared 'host->mrq' > outside of the spinlock protected critical section. That leaves a small > race window during execution of 'tmio_mmc_reset()' where the done_work > handler could grab a pointer to the now invalid 'host->mrq'. Both would > use it to call mmc_request_done() causing problems (see Link). > > However, 'host->mrq' cannot simply be cleared earlier inside the > critical section. That would allow new mrqs to come in asynchronously > while the actual reset of the controller still needs to be done. So, > like 'tmio_mmc_set_ios()', an ERR_PTR is used to prevent new mrqs from > coming in but still avoiding concurrency between work handlers. > > Reported-by: Dirk Behme <dirk.behme@de.bosch.com> > Closes: https://lore.kernel.org/all/20240220061356.3001761-1-dirk.behme@de.bosch.com/ > Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> > Fixes: df3ef2d3c92c ("mmc: protect the tmio_mmc driver against a theoretical race") Tested-by: Dirk Behme <dirk.behme@de.bosch.com> Reviewed-by: Dirk Behme <dirk.behme@de.bosch.com> > --- > > Dirk: could you get this tested on your affected setups? I am somewhat > optimistic that this is already enough. For sure, it is a needed first > step. Testing looks good :) Many thanks! At least the issues we observed before are not seen any more. As we are not exactly sure on the root cause, of course this is not a 100% proof. But as the change looks good, looks like it won't break something and the system behaves good with it I would say we are good to go. I think we could add anything like Cc: stable@vger.kernel.org # 3.0+ ? > drivers/mmc/host/tmio_mmc_core.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/mmc/host/tmio_mmc_core.c b/drivers/mmc/host/tmio_mmc_core.c > index be7f18fd4836..c253d176db69 100644 > --- a/drivers/mmc/host/tmio_mmc_core.c > +++ b/drivers/mmc/host/tmio_mmc_core.c > @@ -259,6 +259,8 @@ static void tmio_mmc_reset_work(struct work_struct *work) > else > mrq->cmd->error = -ETIMEDOUT; > > + /* No new calls yet, but disallow concurrent tmio_mmc_done_work() */ > + host->mrq = ERR_PTR(-EBUSY); > host->cmd = NULL; > host->data = NULL; Thanks again! Dirk
Hi Dirk, > > With the to-be-fixed commit, the reset_work handler cleared 'host->mrq' > > outside of the spinlock protected critical section. That leaves a small > > race window during execution of 'tmio_mmc_reset()' where the done_work > > handler could grab a pointer to the now invalid 'host->mrq'. Both would > > use it to call mmc_request_done() causing problems (see Link). > > > > However, 'host->mrq' cannot simply be cleared earlier inside the > > critical section. That would allow new mrqs to come in asynchronously > > while the actual reset of the controller still needs to be done. So, > > like 'tmio_mmc_set_ios()', an ERR_PTR is used to prevent new mrqs from > > coming in but still avoiding concurrency between work handlers. > > > > Reported-by: Dirk Behme <dirk.behme@de.bosch.com> > > Closes: https://lore.kernel.org/all/20240220061356.3001761-1-dirk.behme@de.bosch.com/ > > Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> > > Fixes: df3ef2d3c92c ("mmc: protect the tmio_mmc driver against a theoretical race") > > Tested-by: Dirk Behme <dirk.behme@de.bosch.com> > Reviewed-by: Dirk Behme <dirk.behme@de.bosch.com> Awesome! Thanks for the super-fast tags! > At least the issues we observed before are not seen any more. As we are not > exactly sure on the root cause, of course this is not a 100% proof. But as > the change looks good, looks like it won't break something and the system > behaves good with it I would say we are good to go. I agree. We don't know if it is all you need. But there definitely was a race window and closing it removes some observed anomalies. Let's hope all of them :) I looked many times at the code and, to the best of my knowledge, don't see side effects. 'host->mrq' stays non-NULL, so new mrqs won't be added like before. Changing it to an ERR_PTR will only affect the check in the done_work handler which is what we want. But, of course, more eyes are always welcome. > I think we could add anything like > > Cc: stable@vger.kernel.org # 3.0+ Yes, we should definitely have that. I would have added it once your testing got good results. This affects every Renesas SDHI or Uniphier SD instance since 3.0 (12 years). Wow! So, thanks a ton for your report and assistance in debugging it. Very much appreciated! And, phew, I am happy that this solution does not make the locking more complex \o/ All the best, Wolfram
diff --git a/drivers/mmc/host/tmio_mmc_core.c b/drivers/mmc/host/tmio_mmc_core.c index be7f18fd4836..c253d176db69 100644 --- a/drivers/mmc/host/tmio_mmc_core.c +++ b/drivers/mmc/host/tmio_mmc_core.c @@ -259,6 +259,8 @@ static void tmio_mmc_reset_work(struct work_struct *work) else mrq->cmd->error = -ETIMEDOUT; + /* No new calls yet, but disallow concurrent tmio_mmc_done_work() */ + host->mrq = ERR_PTR(-EBUSY); host->cmd = NULL; host->data = NULL;