Message ID | 20221214134620.3028726-1-peter.suti@streamunlimited.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp236952wrn; Wed, 14 Dec 2022 06:03:33 -0800 (PST) X-Google-Smtp-Source: AA0mqf5qZCDQNYJP3XgY06/XhpjL/U+PQryu7SRt4O4/CsiC3/X3YY0SQlEtej9g4rguR2wtTNr6 X-Received: by 2002:a17:907:c716:b0:7c0:e7a7:50b with SMTP id ty22-20020a170907c71600b007c0e7a7050bmr21761713ejc.48.1671026612880; Wed, 14 Dec 2022 06:03:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671026612; cv=none; d=google.com; s=arc-20160816; b=bnHDMQQYQtrgoY1DbmXcG+GWzcPwc8RfrKtHzC1jiUBBZWDZNCL60KylYTkCUd1dJf upJsHw888SUkwl7r8XZoxwmwY2EMTgNmExd/biDsxbc0xo3i4Eh6IhxTqavTzkvAI88i Y5+NsCPJqSwB33EcaaOQ5FXFlcQK6TvmmiDHFQAcfJO10l5ShqQStqFugAhrk/YdB6Qv KCRGH8ODOfFkwwG8KFUXdktNAtWP2Yzm2uj50Cb0DmAlUXz0OgNWfWo4iymj6l0/K52r 1g1SGelgkHOt53Nz4BYUnCqSWiN1S7PhpBrV0WSukVrOtgm0gljv2vY0E/FKwxpHSx6b /U4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rduJSBHNOfADwB39FsLCzXunOBvEcaJu6tUh95KODL4=; b=ecncXf3m9X9CWr0U7+tgNVrWNlRPcOEwGWmqD7jEM/9+dyW7HvWkRvvp9ccuj/FIKA 9MGofs137Im/J9fHmEm59nM2hDr8RVI9IV3YHU1llBdCA8gGrmjBQRwjetQxAsX/Ay8v T3/VyxaKM0qh+zTKCpOrn1Zobg/wmUd2AezEYfpOCfAunWzx3COpredbmAfzTDCYjx8d yMU7DzlClwNUL5ILKiAYFXRaabKaXCs6MMK5WVGeXBu4IIr8aaRbGCAsK8gMQUcPizvN X8pk0H+eXIdMEg+oxPiP2JksPvwrNYFPaG5b6AueJdc4vkL+rtCKSYeDT1wa0WeJdnsu H+IQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@streamunlimited.com header.s=google header.b=fdwnhavs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=streamunlimited.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cs16-20020a170906dc9000b007add0e1a187si11540234ejc.594.2022.12.14.06.02.43; Wed, 14 Dec 2022 06:03:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@streamunlimited.com header.s=google header.b=fdwnhavs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=streamunlimited.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238413AbiLNNsI (ORCPT <rfc822;jeantsuru.cumc.mandola@gmail.com> + 99 others); Wed, 14 Dec 2022 08:48:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238313AbiLNNsG (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 14 Dec 2022 08:48:06 -0500 Received: from mail-yb1-xb30.google.com (mail-yb1-xb30.google.com [IPv6:2607:f8b0:4864:20::b30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6E9626AB8 for <linux-kernel@vger.kernel.org>; Wed, 14 Dec 2022 05:48:03 -0800 (PST) Received: by mail-yb1-xb30.google.com with SMTP id e15so3307183ybb.2 for <linux-kernel@vger.kernel.org>; Wed, 14 Dec 2022 05:48:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamunlimited.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rduJSBHNOfADwB39FsLCzXunOBvEcaJu6tUh95KODL4=; b=fdwnhavsGqlgQt7e1eaz36uYU8Z77IBoLK4pevJAqvSoZWgxTQUVZ6VtElmj0yFLJF Ufo+Nsy/BTv1p3N4pQUL4uYYI8A3yXLzI6rGOnVNy7F45EiZT39czJ9slCdQnpiP4cXz PUbm5mvjGFiV7BXlDC5zbiTWA9nhEzN8QR1gM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rduJSBHNOfADwB39FsLCzXunOBvEcaJu6tUh95KODL4=; b=N8d1uCcNQHaLd8lJbYlTPwXlNopAIxUsAjqhiS0WyKjX9v7DYK+B6Inumky9mIIy9H 8Zc81eBXoX/g1CHmrBlqKizbPBXfy2FegqGR01H9ldC7rFG5ucJOynxUIJ8i0Gku/Zbs NH3A5KhdTMg+w+AGMRofKImu9GgIm3w9r4dHwG2Nv5cWr6L1twagHalHQ9ZFWT0OC6Ar bXDE/aU3wQGRNU5vAuYA5jtXFJIu3K8z1SJT49p4rRfb6lKzqa8NU1ocbJghPfM8BtRF 2oW8d/KsibC/C6X75pVfOemzNFy37Lwt7M77hwkAEWj97m4bzKrwp5nOXAImYut8QOm6 izMw== X-Gm-Message-State: ANoB5pmkMNrAyeyhiaIj36p2z1u3lJLbDzQFgYrxOibne0FjncZGAzH7 vv+35fL1hIl6C+Gs/uBgXfA0HA== X-Received: by 2002:a05:6902:147:b0:724:6d18:6c36 with SMTP id p7-20020a056902014700b007246d186c36mr11589008ybh.23.1671025683021; Wed, 14 Dec 2022 05:48:03 -0800 (PST) Received: from localhost.localdomain (vpn.streamunlimited.com. [91.114.0.140]) by smtp.gmail.com with ESMTPSA id f10-20020a05620a408a00b006eeb3165565sm10139602qko.80.2022.12.14.05.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Dec 2022 05:48:02 -0800 (PST) From: Peter Suti <peter.suti@streamunlimited.com> To: Ulf Hansson <ulf.hansson@linaro.org>, Neil Armstrong <neil.armstrong@linaro.org>, Kevin Hilman <khilman@baylibre.com>, Jerome Brunet <jbrunet@baylibre.com>, Martin Blumenstingl <martin.blumenstingl@googlemail.com>, Matthias Brugger <matthias.bgg@gmail.com>, Heiner Kallweit <hkallweit1@gmail.com> Cc: Peter Suti <peter.suti@streamunlimited.com>, linux-mmc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-amlogic@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mediatek@lists.infradead.org Subject: [PATCH v3] mmc: meson-gx: fix SDIO interrupt handling Date: Wed, 14 Dec 2022 14:46:21 +0100 Message-Id: <20221214134620.3028726-1-peter.suti@streamunlimited.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <52861a84-0fe2-37f0-d66a-145f2ebe1d79@gmail.com> References: <52861a84-0fe2-37f0-d66a-145f2ebe1d79@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749464611573751529?= X-GMAIL-MSGID: =?utf-8?q?1752198401934906536?= |
Series |
[v3] mmc: meson-gx: fix SDIO interrupt handling
|
|
Commit Message
Peter Suti
Dec. 14, 2022, 1:46 p.m. UTC
With the interrupt support introduced in commit 066ecde sometimes the
Marvell-8987 wifi chip got stuck using the marvell-sd-uapsta-8987
vendor driver. The cause seems to be that after sending ack to all interrupts
the IRQ_SDIO still happens, but it is ignored.
To work around this, recheck the IRQ_SDIO after meson_mmc_request_done().
Inspired by 9e2582e ("mmc: mediatek: fix SDIO irq issue") which used a
similar fix to handle lost interrupts.
Fixes: 066ecde ("mmc: meson-gx: add SDIO interrupt support")
Signed-off-by: Peter Suti <peter.suti@streamunlimited.com>
---
Changes in v2:
- use spin_lock instead of spin_lock_irqsave
- only reenable interrupts if they were enabled already
Changes in v3:
- Rework the patch based on feedback from Heiner Kallweit.
The IRQ does not happen on 2 CPUs and the hard IRQ is not re-entrant.
But still one SDIO IRQ is lost without this change.
After the ack, reading the SD_EMMC_STATUS BIT(15) is set, but
meson_mmc_irq() is never called again.
The fix is similar to Mediatek msdc_recheck_sdio_irq().
That platform also loses an IRQ in some cases it seems.
drivers/mmc/host/meson-gx-mmc.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
Comments
On 14.12.2022 14:46, Peter Suti wrote: > With the interrupt support introduced in commit 066ecde sometimes the > Marvell-8987 wifi chip got stuck using the marvell-sd-uapsta-8987 > vendor driver. The cause seems to be that after sending ack to all interrupts > the IRQ_SDIO still happens, but it is ignored. > > To work around this, recheck the IRQ_SDIO after meson_mmc_request_done(). > > Inspired by 9e2582e ("mmc: mediatek: fix SDIO irq issue") which used a > similar fix to handle lost interrupts. > The commit description of the referenced fix isn't clear with regard to who's fault it is that an interrupt can be lost. I'd interpret it being a silicon bug rather than a kernel/driver bug. Not sure whether it's the case, but it's possible that both vendors use at least parts of the same IP in the MMC block, and therefore the issue pops up here too. > Fixes: 066ecde ("mmc: meson-gx: add SDIO interrupt support") > > Signed-off-by: Peter Suti <peter.suti@streamunlimited.com> > --- > Changes in v2: > - use spin_lock instead of spin_lock_irqsave > - only reenable interrupts if they were enabled already > > Changes in v3: > - Rework the patch based on feedback from Heiner Kallweit. > The IRQ does not happen on 2 CPUs and the hard IRQ is not re-entrant. > But still one SDIO IRQ is lost without this change. > After the ack, reading the SD_EMMC_STATUS BIT(15) is set, but > meson_mmc_irq() is never called again. > > The fix is similar to Mediatek msdc_recheck_sdio_irq(). > That platform also loses an IRQ in some cases it seems. > > drivers/mmc/host/meson-gx-mmc.c | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c > index 6e5ea0213b47..7d3ee2f9a7f6 100644 > --- a/drivers/mmc/host/meson-gx-mmc.c > +++ b/drivers/mmc/host/meson-gx-mmc.c > @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) > if (ret == IRQ_HANDLED) > meson_mmc_request_done(host->mmc, cmd->mrq); > > + /* > + * Sometimes after we ack all raised interrupts, > + * an IRQ_SDIO can still be pending, which can get lost. > + * A reader may scratch his head here and wonder how the interrupt can get lost, and why adding a workaround instead of eliminating the root cause for losing the interrupt. If you can't provide an explanation why the root cause for losing the interrupt can't be fixed, presumably you would have to say that you're adding a workaround for a suspected silicon bug. > + * To prevent this, recheck the IRQ_SDIO here and schedule > + * it to be processed. > + */ > + raw_status = readl(host->regs + SD_EMMC_STATUS); > + status = raw_status & (IRQ_EN_MASK | IRQ_SDIO); This isn't needed here. Why not simply: status = readl(host->regs + SD_EMMC_STATUS); if (status & IRQ_SDIO) ... > + if (status & IRQ_SDIO) { > + spin_lock(&host->lock); > + __meson_mmc_enable_sdio_irq(host->mmc, 0); > + sdio_signal_irq(host->mmc); > + spin_unlock(&host->lock); > + } > + > return ret; > } >
On Wed, Dec 14, 2022 at 10:33 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > On 14.12.2022 14:46, Peter Suti wrote: > > With the interrupt support introduced in commit 066ecde sometimes the > > Marvell-8987 wifi chip got stuck using the marvell-sd-uapsta-8987 > > vendor driver. The cause seems to be that after sending ack to all interrupts > > the IRQ_SDIO still happens, but it is ignored. > > > > To work around this, recheck the IRQ_SDIO after meson_mmc_request_done(). > > > > Inspired by 9e2582e ("mmc: mediatek: fix SDIO irq issue") which used a > > similar fix to handle lost interrupts. > > > The commit description of the referenced fix isn't clear with regard to > who's fault it is that an interrupt can be lost. I'd interpret it being > a silicon bug rather than a kernel/driver bug. Unfortunately I can't confirm that the referenced bug is in the silicon for the original commit either. However a similar workaround works in this case too which is why I referenced that commit. > Not sure whether it's the case, but it's possible that both vendors use > at least parts of the same IP in the MMC block, and therefore the issue > pops up here too. > > > Fixes: 066ecde ("mmc: meson-gx: add SDIO interrupt support") > > > > Signed-off-by: Peter Suti <peter.suti@streamunlimited.com> > > --- > > Changes in v2: > > - use spin_lock instead of spin_lock_irqsave > > - only reenable interrupts if they were enabled already > > > > Changes in v3: > > - Rework the patch based on feedback from Heiner Kallweit. > > The IRQ does not happen on 2 CPUs and the hard IRQ is not re-entrant. > > But still one SDIO IRQ is lost without this change. > > After the ack, reading the SD_EMMC_STATUS BIT(15) is set, but > > meson_mmc_irq() is never called again. > > > > The fix is similar to Mediatek msdc_recheck_sdio_irq(). > > That platform also loses an IRQ in some cases it seems. > > > > drivers/mmc/host/meson-gx-mmc.c | 16 ++++++++++++++++ > > 1 file changed, 16 insertions(+) > > > > diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c > > index 6e5ea0213b47..7d3ee2f9a7f6 100644 > > --- a/drivers/mmc/host/meson-gx-mmc.c > > +++ b/drivers/mmc/host/meson-gx-mmc.c > > @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) > > if (ret == IRQ_HANDLED) > > meson_mmc_request_done(host->mmc, cmd->mrq); > > > > + /* > > + * Sometimes after we ack all raised interrupts, > > + * an IRQ_SDIO can still be pending, which can get lost. > > + * > > A reader may scratch his head here and wonder how the interrupt can get lost, > and why adding a workaround instead of eliminating the root cause for losing > the interrupt. If you can't provide an explanation why the root cause for > losing the interrupt can't be fixed, presumably you would have to say that > you're adding a workaround for a suspected silicon bug. After talking to the manufacturer, we got the following explanation for this situation: "wifi may have dat1 interrupt coming in, without this the dat1 interrupt would be missed" Supposedly this is fixed in their codebase. Unfortunately we were not able to find out more and can't prepare a patch with a proper explanation. Thank you for reviewing. > > > + * To prevent this, recheck the IRQ_SDIO here and schedule > > + * it to be processed. > > + */ > > + raw_status = readl(host->regs + SD_EMMC_STATUS); > > + status = raw_status & (IRQ_EN_MASK | IRQ_SDIO); > > This isn't needed here. Why not simply: > > status = readl(host->regs + SD_EMMC_STATUS); > if (status & IRQ_SDIO) > ... > > > > + if (status & IRQ_SDIO) { > > + spin_lock(&host->lock); > > + __meson_mmc_enable_sdio_irq(host->mmc, 0); > > + sdio_signal_irq(host->mmc); > > + spin_unlock(&host->lock); > > + } > > + > > return ret; > > } > > >
On 09.01.2023 12:52, Peter Suti wrote: > On Wed, Dec 14, 2022 at 10:33 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 14.12.2022 14:46, Peter Suti wrote: >>> With the interrupt support introduced in commit 066ecde sometimes the >>> Marvell-8987 wifi chip got stuck using the marvell-sd-uapsta-8987 >>> vendor driver. The cause seems to be that after sending ack to all interrupts >>> the IRQ_SDIO still happens, but it is ignored. >>> >>> To work around this, recheck the IRQ_SDIO after meson_mmc_request_done(). >>> >>> Inspired by 9e2582e ("mmc: mediatek: fix SDIO irq issue") which used a >>> similar fix to handle lost interrupts. >>> >> The commit description of the referenced fix isn't clear with regard to >> who's fault it is that an interrupt can be lost. I'd interpret it being >> a silicon bug rather than a kernel/driver bug. > Unfortunately I can't confirm that the referenced bug is in the > silicon for the original commit either. > However a similar workaround works in this case too which is why I > referenced that commit. > >> Not sure whether it's the case, but it's possible that both vendors use >> at least parts of the same IP in the MMC block, and therefore the issue >> pops up here too. >> >>> Fixes: 066ecde ("mmc: meson-gx: add SDIO interrupt support") >>> >>> Signed-off-by: Peter Suti <peter.suti@streamunlimited.com> >>> --- >>> Changes in v2: >>> - use spin_lock instead of spin_lock_irqsave >>> - only reenable interrupts if they were enabled already >>> >>> Changes in v3: >>> - Rework the patch based on feedback from Heiner Kallweit. >>> The IRQ does not happen on 2 CPUs and the hard IRQ is not re-entrant. >>> But still one SDIO IRQ is lost without this change. >>> After the ack, reading the SD_EMMC_STATUS BIT(15) is set, but >>> meson_mmc_irq() is never called again. >>> >>> The fix is similar to Mediatek msdc_recheck_sdio_irq(). >>> That platform also loses an IRQ in some cases it seems. >>> >>> drivers/mmc/host/meson-gx-mmc.c | 16 ++++++++++++++++ >>> 1 file changed, 16 insertions(+) >>> >>> diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c >>> index 6e5ea0213b47..7d3ee2f9a7f6 100644 >>> --- a/drivers/mmc/host/meson-gx-mmc.c >>> +++ b/drivers/mmc/host/meson-gx-mmc.c >>> @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) >>> if (ret == IRQ_HANDLED) >>> meson_mmc_request_done(host->mmc, cmd->mrq); >>> >>> + /* >>> + * Sometimes after we ack all raised interrupts, >>> + * an IRQ_SDIO can still be pending, which can get lost. >>> + * >> >> A reader may scratch his head here and wonder how the interrupt can get lost, >> and why adding a workaround instead of eliminating the root cause for losing >> the interrupt. If you can't provide an explanation why the root cause for >> losing the interrupt can't be fixed, presumably you would have to say that >> you're adding a workaround for a suspected silicon bug. > After talking to the manufacturer, we got the following explanation > for this situation: To which manufacturer did you talk, Marvell or Amlogic? > "wifi may have dat1 interrupt coming in, without this the dat1 > interrupt would be missed" I don't understand this sentence. W/o the interrupt coming in the interrupt would be missed? Can you explain it? > Supposedly this is fixed in their codebase. Which codebase do you mean? A specific vendor driver? Or firmware? > Unfortunately we were not able to find out more and can't prepare a > patch with a proper explanation. The workaround is rather simple, so we should add it. It's just unfortunate that we have no idea about the root cause of the issue. > Thank you for reviewing. >> >>> + * To prevent this, recheck the IRQ_SDIO here and schedule >>> + * it to be processed. >>> + */ >>> + raw_status = readl(host->regs + SD_EMMC_STATUS); >>> + status = raw_status & (IRQ_EN_MASK | IRQ_SDIO); >> >> This isn't needed here. Why not simply: >> >> status = readl(host->regs + SD_EMMC_STATUS); >> if (status & IRQ_SDIO) >> ... >> >> >>> + if (status & IRQ_SDIO) { >>> + spin_lock(&host->lock); >>> + __meson_mmc_enable_sdio_irq(host->mmc, 0); >>> + sdio_signal_irq(host->mmc); >>> + spin_unlock(&host->lock); >>> + } >>> + >>> return ret; >>> } >>> >>
On Thu, Jan 12, 2023 at 10:27 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > On 09.01.2023 12:52, Peter Suti wrote: > > On Wed, Dec 14, 2022 at 10:33 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: > >> > >> On 14.12.2022 14:46, Peter Suti wrote: > >>> > >>> diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c > >>> index 6e5ea0213b47..7d3ee2f9a7f6 100644 > >>> --- a/drivers/mmc/host/meson-gx-mmc.c > >>> +++ b/drivers/mmc/host/meson-gx-mmc.c > >>> @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) > >>> if (ret == IRQ_HANDLED) > >>> meson_mmc_request_done(host->mmc, cmd->mrq); > >>> > >>> + /* > >>> + * Sometimes after we ack all raised interrupts, > >>> + * an IRQ_SDIO can still be pending, which can get lost. > >>> + * > >> > >> A reader may scratch his head here and wonder how the interrupt can get lost, > >> and why adding a workaround instead of eliminating the root cause for losing > >> the interrupt. If you can't provide an explanation why the root cause for > >> losing the interrupt can't be fixed, presumably you would have to say that > >> you're adding a workaround for a suspected silicon bug. > > After talking to the manufacturer, we got the following explanation > > for this situation: > > To which manufacturer did you talk, Marvell or Amlogic? Amlogic > > > "wifi may have dat1 interrupt coming in, without this the dat1 > > interrupt would be missed" > > I don't understand this sentence. W/o the interrupt coming in > the interrupt would be missed? Can you explain it? So the "without this" part of that sentence referred to the patch. Which means that without the patch, the interrupt can be missed. > > > Supposedly this is fixed in their codebase. > > Which codebase do you mean? A specific vendor driver? Or firmware? The vendor driver from openlinux2.amlogic.com handles SDIO interrupts a bit differently. It uses mmc_signal_sdio_irq() > > > Unfortunately we were not able to find out more and can't prepare a > > patch with a proper explanation. > > The workaround is rather simple, so we should add it. > It's just unfortunate that we have no idea about the root cause of the issue. After doing a more long term stress test, it was revealed that this patch is still not sufficient, but only masks the underlying problem better. Reverting 066ecde fixes the issue fully for us (verified overnight with iperf). v1 and v2 also fix the issue, but v3 does not correct the bug when WiFi is stressed for a longer time. And therefore it should not be used. Could you please give us some advice on how to investigate this further?
On 18.01.2023 14:32, Peter Suti wrote: > On Thu, Jan 12, 2023 at 10:27 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 09.01.2023 12:52, Peter Suti wrote: >>> On Wed, Dec 14, 2022 at 10:33 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >>>> >>>> On 14.12.2022 14:46, Peter Suti wrote: >>>>> >>>>> diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c >>>>> index 6e5ea0213b47..7d3ee2f9a7f6 100644 >>>>> --- a/drivers/mmc/host/meson-gx-mmc.c >>>>> +++ b/drivers/mmc/host/meson-gx-mmc.c >>>>> @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) >>>>> if (ret == IRQ_HANDLED) >>>>> meson_mmc_request_done(host->mmc, cmd->mrq); >>>>> >>>>> + /* >>>>> + * Sometimes after we ack all raised interrupts, >>>>> + * an IRQ_SDIO can still be pending, which can get lost. >>>>> + * >>>> >>>> A reader may scratch his head here and wonder how the interrupt can get lost, >>>> and why adding a workaround instead of eliminating the root cause for losing >>>> the interrupt. If you can't provide an explanation why the root cause for >>>> losing the interrupt can't be fixed, presumably you would have to say that >>>> you're adding a workaround for a suspected silicon bug. >>> After talking to the manufacturer, we got the following explanation >>> for this situation: >> >> To which manufacturer did you talk, Marvell or Amlogic? > > Amlogic > >> >>> "wifi may have dat1 interrupt coming in, without this the dat1 >>> interrupt would be missed" >> >> I don't understand this sentence. W/o the interrupt coming in >> the interrupt would be missed? Can you explain it? > > So the "without this" part of that sentence referred to the patch. > Which means that without the patch, the interrupt can be missed. > Did you ask Amlogic for the root cause of the problem? Silicon bug? >> >>> Supposedly this is fixed in their codebase. >> >> Which codebase do you mean? A specific vendor driver? Or firmware? > > The vendor driver from openlinux2.amlogic.com handles SDIO interrupts > a bit differently. It uses mmc_signal_sdio_irq() > This is the legacy API for SDIO irq. >> >>> Unfortunately we were not able to find out more and can't prepare a >>> patch with a proper explanation. >> >> The workaround is rather simple, so we should add it. >> It's just unfortunate that we have no idea about the root cause of the issue. > > After doing a more long term stress test, it was revealed that this > patch is still not sufficient, but only masks the underlying problem > better. Reverting 066ecde fixes the issue fully for us (verified > overnight with iperf). > v1 and v2 also fix the issue, but v3 does not correct the bug when > WiFi is stressed for a longer time. And therefore it should not be > used. > Could you please give us some advice on how to investigate this further? When talking about v2 of the patch, it includes a number of actual changes: - hold lock from start to end of interrupt handler - if SDIO irq was disabled when entering the irq handler I see no change - if SDIO irq was enabled - disable SDIO irq at start of irq handler, no matter which type of interrupt was received -> means disable SDIO irq during irq handler even if some other interrupt was received - if SDIO and another interrupt source flag is set, leave irq handler with SDIO irq enabled. Not sure whether that's intentional, you can use the small patch listed below on top of v2 to check whether SDIO irq still works ok or not. At least to me it's not clear which (or all?) of the changes are needed to work around the problem. So you could check whether SDIO irq still works fine if you remove a single change from the patch. diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c index f4b102b85..711431706 100644 --- a/drivers/mmc/host/meson-gx-mmc.c +++ b/drivers/mmc/host/meson-gx-mmc.c @@ -994,6 +994,7 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) spin_unlock(&host->lock); return IRQ_HANDLED; } + irq_enabled = false; } if (WARN_ON(!cmd))
On 18.01.2023 14:32, Peter Suti wrote: > On Thu, Jan 12, 2023 at 10:27 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 09.01.2023 12:52, Peter Suti wrote: >>> On Wed, Dec 14, 2022 at 10:33 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >>>> >>>> On 14.12.2022 14:46, Peter Suti wrote: >>>>> >>>>> diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c >>>>> index 6e5ea0213b47..7d3ee2f9a7f6 100644 >>>>> --- a/drivers/mmc/host/meson-gx-mmc.c >>>>> +++ b/drivers/mmc/host/meson-gx-mmc.c >>>>> @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) >>>>> if (ret == IRQ_HANDLED) >>>>> meson_mmc_request_done(host->mmc, cmd->mrq); >>>>> >>>>> + /* >>>>> + * Sometimes after we ack all raised interrupts, >>>>> + * an IRQ_SDIO can still be pending, which can get lost. >>>>> + * >>>> >>>> A reader may scratch his head here and wonder how the interrupt can get lost, >>>> and why adding a workaround instead of eliminating the root cause for losing >>>> the interrupt. If you can't provide an explanation why the root cause for >>>> losing the interrupt can't be fixed, presumably you would have to say that >>>> you're adding a workaround for a suspected silicon bug. >>> After talking to the manufacturer, we got the following explanation >>> for this situation: >> >> To which manufacturer did you talk, Marvell or Amlogic? > > Amlogic > >> >>> "wifi may have dat1 interrupt coming in, without this the dat1 >>> interrupt would be missed" >> >> I don't understand this sentence. W/o the interrupt coming in >> the interrupt would be missed? Can you explain it? > > So the "without this" part of that sentence referred to the patch. > Which means that without the patch, the interrupt can be missed. > >> >>> Supposedly this is fixed in their codebase. >> >> Which codebase do you mean? A specific vendor driver? Or firmware? > > The vendor driver from openlinux2.amlogic.com handles SDIO interrupts > a bit differently. It uses mmc_signal_sdio_irq() > >> >>> Unfortunately we were not able to find out more and can't prepare a >>> patch with a proper explanation. >> >> The workaround is rather simple, so we should add it. >> It's just unfortunate that we have no idea about the root cause of the issue. > > After doing a more long term stress test, it was revealed that this > patch is still not sufficient, but only masks the underlying problem > better. Reverting 066ecde fixes the issue fully for us (verified > overnight with iperf). > v1 and v2 also fix the issue, but v3 does not correct the bug when > WiFi is stressed for a longer time. And therefore it should not be > used. > Could you please give us some advice on how to investigate this further? One more thought: When checking device tree I found that my system uses IRQ_TYPE_LEVEL_HIGH for the SDIO interrupt. meson-g12-common.dtsi uses IRQ_TYPE_EDGE_RISING in mainline Linux, however vendor kernel uses IRQ_TYPE_LEVEL_HIGH in meson-g12-common.dtsi. A wrong interrupt trigger type may result in lost interrupts. So you could check which trigger type your system uses for the SDIO interrupt. If it's IRQ_TYPE_EDGE_RISING, re-test with IRQ_TYPE_LEVEL_HIGH.
On Thu, Jan 19, 2023 at 8:37 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > > v1 and v2 also fix the issue, but v3 does not correct the bug when > > WiFi is stressed for a longer time. And therefore it should not be > > used. > > Could you please give us some advice on how to investigate this further? > > One more thought: > When checking device tree I found that my system uses IRQ_TYPE_LEVEL_HIGH > for the SDIO interrupt. meson-g12-common.dtsi uses IRQ_TYPE_EDGE_RISING > in mainline Linux, however vendor kernel uses IRQ_TYPE_LEVEL_HIGH > in meson-g12-common.dtsi. A wrong interrupt trigger type may result in > lost interrupts. > So you could check which trigger type your system uses for the SDIO interrupt. > If it's IRQ_TYPE_EDGE_RISING, re-test with IRQ_TYPE_LEVEL_HIGH. Thank you for this suggestion, this change fixes the problem fully. With the interrupt trigger type set to IRQ_TYPE_LEVEL_HIGH everything works as expected even when WiFI is stressed for a long time.
diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c index 6e5ea0213b47..7d3ee2f9a7f6 100644 --- a/drivers/mmc/host/meson-gx-mmc.c +++ b/drivers/mmc/host/meson-gx-mmc.c @@ -1023,6 +1023,22 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) if (ret == IRQ_HANDLED) meson_mmc_request_done(host->mmc, cmd->mrq); + /* + * Sometimes after we ack all raised interrupts, + * an IRQ_SDIO can still be pending, which can get lost. + * + * To prevent this, recheck the IRQ_SDIO here and schedule + * it to be processed. + */ + raw_status = readl(host->regs + SD_EMMC_STATUS); + status = raw_status & (IRQ_EN_MASK | IRQ_SDIO); + if (status & IRQ_SDIO) { + spin_lock(&host->lock); + __meson_mmc_enable_sdio_irq(host->mmc, 0); + sdio_signal_irq(host->mmc); + spin_unlock(&host->lock); + } + return ret; }