Message ID | 20230517105235.29176-10-ilpo.jarvinen@linux.intel.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1044677vqo; Wed, 17 May 2023 04:06:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7//9XkXn+LUVPBTxKSroSQU++ICHVbCNFSeiI7pQab22caAG2xkF1EdSyWGKABq6bbsS9Z X-Received: by 2002:a17:902:c950:b0:1a1:d54b:71df with SMTP id i16-20020a170902c95000b001a1d54b71dfmr2011815pla.0.1684321603828; Wed, 17 May 2023 04:06:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684321603; cv=none; d=google.com; s=arc-20160816; b=XNYU4BlIrUtzPDtL9wmNS352QECY92sJy4XUTZeRxeaCSNMrP5uf1FgiUlp06AT5Qm Gd/T8ZKvhJmSsPz5+014err59neuVtE4DnrMPVsrcrLD9Av2Xx9Z+AgPrM5k1qZMfMoZ dEOQ06uWPOqCYNeNUrakSgle2zHPU7A32UYvjq45xv6MMvLCKHxkWGa/oIZ4l5OGQTHo ZE7iVueZHqLhnoHS0U4phdPJLRoqK3Jf9pEL/O6kxIoAZxlWIqEpyKHjLu0bvR0SdA2L zE18cR5Ylu0vmcDuuHgbxEwAwgqWn2XQQ0gxldoL6BDTCOvlopX9Zrs9YZg1a3NH1sx6 SfKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2axjmF+AM7q79nZADlnqZG+DqsQaGGOui7TPu6aZKsU=; b=EULbioWUBSt9CHR0Q7jw6wJDzHI8/JuSaH4m0aIpb/foosoT7zSuKqi/xdKVxwFtSK ide7mdgpB2hQtZRlmX0HVwGNzwXvooFq1SbfaftU2osdv8yzxZbOMiMJsqf9ImlcTjvQ AxN9ETRbdsye/DMwrbceiZu3o15vj5Q80VnIs08lisFYzpQycELi1hRMz3Kkc5mbNYPk 7mGygoHssaH3O1VEFLrTRAmanPsBJCh6Pq1/V/jqw0uyt5D9aGY4O4ba6EwGlno8VNQW iKZ09Ae/KnM4QXdFC9JosJXwf80X5KYCuEK2eaZh2pL78fWruVB4L7PEnyn+QEDPwtlD jHUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kQSZsMOP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u2-20020a170902e5c200b0019f33cb669asi22086915plf.615.2023.05.17.04.06.28; Wed, 17 May 2023 04:06:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kQSZsMOP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230514AbjEQK6o (ORCPT <rfc822;pacteraone@gmail.com> + 99 others); Wed, 17 May 2023 06:58:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230216AbjEQK6l (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 17 May 2023 06:58:41 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FE4B61AD; Wed, 17 May 2023 03:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684321090; x=1715857090; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FuFXzjOd8Jq9CElj4g5CQDGQFvJ4smZ3IyuF6JWPb+E=; b=kQSZsMOP7zbF5CIsF2HnMdNXkYQo4aP36Lu9xtRHy9xYKfeyHhzjwgkl 3oAILAs6t+0q6cUuUzN1UNqt6GBKKZ89XWP8DjVBFVbJUeh78oh9XiRYr uZ1AKfjKnv+/aHUXdMIrDHZMRE0iVEIsU8llkpPwlkcOCziLj/AdKrtHo +50FeaZEfiLfpsIjamNihbBv7t9comLqJxJ+MdVbIiXet2paXZnSWr0bK I7b9I1KiJ5tBNBw9OcZsszAQ2Kamo0dfTzJ0o7twnXBDtLodgndiB60wr jzGuB9p6j5QPcA6Ia8eYhOjC3TH32lEOHVHOy6D4o63EZZDHqurwFw2mz Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10712"; a="438071923" X-IronPort-AV: E=Sophos;i="5.99,281,1677571200"; d="scan'208";a="438071923" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 03:54:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10712"; a="734651392" X-IronPort-AV: E=Sophos;i="5.99,281,1677571200"; d="scan'208";a="734651392" Received: from lnstern-mobl1.amr.corp.intel.com (HELO ijarvine-MOBL2.ger.corp.intel.com) ([10.251.221.185]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 03:53:59 -0700 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com> To: linux-pci@vger.kernel.org, Bjorn Helgaas <bhelgaas@google.com>, Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>, Rob Herring <robh@kernel.org>, =?utf-8?q?Krzysztof_Wilczy=C5=84ski?= <kw@linux.com>, Emmanuel Grumbach <emmanuel.grumbach@intel.com>, "Rafael J . Wysocki" <rafael@kernel.org>, Heiner Kallweit <hkallweit1@gmail.com>, Lukas Wunner <lukas@wunner.de>, Kalle Valo <kvalo@kernel.org>, "David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, Michal Kazior <michal.kazior@tieto.com>, Janusz Dziedzic <janusz.dziedzic@tieto.com>, ath10k@lists.infradead.org, linux-wireless@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Dean Luick <dean.luick@cornelisnetworks.com>, Andy Shevchenko <andriy.shevchenko@linux.intel.com>, =?utf-8?q?Ilpo_J=C3=A4?= =?utf-8?q?rvinen?= <ilpo.jarvinen@linux.intel.com>, stable@vger.kernel.org Subject: [PATCH v2 9/9] wifi: ath10k: Use RMW accessors for changing LNKCTL Date: Wed, 17 May 2023 13:52:35 +0300 Message-Id: <20230517105235.29176-10-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230517105235.29176-1-ilpo.jarvinen@linux.intel.com> References: <20230517105235.29176-1-ilpo.jarvinen@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766139210142300560?= X-GMAIL-MSGID: =?utf-8?q?1766139210142300560?= |
Series |
PCI: Improve PCIe Capability RMW concurrency control
|
|
Commit Message
Ilpo Järvinen
May 17, 2023, 10:52 a.m. UTC
Don't assume that only the driver would be accessing LNKCTL. ASPM
policy changes can trigger write to LNKCTL outside of driver's control.
Use RMW capability accessors which does proper locking to avoid losing
concurrent updates to the register value. On restore, clear the ASPMC
field properly.
Fixes: 76d870ed09ab ("ath10k: enable ASPM")
Suggested-by: Lukas Wunner <lukas@wunner.de>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Cc: stable@vger.kernel.org
---
drivers/net/wireless/ath/ath10k/pci.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
Comments
Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes: > Don't assume that only the driver would be accessing LNKCTL. ASPM > policy changes can trigger write to LNKCTL outside of driver's control. > > Use RMW capability accessors which does proper locking to avoid losing > concurrent updates to the register value. On restore, clear the ASPMC > field properly. > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > Suggested-by: Lukas Wunner <lukas@wunner.de> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > Cc: stable@vger.kernel.org Acked-by: Kalle Valo <kvalo@kernel.org>
On Wed, May 17, 2023 at 01:52:35PM +0300, Ilpo Järvinen wrote: > Don't assume that only the driver would be accessing LNKCTL. ASPM > policy changes can trigger write to LNKCTL outside of driver's control. > > Use RMW capability accessors which does proper locking to avoid losing > concurrent updates to the register value. On restore, clear the ASPMC > field properly. > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > Suggested-by: Lukas Wunner <lukas@wunner.de> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > Cc: stable@vger.kernel.org > --- > drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- > 1 file changed, 5 insertions(+), 4 deletions(-) > > diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c > index a7f44f6335fb..9275a672f90c 100644 > --- a/drivers/net/wireless/ath/ath10k/pci.c > +++ b/drivers/net/wireless/ath/ath10k/pci.c > @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) > ath10k_pci_irq_enable(ar); > ath10k_pci_rx_post(ar); > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > - ar_pci->link_ctl); > + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, > + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); > > return 0; > } > @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, > > pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, > &ar_pci->link_ctl); > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); > + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC); These ath drivers all have the form: 1) read LNKCTL 2) save LNKCTL value in ->link_ctl 3) write LNKCTL with "->link_ctl & ~PCI_EXP_LNKCTL_ASPMC" to disable ASPM 4) write LNKCTL with ->link_ctl, presumably to re-enable ASPM These patches close the hole between 1) and 3) where other LNKCTL updates could interfere, which is definitely a good thing. But the hole between 1) and 4) is much bigger and still there. Any update by the PCI core in that interval would be lost. Straw-man proposal: - Change pci_disable_link_state() so it ignores aspm_disabled and always disables ASPM even if platform firmware hasn't granted ownership. Maybe this should warn and taint the kernel. - Change drivers to use pci_disable_link_state() instead of writing LNKCTL directly. Bjorn
On Wed, 24 May 2023, Bjorn Helgaas wrote: > On Wed, May 17, 2023 at 01:52:35PM +0300, Ilpo Järvinen wrote: > > Don't assume that only the driver would be accessing LNKCTL. ASPM > > policy changes can trigger write to LNKCTL outside of driver's control. > > > > Use RMW capability accessors which does proper locking to avoid losing > > concurrent updates to the register value. On restore, clear the ASPMC > > field properly. > > > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > > Suggested-by: Lukas Wunner <lukas@wunner.de> > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > Cc: stable@vger.kernel.org > > --- > > drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- > > 1 file changed, 5 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c > > index a7f44f6335fb..9275a672f90c 100644 > > --- a/drivers/net/wireless/ath/ath10k/pci.c > > +++ b/drivers/net/wireless/ath/ath10k/pci.c > > @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) > > ath10k_pci_irq_enable(ar); > > ath10k_pci_rx_post(ar); > > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > - ar_pci->link_ctl); > > + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > + PCI_EXP_LNKCTL_ASPMC, > > + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); > > > > return 0; > > } > > @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, > > > > pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > &ar_pci->link_ctl); > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); > > + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > + PCI_EXP_LNKCTL_ASPMC); > > These ath drivers all have the form: > > 1) read LNKCTL > 2) save LNKCTL value in ->link_ctl > 3) write LNKCTL with "->link_ctl & ~PCI_EXP_LNKCTL_ASPMC" > to disable ASPM > 4) write LNKCTL with ->link_ctl, presumably to re-enable ASPM > > These patches close the hole between 1) and 3) where other LNKCTL > updates could interfere, which is definitely a good thing. > > But the hole between 1) and 4) is much bigger and still there. Any > update by the PCI core in that interval would be lost. Any update to PCI_EXP_LNKCTL_ASPMC field in that interval is lost yes, the updates to _the other fields_ in LNKCTL are not lost. I know this might result in drivers/pci/pcie/aspm.c disagreeing what the state of the ASPM is (as shown under sysfs) compared with LNKCTL value but the cause can no longer be due racing RMW. Essentially, 4) is seen as an override to what core did if it changed ASPMC in between. Technically, something is still "lost" like you say but for a different reason than this series is trying to fix. > Straw-man proposal: > > - Change pci_disable_link_state() so it ignores aspm_disabled and > always disables ASPM even if platform firmware hasn't granted > ownership. Maybe this should warn and taint the kernel. > > - Change drivers to use pci_disable_link_state() instead of writing > LNKCTL directly. I fully agree that's the direction we should be moving, yes. However, I'm a bit hesitant to take that leap in one step. These drivers currently not only disable ASPM but also re-enable it (assuming we guessed the intent right). If I directly implement that proposal, ASPM is not going to be re-enabled when PCI core does not allowing it. Could it cause some power related regression? My plan is to make another patch series after these to realize exactly what you're proposing. It would allow better to isolate the problems that related to the lack of ASPM. I hope this two step approach is an acceptable way forward? I can of course add those patches on top of these if that would be preferrable.
On Thu, 25 May 2023, Ilpo Järvinen wrote: > On Wed, 24 May 2023, Bjorn Helgaas wrote: > > > On Wed, May 17, 2023 at 01:52:35PM +0300, Ilpo Järvinen wrote: > > > Don't assume that only the driver would be accessing LNKCTL. ASPM > > > policy changes can trigger write to LNKCTL outside of driver's control. > > > > > > Use RMW capability accessors which does proper locking to avoid losing > > > concurrent updates to the register value. On restore, clear the ASPMC > > > field properly. > > > > > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > > > Suggested-by: Lukas Wunner <lukas@wunner.de> > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > > Cc: stable@vger.kernel.org > > > --- > > > drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- > > > 1 file changed, 5 insertions(+), 4 deletions(-) > > > > > > diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c > > > index a7f44f6335fb..9275a672f90c 100644 > > > --- a/drivers/net/wireless/ath/ath10k/pci.c > > > +++ b/drivers/net/wireless/ath/ath10k/pci.c > > > @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) > > > ath10k_pci_irq_enable(ar); > > > ath10k_pci_rx_post(ar); > > > > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > - ar_pci->link_ctl); > > > + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > + PCI_EXP_LNKCTL_ASPMC, > > > + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); > > > > > > return 0; > > > } > > > @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, > > > > > > pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > &ar_pci->link_ctl); > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); > > > + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > + PCI_EXP_LNKCTL_ASPMC); > > > > These ath drivers all have the form: > > > > 1) read LNKCTL > > 2) save LNKCTL value in ->link_ctl > > 3) write LNKCTL with "->link_ctl & ~PCI_EXP_LNKCTL_ASPMC" > > to disable ASPM > > 4) write LNKCTL with ->link_ctl, presumably to re-enable ASPM > > > > These patches close the hole between 1) and 3) where other LNKCTL > > updates could interfere, which is definitely a good thing. > > > > But the hole between 1) and 4) is much bigger and still there. Any > > update by the PCI core in that interval would be lost. > > Any update to PCI_EXP_LNKCTL_ASPMC field in that interval is lost yes, the > updates to _the other fields_ in LNKCTL are not lost. > > I know this might result in drivers/pci/pcie/aspm.c disagreeing what > the state of the ASPM is (as shown under sysfs) compared with LNKCTL > value but the cause can no longer be due racing RMW. Essentially, 4) is > seen as an override to what core did if it changed ASPMC in between. > Technically, something is still "lost" like you say but for a different > reason than this series is trying to fix. > > > Straw-man proposal: > > > > - Change pci_disable_link_state() so it ignores aspm_disabled and > > always disables ASPM even if platform firmware hasn't granted > > ownership. Maybe this should warn and taint the kernel. > > > > - Change drivers to use pci_disable_link_state() instead of writing > > LNKCTL directly. Now that I took a deeper look into what pci_disable_link_state() and pci_enable_link_state() do, I realized they're not really disable/enable pair like I had assumed from their names. Disable adds to ->aspm_disable and flags are never removed from that because enable does not touch aspm_disable at all but has it's own flag variable. This asymmetry looks intentional. So if ath drivers would do pci_disable_link_state() to realize 1)-3), there is no way to undo it in 4). It looks as if ath drivers would actually want to use pci_enable_link_state() with different state parameters to realize what they want to do in 1)-4). Any suggestion which way I should go with these ath drivers here, use pci_enable_link_state()? (There are other drivers where pci_disable_link_state() is very much valid thing to do.)
On Thu, May 25, 2023 at 01:11:51PM +0300, Ilpo Järvinen wrote: > On Wed, 24 May 2023, Bjorn Helgaas wrote: > > On Wed, May 17, 2023 at 01:52:35PM +0300, Ilpo Järvinen wrote: > > > Don't assume that only the driver would be accessing LNKCTL. ASPM > > > policy changes can trigger write to LNKCTL outside of driver's control. > > > > > > Use RMW capability accessors which does proper locking to avoid losing > > > concurrent updates to the register value. On restore, clear the ASPMC > > > field properly. > > > > > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > > > Suggested-by: Lukas Wunner <lukas@wunner.de> > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > > Cc: stable@vger.kernel.org > > > --- > > > drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- > > > 1 file changed, 5 insertions(+), 4 deletions(-) > > > > > > diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c > > > index a7f44f6335fb..9275a672f90c 100644 > > > --- a/drivers/net/wireless/ath/ath10k/pci.c > > > +++ b/drivers/net/wireless/ath/ath10k/pci.c > > > @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) > > > ath10k_pci_irq_enable(ar); > > > ath10k_pci_rx_post(ar); > > > > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > - ar_pci->link_ctl); > > > + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > + PCI_EXP_LNKCTL_ASPMC, > > > + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); > > > > > > return 0; > > > } > > > @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, > > > > > > pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > &ar_pci->link_ctl); > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); > > > + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > + PCI_EXP_LNKCTL_ASPMC); > > > > These ath drivers all have the form: > > > > 1) read LNKCTL > > 2) save LNKCTL value in ->link_ctl > > 3) write LNKCTL with "->link_ctl & ~PCI_EXP_LNKCTL_ASPMC" > > to disable ASPM > > 4) write LNKCTL with ->link_ctl, presumably to re-enable ASPM > > > > These patches close the hole between 1) and 3) where other LNKCTL > > updates could interfere, which is definitely a good thing. > > > > But the hole between 1) and 4) is much bigger and still there. Any > > update by the PCI core in that interval would be lost. > > Any update to PCI_EXP_LNKCTL_ASPMC field in that interval is lost yes, the > updates to _the other fields_ in LNKCTL are not lost. Ah, yes, you're right, I missed the masking to PCI_EXP_LNKCTL_ASPMC in the pcie_capability_clear_word(). > > Straw-man proposal: > > > > - Change pci_disable_link_state() so it ignores aspm_disabled and > > always disables ASPM even if platform firmware hasn't granted > > ownership. Maybe this should warn and taint the kernel. > > > > - Change drivers to use pci_disable_link_state() instead of writing > > LNKCTL directly. > > I fully agree that's the direction we should be moving, yes. However, I'm > a bit hesitant to take that leap in one step. These drivers currently not > only disable ASPM but also re-enable it (assuming we guessed the intent > right). > > If I directly implement that proposal, ASPM is not going to be re-enabled > when PCI core does not allowing it. Could it cause some power related > regression? IIUC the potential problem only happens with: - A platform that enables ASPM but doesn't grant PCIe Capability ownership to the OS, and - A device where we force-disable ASPM, presumably to avoid some hardware defect. I'm not sure this case is worth worrying about. A platform that enables ASPM without allowing the OS to disable it is taking a risk because it can't know about these device defects or even about user preferences. A device that has an ASPM-related defect may use more power than necessary. I think that's to be expected. > My plan is to make another patch series after these to realize exactly > what you're proposing. It would allow better to isolate the problems that > related to the lack of ASPM. > > I hope this two step approach is an acceptable way forward? I can of > course add those patches on top of these if that would be preferrable. I think two steps is OK. It's a little more work for the driver maintainers to review them, but this step is pretty trivial already reviewed (except for the GPUs, which are probably the most important :)). Bjorn
On Fri, May 26, 2023 at 02:48:44PM +0300, Ilpo Järvinen wrote: > On Thu, 25 May 2023, Ilpo Järvinen wrote: > > On Wed, 24 May 2023, Bjorn Helgaas wrote: > > > On Wed, May 17, 2023 at 01:52:35PM +0300, Ilpo Järvinen wrote: > > > > Don't assume that only the driver would be accessing LNKCTL. ASPM > > > > policy changes can trigger write to LNKCTL outside of driver's control. > > > > > > > > Use RMW capability accessors which does proper locking to avoid losing > > > > concurrent updates to the register value. On restore, clear the ASPMC > > > > field properly. > > > > > > > > Fixes: 76d870ed09ab ("ath10k: enable ASPM") > > > > Suggested-by: Lukas Wunner <lukas@wunner.de> > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > > > Cc: stable@vger.kernel.org > > > > --- > > > > drivers/net/wireless/ath/ath10k/pci.c | 9 +++++---- > > > > 1 file changed, 5 insertions(+), 4 deletions(-) > > > > > > > > diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c > > > > index a7f44f6335fb..9275a672f90c 100644 > > > > --- a/drivers/net/wireless/ath/ath10k/pci.c > > > > +++ b/drivers/net/wireless/ath/ath10k/pci.c > > > > @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) > > > > ath10k_pci_irq_enable(ar); > > > > ath10k_pci_rx_post(ar); > > > > > > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > > - ar_pci->link_ctl); > > > > + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > > + PCI_EXP_LNKCTL_ASPMC, > > > > + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); > > > > > > > > return 0; > > > > } > > > > @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, > > > > > > > > pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > > &ar_pci->link_ctl); > > > > - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > > - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); > > > > + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, > > > > + PCI_EXP_LNKCTL_ASPMC); > > > > > > These ath drivers all have the form: > > > > > > 1) read LNKCTL > > > 2) save LNKCTL value in ->link_ctl > > > 3) write LNKCTL with "->link_ctl & ~PCI_EXP_LNKCTL_ASPMC" > > > to disable ASPM > > > 4) write LNKCTL with ->link_ctl, presumably to re-enable ASPM > > > > > > These patches close the hole between 1) and 3) where other LNKCTL > > > updates could interfere, which is definitely a good thing. > > > > > > But the hole between 1) and 4) is much bigger and still there. Any > > > update by the PCI core in that interval would be lost. > > > > Any update to PCI_EXP_LNKCTL_ASPMC field in that interval is lost yes, the > > updates to _the other fields_ in LNKCTL are not lost. > > > > I know this might result in drivers/pci/pcie/aspm.c disagreeing what > > the state of the ASPM is (as shown under sysfs) compared with LNKCTL > > value but the cause can no longer be due racing RMW. Essentially, 4) is > > seen as an override to what core did if it changed ASPMC in between. > > Technically, something is still "lost" like you say but for a different > > reason than this series is trying to fix. > > > > > Straw-man proposal: > > > > > > - Change pci_disable_link_state() so it ignores aspm_disabled and > > > always disables ASPM even if platform firmware hasn't granted > > > ownership. Maybe this should warn and taint the kernel. > > > > > > - Change drivers to use pci_disable_link_state() instead of writing > > > LNKCTL directly. > > Now that I took a deeper look into what pci_disable_link_state() and > pci_enable_link_state() do, I realized they're not really disable/enable > pair like I had assumed from their names. Disable adds to ->aspm_disable > and flags are never removed from that because enable does not touch > aspm_disable at all but has it's own flag variable. This asymmetry looks > intentional. Yes, that's an annoying feature. There's only one caller of pci_enable_link_state(), so it may be possible to make this more symmetric. > So if ath drivers would do pci_disable_link_state() to realize 1)-3), > there is no way to undo it in 4). It looks as if ath drivers would > actually want to use pci_enable_link_state() with different state > parameters to realize what they want to do in 1)-4). Yeah, that does sound like a problem. I don't have any great ideas. Bjorn
diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c index a7f44f6335fb..9275a672f90c 100644 --- a/drivers/net/wireless/ath/ath10k/pci.c +++ b/drivers/net/wireless/ath/ath10k/pci.c @@ -1963,8 +1963,9 @@ static int ath10k_pci_hif_start(struct ath10k *ar) ath10k_pci_irq_enable(ar); ath10k_pci_rx_post(ar); - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, - ar_pci->link_ctl); + pcie_capability_clear_and_set_word(ar_pci->pdev, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, + ar_pci->link_ctl & PCI_EXP_LNKCTL_ASPMC); return 0; } @@ -2821,8 +2822,8 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar, pcie_capability_read_word(ar_pci->pdev, PCI_EXP_LNKCTL, &ar_pci->link_ctl); - pcie_capability_write_word(ar_pci->pdev, PCI_EXP_LNKCTL, - ar_pci->link_ctl & ~PCI_EXP_LNKCTL_ASPMC); + pcie_capability_clear_word(ar_pci->pdev, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC); /* * Bring the target up cleanly.