Message ID | 20240129112710.2852-2-ilpo.jarvinen@linux.intel.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp652325dyb; Mon, 29 Jan 2024 07:45:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IF84WZoQi7t/dEYplAfZX1HosHj3/HMGsmtPnAdJbJ0sM3U6Qgm40PZZCd7X/g4ID82gxtp X-Received: by 2002:a05:6358:7e99:b0:178:7ecb:1573 with SMTP id o25-20020a0563587e9900b001787ecb1573mr618104rwn.15.1706543110940; Mon, 29 Jan 2024 07:45:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706543110; cv=pass; d=google.com; s=arc-20160816; b=lyvG/f1CtRuxCT1B8YQpD93+JDyg4JPKRchEvRzP4Lr/OwQtD4EDiKHeCeqMrjC/fu U4hVf2rxfIbRk/Pc1FNOCmTSzP5TUbhfLelL/GPaDun3MZDzGgO+veLtcexTg08ktwps pXyUN0xeh4u4xEp2GdqiRJIgpgdmbaPrNQFIoPv4RT+9lm3a4IFdE0Al+jyN2WVn9YFu e228N0uJj0r5GeWpoys5FNYC14DMIYkGW/lMbhEEbrmDLyYuZopIAFMcwhSgImf6c38r eNPlSonyBPYtCN7QaFTvVuAi0ogN+74eIKSENOyox5TD2Z4i8zEofNm1OGx7qAN291TV YcMw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=foDMV4q9ZIZEhENjpLaQwkh5ScnxOQaSK8aMiOMbWFg=; fh=uaDhNdmOckUBDf+XjXUykn5C4NpzoLZK8jGmWlP+92w=; b=Z/1pohm+cWQjLxyk7gxVi+C9BPiQEzWD+2gVcXIOgVW0wQ1Y21ztw6Y4iCKMbgoL0V hSHqVWKlUt0iVxdSC/5RfSFNnah3CBhgkNew19fs1/uHKLbkBgPYo2N5hm6/w8sx2qzS oseAkqqxgVBvsvRy+isvXzgKM35A1pCyfDYMwfO6Cmnr/6Vrjbbc+VUOhhfBC76ADt1X K7kiQtBIT7eq2zhxY9q8QeltnU9iAlfbPQClcIDl8Ppyt0UrET4TDL0yUN+yUYCWglul pa7M6GsARQyhIbiPvMBGNxRvqkhC/lzdu8qPdkK8/ZR0TTeS5jkEwuGgB17I8NHsSScn T0ag== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HvXzSg6W; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id l196-20020a633ecd000000b005897813624fsi5863025pga.476.2024.01.29.07.45.10 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 07:45:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HvXzSg6W; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42614-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 0C894B239B4 for <ouuuleilei@gmail.com>; Mon, 29 Jan 2024 11:29:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E02425FEE5; Mon, 29 Jan 2024 11:28:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="HvXzSg6W" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 794C65BAD2; Mon, 29 Jan 2024 11:28:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706527702; cv=none; b=O9A///ks9hxO2aRlNSiEafpWG7tkdsI4j9wwwmj9p5nN01QqwDBETrjqnoDvEK7grDn1Pb9Lz8nrIZYncKVWlZvqAqIC8VJxCqAH15fuHeKdBTeenSaHguSp88OnY0ocPetGISyr/inyQYmeNfmGOYtr9DHlqAS6lP1FdQH94LU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706527702; c=relaxed/simple; bh=vj5fBWYj/huccK5UtSPMJRYI869/VeBvWXPwIF4xpbo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=lYErmWoRvuLQ8b7KgiSsxfsaihWwHQixHN8MnToJ0ka6MiCU7bjQ0WCzDXRUdFRjWGA9VVhTVou/tVTuhDF9jlbfv7JSPGJV40K3WTv9UZ02J2TmckNf5cjiq4TbfwyMpTTHInwYyMxfQxapaZclqyQU2lxhyBsPpUn8QbER07U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=HvXzSg6W; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706527700; x=1738063700; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vj5fBWYj/huccK5UtSPMJRYI869/VeBvWXPwIF4xpbo=; b=HvXzSg6We+qdRerhxyjxwDP6mDeU1GO8EmlP4GocUIKNavC/a+ciyVCT XtrBz54t12k1P3yo2Lol8vGN+r5HtopeA/m3mWJm9fQutvSoRZ6XF8atb +cWwvlK5ZaqvRmHO7BYVXZmOsu4ZQ3+havEmVJNC1kR+1BdFcwAtqr5aP tKq1bX87yiA6fU8o6mukVPS5GhhbXAy1lc04nxsbLp21ZK+jLquhTN88D TyrOocQ3NZXeHSgBciMh8BwVe2LT/90Wvb2U8rx/TUA5NTaa0EbHmVhtA trvfDzJ1UK0RMxSzzju2X5boxUUM/HzdVDkm2l6UBe4HQaApd1NcBHZFn g==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="1857085" X-IronPort-AV: E=Sophos;i="6.05,227,1701158400"; d="scan'208";a="1857085" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2024 03:28:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="858070352" X-IronPort-AV: E=Sophos;i="6.05,227,1701158400"; d="scan'208";a="858070352" Received: from ijarvine-desk1.ger.corp.intel.com (HELO localhost) ([10.94.253.213]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2024 03:28:13 -0800 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com> To: Bjorn Helgaas <bhelgaas@google.com>, linux-pci@vger.kernel.org, "Maciej W. Rozycki" <macro@orcam.me.uk>, linux-kernel@vger.kernel.org Cc: Mika Westerberg <mika.westerberg@linux.intel.com>, =?utf-8?q?Ilpo_J?= =?utf-8?q?=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com> Subject: [PATCH 1/2] PCI: Clear LBMS on resume to avoid Target Speed quirk Date: Mon, 29 Jan 2024 13:27:09 +0200 Message-Id: <20240129112710.2852-2-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240129112710.2852-1-ilpo.jarvinen@linux.intel.com> References: <20240129112710.2852-1-ilpo.jarvinen@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: <linux-kernel.vger.kernel.org> List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org> List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789440149273494623 X-GMAIL-MSGID: 1789440149273494623 |
Series |
PCI: Fix disconnect related issues
|
|
Commit Message
Ilpo Järvinen
Jan. 29, 2024, 11:27 a.m. UTC
While a device is runtime suspended along with its PCIe hierarchy, the device could get disconnected. Because of the suspend, the device disconnection cannot be detected until portdrv/hotplug have resumed. On runtime resume, pcie_wait_for_link_delay() is called: pci_pm_runtime_resume() pci_pm_bridge_power_up_actions() pci_bridge_wait_for_secondary_bus() pcie_wait_for_link_delay() Because the device is already disconnected, this results in cascading failures: 1. pcie_wait_for_link_status() returns -ETIMEDOUT. 2. After the commit a89c82249c37 ("PCI: Work around PCIe link training failures"), pcie_failed_link_retrain() spuriously detects this failure as a Link Retraining failure and attempts the Target Speed trick, which also fails. 3. pci_bridge_wait_for_secondary_bus() then calls pci_dev_wait() which cannot succeed (but waits ~1 minute, delaying the resume). The Target Speed trick (in step 2) is only used if LBMS bit (PCIe r6.1 sec 7.5.3.8) is set. For links that have been operational before suspend, it is well possible that LBMS has been set at the bridge and remains on. Thus, after resume, LBMS does not indicate the link needs the Target Speed quirk. Clear LBMS on resume for bridges to avoid the issue. Fixes: a89c82249c37 ("PCI: Work around PCIe link training failures") Reported-by: Mika Westerberg <mika.westerberg@linux.intel.com> Tested-by: Mika Westerberg <mika.westerberg@linux.intel.com> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> --- drivers/pci/pci-driver.c | 6 ++++++ 1 file changed, 6 insertions(+)
Comments
On Mon, Jan 29, 2024 at 01:27:09PM +0200, Ilpo Järvinen wrote: > While a device is runtime suspended along with its PCIe hierarchy, the > device could get disconnected. Because of the suspend, the device > disconnection cannot be detected until portdrv/hotplug have resumed. On > runtime resume, pcie_wait_for_link_delay() is called: > > pci_pm_runtime_resume() > pci_pm_bridge_power_up_actions() > pci_bridge_wait_for_secondary_bus() > pcie_wait_for_link_delay() > > Because the device is already disconnected, this results in cascading > failures: > > 1. pcie_wait_for_link_status() returns -ETIMEDOUT. > > 2. After the commit a89c82249c37 ("PCI: Work around PCIe link > training failures"), I this this also depends on the merge resolution in 1abb47390350 ("Merge branch 'pci/enumeration'"). Just looking at a89c82249c37 in isolation suggests that pcie_wait_for_link_status() returning -ETIMEDOUT would not cause pcie_wait_for_link_delay() to call pcie_failed_link_retrain(). > pcie_failed_link_retrain() spuriously detects > this failure as a Link Retraining failure and attempts the Target > Speed trick, which also fails. Based on the comment below, I guess "Target Speed trick" probably refers to the "retrain at 2.5GT/s, then remove the speed restriction and retrain again" part of pcie_failed_link_retrain() (which I guess is basically the entire point of the function)? > 3. pci_bridge_wait_for_secondary_bus() then calls pci_dev_wait() which > cannot succeed (but waits ~1 minute, delaying the resume). > > The Target Speed trick (in step 2) is only used if LBMS bit (PCIe r6.1 > sec 7.5.3.8) is set. For links that have been operational before > suspend, it is well possible that LBMS has been set at the bridge and > remains on. Thus, after resume, LBMS does not indicate the link needs > the Target Speed quirk. Clear LBMS on resume for bridges to avoid the > issue. > > Fixes: a89c82249c37 ("PCI: Work around PCIe link training failures") > Reported-by: Mika Westerberg <mika.westerberg@linux.intel.com> > Tested-by: Mika Westerberg <mika.westerberg@linux.intel.com> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > --- > drivers/pci/pci-driver.c | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c > index 51ec9e7e784f..05a114962df3 100644 > --- a/drivers/pci/pci-driver.c > +++ b/drivers/pci/pci-driver.c > @@ -574,6 +574,12 @@ static void pci_pm_bridge_power_up_actions(struct pci_dev *pci_dev) > { > int ret; > > + /* > + * Clear LBMS on resume to avoid spuriously triggering Target Speed > + * quirk in pcie_failed_link_retrain(). > + */ > + pcie_capability_write_word(pci_dev, PCI_EXP_LNKSTA, PCI_EXP_LNKSTA_LBMS); > + > ret = pci_bridge_wait_for_secondary_bus(pci_dev, "resume"); > if (ret) { > /* > -- > 2.39.2 >
On Mon, 29 Jan 2024, Bjorn Helgaas wrote: > On Mon, Jan 29, 2024 at 01:27:09PM +0200, Ilpo Järvinen wrote: > > While a device is runtime suspended along with its PCIe hierarchy, the > > device could get disconnected. Because of the suspend, the device > > disconnection cannot be detected until portdrv/hotplug have resumed. On > > runtime resume, pcie_wait_for_link_delay() is called: > > > > pci_pm_runtime_resume() > > pci_pm_bridge_power_up_actions() > > pci_bridge_wait_for_secondary_bus() > > pcie_wait_for_link_delay() > > > > Because the device is already disconnected, this results in cascading > > failures: > > > > 1. pcie_wait_for_link_status() returns -ETIMEDOUT. > > > > 2. After the commit a89c82249c37 ("PCI: Work around PCIe link > > training failures"), > > I this this also depends on the merge resolution in 1abb47390350 > ("Merge branch 'pci/enumeration'"). Just looking at a89c82249c37 in > isolation suggests that pcie_wait_for_link_status() returning > -ETIMEDOUT would not cause pcie_wait_for_link_delay() to call > pcie_failed_link_retrain(). I was aware of the merge but I seem to have somehow misanalyzed the return values earlier since I cannot anymore reach my earlier conclusion and now ended up agreeing with your analysis that 1abb47390350 broke it. That would imply there is a logic error in 1abb47390350 in addition to the LBMS-logic problem in a89c82249c37 my patch is fixing... However, I cannot pinpoint a single error because there seems to be more than one in the whole code. First of all, this is not true for pcie_failed_link_retrain(): * Return TRUE if the link has been successfully retrained, otherwise FALSE. If LBMS is not set, the Target Speed quirk is not applied but the function still returns true. I think that should be changed to early return false when no LBMS is present. But if I make that change, then pcie_wait_for_link_delay() will do msleep() + return true, and pci_bridge_wait_for_secondary_bus() will call long ~60s pci_dev_wait(). I'll try to come up another patch to cleanup all that return logic so that it actually starts to make some sense. > > pcie_failed_link_retrain() spuriously detects > > this failure as a Link Retraining failure and attempts the Target > > Speed trick, which also fails. > > Based on the comment below, I guess "Target Speed trick" probably > refers to the "retrain at 2.5GT/s, then remove the speed restriction > and retrain again" part of pcie_failed_link_retrain() (which I guess > is basically the entire point of the function)? Yes. I'll change the wording slightly to make it more obvious and put (Target Speed quirk) into parenthesis so I can use it below. > > 3. pci_bridge_wait_for_secondary_bus() then calls pci_dev_wait() which > > cannot succeed (but waits ~1 minute, delaying the resume). > > > > The Target Speed trick (in step 2) is only used if LBMS bit (PCIe r6.1 > > sec 7.5.3.8) is set. For links that have been operational before > > suspend, it is well possible that LBMS has been set at the bridge and > > remains on. Thus, after resume, LBMS does not indicate the link needs > > the Target Speed quirk. Clear LBMS on resume for bridges to avoid the > > issue.
On Fri, 2 Feb 2024, Ilpo Järvinen wrote: > On Thu, 1 Feb 2024, Maciej W. Rozycki wrote: > > On Thu, 1 Feb 2024, Ilpo Järvinen wrote: > > > > > > > What kind of scenario does the LBMS bit get set in that you have a > > > > > trouble with? You write that in your case the downstream device has been > > > > > disconnected while the bug hierarchy was suspended and it is not present > > > > > anymore at resume, is that correct? > > > > > > > > > > But in that case no link negotiation could have been possible and > > > > > consequently the LBMS bit mustn't have been set by hardware (according to > > > > > my understanding of PCIe), because (for compliant, non-broken devices > > > > > anyway) it is only specified to be set for ports that can communicate with > > > > > the other link end (the spec explicitly says there mustn't have been a > > > > > transition through the DL_Down status for the port). > > > > > > > > > > Am I missing something? > > > > > > > > Yes, when resuming the device is already gone but the bridge still has > > > > LBMS set. My understanding is that it was set because it was there > > > > from pre-suspend time but I've not really taken a deep look into it > > > > because the problem and fix seemed obvious. > > > > I've always been confused with the suspend/resume terminology: I'd have > > assumed this would have gone through a power cycle, in which case the LBMS > > bit would have necessarily been cleared in the transition, because its > > required state at power-up/reset is 0, so the value of 1 observed would be > > a result of what has happened solely through the resume stage. Otherwise > > it may make sense to clear the bit in the course of the suspend stage, > > though it wouldn't be race-free I'm afraid. > > I also thought suspend as one possibility but yes, it racy. Mika also > suggested clearing LBMS after each successful retrain but that wouldn't > cover all possible ways to get LBMS set as devices can set it > independently of OS. Keeping it cleared constantly is pretty much what > will happen with the bandwidth controller anyway. > > > > > I read that "without the Port transitioning through DL_Down status" > > > > differently than you, I only interpret that it relates to the two > > > > bullets following it. ...So if retrain bit is set, and link then goes > > > > down, the bullet no longer applies and LBMS should not be set because > > > > there was transition through DL_Down. But I could well be wrong... > > > > What I refer to is that if you suspend your system, remove the device > > that originally caused the quirk to trigger and then resume your system > > with the device absent, > > A small correction here, the quirk didn't trigger initially for the > device, it does that only after resume. And even then quirk is called > only because the link doesn't come up. > > On longer term, I'd actually want to have hotplug resumed earlier so the > disconnect could be detected before/while all this waiting related to link > up is done. But that's very complicated to realize in practice because > hotplug lurks behind portdrv so resuming it earlier isn't going to be > about just moving a few lines around. > > > then LBMS couldn't have been set in the course of > > resume, because the port couldn't have come out of the DL_Down status in > > the absence of the downstream device. Do you interpret it differently? > > That's a good question and I don't have an answer to this yet, that is, > I don't fully understand what happens to those device during runtime > suspend/resume cycle and what is the exact mechanism that preserves the > LBMS bit, I'll look more into it. > > But I agree that if the device goes cold enough and downstream is > disconnected, the port should no longer have a way to reassert LBMS. It seems that the root cause here is that the bridge ports do not enter D3cold but remain in D3hot because of an ACPI power resource that is shared (with Thunderbolt in this case but that's just one example, there could be other similar sharings outside of what PCI controls). There is an attempt to power off the entire hierarchy into D3cold on suspend but the ports won't reach that state. Because the port remains in D3hot, the config space is preserved and LBMS bit along with it. So it seems that we cannot make the assumption that a device enters D3cold just because it was suspended. On the positive side, it was also tested that the logic fix I sent earlier solved the most visible problem which is the delay on resume. It doesn't address the false activation of Target Speed quirk because LBMS bit is still set but the symptoms are no longer the same because of the corrected logic. So to solve the main issue with delay, there's no need to clear the LBMS and the patch you're preparing/testing for pcie_failed_link_retrain() together with the logic fix on its caller are enough to address the first issue. I still think those two should be put into the same commit because the true <-> false changes, if made separately, lead to additional incoherent states but if Bjorn wants them separately, at least they should be put back-to-back so the brokeness is just within a single commit in the history. In addition, my 2nd patch addresses another issue which is unrelated to this despite similar symptoms with extra delay on resume. I'll send v2 of that 2nd path separately with the inverted return value. > > > Because if it is constantly picking another speed, it would mean you get > > > LBMS set over and over again, no? If that happens 34-35 times per second, > > > it should be set already again when we get into that quirk because there > > > was some wait before it gets called. > > > > I'll see if I can experiment with the hardware over the next couple of > > days and come back with my findings. > > Okay thanks. One point I'd be very interested to know if the link actually comes up successfully (even if briefly) because this has large implications whether the quirk can actually be invoked from the bandwidth controller code.
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 51ec9e7e784f..05a114962df3 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -574,6 +574,12 @@ static void pci_pm_bridge_power_up_actions(struct pci_dev *pci_dev) { int ret; + /* + * Clear LBMS on resume to avoid spuriously triggering Target Speed + * quirk in pcie_failed_link_retrain(). + */ + pcie_capability_write_word(pci_dev, PCI_EXP_LNKSTA, PCI_EXP_LNKSTA_LBMS); + ret = pci_bridge_wait_for_secondary_bus(pci_dev, "resume"); if (ret) { /*