[v5,2/2] PCI: Don't assume root ports are power manageable

  Using a USB keyboard or mouse to wakeup the system from s2idle fails when
that xHCI device is connected to a USB-C port for an AMD USB4 router.

Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
all PCIe ports go into D3 during s2idle.

When specific root ports are put into D3 over s2idle on some AMD platforms
it is not possible for the platform to properly identify wakeup sources.
This happens whether the root port goes into D3hot or D3cold.

Comparing registers between Linux and Windows 11 this behavior to put
these specific root ports into D3 at suspend is unique to Linux. On an
affected system Windows does not put those specific root ports into D3
over Modern Standby.

Windows avoids putting Root Ports that are not power manageable (e.g do
not have platform firmware support) into low power states.

Linux shouldn't assume root ports support D3 just because they're on a
machine newer than 2015, the ports should also be deemed power manageable.
Add an extra check explicitly for root ports to ensure D3 isn't selected
for them if they are not power-manageable through platform firmware.

Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
Reported-by: Iain Lane <iain@orangesquash.org.uk>
Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
Acked-by: Rafael J. Wysocki <rafael@kernel.org>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
v4->v5:
 * Add tags
 * Fix title
 * Adjust commit message
v3->v4:
 * Move after refactor
---
 drivers/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)
  

Hi Mario,

The patch itself looks fine, but since I don't have all the power
management details in my head, it would help me a lot to make the
description more concrete.

On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
> Using a USB keyboard or mouse to wakeup the system from s2idle fails when
> that xHCI device is connected to a USB-C port for an AMD USB4 router.

It sounds like the real issue is that "Root Ports in D3hot/D3cold may
not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
are probably not really relevant.  And hopefully even the "AMD
platforms" mentioned below is not relevant.

> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> all PCIe ports go into D3 during s2idle.
> 
> When specific root ports are put into D3 over s2idle on some AMD platforms
> it is not possible for the platform to properly identify wakeup sources.
> This happens whether the root port goes into D3hot or D3cold.

Can we connect this to a spec so it's not just the empirical "some AMD
platforms work like X" observation?

"s2idle" is meaningful on the power management side of the house, but
it doesn't appear in PCI or ACPI specs, so I don't know what it means
here.  I assume the D3hot/D3cold state of the Root Port is the
critical factor, regardless of how it got there.

> Comparing registers between Linux and Windows 11 this behavior to put
> these specific root ports into D3 at suspend is unique to Linux. On an
> affected system Windows does not put those specific root ports into D3
> over Modern Standby.
> 
> Windows avoids putting Root Ports that are not power manageable (e.g do
> not have platform firmware support) into low power states.

The Windows behavior was probably useful to you in debugging, but I
don't really care about these Windows details because I don't think
they help us maintain this in the future.

> Linux shouldn't assume root ports support D3 just because they're on a
> machine newer than 2015, the ports should also be deemed power manageable.
> Add an extra check explicitly for root ports to ensure D3 isn't selected
> for them if they are not power-manageable through platform firmware.

But I *would* like to know specifically what "power manageable" means
here.  I might naively assume that a device with the PCI Power
Management Capability is "power manageable", and that if PME_Support
includes D3hot and D3cold, we're good to go.  But obviously it's more
complicated than that, and I'd like to cite the spec that mentions the
actual things we need here.

> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> Acked-by: Rafael J. Wysocki <rafael@kernel.org>
> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
> v4->v5:
>  * Add tags
>  * Fix title
>  * Adjust commit message
> v3->v4:
>  * Move after refactor
> ---
>  drivers/pci/pci.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index d1fa040bcea7..d293db963327 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3015,6 +3015,14 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>  	if (dmi_check_system(bridge_d3_blacklist))
>  		return false;
>  
> +	/*
> +	 * It's not safe to put root ports that don't support power
> +	 * management into D3.

I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
not be able to signal PME interrupts unless ... <mumble> platform
firmware <mumble> e.g., ACPI method <mumble> ..."

Can we include some of those hints here?

> +	 */
> +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
> +	    !platform_pci_power_manageable(bridge))
> +		return false;
> +
>  	/*
>  	 * It should be safe to put PCIe ports from 2015 or newer
>  	 * to D3.
> -- 
> 2.34.1
>
  

On 6/2/2023 5:20 PM, Bjorn Helgaas wrote:
> Hi Mario,
>
> The patch itself looks fine, but since I don't have all the power
> management details in my head, it would help me a lot to make the
> description more concrete.
OK, please let me know if after reviewing my responses you
would prefer me to take an attempt at rewriting the commit
message or if you can handle changing it.
>
> On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
>> Using a USB keyboard or mouse to wakeup the system from s2idle fails when
>> that xHCI device is connected to a USB-C port for an AMD USB4 router.
> It sounds like the real issue is that "Root Ports in D3hot/D3cold may
> not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
> are probably not really relevant.  And hopefully even the "AMD
> platforms" mentioned below is not relevant.
Yeah.  It comes down to how much you want in the commit
about how we got to this conclusion versus a generic
fix.  I generally like to be verbose about a specific case
something fixes so that when distros decide what to pull
in to their older maintenance kernels they can understand
what's important.
>> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> all PCIe ports go into D3 during s2idle.
>>
>> When specific root ports are put into D3 over s2idle on some AMD platforms
>> it is not possible for the platform to properly identify wakeup sources.
>> This happens whether the root port goes into D3hot or D3cold.
> Can we connect this to a spec so it's not just the empirical "some AMD
> platforms work like X" observation?
>
> "s2idle" is meaningful on the power management side of the house, but
> it doesn't appear in PCI or ACPI specs, so I don't know what it means
> here.  I assume the D3hot/D3cold state of the Root Port is the
> critical factor, regardless of how it got there.

Unfortunately (?) for this particular issue it's only a
critical factor when the system is in s2idle.

PME works fine to wake up the device if the root port is
in either D3hot or D3cold when the system isn't in s2idle.

>
>> Comparing registers between Linux and Windows 11 this behavior to put
>> these specific root ports into D3 at suspend is unique to Linux. On an
>> affected system Windows does not put those specific root ports into D3
>> over Modern Standby.
>>
>> Windows avoids putting Root Ports that are not power manageable (e.g do
>> not have platform firmware support) into low power states.
> The Windows behavior was probably useful to you in debugging, but I
> don't really care about these Windows details because I don't think
> they help us maintain this in the future.
OK.
>> Linux shouldn't assume root ports support D3 just because they're on a
>> machine newer than 2015, the ports should also be deemed power manageable.
>> Add an extra check explicitly for root ports to ensure D3 isn't selected
>> for them if they are not power-manageable through platform firmware.
> But I *would* like to know specifically what "power manageable" means
> here.  I might naively assume that a device with the PCI Power
> Management Capability is "power manageable", and that if PME_Support
> includes D3hot and D3cold, we're good to go.  But obviously it's more
> complicated than that, and I'd like to cite the spec that mentions the
> actual things we need here.
Power manageable through platform firmware means the device
has ACPI methods like like _PR0, _PS0.
>> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>> Reported-by: Iain Lane <iain@orangesquash.org.uk>
>> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
>> Acked-by: Rafael J. Wysocki <rafael@kernel.org>
>> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>> v4->v5:
>>   * Add tags
>>   * Fix title
>>   * Adjust commit message
>> v3->v4:
>>   * Move after refactor
>> ---
>>   drivers/pci/pci.c | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index d1fa040bcea7..d293db963327 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -3015,6 +3015,14 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>>   	if (dmi_check_system(bridge_d3_blacklist))
>>   		return false;
>>   
>> +	/*
>> +	 * It's not safe to put root ports that don't support power
>> +	 * management into D3.
> I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
> not be able to signal PME interrupts unless ... <mumble> platform
> firmware <mumble> e.g., ACPI method <mumble> ..."
>
> Can we include some of those hints here?

I'm cautious about hardcoding logic used by
acpi_bus_get_power_flags() in a comment in case it changes.

How about:

"Root ports in D3 may not be able to reliably signal wakeup
events unless platform firmware signals power management
capabilities".

>
>> +	 */
>> +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
>> +	    !platform_pci_power_manageable(bridge))
>> +		return false;
>> +
>>   	/*
>>   	 * It should be safe to put PCIe ports from 2015 or newer
>>   	 * to D3.
>> -- 
>> 2.34.1
>>
  

On Fri, Jun 02, 2023 at 05:38:37PM -0500, Limonciello, Mario wrote:
> On 6/2/2023 5:20 PM, Bjorn Helgaas wrote:
> > On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
> > > Using a USB keyboard or mouse to wakeup the system from s2idle fails when
> > > that xHCI device is connected to a USB-C port for an AMD USB4 router.
> >
> > It sounds like the real issue is that "Root Ports in D3hot/D3cold may
> > not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
> > are probably not really relevant.  And hopefully even the "AMD
> > platforms" mentioned below is not relevant.
>
> Yeah.  It comes down to how much you want in the commit
> about how we got to this conclusion versus a generic
> fix.  I generally like to be verbose about a specific case
> something fixes so that when distros decide what to pull
> in to their older maintenance kernels they can understand
> what's important.

That's actually my point.  I think this problem probably affects
non-USB devices, non-xHCI devices, non-USB4 routers, etc.

If we can say "Any device below a Root Port in D3hot/D3cold may not
support wakeup if X, Y, Z.  Root Ports may be put in D3hot/D3cold for
sleep/hibernate/s2idle/...",  that's much more actionable.

> > > Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> > > all PCIe ports go into D3 during s2idle.
> > > 
> > > When specific root ports are put into D3 over s2idle on some AMD platforms
> > > it is not possible for the platform to properly identify wakeup sources.
> > > This happens whether the root port goes into D3hot or D3cold.
> >
> > Can we connect this to a spec so it's not just the empirical "some AMD
> > platforms work like X" observation?
> > 
> > "s2idle" is meaningful on the power management side of the house, but
> > it doesn't appear in PCI or ACPI specs, so I don't know what it means
> > here.  I assume the D3hot/D3cold state of the Root Port is the
> > critical factor, regardless of how it got there.
> 
> Unfortunately (?) for this particular issue it's only a
> critical factor when the system is in s2idle.
> 
> PME works fine to wake up the device if the root port is
> in either D3hot or D3cold when the system isn't in s2idle.

So that must mean something other than the Root Port has to be in some
specific state.  "System in s2idle" is not actionable in terms of PCI
maintenance.  It sounds like we just haven't really gotten to the root
cause yet.

> > > Linux shouldn't assume root ports support D3 just because they're on a
> > > machine newer than 2015, the ports should also be deemed power manageable.
> > > Add an extra check explicitly for root ports to ensure D3 isn't selected
> > > for them if they are not power-manageable through platform firmware.
> >
> > But I *would* like to know specifically what "power manageable" means
> > here.  I might naively assume that a device with the PCI Power
> > Management Capability is "power manageable", and that if PME_Support
> > includes D3hot and D3cold, we're good to go.  But obviously it's more
> > complicated than that, and I'd like to cite the spec that mentions the
> > actual things we need here.
>
> Power manageable through platform firmware means the device
> has ACPI methods like like _PR0, _PS0.

What's the connection to wakeup?

> > > +	 * It's not safe to put root ports that don't support power
> > > +	 * management into D3.
> >
> > I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
> > not be able to signal PME interrupts unless ... <mumble> platform
> > firmware <mumble> e.g., ACPI method <mumble> ..."
> > 
> > Can we include some of those hints here?
> 
> I'm cautious about hardcoding logic used by
> acpi_bus_get_power_flags() in a comment in case it changes.
> 
> How about:
> 
> "Root ports in D3 may not be able to reliably signal wakeup
> events unless platform firmware signals power management
> capabilities".

I'm really looking hard for that spec citation :)  Without that, this
just devolves into "this seems to work on these systems."

> > > +	 */
> > > +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
> > > +	    !platform_pci_power_manageable(bridge))
> > > +		return false;
> > > +
> > >   	/*
> > >   	 * It should be safe to put PCIe ports from 2015 or newer
> > >   	 * to D3.
> > > -- 
> > > 2.34.1
> > >
  

On 6/2/2023 6:19 PM, Bjorn Helgaas wrote:
> On Fri, Jun 02, 2023 at 05:38:37PM -0500, Limonciello, Mario wrote:
>> On 6/2/2023 5:20 PM, Bjorn Helgaas wrote:
>>> On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
>>>> Using a USB keyboard or mouse to wakeup the system from s2idle fails when
>>>> that xHCI device is connected to a USB-C port for an AMD USB4 router.
>>> It sounds like the real issue is that "Root Ports in D3hot/D3cold may
>>> not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
>>> are probably not really relevant.  And hopefully even the "AMD
>>> platforms" mentioned below is not relevant.
>> Yeah.  It comes down to how much you want in the commit
>> about how we got to this conclusion versus a generic
>> fix.  I generally like to be verbose about a specific case
>> something fixes so that when distros decide what to pull
>> in to their older maintenance kernels they can understand
>> what's important.
> That's actually my point.  I think this problem probably affects
> non-USB devices, non-xHCI devices, non-USB4 routers, etc.
>
> If we can say "Any device below a Root Port in D3hot/D3cold may not
> support wakeup if X, Y, Z.  Root Ports may be put in D3hot/D3cold for
> sleep/hibernate/s2idle/...",  that's much more actionable.
Completely agree.
>
>>>> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>>>> all PCIe ports go into D3 during s2idle.
>>>>
>>>> When specific root ports are put into D3 over s2idle on some AMD platforms
>>>> it is not possible for the platform to properly identify wakeup sources.
>>>> This happens whether the root port goes into D3hot or D3cold.
>>> Can we connect this to a spec so it's not just the empirical "some AMD
>>> platforms work like X" observation?
>>>
>>> "s2idle" is meaningful on the power management side of the house, but
>>> it doesn't appear in PCI or ACPI specs, so I don't know what it means
>>> here.  I assume the D3hot/D3cold state of the Root Port is the
>>> critical factor, regardless of how it got there.
>> Unfortunately (?) for this particular issue it's only a
>> critical factor when the system is in s2idle.
>>
>> PME works fine to wake up the device if the root port is
>> in either D3hot or D3cold when the system isn't in s2idle.
> So that must mean something other than the Root Port has to be in some
> specific state.  "System in s2idle" is not actionable in terms of PCI
> maintenance.  It sounds like we just haven't really gotten to the root
> cause yet.
The root cause of this behavior is deep in the platform
firmware.  This is why the platform firmware doesn't
advertise power management support for the root port.
>
>>>> Linux shouldn't assume root ports support D3 just because they're on a
>>>> machine newer than 2015, the ports should also be deemed power manageable.
>>>> Add an extra check explicitly for root ports to ensure D3 isn't selected
>>>> for them if they are not power-manageable through platform firmware.
>>> But I *would* like to know specifically what "power manageable" means
>>> here.  I might naively assume that a device with the PCI Power
>>> Management Capability is "power manageable", and that if PME_Support
>>> includes D3hot and D3cold, we're good to go.  But obviously it's more
>>> complicated than that, and I'd like to cite the spec that mentions the
>>> actual things we need here.
>> Power manageable through platform firmware means the device
>> has ACPI methods like like _PR0, _PS0.
> What's the connection to wakeup?
There is also no _PRW (power resources for wake) for this
root port.
>
>>>> +	 * It's not safe to put root ports that don't support power
>>>> +	 * management into D3.
>>> I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
>>> not be able to signal PME interrupts unless ... <mumble> platform
>>> firmware <mumble> e.g., ACPI method <mumble> ..."
>>>
>>> Can we include some of those hints here?
>> I'm cautious about hardcoding logic used by
>> acpi_bus_get_power_flags() in a comment in case it changes.
>>
>> How about:
>>
>> "Root ports in D3 may not be able to reliably signal wakeup
>> events unless platform firmware signals power management
>> capabilities".
> I'm really looking hard for that spec citation :)  Without that, this
> just devolves into "this seems to work on these systems."
I mean that's exactly what the broken logic below this
fix is....

But I think I can get you what you're looking for.

 From this failing system the problematic root port is GP19.

The DSDT has:

Device (GP19) {
     Method (_DSM,..)
     Method (_PRT,..)
     Device (NHI0)
     Device (NHI1)
}

The SSDT has:

Scope (\_SB.PCI0.GP19) {
Method (YS0W,..)
Method (YPRW,..)
Method (RPRM,..)
Method (WPRM,..)
Method (SPDP,..)
Method (SPCH,..)
Method (_STA,..)
Method (_INI,..)
Method (_REG,..)
}

Section 7.3 from the ACPI spec [1] says

"For a device that is power-managed using ACPI, a Definition Block 
contains one or more of the objects found in the table below. Power 
management of a device is done using Power Resource control"

The GP19 device has NONE of the objects mentioned in the table.

Outside of this change, I do think this means acpi_bus_get_power_flags() 
may want to also look for some of the other objects besides _PS0 and 
_PR0 to resolve that a device is power manageable though.

[1] 
https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/device-power-management-objects.html?highlight=pr0#device-power-management-objects

>>>> +	 */
>>>> +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
>>>> +	    !platform_pci_power_manageable(bridge))
>>>> +		return false;
>>>> +
>>>>    	/*
>>>>    	 * It should be safe to put PCIe ports from 2015 or newer
>>>>    	 * to D3.
>>>> -- 
>>>> 2.34.1
>>>>
  

On Sat, Jun 3, 2023 at 12:38 AM Limonciello, Mario
<mario.limonciello@amd.com> wrote:
>
>
> On 6/2/2023 5:20 PM, Bjorn Helgaas wrote:
> > Hi Mario,
> >
> > The patch itself looks fine, but since I don't have all the power
> > management details in my head, it would help me a lot to make the
> > description more concrete.
> OK, please let me know if after reviewing my responses you
> would prefer me to take an attempt at rewriting the commit
> message or if you can handle changing it.
> >
> > On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
> >> Using a USB keyboard or mouse to wakeup the system from s2idle fails when
> >> that xHCI device is connected to a USB-C port for an AMD USB4 router.
> > It sounds like the real issue is that "Root Ports in D3hot/D3cold may
> > not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
> > are probably not really relevant.  And hopefully even the "AMD
> > platforms" mentioned below is not relevant.
> Yeah.  It comes down to how much you want in the commit
> about how we got to this conclusion versus a generic
> fix.  I generally like to be verbose about a specific case
> something fixes so that when distros decide what to pull
> in to their older maintenance kernels they can understand
> what's important.
> >> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> >> all PCIe ports go into D3 during s2idle.
> >>
> >> When specific root ports are put into D3 over s2idle on some AMD platforms
> >> it is not possible for the platform to properly identify wakeup sources.
> >> This happens whether the root port goes into D3hot or D3cold.
> > Can we connect this to a spec so it's not just the empirical "some AMD
> > platforms work like X" observation?
> >
> > "s2idle" is meaningful on the power management side of the house, but
> > it doesn't appear in PCI or ACPI specs, so I don't know what it means
> > here.  I assume the D3hot/D3cold state of the Root Port is the
> > critical factor, regardless of how it got there.
>
> Unfortunately (?) for this particular issue it's only a
> critical factor when the system is in s2idle.
>
> PME works fine to wake up the device if the root port is
> in either D3hot or D3cold when the system isn't in s2idle.

Why doesn't it work fine when the system is in s2idle then?

Getting to the root of this would be really helpful here IMO.

> >
> >> Comparing registers between Linux and Windows 11 this behavior to put
> >> these specific root ports into D3 at suspend is unique to Linux. On an
> >> affected system Windows does not put those specific root ports into D3
> >> over Modern Standby.
> >>
> >> Windows avoids putting Root Ports that are not power manageable (e.g do
> >> not have platform firmware support) into low power states.
> > The Windows behavior was probably useful to you in debugging, but I
> > don't really care about these Windows details because I don't think
> > they help us maintain this in the future.
> OK.
> >> Linux shouldn't assume root ports support D3 just because they're on a
> >> machine newer than 2015, the ports should also be deemed power manageable.
> >> Add an extra check explicitly for root ports to ensure D3 isn't selected
> >> for them if they are not power-manageable through platform firmware.
> > But I *would* like to know specifically what "power manageable" means
> > here.  I might naively assume that a device with the PCI Power
> > Management Capability is "power manageable", and that if PME_Support
> > includes D3hot and D3cold, we're good to go.  But obviously it's more
> > complicated than that, and I'd like to cite the spec that mentions the
> > actual things we need here.
> Power manageable through platform firmware means the device
> has ACPI methods like like _PR0, _PS0.
> >> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
> >> Reported-by: Iain Lane <iain@orangesquash.org.uk>
> >> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
> >> Acked-by: Rafael J. Wysocki <rafael@kernel.org>
> >> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> >> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> >> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> >> ---
> >> v4->v5:
> >>   * Add tags
> >>   * Fix title
> >>   * Adjust commit message
> >> v3->v4:
> >>   * Move after refactor
> >> ---
> >>   drivers/pci/pci.c | 8 ++++++++
> >>   1 file changed, 8 insertions(+)
> >>
> >> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> >> index d1fa040bcea7..d293db963327 100644
> >> --- a/drivers/pci/pci.c
> >> +++ b/drivers/pci/pci.c
> >> @@ -3015,6 +3015,14 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
> >>      if (dmi_check_system(bridge_d3_blacklist))
> >>              return false;
> >>
> >> +    /*
> >> +     * It's not safe to put root ports that don't support power
> >> +     * management into D3.
> > I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
> > not be able to signal PME interrupts unless ... <mumble> platform
> > firmware <mumble> e.g., ACPI method <mumble> ..."
> >
> > Can we include some of those hints here?
>
> I'm cautious about hardcoding logic used by
> acpi_bus_get_power_flags() in a comment in case it changes.
>
> How about:
>
> "Root ports in D3 may not be able to reliably signal wakeup
> events unless platform firmware signals power management
> capabilities".

I would rather write "unless then can be power-managed with the help
of the platform firmware".

The meaning of "signaling" is unclear in this context and even if it
was clear, the platform firmware support actually needs to be used
here, its mere existence is not sufficient AFAICS.
  

On 6/4/2023 6:40 AM, Rafael J. Wysocki wrote:
> On Sat, Jun 3, 2023 at 12:38 AM Limonciello, Mario
> <mario.limonciello@amd.com> wrote:
>>
>> On 6/2/2023 5:20 PM, Bjorn Helgaas wrote:
>>> Hi Mario,
>>>
>>> The patch itself looks fine, but since I don't have all the power
>>> management details in my head, it would help me a lot to make the
>>> description more concrete.
>> OK, please let me know if after reviewing my responses you
>> would prefer me to take an attempt at rewriting the commit
>> message or if you can handle changing it.
>>> On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
>>>> Using a USB keyboard or mouse to wakeup the system from s2idle fails when
>>>> that xHCI device is connected to a USB-C port for an AMD USB4 router.
>>> It sounds like the real issue is that "Root Ports in D3hot/D3cold may
>>> not support wakeup", and the USB, xHCI, USB-C, AMD USB4 router bits
>>> are probably not really relevant.  And hopefully even the "AMD
>>> platforms" mentioned below is not relevant.
>> Yeah.  It comes down to how much you want in the commit
>> about how we got to this conclusion versus a generic
>> fix.  I generally like to be verbose about a specific case
>> something fixes so that when distros decide what to pull
>> in to their older maintenance kernels they can understand
>> what's important.
>>>> Due to commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>>>> all PCIe ports go into D3 during s2idle.
>>>>
>>>> When specific root ports are put into D3 over s2idle on some AMD platforms
>>>> it is not possible for the platform to properly identify wakeup sources.
>>>> This happens whether the root port goes into D3hot or D3cold.
>>> Can we connect this to a spec so it's not just the empirical "some AMD
>>> platforms work like X" observation?
>>>
>>> "s2idle" is meaningful on the power management side of the house, but
>>> it doesn't appear in PCI or ACPI specs, so I don't know what it means
>>> here.  I assume the D3hot/D3cold state of the Root Port is the
>>> critical factor, regardless of how it got there.
>> Unfortunately (?) for this particular issue it's only a
>> critical factor when the system is in s2idle.
>>
>> PME works fine to wake up the device if the root port is
>> in either D3hot or D3cold when the system isn't in s2idle.
> Why doesn't it work fine when the system is in s2idle then?
>
> Getting to the root of this would be really helpful here IMO.
The process of the hardware going into s2idle has a certain
sequence of events by the platform.

This sequence is what causes the PME to not be able to work
during resume.  This issue has been root caused and is
understood by AMD platform designers.

It's why the AML doesn't provide any of those ACPI power
management routines outlined in the ACPI spec.

If the AML is patched to advertise these routines the exact
same issue is reproduced under Windows 11.

>>>> Comparing registers between Linux and Windows 11 this behavior to put
>>>> these specific root ports into D3 at suspend is unique to Linux. On an
>>>> affected system Windows does not put those specific root ports into D3
>>>> over Modern Standby.
>>>>
>>>> Windows avoids putting Root Ports that are not power manageable (e.g do
>>>> not have platform firmware support) into low power states.
>>> The Windows behavior was probably useful to you in debugging, but I
>>> don't really care about these Windows details because I don't think
>>> they help us maintain this in the future.
>> OK.
>>>> Linux shouldn't assume root ports support D3 just because they're on a
>>>> machine newer than 2015, the ports should also be deemed power manageable.
>>>> Add an extra check explicitly for root ports to ensure D3 isn't selected
>>>> for them if they are not power-manageable through platform firmware.
>>> But I *would* like to know specifically what "power manageable" means
>>> here.  I might naively assume that a device with the PCI Power
>>> Management Capability is "power manageable", and that if PME_Support
>>> includes D3hot and D3cold, we're good to go.  But obviously it's more
>>> complicated than that, and I'd like to cite the spec that mentions the
>>> actual things we need here.
>> Power manageable through platform firmware means the device
>> has ACPI methods like like _PR0, _PS0.
>>>> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend")
>>>> Reported-by: Iain Lane <iain@orangesquash.org.uk>
>>>> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121
>>>> Acked-by: Rafael J. Wysocki <rafael@kernel.org>
>>>> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>>>> Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>>>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>> v4->v5:
>>>>    * Add tags
>>>>    * Fix title
>>>>    * Adjust commit message
>>>> v3->v4:
>>>>    * Move after refactor
>>>> ---
>>>>    drivers/pci/pci.c | 8 ++++++++
>>>>    1 file changed, 8 insertions(+)
>>>>
>>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>>> index d1fa040bcea7..d293db963327 100644
>>>> --- a/drivers/pci/pci.c
>>>> +++ b/drivers/pci/pci.c
>>>> @@ -3015,6 +3015,14 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
>>>>       if (dmi_check_system(bridge_d3_blacklist))
>>>>               return false;
>>>>
>>>> +    /*
>>>> +     * It's not safe to put root ports that don't support power
>>>> +     * management into D3.
>>> I assume "it's not safe" really means "Root Ports in D3hot/D3cold may
>>> not be able to signal PME interrupts unless ... <mumble> platform
>>> firmware <mumble> e.g., ACPI method <mumble> ..."
>>>
>>> Can we include some of those hints here?
>> I'm cautious about hardcoding logic used by
>> acpi_bus_get_power_flags() in a comment in case it changes.
>>
>> How about:
>>
>> "Root ports in D3 may not be able to reliably signal wakeup
>> events unless platform firmware signals power management
>> capabilities".
> I would rather write "unless then can be power-managed with the help
> of the platform firmware".
>
> The meaning of "signaling" is unclear in this context and even if it
> was clear, the platform firmware support actually needs to be used
> here, its mere existence is not sufficient AFAICS.
OK thanks!
  

On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
> +	/*
> +	 * It's not safe to put root ports that don't support power
> +	 * management into D3.
> +	 */
> +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
> +	    !platform_pci_power_manageable(bridge))
> +		return false;
> +
>  	/*
>  	 * It should be safe to put PCIe ports from 2015 or newer
>  	 * to D3.

My recollection is that we began suspending Root Ports to D3hot because
otherwise low power states of the whole CPU package could not be reached
on certain Intel CPUs from the 2015+ era.

Do we know if the DSDT of all those systems contains the required ACPI
objects to continue runtime suspending their Root Ports after this change?
Otherwise these systems would experience a power regression.

Thanks,

Lukas
  

On 6/7/2023 3:01 AM, Lukas Wunner wrote:
> On Tue, May 30, 2023 at 11:39:47AM -0500, Mario Limonciello wrote:
>> +	/*
>> +	 * It's not safe to put root ports that don't support power
>> +	 * management into D3.
>> +	 */
>> +	if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT &&
>> +	    !platform_pci_power_manageable(bridge))
>> +		return false;
>> +
>>   	/*
>>   	 * It should be safe to put PCIe ports from 2015 or newer
>>   	 * to D3.
> My recollection is that we began suspending Root Ports to D3hot because
> otherwise low power states of the whole CPU package could not be reached
> on certain Intel CPUs from the 2015+ era.
>
> Do we know if the DSDT of all those systems contains the required ACPI
> objects to continue runtime suspending their Root Ports after this change?
> Otherwise these systems would experience a power regression.
>
> Thanks,
>
> Lukas
You might not have been CC'ed on my earlier patches, but
I was worried about a similar problem and at least in one
of the earlier versions was adjusting the existing
heuristic of ">= 2015" to "Intel & >= 2015".

That being said, this most recent version got R-b and A-b
tags from 3 Intel guys, I would think they cross
referenced reference systems to make that assertion before
adding tags.