[net-next,v4,00/23] net: add EEE support for KSZ9477 and AR8035 with i.MX6

  changes v4:
- remove following helpers:
  mmd_eee_cap_to_ethtool_sup_t
  mmd_eee_adv_to_ethtool_adv_t
  ethtool_adv_to_mmd_eee_adv_t
  and port drivers from this helpers to linkmode helpers.
- rebase against latest net-next
- port phy_init_eee() to genphy_c45_eee_is_active()

changes v3:
- rework some parts of EEE infrastructure and move it to c45 code.
- add supported_eee storage and start using it in EEE code and by the
  micrel driver.
- add EEE support for ar8035 PHY
- add SmartEEE support to FEC i.MX series.

changes v2:
- use phydev->supported instead of reading MII_BMSR regiaster
- fix @get_eee > @set_eee

With this patch series we provide EEE control for KSZ9477 family of switches and
AR8035 with i.MX6 configuration.
According to my tests, on a system with KSZ8563 switch and 100Mbit idle link,
we consume 0,192W less power per port if EEE is enabled.

Oleksij Rempel (23):
  net: dsa: microchip: enable EEE support
  net: phy: add genphy_c45_read_eee_abilities() function
  net: phy: micrel: add ksz9477_get_features()
  net: phy: export phy_check_valid() function
  net: phy: add genphy_c45_ethtool_get/set_eee() support
  net: phy: c22: migrate to genphy_c45_write_eee_adv()
  net: phy: c45: migrate to genphy_c45_write_eee_adv()
  net: phy: migrate phy_init_eee() to genphy_c45_eee_is_active()
  net: phy: start using genphy_c45_ethtool_get/set_eee()
  net: phy: add driver specific get/set_eee support
  net: phy: at803x: implement ethtool access to SmartEEE functionality
  net: phy: at803x: ar8035: fix EEE support for half duplex links
  net: phy: add PHY specifica flag to signal SmartEEE support
  net: phy: at803x: add PHY_SMART_EEE flag to AR8035
  net: phy: add phy_has_smarteee() helper
  net: fec: add support for PHYs with SmartEEE support
  e1000e: replace EEE ethtool helpers to linkmode variants
  igb: replace EEE ethtool helpers to linkmode variants
  igc: replace EEE ethtool helpers to linkmode variants
  tg3: replace EEE ethtool helpers to linkmode variants
  r8152: replace EEE ethtool helpers to linkmode variants
  net: usb: ax88179_178a: replace EEE ethtool helpers to linkmode
    variants
  net: mdio: drop EEE ethtool helpers in favor to linkmode variants

 drivers/net/dsa/microchip/ksz_common.c       |  66 ++++
 drivers/net/ethernet/broadcom/tg3.c          |   9 +-
 drivers/net/ethernet/freescale/fec_main.c    |  22 +-
 drivers/net/ethernet/intel/e1000e/ethtool.c  |  16 +-
 drivers/net/ethernet/intel/igb/igb_ethtool.c |  23 +-
 drivers/net/ethernet/intel/igc/igc_ethtool.c |  12 +-
 drivers/net/phy/at803x.c                     | 142 ++++++++-
 drivers/net/phy/micrel.c                     |  21 ++
 drivers/net/phy/phy-c45.c                    | 298 ++++++++++++++++++-
 drivers/net/phy/phy.c                        | 155 ++--------
 drivers/net/phy/phy_device.c                 |  26 +-
 drivers/net/usb/ax88179_178a.c               |  24 +-
 drivers/net/usb/r8152.c                      |  34 ++-
 include/linux/mdio.h                         | 136 ++++-----
 include/linux/phy.h                          |  28 ++
 include/uapi/linux/mdio.h                    |   8 +
 16 files changed, 760 insertions(+), 260 deletions(-)
  

On Wed, Feb 01, 2023 at 03:58:22PM +0100, Oleksij Rempel wrote:
> With this patch series we provide EEE control for KSZ9477 family of switches and
> AR8035 with i.MX6 configuration.
> According to my tests, on a system with KSZ8563 switch and 100Mbit idle link,
> we consume 0,192W less power per port if EEE is enabled.

What is the code flow through the kernel with EEE? I wasn't able to find
a good explanation about it.

Is it advertised by default, if supported? I guess phy_advertise_supported()
does that.

But is that desirable? Doesn't EEE cause undesired latency for MAC-level
PTP timestamping on an otherwise idle link?
  

On Sat, Feb 04, 2023 at 02:13:32AM +0200, Vladimir Oltean wrote:
> On Wed, Feb 01, 2023 at 03:58:22PM +0100, Oleksij Rempel wrote:
> > With this patch series we provide EEE control for KSZ9477 family of switches and
> > AR8035 with i.MX6 configuration.
> > According to my tests, on a system with KSZ8563 switch and 100Mbit idle link,
> > we consume 0,192W less power per port if EEE is enabled.
> 
> What is the code flow through the kernel with EEE? I wasn't able to find
> a good explanation about it.
> 
> Is it advertised by default, if supported? I guess phy_advertise_supported()
> does that.

Ack.

> But is that desirable? Doesn't EEE cause undesired latency for MAC-level
> PTP timestamping on an otherwise idle link?

Theoretically, MAC controls Low Power Idle states and even with some
"Wake" latency should be fully aware of actual ingress and egress time
stamps.

Practically, right now I do not have such HW to confirm it. My project
is affected by EEE in different ways:
- with EEE PTP has too much jitter
- without EEE, the devices consumes too much power in standby mode with
  WoL enabled. Even switching to 10BaseT less power as 100BaseTX with
  EEE would do.

My view is probably biased by my environment - PTP is relatively rare
use case. EEE saves power (0,2W+0,2W per link in my case). Summary power
saving of all devices is potentially equal to X amount of power plants. 
So, EEE should be enabled by default.

Beside, flow control (enabled by default) affects PTP in some cases too.

May be ptp4l should warn about this options? We should be able to detect
it from user space.

Regards,
Oleksij
  

On Mon, Feb 06, 2023 at 06:47:13AM +0100, Oleksij Rempel wrote:
> On Sat, Feb 04, 2023 at 02:13:32AM +0200, Vladimir Oltean wrote:
> > On Wed, Feb 01, 2023 at 03:58:22PM +0100, Oleksij Rempel wrote:
> > > With this patch series we provide EEE control for KSZ9477 family of switches and
> > > AR8035 with i.MX6 configuration.
> > > According to my tests, on a system with KSZ8563 switch and 100Mbit idle link,
> > > we consume 0,192W less power per port if EEE is enabled.
> > 
> > What is the code flow through the kernel with EEE? I wasn't able to find
> > a good explanation about it.
> > 
> > Is it advertised by default, if supported? I guess phy_advertise_supported()
> > does that.
> 
> Ack.
> 
> > But is that desirable? Doesn't EEE cause undesired latency for MAC-level
> > PTP timestamping on an otherwise idle link?
> 
> Theoretically, MAC controls Low Power Idle states and even with some
> "Wake" latency should be fully aware of actual ingress and egress time
> stamps.

I'm not sure if this is also true with Atheros SmartEEE, where the PHY
is the one who enters LPI mode and buffers packets until it wakes up?

> 
> Practically, right now I do not have such HW to confirm it. My project
> is affected by EEE in different ways:

Doesn't FEC support PTP?

> - with EEE PTP has too much jitter
> - without EEE, the devices consumes too much power in standby mode with
>   WoL enabled. Even switching to 10BaseT less power as 100BaseTX with
>   EEE would do.
> 
> My view is probably biased by my environment - PTP is relatively rare
> use case. EEE saves power (0,2W+0,2W per link in my case). Summary power
> saving of all devices is potentially equal to X amount of power plants. 
> So, EEE should be enabled by default.

I'm not contesting the value of EEE. Just wondering whether it's best
for the kernel, rather than user space, to enable it by default.

> 
> Beside, flow control (enabled by default) affects PTP in some cases too.

You are probably talking about the fact that flow control may affect
end-to-end delay measurements (across switches in a LAN). Yes, but EEE
(or at least SmartEEE) may affect peer-to-peer delay measurements, which
I see as worse.

> 
> May be ptp4l should warn about this options? We should be able to detect
> it from user space.

This isn't necessarily a bad idea, even though it would end up
renegotiating and losing the link.

Maybe it would be good to drag Richard Cochran into the discussion too.
After all he's the one who should agree what should and what shouldn't
ptp4l be concerned with.

> 
> Regards,
> Oleksij
> -- 
> Pengutronix e.K.                           |                             |
> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |
  

> > > What is the code flow through the kernel with EEE? I wasn't able to find
> > > a good explanation about it.
> > > 
> > > Is it advertised by default, if supported? I guess phy_advertise_supported()
> > > does that.

The old flow is poorly defined. If the MAC supports EEE, it should
call phy_init_eee(). That looks at the results of auto-neg and returns
if EEE has been negotiated or not.

However, i'm not aware of any code which disables by default the
advertisement of EEE, or actually enables the negotiation of EEE. So
there are probably a number of PHYs which are EEE capable, connected
to a MAC driver which does not call phy_init_eee() and are advertising
EEE and negotiating EEE. There might also be a subset of that which
are actually doing EEE, despite not calling phy_init_eee().

So the current code is not good, and there is a danger we introduce
power regressions as we sort this out.

The current MAC/PHY API is pretty broken. We probably should be
handling this similar to pause. A MAC which supports pause should call
phy_support_asym_pause() or phy_support_sym_pause() which will cause
the PHY to advertise its supported Pause modes. So we might want to
add a phy_support_eee()? We then want the result of EEE negotiation
available in phydev for when the link_adjust() callback is called.

A quick look at a few MAC drivers seems to indicate many are getting
it wrong and don't actually wait for the result of the auto-neg....

   Andrew
  

On Mon, Feb 06, 2023 at 04:10:38PM +0200, Vladimir Oltean wrote:
> On Mon, Feb 06, 2023 at 06:47:13AM +0100, Oleksij Rempel wrote:
> > On Sat, Feb 04, 2023 at 02:13:32AM +0200, Vladimir Oltean wrote:
> > > On Wed, Feb 01, 2023 at 03:58:22PM +0100, Oleksij Rempel wrote:
> > > > With this patch series we provide EEE control for KSZ9477 family of switches and
> > > > AR8035 with i.MX6 configuration.
> > > > According to my tests, on a system with KSZ8563 switch and 100Mbit idle link,
> > > > we consume 0,192W less power per port if EEE is enabled.
> > > 
> > > What is the code flow through the kernel with EEE? I wasn't able to find
> > > a good explanation about it.
> > > 
> > > Is it advertised by default, if supported? I guess phy_advertise_supported()
> > > does that.
> > 
> > Ack.
> > 
> > > But is that desirable? Doesn't EEE cause undesired latency for MAC-level
> > > PTP timestamping on an otherwise idle link?
> > 
> > Theoretically, MAC controls Low Power Idle states and even with some
> > "Wake" latency should be fully aware of actual ingress and egress time
> > stamps.
> 
> I'm not sure if this is also true with Atheros SmartEEE, where the PHY
> is the one who enters LPI mode and buffers packets until it wakes up?

Yes, you right. With SmartEEE without MAC assistance, PTP will be
broken. At the same time, if MAC is PTP and EEE capable, the same PHY
with SmartEEE disabled should work just fine.

> > Practically, right now I do not have such HW to confirm it. My project
> > is affected by EEE in different ways:
> 
> Doesn't FEC support PTP?

FEC do supports PTP, but do not support EEE on i.MX6/7 variants.

> > - with EEE PTP has too much jitter
> > - without EEE, the devices consumes too much power in standby mode with
> >   WoL enabled. Even switching to 10BaseT less power as 100BaseTX with
> >   EEE would do.
> > 
> > My view is probably biased by my environment - PTP is relatively rare
> > use case. EEE saves power (0,2W+0,2W per link in my case). Summary power
> > saving of all devices is potentially equal to X amount of power plants. 
> > So, EEE should be enabled by default.
> 
> I'm not contesting the value of EEE. Just wondering whether it's best
> for the kernel, rather than user space, to enable it by default.

I woulds say, at the end the switch will decide what functionality will
be advertised. Other nodes should just tell what capabilities they
support.

> > 
> > Beside, flow control (enabled by default) affects PTP in some cases too.
> 
> You are probably talking about the fact that flow control may affect
> end-to-end delay measurements (across switches in a LAN). Yes, but EEE
> (or at least SmartEEE) may affect peer-to-peer delay measurements, which
> I see as worse.

I agree. User space should be notified some how about SmartEEE
functionality. Especially if it is incompatible with some other
functionality like PTP. It took me some time to understand why my PTP sync was
so unstable. SmartEEE was just silently enabled by HW and no EEE related
information was provided to user space.

> > May be ptp4l should warn about this options? We should be able to detect
> > it from user space.
> 
> This isn't necessarily a bad idea, even though it would end up
> renegotiating and losing the link.

My idea was to inform the user, not actively do what ever is needed. It
can conflict with other services or make system administrator scratch the
head without understanding why things magically happen.

> Maybe it would be good to drag Richard Cochran into the discussion too.
> After all he's the one who should agree what should and what shouldn't
> ptp4l be concerned with.

ACK.

Regards,
Oleksij
  

On Mon, Feb 06, 2023 at 04:39:52PM +0100, Andrew Lunn wrote:
> > > > What is the code flow through the kernel with EEE? I wasn't able to find
> > > > a good explanation about it.
> > > > 
> > > > Is it advertised by default, if supported? I guess phy_advertise_supported()
> > > > does that.
> 
> The old flow is poorly defined. If the MAC supports EEE, it should
> call phy_init_eee(). That looks at the results of auto-neg and returns
> if EEE has been negotiated or not.
> 
> However, i'm not aware of any code which disables by default the
> advertisement of EEE, or actually enables the negotiation of EEE. So
> there are probably a number of PHYs which are EEE capable, connected
> to a MAC driver which does not call phy_init_eee() and are advertising
> EEE and negotiating EEE. There might also be a subset of that which
> are actually doing EEE, despite not calling phy_init_eee().
> 
> So the current code is not good, and there is a danger we introduce
> power regressions as we sort this out.
> 
> The current MAC/PHY API is pretty broken. We probably should be
> handling this similar to pause. A MAC which supports pause should call
> phy_support_asym_pause() or phy_support_sym_pause() which will cause
> the PHY to advertise its supported Pause modes. So we might want to
> add a phy_support_eee()? We then want the result of EEE negotiation
> available in phydev for when the link_adjust() callback is called.

Good point.

SmartEEE will be probably a bit more challenging. If MAC do not
advertise EEE support, SmartEEE can be enabled. But it would break PTP
if SmartEEE is active. Except SmartEEE capable PHY implements own PTP
support. In any case, user space will need extra information to
identify potential issues.

> A quick look at a few MAC drivers seems to indicate many are getting
> it wrong and don't actually wait for the result of the auto-neg....

Some ethernet driver trying to do own EEE state detection, and doing
false positive detection on not supported states - for example half
duplex.
  

> SmartEEE will be probably a bit more challenging. If MAC do not
> advertise EEE support, SmartEEE can be enabled. But it would break PTP
> if SmartEEE is active. Except SmartEEE capable PHY implements own PTP
> support. In any case, user space will need extra information to
> identify potential issues.

If we have a MAC driver which does not implement the ethtool set_eee()
and get_eee() ops, and a dev->phydev with the SmartEEE flag set, we
could have net/ethtool/eee.c call direct into phylib.

As for PTP and EEE, maybe we want the core PTP code to try calling
get_eee() and at minimum issue a warning if it is enabled, if it
thinks MAC PTP is being used?

       Andrew