[v5,3/6] net: phy: at803x: add QCA8084 ethernet phy support

  Add qca8084 PHY support, which is four-port PHY with maximum
link capability 2.5G, the features of each port is almost same
as QCA8081 and slave seed config is not needed.

Three kind of interface modes supported by qca8084.
PHY_INTERFACE_MODE_10G_QXGMII, PHY_INTERFACE_MODE_2500BASEX and
PHY_INTERFACE_MODE_SGMII.

The PCS(serdes) and clock are also needed to be configured to
bringup qca8084 PHY, which will be added in the pcs driver.

The additional CDT configurations used for qca8084.

Signed-off-by: Luo Jie <quic_luoj@quicinc.com>
---
 drivers/net/phy/at803x.c | 48 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
  

On Sat, Nov 18, 2023 at 02:27:51PM +0800, Luo Jie wrote:
> Add qca8084 PHY support, which is four-port PHY with maximum
> link capability 2.5G, the features of each port is almost same
> as QCA8081 and slave seed config is not needed.
> 
> Three kind of interface modes supported by qca8084.
> PHY_INTERFACE_MODE_10G_QXGMII, PHY_INTERFACE_MODE_2500BASEX and
> PHY_INTERFACE_MODE_SGMII.

Sorry for joining the conversation late.

I'm trying to get my head around QXGMII. Let me describe what i think
is happening, and then you can correct me....

You have 4 MACs, probably in a switch. The MII interfaces from these
MACs go into a multiplexer, and out comes QXGMII? You then have a
SERDES interface out of the switch and into the PHY package. Inside
the PHY package there is a demultiplexor, giving you four MII
interfaces, one to each PHY in the package.

If you have the PHY SERDES running in 2500BaseX, you have a single
MAC, no mux/demux, and only one PHY is used? The other three are idle.
Same from SGMII?

So the interface mode QXGMII is a property of the package. It is not
really a property of one PHY. Having one PHY using QXGMII and another
SGMII does not work?

     Andrew
  

On Sat, Nov 18, 2023 at 04:51:42PM +0100, Andrew Lunn wrote:
> On Sat, Nov 18, 2023 at 02:27:51PM +0800, Luo Jie wrote:
> > Add qca8084 PHY support, which is four-port PHY with maximum
> > link capability 2.5G, the features of each port is almost same
> > as QCA8081 and slave seed config is not needed.
> > 
> > Three kind of interface modes supported by qca8084.
> > PHY_INTERFACE_MODE_10G_QXGMII, PHY_INTERFACE_MODE_2500BASEX and
> > PHY_INTERFACE_MODE_SGMII.
> 
> Sorry for joining the conversation late.
> 
> I'm trying to get my head around QXGMII. Let me describe what i think
> is happening, and then you can correct me....
> 
> You have 4 MACs, probably in a switch. The MII interfaces from these
> MACs go into a multiplexer, and out comes QXGMII? You then have a
> SERDES interface out of the switch and into the PHY package. Inside
> the PHY package there is a demultiplexor, giving you four MII
> interfaces, one to each PHY in the package.
> 
> If you have the PHY SERDES running in 2500BaseX, you have a single
> MAC, no mux/demux, and only one PHY is used? The other three are idle.
> Same from SGMII?
> 
> So the interface mode QXGMII is a property of the package. It is not
> really a property of one PHY. Having one PHY using QXGMII and another
> SGMII does not work?

10G_QXGMII is defined in the Cisco USXGMII multi-port document as one
of several possibilities for a USXGMII-M link. The Cisco document can
be a little confusing beause it states that 10G_QXGMII supports 10M,
100M, 1G and 2.5G, and then only talks about a 10G and 100M/1G MAC.

For 10G_QXGMII, there are 4 MAC interfaces. These are connected to a
rate "adaption" through symbol replication block, and then on to a
clause 49 PCS block.

There is then a port MUX and framing block, followed by the PMA
serdes which communicates with the remote end over a single pair of
transmit/receive serdes lines.

Each interface also has its own clause 37 autoneg block.

So, for an interface to operate in SGMII mode, it would have to be
muxed to a different path before being presented to the USXGMII-M
block since each interface does not have its own external data lane
- thus that's out of scope of USXGMII-M as documented by Cisco.

Hope this helps.
  

> 10G_QXGMII is defined in the Cisco USXGMII multi-port document as one
> of several possibilities for a USXGMII-M link. The Cisco document can
> be a little confusing beause it states that 10G_QXGMII supports 10M,
> 100M, 1G and 2.5G, and then only talks about a 10G and 100M/1G MAC.
> 
> For 10G_QXGMII, there are 4 MAC interfaces. These are connected to a
> rate "adaption" through symbol replication block, and then on to a
> clause 49 PCS block.
> 
> There is then a port MUX and framing block, followed by the PMA
> serdes which communicates with the remote end over a single pair of
> transmit/receive serdes lines.
> 
> Each interface also has its own clause 37 autoneg block.
> 
> So, for an interface to operate in SGMII mode, it would have to be
> muxed to a different path before being presented to the USXGMII-M
> block since each interface does not have its own external data lane
> - thus that's out of scope of USXGMII-M as documented by Cisco.

Hi Russell

I think it helps.

Where i'm having trouble is deciding if this is actually an interface
mode. Interface mode is a per PHY property. Where as it seems
10G_QXGMII is a property of the USXGMII-M link? Should we be
representing the package with 4 PHYs in it, and specify the package
has a PMA which is using 10G_QXGMII over USXGMII-M? The PHY interface
mode is then internal? Its just the link between the PHY and the MUX?

By saying the interface mode is 10G_QXGMII and not describing the PMA
mode, are we setting ourselves up for problems in the future? Could
there be a PMA interface which could carry different PHY interface
modes?

If we decide we do want to use 10G_QXGMII as an interface made, i
think the driver should be doing some validation. If asked to do
anything else, it should return -EINVAL.

And i don't yet understand how it can also do 1000BaseX and 2500BaseX
and SGMII?

    Andrew
  

On Sat, 18 Nov 2023 14:27:51 +0800 Luo Jie wrote:
> +		/* There are two PCSs available for QCA8084, which support the following
> +		 * interface modes.
> +		 *
> +		 * 1. PHY_INTERFACE_MODE_10G_QXGMII utilizes PCS1 for all available 4 ports,
> +		 * which is for all link speeds.
> +		 *
> +		 * 2. PHY_INTERFACE_MODE_2500BASEX utilizes PCS0 for the fourth port,
> +		 * which is only for the link speed 2500M same as QCA8081.
> +		 *
> +		 * 3. PHY_INTERFACE_MODE_SGMII utilizes PCS0 for the fourth port,
> +		 * which is for the link speed 10M, 100M and 1000M same as QCA8081.

Wrap the comments please. 

Please use checkpatch with --max-line-length=80
  

On 11/19/2023 4:19 AM, Andrew Lunn wrote:
>> 10G_QXGMII is defined in the Cisco USXGMII multi-port document as one
>> of several possibilities for a USXGMII-M link. The Cisco document can
>> be a little confusing beause it states that 10G_QXGMII supports 10M,
>> 100M, 1G and 2.5G, and then only talks about a 10G and 100M/1G MAC.
>>
>> For 10G_QXGMII, there are 4 MAC interfaces. These are connected to a
>> rate "adaption" through symbol replication block, and then on to a
>> clause 49 PCS block.
>>
>> There is then a port MUX and framing block, followed by the PMA
>> serdes which communicates with the remote end over a single pair of
>> transmit/receive serdes lines.
>>
>> Each interface also has its own clause 37 autoneg block.
>>
>> So, for an interface to operate in SGMII mode, it would have to be
>> muxed to a different path before being presented to the USXGMII-M
>> block since each interface does not have its own external data lane
>> - thus that's out of scope of USXGMII-M as documented by Cisco.
> 
> Hi Russell
> 
> I think it helps.
> 
> Where i'm having trouble is deciding if this is actually an interface
> mode. Interface mode is a per PHY property. Where as it seems
> 10G_QXGMII is a property of the USXGMII-M link? Should we be
> representing the package with 4 PHYs in it, and specify the package
> has a PMA which is using 10G_QXGMII over USXGMII-M? The PHY interface
> mode is then internal? Its just the link between the PHY and the MUX?
> 
> By saying the interface mode is 10G_QXGMII and not describing the PMA
> mode, are we setting ourselves up for problems in the future? Could
> there be a PMA interface which could carry different PHY interface
> modes?
> 
> If we decide we do want to use 10G_QXGMII as an interface made, i
> think the driver should be doing some validation. If asked to do
> anything else, it should return -EINVAL.
> 
> And i don't yet understand how it can also do 1000BaseX and 2500BaseX
> and SGMII?
> 
>      Andrew

Hi Andrew,
The interface mode 10G_QXGMII is a type of USXGMII-M, the other modes
such as 20G-QXGMII, 20G-OXGMII...

As for the interface mode 10G-QXGMII, there is a multiplexer for 4 PHYs,
then do 66bit/68bit encode in xpcs and pass to PMA, the link topology:
quad PHY --- multiplexer ---XPCS --- PMA.
the 10G-QXGMII interface block includes multiplexer, XPCS and PMA.

when the PHY works on SGMII mode, then there is no xpcs, the only fourth
PHY of qca8084 can work on SGMII mode, the link topology:
the fourth PHY --- PCS --- PMA, the SGMII block includes PCS and PMA.

Either 10G-QXGMII or SGMII block of qca8084 connects with the interface
block(10G-QXGMII or SGMII) in MAC side.

Here is a problem as Russell mentioned earlier, we need to know which 
PHY device is changing the link status when the 10G-QXGMII mode is used,
since there are 4 PHYs, when one of them has the link change, there is 
no PHY device information passed to the PHYLINK, so the PCS driver don't
which PHY is changing link status and 10G-QXGMII mode don't know which
channel(mapped to PHY) should be configured.

would we add a field such as (int channel) in the struct phy_device?
so we can pass this information to PCS driver when the PHY link changed.

Thanks,
  

On 11/19/2023 8:23 AM, Jakub Kicinski wrote:
> On Sat, 18 Nov 2023 14:27:51 +0800 Luo Jie wrote:
>> +		/* There are two PCSs available for QCA8084, which support the following
>> +		 * interface modes.
>> +		 *
>> +		 * 1. PHY_INTERFACE_MODE_10G_QXGMII utilizes PCS1 for all available 4 ports,
>> +		 * which is for all link speeds.
>> +		 *
>> +		 * 2. PHY_INTERFACE_MODE_2500BASEX utilizes PCS0 for the fourth port,
>> +		 * which is only for the link speed 2500M same as QCA8081.
>> +		 *
>> +		 * 3. PHY_INTERFACE_MODE_SGMII utilizes PCS0 for the fourth port,
>> +		 * which is for the link speed 10M, 100M and 1000M same as QCA8081.
> 
> Wrap the comments please.
> 
> Please use checkpatch with --max-line-length=80

Thanks Jakub, i will add this option for the patch check.
  

On Mon, Nov 20, 2023 at 04:49:59PM +0800, Jie Luo wrote:
> 
> 
> On 11/19/2023 4:19 AM, Andrew Lunn wrote:
> > > 10G_QXGMII is defined in the Cisco USXGMII multi-port document as one
> > > of several possibilities for a USXGMII-M link. The Cisco document can
> > > be a little confusing beause it states that 10G_QXGMII supports 10M,
> > > 100M, 1G and 2.5G, and then only talks about a 10G and 100M/1G MAC.
> > > 
> > > For 10G_QXGMII, there are 4 MAC interfaces. These are connected to a
> > > rate "adaption" through symbol replication block, and then on to a
> > > clause 49 PCS block.
> > > 
> > > There is then a port MUX and framing block, followed by the PMA
> > > serdes which communicates with the remote end over a single pair of
> > > transmit/receive serdes lines.
> > > 
> > > Each interface also has its own clause 37 autoneg block.
> > > 
> > > So, for an interface to operate in SGMII mode, it would have to be
> > > muxed to a different path before being presented to the USXGMII-M
> > > block since each interface does not have its own external data lane
> > > - thus that's out of scope of USXGMII-M as documented by Cisco.
> > 
> > Hi Russell
> > 
> > I think it helps.
> > 
> > Where i'm having trouble is deciding if this is actually an interface
> > mode. Interface mode is a per PHY property. Where as it seems
> > 10G_QXGMII is a property of the USXGMII-M link? Should we be
> > representing the package with 4 PHYs in it, and specify the package
> > has a PMA which is using 10G_QXGMII over USXGMII-M? The PHY interface
> > mode is then internal? Its just the link between the PHY and the MUX?
> > 
> > By saying the interface mode is 10G_QXGMII and not describing the PMA
> > mode, are we setting ourselves up for problems in the future? Could
> > there be a PMA interface which could carry different PHY interface
> > modes?
> > 
> > If we decide we do want to use 10G_QXGMII as an interface made, i
> > think the driver should be doing some validation. If asked to do
> > anything else, it should return -EINVAL.
> > 
> > And i don't yet understand how it can also do 1000BaseX and 2500BaseX
> > and SGMII?
> > 
> >      Andrew
> 
> Hi Andrew,
> The interface mode 10G_QXGMII is a type of USXGMII-M, the other modes
> such as 20G-QXGMII, 20G-OXGMII...
> 
> As for the interface mode 10G-QXGMII, there is a multiplexer for 4 PHYs,
> then do 66bit/68bit encode in xpcs and pass to PMA, the link topology:
> quad PHY --- multiplexer ---XPCS --- PMA.
> the 10G-QXGMII interface block includes multiplexer, XPCS and PMA.

Note that phylink_pcs does *not* cover any PCS on the PHY device side
of the link. It only covers a PCS on the MAC side.

> Here is a problem as Russell mentioned earlier, we need to know which PHY
> device is changing the link status when the 10G-QXGMII mode is used,
> since there are 4 PHYs, when one of them has the link change, there is no
> PHY device information passed to the PHYLINK, so the PCS driver don't
> which PHY is changing link status and 10G-QXGMII mode don't know which
> channel(mapped to PHY) should be configured.
> 
> would we add a field such as (int channel) in the struct phy_device?
> so we can pass this information to PCS driver when the PHY link changed.

Nothing in phylink nor phylib is setup to deal with "channels" within
a PHY. The model assumes that a network interface consists of exactly
one MAC associated with one active PHY.

As there are 4 PHYs, phylib will expect there to be four PHY devices,
and there will be expected to be four phylink instances.
  

> Hi Andrew,
> The interface mode 10G_QXGMII is a type of USXGMII-M, the other modes
> such as 20G-QXGMII, 20G-OXGMII...
> 
> As for the interface mode 10G-QXGMII, there is a multiplexer for 4 PHYs,
> then do 66bit/68bit encode in xpcs and pass to PMA, the link topology:
> quad PHY --- multiplexer ---XPCS --- PMA.
> the 10G-QXGMII interface block includes multiplexer, XPCS and PMA.
> 
> when the PHY works on SGMII mode, then there is no xpcs, the only fourth
> PHY of qca8084 can work on SGMII mode, the link topology:
> the fourth PHY --- PCS --- PMA, the SGMII block includes PCS and PMA.

What i missed is that you have two different PMA blocks. PHY4 can be
muxed to either the QXGMII PMA or the 2500BaseX PMA. This is not clear
in the commit message, and i think why you are getting questions about
how 2500BaseX can work over QXGMII. Please expand you commit message
to explain the architecture in more detail.

So, next question. How do you control what PMA PHY4 is connected to?
Is this going to be based on interface mode? QXGMII it is configured
to use the QXGMII PMA? SGMII, 1000BaseX, and 2500BaseX it is
configured to the other PMA?

> Here is a problem as Russell mentioned earlier, we need to know which PHY
> device is changing the link status when the 10G-QXGMII mode is used,
> since there are 4 PHYs, when one of them has the link change, there is no
> PHY device information passed to the PHYLINK, so the PCS driver don't
> which PHY is changing link status and 10G-QXGMII mode don't know which
> channel(mapped to PHY) should be configured.

This is the first time QXGMII has been seen in mainline, so its good
to explain the background.

Are you saying there is a USXGMII-M level link change status? The link
between the SoC and the PHY package is up/down? If it is down, all
four MAC-PHY links are down. If it is up, it is possible to carry
frames between the SoC and the PHY package, but maybe the PHYs
themselves are down?

Withing the four multiplex streams, is there a per stream link change
indication?

	Andrew
  

On Mon, Nov 20, 2023 at 04:34:55PM +0100, Andrew Lunn wrote:
> Are you saying there is a USXGMII-M level link change status? The link
> between the SoC and the PHY package is up/down? If it is down, all
> four MAC-PHY links are down. If it is up, it is possible to carry
> frames between the SoC and the PHY package, but maybe the PHYs
> themselves are down?

It shouldn't do. Each "channel" in the USXGMII-M link has its own
autoneg block at both ends, each conveys link status independently.

The MAC side structure is:


                            +----------+                +-----+
                    .-XGMII-> Rate     |    PCS         |     |
MAC1 <-MDI-> PHY <-+        | Adaption <--> Clause 49 <->     |
                    `-GMII-->          |                |     |
                            +-----^----+                |     |
                                  |                     |     |
                            +-----v---- +               |     |
                            | Autoneg   |               |     |
                            | Clause 37 |               |     |
                            +-----------+               |     |
                                                        | Mux <--> PMA <-->
                                                        |     |
                                                        .......     USXGMII-M

<------------------------------------------------------>
      These blocks are repeated for each channel

The spec goes on to state that there must be a USXGMII enable bit that
defaults to disabled and the PHY should assume normal XGMII/XFI
operation. When enabled, autoneg follows a slight modification of
clause 37-6.

As far as the USXGMII-M link, I believe 2.7.8 in the USXGMII-M
documentation covers this, which is "hardware autoneg programming
sequence". It states that "if 10G link is lost or regained, the
software is expected to disable autoneg and re-enable autoneg". I
think "10G link" refers to the USXGMII-M connection, which means
the loss of that link shold cause software to intervene in each
of the PCS autoneg blocks. It is, however, rather unclear.
  

On 11/20/2023 5:29 PM, Russell King (Oracle) wrote:
> On Mon, Nov 20, 2023 at 04:49:59PM +0800, Jie Luo wrote:
>>
>>
>> On 11/19/2023 4:19 AM, Andrew Lunn wrote:
>>>> 10G_QXGMII is defined in the Cisco USXGMII multi-port document as one
>>>> of several possibilities for a USXGMII-M link. The Cisco document can
>>>> be a little confusing beause it states that 10G_QXGMII supports 10M,
>>>> 100M, 1G and 2.5G, and then only talks about a 10G and 100M/1G MAC.
>>>>
>>>> For 10G_QXGMII, there are 4 MAC interfaces. These are connected to a
>>>> rate "adaption" through symbol replication block, and then on to a
>>>> clause 49 PCS block.
>>>>
>>>> There is then a port MUX and framing block, followed by the PMA
>>>> serdes which communicates with the remote end over a single pair of
>>>> transmit/receive serdes lines.
>>>>
>>>> Each interface also has its own clause 37 autoneg block.
>>>>
>>>> So, for an interface to operate in SGMII mode, it would have to be
>>>> muxed to a different path before being presented to the USXGMII-M
>>>> block since each interface does not have its own external data lane
>>>> - thus that's out of scope of USXGMII-M as documented by Cisco.
>>>
>>> Hi Russell
>>>
>>> I think it helps.
>>>
>>> Where i'm having trouble is deciding if this is actually an interface
>>> mode. Interface mode is a per PHY property. Where as it seems
>>> 10G_QXGMII is a property of the USXGMII-M link? Should we be
>>> representing the package with 4 PHYs in it, and specify the package
>>> has a PMA which is using 10G_QXGMII over USXGMII-M? The PHY interface
>>> mode is then internal? Its just the link between the PHY and the MUX?
>>>
>>> By saying the interface mode is 10G_QXGMII and not describing the PMA
>>> mode, are we setting ourselves up for problems in the future? Could
>>> there be a PMA interface which could carry different PHY interface
>>> modes?
>>>
>>> If we decide we do want to use 10G_QXGMII as an interface made, i
>>> think the driver should be doing some validation. If asked to do
>>> anything else, it should return -EINVAL.
>>>
>>> And i don't yet understand how it can also do 1000BaseX and 2500BaseX
>>> and SGMII?
>>>
>>>       Andrew
>>
>> Hi Andrew,
>> The interface mode 10G_QXGMII is a type of USXGMII-M, the other modes
>> such as 20G-QXGMII, 20G-OXGMII...
>>
>> As for the interface mode 10G-QXGMII, there is a multiplexer for 4 PHYs,
>> then do 66bit/68bit encode in xpcs and pass to PMA, the link topology:
>> quad PHY --- multiplexer ---XPCS --- PMA.
>> the 10G-QXGMII interface block includes multiplexer, XPCS and PMA.
> 
> Note that phylink_pcs does *not* cover any PCS on the PHY device side
> of the link. It only covers a PCS on the MAC side.

Ok, even there is only one XPCS multiplex with 4 MACs, we should create
4 PCS instances for 4 MACs.

> 
>> Here is a problem as Russell mentioned earlier, we need to know which PHY
>> device is changing the link status when the 10G-QXGMII mode is used,
>> since there are 4 PHYs, when one of them has the link change, there is no
>> PHY device information passed to the PHYLINK, so the PCS driver don't
>> which PHY is changing link status and 10G-QXGMII mode don't know which
>> channel(mapped to PHY) should be configured.
>>
>> would we add a field such as (int channel) in the struct phy_device?
>> so we can pass this information to PCS driver when the PHY link changed.
> 
> Nothing in phylink nor phylib is setup to deal with "channels" within
> a PHY. The model assumes that a network interface consists of exactly
> one MAC associated with one active PHY.
> 
> As there are 4 PHYs, phylib will expect there to be four PHY devices,
> and there will be expected to be four phylink instances.
> 
make sense, thanks Russell.
  

On 11/20/2023 11:34 PM, Andrew Lunn wrote:
>> Hi Andrew,
>> The interface mode 10G_QXGMII is a type of USXGMII-M, the other modes
>> such as 20G-QXGMII, 20G-OXGMII...
>>
>> As for the interface mode 10G-QXGMII, there is a multiplexer for 4 PHYs,
>> then do 66bit/68bit encode in xpcs and pass to PMA, the link topology:
>> quad PHY --- multiplexer ---XPCS --- PMA.
>> the 10G-QXGMII interface block includes multiplexer, XPCS and PMA.
>>
>> when the PHY works on SGMII mode, then there is no xpcs, the only fourth
>> PHY of qca8084 can work on SGMII mode, the link topology:
>> the fourth PHY --- PCS --- PMA, the SGMII block includes PCS and PMA.
> 
> What i missed is that you have two different PMA blocks. PHY4 can be
> muxed to either the QXGMII PMA or the 2500BaseX PMA. This is not clear
> in the commit message, and i think why you are getting questions about
> how 2500BaseX can work over QXGMII. Please expand you commit message
> to explain the architecture in more detail.

Ok, Andrew, i will give more detail about the pcs utilized in the next
patch set.

when pcs is configured to QXGMII mode, quad phy can reach to maximum
speed 2.5G, it is still in qxgmii mode.

when pcs is configured to SGMII mode, the fourth PHY can reach to
maximum speed 2.5G(2500BaseT) that is reached by increasing the clock
rate to 312.5MHZ from 125MHZ of 1G speed, but there is no corresponding
interface mode can be used to reflect this 2.5G speed mode(sgmii+), so
i use 2500BaseX to reflect this 2.5G link speed, which is same as
qca8081 PHY. Actually we should add a new interface mode such as sgmii+
to reflect this 2.5G speed of sgmii, so that the PHYLINK can support
all supported link speeds(10/100/1000/2500M) with the interface mode
sgmii+, currently 2500BaseX only advertise 2.5G, sgmii advertise
10/100/1000 in the PHYLINK framework. but actually qca808x supports
10/100/1000/2500 speed. how do you think?

> 
> So, next question. How do you control what PMA PHY4 is connected to?
> Is this going to be based on interface mode? QXGMII it is configured
> to use the QXGMII PMA? SGMII, 1000BaseX, and 2500BaseX it is
> configured to the other PMA?

Yes, there are two different PCS to work on qxgmii and sgmii mode.
one PCS can be connected with quad phy, another pcs can be connected
with PHY4.

> 
>> Here is a problem as Russell mentioned earlier, we need to know which PHY
>> device is changing the link status when the 10G-QXGMII mode is used,
>> since there are 4 PHYs, when one of them has the link change, there is no
>> PHY device information passed to the PHYLINK, so the PCS driver don't
>> which PHY is changing link status and 10G-QXGMII mode don't know which
>> channel(mapped to PHY) should be configured.
> 
> This is the first time QXGMII has been seen in mainline, so its good
> to explain the background.

 From PCS hardware perspective, there is only one QXGMII mode instance
for 4 PHYs and MACs, and the each PHY is related with the channel of
QXGMII.

> 
> Are you saying there is a USXGMII-M level link change status? The link
> between the SoC and the PHY package is up/down? If it is down, all
> four MAC-PHY links are down. If it is up, it is possible to carry
> frames between the SoC and the PHY package, but maybe the PHYs
> themselves are down?
> 

There is a per channel(for each PHY) configurations in XPCS XGMII mode,
if one PHY has the link change, the related channel should be
configured for this PHY, and multiplex is also used for the connected 4
MACs, the link change on one phy does not effect the other PHYs.

If the PHY is down, the pipeline of the related channel can't transfer
data, the phy link status is reflected to XPCS.

> Withing the four multiplex streams, is there a per stream link change
> indication?

Yes, it is per channel(for each PHY) link change indication in QXGMII.

> 
> 	Andrew
  

On 11/21/2023 12:18 AM, Russell King (Oracle) wrote:
> On Mon, Nov 20, 2023 at 04:34:55PM +0100, Andrew Lunn wrote:
>> Are you saying there is a USXGMII-M level link change status? The link
>> between the SoC and the PHY package is up/down? If it is down, all
>> four MAC-PHY links are down. If it is up, it is possible to carry
>> frames between the SoC and the PHY package, but maybe the PHYs
>> themselves are down?
> 
> It shouldn't do. Each "channel" in the USXGMII-M link has its own
> autoneg block at both ends, each conveys link status independently.
> 
> The MAC side structure is:
> 
> 
>                              +----------+                +-----+
>                      .-XGMII-> Rate     |    PCS         |     |
> MAC1 <-MDI-> PHY <-+        | Adaption <--> Clause 49 <->     |
>                      `-GMII-->          |                |     |
>                              +-----^----+                |     |
>                                    |                     |     |
>                              +-----v---- +               |     |
>                              | Autoneg   |               |     |
>                              | Clause 37 |               |     |
>                              +-----------+               |     |
>                                                          | Mux <--> PMA <-->
>                                                          |     |
>                                                          .......     USXGMII-M
> 
> <------------------------------------------------------>
>        These blocks are repeated for each channel
> 
> The spec goes on to state that there must be a USXGMII enable bit that
> defaults to disabled and the PHY should assume normal XGMII/XFI
> operation. When enabled, autoneg follows a slight modification of
> clause 37-6.
> 
> As far as the USXGMII-M link, I believe 2.7.8 in the USXGMII-M
> documentation covers this, which is "hardware autoneg programming
> sequence". It states that "if 10G link is lost or regained, the
> software is expected to disable autoneg and re-enable autoneg". I
> think "10G link" refers to the USXGMII-M connection, which means
> the loss of that link shold cause software to intervene in each
> of the PCS autoneg blocks. It is, however, rather unclear.
> 

The link status of PHY is updated, software should do the corresponding
QXGMII mode configuration per channel for this PHY.

The PCS QXGMII configuration reflects the current link status of the 
connected PHY.
  

On Tue, Nov 21, 2023 at 07:10:08PM +0800, Jie Luo wrote:
> when pcs is configured to SGMII mode, the fourth PHY can reach to
> maximum speed 2.5G(2500BaseT) that is reached by increasing the clock
> rate to 312.5MHZ from 125MHZ of 1G speed, but there is no corresponding
> interface mode can be used to reflect this 2.5G speed mode(sgmii+)

So this comes up again. 2.5G SGMII? What is that?

Let's start off with the basics. SGMII is Cisco's modification of
1000base-X. The two are broadly compatible in that they can communicate
with each other provided that the inband control word is disregarded.

2500base-X is generally implemented as 1000base-X over-clocked by 2.5x.
Some manufacturers state that the inband control word is not supported.
Others say it can be used. This disparity comes from the lack of early
IEEE standardisation of this protocol.

Cisco SGMII as defined is a 10M/100M/1G protocol operating at 125MHz
with a fixed underlying baud rate of 1250Mbaud. Slower speeds are
achieved via symbol replication by 10x or 100x. The inband control
word is modified in order to convey this speed information, as well
as duplex and sometimes also other vendor extensions.

Switching SGMII to be clocked 2.5x faster means that a partner that
expects SGMII at normal speed sees garbage - it can't recognise the
waveform. Therefore, it is not possible for inband to convey any
information. Many vendors explicitly state that symbol replication
is not supported when "SGMII" is clocked at 2.5x.

All variants of whatever the vendor calls the 2.5G mode tend to use
the SGMII term because... it's Serial Gigabit... and SGMII even gets
used by vendors to describe the interface used for 1000base-X.
Vendors use terms like "HS-SGMII" and other stuff to describe their
2.5x mode. Some use "2500base-X". Yours seems to use "SGMII+".

SGMII without inband signalling is basically the same as 1000base-X.
Therefore, SGMII clocked at 2.5x the speed is basically the same as
2500base-X without inband signalling.

So, the whole area is totally confused, and one should not get too
hung up on the terminology that vendors are using, but go back to
precisely what's going on at the hardware level.

We have raised this point almost every time someone talks about an
up-clocked "SGMII".


> Actually we should add a new interface mode such as sgmii+
> to reflect this 2.5G speed of sgmii

Only if there really is something different about it. For example,
if it were Cisco SGMII modified to operate always at 312.5MHz with
inband signalling updated to signal the four speeds. That would
definitely be a different protocol.

However, it's not that. What it actually is is Cisco SGMII when
operating at 10M/100M/1G speeds, and 2500base-X without inband
signalling when operating at 2.5G speed.

We have PHYs that support this (and more) which we support. PHYs
that switch between 10GBASE-R, 5GBASE-R, 2500BASE-X and Cisco SGMII
depending on the speed that was negotiated on the media. There is
no definition of a single interface mode that covers all those,
because it isn't a single interface mode. It's four separate modes
that the PHY switches between - and this is no different from what
is happening with your PHY.

Ultimately, you will need a way to use inband signalling with Cisco
SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
operating at 2.5G speeds, and that is done via the PHY driver
updating phydev->interface.

What we do need is some way for the PHY to also tell the PCS/MAC
whether inband should be used. This is something I keep bringing up
and now that we have PCS drivers revised to use the value from
phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
we can now think about signalling to PCS drivers whether inband mode
needs to be turned off when switching between modes.

There have been patches in the past that allow inband mode to be
queried from phylib, and this is another important component in
properly dealing with PHYs that need to use inband signalling with
Cisco SGMII, but do not support inband signalling when operating at
2.5G speeds. The problem when operating at 2.5G speed is that the
base-X protocols are normally for use over fibre, which is the media,
and therefore the ethtool Autoneg bit should define whether inband
gets used or not. However, in the case of a PHY using 2500base-X,
the Autoneg bit continues to define whether autonegotiation should
be used on the media, and in this case it's the media side of the
PHY rather than the 2500base-X link.

So, when using a 2500base-X link to a PHY, we need to disregard the
Autoneg bit, but that then raises the question about how we should
configure it - and one solution to that would be to entire of phylib
what the PHY wants to do. Another is to somehow ask the PCS driver
whether it supports inband signalling at 2500base-X, and resolve
those capabilities.

That is my view where we need to get to in order to properly resolve
the ongoing issues about 2500base-X and PHYs that make use of that.
  

On 11/21/2023 7:52 PM, Russell King (Oracle) wrote:
> On Tue, Nov 21, 2023 at 07:10:08PM +0800, Jie Luo wrote:
>> when pcs is configured to SGMII mode, the fourth PHY can reach to
>> maximum speed 2.5G(2500BaseT) that is reached by increasing the clock
>> rate to 312.5MHZ from 125MHZ of 1G speed, but there is no corresponding
>> interface mode can be used to reflect this 2.5G speed mode(sgmii+)
> 
> So this comes up again. 2.5G SGMII? What is that?

2.5G SGMII here is reached by increasing the PCS clock rate to 312.5MHZ
from sgmii with the clock rate 125MHZ.

> 
> Let's start off with the basics. SGMII is Cisco's modification of
> 1000base-X. The two are broadly compatible in that they can communicate
> with each other provided that the inband control word is disregarded.
> 
> 2500base-X is generally implemented as 1000base-X over-clocked by 2.5x.
> Some manufacturers state that the inband control word is not supported.
> Others say it can be used. This disparity comes from the lack of early
> IEEE standardisation of this protocol.
> 
> Cisco SGMII as defined is a 10M/100M/1G protocol operating at 125MHz
> with a fixed underlying baud rate of 1250Mbaud. Slower speeds are
> achieved via symbol replication by 10x or 100x. The inband control
> word is modified in order to convey this speed information, as well
> as duplex and sometimes also other vendor extensions.
> 
> Switching SGMII to be clocked 2.5x faster means that a partner that
> expects SGMII at normal speed sees garbage - it can't recognise the
> waveform. Therefore, it is not possible for inband to convey any
> information. Many vendors explicitly state that symbol replication
> is not supported when "SGMII" is clocked at 2.5x.
> 
> All variants of whatever the vendor calls the 2.5G mode tend to use
> the SGMII term because... it's Serial Gigabit... and SGMII even gets
> used by vendors to describe the interface used for 1000base-X.
> Vendors use terms like "HS-SGMII" and other stuff to describe their
> 2.5x mode. Some use "2500base-X". Yours seems to use "SGMII+".

i use the term SGMII+ because of the 2.5G speed achieved by operating
the clock rate in 312.5MHZ from 1G speed of 125MHZ.

> 
> SGMII without inband signalling is basically the same as 1000base-X.
> Therefore, SGMII clocked at 2.5x the speed is basically the same as
> 2500base-X without inband signalling.

SGMII+ disable the autoneg when the serdes works 312.5MHZ for the 2.5
link speed.

> 
> So, the whole area is totally confused, and one should not get too
> hung up on the terminology that vendors are using, but go back to
> precisely what's going on at the hardware level.
> 
> We have raised this point almost every time someone talks about an
> up-clocked "SGMII".
> 
> 
>> Actually we should add a new interface mode such as sgmii+
>> to reflect this 2.5G speed of sgmii
> 
> Only if there really is something different about it. For example,
> if it were Cisco SGMII modified to operate always at 312.5MHz with
> inband signalling updated to signal the four speeds. That would
> definitely be a different protocol.

For qca808x PHY, the clock rate is different between sgmii(125MHZ for
speed 10/100/1000M) and sgmii+(312.5MHZ for 2.5G).

> 
> However, it's not that. What it actually is is Cisco SGMII when
> operating at 10M/100M/1G speeds, and 2500base-X without inband
> signalling when operating at 2.5G speed.

Yes, Russell, this is what qca808x PHY works on.

> 
> We have PHYs that support this (and more) which we support. PHYs
> that switch between 10GBASE-R, 5GBASE-R, 2500BASE-X and Cisco SGMII
> depending on the speed that was negotiated on the media. There is
> no definition of a single interface mode that covers all those,
> because it isn't a single interface mode. It's four separate modes
> that the PHY switches between - and this is no different from what
> is happening with your PHY.

This is indeed two different modes switched between 2.5G and other
speeds(10/100/1000M).

> 
> Ultimately, you will need a way to use inband signalling with Cisco
> SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
> operating at 2.5G speeds, and that is done via the PHY driver
> updating phydev->interface.
> 
> What we do need is some way for the PHY to also tell the PCS/MAC
> whether inband should be used. This is something I keep bringing up
> and now that we have PCS drivers revised to use the value from
> phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
> we can now think about signalling to PCS drivers whether inband mode
> needs to be turned off when switching between modes.

Yes, we can switch the interface mode according to the current link
speed in the pcs driver.
but the issue is that the phy-mode i specified for the PHYLINK,
if phy-mode is sgmii, the support capability is limited to maximum
capability 1G during the PHYLINK setup and i can't configure it to 2.5G
dynamically, if the phy-mode is 2500base-x, then PHY capability will
be modified to only support 2.5G, other speeds can't be linked up.

> 
> There have been patches in the past that allow inband mode to be
> queried from phylib, and this is another important component in
> properly dealing with PHYs that need to use inband signalling with
> Cisco SGMII, but do not support inband signalling when operating at
> 2.5G speeds. The problem when operating at 2.5G speed is that the
> base-X protocols are normally for use over fibre, which is the media,
> and therefore the ethtool Autoneg bit should define whether inband
> gets used or not. However, in the case of a PHY using 2500base-X,
> the Autoneg bit continues to define whether autonegotiation should
> be used on the media, and in this case it's the media side of the
> PHY rather than the 2500base-X link.
> 
> So, when using a 2500base-X link to a PHY, we need to disregard the
> Autoneg bit, but that then raises the question about how we should
> configure it - and one solution to that would be to entire of phylib
> what the PHY wants to do. Another is to somehow ask the PCS driver
> whether it supports inband signalling at 2500base-X, and resolve
> those capabilities.

For the qca808x PHY, when it is linked in 2.5G, the autoneg is also
disabled in PCS hardware, so the sgmii+ of qca808x PHY is almost
same as 2500base-X.

> 
> That is my view where we need to get to in order to properly resolve
> the ongoing issues about 2500base-X and PHYs that make use of that.
> 

Thanks Russell for the detail information about the SGMII and 2500BASE-X
mode, it is really helpful.

For now, the issue is "the Supported link modes" of PHY is limited when
the interface mode(such as sgmii) is used to setup PHYLINK, which leads
to the PHY can't link in the speed(such as 2.5G) of the different
interface mode(sgmii+).
  

On Thu, Nov 23, 2023 at 06:57:59PM +0800, Jie Luo wrote:
> On 11/21/2023 7:52 PM, Russell King (Oracle) wrote:
> > Ultimately, you will need a way to use inband signalling with Cisco
> > SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
> > operating at 2.5G speeds, and that is done via the PHY driver
> > updating phydev->interface.
> > 
> > What we do need is some way for the PHY to also tell the PCS/MAC
> > whether inband should be used. This is something I keep bringing up
> > and now that we have PCS drivers revised to use the value from
> > phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
> > we can now think about signalling to PCS drivers whether inband mode
> > needs to be turned off when switching between modes.
> 
> Yes, we can switch the interface mode according to the current link
> speed in the pcs driver.
> but the issue is that the phy-mode i specified for the PHYLINK,
> if phy-mode is sgmii, the support capability is limited to maximum
> capability 1G during the PHYLINK setup and i can't configure it to 2.5G
> dynamically, if the phy-mode is 2500base-x, then PHY capability will
> be modified to only support 2.5G, other speeds can't be linked up.

So you need my patches that add "possible_interfaces" to phylib so you
can tell phylink that you will be switching between SGMII and
2500base-X. Please see the RFC posting of those patches I sent
yesterday and try them out - you will need to modify your phylib
driver to fill in phydev->possible_interfaces.

> > There have been patches in the past that allow inband mode to be
> > queried from phylib, and this is another important component in
> > properly dealing with PHYs that need to use inband signalling with
> > Cisco SGMII, but do not support inband signalling when operating at
> > 2.5G speeds. The problem when operating at 2.5G speed is that the
> > base-X protocols are normally for use over fibre, which is the media,
> > and therefore the ethtool Autoneg bit should define whether inband
> > gets used or not. However, in the case of a PHY using 2500base-X,
> > the Autoneg bit continues to define whether autonegotiation should
> > be used on the media, and in this case it's the media side of the
> > PHY rather than the 2500base-X link.
> > 
> > So, when using a 2500base-X link to a PHY, we need to disregard the
> > Autoneg bit, but that then raises the question about how we should
> > configure it - and one solution to that would be to entire of phylib
> > what the PHY wants to do. Another is to somehow ask the PCS driver
> > whether it supports inband signalling at 2500base-X, and resolve
> > those capabilities.
> 
> For the qca808x PHY, when it is linked in 2.5G, the autoneg is also
> disabled in PCS hardware, so the sgmii+ of qca808x PHY is almost
> same as 2500base-X.

Not "almost". It _is_ the same. This is the point I've been trying
to get across to you. Without inband signalling, 1000base-X and SGMII
(when operating at 1G) are _identical_ and entirely compatible.

You've said that your 2.5G "SGMII" mode has inband signalling disabled,
and thus it without inband signalling, 2500base-X and this 2.5G mode
are again identical and entirely compatible. There's no "almost" about
it.
  

On 11/23/2023 8:01 PM, Russell King (Oracle) wrote:
> On Thu, Nov 23, 2023 at 06:57:59PM +0800, Jie Luo wrote:
>> On 11/21/2023 7:52 PM, Russell King (Oracle) wrote:
>>> Ultimately, you will need a way to use inband signalling with Cisco
>>> SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
>>> operating at 2.5G speeds, and that is done via the PHY driver
>>> updating phydev->interface.
>>>
>>> What we do need is some way for the PHY to also tell the PCS/MAC
>>> whether inband should be used. This is something I keep bringing up
>>> and now that we have PCS drivers revised to use the value from
>>> phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
>>> we can now think about signalling to PCS drivers whether inband mode
>>> needs to be turned off when switching between modes.
>>
>> Yes, we can switch the interface mode according to the current link
>> speed in the pcs driver.
>> but the issue is that the phy-mode i specified for the PHYLINK,
>> if phy-mode is sgmii, the support capability is limited to maximum
>> capability 1G during the PHYLINK setup and i can't configure it to 2.5G
>> dynamically, if the phy-mode is 2500base-x, then PHY capability will
>> be modified to only support 2.5G, other speeds can't be linked up.
> 
> So you need my patches that add "possible_interfaces" to phylib so you
> can tell phylink that you will be switching between SGMII and
> 2500base-X. Please see the RFC posting of those patches I sent
> yesterday and try them out - you will need to modify your phylib
> driver to fill in phydev->possible_interfaces.

Your patches work on my board, thanks Russell.

> 
>>> There have been patches in the past that allow inband mode to be
>>> queried from phylib, and this is another important component in
>>> properly dealing with PHYs that need to use inband signalling with
>>> Cisco SGMII, but do not support inband signalling when operating at
>>> 2.5G speeds. The problem when operating at 2.5G speed is that the
>>> base-X protocols are normally for use over fibre, which is the media,
>>> and therefore the ethtool Autoneg bit should define whether inband
>>> gets used or not. However, in the case of a PHY using 2500base-X,
>>> the Autoneg bit continues to define whether autonegotiation should
>>> be used on the media, and in this case it's the media side of the
>>> PHY rather than the 2500base-X link.
>>>
>>> So, when using a 2500base-X link to a PHY, we need to disregard the
>>> Autoneg bit, but that then raises the question about how we should
>>> configure it - and one solution to that would be to entire of phylib
>>> what the PHY wants to do. Another is to somehow ask the PCS driver
>>> whether it supports inband signalling at 2500base-X, and resolve
>>> those capabilities.
>>
>> For the qca808x PHY, when it is linked in 2.5G, the autoneg is also
>> disabled in PCS hardware, so the sgmii+ of qca808x PHY is almost
>> same as 2500base-X.
> 
> Not "almost". It _is_ the same. This is the point I've been trying
> to get across to you. Without inband signalling, 1000base-X and SGMII
> (when operating at 1G) are _identical_ and entirely compatible.
> 
> You've said that your 2.5G "SGMII" mode has inband signalling disabled,
> and thus it without inband signalling, 2500base-X and this 2.5G mode
> are again identical and entirely compatible. There's no "almost" about
> it.
> 
> 
Yes, confirmed with HW guy, they work on the same way.
  

On Fri, Nov 24, 2023 at 05:47:04PM +0800, Jie Luo wrote:
> 
> 
> On 11/23/2023 8:01 PM, Russell King (Oracle) wrote:
> > On Thu, Nov 23, 2023 at 06:57:59PM +0800, Jie Luo wrote:
> > > On 11/21/2023 7:52 PM, Russell King (Oracle) wrote:
> > > > Ultimately, you will need a way to use inband signalling with Cisco
> > > > SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
> > > > operating at 2.5G speeds, and that is done via the PHY driver
> > > > updating phydev->interface.
> > > > 
> > > > What we do need is some way for the PHY to also tell the PCS/MAC
> > > > whether inband should be used. This is something I keep bringing up
> > > > and now that we have PCS drivers revised to use the value from
> > > > phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
> > > > we can now think about signalling to PCS drivers whether inband mode
> > > > needs to be turned off when switching between modes.
> > > 
> > > Yes, we can switch the interface mode according to the current link
> > > speed in the pcs driver.
> > > but the issue is that the phy-mode i specified for the PHYLINK,
> > > if phy-mode is sgmii, the support capability is limited to maximum
> > > capability 1G during the PHYLINK setup and i can't configure it to 2.5G
> > > dynamically, if the phy-mode is 2500base-x, then PHY capability will
> > > be modified to only support 2.5G, other speeds can't be linked up.
> > 
> > So you need my patches that add "possible_interfaces" to phylib so you
> > can tell phylink that you will be switching between SGMII and
> > 2500base-X. Please see the RFC posting of those patches I sent
> > yesterday and try them out - you will need to modify your phylib
> > driver to fill in phydev->possible_interfaces.
> 
> Your patches work on my board, thanks Russell.

Please can you reply to the covering email for that series giving your
tested-by? Thanks.
  

On 11/24/2023 5:53 PM, Russell King (Oracle) wrote:
> On Fri, Nov 24, 2023 at 05:47:04PM +0800, Jie Luo wrote:
>>
>>
>> On 11/23/2023 8:01 PM, Russell King (Oracle) wrote:
>>> On Thu, Nov 23, 2023 at 06:57:59PM +0800, Jie Luo wrote:
>>>> On 11/21/2023 7:52 PM, Russell King (Oracle) wrote:
>>>>> Ultimately, you will need a way to use inband signalling with Cisco
>>>>> SGMII for 10M/100M/1G speeds, and then switch to 2500base-X when
>>>>> operating at 2.5G speeds, and that is done via the PHY driver
>>>>> updating phydev->interface.
>>>>>
>>>>> What we do need is some way for the PHY to also tell the PCS/MAC
>>>>> whether inband should be used. This is something I keep bringing up
>>>>> and now that we have PCS drivers revised to use the value from
>>>>> phylink_pcs_neg_mode() _and_ a consistent implementation amongst them
>>>>> we can now think about signalling to PCS drivers whether inband mode
>>>>> needs to be turned off when switching between modes.
>>>>
>>>> Yes, we can switch the interface mode according to the current link
>>>> speed in the pcs driver.
>>>> but the issue is that the phy-mode i specified for the PHYLINK,
>>>> if phy-mode is sgmii, the support capability is limited to maximum
>>>> capability 1G during the PHYLINK setup and i can't configure it to 2.5G
>>>> dynamically, if the phy-mode is 2500base-x, then PHY capability will
>>>> be modified to only support 2.5G, other speeds can't be linked up.
>>>
>>> So you need my patches that add "possible_interfaces" to phylib so you
>>> can tell phylink that you will be switching between SGMII and
>>> 2500base-X. Please see the RFC posting of those patches I sent
>>> yesterday and try them out - you will need to modify your phylib
>>> driver to fill in phydev->possible_interfaces.
>>
>> Your patches work on my board, thanks Russell.
> 
> Please can you reply to the covering email for that series giving your
> tested-by? Thanks.
> 
Ok.