[net-next,v2,00/21] net: ravb: Add suspend to RAM and runtime PM support for RZ/G3S

  From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>

Hi,

This series adds suspend to RAM and runtime PM support for Ethernet
IP available on the RZ/G3S (R9A08G045) SoC.

As there are IP versions that switch to module standby when disabling
the clocks, and because of module standby IP switches to reset and
the register content is lost, to be able to have runtime PM supported
for all IP variants, the configuration operations were moved all to
ravb_open()/ravb_close() letting the ravb_probe() and ravb_remove()
to deal with resource parsing and allocation/free.

The ethtool and IOCTL APIs that could have been run asyncronously
were adapted to return if the interface is down. As explained in
each individual commits description, this should be harmless.

Along with it, the series contains preparatory cleanups.

The series has been tested on the boards with the following device trees:
- r8a7742-iwg21d-q7.dts
- r8a774a1-hihope-rzg2m-ex.dts 
- r9a07g043u11-smarc-rzg2ul.dts
- r9a07g054l2-smarc-rzv2l.dts
- r9a07g044l2-smarc-rzg2l.dts

Patches are based on series at [1].

Thank you,
Claudiu Beznea

Changes in v2:
- rework the driver (mainly, ravb_open() contains now only resource
  allocation and parsing leaving the settings to ravb_open(); ravb_remove()
  has been adapted accordingly) to be able to use runtime PM for all
  IP variants; due to this number of patches increased
- adjust previous series to review comments
- collected tags
- populated driver's own runtime PM ops with enable/disable of reference
  clock

[1] https://lore.kernel.org/all/20231214113137.2450292-1-claudiu.beznea.uj@bp.renesas.com/

Claudiu Beznea (21):
  net: ravb: Let IP-specific receive function to interrogate descriptors
  net: ravb: Rely on PM domain to enable gptp_clk
  net: ravb: Make reset controller support mandatory
  net: ravb: Switch to SYSTEM_SLEEP_PM_OPS()/RUNTIME_PM_OPS() and
    pm_ptr()
  net: ravb: Use tabs instead of spaces
  net: ravb: Assert/de-assert reset on suspend/resume
  net: ravb: Move reference clock enable/disable on runtime PM APIs
  net: ravb: Move the IRQs get and request in the probe function
  net: ravb: Split GTI computation and set operations
  net: ravb: Move delay mode set in the driver's ndo_open API
  net: ravb: Move DBAT configuration to the driver's ndo_open API
  net: ravb: Move ptp initialization in the driver's ndo_open API for
    ccc_gac platorms
  net: ravb: Set config mode in ndo_open and reset mode in ndo_close
  net: ravb: Simplify ravb_suspend()
  net: ravb: Simplify ravb_resume()
  net: ravb: Keep the reverse order of operations in ravb_close()
  net: ravb: Keep clock request operations grouped together
  net: ravb: Return cached statistics if the interface is down
  net: ravb: Do not set promiscuous mode if the interface is down
  net: ravb: Do not apply RX CSUM settings to hardware if the interface
    is down
  net: ravb: Add runtime PM support

 drivers/net/ethernet/renesas/ravb.h      |   2 +
 drivers/net/ethernet/renesas/ravb_main.c | 783 ++++++++++++-----------
 2 files changed, 417 insertions(+), 368 deletions(-)
  

On 14.12.2023 13:45, Claudiu wrote:
> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> Hi,
> 
> This series adds suspend to RAM and runtime PM support for Ethernet
> IP available on the RZ/G3S (R9A08G045) SoC.
> 
> As there are IP versions that switch to module standby when disabling
> the clocks, and because of module standby IP switches to reset and
> the register content is lost, to be able to have runtime PM supported
> for all IP variants, the configuration operations were moved all to
> ravb_open()/ravb_close() letting the ravb_probe() and ravb_remove()
> to deal with resource parsing and allocation/free.
> 
> The ethtool and IOCTL APIs that could have been run asyncronously
> were adapted to return if the interface is down. As explained in
> each individual commits description, this should be harmless.
> 
> Along with it, the series contains preparatory cleanups.
> 
> The series has been tested on the boards with the following device trees:
> - r8a7742-iwg21d-q7.dts
> - r8a774a1-hihope-rzg2m-ex.dts 
> - r9a07g043u11-smarc-rzg2ul.dts
> - r9a07g054l2-smarc-rzv2l.dts
> - r9a07g044l2-smarc-rzg2l.dts

And also (I forgot to mention):
- r9a08g045s33-smarc.dts (RZ/G3S).

> 
> Patches are based on series at [1].
> 
> Thank you,
> Claudiu Beznea
> 
> Changes in v2:
> - rework the driver (mainly, ravb_open() contains now only resource
>   allocation and parsing leaving the settings to ravb_open(); ravb_remove()
>   has been adapted accordingly) to be able to use runtime PM for all
>   IP variants; due to this number of patches increased
> - adjust previous series to review comments
> - collected tags
> - populated driver's own runtime PM ops with enable/disable of reference
>   clock
> 
> [1] https://lore.kernel.org/all/20231214113137.2450292-1-claudiu.beznea.uj@bp.renesas.com/
> 
> Claudiu Beznea (21):
>   net: ravb: Let IP-specific receive function to interrogate descriptors
>   net: ravb: Rely on PM domain to enable gptp_clk
>   net: ravb: Make reset controller support mandatory
>   net: ravb: Switch to SYSTEM_SLEEP_PM_OPS()/RUNTIME_PM_OPS() and
>     pm_ptr()
>   net: ravb: Use tabs instead of spaces
>   net: ravb: Assert/de-assert reset on suspend/resume
>   net: ravb: Move reference clock enable/disable on runtime PM APIs
>   net: ravb: Move the IRQs get and request in the probe function
>   net: ravb: Split GTI computation and set operations
>   net: ravb: Move delay mode set in the driver's ndo_open API
>   net: ravb: Move DBAT configuration to the driver's ndo_open API
>   net: ravb: Move ptp initialization in the driver's ndo_open API for
>     ccc_gac platorms
>   net: ravb: Set config mode in ndo_open and reset mode in ndo_close
>   net: ravb: Simplify ravb_suspend()
>   net: ravb: Simplify ravb_resume()
>   net: ravb: Keep the reverse order of operations in ravb_close()
>   net: ravb: Keep clock request operations grouped together
>   net: ravb: Return cached statistics if the interface is down
>   net: ravb: Do not set promiscuous mode if the interface is down
>   net: ravb: Do not apply RX CSUM settings to hardware if the interface
>     is down
>   net: ravb: Add runtime PM support
> 
>  drivers/net/ethernet/renesas/ravb.h      |   2 +
>  drivers/net/ethernet/renesas/ravb_main.c | 783 ++++++++++++-----------
>  2 files changed, 417 insertions(+), 368 deletions(-)
>
  

On 14.12.2023 21:26, Jakub Kicinski wrote:
> On Thu, 14 Dec 2023 13:45:39 +0200 Claudiu wrote:
>> Subject: [PATCH net-next v2 00/21] 
> 
> We got 260 patches in the review queue. Please pace your patches:
> https://www.kernel.org/doc/html/next/process/maintainer-netdev.html#tl-dr
> 
>> Patches are based on series at [1].
>>
>> [1] https://lore.kernel.org/all/20231214113137.2450292-1-claudiu.beznea.uj@bp.renesas.com/
> 
> Meaning there's a dependency we're supposed to track?

The intention was to have a review on this series (from driver's
maintainers) while the fixes are integrated, if any.

> You have to wait for fixes to land, we marge the trees every week.

The intention was to let the reviewers know they should apply [1] (if any)
for reviewing this series.

Sorry for any inconvenience,
Claudiu Beznea
  

On Fri, 15 Dec 2023 11:44:32 +0200 claudiu beznea wrote:
> > You have to wait for fixes to land, we marge the trees every week.  
> 
> The intention was to let the reviewers know they should apply [1] (if any)
> for reviewing this series.

If there's a dependency please post the "later" thing as RFC.
We can't apply it, and it saves us clicking it away in patchwork.
  

On 12/14/23 2:45 PM, Claudiu wrote:

> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> Delay parse and set were done in the driver's probe API. As some IP

   Parsing and setting?

> variants switch to reset mode (and thus registers' content is lost) when
> setting clocks (due to module standby functionality) to be able to
> implement runtime PM keep the delay parsing in the driver's probe function
> and move the delay apply function to the driver's ndo_open API.

   Applying?

> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
[...]

> diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
> index 5e01e03e1b43..04eaa1967651 100644
> --- a/drivers/net/ethernet/renesas/ravb_main.c
> +++ b/drivers/net/ethernet/renesas/ravb_main.c
[...]
> @@ -1806,6 +1821,8 @@ static int ravb_open(struct net_device *ndev)
>  	if (info->nc_queues)
>  		napi_enable(&priv->napi[RAVB_NC]);
>  
> +	ravb_set_delay_mode(ndev);
> +

   I suspect this belongs in ravb_dmac_init() now...

>  	/* Device init */
>  	error = ravb_dmac_init(ndev);
>  	if (error)
[...]

MRB, Sergey
  

On 12/14/23 2:45 PM, Claudiu wrote:

> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> Delay parse and set were done in the driver's probe API. As some IP

   Parsing and setting?

> variants switch to reset mode (and thus registers' content is lost) when

   Register.

> setting clocks (due to module standby functionality) to be able to
> implement runtime PM keep the delay parsing in the driver's probe function
> and move the delay apply function to the driver's ndo_open API.

   Applying?

> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
[...]

> diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
> index 5e01e03e1b43..04eaa1967651 100644
> --- a/drivers/net/ethernet/renesas/ravb_main.c
> +++ b/drivers/net/ethernet/renesas/ravb_main.c
[...]
> @@ -1806,6 +1821,8 @@ static int ravb_open(struct net_device *ndev)
>  	if (info->nc_queues)
>  		napi_enable(&priv->napi[RAVB_NC]);
>  
> +	ravb_set_delay_mode(ndev);
> +

   I suspect this belongs in ravb_dmac_init() now...

>  	/* Device init */
>  	error = ravb_dmac_init(ndev);
>  	if (error)
[...]

MRB, Sergey
  

On 12/14/23 2:45 PM, Claudiu wrote:

> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> ravb_set_gti() was computing the value of GTI based on the reference clock
> rate and then applied it to register. This was done on the driver's probe
> function. In order to implement runtime PM for all IP variants (as some IP
> variants switches to reset operation mode (and thus the register's content

   Again, operating mode...

> is lost) when module standby is configured through clock APIs) the GTI was

   The GTI what? Setup?

> split in 2 parts: one computing the value of the GTI register (done in the
> driver's probe function) and one applying the computed value to register
> (done in the driver's ndo_open API).
> 
> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
[...]

> diff --git a/drivers/net/ethernet/renesas/ravb.h b/drivers/net/ethernet/renesas/ravb.h
> index e0f8276cffed..76202395b68d 100644
> --- a/drivers/net/ethernet/renesas/ravb.h
> +++ b/drivers/net/ethernet/renesas/ravb.h
> @@ -1106,6 +1106,8 @@ struct ravb_private {
>  
>  	const struct ravb_hw_info *info;
>  	struct reset_control *rstc;
> +
> +	uint64_t gti_tiv;

   Please use the kernel type, u64; uint64_t is for userland, IIRC.

[...]
> diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
> index d7f6e8ea8e79..5e01e03e1b43 100644
> --- a/drivers/net/ethernet/renesas/ravb_main.c
> +++ b/drivers/net/ethernet/renesas/ravb_main.c
> @@ -1750,6 +1750,51 @@ static int ravb_set_reset_mode(struct net_device *ndev)
>  	return error;
>  }
>  
> +static int ravb_set_gti(struct net_device *ndev)
> +{
[...]
> +	/* Request GTI loading */
> +	ravb_modify(ndev, GCCR, GCCR_LTI, GCCR_LTI);
> +
> +	/* Check completion status. */
> +	return ravb_wait(ndev, GCCR, GCCR_LTI, 0);

   Is this really necessary?

[...]
> @@ -1767,6 +1812,10 @@ static int ravb_open(struct net_device *ndev)
>  		goto out_napi_off;
>  	ravb_emac_init(ndev);
>  
> +	error = ravb_set_gti(ndev);
> +	if (error)
> +		goto out_dma_stop;
> +

   Hm... belongs in ravb_dmac_init() now, as it seems... 

[...]

MBR, Sergey
  

On 12/14/23 2:45 PM, Claudiu wrote:

> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> As some IP variants switch to reset mode (and thus registers' content is

   Register.

> lost) when setting clocks (due to module standby functionality) to be able
> to implement runtime PM and save more power, set the IP's operation mode to

   Operating.

> reset at the end of the probe. Along with it, in the ndo_open API the IP
> will be switched to configuration, then operational mode. In the ndo_close
> API, the IP will be switched back to reset mode. This allows implementing
> runtime PM and, along with it, save more power when the IP is not used.
> 
> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
[..]

> diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
> index db9222fc57c2..31a1f8a83652 100644
> --- a/drivers/net/ethernet/renesas/ravb_main.c
> +++ b/drivers/net/ethernet/renesas/ravb_main.c
[...]
> @@ -1821,13 +1845,19 @@ static int ravb_open(struct net_device *ndev)
>  	if (info->nc_queues)
>  		napi_enable(&priv->napi[RAVB_NC]);
>  
> +	/* Set AVB config mode */
> +	error = ravb_set_config_mode(ndev);
> +	if (error)
> +		goto out_napi_off;
> +

   I suspect this too belongs in ravb_dmac_init() now...

[...]
> @@ -2875,19 +2886,30 @@ static int ravb_probe(struct platform_device *pdev)
>  
>  	device_set_wakeup_capable(&pdev->dev, 1);
>  
> +	/* Reset MAC as the module will be runtime disabled at this moment.
> +	 * This saves power. MAC will be switched back to configuration mode
> +	 * in ravb_open().
> +	 */
> +	error = ravb_set_reset_mode(ndev);
> +	if (error)
> +		goto out_netdev_unregister;
> +

   I think this now races with the register_netdev() call above (the device
can be opened before it returns)! Should be called before register_netdev()...

[...]

MBR, Sergey
  

On 12/14/23 2:45 PM, Claudiu wrote:

> From: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> 
> Keep clock request operations grouped togeter to have all clock-related
> code in a single place. This makes the code simpler to follow.
> 
> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@bp.renesas.com>
> ---
> 
> Changes in v2:
> - none; this patch is new
> 
>  drivers/net/ethernet/renesas/ravb_main.c | 28 ++++++++++++------------
>  1 file changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
> index 38999ef1ea85..a2a64c22ec41 100644
> --- a/drivers/net/ethernet/renesas/ravb_main.c
> +++ b/drivers/net/ethernet/renesas/ravb_main.c
> @@ -2768,6 +2768,20 @@ static int ravb_probe(struct platform_device *pdev)
>  	if (error)
>  		goto out_reset_assert;
>  
> +	priv->clk = devm_clk_get(&pdev->dev, NULL);
> +	if (IS_ERR(priv->clk)) {
> +		error = PTR_ERR(priv->clk);
> +		goto out_reset_assert;
> +	}
> +
> +	if (info->gptp_ref_clk) {
> +		priv->gptp_clk = devm_clk_get(&pdev->dev, "gptp");
> +		if (IS_ERR(priv->gptp_clk)) {
> +			error = PTR_ERR(priv->gptp_clk);
> +			goto out_reset_assert;
> +		}
> +	}
> +
>  	priv->refclk = devm_clk_get_optional(&pdev->dev, "refclk");
>  	if (IS_ERR(priv->refclk)) {
>  		error = PTR_ERR(priv->refclk);

   Hmm... I think we currently have all these calls in one place.
Perhaps you just shouldn't have moved this code around?

MBR, Sergey