[RFC,net-next,v2,0/7] Improve GbEth performance on Renesas RZ/G2L and related SoCs

  This series aims to improve peformance of the GbEth IP in the Renesas
RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
the way, we do some refactoring and ensure that napi_complete_done() is
used in accordance with the NAPI documentation for both GbEth and R-Car
code paths.

Performance improvment mainly comes from enabling SW IRQ Coalescing for
all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
but our goal is to set sensible defaults which get good performance on
the affected SoCs.

The performance impact of this series on iperf3 testing is as follows:
  * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
      * Bidirectional and TCP RX: 6.5% less CPU usage
      * UDP RX: 10% less CPU usage

  * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
    cases except UDP TX, which suffers a slight loss:
      * TCP TX: 32% more throughput
      * TCP RX: 11% more throughput
      * UDP TX: 10% less throughput
      * UDP RX: 10183% more throughput - the previous throughput of
        1.06Mbps is what prompted this work.

  * RZ/G2N CPU usage and Ethernet throughput is unchanged (tested as a
    representative of the SoCs which use the R-Car based RAVB IP).

This series depends on:
  * "net: ravb: Let IP-specific receive function to interrogate descriptors" v6
    https://lore.kernel.org/all/20240202084136.3426492-2-claudiu.beznea.uj@bp.renesas.com/

To get the results shown above, you'll also need:
  * "topology: Set capacity_freq_ref in all cases"
    https://lore.kernel.org/all/20240117190545.596057-1-vincent.guittot@linaro.org/

  * "ravb: Add Rx checksum offload support" v4
    https://lore.kernel.org/all/20240203142559.130466-2-biju.das.jz@bp.renesas.com/

  * "ravb: Add Tx checksum offload support" v4
    https://lore.kernel.org/all/20240203142559.130466-3-biju.das.jz@bp.renesas.com/

Work in this area will continue, in particular we expect to improve
TCP/UDP RX performance further with future changes to RX buffer
handling.

Changes v1->v2:
  * Marked as RFC as the series depends on unmerged patches.
  * Refactored R-Car code paths as well as GbEth code paths.
  * Updated references to the patches this series depends on.

Paul Barker (7):
  net: ravb: Simplify poll & receive functions
  net: ravb: Count packets instead of descriptors in RX path
  net: ravb: Always process TX descriptor ring
  net: ravb: Always update error counters
  net: ravb: Align poll function with NAPI docs
  net: ravb: Enable SW IRQ Coalescing for GbEth
  net: ravb: Use NAPI threaded mode on 1-core CPUs with GbEth IP

 drivers/net/ethernet/renesas/ravb.h      |  3 +-
 drivers/net/ethernet/renesas/ravb_main.c | 92 ++++++++++++------------
 2 files changed, 46 insertions(+), 49 deletions(-)
  

On 2/6/24 12:19 PM, Paul Barker wrote:

> This series aims to improve peformance of the GbEth IP in the Renesas

   You didn't fix the typo in "peformance"... :-/

> RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
> the way, we do some refactoring and ensure that napi_complete_done() is
> used in accordance with the NAPI documentation for both GbEth and R-Car
> code paths.
> 
> Performance improvment mainly comes from enabling SW IRQ Coalescing for

   And in "improvment" too... :-/

> all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
> using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
> but our goal is to set sensible defaults which get good performance on
> the affected SoCs.
> 
> The performance impact of this series on iperf3 testing is as follows:
>   * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
>       * Bidirectional and TCP RX: 6.5% less CPU usage
>       * UDP RX: 10% less CPU usage
> 
>   * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
>     cases except UDP TX, which suffers a slight loss:
>       * TCP TX: 32% more throughput
>       * TCP RX: 11% more throughput
>       * UDP TX: 10% less throughput
>       * UDP RX: 10183% more throughput - the previous throughput of

   So this is a real figure? I thought you forgot to erase 10... :-)

>         1.06Mbps is what prompted this work.
> 
>   * RZ/G2N CPU usage and Ethernet throughput is unchanged (tested as a
>     representative of the SoCs which use the R-Car based RAVB IP).
> 
> This series depends on:
>   * "net: ravb: Let IP-specific receive function to interrogate descriptors" v6
>     https://lore.kernel.org/all/20240202084136.3426492-2-claudiu.beznea.uj@bp.renesas.com/

   This one has been merged now, so you can drop RFC...

> To get the results shown above, you'll also need:
>   * "topology: Set capacity_freq_ref in all cases"
>     https://lore.kernel.org/all/20240117190545.596057-1-vincent.guittot@linaro.org/
> 
>   * "ravb: Add Rx checksum offload support" v4
>     https://lore.kernel.org/all/20240203142559.130466-2-biju.das.jz@bp.renesas.com/
> 
>   * "ravb: Add Tx checksum offload support" v4
>     https://lore.kernel.org/all/20240203142559.130466-3-biju.das.jz@bp.renesas.com/

   These two have been merged too...

> Work in this area will continue, in particular we expect to improve
> TCP/UDP RX performance further with future changes to RX buffer
> handling.
> 
> Changes v1->v2:
>   * Marked as RFC as the series depends on unmerged patches.
>   * Refactored R-Car code paths as well as GbEth code paths.
>   * Updated references to the patches this series depends on.
> 
> Paul Barker (7):
>   net: ravb: Simplify poll & receive functions

   The below 3 commits fix issues in the GbEth code, so should
be redone against net.git and posted separately from this series...

>   net: ravb: Count packets instead of descriptors in RX path
>   net: ravb: Always process TX descriptor ring
>   net: ravb: Always update error counters

[...]

MBR, Sergey
  

On 10/02/2024 19:36, Sergey Shtylyov wrote:
> On 2/6/24 12:19 PM, Paul Barker wrote:
> 
>> This series aims to improve peformance of the GbEth IP in the Renesas
> 
>    You didn't fix the typo in "peformance"... :-/
> 
>> RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
>> the way, we do some refactoring and ensure that napi_complete_done() is
>> used in accordance with the NAPI documentation for both GbEth and R-Car
>> code paths.
>>
>> Performance improvment mainly comes from enabling SW IRQ Coalescing for
> 
>    And in "improvment" too... :-/

I'll fix this and the above type in v3.

> 
>> all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
>> using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
>> but our goal is to set sensible defaults which get good performance on
>> the affected SoCs.
>>
>> The performance impact of this series on iperf3 testing is as follows:
>>   * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
>>       * Bidirectional and TCP RX: 6.5% less CPU usage
>>       * UDP RX: 10% less CPU usage
>>
>>   * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
>>     cases except UDP TX, which suffers a slight loss:
>>       * TCP TX: 32% more throughput
>>       * TCP RX: 11% more throughput
>>       * UDP TX: 10% less throughput
>>       * UDP RX: 10183% more throughput - the previous throughput of
> 
>    So this is a real figure? I thought you forgot to erase 10... :-)

Yes, throughput went from 1.06Mbps to 109Mbps for the RZ/G2UL with these
changes.

Initial testing shows that goes up again to 485Mbps with the next patch
series I'm working on to reduce RX buffer sizes.

Biju's work on checksum offload also helps a lot with these numbers, I
can't take all the credit.

> 
>>         1.06Mbps is what prompted this work.
>>
>>   * RZ/G2N CPU usage and Ethernet throughput is unchanged (tested as a
>>     representative of the SoCs which use the R-Car based RAVB IP).
>>
>> This series depends on:
>>   * "net: ravb: Let IP-specific receive function to interrogate descriptors" v6
>>     https://lore.kernel.org/all/20240202084136.3426492-2-claudiu.beznea.uj@bp.renesas.com/
> 
>    This one has been merged now, so you can drop RFC...
> 
>> To get the results shown above, you'll also need:
>>   * "topology: Set capacity_freq_ref in all cases"
>>     https://lore.kernel.org/all/20240117190545.596057-1-vincent.guittot@linaro.org/
>>
>>   * "ravb: Add Rx checksum offload support" v4
>>     https://lore.kernel.org/all/20240203142559.130466-2-biju.das.jz@bp.renesas.com/
>>
>>   * "ravb: Add Tx checksum offload support" v4
>>     https://lore.kernel.org/all/20240203142559.130466-3-biju.das.jz@bp.renesas.com/
> 
>    These two have been merged too...
> 
>> Work in this area will continue, in particular we expect to improve
>> TCP/UDP RX performance further with future changes to RX buffer
>> handling.
>>
>> Changes v1->v2:
>>   * Marked as RFC as the series depends on unmerged patches.
>>   * Refactored R-Car code paths as well as GbEth code paths.
>>   * Updated references to the patches this series depends on.
>>
>> Paul Barker (7):
>>   net: ravb: Simplify poll & receive functions
> 
>    The below 3 commits fix issues in the GbEth code, so should
> be redone against net.git and posted separately from this series...
> 
>>   net: ravb: Count packets instead of descriptors in RX path
>>   net: ravb: Always process TX descriptor ring
>>   net: ravb: Always update error counters

I'll split out and re-submit these as bug fixes. "net: ravb: Count
packets instead of descriptors in RX path" will require a bit of rework
so it doesn't depend on the first patch of the series ("net: ravb:
Simplify poll & receive functions") so you'll probably want to re-review
when I send it.

Then I'll re-send the rest as a non-RFC series.

> 
> [...]
> 
> MBR, Sergey

Thanks for the review!
Paul
  

On 2/12/24 2:52 PM, Paul Barker wrote:
[...]

>>> This series aims to improve peformance of the GbEth IP in the Renesas
>>
>>    You didn't fix the typo in "peformance"... :-/
>>
>>> RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
>>> the way, we do some refactoring and ensure that napi_complete_done() is
>>> used in accordance with the NAPI documentation for both GbEth and R-Car
>>> code paths.
>>>
>>> Performance improvment mainly comes from enabling SW IRQ Coalescing for
>>
>>    And in "improvment" too... :-/
> 
> I'll fix this and the above type in v3.

   TIA! Chances are this will end up in the merge commit...

>>> all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
>>> using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
>>> but our goal is to set sensible defaults which get good performance on
>>> the affected SoCs.
>>>
>>> The performance impact of this series on iperf3 testing is as follows:
>>>   * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
>>>       * Bidirectional and TCP RX: 6.5% less CPU usage
>>>       * UDP RX: 10% less CPU usage
>>>
>>>   * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
>>>     cases except UDP TX, which suffers a slight loss:
>>>       * TCP TX: 32% more throughput
>>>       * TCP RX: 11% more throughput
>>>       * UDP TX: 10% less throughput
>>>       * UDP RX: 10183% more throughput - the previous throughput of
>>
>>    So this is a real figure? I thought you forgot to erase 10... :-)
> 
> Yes, throughput went from 1.06Mbps to 109Mbps for the RZ/G2UL with these
> changes.

   Hm, that gives me even 10283%! :-)

> Initial testing shows that goes up again to 485Mbps with the next patch
> series I'm working on to reduce RX buffer sizes.

   Oh, wow! :-)

> Biju's work on checksum offload also helps a lot with these numbers, I
> can't take all the credit.

   Took 5 versions to merge, unfortunately... :-/

[...]

>>> Work in this area will continue, in particular we expect to improve
>>> TCP/UDP RX performance further with future changes to RX buffer
>>> handling.
>>>
>>> Changes v1->v2:
>>>   * Marked as RFC as the series depends on unmerged patches.
>>>   * Refactored R-Car code paths as well as GbEth code paths.
>>>   * Updated references to the patches this series depends on.
>>>
>>> Paul Barker (7):
>>>   net: ravb: Simplify poll & receive functions
>>
>>    The below 3 commits fix issues in the GbEth code, so should
>> be redone against net.git and posted separately from this series...
>>
>>>   net: ravb: Count packets instead of descriptors in RX path
>>>   net: ravb: Always process TX descriptor ring
>>>   net: ravb: Always update error counters
> 
> I'll split out and re-submit these as bug fixes. "net: ravb: Count
> packets instead of descriptors in RX path" will require a bit of rework
> so it doesn't depend on the first patch of the series ("net: ravb:
> Simplify poll & receive functions") so you'll probably want to re-review
> when I send it.

   Yes, I figured that at least the 1st patch would need to be reworked...

> Then I'll re-send the rest as a non-RFC series.

   Won't they need to be rebased against 3 fixes?

[...]

> Thanks for the review!
> Paul

MBR, Sergey
  

On 2/12/24 11:53 PM, Sergey Shtylyov wrote:
[...]

>>>> This series aims to improve peformance of the GbEth IP in the Renesas
>>>
>>>    You didn't fix the typo in "peformance"... :-/
>>>
>>>> RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
>>>> the way, we do some refactoring and ensure that napi_complete_done() is
>>>> used in accordance with the NAPI documentation for both GbEth and R-Car
>>>> code paths.
>>>>
>>>> Performance improvment mainly comes from enabling SW IRQ Coalescing for
>>>
>>>    And in "improvment" too... :-/
>>
>> I'll fix this and the above type in v3.
> 
>    TIA! Chances are this will end up in the merge commit...
> 
>>>> all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
>>>> using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
>>>> but our goal is to set sensible defaults which get good performance on
>>>> the affected SoCs.
>>>>
>>>> The performance impact of this series on iperf3 testing is as follows:
>>>>   * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
>>>>       * Bidirectional and TCP RX: 6.5% less CPU usage
>>>>       * UDP RX: 10% less CPU usage
>>>>
>>>>   * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
>>>>     cases except UDP TX, which suffers a slight loss:
>>>>       * TCP TX: 32% more throughput
>>>>       * TCP RX: 11% more throughput
>>>>       * UDP TX: 10% less throughput
>>>>       * UDP RX: 10183% more throughput - the previous throughput of
>>>
>>>    So this is a real figure? I thought you forgot to erase 10... :-)
>>
>> Yes, throughput went from 1.06Mbps to 109Mbps for the RZ/G2UL with these
>> changes.
> 
>    Hm, that gives me even 10283%! :-)

   Stupid me, forgot to subtract 100%... :-)

[...]

MBR, Sergey
  

On 12/02/2024 20:53, Sergey Shtylyov wrote:
> On 2/12/24 2:52 PM, Paul Barker wrote:
> [...]
> 
>>>> This series aims to improve peformance of the GbEth IP in the Renesas
>>>
>>>    You didn't fix the typo in "peformance"... :-/
>>>
>>>> RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
>>>> the way, we do some refactoring and ensure that napi_complete_done() is
>>>> used in accordance with the NAPI documentation for both GbEth and R-Car
>>>> code paths.
>>>>
>>>> Performance improvment mainly comes from enabling SW IRQ Coalescing for
>>>
>>>    And in "improvment" too... :-/
>>
>> I'll fix this and the above type in v3.
> 
>    TIA! Chances are this will end up in the merge commit...
> 
>>>> all SoCs using the GbEth IP, and NAPI Threaded mode for single core SoCs
>>>> using the GbEth IP. These can be enabled/disabled at runtime via sysfs,
>>>> but our goal is to set sensible defaults which get good performance on
>>>> the affected SoCs.
>>>>
>>>> The performance impact of this series on iperf3 testing is as follows:
>>>>   * RZ/G2L Ethernet throughput is unchanged, but CPU usage drops:
>>>>       * Bidirectional and TCP RX: 6.5% less CPU usage
>>>>       * UDP RX: 10% less CPU usage
>>>>
>>>>   * RZ/G2UL and RZ/G3S Ethernet throughput is increased for all test
>>>>     cases except UDP TX, which suffers a slight loss:
>>>>       * TCP TX: 32% more throughput
>>>>       * TCP RX: 11% more throughput
>>>>       * UDP TX: 10% less throughput
>>>>       * UDP RX: 10183% more throughput - the previous throughput of
>>>
>>>    So this is a real figure? I thought you forgot to erase 10... :-)
>>
>> Yes, throughput went from 1.06Mbps to 109Mbps for the RZ/G2UL with these
>> changes.
> 
>    Hm, that gives me even 10283%! :-)
> 
>> Initial testing shows that goes up again to 485Mbps with the next patch
>> series I'm working on to reduce RX buffer sizes.
> 
>    Oh, wow! :-)
> 
>> Biju's work on checksum offload also helps a lot with these numbers, I
>> can't take all the credit.
> 
>    Took 5 versions to merge, unfortunately... :-/
> 
> [...]
> 
>>>> Work in this area will continue, in particular we expect to improve
>>>> TCP/UDP RX performance further with future changes to RX buffer
>>>> handling.
>>>>
>>>> Changes v1->v2:
>>>>   * Marked as RFC as the series depends on unmerged patches.
>>>>   * Refactored R-Car code paths as well as GbEth code paths.
>>>>   * Updated references to the patches this series depends on.
>>>>
>>>> Paul Barker (7):
>>>>   net: ravb: Simplify poll & receive functions
>>>
>>>    The below 3 commits fix issues in the GbEth code, so should
>>> be redone against net.git and posted separately from this series...
>>>
>>>>   net: ravb: Count packets instead of descriptors in RX path
>>>>   net: ravb: Always process TX descriptor ring
>>>>   net: ravb: Always update error counters
>>
>> I'll split out and re-submit these as bug fixes. "net: ravb: Count
>> packets instead of descriptors in RX path" will require a bit of rework
>> so it doesn't depend on the first patch of the series ("net: ravb:
>> Simplify poll & receive functions") so you'll probably want to re-review
>> when I send it.
> 
>    Yes, I figured that at least the 1st patch would need to be reworked...
> 
>> Then I'll re-send the rest as a non-RFC series.
> 
>    Won't they need to be rebased against 3 fixes?

Yes, the rest will need rebasing.

We need to test gPTP on an RZ/G2N board with these changes first. We're
working on it and I'll let you know the status soon. I should be able to
send at least one bugfix in a way that doesn't affect RZ/G2N & R-Car
boards though...

Thanks,