[RFC,0/5] Enable multiple MCAN on AM62x

  On AM62x there is one MCAN in MAIN domain and two in MCU domain.
The MCANs in MCU domain were not enabled since there is no
hardware interrupt routed to A53 GIC interrupt controller.
Therefore A53 Linux cannot be interrupted by MCU MCANs.

This solution instantiates a hrtimer with 1 ms polling interval
for a MCAN when there is no hardware interrupt. This hrtimer
generates a recurring software interrupt which allows to call the
isr. The isr will check if there is pending transaction by reading
a register and proceed normally if there is.

On AM62x this series enables two MCU MCAN which will use the hrtimer
implementation. MCANs with hardware interrupt routed to A53 Linux
will continue to use the hardware interrupt as expected.

Timer polling method was tested on both classic CAN and CAN-FD
at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
switching.

Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
1 MBPS timer polling interval is the better timer polling interval
since it has comparable latency to hardware interrupt with the worse
case being 1ms + CAN frame propagation time and CPU load is not
substantial. Latency can be improved further with less than 1 ms
polling intervals, howerver it is at the cost of CPU usage since CPU
load increases at 0.5 ms and lower polling periods than 1ms.

Note that in terms of power, enabling MCU MCANs with timer-polling
implementation might have negative impact since we will have to wake
up every 1 ms whether there are CAN packets pending in the RX FIFO or
not. This might prevent the CPU from entering into deeper idle states
for extended periods of time.

This patch series depends on 'Enable CAN PHY transceiver driver':
https://lore.kernel.org/lkml/775ec9ce-7668-429c-a977-6c8995968d6e@app.fastmail.com/T/

Judith Mendez (5):
  arm64: dts: ti: Add AM62x MCAN MAIN domain transceiver overlay
  arm64: defconfig: Enable MCAN driver
  dt-binding: can: m_can: Remove required interrupt attributes
  arm64: dts: ti: Enable multiple MCAN for AM62x in MCU MCAN overlay
  can: m_can: Add hrtimer to generate software interrupt

 .../bindings/net/can/bosch,m_can.yaml         |  2 -
 arch/arm64/boot/dts/ti/Makefile               |  2 +
 .../boot/dts/ti/k3-am625-sk-mcan-main.dtso    | 35 +++++++++
 .../boot/dts/ti/k3-am625-sk-mcan-mcu.dtso     | 75 +++++++++++++++++++
 arch/arm64/configs/defconfig                  |  2 +
 drivers/net/can/m_can/m_can.c                 | 24 +++++-
 drivers/net/can/m_can/m_can.h                 |  3 +
 drivers/net/can/m_can/m_can_platform.c        |  9 ++-
 8 files changed, 146 insertions(+), 6 deletions(-)
 create mode 100644 arch/arm64/boot/dts/ti/k3-am625-sk-mcan-main.dtso
 create mode 100644 arch/arm64/boot/dts/ti/k3-am625-sk-mcan-mcu.dtso
  

Hi Judith,

On 14/04/23 04:00, Judith Mendez wrote:
> Judith Mendez (5):
>   arm64: dts: ti: Add AM62x MCAN MAIN domain transceiver overlay
>   arm64: defconfig: Enable MCAN driver
>   dt-binding: can: m_can: Remove required interrupt attributes
>   arm64: dts: ti: Enable multiple MCAN for AM62x in MCU MCAN overlay
>   can: m_can: Add hrtimer to generate software interrupt

This is fine for RFC, but next time, please split DT and defconfig
changes (1/5,2/5, and 4/5) to separate series as they have to go via
arm64 tree.
  

On 13.04.2023 17:30:46, Judith Mendez wrote:
> On AM62x there is one MCAN in MAIN domain and two in MCU domain.
> The MCANs in MCU domain were not enabled since there is no
> hardware interrupt routed to A53 GIC interrupt controller.
> Therefore A53 Linux cannot be interrupted by MCU MCANs.

Is this a general hardware limitation, that effects all MCU domain
peripherals? Is there a mailbox mechanism between the MCU and the MAIN
domain, would it be possible to pass the IRQ with a small firmware on
the MCU? Anyways, that's future optimization.

> This solution instantiates a hrtimer with 1 ms polling interval
> for a MCAN when there is no hardware interrupt. This hrtimer
> generates a recurring software interrupt which allows to call the
> isr. The isr will check if there is pending transaction by reading
> a register and proceed normally if there is.
> 
> On AM62x this series enables two MCU MCAN which will use the hrtimer
> implementation. MCANs with hardware interrupt routed to A53 Linux
> will continue to use the hardware interrupt as expected.
> 
> Timer polling method was tested on both classic CAN and CAN-FD
> at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
> switching.
> 
> Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
> 1 MBPS timer polling interval is the better timer polling interval
> since it has comparable latency to hardware interrupt with the worse
> case being 1ms + CAN frame propagation time and CPU load is not
> substantial. Latency can be improved further with less than 1 ms
> polling intervals, howerver it is at the cost of CPU usage since CPU
> load increases at 0.5 ms and lower polling periods than 1ms.

Some Linux input drivers have the property poll-interval, would it make
sense to ass this here too?

> Note that in terms of power, enabling MCU MCANs with timer-polling
> implementation might have negative impact since we will have to wake
> up every 1 ms whether there are CAN packets pending in the RX FIFO or
> not. This might prevent the CPU from entering into deeper idle states
> for extended periods of time.
> 
> This patch series depends on 'Enable CAN PHY transceiver driver':
> https://lore.kernel.org/lkml/775ec9ce-7668-429c-a977-6c8995968d6e@app.fastmail.com/T/

Marc
  

Hello Marc,

On 4/14/2023 12:49 PM, Marc Kleine-Budde wrote:
> On 13.04.2023 17:30:46, Judith Mendez wrote:
>> On AM62x there is one MCAN in MAIN domain and two in MCU domain.
>> The MCANs in MCU domain were not enabled since there is no
>> hardware interrupt routed to A53 GIC interrupt controller.
>> Therefore A53 Linux cannot be interrupted by MCU MCANs.
> 
> Is this a general hardware limitation, that effects all MCU domain
> peripherals? Is there a mailbox mechanism between the MCU and the MAIN
> domain, would it be possible to pass the IRQ with a small firmware on
> the MCU? Anyways, that's future optimization.
> 

This is a hardware limitation that affects AM62x SoC and has been 
carried over to at least 1 other SoC. Using the MCU is an idea that we 
have juggled around for a while, we will definitely keep it in mind for 
future optimization. Thanks for your feedback.

>> This solution instantiates a hrtimer with 1 ms polling interval
>> for a MCAN when there is no hardware interrupt. This hrtimer
>> generates a recurring software interrupt which allows to call the
>> isr. The isr will check if there is pending transaction by reading
>> a register and proceed normally if there is.
>>
>> On AM62x this series enables two MCU MCAN which will use the hrtimer
>> implementation. MCANs with hardware interrupt routed to A53 Linux
>> will continue to use the hardware interrupt as expected.
>>
>> Timer polling method was tested on both classic CAN and CAN-FD
>> at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
>> switching.
>>
>> Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
>> 1 MBPS timer polling interval is the better timer polling interval
>> since it has comparable latency to hardware interrupt with the worse
>> case being 1ms + CAN frame propagation time and CPU load is not
>> substantial. Latency can be improved further with less than 1 ms
>> polling intervals, howerver it is at the cost of CPU usage since CPU
>> load increases at 0.5 ms and lower polling periods than 1ms.
> 
> Some Linux input drivers have the property poll-interval, would it make
> sense to ass this here too?
> 
>> Note that in terms of power, enabling MCU MCANs with timer-polling
>> implementation might have negative impact since we will have to wake
>> up every 1 ms whether there are CAN packets pending in the RX FIFO or
>> not. This might prevent the CPU from entering into deeper idle states
>> for extended periods of time.
>>
>> This patch series depends on 'Enable CAN PHY transceiver driver':
>> https://lore.kernel.org/lkml/775ec9ce-7668-429c-a977-6c8995968d6e@app.fastmail.com/T/
> 
> Marc
> 

regards,
Judith
  

On 18.04.2023 11:15:35, Mendez, Judith wrote:
> Hello Marc,
> 
> On 4/14/2023 12:49 PM, Marc Kleine-Budde wrote:
> > On 13.04.2023 17:30:46, Judith Mendez wrote:
> > > On AM62x there is one MCAN in MAIN domain and two in MCU domain.
> > > The MCANs in MCU domain were not enabled since there is no
> > > hardware interrupt routed to A53 GIC interrupt controller.
> > > Therefore A53 Linux cannot be interrupted by MCU MCANs.
> > 
> > Is this a general hardware limitation, that effects all MCU domain
> > peripherals? Is there a mailbox mechanism between the MCU and the MAIN
> > domain, would it be possible to pass the IRQ with a small firmware on
> > the MCU? Anyways, that's future optimization.
> 
> This is a hardware limitation that affects AM62x SoC and has been carried
> over to at least 1 other SoC. Using the MCU is an idea that we have juggled
> around for a while, we will definitely keep it in mind for future
> optimization. Thanks for your feedback.

Once you have a proper IRQ de-multiplexer, you can integrate it into the
system with a DT change only. No need for changes in the m_can driver.

> > > This solution instantiates a hrtimer with 1 ms polling interval
> > > for a MCAN when there is no hardware interrupt. This hrtimer
> > > generates a recurring software interrupt which allows to call the
> > > isr. The isr will check if there is pending transaction by reading
> > > a register and proceed normally if there is.
> > > 
> > > On AM62x this series enables two MCU MCAN which will use the hrtimer
> > > implementation. MCANs with hardware interrupt routed to A53 Linux
> > > will continue to use the hardware interrupt as expected.
> > > 
> > > Timer polling method was tested on both classic CAN and CAN-FD
> > > at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
> > > switching.
> > > 
> > > Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
> > > 1 MBPS timer polling interval is the better timer polling interval
> > > since it has comparable latency to hardware interrupt with the worse
> > > case being 1ms + CAN frame propagation time and CPU load is not
> > > substantial. Latency can be improved further with less than 1 ms
> > > polling intervals, howerver it is at the cost of CPU usage since CPU
> > > load increases at 0.5 ms and lower polling periods than 1ms.

Have you seen my suggestion of the poll-interval?

Some Linux input drivers have the property poll-interval, would it make
sense to ass this here too?

Marc
  

Hello Vignesh,

On 4/14/2023 1:12 AM, Vignesh Raghavendra wrote:
> Hi Judith,
> 
> On 14/04/23 04:00, Judith Mendez wrote:
>> Judith Mendez (5):
>>    arm64: dts: ti: Add AM62x MCAN MAIN domain transceiver overlay
>>    arm64: defconfig: Enable MCAN driver
>>    dt-binding: can: m_can: Remove required interrupt attributes
>>    arm64: dts: ti: Enable multiple MCAN for AM62x in MCU MCAN overlay
>>    can: m_can: Add hrtimer to generate software interrupt
> 
> This is fine for RFC, but next time, please split DT and defconfig
> changes (1/5,2/5, and 4/5) to separate series as they have to go via
> arm64 tree.

Thanks, will do in the next respin.
  

Hello Marc,

On 4/19/2023 1:10 AM, Marc Kleine-Budde wrote:
> On 18.04.2023 11:15:35, Mendez, Judith wrote:
>> Hello Marc,
>>
>> On 4/14/2023 12:49 PM, Marc Kleine-Budde wrote:
>>> On 13.04.2023 17:30:46, Judith Mendez wrote:
>>>> On AM62x there is one MCAN in MAIN domain and two in MCU domain.
>>>> The MCANs in MCU domain were not enabled since there is no
>>>> hardware interrupt routed to A53 GIC interrupt controller.
>>>> Therefore A53 Linux cannot be interrupted by MCU MCANs.
>>>
>>> Is this a general hardware limitation, that effects all MCU domain
>>> peripherals? Is there a mailbox mechanism between the MCU and the MAIN
>>> domain, would it be possible to pass the IRQ with a small firmware on
>>> the MCU? Anyways, that's future optimization.
>>
>> This is a hardware limitation that affects AM62x SoC and has been carried
>> over to at least 1 other SoC. Using the MCU is an idea that we have juggled
>> around for a while, we will definitely keep it in mind for future
>> optimization. Thanks for your feedback.
> 
> Once you have a proper IRQ de-multiplexer, you can integrate it into the
> system with a DT change only. No need for changes in the m_can driver.
> 

Is this a recommendation for the current patch?

The reason I am asking is because adding firmware for the M4 to forward
a mailbox with the IRQ to the A53 sounds like a good idea and we have 
been juggling the idea, but it is not an ideal solution if customers are
using the M4 for other purposes like safety.

>>>> This solution instantiates a hrtimer with 1 ms polling interval
>>>> for a MCAN when there is no hardware interrupt. This hrtimer
>>>> generates a recurring software interrupt which allows to call the
>>>> isr. The isr will check if there is pending transaction by reading
>>>> a register and proceed normally if there is.
>>>>
>>>> On AM62x this series enables two MCU MCAN which will use the hrtimer
>>>> implementation. MCANs with hardware interrupt routed to A53 Linux
>>>> will continue to use the hardware interrupt as expected.
>>>>
>>>> Timer polling method was tested on both classic CAN and CAN-FD
>>>> at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
>>>> switching.
>>>>
>>>> Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
>>>> 1 MBPS timer polling interval is the better timer polling interval
>>>> since it has comparable latency to hardware interrupt with the worse
>>>> case being 1ms + CAN frame propagation time and CPU load is not
>>>> substantial. Latency can be improved further with less than 1 ms
>>>> polling intervals, howerver it is at the cost of CPU usage since CPU
>>>> load increases at 0.5 ms and lower polling periods than 1ms.
> 
> Have you seen my suggestion of the poll-interval?
> 
> Some Linux input drivers have the property poll-interval, would it make
> sense to ass this here too?

Looking at some examples, I do think we could implement this 
poll-interval attribute, then read in the driver and initialize the 
hrtimer based on this. I like the idea to submit as a future 
optimization patch, thanks!

regards,
Judith
  

On 19.04.2023 15:40:24, Mendez, Judith wrote:
> Hello Marc,
> 
> On 4/19/2023 1:10 AM, Marc Kleine-Budde wrote:
> > On 18.04.2023 11:15:35, Mendez, Judith wrote:
> > > Hello Marc,
> > > 
> > > On 4/14/2023 12:49 PM, Marc Kleine-Budde wrote:
> > > > On 13.04.2023 17:30:46, Judith Mendez wrote:
> > > > > On AM62x there is one MCAN in MAIN domain and two in MCU domain.
> > > > > The MCANs in MCU domain were not enabled since there is no
> > > > > hardware interrupt routed to A53 GIC interrupt controller.
> > > > > Therefore A53 Linux cannot be interrupted by MCU MCANs.
> > > > 
> > > > Is this a general hardware limitation, that effects all MCU domain
> > > > peripherals? Is there a mailbox mechanism between the MCU and the MAIN
> > > > domain, would it be possible to pass the IRQ with a small firmware on
> > > > the MCU? Anyways, that's future optimization.
> > > 
> > > This is a hardware limitation that affects AM62x SoC and has been carried
> > > over to at least 1 other SoC. Using the MCU is an idea that we have juggled
> > > around for a while, we will definitely keep it in mind for future
> > > optimization. Thanks for your feedback.
> > 
> > Once you have a proper IRQ de-multiplexer, you can integrate it into the
> > system with a DT change only. No need for changes in the m_can driver.
> > 
> 
> Is this a recommendation for the current patch?

It is a recommendation on how to get around the hardware limitation,
instead of falling back to polling.

> The reason I am asking is because adding firmware for the M4 to forward
> a mailbox with the IRQ to the A53 sounds like a good idea and we have been
> juggling the idea, but it is not an ideal solution if customers are
> using the M4 for other purposes like safety.

Of course, the feasibility of this approach depends on your system
design.

> > > > > This solution instantiates a hrtimer with 1 ms polling interval
> > > > > for a MCAN when there is no hardware interrupt. This hrtimer
> > > > > generates a recurring software interrupt which allows to call the
> > > > > isr. The isr will check if there is pending transaction by reading
> > > > > a register and proceed normally if there is.
> > > > > 
> > > > > On AM62x this series enables two MCU MCAN which will use the hrtimer
> > > > > implementation. MCANs with hardware interrupt routed to A53 Linux
> > > > > will continue to use the hardware interrupt as expected.
> > > > > 
> > > > > Timer polling method was tested on both classic CAN and CAN-FD
> > > > > at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
> > > > > switching.
> > > > > 
> > > > > Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
> > > > > 1 MBPS timer polling interval is the better timer polling interval
> > > > > since it has comparable latency to hardware interrupt with the worse
> > > > > case being 1ms + CAN frame propagation time and CPU load is not
> > > > > substantial. Latency can be improved further with less than 1 ms
> > > > > polling intervals, howerver it is at the cost of CPU usage since CPU
> > > > > load increases at 0.5 ms and lower polling periods than 1ms.
> > 
> > Have you seen my suggestion of the poll-interval?
> > 
> > Some Linux input drivers have the property poll-interval, would it make
> > sense to ass this here too?
> 
> Looking at some examples, I do think we could implement this poll-interval
> attribute, then read in the driver and initialize the hrtimer based on this.
> I like the idea to submit as a future optimization patch, thanks!

I would like to have the DT bindings in place, as handling legacy DT
without poll interval adds unnecessary complexity.

regards,
Marc
  

Hello Marc,

On 4/20/2023 4:36 AM, Marc Kleine-Budde wrote:
> On 19.04.2023 15:40:24, Mendez, Judith wrote:
>> Hello Marc,
>>
>> On 4/19/2023 1:10 AM, Marc Kleine-Budde wrote:
>>> On 18.04.2023 11:15:35, Mendez, Judith wrote:
>>>> Hello Marc,
>>>>
>>>> On 4/14/2023 12:49 PM, Marc Kleine-Budde wrote:
>>>>> On 13.04.2023 17:30:46, Judith Mendez wrote:
>>>>>> On AM62x there is one MCAN in MAIN domain and two in MCU domain.
>>>>>> The MCANs in MCU domain were not enabled since there is no
>>>>>> hardware interrupt routed to A53 GIC interrupt controller.
>>>>>> Therefore A53 Linux cannot be interrupted by MCU MCANs.
>>>>>
>>>>> Is this a general hardware limitation, that effects all MCU domain
>>>>> peripherals? Is there a mailbox mechanism between the MCU and the MAIN
>>>>> domain, would it be possible to pass the IRQ with a small firmware on
>>>>> the MCU? Anyways, that's future optimization.
>>>>
>>>> This is a hardware limitation that affects AM62x SoC and has been carried
>>>> over to at least 1 other SoC. Using the MCU is an idea that we have juggled
>>>> around for a while, we will definitely keep it in mind for future
>>>> optimization. Thanks for your feedback.
>>>
>>> Once you have a proper IRQ de-multiplexer, you can integrate it into the
>>> system with a DT change only. No need for changes in the m_can driver.
>>>
>>
>> Is this a recommendation for the current patch?
> 
> It is a recommendation on how to get around the hardware limitation,
> instead of falling back to polling.
> 
>> The reason I am asking is because adding firmware for the M4 to forward
>> a mailbox with the IRQ to the A53 sounds like a good idea and we have been
>> juggling the idea, but it is not an ideal solution if customers are
>> using the M4 for other purposes like safety.
> 
> Of course, the feasibility of this approach depends on your system
> design.

I understand your concern. Like mentioned, using the M4 approach may not 
be the best solution since some customers use the M4 for various reasons 
that could provide problems for this design.

I think the best way to go would be to enable polling + your suggestion 
of using poll-interval in device tree. If poll-interval is specified, 
then we can enable polling mode for MCAN.

>>>>>> This solution instantiates a hrtimer with 1 ms polling interval
>>>>>> for a MCAN when there is no hardware interrupt. This hrtimer
>>>>>> generates a recurring software interrupt which allows to call the
>>>>>> isr. The isr will check if there is pending transaction by reading
>>>>>> a register and proceed normally if there is.
>>>>>>
>>>>>> On AM62x this series enables two MCU MCAN which will use the hrtimer
>>>>>> implementation. MCANs with hardware interrupt routed to A53 Linux
>>>>>> will continue to use the hardware interrupt as expected.
>>>>>>
>>>>>> Timer polling method was tested on both classic CAN and CAN-FD
>>>>>> at 125 KBPS, 250 KBPS, 1 MBPS and 2.5 MBPS with 4 MBPS bitrate
>>>>>> switching.
>>>>>>
>>>>>> Letency and CPU load benchmarks were tested on 3x MCAN on AM62x.
>>>>>> 1 MBPS timer polling interval is the better timer polling interval
>>>>>> since it has comparable latency to hardware interrupt with the worse
>>>>>> case being 1ms + CAN frame propagation time and CPU load is not
>>>>>> substantial. Latency can be improved further with less than 1 ms
>>>>>> polling intervals, howerver it is at the cost of CPU usage since CPU
>>>>>> load increases at 0.5 ms and lower polling periods than 1ms.
>>>
>>> Have you seen my suggestion of the poll-interval?
>>>
>>> Some Linux input drivers have the property poll-interval, would it make
>>> sense to ass this here too?
>>
>> Looking at some examples, I do think we could implement this poll-interval
>> attribute, then read in the driver and initialize the hrtimer based on this.
>> I like the idea to submit as a future optimization patch, thanks!
> 
> I would like to have the DT bindings in place, as handling legacy DT
> without poll interval adds unnecessary complexity.

Understood, thanks so much for your feedback.

regards,
Judith