cdx: add MSI support for CDX bus

  Add CDX-MSI domain with gic-its domain as parent, to support MSI
for CDX devices. CDX devices allocate MSIs from the CDX domain.
Also, introduce APIs to alloc and free IRQs for CDX domain.

Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
Signed-off-by: Nikhil Agarwal <nikhil.agarwal@amd.com>
Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
Reviewed-by: Pieter Jansen van Vuuren <pieter.jansen-van-vuuren@amd.com>
---

This is a rework of CDX-MSI RFC sent at:
https://lore.kernel.org/lkml/20221014044049.2557085-8-nipun.gupta@amd.com/
The change was separated out from CDX bus series and has now been
validated on Versal-net platform.

 drivers/cdx/Kconfig                     |   1 +
 drivers/cdx/Makefile                    |   2 +-
 drivers/cdx/cdx.c                       |   6 +
 drivers/cdx/cdx.h                       |  10 ++
 drivers/cdx/cdx_msi.c                   | 153 ++++++++++++++++++++++++
 drivers/cdx/controller/cdx_controller.c |  23 ++++
 drivers/cdx/controller/mc_cdx_pcol.h    |  58 +++++++++
 drivers/cdx/controller/mcdi_functions.c |  19 +++
 drivers/cdx/controller/mcdi_functions.h |  20 ++++
 include/linux/cdx/cdx_bus.h             |  26 ++++
 kernel/irq/msi.c                        |   1 +
 11 files changed, 318 insertions(+), 1 deletion(-)
 create mode 100644 drivers/cdx/cdx_msi.c
  

On Mon, May 08 2023 at 19:39, Nipun Gupta wrote:
> Add CDX-MSI domain with gic-its domain as parent, to support MSI
> for CDX devices. CDX devices allocate MSIs from the CDX domain.
> Also, introduce APIs to alloc and free IRQs for CDX domain.

This lacks any information why this needs to have a separate irq domain
and what this is about. Changelogs need to be self explanatory and
providing a link to some RFC series which might have more information
does not cut it.

Just for the record. I complained about the useless changelog in that
RFC series already.

> Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> Signed-off-by: Nikhil Agarwal <nikhil.agarwal@amd.com>
> Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>

This Signed-off-by chain is broken. If this is intended to denote
co-authorship, then please follow:

  https://www.kernel.org/doc/html/latest/process/submitting-patches.html#when-to-use-acked-by-cc-and-co-developed-by

> +static struct irq_chip cdx_msi_irq_chip = {
> +	.name			= "CDX-MSI",
> +	.irq_mask		= irq_chip_mask_parent,
> +	.irq_unmask		= irq_chip_unmask_parent,
> +	.irq_eoi		= irq_chip_eoi_parent,
> +	.irq_set_affinity	= msi_domain_set_affinity,
> +	.irq_write_msi_msg	= cdx_msi_write_msg
> +};

The only real CDX specific functionality here is a CDX specific
irq_write_msi_msg() callback, right?

And I gave you a pointer how this should be handled, but instead of
helping this effort along you go off and implement it differently just
because. Sigh!

Thanks,

        tglx
  

> -----Original Message-----
> From: Thomas Gleixner <tglx@linutronix.de>
> Sent: Tuesday, May 9, 2023 1:32 PM
> To: Gupta, Nipun <Nipun.Gupta@amd.com>; gregkh@linuxfoundation.org;
> maz@kernel.org; jgg@ziepe.ca; linux-kernel@vger.kernel.org
> Cc: git (AMD-Xilinx) <git@amd.com>; Anand, Harpreet
> <harpreet.anand@amd.com>; Jansen Van Vuuren, Pieter <pieter.jansen-
> van-vuuren@amd.com>; Agarwal, Nikhil <nikhil.agarwal@amd.com>; Simek,
> Michal <michal.simek@amd.com>; Gangurde, Abhijit
> <abhijit.gangurde@amd.com>; Cascon, Pablo <pablo.cascon@amd.com>;
> Gupta, Nipun <Nipun.Gupta@amd.com>
> Subject: Re: [PATCH] cdx: add MSI support for CDX bus
> 
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
> 
> 
> On Mon, May 08 2023 at 19:39, Nipun Gupta wrote:
> > Add CDX-MSI domain with gic-its domain as parent, to support MSI
> > for CDX devices. CDX devices allocate MSIs from the CDX domain.
> > Also, introduce APIs to alloc and free IRQs for CDX domain.
> 
> This lacks any information why this needs to have a separate irq domain
> and what this is about. Changelogs need to be self explanatory and
> providing a link to some RFC series which might have more information
> does not cut it.
> 
> Just for the record. I complained about the useless changelog in that
> RFC series already.
> 
> > Signed-off-by: Nipun Gupta <nipun.gupta@amd.com>
> > Signed-off-by: Nikhil Agarwal <nikhil.agarwal@amd.com>
> > Signed-off-by: Abhijit Gangurde <abhijit.gangurde@amd.com>
> 
> This Signed-off-by chain is broken. If this is intended to denote
> co-authorship, then please follow:
> 
>   https://www.kernel.org/doc/html/latest/process/submitting-
> patches.html#when-to-use-acked-by-cc-and-co-developed-by
> 
> > +static struct irq_chip cdx_msi_irq_chip = {
> > +     .name                   = "CDX-MSI",
> > +     .irq_mask               = irq_chip_mask_parent,
> > +     .irq_unmask             = irq_chip_unmask_parent,
> > +     .irq_eoi                = irq_chip_eoi_parent,
> > +     .irq_set_affinity       = msi_domain_set_affinity,
> > +     .irq_write_msi_msg      = cdx_msi_write_msg
> > +};
> 
> The only real CDX specific functionality here is a CDX specific
> irq_write_msi_msg() callback, right?
> 
> And I gave you a pointer how this should be handled, but instead of
> helping this effort along you go off and implement it differently just
> because. Sigh!

Hi Thomas,

As you rightly mentioned the irq_chip has only irq_write_msi_msg() as
callback, but there is also cdx_msi_prepare() in msi_domain_ops which
needs to fetch device ID from CDX device, due to which we are currently
using separate CDX domain.

IIUC, as per your suggestion we should have CDX bus token added into 
its_init_dev_msi_info() of
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/tree/drivers/irqchip/irq-gic-v3-its-msi-parent.c?h=devmsi-arm,
and register CDX specific 'msi_prepare' here; so that we can use
msi_create_device_irq_domain() to create a per device domain?

Thanks,
Nipun

> 
> Thanks,
> 
>         tglx
  

Nipun!

On Tue, May 09 2023 at 11:06, Nipun Gupta wrote:
>> -----Original Message-----
>> From: Thomas Gleixner <tglx@linutronix.de>
>> Sent: Tuesday, May 9, 2023 1:32 PM
>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; gregkh@linuxfoundation.org;
>> maz@kernel.org; jgg@ziepe.ca; linux-kernel@vger.kernel.org

Can you please fix your mail client to not copy half of the mail header
into your reply?

>> Caution: This message originated from an External Source. Use proper
>> caution when opening attachments, clicking links, or responding.

That's also relevant information for me, right?
 
>> The only real CDX specific functionality here is a CDX specific
>> irq_write_msi_msg() callback, right?
>> 
>> And I gave you a pointer how this should be handled, but instead of
>> helping this effort along you go off and implement it differently just
>> because. Sigh!
>
> As you rightly mentioned the irq_chip has only irq_write_msi_msg() as
> callback, but there is also cdx_msi_prepare() in msi_domain_ops which
> needs to fetch device ID from CDX device, due to which we are currently
> using separate CDX domain.

Sure. But where is that information in the changelog?

> IIUC, as per your suggestion we should have CDX bus token added into 
> its_init_dev_msi_info() of
> https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/tree/drivers/irqchip/irq-gic-v3-its-msi-parent.c?h=devmsi-arm,
> and register CDX specific 'msi_prepare' here; so that we can use
> msi_create_device_irq_domain() to create a per device domain?

Correct.

I'm not insisting on that, but you could at least have had the courtesy
of responding to my review reply and explain to me why you want to solve
it differently and why my suggestion is not the right solution.

Alternatively you could have added that information in the changelog or
cover letter.

So in summary you ignored _all_ review comments I made, went off and did
something different and provided a slightly different useless changelog
with the extra add on of a broken Signed-off-by chain.

Feel free to ignore my reviews and the documentation which we put out
there to make collaboration feasible for both sides, but please don't be
upset when I ignore you and your patches in return.

Thanks,

        tglx
  

On 5/10/2023 3:31 AM, Thomas Gleixner wrote:
> 
> Nipun!
> 
> On Tue, May 09 2023 at 11:06, Nipun Gupta wrote:
>>> -----Original Message-----
>>> From: Thomas Gleixner <tglx@linutronix.de>
>>> Sent: Tuesday, May 9, 2023 1:32 PM
>>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; gregkh@linuxfoundation.org;
>>> maz@kernel.org; jgg@ziepe.ca; linux-kernel@vger.kernel.org
> 
> Can you please fix your mail client to not copy half of the mail header
> into your reply?

Sure. Got it fixed.

> 
>>> Caution: This message originated from an External Source. Use proper
>>> caution when opening attachments, clicking links, or responding.
> 
> That's also relevant information for me, right?

Sorry to submit with this text, have already contacted concerned 
internal team regarding removal of this text. Have removed it manually 
for now.

> 
>>> The only real CDX specific functionality here is a CDX specific
>>> irq_write_msi_msg() callback, right?
>>>
>>> And I gave you a pointer how this should be handled, but instead of
>>> helping this effort along you go off and implement it differently just
>>> because. Sigh!
>>
>> As you rightly mentioned the irq_chip has only irq_write_msi_msg() as
>> callback, but there is also cdx_msi_prepare() in msi_domain_ops which
>> needs to fetch device ID from CDX device, due to which we are currently
>> using separate CDX domain.
> 
> Sure. But where is that information in the changelog?
> 
>> IIUC, as per your suggestion we should have CDX bus token added into
>> its_init_dev_msi_info() of
>> https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/tree/drivers/irqchip/irq-gic-v3-its-msi-parent.c?h=devmsi-arm,
>> and register CDX specific 'msi_prepare' here; so that we can use
>> msi_create_device_irq_domain() to create a per device domain?
> 
> Correct.
> 
> I'm not insisting on that, but you could at least have had the courtesy
> of responding to my review reply and explain to me why you want to solve
> it differently and why my suggestion is not the right solution.
> 
> Alternatively you could have added that information in the changelog or
> cover letter.
> 
> So in summary you ignored _all_ review comments I made, went off and did
> something different and provided a slightly different useless changelog
> with the extra add on of a broken Signed-off-by chain.
> 
> Feel free to ignore my reviews and the documentation which we put out
> there to make collaboration feasible for both sides, but please don't be
> upset when I ignore you and your patches in return.

Sincere apology for not responding to the earlier comments. Intention 
was never to ignore the review comments. Appreciate your vast changes 
regarding the MSI, and the patch series you shared took time to 
understand (provided other things as well), and it was quite late to 
reply. I understand that even in this case atleast I should have added 
this as part of the cover-letter.

IMHO, use-case for MSI in CDX subsystem is a bit different from per 
device MSI domain. Here we are trying to create a domain per CDX 
controller which is attached to a MSI controller, and all devices on a 
particular CDX controller will have same mechanism of write MSI message. 
Also, the current CDX controller that we have added has a different 
mechanism for MSI prepare (it gets requester ID from firmware).

In your opinion is there any advantage in moving to a per device domain 
for CDX devices? We can definitely rethink the implementation of MSI in 
CDX subsystem.

Thanks,
Nipun

> 
> Thanks,
> 
>          tglx
> 
>
  

Nipun!

On Wed, May 10 2023 at 19:34, Nipun Gupta wrote:
> On 5/10/2023 3:31 AM, Thomas Gleixner wrote:
>> I'm not insisting on that, but you could at least have had the courtesy
>> of responding to my review reply and explain to me why you want to solve
>> it differently and why my suggestion is not the right solution.
>> 
>> Alternatively you could have added that information in the changelog or
>> cover letter.
>> 
>> So in summary you ignored _all_ review comments I made, went off and did
>> something different and provided a slightly different useless changelog
>> with the extra add on of a broken Signed-off-by chain.
>> 
>> Feel free to ignore my reviews and the documentation which we put out
>> there to make collaboration feasible for both sides, but please don't be
>> upset when I ignore you and your patches in return.
>
> Sincere apology for not responding to the earlier comments. Intention 
> was never to ignore the review comments. Appreciate your vast changes 
> regarding the MSI, and the patch series you shared took time to 
> understand (provided other things as well), and it was quite late to 
> reply. I understand that even in this case atleast I should have added 
> this as part of the cover-letter.

Fair enough. All settled.

> IMHO, use-case for MSI in CDX subsystem is a bit different from per 
> device MSI domain. Here we are trying to create a domain per CDX 
> controller which is attached to a MSI controller, and all devices on a 
> particular CDX controller will have same mechanism of write MSI
> message.

That was exactly the same assumption which PCI/MSI and other MSI
implementations made. It turned out to be the wrong abstraction.

CDX is not any different than PCI. The actual "interrupt chip" is not
part of the bus, it's part of the device and pretending that it is a bus
specific thing is just running in to the same cul-de-sac sooner than
later.

> Also, the current CDX controller that we have added has a different 
> mechanism for MSI prepare (it gets requester ID from firmware).

That's not an argument, that's just an implementation detail.

> In your opinion is there any advantage in moving to a per device domain 
> for CDX devices? We can definitely rethink the implementation of MSI in 
> CDX subsystem.

See above.

While talking about implementation and design. I actually got curious
and looked at CDX because I was amazed about the gazillion indirections
in that msi_write_msg() callback.

So this ends up doing:

   cdx->ops->dev_configure(cdx, ...)
     cdx_configure_device()
       cdx_mcdi_write_msi()
         cdx_mcdi_rpc_async()
           kmalloc()                            <- FAIL #1
           cdx_mcdi_rpc_async_internal()
              queue_work()                      <- FAIL #2

#1) That kmalloc() uses GFP_ATOMIC, but this is invoked deep in the guts
    of interrupt handling with locks held and interrupts disabled.

    Aside of the fact that this breaks on PREEMPT_RT, such allocations
    are generally frowned upon. As a consequence the kref_put()s in the
    error paths of cdx_mcdi_rpc_async_internal() will blow up on RT
    too.

    I know that Xilinx stated publicly that they don't support RT, but
    RT is not that far out to be supported in mainline and aside of that
    I know for sure that quite a lot of Xilinx customers use PREEMPT_RT
    nevertheless.

#2) That's actually the worse part of it and completely broken versus
    device setup

    probe()
      cdx_msi_domain_alloc_irqs()
      ...
      request_irq() {
        ...
        irq_activate()
          irq_chip_write_msi_msg()
            ...
            queue_work()
          ...
      }

      enable_irq_in_device()
      
        <- device raises interrupt and eventually uses an uninitialized
           MSI message because the scheduled work has not yet completed.

    That's going to be a nightmare to debug and it's going to happen
    once in a blue moon out in the field.

The interrupt subsystem already can handle update mechanisms which
require sleepable context:

   irq_bus_lock() and irq_bus_sync_unlock() irqchip callbacks

They were initially implemented to deal with interrupt chips which are
configured via I2C, SPI etc. 

How does that work?

On entry to interrupt management functions the sequence is:

    if (desc->irq_data.chip->irq_bus_lock)
       desc->irq_data.chip->irq_bus_lock(...)
    raw_spin_lock_irq(&desc->lock);

and on exit:

    raw_spin_unlock_irq(&desc->lock);
    if (desc->irq_data.chip->irq_bus_sync_unlock)
       desc->irq_data.chip->irq_bus_sync_unlock(...)

irq_bus_lock() usually just acquires a mutex.

The other irqchip callbacks just cache the relevant information, but do
not execute the bus transaction because that is not possible with
desc->lock held.

In the irq_bus_sync_unlock() they execute the bus transaction with the
cached information before dropping the mutex.

So you can solve #1 and #2 with that. Your msi_write_msg() callback will
just save the message and set some internal flag that it needs to be
written out in the irq_bus_sync_unlock() callback.

See?

IIRC, there is a gap vs. interrupt affinity setting from user space,
which is irrelevant for I2C, SPI etc. configured interrupt chips as they
raise interrupt via an SoC interrupt pin and that's the entity which
does the affinity management w/o requiring I2C/SPI. IIRC I posted a
patch snippet to that effect in one of those lengthy PCI/MSI/IMS threads
because that is also required for MSI storage which happens to be in
queue memory and needs to be synchronized via some command channel. But
I can't be bothered to search for it as it's a no-brainer to fix that
up.

Thanks,

        tglx
  

Nipun!

On Thu, May 11 2023 at 00:29, Thomas Gleixner wrote:
> On Wed, May 10 2023 at 19:34, Nipun Gupta wrote:
> #2) That's actually the worse part of it and completely broken versus
>     device setup
>
>     probe()
>       cdx_msi_domain_alloc_irqs()
>       ...
>       request_irq() {
>         ...
>         irq_activate()
>           irq_chip_write_msi_msg()
>             ...
>             queue_work()
>           ...
>       }
>
>       enable_irq_in_device()
>       
>         <- device raises interrupt and eventually uses an uninitialized
>            MSI message because the scheduled work has not yet completed.
>
>     That's going to be a nightmare to debug and it's going to happen
>     once in a blue moon out in the field.

Here is another failure scenario:

cpu_down()
  // No scheduling possible as this happens in stomp_machine() context
  __cpu_disable()
    // Point of no return
    set_cpu_online(cpu, false);
    irq_migrate_all_off_this_cpu()
      ...
      set_affinity()

That works with your current implementation by some definition of works
because you schedule the affinity change async.

But what makes sure that this takes effect _before_ the CPU goes into
lala land and cannot respond to an interrupt anymore?

Nothing, other than pure luck, which is not really a solid engineering
principle.

While the regular operations can be fixed via the bus_lock mechanics,
this one falls flat on its nose because in that context this can't
schedule anymore so acquiring a mutex or waiting for some async muck to
complete is a no-no.

Not the end of the world though. There is a way to handle that
gracefully and we need that for the PCI/IMS implementations which will
do that via command queues too.

irq_migrate_all_off_this_cpu() is invoked from stomp_machine() context
with interrupts disabled because the current x86 interrupt management
hardware trainwreck requires it at least in the case that interrupt
remapping is not available.

Of course that got copied to all other architectures... Whether they
really require it or not from a hardware POV is a different
story. Though they all rely on the stomp_machine() context which
prevents concurrent modifications.

So from a historical POV all of this makes sense to some extent, but
that does not prevent us to fix this and make it an two stage process
which can actually handle both worlds.

I'm way too tired to think that through, but you get the idea and you
are welcome to beat me to it.

Thanks,

        tglx
  

On 5/11/2023 3:59 AM, Thomas Gleixner wrote:
> 
> Nipun!
> 
> On Wed, May 10 2023 at 19:34, Nipun Gupta wrote:
>> On 5/10/2023 3:31 AM, Thomas Gleixner wrote:
>>> I'm not insisting on that, but you could at least have had the courtesy
>>> of responding to my review reply and explain to me why you want to solve
>>> it differently and why my suggestion is not the right solution.
>>>
>>> Alternatively you could have added that information in the changelog or
>>> cover letter.
>>>
>>> So in summary you ignored _all_ review comments I made, went off and did
>>> something different and provided a slightly different useless changelog
>>> with the extra add on of a broken Signed-off-by chain.
>>>
>>> Feel free to ignore my reviews and the documentation which we put out
>>> there to make collaboration feasible for both sides, but please don't be
>>> upset when I ignore you and your patches in return.
>>
>> Sincere apology for not responding to the earlier comments. Intention
>> was never to ignore the review comments. Appreciate your vast changes
>> regarding the MSI, and the patch series you shared took time to
>> understand (provided other things as well), and it was quite late to
>> reply. I understand that even in this case atleast I should have added
>> this as part of the cover-letter.
> 
> Fair enough. All settled.
> 
>> IMHO, use-case for MSI in CDX subsystem is a bit different from per
>> device MSI domain. Here we are trying to create a domain per CDX
>> controller which is attached to a MSI controller, and all devices on a
>> particular CDX controller will have same mechanism of write MSI
>> message.
> 
> That was exactly the same assumption which PCI/MSI and other MSI
> implementations made. It turned out to be the wrong abstraction.
> 
> CDX is not any different than PCI. The actual "interrupt chip" is not
> part of the bus, it's part of the device and pretending that it is a bus
> specific thing is just running in to the same cul-de-sac sooner than
> later.

I understand your viewpoint, but would state that CDX bus is somewhat 
different than PCI in the sense that firmware is a controller for
all the devices and their configuration. CDX bus controller sends all 
the write_msi_msg commands to firmware running on RPU over the RPmsg and 
it is the firmware which interfaces with actual devices to pass this 
information to devices in a way agreed between firmware and device. The 
only way to pass MSI information to device is via firmware and CDX bus 
controller is only entity which can communicate with the firmware for this.

> 
>> Also, the current CDX controller that we have added has a different
>> mechanism for MSI prepare (it gets requester ID from firmware).
> 
> That's not an argument, that's just an implementation detail.
> 
>> In your opinion is there any advantage in moving to a per device domain
>> for CDX devices? We can definitely rethink the implementation of MSI in
>> CDX subsystem.
> 
> See above.
> 
> While talking about implementation and design. I actually got curious
> and looked at CDX because I was amazed about the gazillion indirections
> in that msi_write_msg() callback.
> 
> So this ends up doing:
> 
>     cdx->ops->dev_configure(cdx, ...)
>       cdx_configure_device()
>         cdx_mcdi_write_msi()
>           cdx_mcdi_rpc_async()
>             kmalloc()                            <- FAIL #1
>             cdx_mcdi_rpc_async_internal()
>                queue_work()                      <- FAIL #2
> 
> #1) That kmalloc() uses GFP_ATOMIC, but this is invoked deep in the guts
>      of interrupt handling with locks held and interrupts disabled.
> 
>      Aside of the fact that this breaks on PREEMPT_RT, such allocations
>      are generally frowned upon. As a consequence the kref_put()s in the
>      error paths of cdx_mcdi_rpc_async_internal() will blow up on RT
>      too.
> 
>      I know that Xilinx stated publicly that they don't support RT, but
>      RT is not that far out to be supported in mainline and aside of that
>      I know for sure that quite a lot of Xilinx customers use PREEMPT_RT
>      nevertheless.
> 
> #2) That's actually the worse part of it and completely broken versus
>      device setup
> 
>      probe()
>        cdx_msi_domain_alloc_irqs()
>        ...
>        request_irq() {
>          ...
>          irq_activate()
>            irq_chip_write_msi_msg()
>              ...
>              queue_work()
>            ...
>        }
> 
>        enable_irq_in_device()
> 
>          <- device raises interrupt and eventually uses an uninitialized
>             MSI message because the scheduled work has not yet completed.
> 
>      That's going to be a nightmare to debug and it's going to happen
>      once in a blue moon out in the field.
> 
> The interrupt subsystem already can handle update mechanisms which
> require sleepable context:
> 
>     irq_bus_lock() and irq_bus_sync_unlock() irqchip callbacks
> 
> They were initially implemented to deal with interrupt chips which are
> configured via I2C, SPI etc.
> 
> How does that work?
> 
> On entry to interrupt management functions the sequence is:
> 
>      if (desc->irq_data.chip->irq_bus_lock)
>         desc->irq_data.chip->irq_bus_lock(...)
>      raw_spin_lock_irq(&desc->lock);
> 
> and on exit:
> 
>      raw_spin_unlock_irq(&desc->lock);
>      if (desc->irq_data.chip->irq_bus_sync_unlock)
>         desc->irq_data.chip->irq_bus_sync_unlock(...)
> 
> irq_bus_lock() usually just acquires a mutex.
> 
> The other irqchip callbacks just cache the relevant information, but do
> not execute the bus transaction because that is not possible with
> desc->lock held.
> 
> In the irq_bus_sync_unlock() they execute the bus transaction with the
> cached information before dropping the mutex.
> 
> So you can solve #1 and #2 with that. Your msi_write_msg() callback will
> just save the message and set some internal flag that it needs to be
> written out in the irq_bus_sync_unlock() callback.
> 
> See?
> 
> IIRC, there is a gap vs. interrupt affinity setting from user space,
> which is irrelevant for I2C, SPI etc. configured interrupt chips as they
> raise interrupt via an SoC interrupt pin and that's the entity which
> does the affinity management w/o requiring I2C/SPI. IIRC I posted a
> patch snippet to that effect in one of those lengthy PCI/MSI/IMS threads
> because that is also required for MSI storage which happens to be in
> queue memory and needs to be synchronized via some command channel. But
> I can't be bothered to search for it as it's a no-brainer to fix that
> up.

Thanks for this analysis and pointing the hidden crucial issues with the 
implementation. These needs to be fixed.

As per your suggestion, we can add Firmware interaction code in the 
irq_bus_sync_xx APIs. Another option is to change the 
cdx_mcdi_rpc_async() API to atomic synchronous API. We are evaluating 
both the solutions and will update the implementation accordingly.

Thanks,
Nipun

> 
> Thanks,
> 
>          tglx
  

Nipun!

On Fri, May 12 2023 at 19:50, Nipun Gupta wrote:
> On 5/11/2023 3:59 AM, Thomas Gleixner wrote:
>> CDX is not any different than PCI. The actual "interrupt chip" is not
>> part of the bus, it's part of the device and pretending that it is a bus
>> specific thing is just running in to the same cul-de-sac sooner than
>> later.
>
> I understand your viewpoint, but would state that CDX bus is somewhat 
> different than PCI in the sense that firmware is a controller for
> all the devices and their configuration. CDX bus controller sends all 
> the write_msi_msg commands to firmware running on RPU over the RPmsg and 
> it is the firmware which interfaces with actual devices to pass this 
> information to devices in a way agreed between firmware and device. The 
> only way to pass MSI information to device is via firmware and CDX bus 
> controller is only entity which can communicate with the firmware for
> this.

Fair enough, but we wouldn't had this dicussion if the above information
would have been part of the changelog. See?

>> IIRC, there is a gap vs. interrupt affinity setting from user space,
>> which is irrelevant for I2C, SPI etc. configured interrupt chips as they
>> raise interrupt via an SoC interrupt pin and that's the entity which
>> does the affinity management w/o requiring I2C/SPI. IIRC I posted a
>> patch snippet to that effect in one of those lengthy PCI/MSI/IMS threads
>> because that is also required for MSI storage which happens to be in
>> queue memory and needs to be synchronized via some command channel. But
>> I can't be bothered to search for it as it's a no-brainer to fix that
>> up.
>
> Thanks for this analysis and pointing the hidden crucial issues with the 
> implementation. These needs to be fixed.
>
> As per your suggestion, we can add Firmware interaction code in the
> irq_bus_sync_xx APIs. Another option is to change the
> cdx_mcdi_rpc_async() API to atomic synchronous API.

I'm not a great fan of that. Depending on how long this update takes the
CPU will busy wait for it to complete with interrupts disabled and locks
held.

> We are evaluating both the solutions and will update the
> implementation accordingly.

Thanks,
 
         tglx
  

On 5/12/2023 11:45 PM, Thomas Gleixner wrote:
> 
> Nipun!
> 
> On Fri, May 12 2023 at 19:50, Nipun Gupta wrote:
>> On 5/11/2023 3:59 AM, Thomas Gleixner wrote:
>>> CDX is not any different than PCI. The actual "interrupt chip" is not
>>> part of the bus, it's part of the device and pretending that it is a bus
>>> specific thing is just running in to the same cul-de-sac sooner than
>>> later.
>>
>> I understand your viewpoint, but would state that CDX bus is somewhat
>> different than PCI in the sense that firmware is a controller for
>> all the devices and their configuration. CDX bus controller sends all
>> the write_msi_msg commands to firmware running on RPU over the RPmsg and
>> it is the firmware which interfaces with actual devices to pass this
>> information to devices in a way agreed between firmware and device. The
>> only way to pass MSI information to device is via firmware and CDX bus
>> controller is only entity which can communicate with the firmware for
>> this.
> 
> Fair enough, but we wouldn't had this dicussion if the above information
> would have been part of the changelog. See?

Yes, you are correct, we will make sure to update and add more info in 
the changelog in next spin.

> 
>>> IIRC, there is a gap vs. interrupt affinity setting from user space,
>>> which is irrelevant for I2C, SPI etc. configured interrupt chips as they
>>> raise interrupt via an SoC interrupt pin and that's the entity which
>>> does the affinity management w/o requiring I2C/SPI. IIRC I posted a
>>> patch snippet to that effect in one of those lengthy PCI/MSI/IMS threads
>>> because that is also required for MSI storage which happens to be in
>>> queue memory and needs to be synchronized via some command channel. But
>>> I can't be bothered to search for it as it's a no-brainer to fix that
>>> up.
>>
>> Thanks for this analysis and pointing the hidden crucial issues with the
>> implementation. These needs to be fixed.
>>
>> As per your suggestion, we can add Firmware interaction code in the
>> irq_bus_sync_xx APIs. Another option is to change the
>> cdx_mcdi_rpc_async() API to atomic synchronous API.
> 
> I'm not a great fan of that. Depending on how long this update takes the
> CPU will busy wait for it to complete with interrupts disabled and locks
> held.

Agree. we are also inclined towards using irq_bus_sync_xx APIs. This 
would definitely solve the issue (#1 and #2) for the setup_irq which you 
mentioned.

For MSI affinity, since GIC-ITS always return IRQ_SET_MASK_OK_DONE, the 
irq_chip_write_msi_msg does not get called.

msi_domain_set_affinity(...)
     ret = parent->chip->irq_set_affinity(...);
     // For GIC ITS it always return IRQ_SET_MASK_OK_DONE
     if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
         irq_chip_write_msi_msg(...);
     }
Since CDX bus is specific to ARM and uses GIC ITS, it seems we do not 
need to do anything here. Can you please share your opinion on this?

Thanks,
Nipun

> 
>> We are evaluating both the solutions and will update the
>> implementation accordingly.
> 
> Thanks,
> 
>           tglx
  

Nipun!

On Mon, May 15 2023 at 18:39, Nipun Gupta wrote:
> On 5/12/2023 11:45 PM, Thomas Gleixner wrote:
>> On Fri, May 12 2023 at 19:50, Nipun Gupta wrote:
>>>
>>> As per your suggestion, we can add Firmware interaction code in the
>>> irq_bus_sync_xx APIs. Another option is to change the
>>> cdx_mcdi_rpc_async() API to atomic synchronous API.
>> 
>> I'm not a great fan of that. Depending on how long this update takes the
>> CPU will busy wait for it to complete with interrupts disabled and locks
>> held.
>
> Agree. we are also inclined towards using irq_bus_sync_xx APIs. This 
> would definitely solve the issue (#1 and #2) for the setup_irq which you 
> mentioned.
>
> For MSI affinity, since GIC-ITS always return IRQ_SET_MASK_OK_DONE, the 
> irq_chip_write_msi_msg does not get called.
>
> msi_domain_set_affinity(...)
>      ret = parent->chip->irq_set_affinity(...);
>      // For GIC ITS it always return IRQ_SET_MASK_OK_DONE
>      if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
>          irq_chip_write_msi_msg(...);
>      }
> Since CDX bus is specific to ARM and uses GIC ITS, it seems we do not 
> need to do anything here. Can you please share your opinion on this?

That makes sense.

Thanks,

        tglx