[v5,0/5] PCI: endpoint: Rework the EPC to EPF notification

  Hello,

During the review of the patch that fixes DBI access in PCI EP, Rob
suggested [1] using a fixed interface for passing the events from EPC to
EPF instead of the in-kernel notifiers.

This series introduces a simple callback based mechanism for passing the
events from EPC to EPF. This interface is chosen for satisfying the below
requirements:

1. The notification has to reach the EPF drivers without any additional
latency.
2. The context of the caller (EPC) needs to be preserved while passing the
notifications.

With the existing notifier mechanism, the 1st case can be satisfied since
notifiers aren't adding any huge overhead. But the 2nd case is clearly not
satisfied, because the current atomic notifiers forces the EPF
notification context to be atomic even though the caller (EPC) may not be
in atomic context. In the notification function, the EPF drivers are
required to call several EPC APIs that might sleep and this triggers a
sleeping in atomic bug during runtime.

The above issue could be fixed by using a blocking notifier instead of
atomic, but that proposal was not accepted either [2].

So instead of working around the issues within the notifiers, let's get rid
of it and use the callback mechanism.

NOTE: DRA7xx and TEGRA194 drivers are only compile tested. Testing this series
on the real platforms is greatly appreciated.

Thanks,
Mani

[1] https://lore.kernel.org/all/20220802072426.GA2494@thinkpad/T/#mfa3a5b3a9694798a562c36b228f595b6a571477d
[2] https://lore.kernel.org/all/20220228055240.24774-1-manivannan.sadhasivam@linaro.org

Changes in v5:

* Collected review tag from Vidya
* Fixed the issue reported by Kbot regarding missing declaration

Changes in v4:

* Added check for the presence of event_ops before involing the callbacks (Kishon)
* Added return with IRQ_WAKE_THREAD when link_up event is found in the hard irq
  handler of tegra194 driver (Vidya)
* Collected review tags

Changes in v3:

* As Kishon spotted, fixed the DRA7xx driver and also the TEGRA194 driver to
  call the LINK_UP callback in threaded IRQ handler.

Changes in v2:

* Introduced a new "list_lock" for protecting the epc->pci_epf list and
  used it in the callback mechanism.

Manivannan Sadhasivam (5):
  PCI: dra7xx: Use threaded IRQ handler for "dra7xx-pcie-main" IRQ
  PCI: tegra194: Move dw_pcie_ep_linkup() to threaded IRQ handler
  PCI: endpoint: Use a separate lock for protecting epc->pci_epf list
  PCI: endpoint: Use callback mechanism for passing events from EPC to
    EPF
  PCI: endpoint: Use link_up() callback in place of LINK_UP notifier

 drivers/pci/controller/dwc/pci-dra7xx.c       |  2 +-
 drivers/pci/controller/dwc/pcie-tegra194.c    |  9 ++++-
 drivers/pci/endpoint/functions/pci-epf-test.c | 38 ++++++-------------
 drivers/pci/endpoint/pci-epc-core.c           | 32 ++++++++++++----
 include/linux/pci-epc.h                       | 10 +----
 include/linux/pci-epf.h                       | 19 ++++++----
 6 files changed, 59 insertions(+), 51 deletions(-)
  

On Tue, Jan 24, 2023 at 12:41:53PM +0530, Manivannan Sadhasivam wrote:
> Hello,
> 
> During the review of the patch that fixes DBI access in PCI EP, Rob
> suggested [1] using a fixed interface for passing the events from EPC to
> EPF instead of the in-kernel notifiers.
> 
> This series introduces a simple callback based mechanism for passing the
> events from EPC to EPF. This interface is chosen for satisfying the below
> requirements:
> 
> 1. The notification has to reach the EPF drivers without any additional
> latency.
> 2. The context of the caller (EPC) needs to be preserved while passing the
> notifications.
> 
> With the existing notifier mechanism, the 1st case can be satisfied since
> notifiers aren't adding any huge overhead. But the 2nd case is clearly not
> satisfied, because the current atomic notifiers forces the EPF
> notification context to be atomic even though the caller (EPC) may not be
> in atomic context. In the notification function, the EPF drivers are
> required to call several EPC APIs that might sleep and this triggers a
> sleeping in atomic bug during runtime.
> 
> The above issue could be fixed by using a blocking notifier instead of
> atomic, but that proposal was not accepted either [2].
> 
> So instead of working around the issues within the notifiers, let's get rid
> of it and use the callback mechanism.
> 
> NOTE: DRA7xx and TEGRA194 drivers are only compile tested. Testing this series
> on the real platforms is greatly appreciated.
> 

Lorenzo, all patches in this series got review tags. Can you please merge now?

Thanks,
Mani

> Thanks,
> Mani
> 
> [1] https://lore.kernel.org/all/20220802072426.GA2494@thinkpad/T/#mfa3a5b3a9694798a562c36b228f595b6a571477d
> [2] https://lore.kernel.org/all/20220228055240.24774-1-manivannan.sadhasivam@linaro.org
> 
> Changes in v5:
> 
> * Collected review tag from Vidya
> * Fixed the issue reported by Kbot regarding missing declaration
> 
> Changes in v4:
> 
> * Added check for the presence of event_ops before involing the callbacks (Kishon)
> * Added return with IRQ_WAKE_THREAD when link_up event is found in the hard irq
>   handler of tegra194 driver (Vidya)
> * Collected review tags
> 
> Changes in v3:
> 
> * As Kishon spotted, fixed the DRA7xx driver and also the TEGRA194 driver to
>   call the LINK_UP callback in threaded IRQ handler.
> 
> Changes in v2:
> 
> * Introduced a new "list_lock" for protecting the epc->pci_epf list and
>   used it in the callback mechanism.
> 
> Manivannan Sadhasivam (5):
>   PCI: dra7xx: Use threaded IRQ handler for "dra7xx-pcie-main" IRQ
>   PCI: tegra194: Move dw_pcie_ep_linkup() to threaded IRQ handler
>   PCI: endpoint: Use a separate lock for protecting epc->pci_epf list
>   PCI: endpoint: Use callback mechanism for passing events from EPC to
>     EPF
>   PCI: endpoint: Use link_up() callback in place of LINK_UP notifier
> 
>  drivers/pci/controller/dwc/pci-dra7xx.c       |  2 +-
>  drivers/pci/controller/dwc/pcie-tegra194.c    |  9 ++++-
>  drivers/pci/endpoint/functions/pci-epf-test.c | 38 ++++++-------------
>  drivers/pci/endpoint/pci-epc-core.c           | 32 ++++++++++++----
>  include/linux/pci-epc.h                       | 10 +----
>  include/linux/pci-epf.h                       | 19 ++++++----
>  6 files changed, 59 insertions(+), 51 deletions(-)
> 
> -- 
> 2.25.1
>
  

On Tue, Jan 24, 2023 at 12:46:11PM +0530, Manivannan Sadhasivam wrote:
> On Tue, Jan 24, 2023 at 12:41:53PM +0530, Manivannan Sadhasivam wrote:
> > Hello,
> > 
> > During the review of the patch that fixes DBI access in PCI EP, Rob
> > suggested [1] using a fixed interface for passing the events from EPC to
> > EPF instead of the in-kernel notifiers.
> > 
> > This series introduces a simple callback based mechanism for passing the
> > events from EPC to EPF. This interface is chosen for satisfying the below
> > requirements:
> > 
> > 1. The notification has to reach the EPF drivers without any additional
> > latency.
> > 2. The context of the caller (EPC) needs to be preserved while passing the
> > notifications.
> > 
> > With the existing notifier mechanism, the 1st case can be satisfied since
> > notifiers aren't adding any huge overhead. But the 2nd case is clearly not
> > satisfied, because the current atomic notifiers forces the EPF
> > notification context to be atomic even though the caller (EPC) may not be
> > in atomic context. In the notification function, the EPF drivers are
> > required to call several EPC APIs that might sleep and this triggers a
> > sleeping in atomic bug during runtime.
> > 
> > The above issue could be fixed by using a blocking notifier instead of
> > atomic, but that proposal was not accepted either [2].
> > 
> > So instead of working around the issues within the notifiers, let's get rid
> > of it and use the callback mechanism.
> > 
> > NOTE: DRA7xx and TEGRA194 drivers are only compile tested. Testing this series
> > on the real platforms is greatly appreciated.
> > 
> 
> Lorenzo, all patches in this series got review tags. Can you please merge now?
> 

Krzysztof, any update on this series?

Thanks,
Mani

> Thanks,
> Mani
> 
> > Thanks,
> > Mani
> > 
> > [1] https://lore.kernel.org/all/20220802072426.GA2494@thinkpad/T/#mfa3a5b3a9694798a562c36b228f595b6a571477d
> > [2] https://lore.kernel.org/all/20220228055240.24774-1-manivannan.sadhasivam@linaro.org
> > 
> > Changes in v5:
> > 
> > * Collected review tag from Vidya
> > * Fixed the issue reported by Kbot regarding missing declaration
> > 
> > Changes in v4:
> > 
> > * Added check for the presence of event_ops before involing the callbacks (Kishon)
> > * Added return with IRQ_WAKE_THREAD when link_up event is found in the hard irq
> >   handler of tegra194 driver (Vidya)
> > * Collected review tags
> > 
> > Changes in v3:
> > 
> > * As Kishon spotted, fixed the DRA7xx driver and also the TEGRA194 driver to
> >   call the LINK_UP callback in threaded IRQ handler.
> > 
> > Changes in v2:
> > 
> > * Introduced a new "list_lock" for protecting the epc->pci_epf list and
> >   used it in the callback mechanism.
> > 
> > Manivannan Sadhasivam (5):
> >   PCI: dra7xx: Use threaded IRQ handler for "dra7xx-pcie-main" IRQ
> >   PCI: tegra194: Move dw_pcie_ep_linkup() to threaded IRQ handler
> >   PCI: endpoint: Use a separate lock for protecting epc->pci_epf list
> >   PCI: endpoint: Use callback mechanism for passing events from EPC to
> >     EPF
> >   PCI: endpoint: Use link_up() callback in place of LINK_UP notifier
> > 
> >  drivers/pci/controller/dwc/pci-dra7xx.c       |  2 +-
> >  drivers/pci/controller/dwc/pcie-tegra194.c    |  9 ++++-
> >  drivers/pci/endpoint/functions/pci-epf-test.c | 38 ++++++-------------
> >  drivers/pci/endpoint/pci-epc-core.c           | 32 ++++++++++++----
> >  include/linux/pci-epc.h                       | 10 +----
> >  include/linux/pci-epf.h                       | 19 ++++++----
> >  6 files changed, 59 insertions(+), 51 deletions(-)
> > 
> > -- 
> > 2.25.1
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்
  

Hello,

> > > During the review of the patch that fixes DBI access in PCI EP, Rob
> > > suggested [1] using a fixed interface for passing the events from EPC to
> > > EPF instead of the in-kernel notifiers.
> > > 
> > > This series introduces a simple callback based mechanism for passing the
> > > events from EPC to EPF. This interface is chosen for satisfying the below
> > > requirements:
> > > 
> > > 1. The notification has to reach the EPF drivers without any additional
> > > latency.
> > > 2. The context of the caller (EPC) needs to be preserved while passing the
> > > notifications.
> > > 
> > > With the existing notifier mechanism, the 1st case can be satisfied since
> > > notifiers aren't adding any huge overhead. But the 2nd case is clearly not
> > > satisfied, because the current atomic notifiers forces the EPF
> > > notification context to be atomic even though the caller (EPC) may not be
> > > in atomic context. In the notification function, the EPF drivers are
> > > required to call several EPC APIs that might sleep and this triggers a
> > > sleeping in atomic bug during runtime.
> > > 
> > > The above issue could be fixed by using a blocking notifier instead of
> > > atomic, but that proposal was not accepted either [2].
> > > 
> > > So instead of working around the issues within the notifiers, let's get rid
> > > of it and use the callback mechanism.
> > > 
> > > NOTE: DRA7xx and TEGRA194 drivers are only compile tested. Testing this series
> > > on the real platforms is greatly appreciated.
> > > 
> > 
> > Lorenzo, all patches in this series got review tags. Can you please merge now?
> > 
> 
> Krzysztof, any update on this series?

Sorry for the late reply.  I just realised that my question from a few days
ago has yet to make it to the mailing list. Again, I apologise for keeping
you waiting.


Nevertheless, I was asking whether there would be any "Fixes:" tags to add
and if we should let the stable maintainers know since this fixes an issue
that might be worth back-porting to older kernels.


Let me know.  Otherwise, everything looks good!  Thank you a lot!
 
	Krzysztof
  

Hello,

> During the review of the patch that fixes DBI access in PCI EP, Rob
> suggested [1] using a fixed interface for passing the events from EPC to
> EPF instead of the in-kernel notifiers.
> 
> This series introduces a simple callback based mechanism for passing the
> events from EPC to EPF. This interface is chosen for satisfying the below
> requirements:
> 
> 1. The notification has to reach the EPF drivers without any additional
> latency.
> 2. The context of the caller (EPC) needs to be preserved while passing the
> notifications.
> 
> With the existing notifier mechanism, the 1st case can be satisfied since
> notifiers aren't adding any huge overhead. But the 2nd case is clearly not
> satisfied, because the current atomic notifiers forces the EPF
> notification context to be atomic even though the caller (EPC) may not be
> in atomic context. In the notification function, the EPF drivers are
> required to call several EPC APIs that might sleep and this triggers a
> sleeping in atomic bug during runtime.
> 
> The above issue could be fixed by using a blocking notifier instead of
> atomic, but that proposal was not accepted either [2].
> 
> So instead of working around the issues within the notifiers, let's get rid
> of it and use the callback mechanism.
> 
> NOTE: DRA7xx and TEGRA194 drivers are only compile tested. Testing this series
> on the real platforms is greatly appreciated.
[...]

Applied to pci/endpoint, thank you!

[01/05] PCI: dra7xx: Use threaded IRQ handler for "dra7xx-pcie-main" IRQ
        https://git.kernel.org/pci/pci/c/da87d35a6e51

[02/05] PCI: tegra194: Move dw_pcie_ep_linkup() to threaded IRQ handler
        https://git.kernel.org/pci/pci/c/c2cc5cdda46c

[03/05] PCI: endpoint: Use a separate lock for protecting epc->pci_epf list
        https://git.kernel.org/pci/pci/c/d6dd5bafaabf

[04/05] PCI: endpoint: Use callback mechanism for passing events from EPC to EPF
        https://git.kernel.org/pci/pci/c/838125b07e77

[05/05] PCI: endpoint: Use link_up() callback in place of LINK_UP notifier
        https://git.kernel.org/pci/pci/c/f5edd8715e2e

	Krzysztof