[V11,0/5] Generate device tree node for pci devices

  This patch series introduces OF overlay support for PCI devices which
primarily addresses two use cases. First, it provides a data driven method
to describe hardware peripherals that are present in a PCI endpoint and
hence can be accessed by the PCI host. Second, it allows reuse of a OF
compatible driver -- often used in SoC platforms -- in a PCI host based
system.

There are 2 series devices rely on this patch:

  1) Xilinx Alveo Accelerator cards (FPGA based device)
  2) Microchip LAN9662 Ethernet Controller

     Please see: https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@bootlin.com/

Normally, the PCI core discovers PCI devices and their BARs using the
PCI enumeration process. However, the process does not provide a way to
discover the hardware peripherals that are present in a PCI device, and
which can be accessed through the PCI BARs. Also, the enumeration process
does not provide a way to associate MSI-X vectors of a PCI device with the
hardware peripherals that are present in the device. PCI device drivers
often use header files to describe the hardware peripherals and their
resources as there is no standard data driven way to do so. This patch
series proposes to use flattened device tree blob to describe the
peripherals in a data driven way. Based on previous discussion, using
device tree overlay is the best way to unflatten the blob and populate
platform devices. To use device tree overlay, there are three obvious
problems that need to be resolved.

First, we need to create a base tree for non-DT system such as x86_64. A
patch series has been submitted for this:
https://lore.kernel.org/lkml/20220624034327.2542112-1-frowand.list@gmail.com/
https://lore.kernel.org/lkml/20220216050056.311496-1-lizhi.hou@xilinx.com/

Second, a device tree node corresponding to the PCI endpoint is required
for overlaying the flattened device tree blob for that PCI endpoint.
Because PCI is a self-discoverable bus, a device tree node is usually not
created for PCI devices. This series adds support to generate a device
tree node for a PCI device which advertises itself using PCI quirks
infrastructure.

Third, we need to generate device tree nodes for PCI bridges since a child
PCI endpoint may choose to have a device tree node created.

This patch series is made up of three patches.

The first patch is adding OF interface to create or destroy OF node
dynamically.

The second patch introduces a kernel option, CONFIG_PCI_DYNAMIC_OF_NODES.
When the option is turned on, the kernel will generate device tree nodes
for all PCI bridges unconditionally. The patch also shows how to use the
PCI quirks infrastructure, DECLARE_PCI_FIXUP_FINAL to generate a device
tree node for a device. Specifically, the patch generates a device tree
node for Xilinx Alveo U50 PCIe accelerator device. The generated device
tree nodes do not have any property.

The third patch adds basic properties ('reg', 'compatible' and
'device_type') to the dynamically generated device tree nodes. More
properties can be added in the future.

Here is the example of device tree nodes generated within the ARM64 QEMU.

# lspci -t
-[0000:00]-+-00.0
           +-01.0
           +-03.0-[01-03]----00.0-[02-03]----00.0-[03]----00.0
           +-03.1-[04]--
           \-04.0-[05-06]----00.0-[06]--

Without CONFIG_PCI_DYNAMIC_OF_NODES

# tree /sys/firmware/devicetree/base/pcie@10000000/
/sys/firmware/devicetree/base/pcie@10000000/
|-- #address-cells
|-- #interrupt-cells
|-- #size-cells
|-- bus-range
|-- compatible
|-- device_type
|-- dma-coherent
|-- interrupt-map
|-- interrupt-map-mask
|-- linux,pci-domain
|-- msi-map
|-- name
|-- ranges
`-- reg

With CONFIG_PCI_DYNAMIC_OF_NODES

# tree /sys/firmware/devicetree/base/pcie@10000000/
/sys/firmware/devicetree/base/pcie@10000000/
|-- #address-cells
|-- #interrupt-cells
|-- #size-cells
|-- bus-range
|-- compatible
|-- device_type
|-- dma-coherent
|-- interrupt-map
|-- interrupt-map-mask
|-- linux,pci-domain
|-- msi-map
|-- name
|-- pci@3,0
|   |-- #address-cells
|   |-- #size-cells
|   |-- bus-range
|   |-- compatible
|   |-- device_type
|   |-- pci@0,0
|   |   |-- #address-cells
|   |   |-- #size-cells
|   |   |-- bus-range
|   |   |-- compatible
|   |   |-- device_type
|   |   |-- pci@0,0
|   |   |   |-- #address-cells
|   |   |   |-- #size-cells
|   |   |   |-- bus-range
|   |   |   |-- compatible
|   |   |   |-- dev@0,0
|   |   |   |   |-- #address-cells
|   |   |   |   |-- #size-cells
|   |   |   |   |-- compatible
|   |   |   |   |-- ranges
|   |   |   |   `-- reg
|   |   |   |-- device_type
|   |   |   |-- ranges
|   |   |   `-- reg
|   |   |-- ranges
|   |   `-- reg
|   |-- ranges
|   `-- reg
|-- pci@3,1
|   |-- #address-cells
|   |-- #size-cells
|   |-- bus-range
|   |-- compatible
|   |-- device_type
|   |-- ranges
|   `-- reg
|-- pci@4,0
|   |-- #address-cells
|   |-- #size-cells
|   |-- bus-range
|   |-- compatible
|   |-- device_type
|   |-- pci@0,0
|   |   |-- #address-cells
|   |   |-- #size-cells
|   |   |-- bus-range
|   |   |-- compatible
|   |   |-- device_type
|   |   |-- ranges
|   |   `-- reg
|   |-- ranges
|   `-- reg
|-- ranges
`-- reg

Changes since v10:
- Remove 'dynamic' property

Changes since v9:
- Introduce 'dynamic' property to identify dynamically generated device tree
  node for PCI device
- Added 'bus-range' property to remove dtc warnings
- Minor code review fixes

Changes since v8:
- Added patches to create unit test to verifying address translation
    The test relies on QEMU PCI Test Device, please see
        https://github.com/houlz0507/xoclv2/blob/pci-dt-0329/pci-dt-patch-0329/README
    for test setup
- Minor code review fixes

Changes since v7:
- Modified dynamic node creation interfaces
- Added unittest for new added interfaces

Changes since v6:
- Removed single line wrapper functions
- Added Signed-off-by Clément Léger <clement.leger@bootlin.com>

Changes since v5:
- Fixed code review comments
- Fixed incorrect 'ranges' and 'reg' properties

Changes since RFC v4:
- Fixed code review comments

Changes since RFC v3:
- Split the Xilinx Alveo U50 PCI quirk to a separate patch
- Minor changes in commit description and code comment

Changes since RFC v2:
- Merged patch 3 with patch 2
- Added OF interfaces of_changeset_add_prop_* and use them to create
  properties.
- Added '#address-cells', '#size-cells' and 'ranges' properties.

Changes since RFC v1:
- Added one patch to create basic properties.
- To move DT related code out of PCI subsystem, replaced of_node_alloc()
  with of_create_node()/of_destroy_node()

Lizhi Hou (5):
  of: dynamic: Add interfaces for creating device node dynamically
  PCI: Create device tree node for bridge
  PCI: Add quirks to generate device tree node for Xilinx Alveo U50
  of: overlay: Extend of_overlay_fdt_apply() to specify the target node
  of: unittest: Add pci_dt_testdrv pci driver

 drivers/of/dynamic.c                          | 164 +++++++++++++
 drivers/of/overlay.c                          |  42 +++-
 drivers/of/unittest-data/Makefile             |   3 +-
 .../of/unittest-data/overlay_pci_node.dtso    |  22 ++
 drivers/of/unittest.c                         | 211 +++++++++++++++-
 drivers/pci/Kconfig                           |  12 +
 drivers/pci/Makefile                          |   1 +
 drivers/pci/bus.c                             |   2 +
 drivers/pci/of.c                              |  96 +++++++-
 drivers/pci/of_property.c                     | 232 ++++++++++++++++++
 drivers/pci/pci.h                             |  12 +
 drivers/pci/quirks.c                          |  12 +
 drivers/pci/remove.c                          |   1 +
 include/linux/of.h                            |  25 +-
 14 files changed, 818 insertions(+), 17 deletions(-)
 create mode 100644 drivers/of/unittest-data/overlay_pci_node.dtso
 create mode 100644 drivers/pci/of_property.c
  

On Tue, Jul 25, 2023 at 4:15 PM Lizhi Hou <lizhi.hou@amd.com> wrote:
>
> The PCI endpoint device such as Xilinx Alveo PCI card maps the register
> spaces from multiple hardware peripherals to its PCI BAR. Normally,
> the PCI core discovers devices and BARs using the PCI enumeration process.
> There is no infrastructure to discover the hardware peripherals that are
> present in a PCI device, and which can be accessed through the PCI BARs.
>
> Apparently, the device tree framework requires a device tree node for the
> PCI device. Thus, it can generate the device tree nodes for hardware
> peripherals underneath. Because PCI is self discoverable bus, there might
> not be a device tree node created for PCI devices. Furthermore, if the PCI
> device is hot pluggable, when it is plugged in, the device tree nodes for
> its parent bridges are required. Add support to generate device tree node
> for PCI bridges.
>
> Add an of_pci_make_dev_node() interface that can be used to create device
> tree node for PCI devices.
>
> Add a PCI_DYNAMIC_OF_NODES config option. When the option is turned on,
> the kernel will generate device tree nodes for PCI bridges unconditionally.
>
> Initially, add the basic properties for the dynamically generated device
> tree nodes which include #address-cells, #size-cells, device_type,
> compatible, ranges, reg.
>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> Signed-off-by: Lizhi Hou <lizhi.hou@amd.com>
> ---
>  drivers/pci/Kconfig       |  12 ++
>  drivers/pci/Makefile      |   1 +
>  drivers/pci/bus.c         |   2 +
>  drivers/pci/of.c          |  96 +++++++++++++++-
>  drivers/pci/of_property.c | 232 ++++++++++++++++++++++++++++++++++++++
>  drivers/pci/pci.h         |  12 ++
>  drivers/pci/remove.c      |   1 +
>  7 files changed, 354 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/pci/of_property.c
>
> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index 3c07d8d214b3..49bd09c7dd0a 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -194,6 +194,18 @@ config PCI_HYPERV
>           The PCI device frontend driver allows the kernel to import arbitrary
>           PCI devices from a PCI backend to support PCI driver domains.
>
> +config PCI_DYNAMIC_OF_NODES
> +       bool "Create device tree nodes for PCI devices"
> +       depends on OF
> +       select OF_DYNAMIC
> +       help
> +         This option enables support for generating device tree nodes for some
> +         PCI devices. Thus, the driver of this kind can load and overlay
> +         flattened device tree for its downstream devices.
> +
> +         Once this option is selected, the device tree nodes will be generated
> +         for all PCI bridges.
> +
>  choice
>         prompt "PCI Express hierarchy optimization setting"
>         default PCIE_BUS_DEFAULT
> diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
> index 2680e4c92f0a..cc8b4e01e29d 100644
> --- a/drivers/pci/Makefile
> +++ b/drivers/pci/Makefile
> @@ -32,6 +32,7 @@ obj-$(CONFIG_PCI_P2PDMA)      += p2pdma.o
>  obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
>  obj-$(CONFIG_VGA_ARB)          += vgaarb.o
>  obj-$(CONFIG_PCI_DOE)          += doe.o
> +obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o
>
>  # Endpoint library must be initialized before its users
>  obj-$(CONFIG_PCI_ENDPOINT)     += endpoint/
> diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> index 5bc81cc0a2de..ab7d06cd0099 100644
> --- a/drivers/pci/bus.c
> +++ b/drivers/pci/bus.c
> @@ -340,6 +340,8 @@ void pci_bus_add_device(struct pci_dev *dev)
>          */
>         pcibios_bus_add_device(dev);
>         pci_fixup_device(pci_fixup_final, dev);
> +       if (pci_is_bridge(dev))
> +               of_pci_make_dev_node(dev);
>         pci_create_sysfs_dev_files(dev);
>         pci_proc_attach_device(dev);
>         pci_bridge_d3_update(dev);
> diff --git a/drivers/pci/of.c b/drivers/pci/of.c
> index e51219f9f523..11d3be165e32 100644
> --- a/drivers/pci/of.c
> +++ b/drivers/pci/of.c
> @@ -495,8 +495,21 @@ static int of_irq_parse_pci(const struct pci_dev *pdev, struct of_phandle_args *
>                  * to rely on this function (you ship a firmware that doesn't
>                  * create device nodes for all PCI devices).
>                  */
> -               if (ppnode)
> -                       break;
> +               if (ppnode) {
> +                       /*
> +                        * When PCI_DYNAMIC_OF_NODES is on, a device tree
> +                        * node will be generated for PCI bridge. For the
> +                        * dynamically generated node, interrupt mapping is
> +                        * not supported. Thus, it needs to check interrupt-map
> +                        * property and set ppnode to NULL to do standard
> +                        * swizzling if interrupt-map does not present.
> +                        */
> +                       if (IS_ENABLED(CONFIG_PCI_DYNAMIC_OF_NODES) &&
> +                           !of_property_present(ppnode, "interrupt-map"))
> +                               ppnode = NULL;

We cannot use a kconfig option to determine behavior. You don't get to
decide the value of the kconfig option. The OS distro does. As I've
said in the past, the kconfig option is not a long term solution. You
need things to work the same way whether PCI nodes were populated
before the kernel runs or dynamically.

Perhaps what you need to do is read PCI_INTERRUPT_PIN and if it's
non-zero for a device, populate 'interrupts' property using the value.
Then the standard DT interrupt parsing code should work. That code
will walk up nodes until it finds the host bridge interrupt-map.

Rob