[v13,1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod

  In drivers/ntb/hw/epf/ntb_hw_epf.c
ntb_epf_peer_db_set()
{
   ...
   db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num));
   writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
               db_offset);
   ...
}

The door register offset's formular is
	offset = db_entry_size * interrupt_num + db_offset[interrupt_number]

Previous db_entry_size is 4, all db_offset is 0.
	irq | offset
       --------------
         0     0
         1     4
         2     8
        ...

Change to db_entry_size is 0 and db_offset is 0, 4, 8, ...
So we can get the same map value between irq and offset. This will be
convenience for hardware doorbell register memory map.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
  

On Thu, Nov 24, 2022 at 12:50:35AM -0500, Frank Li wrote:
> In drivers/ntb/hw/epf/ntb_hw_epf.c
> ntb_epf_peer_db_set()
> {
>    ...
>    db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num));
>    writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
>                db_offset);
>    ...
> }
> 
> The door register offset's formular is
> 	offset = db_entry_size * interrupt_num + db_offset[interrupt_number]

You did not mention the DB BAR here. Without that, this calculation doesn't
make sense.

> 
> Previous db_entry_size is 4, all db_offset is 0.

s/Previous/Previously

> 	irq | offset
>        --------------
>          0     0
>          1     4
>          2     8
>         ...
> 
> Change to db_entry_size is 0 and db_offset is 0, 4, 8, ...
> So we can get the same map value between irq and offset. This will be
> convenience for hardware doorbell register memory map.
> 

In your irq-imx-mu-msi.c driver, the msi_address is calculated as:

```
u64 addr = msi_data->msiir_addr + 4 * data->hwirq;
```

So the MSI addresses itself are of 4 bytes width. So the offsets will be
separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses
as they are 4 bytes apart.

So you want to change the offset to 0, 4, 8,... by zeroing db_entry_size,
right?

> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index 04698e7995a5..0d744975f815 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
>  	ctrl->num_mws = ntb->num_mws;
>  	ntb->spad_size = spad_size;
>  
> -	ctrl->db_entry_size = sizeof(u32);
> +	ctrl->db_entry_size = 0;
>  
>  	for (i = 0; i < ntb->db_count; i++) {
>  		ntb->reg->db_data[i] = 1 + i;
> -		ntb->reg->db_offset[i] = 0;
> +		ntb->reg->db_offset[i] = sizeof(u32) * i;

If my above understanding is correct, then you could just reassign
"db_entry_size" in epf_ntb_epc_msi_init().

Thanks,
Mani

>  	}
>  
>  	return 0;
> -- 
> 2.34.1
>
  

> -----Original Message-----
> From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> Sent: Thursday, November 24, 2022 3:19 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: lpieralisi@kernel.org; Aisheng Dong <aisheng.dong@nxp.com>;
> bhelgaas@google.com; devicetree@vger.kernel.org; festevam@gmail.com;
> imx@lists.linux.dev; jdmason@kudzu.us; kernel@pengutronix.de;
> kishon@ti.com; krzysztof.kozlowski+dt@linaro.org; kw@linux.com; linux-
> arm-kernel@lists.infradead.org; dl-linux-imx <linux-imx@nxp.com>; linux-
> kernel@vger.kernel.org; linux-pci@vger.kernel.org;
> lorenzo.pieralisi@arm.com; lznuaa@gmail.com; maz@kernel.org;
> ntb@lists.linux.dev; Peng Fan <peng.fan@nxp.com>; robh+dt@kernel.org;
> s.hauer@pengutronix.de; shawnguo@kernel.org; tglx@linutronix.de
> Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change
> doorbell register offset calc mathod
> 
> Caution: EXT Email
> 
> On Thu, Nov 24, 2022 at 12:50:35AM -0500, Frank Li wrote:
> > In drivers/ntb/hw/epf/ntb_hw_epf.c
> > ntb_epf_peer_db_set()
> > {
> >    ...
> >    db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num));
> >    writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
> >                db_offset);
> >    ...
> > }
> >
> > The door register offset's formular is
> >       offset = db_entry_size * interrupt_num + db_offset[interrupt_number]
> 
> You did not mention the DB BAR here. Without that, this calculation doesn't
> make sense.

Doorbell register offset should means Base on DB BAR. 
How about "The formula of  door register offset refer to DB BAR"?

> 
> >
> > Previous db_entry_size is 4, all db_offset is 0.
> 
> s/Previous/Previously
> 
> >       irq | offset
> >        --------------
> >          0     0
> >          1     4
> >          2     8
> >         ...
> >
> > Change to db_entry_size is 0 and db_offset is 0, 4, 8, ...
> > So we can get the same map value between irq and offset. This will be
> > convenience for hardware doorbell register memory map.
> >
> 
> In your irq-imx-mu-msi.c driver, the msi_address is calculated as:
> 
> ```
> u64 addr = msi_data->msiir_addr + 4 * data->hwirq;
> ```
> 
> So the MSI addresses itself are of 4 bytes width. So the offsets will be
> separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses
> as they are 4 bytes apart.

Addr is absolute physical IO address, which increased by 4. But it doesn't matter.
It should be okay if range is between 2^32.

> 
> So you want to change the offset to 0, 4, 8,... by zeroing db_entry_size,
> right?

I want to directly using db_offset[irq] value as offset. It will be simple. 

I am not sure why ntb_hw_epf.c use below formular.   
 "Db_offset[irq] + irq * db_entry_size"

Db_entry_size = 0 will be simple,  all offset will be controlled by db_offset[]

You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.

> 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > index 04698e7995a5..0d744975f815 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > @@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct
> epf_ntb *ntb)
> >       ctrl->num_mws = ntb->num_mws;
> >       ntb->spad_size = spad_size;
> >
> > -     ctrl->db_entry_size = sizeof(u32);
> > +     ctrl->db_entry_size = 0;
> >
> >       for (i = 0; i < ntb->db_count; i++) {
> >               ntb->reg->db_data[i] = 1 + i;
> > -             ntb->reg->db_offset[i] = 0;
> > +             ntb->reg->db_offset[i] = sizeof(u32) * i;
> 
> If my above understanding is correct, then you could just reassign
> "db_entry_size" in epf_ntb_epc_msi_init().

Yes, that's one method.
I want to use one method to calc db offset for both software polling
and MSI.  So overall logic should be simple. 

Frank Li

> 
> Thanks,
> Mani
> 
> >       }
> >
> >       return 0;
> > --
> > 2.34.1
> >
> 
> --
> மணிவண்ணன் சதாசிவம்
  

On Thu, Nov 24, 2022 at 05:49:32PM +0000, Frank Li wrote:
> 
> 
> > -----Original Message-----
> > From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > Sent: Thursday, November 24, 2022 3:19 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: lpieralisi@kernel.org; Aisheng Dong <aisheng.dong@nxp.com>;
> > bhelgaas@google.com; devicetree@vger.kernel.org; festevam@gmail.com;
> > imx@lists.linux.dev; jdmason@kudzu.us; kernel@pengutronix.de;
> > kishon@ti.com; krzysztof.kozlowski+dt@linaro.org; kw@linux.com; linux-
> > arm-kernel@lists.infradead.org; dl-linux-imx <linux-imx@nxp.com>; linux-
> > kernel@vger.kernel.org; linux-pci@vger.kernel.org;
> > lorenzo.pieralisi@arm.com; lznuaa@gmail.com; maz@kernel.org;
> > ntb@lists.linux.dev; Peng Fan <peng.fan@nxp.com>; robh+dt@kernel.org;
> > s.hauer@pengutronix.de; shawnguo@kernel.org; tglx@linutronix.de
> > Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change
> > doorbell register offset calc mathod
> > 
> > Caution: EXT Email
> > 
> > On Thu, Nov 24, 2022 at 12:50:35AM -0500, Frank Li wrote:
> > > In drivers/ntb/hw/epf/ntb_hw_epf.c
> > > ntb_epf_peer_db_set()
> > > {
> > >    ...
> > >    db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num));
> > >    writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
> > >                db_offset);
> > >    ...
> > > }
> > >
> > > The door register offset's formular is
> > >       offset = db_entry_size * interrupt_num + db_offset[interrupt_number]
> > 
> > You did not mention the DB BAR here. Without that, this calculation doesn't
> > make sense.
> 
> Doorbell register offset should means Base on DB BAR. 
> How about "The formula of  door register offset refer to DB BAR"?

"Doobell register offset in DB BAR is calculated using:"

> 
> > 
> > >
> > > Previous db_entry_size is 4, all db_offset is 0.
> > 
> > s/Previous/Previously
> > 
> > >       irq | offset
> > >        --------------
> > >          0     0
> > >          1     4
> > >          2     8
> > >         ...
> > >
> > > Change to db_entry_size is 0 and db_offset is 0, 4, 8, ...
> > > So we can get the same map value between irq and offset. This will be
> > > convenience for hardware doorbell register memory map.
> > >
> > 
> > In your irq-imx-mu-msi.c driver, the msi_address is calculated as:
> > 
> > ```
> > u64 addr = msi_data->msiir_addr + 4 * data->hwirq;
> > ```
> > 
> > So the MSI addresses itself are of 4 bytes width. So the offsets will be
> > separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses
> > as they are 4 bytes apart.
> 
> Addr is absolute physical IO address, which increased by 4. But it doesn't matter.
> It should be okay if range is between 2^32.
> 
> > 
> > So you want to change the offset to 0, 4, 8,... by zeroing db_entry_size,
> > right?
> 
> I want to directly using db_offset[irq] value as offset. It will be simple. 
> 
> I am not sure why ntb_hw_epf.c use below formular.   
>  "Db_offset[irq] + irq * db_entry_size"
> 
> Db_entry_size = 0 will be simple,  all offset will be controlled by db_offset[]
> 
> You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.
> 
> > 
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > index 04698e7995a5..0d744975f815 100644
> > > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > @@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct
> > epf_ntb *ntb)
> > >       ctrl->num_mws = ntb->num_mws;
> > >       ntb->spad_size = spad_size;
> > >
> > > -     ctrl->db_entry_size = sizeof(u32);
> > > +     ctrl->db_entry_size = 0;
> > >
> > >       for (i = 0; i < ntb->db_count; i++) {
> > >               ntb->reg->db_data[i] = 1 + i;
> > > -             ntb->reg->db_offset[i] = 0;
> > > +             ntb->reg->db_offset[i] = sizeof(u32) * i;
> > 
> > If my above understanding is correct, then you could just reassign
> > "db_entry_size" in epf_ntb_epc_msi_init().
> 
> Yes, that's one method.
> I want to use one method to calc db offset for both software polling
> and MSI.  So overall logic should be simple. 
> 

I think it is better to leave db_entry_size for polling as it is and modify it
for MSI alone.

Thanks,
Mani

> Frank Li
> 
> > 
> > Thanks,
> > Mani
> > 
> > >       }
> > >
> > >       return 0;
> > > --
> > > 2.34.1
> > >
> > 
> > --
> > மணிவண்ணன் சதாசிவம்
  

> -----Original Message-----
> From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> Sent: Thursday, November 24, 2022 12:51 PM
> To: Frank Li <frank.li@nxp.com>
> Cc: lpieralisi@kernel.org; Aisheng Dong <aisheng.dong@nxp.com>;
> bhelgaas@google.com; devicetree@vger.kernel.org; festevam@gmail.com;
> imx@lists.linux.dev; jdmason@kudzu.us; kernel@pengutronix.de;
> krzysztof.kozlowski+dt@linaro.org; kw@linux.com; linux-arm-
> kernel@lists.infradead.org; dl-linux-imx <linux-imx@nxp.com>; linux-
> kernel@vger.kernel.org; linux-pci@vger.kernel.org;
> lorenzo.pieralisi@arm.com; lznuaa@gmail.com; maz@kernel.org;
> ntb@lists.linux.dev; Peng Fan <peng.fan@nxp.com>; robh+dt@kernel.org;
> s.hauer@pengutronix.de; shawnguo@kernel.org; tglx@linutronix.de;
> kishon@kernel.org
> Subject: Re: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change
> doorbell register offset calc mathod
> 
> Caution: EXT Email
> 
> On Thu, Nov 24, 2022 at 05:49:32PM +0000, Frank Li wrote:
> >
> >
> > > -----Original Message-----
> > > From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > > Sent: Thursday, November 24, 2022 3:19 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: lpieralisi@kernel.org; Aisheng Dong <aisheng.dong@nxp.com>;
> > > bhelgaas@google.com; devicetree@vger.kernel.org;
> festevam@gmail.com;
> > > imx@lists.linux.dev; jdmason@kudzu.us; kernel@pengutronix.de;
> > > kishon@ti.com; krzysztof.kozlowski+dt@linaro.org; kw@linux.com; linux-
> > > arm-kernel@lists.infradead.org; dl-linux-imx <linux-imx@nxp.com>;
> linux-
> > > kernel@vger.kernel.org; linux-pci@vger.kernel.org;
> > > lorenzo.pieralisi@arm.com; lznuaa@gmail.com; maz@kernel.org;
> > > ntb@lists.linux.dev; Peng Fan <peng.fan@nxp.com>; robh+dt@kernel.org;
> > > s.hauer@pengutronix.de; shawnguo@kernel.org; tglx@linutronix.de
> > > Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change
> > > doorbell register offset calc mathod
> > >
> > > Caution: EXT Email
> > >
> > > On Thu, Nov 24, 2022 at 12:50:35AM -0500, Frank Li wrote:
> > > > In drivers/ntb/hw/epf/ntb_hw_epf.c
> > > > ntb_epf_peer_db_set()
> > > > {
> > > >    ...
> > > >    db_offset = readl(ndev->ctrl_reg +
> NTB_EPF_DB_OFFSET(interrupt_num));
> > > >    writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
> > > >                db_offset);
> > > >    ...
> > > > }
> > > >
> > > > The door register offset's formular is
> > > >       offset = db_entry_size * interrupt_num +
> db_offset[interrupt_number]
> > >
> > > You did not mention the DB BAR here. Without that, this calculation
> doesn't
> > > make sense.
> >
> > Doorbell register offset should means Base on DB BAR.
> > How about "The formula of  door register offset refer to DB BAR"?
> 
> "Doobell register offset in DB BAR is calculated using:"
> 
> >
> > >
> > > >
> > > > Previous db_entry_size is 4, all db_offset is 0.
> > >
> > > s/Previous/Previously
> > >
> > > >       irq | offset
> > > >        --------------
> > > >          0     0
> > > >          1     4
> > > >          2     8
> > > >         ...
> > > >
> > > > Change to db_entry_size is 0 and db_offset is 0, 4, 8, ...
> > > > So we can get the same map value between irq and offset. This will be
> > > > convenience for hardware doorbell register memory map.
> > > >
> > >
> > > In your irq-imx-mu-msi.c driver, the msi_address is calculated as:
> > >
> > > ```
> > > u64 addr = msi_data->msiir_addr + 4 * data->hwirq;
> > > ```
> > >
> > > So the MSI addresses itself are of 4 bytes width. So the offsets will be
> > > separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI
> addresses
> > > as they are 4 bytes apart.
> >
> > Addr is absolute physical IO address, which increased by 4. But it doesn't
> matter.
> > It should be okay if range is between 2^32.
> >
> > >
> > > So you want to change the offset to 0, 4, 8,... by zeroing db_entry_size,
> > > right?
> >
> > I want to directly using db_offset[irq] value as offset. It will be simple.
> >
> > I am not sure why ntb_hw_epf.c use below formular.
> >  "Db_offset[irq] + irq * db_entry_size"
> >
> > Db_entry_size = 0 will be simple,  all offset will be controlled by db_offset[]
> >
> > You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.
> >
> > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++--
> > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > > index 04698e7995a5..0d744975f815 100644
> > > > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > > > @@ -461,11 +461,11 @@ static int
> epf_ntb_config_spad_bar_alloc(struct
> > > epf_ntb *ntb)
> > > >       ctrl->num_mws = ntb->num_mws;
> > > >       ntb->spad_size = spad_size;
> > > >
> > > > -     ctrl->db_entry_size = sizeof(u32);
> > > > +     ctrl->db_entry_size = 0;
> > > >
> > > >       for (i = 0; i < ntb->db_count; i++) {
> > > >               ntb->reg->db_data[i] = 1 + i;
> > > > -             ntb->reg->db_offset[i] = 0;
> > > > +             ntb->reg->db_offset[i] = sizeof(u32) * i;
> > >
> > > If my above understanding is correct, then you could just reassign
> > > "db_entry_size" in epf_ntb_epc_msi_init().
> >
> > Yes, that's one method.
> > I want to use one method to calc db offset for both software polling
> > and MSI.  So overall logic should be simple.
> >
> 
> I think it is better to leave db_entry_size for polling as it is and modify it
> for MSI alone.

Okay, that means needn't this patch at all.

> 
> Thanks,
> Mani
> 
> > Frank Li
> >
> > >
> > > Thanks,
> > > Mani
> > >
> > > >       }
> > > >
> > > >       return 0;
> > > > --
> > > > 2.34.1
> > > >
> > >
> > > --
> > > மணிவண்ணன் சதாசிவம்
> 
> --
> மணிவண்ணன் சதாசிவம்