[0/2] Add qcom hvc/shmem transport

  This change introduce a new transport channel for Qualcomm virtual
platforms. The transport is mechanically similar to ARM_SCMI_TRANSPORT_SMC.
The difference between the two transports is that a parameter is passed in
the hypervisor call to identify which doorbell to assert. This parameter is
dynamically generated at runtime on the device and insuitable to pass via
the devicetree.

The function ID and parameter are stored by firmware in the shmem region.

This has been tested on ARM64 virtual Qualcomm platform.

Nikunj Kela (2):
  dt-bindings: arm: Add qcom specific hvc transport for SCMI
  firmware: arm_scmi: Add qcom hvc/shmem transport

 .../bindings/firmware/arm,scmi.yaml           |  69 +++++
 drivers/firmware/arm_scmi/Kconfig             |  13 +
 drivers/firmware/arm_scmi/Makefile            |   1 +
 drivers/firmware/arm_scmi/common.h            |   3 +
 drivers/firmware/arm_scmi/driver.c            |   4 +
 drivers/firmware/arm_scmi/qcom_hvc.c          | 241 ++++++++++++++++++
 6 files changed, 331 insertions(+)
 create mode 100644 drivers/firmware/arm_scmi/qcom_hvc.c
  

On 18.07.2023 18:08, Nikunj Kela wrote:
> This change introduce a new transport channel for Qualcomm virtual
> platforms. The transport is mechanically similar to ARM_SCMI_TRANSPORT_SMC.
> The difference between the two transports is that a parameter is passed in
> the hypervisor call to identify which doorbell to assert. This parameter is
> dynamically generated at runtime on the device and insuitable to pass via
> the devicetree.
> 
> The function ID and parameter are stored by firmware in the shmem region.
> 
> This has been tested on ARM64 virtual Qualcomm platform.
What can we test it on?

Konrad
  

Gentle Ping!

On 9/11/2023 12:43 PM, Nikunj Kela wrote:
> This change augments smc transport to include support for Qualcomm virtual
> platforms by passing a parameter(capability-id) in the hypervisor call to
> identify which doorbell to assert. This parameter is dynamically generated
> at runtime on the device and insuitable to pass via the devicetree.
>
> The function ID and parameter are stored by firmware in the shmem region.
>
> This has been tested on ARM64 virtual Qualcomm platform.
>
> ---
> v4 -> port the changes into smc.c
>
> v3 -> fix the compilation error reported by the test bot,
>        add support for polling based instances
>
> v2 -> use allOf construct in dtb schema,
>        remove wrappers from mutexes,
>        use architecture independent channel layout
>
> v1 -> original patches
>
> Nikunj Kela (4):
>    firmware: arm_scmi: Add polling support for completion in smc
>    dt-bindings: arm: convert nested if-else construct to allOf
>    dt-bindings: arm: Add new compatible for smc/hvc transport for SCMI
>    firmware: arm_scmi: Add qcom hvc/shmem transport support
>
>   .../bindings/firmware/arm,scmi.yaml           | 67 +++++++++++--------
>   drivers/firmware/arm_scmi/Kconfig             | 14 ++++
>   drivers/firmware/arm_scmi/driver.c            |  1 +
>   drivers/firmware/arm_scmi/smc.c               | 62 +++++++++++++++--
>   4 files changed, 110 insertions(+), 34 deletions(-)
>
  

On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
> Gentle Ping!
> 

I will take a look at this later this week. That said, I am unable be
gauge the urgency based on you ping here. You have shown the same urgency
last time for a feature that I queued promptly just to know that it was
abandon within couple of days. So I don't want to rush here simply based
on the number of pings here. I need to understand that it is really that
important. For now, I am thinking of skipping even v6.7 just to allow
some time for Qcom to make up its mind and be absolutely sure this is what
they *really* want this time.
  

On Mon, Sep 18, 2023 at 04:15:52PM +0100, Sudeep Holla wrote:
> On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
> > Gentle Ping!
> > 
> 
> I will take a look at this later this week. That said, I am unable be
> gauge the urgency based on you ping here. You have shown the same urgency
> last time for a feature that I queued promptly just to know that it was
> abandon within couple of days. So I don't want to rush here simply based
> on the number of pings here. I need to understand that it is really that
> important. For now, I am thinking of skipping even v6.7 just to allow
> some time for Qcom to make up its mind and be absolutely sure this is what
> they *really* want this time.

Hi Sudeep,

Red Hat is interested in this patch set. Qualcomm is moving one of their
automotive platforms over to use SCMI and this will appear in that
product.

Brian
  

On 18/09/2023 17:01, Nikunj Kela wrote:
> Gentle Ping!

Whatever is written with exclamation mark is not really gentle.
Especially for second time... and 7 days after posting. 7 days and you ping.

Please relax, and help out by reviewing other patches on the mailing
lists in order to relieve the burden of maintainers and move your
patches higher up the list.

Best regards,
Krzysztof
  

On Mon, Sep 18, 2023 at 11:54:25AM -0400, Brian Masney wrote:
> On Mon, Sep 18, 2023 at 04:15:52PM +0100, Sudeep Holla wrote:
> > On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
> > > Gentle Ping!
> > >
> >
> > I will take a look at this later this week. That said, I am unable be
> > gauge the urgency based on you ping here. You have shown the same urgency
> > last time for a feature that I queued promptly just to know that it was
> > abandon within couple of days. So I don't want to rush here simply based
> > on the number of pings here. I need to understand that it is really that
> > important. For now, I am thinking of skipping even v6.7 just to allow
> > some time for Qcom to make up its mind and be absolutely sure this is what
> > they *really* want this time.
>
> Hi Sudeep,
>
> Red Hat is interested in this patch set. Qualcomm is moving one of their
> automotive platforms over to use SCMI and this will appear in that
> product.
>

Thanks Brian, I trust Redhat over Qcom 😄. I will try to review and enable
progress later this week. We can try to target next merge window.

--
Regards,
Sudeep
  

On 9/19/2023 1:56 AM, Sudeep Holla wrote:
> On Mon, Sep 18, 2023 at 11:54:25AM -0400, Brian Masney wrote:
>> On Mon, Sep 18, 2023 at 04:15:52PM +0100, Sudeep Holla wrote:
>>> On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
>>>> Gentle Ping!
>>>>
>>> I will take a look at this later this week. That said, I am unable be
>>> gauge the urgency based on you ping here. You have shown the same urgency
>>> last time for a feature that I queued promptly just to know that it was
>>> abandon within couple of days. So I don't want to rush here simply based
>>> on the number of pings here. I need to understand that it is really that
>>> important. For now, I am thinking of skipping even v6.7 just to allow
>>> some time for Qcom to make up its mind and be absolutely sure this is what
>>> they *really* want this time.
>> Hi Sudeep,
>>
>> Red Hat is interested in this patch set. Qualcomm is moving one of their
>> automotive platforms over to use SCMI and this will appear in that
>> product.
>>
> Thanks Brian, I trust Redhat over Qcom 😄. I will try to review and enable
> progress later this week. We can try to target next merge window.
>
> --
> Regards,
> Sudeep
Gentle Ping...
  

On Mon, Oct 02, 2023 at 10:31:27AM -0700, Nikunj Kela wrote:
> 
> On 9/19/2023 1:56 AM, Sudeep Holla wrote:
> > On Mon, Sep 18, 2023 at 11:54:25AM -0400, Brian Masney wrote:
> > > On Mon, Sep 18, 2023 at 04:15:52PM +0100, Sudeep Holla wrote:
> > > > On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
> > > > > Gentle Ping!
> > > > > 
> > > > I will take a look at this later this week. That said, I am unable be
> > > > gauge the urgency based on you ping here. You have shown the same urgency
> > > > last time for a feature that I queued promptly just to know that it was
> > > > abandon within couple of days. So I don't want to rush here simply based
> > > > on the number of pings here. I need to understand that it is really that
> > > > important. For now, I am thinking of skipping even v6.7 just to allow
> > > > some time for Qcom to make up its mind and be absolutely sure this is what
> > > > they *really* want this time.
> > > Hi Sudeep,
> > > 
> > > Red Hat is interested in this patch set. Qualcomm is moving one of their
> > > automotive platforms over to use SCMI and this will appear in that
> > > product.
> > > 
> > Thanks Brian, I trust Redhat over Qcom 😄. I will try to review and enable
> > progress later this week. We can try to target next merge window.
> > 
> > --
> > Regards,
> > Sudeep
> Gentle Ping...

Looking at this tomorrow.

Thanks,
Cristian
  

On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
> Currently, the return from the smc call assumes the completion of
> the scmi request. However this may not be true in virtual platforms
> that are using hvc doorbell.
> 
> This change adds a Kconfig to enable the polling for the request
> completion.
> 
> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> ---
>  drivers/firmware/arm_scmi/Kconfig | 14 ++++++++++++++
>  drivers/firmware/arm_scmi/smc.c   | 15 ++++++++++++++-
>  2 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/firmware/arm_scmi/Kconfig b/drivers/firmware/arm_scmi/Kconfig
> index ea0f5083ac47..771d60f8319f 100644
> --- a/drivers/firmware/arm_scmi/Kconfig
> +++ b/drivers/firmware/arm_scmi/Kconfig
> @@ -125,6 +125,20 @@ config ARM_SCMI_TRANSPORT_SMC_ATOMIC_ENABLE
>  	  in atomic context too, at the price of using a number of busy-waiting
>  	  primitives all over instead. If unsure say N.
>  
> +config ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
> +	bool "Enable polling support for SCMI SMC transport"
> +	depends on ARM_SCMI_TRANSPORT_SMC
> +	help
> +	  Enable completion polling support for SCMI SMC based transport.
> +
> +	  If you want the SCMI SMC based transport to poll for the completion,
> +	  answer Y.
> +	  Enabling completion polling might be desired in the absence of the a2p
> +	  irq when the return from smc/hvc call doesn't indicate the completion
> +	  of the SCMI requests. This might be useful for instances used in
> +	  virtual platforms.
> +	  If unsure say N.
> +
>  config ARM_SCMI_TRANSPORT_VIRTIO
>  	bool "SCMI transport based on VirtIO"
>  	depends on VIRTIO=y || VIRTIO=ARM_SCMI_PROTOCOL
> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> index c193516a254d..0a0b7e401159 100644
> --- a/drivers/firmware/arm_scmi/smc.c
> +++ b/drivers/firmware/arm_scmi/smc.c
> @@ -250,6 +250,16 @@ static void smc_mark_txdone(struct scmi_chan_info *cinfo, int ret,
>  	smc_channel_lock_release(scmi_info);
>  }
>  
> +#ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
> +static bool
> +smc_poll_done(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
> +{
> +	struct scmi_smc *scmi_info = cinfo->transport_info;
> +
> +	return shmem_poll_done(scmi_info->shmem, xfer);
> +}
> +#endif
> +
>  static const struct scmi_transport_ops scmi_smc_ops = {
>  	.chan_available = smc_chan_available,
>  	.chan_setup = smc_chan_setup,
> @@ -257,6 +267,9 @@ static const struct scmi_transport_ops scmi_smc_ops = {
>  	.send_message = smc_send_message,
>  	.mark_txdone = smc_mark_txdone,
>  	.fetch_response = smc_fetch_response,
> +#ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
> +	.poll_done = smc_poll_done,
> +#endif
>  };
>  
>  const struct scmi_desc scmi_smc_desc = {
> @@ -272,6 +285,6 @@ const struct scmi_desc scmi_smc_desc = {
>  	 * for the issued command will be immmediately ready to be fetched
>  	 * from the shared memory area.
>  	 */
> -	.sync_cmds_completed_on_ret = true,
> +	.sync_cmds_completed_on_ret = !IS_ENABLED(CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION),
>  	.atomic_enabled = IS_ENABLED(CONFIG_ARM_SCMI_TRANSPORT_SMC_ATOMIC_ENABLE),

From a Linux distributor viewpoint, it would be nice if this was
determined at runtime, rather than at compile time. We generate a single
kernel binary that's used on systems from multiple hardware
manufacturers. We'd run into an issue if one company required this, but
another one didn't. We may potentially run into this same type of issue
with the upstream arm64 defconfig.

Brian
  

On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> This change adds the support for SCMI message exchange on Qualcomm
> virtual platforms.
> 
> The hypervisor associates an object-id also known as capability-id
> with each hvc doorbell object. The capability-id is used to identify the
> doorbell from the VM's capability namespace, similar to a file-descriptor.
> 
> The hypervisor, in addition to the function-id, expects the capability-id
> to be passed in x1 register when HVC call is invoked.
> 
> The function-id & capability-id are allocated by the hypervisor on bootup
> and are stored in the shmem region by the firmware before starting Linux.
> 
> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> ---
>  drivers/firmware/arm_scmi/driver.c |  1 +
>  drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
>  2 files changed, 43 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 87383c05424b..ea344bc6ae49 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
>  #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
>  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
>  	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
> +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
>  #endif
>  #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
>  	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> index 0a0b7e401159..94ec07fdc14a 100644
> --- a/drivers/firmware/arm_scmi/smc.c
> +++ b/drivers/firmware/arm_scmi/smc.c
> @@ -50,6 +50,9 @@
>   * @func_id: smc/hvc call function id
>   * @param_page: 4K page number of the shmem channel
>   * @param_offset: Offset within the 4K page of the shmem channel
> + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
> + *	    platforms
> + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
>   */
>  
>  struct scmi_smc {
> @@ -63,6 +66,8 @@ struct scmi_smc {
>  	u32 func_id;
>  	u32 param_page;
>  	u32 param_offset;
> +	u64 cap_id;
> +	bool qcom_xport;
>  };

[snip]

>  static irqreturn_t smc_msg_done_isr(int irq, void *data)
> @@ -129,6 +134,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>  	struct resource res;
>  	struct device_node *np;
>  	u32 func_id;
> +	u64 cap_id;
>  	int ret;

[snip]

> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
> +#ifdef CONFIG_ARM64
> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
> +#else
> +		/* capability-id is 32 bit wide on 32bit machines */
> +		cap_id = readl((void __iomem *)(scmi_info->shmem) + size - 8);
> +#endif

The 32 bit case is defined as a u64 in two places above.

> +
> +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
> +		 *     +-------+
> +		 *     |       |
> +		 *     | shmem |
> +		 *     |       |
> +		 *     |       |
> +		 *     +-------+ <-- (size - 16)
> +		 *     | funcId|
> +		 *     +-------+ <-- (size - 8)
> +		 *     | capId |
> +		 *     +-------+ <-- size
> +		 */

Personally I'd add one more space to the right side of the table after
funcId.

> -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
> -			     &res);
> +	if (scmi_info->qcom_xport)
> +		arm_smccc_1_1_hvc(scmi_info->func_id, cap_id, 0, 0, 0, 0, 0, 0,
> +				  &res);
> +	else
> +		arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0,
> +				     0, 0, &res);

Does it make sense to call this variable qcom_xport? Would hvc_xport be
a more appropriate name?

Brian
  

On 10/2/2023 11:18 AM, Brian Masney wrote:
> On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
>> Currently, the return from the smc call assumes the completion of
>> the scmi request. However this may not be true in virtual platforms
>> that are using hvc doorbell.
>>
>> This change adds a Kconfig to enable the polling for the request
>> completion.
>>
>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>> ---
>>   drivers/firmware/arm_scmi/Kconfig | 14 ++++++++++++++
>>   drivers/firmware/arm_scmi/smc.c   | 15 ++++++++++++++-
>>   2 files changed, 28 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/Kconfig b/drivers/firmware/arm_scmi/Kconfig
>> index ea0f5083ac47..771d60f8319f 100644
>> --- a/drivers/firmware/arm_scmi/Kconfig
>> +++ b/drivers/firmware/arm_scmi/Kconfig
>> @@ -125,6 +125,20 @@ config ARM_SCMI_TRANSPORT_SMC_ATOMIC_ENABLE
>>   	  in atomic context too, at the price of using a number of busy-waiting
>>   	  primitives all over instead. If unsure say N.
>>   
>> +config ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
>> +	bool "Enable polling support for SCMI SMC transport"
>> +	depends on ARM_SCMI_TRANSPORT_SMC
>> +	help
>> +	  Enable completion polling support for SCMI SMC based transport.
>> +
>> +	  If you want the SCMI SMC based transport to poll for the completion,
>> +	  answer Y.
>> +	  Enabling completion polling might be desired in the absence of the a2p
>> +	  irq when the return from smc/hvc call doesn't indicate the completion
>> +	  of the SCMI requests. This might be useful for instances used in
>> +	  virtual platforms.
>> +	  If unsure say N.
>> +
>>   config ARM_SCMI_TRANSPORT_VIRTIO
>>   	bool "SCMI transport based on VirtIO"
>>   	depends on VIRTIO=y || VIRTIO=ARM_SCMI_PROTOCOL
>> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
>> index c193516a254d..0a0b7e401159 100644
>> --- a/drivers/firmware/arm_scmi/smc.c
>> +++ b/drivers/firmware/arm_scmi/smc.c
>> @@ -250,6 +250,16 @@ static void smc_mark_txdone(struct scmi_chan_info *cinfo, int ret,
>>   	smc_channel_lock_release(scmi_info);
>>   }
>>   
>> +#ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
>> +static bool
>> +smc_poll_done(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer)
>> +{
>> +	struct scmi_smc *scmi_info = cinfo->transport_info;
>> +
>> +	return shmem_poll_done(scmi_info->shmem, xfer);
>> +}
>> +#endif
>> +
>>   static const struct scmi_transport_ops scmi_smc_ops = {
>>   	.chan_available = smc_chan_available,
>>   	.chan_setup = smc_chan_setup,
>> @@ -257,6 +267,9 @@ static const struct scmi_transport_ops scmi_smc_ops = {
>>   	.send_message = smc_send_message,
>>   	.mark_txdone = smc_mark_txdone,
>>   	.fetch_response = smc_fetch_response,
>> +#ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION
>> +	.poll_done = smc_poll_done,
>> +#endif
>>   };
>>   
>>   const struct scmi_desc scmi_smc_desc = {
>> @@ -272,6 +285,6 @@ const struct scmi_desc scmi_smc_desc = {
>>   	 * for the issued command will be immmediately ready to be fetched
>>   	 * from the shared memory area.
>>   	 */
>> -	.sync_cmds_completed_on_ret = true,
>> +	.sync_cmds_completed_on_ret = !IS_ENABLED(CONFIG_ARM_SCMI_TRANSPORT_SMC_POLL_COMPLETION),
>>   	.atomic_enabled = IS_ENABLED(CONFIG_ARM_SCMI_TRANSPORT_SMC_ATOMIC_ENABLE),
>  From a Linux distributor viewpoint, it would be nice if this was
> determined at runtime, rather than at compile time. We generate a single
> kernel binary that's used on systems from multiple hardware
> manufacturers. We'd run into an issue if one company required this, but
> another one didn't. We may potentially run into this same type of issue
> with the upstream arm64 defconfig.
>
> Brian
This is a transport dependent property. Either the transport supports 
"completion on return of the smc call" or not. For a given platform, 
this will be fixed for all channels. This is similar to

CONFIG_ARM_SCMI_TRANSPORT_SMC_ATOMIC_ENABLE which is also a Kconfig.
  

On Mon, Oct 02, 2023 at 02:34:06PM -0400, Brian Masney wrote:
> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> > +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
> > +#ifdef CONFIG_ARM64
> > +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
> > +#else
> > +		/* capability-id is 32 bit wide on 32bit machines */
> > +		cap_id = readl((void __iomem *)(scmi_info->shmem) + size - 8);
> > +#endif
> 
> The 32 bit case is defined as a u64 in two places above.

Also should the 32 bit case be 'size - 4' instead of 'size - 8'? Sorry
I just noticed that as soon as I pressed send.

Brian
  

On 10/2/2023 11:34 AM, Brian Masney wrote:
> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
>> This change adds the support for SCMI message exchange on Qualcomm
>> virtual platforms.
>>
>> The hypervisor associates an object-id also known as capability-id
>> with each hvc doorbell object. The capability-id is used to identify the
>> doorbell from the VM's capability namespace, similar to a file-descriptor.
>>
>> The hypervisor, in addition to the function-id, expects the capability-id
>> to be passed in x1 register when HVC call is invoked.
>>
>> The function-id & capability-id are allocated by the hypervisor on bootup
>> and are stored in the shmem region by the firmware before starting Linux.
>>
>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>> ---
>>   drivers/firmware/arm_scmi/driver.c |  1 +
>>   drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
>>   2 files changed, 43 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 87383c05424b..ea344bc6ae49 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
>>   #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
>>   	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
>>   	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
>> +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
>>   #endif
>>   #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
>>   	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
>> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
>> index 0a0b7e401159..94ec07fdc14a 100644
>> --- a/drivers/firmware/arm_scmi/smc.c
>> +++ b/drivers/firmware/arm_scmi/smc.c
>> @@ -50,6 +50,9 @@
>>    * @func_id: smc/hvc call function id
>>    * @param_page: 4K page number of the shmem channel
>>    * @param_offset: Offset within the 4K page of the shmem channel
>> + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
>> + *	    platforms
>> + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
>>    */
>>   
>>   struct scmi_smc {
>> @@ -63,6 +66,8 @@ struct scmi_smc {
>>   	u32 func_id;
>>   	u32 param_page;
>>   	u32 param_offset;
>> +	u64 cap_id;
>> +	bool qcom_xport;
>>   };
> [snip]
>
>>   static irqreturn_t smc_msg_done_isr(int irq, void *data)
>> @@ -129,6 +134,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>   	struct resource res;
>>   	struct device_node *np;
>>   	u32 func_id;
>> +	u64 cap_id;
>>   	int ret;
> [snip]
>
>> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
>> +#ifdef CONFIG_ARM64
>> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
>> +#else
>> +		/* capability-id is 32 bit wide on 32bit machines */
>> +		cap_id = readl((void __iomem *)(scmi_info->shmem) + size - 8);
>> +#endif
> The 32 bit case is defined as a u64 in two places above.

That is done to make sure the size of the structure in memory is not 
architecture dependent. This was recommended in one of the previous 
version of this patch.


>
>> +
>> +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
>> +		 *     +-------+
>> +		 *     |       |
>> +		 *     | shmem |
>> +		 *     |       |
>> +		 *     |       |
>> +		 *     +-------+ <-- (size - 16)
>> +		 *     | funcId|
>> +		 *     +-------+ <-- (size - 8)
>> +		 *     | capId |
>> +		 *     +-------+ <-- size
>> +		 */
> Personally I'd add one more space to the right side of the table after
> funcId.

I could do that but then in 32bit case, you would want one more space 
right after cap-id since it is 32 bit on 32 bit platform. If it helps, I 
can have two lay out one for 32bit and one for 64 bit.


>> -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
>> -			     &res);
>> +	if (scmi_info->qcom_xport)
>> +		arm_smccc_1_1_hvc(scmi_info->func_id, cap_id, 0, 0, 0, 0, 0, 0,
>> +				  &res);
>> +	else
>> +		arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0,
>> +				     0, 0, &res);
> Does it make sense to call this variable qcom_xport? Would hvc_xport be
> a more appropriate name?
>
> Brian

Cap-id is QCOM specific ABI parameter not HVC.

>
  

On 10/2/2023 11:39 AM, Brian Masney wrote:
> On Mon, Oct 02, 2023 at 02:34:06PM -0400, Brian Masney wrote:
>> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
>>> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
>>> +#ifdef CONFIG_ARM64
>>> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
>>> +#else
>>> +		/* capability-id is 32 bit wide on 32bit machines */
>>> +		cap_id = readl((void __iomem *)(scmi_info->shmem) + size - 8);
>>> +#endif
>> The 32 bit case is defined as a u64 in two places above.
> Also should the 32 bit case be 'size - 4' instead of 'size - 8'? Sorry
> I just noticed that as soon as I pressed send.
>
> Brian

I already addressed this in one of your previous comments. We are 
keeping last 16 bytes reserved for these two parameters regardless of 
the architecture.
  

On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
> Currently, the return from the smc call assumes the completion of
> the scmi request. However this may not be true in virtual platforms
> that are using hvc doorbell.
>

Hmm, it is expectation from SMCCC for the fast calls. Is you HVC FID
not a fast call. AFAIK, only TOS use yielding calls. Are you using them
here ? If not, this must complete when the SMC/HVC returns. We added
support for platforms indicating the same via interrupt.

I would like to avoid adding this build config. Why does it require polling ?
Broken firmware ? I would add a compatible for that. Or if the qcom always
wants to do this way, just make it specific to the qcom compatible.

I would avoid a config flag as it needs to be always enabled for single
image and affects other platforms as well. So please drop this change.
If this is absolutely needed, just add additional property which DT
maintainers may not like as it is more like a policy or just make it
compatible specific.

--
Regards,
Sudeep
  

On Mon, Oct 02, 2023 at 10:31:27AM -0700, Nikunj Kela wrote:
> 
> On 9/19/2023 1:56 AM, Sudeep Holla wrote:
> > On Mon, Sep 18, 2023 at 11:54:25AM -0400, Brian Masney wrote:
> > > On Mon, Sep 18, 2023 at 04:15:52PM +0100, Sudeep Holla wrote:
> > > > On Mon, Sep 18, 2023 at 08:01:26AM -0700, Nikunj Kela wrote:
> > > > > Gentle Ping!
> > > > > 
> > > > I will take a look at this later this week. That said, I am unable be
> > > > gauge the urgency based on you ping here. You have shown the same urgency
> > > > last time for a feature that I queued promptly just to know that it was
> > > > abandon within couple of days. So I don't want to rush here simply based
> > > > on the number of pings here. I need to understand that it is really that
> > > > important. For now, I am thinking of skipping even v6.7 just to allow
> > > > some time for Qcom to make up its mind and be absolutely sure this is what
> > > > they *really* want this time.
> > > Hi Sudeep,
> > > 
> > > Red Hat is interested in this patch set. Qualcomm is moving one of their
> > > automotive platforms over to use SCMI and this will appear in that
> > > product.
> > > 
> > Thanks Brian, I trust Redhat over Qcom 😄. I will try to review and enable
> > progress later this week. We can try to target next merge window.
> > 
> > --
> > Regards,
> > Sudeep
> Gentle Ping...

Sorry for the delay, both me and Cristian looking at this now.
  

On Mon, Sep 11, 2023 at 12:43:58PM -0700, Nikunj Kela wrote:
> Introduce compatible "qcom,scmi-hvc-shmem" for SCMI smc/hvc
> transport channel for Qualcomm virtual platforms.
> The compatible mandates a shared memory channel.
> 
> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> ---
>  .../devicetree/bindings/firmware/arm,scmi.yaml       | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> index 8d54ea768d38..4090240f45b1 100644
> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> @@ -45,6 +45,9 @@ properties:
>        - description: SCMI compliant firmware with OP-TEE transport
>          items:
>            - const: linaro,scmi-optee
> +      - description: SCMI compliant firmware with Qualcomm hvc/shmem transport
> +        items:
> +          - const: qcom,scmi-hvc-shmem

Can it be simply "qcom,scmi-smc" for 2 reasons ?
1. We don't support SMC/HVC without shmem, so what is your argument to add
   '-shmem' in the compatible here ?
2. The exact conduit(SMC/HVC) used is detected runtime, so I prefer to keep
  '-smc' instead of '-hvc' in the compatible just to avoid giving an illusion
  that HVC is the conduit chosen here based on the compatible. It can be true
  for other reason but I don't want to mislead here by using HVC.
>
>    interrupts:
>      description:
> @@ -320,6 +323,15 @@ allOf:
>        required:
>          - linaro,optee-channel-id
>  
> +  - if:
> +      properties:
> +        compatible:
> +          contains:
> +            const: qcom,scmi-hvc-shmem
> +    then:
> +      required:
> +        - shmem
> +
>  examples:
>    - |
>      firmware {
> -- 
> 2.17.1
>
  

On Mon, Oct 02, 2023 at 11:42:22AM -0700, Nikunj Kela wrote:
> 
> On 10/2/2023 11:34 AM, Brian Masney wrote:
> > On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> > > This change adds the support for SCMI message exchange on Qualcomm
> > > virtual platforms.
> > > 
> > > The hypervisor associates an object-id also known as capability-id
> > > with each hvc doorbell object. The capability-id is used to identify the
> > > doorbell from the VM's capability namespace, similar to a file-descriptor.
> > > 
> > > The hypervisor, in addition to the function-id, expects the capability-id
> > > to be passed in x1 register when HVC call is invoked.
> > > 
> > > The function-id & capability-id are allocated by the hypervisor on bootup
> > > and are stored in the shmem region by the firmware before starting Linux.
> > > 
> > > Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> > > ---
> > >   drivers/firmware/arm_scmi/driver.c |  1 +
> > >   drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
> > >   2 files changed, 43 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > > index 87383c05424b..ea344bc6ae49 100644
> > > --- a/drivers/firmware/arm_scmi/driver.c
> > > +++ b/drivers/firmware/arm_scmi/driver.c
> > > @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
> > >   #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
> > >   	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> > >   	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
> > > +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
> > >   #endif
> > >   #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
> > >   	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> > > diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> > > index 0a0b7e401159..94ec07fdc14a 100644
> > > --- a/drivers/firmware/arm_scmi/smc.c
> > > +++ b/drivers/firmware/arm_scmi/smc.c
> > > @@ -50,6 +50,9 @@
> > >    * @func_id: smc/hvc call function id
> > >    * @param_page: 4K page number of the shmem channel
> > >    * @param_offset: Offset within the 4K page of the shmem channel
> > > + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
> > > + *	    platforms
> > > + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
> > >    */
> > >   struct scmi_smc {
> > > @@ -63,6 +66,8 @@ struct scmi_smc {
> > >   	u32 func_id;
> > >   	u32 param_page;
> > >   	u32 param_offset;
> > > +	u64 cap_id;
> > > +	bool qcom_xport;
> > >   };
> > [snip]
> > 
> > >   static irqreturn_t smc_msg_done_isr(int irq, void *data)
> > > @@ -129,6 +134,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> > >   	struct resource res;
> > >   	struct device_node *np;
> > >   	u32 func_id;
> > > +	u64 cap_id;
> > >   	int ret;
> > [snip]
> > 
> > > +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
> > > +#ifdef CONFIG_ARM64
> > > +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
> > > +#else
> > > +		/* capability-id is 32 bit wide on 32bit machines */
> > > +		cap_id = readl((void __iomem *)(scmi_info->shmem) + size - 8);
> > > +#endif
> > The 32 bit case is defined as a u64 in two places above.
> 
> That is done to make sure the size of the structure in memory is not
> architecture dependent. This was recommended in one of the previous version
> of this patch.
> 
> 
> > 
> > > +
> > > +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
> > > +		 *     +-------+
> > > +		 *     |       |
> > > +		 *     | shmem |
> > > +		 *     |       |
> > > +		 *     |       |
> > > +		 *     +-------+ <-- (size - 16)
> > > +		 *     | funcId|
> > > +		 *     +-------+ <-- (size - 8)
> > > +		 *     | capId |
> > > +		 *     +-------+ <-- size
> > > +		 */
> > Personally I'd add one more space to the right side of the table after
> > funcId.
> 
> I could do that but then in 32bit case, you would want one more space right
> after cap-id since it is 32 bit on 32 bit platform. If it helps, I can have
> two lay out one for 32bit and one for 64 bit.
>

IIUC, it was just a cosmetic change requested. Instead of this,

		+-------+ <-- (size - 16)
		| funcId|
		+-------+ <-- (size - 8)

something like this, just a extra space after 'funcId' text.

		+--------+ <-- (size - 16)
		| funcId |
		+--------+ <-- (size - 8)
  

On Tue, Oct 03, 2023 at 11:33:17AM +0100, Sudeep Holla wrote:
> On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
> > Currently, the return from the smc call assumes the completion of
> > the scmi request. However this may not be true in virtual platforms
> > that are using hvc doorbell.
> >
> 
> Hmm, it is expectation from SMCCC for the fast calls. Is you HVC FID
> not a fast call. AFAIK, only TOS use yielding calls. Are you using them
> here ? If not, this must complete when the SMC/HVC returns. We added
> support for platforms indicating the same via interrupt.
> 
> I would like to avoid adding this build config. Why does it require polling ?
> Broken firmware ? I would add a compatible for that. Or if the qcom always
> wants to do this way, just make it specific to the qcom compatible.
> 
> I would avoid a config flag as it needs to be always enabled for single
> image and affects other platforms as well. So please drop this change.
> If this is absolutely needed, just add additional property which DT
> maintainers may not like as it is more like a policy or just make it
> compatible specific.
> 

Not sure if it could be acceptable or controversial, BUT if there is the
need somehow to support polling for yielding calls (depending on the location
of the SCMI server), should not we think about doing this by just looking up
dynamically the fast-call bits in the provided FID ?

Why we need another binding, given that the FID is currently already
statically provided by the DT itself (via smc-id) or dynamically by the
hypervisor at setup by the changes in this series and the SMCCC spec
clearly defines how the IDs are supposed to be formed for
fast-atomic-calls ?

This way we could enforce the compliance with the SMCCC spec tooo...
...for sure it would require a bit of work in the core, though, given the
const nature of some of this structures.

Thanks,
Cristian
  

On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> This change adds the support for SCMI message exchange on Qualcomm
> virtual platforms.
> 
> The hypervisor associates an object-id also known as capability-id
> with each hvc doorbell object. The capability-id is used to identify the
> doorbell from the VM's capability namespace, similar to a file-descriptor.
> 
> The hypervisor, in addition to the function-id, expects the capability-id
> to be passed in x1 register when HVC call is invoked.
> 
> The function-id & capability-id are allocated by the hypervisor on bootup
> and are stored in the shmem region by the firmware before starting Linux.
> 
> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> ---
>  drivers/firmware/arm_scmi/driver.c |  1 +
>  drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
>  2 files changed, 43 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> index 87383c05424b..ea344bc6ae49 100644
> --- a/drivers/firmware/arm_scmi/driver.c
> +++ b/drivers/firmware/arm_scmi/driver.c
> @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
>  #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
>  	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
>  	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
> +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
>  #endif
>  #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
>  	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> index 0a0b7e401159..94ec07fdc14a 100644
> --- a/drivers/firmware/arm_scmi/smc.c
> +++ b/drivers/firmware/arm_scmi/smc.c
> @@ -50,6 +50,9 @@
>   * @func_id: smc/hvc call function id
>   * @param_page: 4K page number of the shmem channel
>   * @param_offset: Offset within the 4K page of the shmem channel
> + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
> + *	    platforms
> + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
>   */
>  
>  struct scmi_smc {
> @@ -63,6 +66,8 @@ struct scmi_smc {
>  	u32 func_id;
>  	u32 param_page;
>  	u32 param_offset;
> +	u64 cap_id;

Can it be unsigned long instead so that it just works for both 32 and 64 bit.

> +	bool qcom_xport;

Do we really need this ?

>  };
>  
>  static irqreturn_t smc_msg_done_isr(int irq, void *data)
> @@ -129,6 +134,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>  	struct resource res;
>  	struct device_node *np;
>  	u32 func_id;
> +	u64 cap_id;

Ditto..

>  	int ret;
>  
>  	if (!tx)
> @@ -158,9 +164,34 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>  		return -EADDRNOTAVAIL;
>  	}
>  
> -	ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
> -	if (ret < 0)
> -		return ret;
> +	if (of_device_is_compatible(dev->of_node, "qcom,scmi-hvc-shmem")) {
> +		scmi_info->qcom_xport = true;
> +
> +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
> +		 *     +-------+
> +		 *     |       |
> +		 *     | shmem |
> +		 *     |       |
> +		 *     |       |
> +		 *     +-------+ <-- (size - 16)
> +		 *     | funcId|
> +		 *     +-------+ <-- (size - 8)
> +		 *     | capId |
> +		 *     +-------+ <-- size
> +		 */
> +
> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);

So unlike 'arm,scmi-smc', you don't want 'arm,smc-id' in the DT ? Any
particular reason ? Just to get both FID and cap ID from shmem ?

> +#ifdef CONFIG_ARM64

I would rather make this arch agnostic using CONFIG_64BIT

> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);

Do you need __iomem typecast here ? Is scmi_info->shmem not already __iomem ?
Also scmi_info->shmem is ioremapped just few steps above and you are using
read* here, is that safe ?

> +#else
> +		/* capability-id is 32 bit wide on 32bit machines */
> +		cap_id = rieadl((void __iomem *)(scmi_info->shmem) + size - 8);

Other thought once you move for u64 to unsigned long you need not have
#ifdeffery, just do copy of sizeof(unsigned long)

> +#endif
> +	} else {
> +		ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
> +		if (ret < 0)
> +			return ret;
> +	}
>  
>  	if (of_device_is_compatible(dev->of_node, "arm,scmi-smc-param")) {
>  		scmi_info->param_page = SHMEM_PAGE(res.start);
> @@ -184,6 +215,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>  	}
>  
>  	scmi_info->func_id = func_id;
> +	scmi_info->cap_id = cap_id;
>  	scmi_info->cinfo = cinfo;
>  	smc_channel_lock_init(scmi_info);
>  	cinfo->transport_info = scmi_info;
> @@ -213,6 +245,7 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>  	struct arm_smccc_res res;
>  	unsigned long page = scmi_info->param_page;
>  	unsigned long offset = scmi_info->param_offset;
> +	unsigned long cap_id = (unsigned long)scmi_info->cap_id;
>  
>  	/*
>  	 * Channel will be released only once response has been
> @@ -222,8 +255,12 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>  
>  	shmem_tx_prepare(scmi_info->shmem, xfer, cinfo);
>  
> -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
> -			     &res);
> +	if (scmi_info->qcom_xport)

Just make sure cap_id is set only for qcom and just use that as your flag.
No point in setting always true scmi_info->qcom_xport and using it here.

> +		arm_smccc_1_1_hvc(scmi_info->func_id, cap_id, 0, 0, 0, 0, 0, 0,
> +				  &res);
> +	else
> +		arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0,
> +				     0, 0, &res);
>  
>  	/* Only SMCCC_RET_NOT_SUPPORTED is valid error code */
>  	if (res.a0) {
> -- 
> 2.17.1
>
  

On 10/3/2023 3:33 AM, Sudeep Holla wrote:
> On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
>> Currently, the return from the smc call assumes the completion of
>> the scmi request. However this may not be true in virtual platforms
>> that are using hvc doorbell.
>>
> Hmm, it is expectation from SMCCC for the fast calls. Is you HVC FID
> not a fast call. AFAIK, only TOS use yielding calls. Are you using them
> here ? If not, this must complete when the SMC/HVC returns. We added
> support for platforms indicating the same via interrupt.
>
> I would like to avoid adding this build config. Why does it require polling ?
> Broken firmware ? I would add a compatible for that. Or if the qcom always
> wants to do this way, just make it specific to the qcom compatible.
>
> I would avoid a config flag as it needs to be always enabled for single
> image and affects other platforms as well. So please drop this change.
> If this is absolutely needed, just add additional property which DT
> maintainers may not like as it is more like a policy or just make it
> compatible specific.
>
> --
> Regards,
> Sudeep
We are using Fast call FID. We are using completion IRQ for all the scmi 
instances except one where we need to communicate with the server when 
GIC is in suspended state in HLOS. We will need to poll the channel for 
completion in that use case. I am open to suggestions.
  

On 10/3/2023 3:44 AM, Sudeep Holla wrote:
> On Mon, Sep 11, 2023 at 12:43:58PM -0700, Nikunj Kela wrote:
>> Introduce compatible "qcom,scmi-hvc-shmem" for SCMI smc/hvc
>> transport channel for Qualcomm virtual platforms.
>> The compatible mandates a shared memory channel.
>>
>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>> Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>> ---
>>   .../devicetree/bindings/firmware/arm,scmi.yaml       | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>> index 8d54ea768d38..4090240f45b1 100644
>> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>> @@ -45,6 +45,9 @@ properties:
>>         - description: SCMI compliant firmware with OP-TEE transport
>>           items:
>>             - const: linaro,scmi-optee
>> +      - description: SCMI compliant firmware with Qualcomm hvc/shmem transport
>> +        items:
>> +          - const: qcom,scmi-hvc-shmem
> Can it be simply "qcom,scmi-smc" for 2 reasons ?
> 1. We don't support SMC/HVC without shmem, so what is your argument to add
>     '-shmem' in the compatible here ?

In our platforms, there are multiple ways to allocate memory. One is 
preallocated shmem as used here, another is dynamically by hypervisor 
APIs. shmem was to just to indicate it is preallocated.


> 2. The exact conduit(SMC/HVC) used is detected runtime, so I prefer to keep
>    '-smc' instead of '-hvc' in the compatible just to avoid giving an illusion
>    that HVC is the conduit chosen here based on the compatible. It can be true
>    for other reason but I don't want to mislead here by using HVC.

IUUC, currently, conduit comes from PSCI dt node. We have been using smc 
for PSCI but want to use hvc here. That being said, I am fine to explore 
if we can change PSCI to use hvc too.


>>     interrupts:
>>       description:
>> @@ -320,6 +323,15 @@ allOf:
>>         required:
>>           - linaro,optee-channel-id
>>   
>> +  - if:
>> +      properties:
>> +        compatible:
>> +          contains:
>> +            const: qcom,scmi-hvc-shmem
>> +    then:
>> +      required:
>> +        - shmem
>> +
>>   examples:
>>     - |
>>       firmware {
>> -- 
>> 2.17.1
>>
  

On 10/3/2023 4:19 AM, Sudeep Holla wrote:
> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
>> This change adds the support for SCMI message exchange on Qualcomm
>> virtual platforms.
>>
>> The hypervisor associates an object-id also known as capability-id
>> with each hvc doorbell object. The capability-id is used to identify the
>> doorbell from the VM's capability namespace, similar to a file-descriptor.
>>
>> The hypervisor, in addition to the function-id, expects the capability-id
>> to be passed in x1 register when HVC call is invoked.
>>
>> The function-id & capability-id are allocated by the hypervisor on bootup
>> and are stored in the shmem region by the firmware before starting Linux.
>>
>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>> ---
>>   drivers/firmware/arm_scmi/driver.c |  1 +
>>   drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
>>   2 files changed, 43 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>> index 87383c05424b..ea344bc6ae49 100644
>> --- a/drivers/firmware/arm_scmi/driver.c
>> +++ b/drivers/firmware/arm_scmi/driver.c
>> @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
>>   #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
>>   	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
>>   	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
>> +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
>>   #endif
>>   #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
>>   	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
>> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
>> index 0a0b7e401159..94ec07fdc14a 100644
>> --- a/drivers/firmware/arm_scmi/smc.c
>> +++ b/drivers/firmware/arm_scmi/smc.c
>> @@ -50,6 +50,9 @@
>>    * @func_id: smc/hvc call function id
>>    * @param_page: 4K page number of the shmem channel
>>    * @param_offset: Offset within the 4K page of the shmem channel
>> + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
>> + *	    platforms
>> + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
>>    */
>>   
>>   struct scmi_smc {
>> @@ -63,6 +66,8 @@ struct scmi_smc {
>>   	u32 func_id;
>>   	u32 param_page;
>>   	u32 param_offset;
>> +	u64 cap_id;
> Can it be unsigned long instead so that it just works for both 32 and 64 bit.

My first version of this patch was ulong but Bjorn suggested to make 
this structure size fixed i.e. architecture independent. Hence changed 
it to u64. If you are ok with ulong, I can change it back to ulong.


>
>> +	bool qcom_xport;
> Do we really need this ?

Not if we initialize it with a negative value since 0 is a valid value 
for cap-id.


>
>>   };
>>   
>>   static irqreturn_t smc_msg_done_isr(int irq, void *data)
>> @@ -129,6 +134,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>   	struct resource res;
>>   	struct device_node *np;
>>   	u32 func_id;
>> +	u64 cap_id;
> Ditto..

Answered in earlier comment.


>>   	int ret;
>>   
>>   	if (!tx)
>> @@ -158,9 +164,34 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>   		return -EADDRNOTAVAIL;
>>   	}
>>   
>> -	ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
>> -	if (ret < 0)
>> -		return ret;
>> +	if (of_device_is_compatible(dev->of_node, "qcom,scmi-hvc-shmem")) {
>> +		scmi_info->qcom_xport = true;
>> +
>> +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
>> +		 *     +-------+
>> +		 *     |       |
>> +		 *     | shmem |
>> +		 *     |       |
>> +		 *     |       |
>> +		 *     +-------+ <-- (size - 16)
>> +		 *     | funcId|
>> +		 *     +-------+ <-- (size - 8)
>> +		 *     | capId |
>> +		 *     +-------+ <-- size
>> +		 */
>> +
>> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
> So unlike 'arm,scmi-smc', you don't want 'arm,smc-id' in the DT ? Any
> particular reason ? Just to get both FID and cap ID from shmem ?

I could use smc-id binding for func-id, it's just two parameters will 
come from two different places so thought of keeping everything at one 
place to maintain consistency.  Since DT can't take cap-id, I decided to 
move func-id. I am fine if you want me to use smc-id binding.


>> +#ifdef CONFIG_ARM64
> I would rather make this arch agnostic using CONFIG_64BIT
ok.
>
>> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
> Do you need __iomem typecast here ? Is scmi_info->shmem not already __iomem ?
> Also scmi_info->shmem is ioremapped just few steps above and you are using
> read* here, is that safe ?

I saw some compilation warnings without __iomem. I will use ioread* API 
instead of read*.


>
>> +#else
>> +		/* capability-id is 32 bit wide on 32bit machines */
>> +		cap_id = rieadl((void __iomem *)(scmi_info->shmem) + size - 8);
> Other thought once you move for u64 to unsigned long you need not have
> #ifdeffery, just do copy of sizeof(unsigned long)
Right, my first version was like that only.
>
>> +#endif
>> +	} else {
>> +		ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
>> +		if (ret < 0)
>> +			return ret;
>> +	}
>>   
>>   	if (of_device_is_compatible(dev->of_node, "arm,scmi-smc-param")) {
>>   		scmi_info->param_page = SHMEM_PAGE(res.start);
>> @@ -184,6 +215,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>   	}
>>   
>>   	scmi_info->func_id = func_id;
>> +	scmi_info->cap_id = cap_id;
>>   	scmi_info->cinfo = cinfo;
>>   	smc_channel_lock_init(scmi_info);
>>   	cinfo->transport_info = scmi_info;
>> @@ -213,6 +245,7 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>>   	struct arm_smccc_res res;
>>   	unsigned long page = scmi_info->param_page;
>>   	unsigned long offset = scmi_info->param_offset;
>> +	unsigned long cap_id = (unsigned long)scmi_info->cap_id;
>>   
>>   	/*
>>   	 * Channel will be released only once response has been
>> @@ -222,8 +255,12 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>>   
>>   	shmem_tx_prepare(scmi_info->shmem, xfer, cinfo);
>>   
>> -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
>> -			     &res);
>> +	if (scmi_info->qcom_xport)
> Just make sure cap_id is set only for qcom and just use that as your flag.
> No point in setting always true scmi_info->qcom_xport and using it here.
ok, I can remove that. Though 0 is a valid value for cap-id so will have 
to init cap-id with a negative value.
>
>> +		arm_smccc_1_1_hvc(scmi_info->func_id, cap_id, 0, 0, 0, 0, 0, 0,
>> +				  &res);
>> +	else
>> +		arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0,
>> +				     0, 0, &res);
>>   
>>   	/* Only SMCCC_RET_NOT_SUPPORTED is valid error code */
>>   	if (res.a0) {
>> -- 
>> 2.17.1
>>
  

On Tue, Oct 03, 2023 at 08:59:45AM -0700, Nikunj Kela wrote:
> 
> On 10/3/2023 3:44 AM, Sudeep Holla wrote:
> > On Mon, Sep 11, 2023 at 12:43:58PM -0700, Nikunj Kela wrote:
> > > Introduce compatible "qcom,scmi-hvc-shmem" for SCMI smc/hvc
> > > transport channel for Qualcomm virtual platforms.
> > > The compatible mandates a shared memory channel.
> > > 
> > > Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> > > Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> > > ---
> > >   .../devicetree/bindings/firmware/arm,scmi.yaml       | 12 ++++++++++++
> > >   1 file changed, 12 insertions(+)
> > > 
> > > diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > index 8d54ea768d38..4090240f45b1 100644
> > > --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > @@ -45,6 +45,9 @@ properties:
> > >         - description: SCMI compliant firmware with OP-TEE transport
> > >           items:
> > >             - const: linaro,scmi-optee
> > > +      - description: SCMI compliant firmware with Qualcomm hvc/shmem transport
> > > +        items:
> > > +          - const: qcom,scmi-hvc-shmem
> > Can it be simply "qcom,scmi-smc" for 2 reasons ?
> > 1. We don't support SMC/HVC without shmem, so what is your argument to add
> >     '-shmem' in the compatible here ?
> 
> In our platforms, there are multiple ways to allocate memory. One is
> preallocated shmem as used here, another is dynamically by hypervisor APIs.
> shmem was to just to indicate it is preallocated.
>

Let us keep it without shmem. If it is dynamically allocated, you must not
need another compatible as you can check it at the runtime.

> 
> > 2. The exact conduit(SMC/HVC) used is detected runtime, so I prefer to keep
> >    '-smc' instead of '-hvc' in the compatible just to avoid giving an illusion
> >    that HVC is the conduit chosen here based on the compatible. It can be true
> >    for other reason but I don't want to mislead here by using HVC.
> 
> IUUC, currently, conduit comes from PSCI dt node. We have been using smc for
> PSCI but want to use hvc here. That being said, I am fine to explore if we
> can change PSCI to use hvc too.
>

I think only OPTEE has explicit conduit other than PSCI and it is continued
for legacy/compatibility reasons IIUC and IIRC. Anything else depends on
the conduit used by PSCI to be consistent. So yes you need to use what the
PSCI conduit is and you don't need the extra information from the DT either
as new property or in the compatible.
  

On Tue, Oct 03, 2023 at 09:16:27AM -0700, Nikunj Kela wrote:
> 
> On 10/3/2023 4:19 AM, Sudeep Holla wrote:
> > On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> > > This change adds the support for SCMI message exchange on Qualcomm
> > > virtual platforms.
> > > 
> > > The hypervisor associates an object-id also known as capability-id
> > > with each hvc doorbell object. The capability-id is used to identify the
> > > doorbell from the VM's capability namespace, similar to a file-descriptor.
> > > 
> > > The hypervisor, in addition to the function-id, expects the capability-id
> > > to be passed in x1 register when HVC call is invoked.
> > > 
> > > The function-id & capability-id are allocated by the hypervisor on bootup
> > > and are stored in the shmem region by the firmware before starting Linux.
> > > 
> > > Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
> > > ---
> > >   drivers/firmware/arm_scmi/driver.c |  1 +
> > >   drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
> > >   2 files changed, 43 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
> > > index 87383c05424b..ea344bc6ae49 100644
> > > --- a/drivers/firmware/arm_scmi/driver.c
> > > +++ b/drivers/firmware/arm_scmi/driver.c
> > > @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
> > >   #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
> > >   	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
> > >   	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
> > > +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
> > >   #endif
> > >   #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
> > >   	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
> > > diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> > > index 0a0b7e401159..94ec07fdc14a 100644
> > > --- a/drivers/firmware/arm_scmi/smc.c
> > > +++ b/drivers/firmware/arm_scmi/smc.c
> > > @@ -50,6 +50,9 @@
> > >    * @func_id: smc/hvc call function id
> > >    * @param_page: 4K page number of the shmem channel
> > >    * @param_offset: Offset within the 4K page of the shmem channel
> > > + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
> > > + *	    platforms
> > > + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
> > >    */
> > >   struct scmi_smc {
> > > @@ -63,6 +66,8 @@ struct scmi_smc {
> > >   	u32 func_id;
> > >   	u32 param_page;
> > >   	u32 param_offset;
> > > +	u64 cap_id;
> > Can it be unsigned long instead so that it just works for both 32 and 64 bit.
> 
> My first version of this patch was ulong but Bjorn suggested to make this
> structure size fixed i.e. architecture independent. Hence changed it to u64.
> If you are ok with ulong, I can change it back to ulong.
>

SMCCC pre-v1.2 used the common structure in that way. I don't see any issue
with that. I haven't followed Bjorn suggestions/comments though.

> 
> > 
> > > +	bool qcom_xport;
> > Do we really need this ?
> 
> Not if we initialize it with a negative value since 0 is a valid value for
> cap-id.
>

Fine with negative value(-EINVAL may be).

> > >   	int ret;
> > >   	if (!tx)
> > > @@ -158,9 +164,34 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> > >   		return -EADDRNOTAVAIL;
> > >   	}
> > > -	ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
> > > -	if (ret < 0)
> > > -		return ret;
> > > +	if (of_device_is_compatible(dev->of_node, "qcom,scmi-hvc-shmem")) {
> > > +		scmi_info->qcom_xport = true;
> > > +
> > > +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
> > > +		 *     +-------+
> > > +		 *     |       |
> > > +		 *     | shmem |
> > > +		 *     |       |
> > > +		 *     |       |
> > > +		 *     +-------+ <-- (size - 16)
> > > +		 *     | funcId|
> > > +		 *     +-------+ <-- (size - 8)
> > > +		 *     | capId |
> > > +		 *     +-------+ <-- size
> > > +		 */
> > > +
> > > +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
> > So unlike 'arm,scmi-smc', you don't want 'arm,smc-id' in the DT ? Any
> > particular reason ? Just to get both FID and cap ID from shmem ?
>

I am fine either way. If you use from DT(via arm,smc-id), then "qcom,scmi"
can be just addition compatible that expects you to read cap-id from the
shmem. May need adjustment in the binding as you allow both
"qcom,scmi-smc", "arm,scmi-smc". I will leave the details to you.

> I could use smc-id binding for func-id, it's just two parameters will come
> from two different places so thought of keeping everything at one place to
> maintain consistency.  Since DT can't take cap-id, I decided to move
> func-id. I am fine if you want me to use smc-id binding.
>

Up to you. If you want to make "qcom,scmi-smc" and "arm,scmi-smc"
compatible in way in that way or you can keep it incompatible as you have
proposed in this patch set.

> 
> > > +#ifdef CONFIG_ARM64
> > I would rather make this arch agnostic using CONFIG_64BIT
> ok.
> > 
> > > +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
> > Do you need __iomem typecast here ? Is scmi_info->shmem not already __iomem ?
> > Also scmi_info->shmem is ioremapped just few steps above and you are using
> > read* here, is that safe ?
> 
> I saw some compilation warnings without __iomem. I will use ioread* API
> instead of read*.
>

That was the clue that you were using __iomem with read* calls IMO.

> 
> > 
> > > +#else
> > > +		/* capability-id is 32 bit wide on 32bit machines */
> > > +		cap_id = rieadl((void __iomem *)(scmi_info->shmem) + size - 8);
> > Other thought once you move for u64 to unsigned long you need not have
> > #ifdeffery, just do copy of sizeof(unsigned long)
> Right, my first version was like that only.

OK

> > 
> > > +#endif
> > > +	} else {
> > > +		ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
> > > +		if (ret < 0)
> > > +			return ret;
> > > +	}
> > >   	if (of_device_is_compatible(dev->of_node, "arm,scmi-smc-param")) {
> > >   		scmi_info->param_page = SHMEM_PAGE(res.start);
> > > @@ -184,6 +215,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
> > >   	}
> > >   	scmi_info->func_id = func_id;
> > > +	scmi_info->cap_id = cap_id;
> > >   	scmi_info->cinfo = cinfo;
> > >   	smc_channel_lock_init(scmi_info);
> > >   	cinfo->transport_info = scmi_info;
> > > @@ -213,6 +245,7 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
> > >   	struct arm_smccc_res res;
> > >   	unsigned long page = scmi_info->param_page;
> > >   	unsigned long offset = scmi_info->param_offset;
> > > +	unsigned long cap_id = (unsigned long)scmi_info->cap_id;
> > >   	/*
> > >   	 * Channel will be released only once response has been
> > > @@ -222,8 +255,12 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
> > >   	shmem_tx_prepare(scmi_info->shmem, xfer, cinfo);
> > > -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
> > > -			     &res);
> > > +	if (scmi_info->qcom_xport)
> > Just make sure cap_id is set only for qcom and just use that as your flag.
> > No point in setting always true scmi_info->qcom_xport and using it here.
> ok, I can remove that. Though 0 is a valid value for cap-id so will have to
> init cap-id with a negative value.

Yes as mentioned above.
  

On Tue, Oct 03, 2023 at 08:53:20AM -0700, Nikunj Kela wrote:
> 
> On 10/3/2023 3:33 AM, Sudeep Holla wrote:
> > On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
> > > Currently, the return from the smc call assumes the completion of
> > > the scmi request. However this may not be true in virtual platforms
> > > that are using hvc doorbell.
> > > 
> > Hmm, it is expectation from SMCCC for the fast calls. Is you HVC FID
> > not a fast call. AFAIK, only TOS use yielding calls. Are you using them
> > here ? If not, this must complete when the SMC/HVC returns. We added
> > support for platforms indicating the same via interrupt.
> > 
> > I would like to avoid adding this build config. Why does it require polling ?
> > Broken firmware ? I would add a compatible for that. Or if the qcom always
> > wants to do this way, just make it specific to the qcom compatible.
> > 
> > I would avoid a config flag as it needs to be always enabled for single
> > image and affects other platforms as well. So please drop this change.
> > If this is absolutely needed, just add additional property which DT
> > maintainers may not like as it is more like a policy or just make it
> > compatible specific.
> > 
> > --
> > Regards,
> > Sudeep
> We are using Fast call FID. We are using completion IRQ for all the scmi
> instances except one where we need to communicate with the server when GIC
> is in suspended state in HLOS. We will need to poll the channel for
> completion in that use case. I am open to suggestions.

IIUC, for the sake of that one corner case, you have added the polling
Kconfig and will be enabled for all the case and even on other platforms
in a single Image. I think we could be something better, no ?

Please share details on that one corner case.
Is it in the scmi drivers already ? If so, specifics please.
If no, again provide details on how you plan to use. We do have ways
to make a polling call, but haven't mixed it with interrupt based calls
for a reason, but we can revisit if it makes sense.

--
Regards,
Sudeep
  

On 10/4/2023 9:06 AM, Sudeep Holla wrote:
> On Tue, Oct 03, 2023 at 09:16:27AM -0700, Nikunj Kela wrote:
>> On 10/3/2023 4:19 AM, Sudeep Holla wrote:
>>> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
>>>> This change adds the support for SCMI message exchange on Qualcomm
>>>> virtual platforms.
>>>>
>>>> The hypervisor associates an object-id also known as capability-id
>>>> with each hvc doorbell object. The capability-id is used to identify the
>>>> doorbell from the VM's capability namespace, similar to a file-descriptor.
>>>>
>>>> The hypervisor, in addition to the function-id, expects the capability-id
>>>> to be passed in x1 register when HVC call is invoked.
>>>>
>>>> The function-id & capability-id are allocated by the hypervisor on bootup
>>>> and are stored in the shmem region by the firmware before starting Linux.
>>>>
>>>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>>>> ---
>>>>    drivers/firmware/arm_scmi/driver.c |  1 +
>>>>    drivers/firmware/arm_scmi/smc.c    | 47 ++++++++++++++++++++++++++----
>>>>    2 files changed, 43 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
>>>> index 87383c05424b..ea344bc6ae49 100644
>>>> --- a/drivers/firmware/arm_scmi/driver.c
>>>> +++ b/drivers/firmware/arm_scmi/driver.c
>>>> @@ -2915,6 +2915,7 @@ static const struct of_device_id scmi_of_match[] = {
>>>>    #ifdef CONFIG_ARM_SCMI_TRANSPORT_SMC
>>>>    	{ .compatible = "arm,scmi-smc", .data = &scmi_smc_desc},
>>>>    	{ .compatible = "arm,scmi-smc-param", .data = &scmi_smc_desc},
>>>> +	{ .compatible = "qcom,scmi-hvc-shmem", .data = &scmi_smc_desc},
>>>>    #endif
>>>>    #ifdef CONFIG_ARM_SCMI_TRANSPORT_VIRTIO
>>>>    	{ .compatible = "arm,scmi-virtio", .data = &scmi_virtio_desc},
>>>> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
>>>> index 0a0b7e401159..94ec07fdc14a 100644
>>>> --- a/drivers/firmware/arm_scmi/smc.c
>>>> +++ b/drivers/firmware/arm_scmi/smc.c
>>>> @@ -50,6 +50,9 @@
>>>>     * @func_id: smc/hvc call function id
>>>>     * @param_page: 4K page number of the shmem channel
>>>>     * @param_offset: Offset within the 4K page of the shmem channel
>>>> + * @cap_id: hvc doorbell's capability id to be used on Qualcomm virtual
>>>> + *	    platforms
>>>> + * @qcom_xport: Flag to indicate the transport on Qualcomm virtual platforms
>>>>     */
>>>>    struct scmi_smc {
>>>> @@ -63,6 +66,8 @@ struct scmi_smc {
>>>>    	u32 func_id;
>>>>    	u32 param_page;
>>>>    	u32 param_offset;
>>>> +	u64 cap_id;
>>> Can it be unsigned long instead so that it just works for both 32 and 64 bit.
>> My first version of this patch was ulong but Bjorn suggested to make this
>> structure size fixed i.e. architecture independent. Hence changed it to u64.
>> If you are ok with ulong, I can change it back to ulong.
>>
> SMCCC pre-v1.2 used the common structure in that way. I don't see any issue
> with that. I haven't followed Bjorn suggestions/comments though.
Ok.
>>>> +	bool qcom_xport;
>>> Do we really need this ?
>> Not if we initialize it with a negative value since 0 is a valid value for
>> cap-id.
>>
> Fine with negative value(-EINVAL may be).
Ok.
>>>>    	int ret;
>>>>    	if (!tx)
>>>> @@ -158,9 +164,34 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>>>    		return -EADDRNOTAVAIL;
>>>>    	}
>>>> -	ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
>>>> -	if (ret < 0)
>>>> -		return ret;
>>>> +	if (of_device_is_compatible(dev->of_node, "qcom,scmi-hvc-shmem")) {
>>>> +		scmi_info->qcom_xport = true;
>>>> +
>>>> +		/* The func-id & capability-id are kept in last 16 bytes of shmem.
>>>> +		 *     +-------+
>>>> +		 *     |       |
>>>> +		 *     | shmem |
>>>> +		 *     |       |
>>>> +		 *     |       |
>>>> +		 *     +-------+ <-- (size - 16)
>>>> +		 *     | funcId|
>>>> +		 *     +-------+ <-- (size - 8)
>>>> +		 *     | capId |
>>>> +		 *     +-------+ <-- size
>>>> +		 */
>>>> +
>>>> +		func_id = readl((void __iomem *)(scmi_info->shmem) + size - 16);
>>> So unlike 'arm,scmi-smc', you don't want 'arm,smc-id' in the DT ? Any
>>> particular reason ? Just to get both FID and cap ID from shmem ?
> I am fine either way. If you use from DT(via arm,smc-id), then "qcom,scmi"
> can be just addition compatible that expects you to read cap-id from the
> shmem. May need adjustment in the binding as you allow both
> "qcom,scmi-smc", "arm,scmi-smc". I will leave the details to you.
Ok.
>> I could use smc-id binding for func-id, it's just two parameters will come
>> from two different places so thought of keeping everything at one place to
>> maintain consistency.  Since DT can't take cap-id, I decided to move
>> func-id. I am fine if you want me to use smc-id binding.
>>
> Up to you. If you want to make "qcom,scmi-smc" and "arm,scmi-smc"
> compatible in way in that way or you can keep it incompatible as you have
> proposed in this patch set.
Ok.
>>>> +#ifdef CONFIG_ARM64
>>> I would rather make this arch agnostic using CONFIG_64BIT
>> ok.
>>>> +		cap_id = readq((void __iomem *)(scmi_info->shmem) + size - 8);
>>> Do you need __iomem typecast here ? Is scmi_info->shmem not already __iomem ?
>>> Also scmi_info->shmem is ioremapped just few steps above and you are using
>>> read* here, is that safe ?
>> I saw some compilation warnings without __iomem. I will use ioread* API
>> instead of read*.
>>
> That was the clue that you were using __iomem with read* calls IMO.
Ok.
>>>> +#else
>>>> +		/* capability-id is 32 bit wide on 32bit machines */
>>>> +		cap_id = rieadl((void __iomem *)(scmi_info->shmem) + size - 8);
>>> Other thought once you move for u64 to unsigned long you need not have
>>> #ifdeffery, just do copy of sizeof(unsigned long)
>> Right, my first version was like that only.
> OK
>
>>>> +#endif
>>>> +	} else {
>>>> +		ret = of_property_read_u32(dev->of_node, "arm,smc-id", &func_id);
>>>> +		if (ret < 0)
>>>> +			return ret;
>>>> +	}
>>>>    	if (of_device_is_compatible(dev->of_node, "arm,scmi-smc-param")) {
>>>>    		scmi_info->param_page = SHMEM_PAGE(res.start);
>>>> @@ -184,6 +215,7 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
>>>>    	}
>>>>    	scmi_info->func_id = func_id;
>>>> +	scmi_info->cap_id = cap_id;
>>>>    	scmi_info->cinfo = cinfo;
>>>>    	smc_channel_lock_init(scmi_info);
>>>>    	cinfo->transport_info = scmi_info;
>>>> @@ -213,6 +245,7 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>>>>    	struct arm_smccc_res res;
>>>>    	unsigned long page = scmi_info->param_page;
>>>>    	unsigned long offset = scmi_info->param_offset;
>>>> +	unsigned long cap_id = (unsigned long)scmi_info->cap_id;
>>>>    	/*
>>>>    	 * Channel will be released only once response has been
>>>> @@ -222,8 +255,12 @@ static int smc_send_message(struct scmi_chan_info *cinfo,
>>>>    	shmem_tx_prepare(scmi_info->shmem, xfer, cinfo);
>>>> -	arm_smccc_1_1_invoke(scmi_info->func_id, page, offset, 0, 0, 0, 0, 0,
>>>> -			     &res);
>>>> +	if (scmi_info->qcom_xport)
>>> Just make sure cap_id is set only for qcom and just use that as your flag.
>>> No point in setting always true scmi_info->qcom_xport and using it here.
>> ok, I can remove that. Though 0 is a valid value for cap-id so will have to
>> init cap-id with a negative value.
> Yes as mentioned above.
Ok.
>
  

On 10/4/2023 9:11 AM, Sudeep Holla wrote:
> On Tue, Oct 03, 2023 at 08:53:20AM -0700, Nikunj Kela wrote:
>> On 10/3/2023 3:33 AM, Sudeep Holla wrote:
>>> On Mon, Sep 11, 2023 at 12:43:56PM -0700, Nikunj Kela wrote:
>>>> Currently, the return from the smc call assumes the completion of
>>>> the scmi request. However this may not be true in virtual platforms
>>>> that are using hvc doorbell.
>>>>
>>> Hmm, it is expectation from SMCCC for the fast calls. Is you HVC FID
>>> not a fast call. AFAIK, only TOS use yielding calls. Are you using them
>>> here ? If not, this must complete when the SMC/HVC returns. We added
>>> support for platforms indicating the same via interrupt.
>>>
>>> I would like to avoid adding this build config. Why does it require polling ?
>>> Broken firmware ? I would add a compatible for that. Or if the qcom always
>>> wants to do this way, just make it specific to the qcom compatible.
>>>
>>> I would avoid a config flag as it needs to be always enabled for single
>>> image and affects other platforms as well. So please drop this change.
>>> If this is absolutely needed, just add additional property which DT
>>> maintainers may not like as it is more like a policy or just make it
>>> compatible specific.
>>>
>>> --
>>> Regards,
>>> Sudeep
>> We are using Fast call FID. We are using completion IRQ for all the scmi
>> instances except one where we need to communicate with the server when GIC
>> is in suspended state in HLOS. We will need to poll the channel for
>> completion in that use case. I am open to suggestions.
> IIUC, for the sake of that one corner case, you have added the polling
> Kconfig and will be enabled for all the case and even on other platforms
> in a single Image. I think we could be something better, no ?
>
> Please share details on that one corner case.
> Is it in the scmi drivers already ? If so, specifics please.
> If no, again provide details on how you plan to use. We do have ways
> to make a polling call, but haven't mixed it with interrupt based calls
> for a reason, but we can revisit if it makes sense.
>
> --
> Regards,
> Sudeep

Ok. I will discard this patch for now from this series and will explore 
alternative ways instead of polling that might work in our usecase. If 
required, will provide you with more details in a separate patch. Thanks!
  

On 10/4/2023 8:53 AM, Sudeep Holla wrote:
> On Tue, Oct 03, 2023 at 08:59:45AM -0700, Nikunj Kela wrote:
>> On 10/3/2023 3:44 AM, Sudeep Holla wrote:
>>> On Mon, Sep 11, 2023 at 12:43:58PM -0700, Nikunj Kela wrote:
>>>> Introduce compatible "qcom,scmi-hvc-shmem" for SCMI smc/hvc
>>>> transport channel for Qualcomm virtual platforms.
>>>> The compatible mandates a shared memory channel.
>>>>
>>>> Signed-off-by: Nikunj Kela <quic_nkela@quicinc.com>
>>>> Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>>>> ---
>>>>    .../devicetree/bindings/firmware/arm,scmi.yaml       | 12 ++++++++++++
>>>>    1 file changed, 12 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>>>> index 8d54ea768d38..4090240f45b1 100644
>>>> --- a/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>>>> +++ b/Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>>>> @@ -45,6 +45,9 @@ properties:
>>>>          - description: SCMI compliant firmware with OP-TEE transport
>>>>            items:
>>>>              - const: linaro,scmi-optee
>>>> +      - description: SCMI compliant firmware with Qualcomm hvc/shmem transport
>>>> +        items:
>>>> +          - const: qcom,scmi-hvc-shmem
>>> Can it be simply "qcom,scmi-smc" for 2 reasons ?
>>> 1. We don't support SMC/HVC without shmem, so what is your argument to add
>>>      '-shmem' in the compatible here ?
>> In our platforms, there are multiple ways to allocate memory. One is
>> preallocated shmem as used here, another is dynamically by hypervisor APIs.
>> shmem was to just to indicate it is preallocated.
>>
> Let us keep it without shmem. If it is dynamically allocated, you must not
> need another compatible as you can check it at the runtime.
>
>>> 2. The exact conduit(SMC/HVC) used is detected runtime, so I prefer to keep
>>>     '-smc' instead of '-hvc' in the compatible just to avoid giving an illusion
>>>     that HVC is the conduit chosen here based on the compatible. It can be true
>>>     for other reason but I don't want to mislead here by using HVC.
>> IUUC, currently, conduit comes from PSCI dt node. We have been using smc for
>> PSCI but want to use hvc here. That being said, I am fine to explore if we
>> can change PSCI to use hvc too.
>>
> I think only OPTEE has explicit conduit other than PSCI and it is continued
> for legacy/compatibility reasons IIUC and IIRC. Anything else depends on
> the conduit used by PSCI to be consistent. So yes you need to use what the
> PSCI conduit is and you don't need the extra information from the DT either
> as new property or in the compatible.

Ok, will use conduit then. Thanks!


>
  

On Wed, Oct 04, 2023 at 05:06:30PM +0100, Sudeep Holla wrote:
> On Tue, Oct 03, 2023 at 09:16:27AM -0700, Nikunj Kela wrote:
> > On 10/3/2023 4:19 AM, Sudeep Holla wrote:
> > > On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> > > > diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
[..]
> > > > @@ -63,6 +66,8 @@ struct scmi_smc {
> > > >   	u32 func_id;
> > > >   	u32 param_page;
> > > >   	u32 param_offset;
> > > > +	u64 cap_id;
> > > Can it be unsigned long instead so that it just works for both 32 and 64 bit.
> > 
> > My first version of this patch was ulong but Bjorn suggested to make this
> > structure size fixed i.e. architecture independent. Hence changed it to u64.
> > If you are ok with ulong, I can change it back to ulong.
> >
> 
> SMCCC pre-v1.2 used the common structure in that way. I don't see any issue
> with that. I haven't followed Bjorn suggestions/comments though.
> 

My request was that funcId and capId is an ABI between the firmware and
the OS, so I'd like for that to use well defined, fixed sized, data
types - if nothing else just for documentation purpose.

These values will be truncated when passed to arm_smccc_1_1_invoke()
anyways, so I don't have any opinion against using unsigned long here...


PS. I understand why func_id is u32, but why are param_page and
param_offset u32?

Regards,
Bjorn
  

On 10/5/2023 3:20 PM, Bjorn Andersson wrote:
> On Wed, Oct 04, 2023 at 05:06:30PM +0100, Sudeep Holla wrote:
>> On Tue, Oct 03, 2023 at 09:16:27AM -0700, Nikunj Kela wrote:
>>> On 10/3/2023 4:19 AM, Sudeep Holla wrote:
>>>> On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
>>>>> diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> [..]
>>>>> @@ -63,6 +66,8 @@ struct scmi_smc {
>>>>>    	u32 func_id;
>>>>>    	u32 param_page;
>>>>>    	u32 param_offset;
>>>>> +	u64 cap_id;
>>>> Can it be unsigned long instead so that it just works for both 32 and 64 bit.
>>> My first version of this patch was ulong but Bjorn suggested to make this
>>> structure size fixed i.e. architecture independent. Hence changed it to u64.
>>> If you are ok with ulong, I can change it back to ulong.
>>>
>> SMCCC pre-v1.2 used the common structure in that way. I don't see any issue
>> with that. I haven't followed Bjorn suggestions/comments though.
>>
> My request was that funcId and capId is an ABI between the firmware and
> the OS, so I'd like for that to use well defined, fixed sized, data
> types - if nothing else just for documentation purpose.
>
> These values will be truncated when passed to arm_smccc_1_1_invoke()
> anyways, so I don't have any opinion against using unsigned long here...
>
>
> PS. I understand why func_id is u32, but why are param_page and
> param_offset u32?

That was done to keep it uniform across smc32/smc64 conventions.

>
> Regards,
> Bjorn
  

On Thu, Oct 05, 2023 at 03:20:16PM -0700, Bjorn Andersson wrote:
> On Wed, Oct 04, 2023 at 05:06:30PM +0100, Sudeep Holla wrote:
> > On Tue, Oct 03, 2023 at 09:16:27AM -0700, Nikunj Kela wrote:
> > > On 10/3/2023 4:19 AM, Sudeep Holla wrote:
> > > > On Mon, Sep 11, 2023 at 12:43:59PM -0700, Nikunj Kela wrote:
> > > > > diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
> [..]
> > > > > @@ -63,6 +66,8 @@ struct scmi_smc {
> > > > >   	u32 func_id;
> > > > >   	u32 param_page;
> > > > >   	u32 param_offset;
> > > > > +	u64 cap_id;
> > > > Can it be unsigned long instead so that it just works for both 32 and 64 bit.
> > > 
> > > My first version of this patch was ulong but Bjorn suggested to make this
> > > structure size fixed i.e. architecture independent. Hence changed it to u64.
> > > If you are ok with ulong, I can change it back to ulong.
> > >
> > 
> > SMCCC pre-v1.2 used the common structure in that way. I don't see any issue
> > with that. I haven't followed Bjorn suggestions/comments though.
> > 
> 
> My request was that funcId and capId is an ABI between the firmware and
> the OS, so I'd like for that to use well defined, fixed sized, data
> types - if nothing else just for documentation purpose.
> 
> These values will be truncated when passed to arm_smccc_1_1_invoke()
> anyways, so I don't have any opinion against using unsigned long here...
> 
> 
> PS. I understand why func_id is u32, but why are param_page and
> param_offset u32?
> 

Good point. Sorry I somehow missed your original comment, my bad.
Yes, it is good to be consistent. Sorry if I added any confusion by
missing o read your comment and understanding it before I responded.

--
Regards,
Sudeep