[v1] KVM: x86: add KVM_CAP_DEVICE_CTRL

  KVM_CAP_DEVICE_CTRL allows userspace to create emulated device in KVM.
For example, userspace VFIO implementation needs to create a kvm_device
(i.e. KVM_DEV_TYPE_VFIO) on x86. So add the cap to allow userspace for
such use cases.

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 arch/x86/kvm/x86.c | 1 +
 1 file changed, 1 insertion(+)
  

On Thu, Dec 15, 2022, Wei Wang wrote:
> KVM_CAP_DEVICE_CTRL allows userspace to create emulated device in KVM.
> For example, userspace VFIO implementation needs to create a kvm_device
> (i.e. KVM_DEV_TYPE_VFIO) on x86. So add the cap to allow userspace for
> such use cases.
> 
> Signed-off-by: Wei Wang <wei.w.wang@intel.com>
> ---
>  arch/x86/kvm/x86.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 69227f77b201..1cdc4469652c 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -4410,6 +4410,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>  	case KVM_CAP_VAPIC:
>  	case KVM_CAP_ENABLE_CAP:
>  	case KVM_CAP_VM_DISABLE_NX_HUGE_PAGES:
> +	case KVM_CAP_DEVICE_CTRL:

Rather than hardcode this in x86, I think it would be better to add an #ifdef'd
version in the generic check.  E.g. if MIPS or RISC-V ever gains KVM_VFIO support
then they'll need to enumerate KVM_CAP_DEVICE_CTRL too, and odds are we'll forget
to to do.

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 13e88297f999..f70b9cea95d9 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4525,6 +4525,10 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
        case KVM_CAP_BINARY_STATS_FD:
        case KVM_CAP_SYSTEM_EVENT_DATA:
                return 1;
+#ifdef CONFIG_KVM_VFIO
+       case KVM_CAP_DEVICE_CTRL:
+               return 1;
+#endif
        default:
                break;
        }

The other potentially bad idea would be to detect the presence of a device_ops and
delete all of the arch hooks, e.g. 

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 9c5573bc4614..190e9c3b10a7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -212,7 +212,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
                r = vgic_present;
                break;
        case KVM_CAP_IOEVENTFD:
-       case KVM_CAP_DEVICE_CTRL:
        case KVM_CAP_USER_MEMORY:
        case KVM_CAP_SYNC_MMU:
        case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 04494a4fb37a..21f9fbe96f6a 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -541,7 +541,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
        case KVM_CAP_ENABLE_CAP:
        case KVM_CAP_ONE_REG:
        case KVM_CAP_IOEVENTFD:
-       case KVM_CAP_DEVICE_CTRL:
        case KVM_CAP_IMMEDIATE_EXIT:
        case KVM_CAP_SET_GUEST_DEBUG:
                r = 1;
diff --git a/arch/riscv/kvm/vm.c b/arch/riscv/kvm/vm.c
index 65a964d7e70d..6efe93b282e1 100644
--- a/arch/riscv/kvm/vm.c
+++ b/arch/riscv/kvm/vm.c
@@ -57,7 +57,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 
        switch (ext) {
        case KVM_CAP_IOEVENTFD:
-       case KVM_CAP_DEVICE_CTRL:
        case KVM_CAP_USER_MEMORY:
        case KVM_CAP_SYNC_MMU:
        case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index e4890e04b210..191d220b6a30 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -567,7 +567,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
        case KVM_CAP_ENABLE_CAP:
        case KVM_CAP_S390_CSS_SUPPORT:
        case KVM_CAP_IOEVENTFD:
-       case KVM_CAP_DEVICE_CTRL:
        case KVM_CAP_S390_IRQCHIP:
        case KVM_CAP_VM_ATTRIBUTES:
        case KVM_CAP_MP_STATE:
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 13e88297f999..99e3da9ce42d 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -4525,6 +4525,15 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
        case KVM_CAP_BINARY_STATS_FD:
        case KVM_CAP_SYSTEM_EVENT_DATA:
                return 1;
+       case KVM_CAP_DEVICE_CTRL: {
+               int i;
+
+               for (i = 0; i < ARRAY_SIZE(kvm_device_ops_table); ++) {
+                       if (kvm_device_ops_table[i])
+                               return 1;
+               }
+               return 0;
+       }
        default:
                break;
        }
  

On Saturday, December 17, 2022 1:13 AM, Sean Christopherson wrote:
> Rather than hardcode this in x86, I think it would be better to add an #ifdef'd
> version in the generic check.  E.g. if MIPS or RISC-V ever gains KVM_VFIO
> support then they'll need to enumerate KVM_CAP_DEVICE_CTRL too, and odds
> are we'll forget to to do.
> 
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index
> 13e88297f999..f70b9cea95d9 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -4525,6 +4525,10 @@ static long
> kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
>         case KVM_CAP_BINARY_STATS_FD:
>         case KVM_CAP_SYSTEM_EVENT_DATA:
>                 return 1;
> +#ifdef CONFIG_KVM_VFIO
> +       case KVM_CAP_DEVICE_CTRL:
> +               return 1;
> +#endif
>         default:
>                 break;
>         }
> 
> The other potentially bad idea would be to detect the presence of a
> device_ops and delete all of the arch hooks, e.g.
> 
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index
> 9c5573bc4614..190e9c3b10a7 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -212,7 +212,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm,
> long ext)
>                 r = vgic_present;
>                 break;
>         case KVM_CAP_IOEVENTFD:
> -       case KVM_CAP_DEVICE_CTRL:
>         case KVM_CAP_USER_MEMORY:
>         case KVM_CAP_SYNC_MMU:
>         case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 04494a4fb37a..21f9fbe96f6a 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -541,7 +541,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm,
> long ext)
>         case KVM_CAP_ENABLE_CAP:
>         case KVM_CAP_ONE_REG:
>         case KVM_CAP_IOEVENTFD:
> -       case KVM_CAP_DEVICE_CTRL:
>         case KVM_CAP_IMMEDIATE_EXIT:
>         case KVM_CAP_SET_GUEST_DEBUG:
>                 r = 1;
> diff --git a/arch/riscv/kvm/vm.c b/arch/riscv/kvm/vm.c index
> 65a964d7e70d..6efe93b282e1 100644
> --- a/arch/riscv/kvm/vm.c
> +++ b/arch/riscv/kvm/vm.c
> @@ -57,7 +57,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm,
> long ext)
> 
>         switch (ext) {
>         case KVM_CAP_IOEVENTFD:
> -       case KVM_CAP_DEVICE_CTRL:
>         case KVM_CAP_USER_MEMORY:
>         case KVM_CAP_SYNC_MMU:
>         case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index
> e4890e04b210..191d220b6a30 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -567,7 +567,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm,
> long ext)
>         case KVM_CAP_ENABLE_CAP:
>         case KVM_CAP_S390_CSS_SUPPORT:
>         case KVM_CAP_IOEVENTFD:
> -       case KVM_CAP_DEVICE_CTRL:
>         case KVM_CAP_S390_IRQCHIP:
>         case KVM_CAP_VM_ATTRIBUTES:
>         case KVM_CAP_MP_STATE:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index
> 13e88297f999..99e3da9ce42d 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -4525,6 +4525,15 @@ static long
> kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
>         case KVM_CAP_BINARY_STATS_FD:
>         case KVM_CAP_SYSTEM_EVENT_DATA:
>                 return 1;
> +       case KVM_CAP_DEVICE_CTRL: {
> +               int i;
> +
> +               for (i = 0; i < ARRAY_SIZE(kvm_device_ops_table); ++) {
> +                       if (kvm_device_ops_table[i])
> +                               return 1;
> +               }
> +               return 0;
> +       }
>         default:
>                 break;
>         }

Yes, it looks better to move it to the generic check, but I'm not sure if it would be necessary to do the per-device check here either via CONFIG_KVM_VFIO (for example, if more non-arch-specific usages are added, we would end up with lots of such #ifdef to be added, which doesn't seem nice) or kvm_device_ops_table.

I think fundamentally KVM_CAP_DEVICE_CTRL is used to check if the generic kvm_device framework (e.g. KVM_CREATE_DEVICE) is supported by KVM (older KVM before 2013 doesn't have it). The per-device type (KVM_DEV_TYPE_VFIO, KVM_DEV_TYPE_ARM_PV_TIME etc.) support can be checked via KVM_CREATE_DEVICE, which reports -ENODEV if the device type doesn't have an entry in kvm_device_ops_table.
  

On Mon, Dec 19, 2022, Wang, Wei W wrote:
> On Saturday, December 17, 2022 1:13 AM, Sean Christopherson wrote:
> > Rather than hardcode this in x86, I think it would be better to add an #ifdef'd
> > version in the generic check.  E.g. if MIPS or RISC-V ever gains KVM_VFIO
> > support then they'll need to enumerate KVM_CAP_DEVICE_CTRL too, and odds
> > are we'll forget to to do.

...

> > The other potentially bad idea would be to detect the presence of a
> > device_ops and delete all of the arch hooks, e.g.

> Yes, it looks better to move it to the generic check, but I'm not sure if it
> would be necessary to do the per-device check here either via CONFIG_KVM_VFIO
> (for example, if more non-arch-specific usages are added, we would end up
> with lots of such #ifdef to be added, which doesn't seem nice) or
> kvm_device_ops_table.
> 
> I think fundamentally KVM_CAP_DEVICE_CTRL is used to check if the generic
> kvm_device framework (e.g. KVM_CREATE_DEVICE) is supported by KVM (older KVM
> before 2013 doesn't have it). The per-device type (KVM_DEV_TYPE_VFIO,
> KVM_DEV_TYPE_ARM_PV_TIME etc.) support can be checked via KVM_CREATE_DEVICE,
> which reports -ENODEV if the device type doesn't have an entry in
> kvm_device_ops_table.

If that's how we want to retroactively define things, then KVM should unconditionally
return 1/true for KVM_CAP_DEVICE_CTRL since KVM_CREATE_DEVICE is provided by
generic code.
  

On Tuesday, December 20, 2022 4:36 AM, Sean Christopherson wrote:
> > Yes, it looks better to move it to the generic check, but I'm not sure
> > if it would be necessary to do the per-device check here either via
> > CONFIG_KVM_VFIO (for example, if more non-arch-specific usages are
> > added, we would end up with lots of such #ifdef to be added, which
> > doesn't seem nice) or kvm_device_ops_table.
> >
> > I think fundamentally KVM_CAP_DEVICE_CTRL is used to check if the
> > generic kvm_device framework (e.g. KVM_CREATE_DEVICE) is supported by
> > KVM (older KVM before 2013 doesn't have it). The per-device type
> > (KVM_DEV_TYPE_VFIO, KVM_DEV_TYPE_ARM_PV_TIME etc.) support can be
> > checked via KVM_CREATE_DEVICE, which reports -ENODEV if the device
> > type doesn't have an entry in kvm_device_ops_table.
> 
> If that's how we want to retroactively define things, then KVM should
> unconditionally return 1/true for KVM_CAP_DEVICE_CTRL since
> KVM_CREATE_DEVICE is provided by generic code.

Yes. Also, since we have KVM_DEV_TYPE_VFIO the generic use case, it should be better
to move the CAP check to the generic kvm_vm_ioctl_check_extension_generic.