[RFC,v5,0/6] KVM: x86: add per-vCPU exits disable capability

  Summary
===========
Introduce support of vCPU-scoped ioctl with KVM_CAP_X86_DISABLE_EXITS
cap for disabling exits to enable finer-grained VM exits disabling
on per vCPU scales instead of whole guest. This patch series enabled
the vCPU-scoped exits control and toggling.

Motivation
============
In use cases like Windows guest running heavy CPU-bound
workloads, disabling HLT VM-exits could mitigate host sched ctx switch
overhead. Simply HLT disabling on all vCPUs could bring
performance benefits, but if no pCPUs reserved for host threads, could
happened to the forced preemption as host does not know the time to do
the schedule for other host threads want to run. With this patch, we
could only disable part of vCPUs HLT exits for one guest, this still
keeps performance benefits, and also shows resiliency to host stressing
workload running at the same time.

Performance and Testing
=========================
In the host stressing workload experiment with Windows guest heavy
CPU-bound workloads, it shows good resiliency and having the ~3%
performance improvement. E.g. Passmark running in a Windows guest
with this patch disabling HLT exits on only half of vCPUs still
showing 2.4% higher main score v/s baseline.

Tested everything on AMD machines.

v4->v5 :
- Drop the usage of KVM request, keep the VM-scoped exits disable
  as the existing design, and only allow per-vCPU settings to
  override the per-VM settings (Sean Christopherson)
- Refactor the disable exits selftest without introducing any
  new prerequisite patch, tests per-vCPU exits disable and overrides,
  and per-VM exits disable

v3->v4 (Chao Gao) :
- Use kvm vCPU request KVM_REQ_DISABLE_EXIT to perform the arch
  VMCS updating (patch 5)
- Fix selftests redundant arguments (patch 7)
- Merge overlapped fix bits from patch 4 to patch 3

v2->v3 (Sean Christopherson) :
- Reject KVM_CAP_X86_DISABLE_EXITS if userspace disable MWAIT exits
  when MWAIT is not allowed in guest (patch 3)
- Make userspace able to re-enable previously disabled exits (patch 4)
- Add mwait/pause/cstate exits flag toggling instead of only hlt
  exits (patch 5)
- Add selftests for KVM_CAP_X86_DISABLE_EXITS (patch 7)

v1->v2 (Sean Christopherson) :
- Add explicit restriction for VM-scoped exits disabling to be called
  before vCPUs creation (patch 1)
- Use vCPU ioctl instead of 64bit vCPU bitmask (patch 5), and make exits
  disable flags check purely for vCPU instead of VM (patch 2)

Best Regards,
Kechen

Kechen Lu (3):
  KVM: x86: Move *_in_guest power management flags to vCPU scope
  KVM: x86: add vCPU scoped toggling for disabled exits
  KVM: selftests: Add tests for VM and vCPU cap
    KVM_CAP_X86_DISABLE_EXITS

Sean Christopherson (3):
  KVM: x86: only allow exits disable before vCPUs created
  KVM: x86: Reject disabling of MWAIT interception when not allowed
  KVM: x86: Let userspace re-enable previously disabled exits

 Documentation/virt/kvm/api.rst                |   8 +-
 arch/x86/include/asm/kvm-x86-ops.h            |   1 +
 arch/x86/include/asm/kvm_host.h               |   7 +
 arch/x86/kvm/cpuid.c                          |   4 +-
 arch/x86/kvm/lapic.c                          |   7 +-
 arch/x86/kvm/svm/nested.c                     |   4 +-
 arch/x86/kvm/svm/svm.c                        |  42 +-
 arch/x86/kvm/vmx/vmx.c                        |  53 +-
 arch/x86/kvm/x86.c                            |  69 ++-
 arch/x86/kvm/x86.h                            |  16 +-
 include/uapi/linux/kvm.h                      |   4 +-
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/x86_64/disable_exits_test.c | 457 ++++++++++++++++++
 13 files changed, 626 insertions(+), 47 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/disable_exits_test.c
  

On Fri, 13 Jan 2023 22:01:08 +0000
Kechen Lu <kechenl@nvidia.com> wrote:

Hi:

checkpatch.pl throws a lot of warning and errors when I was trying
this series. Can you fix them?

total: 470 errors, 22 warnings, 464 lines checked

> Summary
> ===========
> Introduce support of vCPU-scoped ioctl with KVM_CAP_X86_DISABLE_EXITS
> cap for disabling exits to enable finer-grained VM exits disabling
> on per vCPU scales instead of whole guest. This patch series enabled
> the vCPU-scoped exits control and toggling.
> 
> Motivation
> ============
> In use cases like Windows guest running heavy CPU-bound
> workloads, disabling HLT VM-exits could mitigate host sched ctx switch
> overhead. Simply HLT disabling on all vCPUs could bring
> performance benefits, but if no pCPUs reserved for host threads, could
> happened to the forced preemption as host does not know the time to do
> the schedule for other host threads want to run. With this patch, we
> could only disable part of vCPUs HLT exits for one guest, this still
> keeps performance benefits, and also shows resiliency to host stressing
> workload running at the same time.
> 
> Performance and Testing
> =========================
> In the host stressing workload experiment with Windows guest heavy
> CPU-bound workloads, it shows good resiliency and having the ~3%
> performance improvement. E.g. Passmark running in a Windows guest
> with this patch disabling HLT exits on only half of vCPUs still
> showing 2.4% higher main score v/s baseline.
> 
> Tested everything on AMD machines.
> 
> v4->v5 :
> - Drop the usage of KVM request, keep the VM-scoped exits disable
>   as the existing design, and only allow per-vCPU settings to
>   override the per-VM settings (Sean Christopherson)
> - Refactor the disable exits selftest without introducing any
>   new prerequisite patch, tests per-vCPU exits disable and overrides,
>   and per-VM exits disable
> 
> v3->v4 (Chao Gao) :
> - Use kvm vCPU request KVM_REQ_DISABLE_EXIT to perform the arch
>   VMCS updating (patch 5)
> - Fix selftests redundant arguments (patch 7)
> - Merge overlapped fix bits from patch 4 to patch 3
> 
> v2->v3 (Sean Christopherson) :
> - Reject KVM_CAP_X86_DISABLE_EXITS if userspace disable MWAIT exits
>   when MWAIT is not allowed in guest (patch 3)
> - Make userspace able to re-enable previously disabled exits (patch 4)
> - Add mwait/pause/cstate exits flag toggling instead of only hlt
>   exits (patch 5)
> - Add selftests for KVM_CAP_X86_DISABLE_EXITS (patch 7)
> 
> v1->v2 (Sean Christopherson) :
> - Add explicit restriction for VM-scoped exits disabling to be called
>   before vCPUs creation (patch 1)
> - Use vCPU ioctl instead of 64bit vCPU bitmask (patch 5), and make exits
>   disable flags check purely for vCPU instead of VM (patch 2)
> 
> Best Regards,
> Kechen
> 
> Kechen Lu (3):
>   KVM: x86: Move *_in_guest power management flags to vCPU scope
>   KVM: x86: add vCPU scoped toggling for disabled exits
>   KVM: selftests: Add tests for VM and vCPU cap
>     KVM_CAP_X86_DISABLE_EXITS
> 
> Sean Christopherson (3):
>   KVM: x86: only allow exits disable before vCPUs created
>   KVM: x86: Reject disabling of MWAIT interception when not allowed
>   KVM: x86: Let userspace re-enable previously disabled exits
> 
>  Documentation/virt/kvm/api.rst                |   8 +-
>  arch/x86/include/asm/kvm-x86-ops.h            |   1 +
>  arch/x86/include/asm/kvm_host.h               |   7 +
>  arch/x86/kvm/cpuid.c                          |   4 +-
>  arch/x86/kvm/lapic.c                          |   7 +-
>  arch/x86/kvm/svm/nested.c                     |   4 +-
>  arch/x86/kvm/svm/svm.c                        |  42 +-
>  arch/x86/kvm/vmx/vmx.c                        |  53 +-
>  arch/x86/kvm/x86.c                            |  69 ++-
>  arch/x86/kvm/x86.h                            |  16 +-
>  include/uapi/linux/kvm.h                      |   4 +-
>  tools/testing/selftests/kvm/Makefile          |   1 +
>  .../selftests/kvm/x86_64/disable_exits_test.c | 457 ++++++++++++++++++
>  13 files changed, 626 insertions(+), 47 deletions(-)
>  create mode 100644 tools/testing/selftests/kvm/x86_64/disable_exits_test.c
>
  

On Wed, 18 Jan 2023 10:30:03 +0200
Zhi Wang <zhi.wang.linux@gmail.com> wrote:

Hi:

No sure why the test never finishes on my testing machine. Will take a look
later today.

My CPU: 
model name      : Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz

branch kvm.git/master top commit:
310bc39546a435c83cc27a0eba878afac0d74714

-----

[inno@inno-lk-server x86_64]$ time ./disable_exits_test
VM-scoped tests start
Halter vCPU thread started
vCPU thread running vCPU 0
Halter vCPU thread reported its first HLT executed after 1 seconds.
vCPU thread running vCPU 1
Halter vCPU had 10 halt exits
Guest records 10 HLTs executed, waked 9 times
Halter vCPU thread started
vCPU thread running vCPU 0



^C

real    19m0.923s
user    37m56.512s
sys     0m1.086s

-----

> On Fri, 13 Jan 2023 22:01:08 +0000
> Kechen Lu <kechenl@nvidia.com> wrote:
> 
> Hi:
> 
> checkpatch.pl throws a lot of warning and errors when I was trying
> this series. Can you fix them?
> 
> total: 470 errors, 22 warnings, 464 lines checked
> 
> > Summary
> > ===========
> > Introduce support of vCPU-scoped ioctl with KVM_CAP_X86_DISABLE_EXITS
> > cap for disabling exits to enable finer-grained VM exits disabling
> > on per vCPU scales instead of whole guest. This patch series enabled
> > the vCPU-scoped exits control and toggling.
> > 
> > Motivation
> > ============
> > In use cases like Windows guest running heavy CPU-bound
> > workloads, disabling HLT VM-exits could mitigate host sched ctx switch
> > overhead. Simply HLT disabling on all vCPUs could bring
> > performance benefits, but if no pCPUs reserved for host threads, could
> > happened to the forced preemption as host does not know the time to do
> > the schedule for other host threads want to run. With this patch, we
> > could only disable part of vCPUs HLT exits for one guest, this still
> > keeps performance benefits, and also shows resiliency to host stressing
> > workload running at the same time.
> > 
> > Performance and Testing
> > =========================
> > In the host stressing workload experiment with Windows guest heavy
> > CPU-bound workloads, it shows good resiliency and having the ~3%
> > performance improvement. E.g. Passmark running in a Windows guest
> > with this patch disabling HLT exits on only half of vCPUs still
> > showing 2.4% higher main score v/s baseline.
> > 
> > Tested everything on AMD machines.
> > 
> > v4->v5 :
> > - Drop the usage of KVM request, keep the VM-scoped exits disable
> >   as the existing design, and only allow per-vCPU settings to
> >   override the per-VM settings (Sean Christopherson)
> > - Refactor the disable exits selftest without introducing any
> >   new prerequisite patch, tests per-vCPU exits disable and overrides,
> >   and per-VM exits disable
> > 
> > v3->v4 (Chao Gao) :
> > - Use kvm vCPU request KVM_REQ_DISABLE_EXIT to perform the arch
> >   VMCS updating (patch 5)
> > - Fix selftests redundant arguments (patch 7)
> > - Merge overlapped fix bits from patch 4 to patch 3
> > 
> > v2->v3 (Sean Christopherson) :
> > - Reject KVM_CAP_X86_DISABLE_EXITS if userspace disable MWAIT exits
> >   when MWAIT is not allowed in guest (patch 3)
> > - Make userspace able to re-enable previously disabled exits (patch 4)
> > - Add mwait/pause/cstate exits flag toggling instead of only hlt
> >   exits (patch 5)
> > - Add selftests for KVM_CAP_X86_DISABLE_EXITS (patch 7)
> > 
> > v1->v2 (Sean Christopherson) :
> > - Add explicit restriction for VM-scoped exits disabling to be called
> >   before vCPUs creation (patch 1)
> > - Use vCPU ioctl instead of 64bit vCPU bitmask (patch 5), and make exits
> >   disable flags check purely for vCPU instead of VM (patch 2)
> > 
> > Best Regards,
> > Kechen
> > 
> > Kechen Lu (3):
> >   KVM: x86: Move *_in_guest power management flags to vCPU scope
> >   KVM: x86: add vCPU scoped toggling for disabled exits
> >   KVM: selftests: Add tests for VM and vCPU cap
> >     KVM_CAP_X86_DISABLE_EXITS
> > 
> > Sean Christopherson (3):
> >   KVM: x86: only allow exits disable before vCPUs created
> >   KVM: x86: Reject disabling of MWAIT interception when not allowed
> >   KVM: x86: Let userspace re-enable previously disabled exits
> > 
> >  Documentation/virt/kvm/api.rst                |   8 +-
> >  arch/x86/include/asm/kvm-x86-ops.h            |   1 +
> >  arch/x86/include/asm/kvm_host.h               |   7 +
> >  arch/x86/kvm/cpuid.c                          |   4 +-
> >  arch/x86/kvm/lapic.c                          |   7 +-
> >  arch/x86/kvm/svm/nested.c                     |   4 +-
> >  arch/x86/kvm/svm/svm.c                        |  42 +-
> >  arch/x86/kvm/vmx/vmx.c                        |  53 +-
> >  arch/x86/kvm/x86.c                            |  69 ++-
> >  arch/x86/kvm/x86.h                            |  16 +-
> >  include/uapi/linux/kvm.h                      |   4 +-
> >  tools/testing/selftests/kvm/Makefile          |   1 +
> >  .../selftests/kvm/x86_64/disable_exits_test.c | 457 ++++++++++++++++++
> >  13 files changed, 626 insertions(+), 47 deletions(-)
> >  create mode 100644 tools/testing/selftests/kvm/x86_64/disable_exits_test.c
> > 
>