[RFC,v2,00/11] Intel IA32_SPEC_CTRL Virtualization

  Changes since RFC v1:
 * add two kselftests (patch 10-11)
 * set virtual MSRs also on APs [Pawan]
 * enable "virtualize IA32_SPEC_CTRL" for L2 to prevent L2 from changing
   some bits of IA32_SPEC_CTRL (patch 4)
 * other misc cleanup and cosmetic changes

RFC v1: https://lore.kernel.org/lkml/20221210160046.2608762-1-chen.zhang@intel.com/


This series introduces "virtualize IA32_SPEC_CTRL" support. Here are
introduction and use cases of this new feature.

### Virtualize IA32_SPEC_CTRL

"Virtualize IA32_SPEC_CTRL" [1] is a new VMX feature on Intel CPUs. This feature
allows VMM to lock some bits of IA32_SPEC_CTRL MSR even when the MSR is
pass-thru'd to a guest.


### Use cases of "virtualize IA32_SPEC_CTRL" [2]

Software mitigations like Retpoline and software BHB-clearing sequence depend on
CPU microarchitectures. And guest cannot know exactly the underlying
microarchitecture. When a guest is migrated between processors of different
microarchitectures, software mitigations which work perfectly on previous
microachitecture may be not effective on the new one. To fix the problem, some
hardware mitigations should be used in conjunction with software mitigations.
Using virtual IA32_SPEC_CTRL, VMM can enforce hardware mitigations transparently
to guests and avoid those hardware mitigations being unintentionally disabled
when guest changes IA32_SPEC_CTRL MSR.


### Intention of this series

This series adds the capability of enforcing hardware mitigations for guests
transparently and efficiently (i.e., without intecepting IA32_SPEC_CTRL MSR
accesses) to kvm. The capability can be used to solve the VM migration issue in
a pool consisting of processors of different microarchitectures.

Specifically, below are two target scenarios of this series:

Scenario 1: If retpoline is used by a VM to mitigate IMBTI in CPL0, VMM can set
	    RRSBA_DIS_S on parts enumerates RRSBA. Note that the VM is presented
	    with a microarchitecture doesn't enumerate RRSBA.

Scenario 2: If a VM uses software BHB-clearing sequence on transitions into CPL0
	    to mitigate BHI, VMM can use "virtualize IA32_SPEC_CTRL" to set
	    BHI_DIS_S on new parts which doesn't enumerate BHI_NO.

Intel defines some virtual MSRs [2] for guests to report in-use software
mitigations. This allows guests to opt in VMM's deploying hardware mitigations
for them if the guests are either running or later migrated to a system on which
in-use software mitigations are not effective. The virtual MSRs interface is
also added in this series.

### Organization of this series

1. Patch 1-3	Advertise RRSBA_CTRL and BHI_CTRL to guest
2. Patch 4	Add "virtualize IA32_SPEC_CTRL" support
3. Patch 5-9	Allow guests to report in-use software mitigations to KVM so
                that KVM can enable hardware mitigations for guests.
4. Patch 10-11	Add kselftest for virtual MSRs and IA32_SPEC_CTRL

[1]: https://cdrdv2.intel.com/v1/dl/getContent/671368 Ref. #319433-047 Chapter 12
[2]: https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

Chao Gao (3):
  KVM: VMX: Advertise MITI_ENUM_RETPOLINE_S_SUPPORT
  KVM: selftests: Add tests for virtual enumeration/mitigation MSRs
  KVM: selftests: Add tests for IA32_SPEC_CTRL MSR

Pawan Gupta (1):
  x86/bugs: Use Virtual MSRs to request hardware mitigations

Zhang Chen (7):
  x86/msr-index: Add bit definitions for BHI_DIS_S and BHI_NO
  KVM: x86: Advertise CPUID.7.2.EDX and RRSBA_CTRL support
  KVM: x86: Advertise BHI_CTRL support
  KVM: VMX: Add IA32_SPEC_CTRL virtualization support
  KVM: x86: Advertise ARCH_CAP_VIRTUAL_ENUM support
  KVM: VMX: Advertise MITIGATION_CTRL support
  KVM: VMX: Advertise MITI_CTRL_BHB_CLEAR_SEQ_S_SUPPORT

 arch/x86/include/asm/msr-index.h              |  33 +++-
 arch/x86/include/asm/vmx.h                    |   5 +
 arch/x86/include/asm/vmxfeatures.h            |   2 +
 arch/x86/kernel/cpu/bugs.c                    |  25 +++
 arch/x86/kvm/cpuid.c                          |  22 ++-
 arch/x86/kvm/reverse_cpuid.h                  |   8 +
 arch/x86/kvm/svm/svm.c                        |   3 +
 arch/x86/kvm/vmx/capabilities.h               |   5 +
 arch/x86/kvm/vmx/nested.c                     |  13 ++
 arch/x86/kvm/vmx/vmcs.h                       |   2 +
 arch/x86/kvm/vmx/vmx.c                        | 112 ++++++++++-
 arch/x86/kvm/vmx/vmx.h                        |  43 ++++-
 arch/x86/kvm/x86.c                            |  19 +-
 tools/arch/x86/include/asm/msr-index.h        |  37 +++-
 tools/testing/selftests/kvm/Makefile          |   2 +
 .../selftests/kvm/include/x86_64/processor.h  |   5 +
 .../selftests/kvm/x86_64/spec_ctrl_msr_test.c | 178 ++++++++++++++++++
 .../kvm/x86_64/virtual_mitigation_msr_test.c  | 175 +++++++++++++++++
 18 files changed, 676 insertions(+), 13 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/spec_ctrl_msr_test.c
 create mode 100644 tools/testing/selftests/kvm/x86_64/virtual_mitigation_msr_test.c


base-commit: 400d2132288edbd6d500f45eab5d85526ca94e46
  

On 4/14/2023 2:25 PM, Chao Gao wrote:
> Changes since RFC v1:
>   * add two kselftests (patch 10-11)
>   * set virtual MSRs also on APs [Pawan]
>   * enable "virtualize IA32_SPEC_CTRL" for L2 to prevent L2 from changing
>     some bits of IA32_SPEC_CTRL (patch 4)
>   * other misc cleanup and cosmetic changes
>
> RFC v1: https://lore.kernel.org/lkml/20221210160046.2608762-1-chen.zhang@intel.com/
>
>
> This series introduces "virtualize IA32_SPEC_CTRL" support. Here are
> introduction and use cases of this new feature.
>
> ### Virtualize IA32_SPEC_CTRL
>
> "Virtualize IA32_SPEC_CTRL" [1] is a new VMX feature on Intel CPUs. This feature
> allows VMM to lock some bits of IA32_SPEC_CTRL MSR even when the MSR is
> pass-thru'd to a guest.
>
>
> ### Use cases of "virtualize IA32_SPEC_CTRL" [2]
>
> Software mitigations like Retpoline and software BHB-clearing sequence depend on
> CPU microarchitectures. And guest cannot know exactly the underlying
> microarchitecture. When a guest is migrated between processors of different
> microarchitectures, software mitigations which work perfectly on previous
> microachitecture may be not effective on the new one. To fix the problem, some
> hardware mitigations should be used in conjunction with software mitigations.

So even the hardware mitigations are enabled, the software mitigations 
are still needed, right?


> Using virtual IA32_SPEC_CTRL, VMM can enforce hardware mitigations transparently
> to guests and avoid those hardware mitigations being unintentionally disabled
> when guest changes IA32_SPEC_CTRL MSR.
>
>
> ### Intention of this series
>
> This series adds the capability of enforcing hardware mitigations for guests
> transparently and efficiently (i.e., without intecepting IA32_SPEC_CTRL MSR

/s/intecepting/intercepting


> accesses) to kvm. The capability can be used to solve the VM migration issue in
> a pool consisting of processors of different microarchitectures.
>
> Specifically, below are two target scenarios of this series:
>
> Scenario 1: If retpoline is used by a VM to mitigate IMBTI in CPL0, VMM can set
> 	    RRSBA_DIS_S on parts enumerates RRSBA. Note that the VM is presented
> 	    with a microarchitecture doesn't enumerate RRSBA.
>
> Scenario 2: If a VM uses software BHB-clearing sequence on transitions into CPL0
> 	    to mitigate BHI, VMM can use "virtualize IA32_SPEC_CTRL" to set
> 	    BHI_DIS_S on new parts which doesn't enumerate BHI_NO.
>
> Intel defines some virtual MSRs [2] for guests to report in-use software
> mitigations. This allows guests to opt in VMM's deploying hardware mitigations
> for them if the guests are either running or later migrated to a system on which
> in-use software mitigations are not effective. The virtual MSRs interface is
> also added in this series.
>
> ### Organization of this series
>
> 1. Patch 1-3	Advertise RRSBA_CTRL and BHI_CTRL to guest
> 2. Patch 4	Add "virtualize IA32_SPEC_CTRL" support
> 3. Patch 5-9	Allow guests to report in-use software mitigations to KVM so
>                  that KVM can enable hardware mitigations for guests.
> 4. Patch 10-11	Add kselftest for virtual MSRs and IA32_SPEC_CTRL
>
> [1]: https://cdrdv2.intel.com/v1/dl/getContent/671368 Ref. #319433-047 Chapter 12
> [2]: https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html
>
> Chao Gao (3):
>    KVM: VMX: Advertise MITI_ENUM_RETPOLINE_S_SUPPORT
>    KVM: selftests: Add tests for virtual enumeration/mitigation MSRs
>    KVM: selftests: Add tests for IA32_SPEC_CTRL MSR
>
> Pawan Gupta (1):
>    x86/bugs: Use Virtual MSRs to request hardware mitigations
>
> Zhang Chen (7):
>    x86/msr-index: Add bit definitions for BHI_DIS_S and BHI_NO
>    KVM: x86: Advertise CPUID.7.2.EDX and RRSBA_CTRL support
>    KVM: x86: Advertise BHI_CTRL support
>    KVM: VMX: Add IA32_SPEC_CTRL virtualization support
>    KVM: x86: Advertise ARCH_CAP_VIRTUAL_ENUM support
>    KVM: VMX: Advertise MITIGATION_CTRL support
>    KVM: VMX: Advertise MITI_CTRL_BHB_CLEAR_SEQ_S_SUPPORT
>
>   arch/x86/include/asm/msr-index.h              |  33 +++-
>   arch/x86/include/asm/vmx.h                    |   5 +
>   arch/x86/include/asm/vmxfeatures.h            |   2 +
>   arch/x86/kernel/cpu/bugs.c                    |  25 +++
>   arch/x86/kvm/cpuid.c                          |  22 ++-
>   arch/x86/kvm/reverse_cpuid.h                  |   8 +
>   arch/x86/kvm/svm/svm.c                        |   3 +
>   arch/x86/kvm/vmx/capabilities.h               |   5 +
>   arch/x86/kvm/vmx/nested.c                     |  13 ++
>   arch/x86/kvm/vmx/vmcs.h                       |   2 +
>   arch/x86/kvm/vmx/vmx.c                        | 112 ++++++++++-
>   arch/x86/kvm/vmx/vmx.h                        |  43 ++++-
>   arch/x86/kvm/x86.c                            |  19 +-
>   tools/arch/x86/include/asm/msr-index.h        |  37 +++-
>   tools/testing/selftests/kvm/Makefile          |   2 +
>   .../selftests/kvm/include/x86_64/processor.h  |   5 +
>   .../selftests/kvm/x86_64/spec_ctrl_msr_test.c | 178 ++++++++++++++++++
>   .../kvm/x86_64/virtual_mitigation_msr_test.c  | 175 +++++++++++++++++
>   18 files changed, 676 insertions(+), 13 deletions(-)
>   create mode 100644 tools/testing/selftests/kvm/x86_64/spec_ctrl_msr_test.c
>   create mode 100644 tools/testing/selftests/kvm/x86_64/virtual_mitigation_msr_test.c
>
>
> base-commit: 400d2132288edbd6d500f45eab5d85526ca94e46
  

On Fri, Apr 14, 2023 at 05:51:43PM +0800, Binbin Wu wrote:
> 
> On 4/14/2023 2:25 PM, Chao Gao wrote:
> > Changes since RFC v1:
> >   * add two kselftests (patch 10-11)
> >   * set virtual MSRs also on APs [Pawan]
> >   * enable "virtualize IA32_SPEC_CTRL" for L2 to prevent L2 from changing
> >     some bits of IA32_SPEC_CTRL (patch 4)
> >   * other misc cleanup and cosmetic changes
> > 
> > RFC v1: https://lore.kernel.org/lkml/20221210160046.2608762-1-chen.zhang@intel.com/
> > 
> > 
> > This series introduces "virtualize IA32_SPEC_CTRL" support. Here are
> > introduction and use cases of this new feature.
> > 
> > ### Virtualize IA32_SPEC_CTRL
> > 
> > "Virtualize IA32_SPEC_CTRL" [1] is a new VMX feature on Intel CPUs. This feature
> > allows VMM to lock some bits of IA32_SPEC_CTRL MSR even when the MSR is
> > pass-thru'd to a guest.
> > 
> > 
> > ### Use cases of "virtualize IA32_SPEC_CTRL" [2]
> > 
> > Software mitigations like Retpoline and software BHB-clearing sequence depend on
> > CPU microarchitectures. And guest cannot know exactly the underlying
> > microarchitecture. When a guest is migrated between processors of different
> > microarchitectures, software mitigations which work perfectly on previous
> > microachitecture may be not effective on the new one. To fix the problem, some
> > hardware mitigations should be used in conjunction with software mitigations.
> 
> So even the hardware mitigations are enabled, the software mitigations are
> still needed, right?

Retpoline mitigation is not fully effective when RET can take prediction
from an alternate predictor. Newer hardware provides a way to disable
this behavior (using RRSBA_DIS_S bit in MSR SPEC_CTRL).

eIBRS is the preferred way to mitigate BTI, but for some reason when a
guest has deployed retpoline, VMM can make it more effective by
deploying the relevant hardware control. That is why the above text
says:

  "... hardware mitigations should be used in conjunction with software
  mitigations."