[v2,0/2] KVM: arm64: Support for Arm v8.8 memcpy instructions in KVM guests

  Hi,

This is v2 of the series to allow using the new Arm memory copy instructions
in KVM guests. See v1 for more information [1].

Changes in v2:
 - Dropped HCRX_EL2 vcpu field
 - Rebased onto v6.6-rc2

Thanks,
Kristina

[1] https://lore.kernel.org/kvmarm/20230915124840.474888-1-kristina.martsenko@arm.com/

Kristina Martsenko (2):
  KVM: arm64: Add handler for MOPS exceptions
  KVM: arm64: Expose MOPS instructions to guests

 arch/arm64/include/asm/kvm_arm.h              |  4 +-
 arch/arm64/include/asm/traps.h                | 54 ++++++++++++++++++-
 arch/arm64/kernel/traps.c                     | 48 +----------------
 arch/arm64/kvm/hyp/include/hyp/switch.h       | 17 ++++++
 .../arm64/kvm/hyp/include/nvhe/fixed_config.h |  3 +-
 arch/arm64/kvm/hyp/nvhe/switch.c              |  2 +
 arch/arm64/kvm/hyp/vhe/switch.c               |  1 +
 arch/arm64/kvm/sys_regs.c                     |  1 -
 8 files changed, 78 insertions(+), 52 deletions(-)


base-commit: ce9ecca0238b140b88f43859b211c9fdfd8e5b70
  

Hi Kristina,

On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote:
> Hi,
> 
> This is v2 of the series to allow using the new Arm memory copy instructions
> in KVM guests. See v1 for more information [1].


Thanks for sending out the series. I've been thinking about what the
architecture says for MOPS, and I wonder if what's currently in the
Arm ARM is clear enough for EL1 software to be written robustly.

While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS
exceptions from EL1, there's no such control for EL0. So when vCPU
migration occurs EL1 could get an unexpected MOPS exception, even for a
process that was pinned to a single (virtual) CPU implementation.

Additionally, the wording of I_NXHPS seems to suggest that EL2 handling
of MOPS exceptions is only expected in certain circumstances where EL1 is
incapable of handling an exception. Is the unwritten expectation then
that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1
and EL0, even if EL1 did not migrate the PE context?

Perhaps I'm being pedantic, but I'd really like for there to be some
documentation that suggests MOPS exceptions can happen due to context
migration done by a higher EL as that is the only option in the context
of virtualization.
  

On 27/09/2023 07:00, Oliver Upton wrote:
> Hi Kristina,

Hi Oliver,

> 
> On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote:
>> Hi,
>>
>> This is v2 of the series to allow using the new Arm memory copy instructions
>> in KVM guests. See v1 for more information [1].
> 
> 
> Thanks for sending out the series. I've been thinking about what the
> architecture says for MOPS, and I wonder if what's currently in the
> Arm ARM is clear enough for EL1 software to be written robustly.
> 
> While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS
> exceptions from EL1, there's no such control for EL0. So when vCPU
> migration occurs EL1 could get an unexpected MOPS exception, even for a
> process that was pinned to a single (virtual) CPU implementation.
> 
> Additionally, the wording of I_NXHPS seems to suggest that EL2 handling
> of MOPS exceptions is only expected in certain circumstances where EL1 is
> incapable of handling an exception. Is the unwritten expectation then
> that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1
> and EL0, even if EL1 did not migrate the PE context?
> 
> Perhaps I'm being pedantic, but I'd really like for there to be some
> documentation that suggests MOPS exceptions can happen due to context
> migration done by a higher EL as that is the only option in the context
> of virtualization.

That's a good point. This shouldn't affect Linux guests as Linux is
always able to handle a MOPS exception coming from EL0. But it would
affect any non-Linux guest that pins all its EL0 tasks and doesn't
implement a handler. It's not clear to me what the expectation for
guests is, I'll ask the architects to clarify and get back to you.

Thanks for the feedback!

Kristina
  

On Thu, Sep 28, 2023 at 05:55:39PM +0100, Kristina Martsenko wrote:

[...]

> > Perhaps I'm being pedantic, but I'd really like for there to be some
> > documentation that suggests MOPS exceptions can happen due to context
> > migration done by a higher EL as that is the only option in the context
> > of virtualization.
> 
> That's a good point. This shouldn't affect Linux guests as Linux is
> always able to handle a MOPS exception coming from EL0. But it would
> affect any non-Linux guest that pins all its EL0 tasks and doesn't
> implement a handler. It's not clear to me what the expectation for
> guests is, I'll ask the architects to clarify and get back to you.

That'd be excellent, thanks! All I'm looking for is something to point
folks at if/when they complain about MOPS behavior in KVM guests.
  

On Thu, 28 Sep 2023 17:55:39 +0100,
Kristina Martsenko <kristina.martsenko@arm.com> wrote:
> 
> On 27/09/2023 07:00, Oliver Upton wrote:
> > Hi Kristina,
> 
> Hi Oliver,
> 
> > 
> > On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote:
> >> Hi,
> >>
> >> This is v2 of the series to allow using the new Arm memory copy instructions
> >> in KVM guests. See v1 for more information [1].
> > 
> > 
> > Thanks for sending out the series. I've been thinking about what the
> > architecture says for MOPS, and I wonder if what's currently in the
> > Arm ARM is clear enough for EL1 software to be written robustly.
> > 
> > While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS
> > exceptions from EL1, there's no such control for EL0. So when vCPU
> > migration occurs EL1 could get an unexpected MOPS exception, even for a
> > process that was pinned to a single (virtual) CPU implementation.
> > 
> > Additionally, the wording of I_NXHPS seems to suggest that EL2 handling
> > of MOPS exceptions is only expected in certain circumstances where EL1 is
> > incapable of handling an exception. Is the unwritten expectation then
> > that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1
> > and EL0, even if EL1 did not migrate the PE context?
> > 
> > Perhaps I'm being pedantic, but I'd really like for there to be some
> > documentation that suggests MOPS exceptions can happen due to context
> > migration done by a higher EL as that is the only option in the context
> > of virtualization.
> 
> That's a good point. This shouldn't affect Linux guests as Linux is
> always able to handle a MOPS exception coming from EL0. But it would
> affect any non-Linux guest that pins all its EL0 tasks and doesn't
> implement a handler. It's not clear to me what the expectation for
> guests is, I'll ask the architects to clarify and get back to you.

My understanding is that MCE2 should always be set if the hypervisor
can migrate vcpus across implementations behind EL1's back, and that
in this context, EL1 never sees such an exception.

I guess the only case where we could let EL1 handle such exception is
by only setting MCE2 on the first entry into the guest after a vcpu
migration (and clear it after that). Is it worth the effort?
Absolutely not.

	M.
  

On 29/09/2023 10:29, Marc Zyngier wrote:
> On Thu, 28 Sep 2023 17:55:39 +0100,
> Kristina Martsenko <kristina.martsenko@arm.com> wrote:
>>
>> On 27/09/2023 07:00, Oliver Upton wrote:
>>>
>>> On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote:
>>>> Hi,
>>>>
>>>> This is v2 of the series to allow using the new Arm memory copy instructions
>>>> in KVM guests. See v1 for more information [1].
>>>
>>>
>>> Thanks for sending out the series. I've been thinking about what the
>>> architecture says for MOPS, and I wonder if what's currently in the
>>> Arm ARM is clear enough for EL1 software to be written robustly.
>>>
>>> While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS
>>> exceptions from EL1, there's no such control for EL0. So when vCPU
>>> migration occurs EL1 could get an unexpected MOPS exception, even for a
>>> process that was pinned to a single (virtual) CPU implementation.
>>>
>>> Additionally, the wording of I_NXHPS seems to suggest that EL2 handling
>>> of MOPS exceptions is only expected in certain circumstances where EL1 is
>>> incapable of handling an exception. Is the unwritten expectation then
>>> that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1
>>> and EL0, even if EL1 did not migrate the PE context?
>>>
>>> Perhaps I'm being pedantic, but I'd really like for there to be some
>>> documentation that suggests MOPS exceptions can happen due to context
>>> migration done by a higher EL as that is the only option in the context
>>> of virtualization.
>>
>> That's a good point. This shouldn't affect Linux guests as Linux is
>> always able to handle a MOPS exception coming from EL0. But it would
>> affect any non-Linux guest that pins all its EL0 tasks and doesn't
>> implement a handler. It's not clear to me what the expectation for
>> guests is, I'll ask the architects to clarify and get back to you.
> 
> My understanding is that MCE2 should always be set if the hypervisor
> can migrate vcpus across implementations behind EL1's back, and that
> in this context, EL1 never sees such an exception.

Notice that MCE2 only traps exceptions from EL1, not from EL0.
Exceptions from EL0 always go to EL1. Even if MCE2 is always set, EL1
will see the exception when the hypervisor migrates the vcpu while the
vcpu is executing a MOPS instruction in EL0.

Thanks,
Kristina
  

On Fri, 29 Sep 2023 15:51:32 +0100,
Kristina Martsenko <kristina.martsenko@arm.com> wrote:
> 
> On 29/09/2023 10:29, Marc Zyngier wrote:
> > On Thu, 28 Sep 2023 17:55:39 +0100,
> > Kristina Martsenko <kristina.martsenko@arm.com> wrote:
> >>
> >> On 27/09/2023 07:00, Oliver Upton wrote:
> >>>
> >>> On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote:
> >>>> Hi,
> >>>>
> >>>> This is v2 of the series to allow using the new Arm memory copy instructions
> >>>> in KVM guests. See v1 for more information [1].
> >>>
> >>>
> >>> Thanks for sending out the series. I've been thinking about what the
> >>> architecture says for MOPS, and I wonder if what's currently in the
> >>> Arm ARM is clear enough for EL1 software to be written robustly.
> >>>
> >>> While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS
> >>> exceptions from EL1, there's no such control for EL0. So when vCPU
> >>> migration occurs EL1 could get an unexpected MOPS exception, even for a
> >>> process that was pinned to a single (virtual) CPU implementation.
> >>>
> >>> Additionally, the wording of I_NXHPS seems to suggest that EL2 handling
> >>> of MOPS exceptions is only expected in certain circumstances where EL1 is
> >>> incapable of handling an exception. Is the unwritten expectation then
> >>> that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1
> >>> and EL0, even if EL1 did not migrate the PE context?
> >>>
> >>> Perhaps I'm being pedantic, but I'd really like for there to be some
> >>> documentation that suggests MOPS exceptions can happen due to context
> >>> migration done by a higher EL as that is the only option in the context
> >>> of virtualization.
> >>
> >> That's a good point. This shouldn't affect Linux guests as Linux is
> >> always able to handle a MOPS exception coming from EL0. But it would
> >> affect any non-Linux guest that pins all its EL0 tasks and doesn't
> >> implement a handler. It's not clear to me what the expectation for
> >> guests is, I'll ask the architects to clarify and get back to you.
> > 
> > My understanding is that MCE2 should always be set if the hypervisor
> > can migrate vcpus across implementations behind EL1's back, and that
> > in this context, EL1 never sees such an exception.
> 
> Notice that MCE2 only traps exceptions from EL1, not from EL0.
> Exceptions from EL0 always go to EL1. Even if MCE2 is always set, EL1
> will see the exception when the hypervisor migrates the vcpu while the
> vcpu is executing a MOPS instruction in EL0.

Ah, good point. I stand corrected.

	M.
  

On Fri, 22 Sep 2023 12:25:06 +0100,
Kristina Martsenko <kristina.martsenko@arm.com> wrote:
> 
> Hi,
> 
> This is v2 of the series to allow using the new Arm memory copy instructions
> in KVM guests. See v1 for more information [1].
> 
> Changes in v2:
>  - Dropped HCRX_EL2 vcpu field
>  - Rebased onto v6.6-rc2

FWIW, and despite my misgivings about the architecture:

Reviewed-by: Marc Zyngier <maz@kernel.org>

	M.
  

On Fri, 22 Sep 2023 12:25:06 +0100, Kristina Martsenko wrote:
> This is v2 of the series to allow using the new Arm memory copy instructions
> in KVM guests. See v1 for more information [1].
> 
> Changes in v2:
>  - Dropped HCRX_EL2 vcpu field
>  - Rebased onto v6.6-rc2
> 
> [...]

Applied to kvmarm/next, thanks!

[1/2] KVM: arm64: Add handler for MOPS exceptions
      https://git.kernel.org/kvmarm/kvmarm/c/17b8ac23488b
[2/2] KVM: arm64: Expose MOPS instructions to guests
      https://git.kernel.org/kvmarm/kvmarm/c/a24015b6cc66

--
Best,
Oliver