[v2,4/6] dt-bindings: Add RISC-V misaligned access performance

  From: Palmer Dabbelt <palmer@rivosinc.com>

This key allows device trees to specify the performance of misaligned
accesses to main memory regions from each CPU in the system.

Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
Signed-off-by: Evan Green <evan@rivosinc.com>
---

(no changes since v1)

 Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
  

On Mon, 06 Feb 2023 12:14:53 -0800, Evan Green wrote:
> From: Palmer Dabbelt <palmer@rivosinc.com>
> 
> This key allows device trees to specify the performance of misaligned
> accesses to main memory regions from each CPU in the system.
> 
> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> Signed-off-by: Evan Green <evan@rivosinc.com>
> ---
> 
> (no changes since v1)
> 
>  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 

My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
on your patch (DT_CHECKER_FLAGS is new in v5.13):

yamllint warnings/errors:
./Documentation/devicetree/bindings/riscv/cpus.yaml:91:72: [error] syntax error: mapping values are not allowed here (syntax)

dtschema/dtc warnings/errors:
make[1]: *** Deleting file 'Documentation/devicetree/bindings/riscv/cpus.example.dts'
Documentation/devicetree/bindings/riscv/cpus.yaml:91:72: mapping values are not allowed here
make[1]: *** [Documentation/devicetree/bindings/Makefile:26: Documentation/devicetree/bindings/riscv/cpus.example.dts] Error 1
make[1]: *** Waiting for unfinished jobs....
./Documentation/devicetree/bindings/riscv/cpus.yaml:91:72: mapping values are not allowed here
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/riscv/cpus.yaml: ignoring, error parsing file
make: *** [Makefile:1508: dt_binding_check] Error 2

doc reference errors (make refcheckdocs):

See https://patchwork.ozlabs.org/project/devicetree-bindings/patch/20230206201455.1790329-5-evan@rivosinc.com

The base for the series is generally the latest rc1. A different dependency
should be noted in *this* patch.

If you already ran 'make dt_binding_check' and didn't see the above
error(s), then make sure 'yamllint' is installed and dt-schema is up to
date:

pip3 install dtschema --upgrade

Please check and re-submit after running the above command yourself. Note
that DT_SCHEMA_FILES can be set to your schema file to speed up checking
your schema. However, it must be unset to test all examples with your schema.
  

On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
> From: Palmer Dabbelt <palmer@rivosinc.com>
> 
> This key allows device trees to specify the performance of misaligned
> accesses to main memory regions from each CPU in the system.
> 
> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> Signed-off-by: Evan Green <evan@rivosinc.com>
> ---
> 
> (no changes since v1)
> 
>  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml b/Documentation/devicetree/bindings/riscv/cpus.yaml
> index c6720764e765..2c09bd6f2927 100644
> --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
> +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
> @@ -85,6 +85,21 @@ properties:
>      $ref: "/schemas/types.yaml#/definitions/string"
>      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
>  
> +  riscv,misaligned-access-performance:
> +    description:
> +      Identifies the performance of misaligned memory accesses to main memory
> +      regions.  There are three flavors of unaligned access performance: "emulated"
> +      means that misaligned accesses are emulated via software and thus
> +      extremely slow, "slow" means that misaligned accesses are supported by
> +      hardware but still slower that aligned accesses sequences, and "fast"
> +      means that misaligned accesses are as fast or faster than the
> +      cooresponding aligned accesses sequences.
> +    $ref: "/schemas/types.yaml#/definitions/string"
> +    enum:
> +      - emulated
> +      - slow
> +      - fast

I don't think this belongs in DT. (I'm not sure about a userspace 
interface either.)

Can't this be tested and determined at runtime? Do misaligned accesses 
and compare the performance. We already do this for things like memcpy 
or crypto implementation selection.

Rob
  

From: Rob Herring
> Sent: 07 February 2023 17:06
> 
> On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
> > From: Palmer Dabbelt <palmer@rivosinc.com>
> >
> > This key allows device trees to specify the performance of misaligned
> > accesses to main memory regions from each CPU in the system.
> >
> > Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> > Signed-off-by: Evan Green <evan@rivosinc.com>
> > ---
> >
> > (no changes since v1)
> >
> >  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml
> b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > index c6720764e765..2c09bd6f2927 100644
> > --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
> > +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > @@ -85,6 +85,21 @@ properties:
> >      $ref: "/schemas/types.yaml#/definitions/string"
> >      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
> >
> > +  riscv,misaligned-access-performance:
> > +    description:
> > +      Identifies the performance of misaligned memory accesses to main memory
> > +      regions.  There are three flavors of unaligned access performance: "emulated"
> > +      means that misaligned accesses are emulated via software and thus
> > +      extremely slow, "slow" means that misaligned accesses are supported by
> > +      hardware but still slower that aligned accesses sequences, and "fast"
> > +      means that misaligned accesses are as fast or faster than the
> > +      cooresponding aligned accesses sequences.
> > +    $ref: "/schemas/types.yaml#/definitions/string"
> > +    enum:
> > +      - emulated
> > +      - slow
> > +      - fast
> 
> I don't think this belongs in DT. (I'm not sure about a userspace
> interface either.)
> 
> Can't this be tested and determined at runtime? Do misaligned accesses
> and compare the performance. We already do this for things like memcpy
> or crypto implementation selection.

There is also an long discussion about misaligned accesses
for loooongarch.

Basically if you want to run a common kernel (and userspace)
you have to default to compiling everything with -mno-stict-align
so that the compiler generates byte accesses for anything
marked 'packed' (etc).

Run-time tests can optimise some hot-spots.

In any case 'slow' is probably pointless - unless the accesses
take more than 1 or 2 extra cycles.

Oh, and you really never, ever want to emulate them.

Technically misaligned reads on (some) x86-64 cpu are slower
than aligned ones, but the difference is marginal.
I've measured two 64bit misaligned reads every clock.
But it is consistently slower by much less than one clock
per cache line.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
  

On Wed, 08 Feb 2023 04:45:10 PST (-0800), David.Laight@ACULAB.COM wrote:
> From: Rob Herring
>> Sent: 07 February 2023 17:06
>>
>> On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
>> > From: Palmer Dabbelt <palmer@rivosinc.com>
>> >
>> > This key allows device trees to specify the performance of misaligned
>> > accesses to main memory regions from each CPU in the system.
>> >
>> > Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
>> > Signed-off-by: Evan Green <evan@rivosinc.com>
>> > ---
>> >
>> > (no changes since v1)
>> >
>> >  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
>> >  1 file changed, 15 insertions(+)
>> >
>> > diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml
>> b/Documentation/devicetree/bindings/riscv/cpus.yaml
>> > index c6720764e765..2c09bd6f2927 100644
>> > --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
>> > +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
>> > @@ -85,6 +85,21 @@ properties:
>> >      $ref: "/schemas/types.yaml#/definitions/string"
>> >      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
>> >
>> > +  riscv,misaligned-access-performance:
>> > +    description:
>> > +      Identifies the performance of misaligned memory accesses to main memory
>> > +      regions.  There are three flavors of unaligned access performance: "emulated"
>> > +      means that misaligned accesses are emulated via software and thus
>> > +      extremely slow, "slow" means that misaligned accesses are supported by
>> > +      hardware but still slower that aligned accesses sequences, and "fast"
>> > +      means that misaligned accesses are as fast or faster than the
>> > +      cooresponding aligned accesses sequences.
>> > +    $ref: "/schemas/types.yaml#/definitions/string"
>> > +    enum:
>> > +      - emulated
>> > +      - slow
>> > +      - fast
>>
>> I don't think this belongs in DT. (I'm not sure about a userspace
>> interface either.)

[Kind of answered below.]

>> Can't this be tested and determined at runtime? Do misaligned accesses
>> and compare the performance. We already do this for things like memcpy
>> or crypto implementation selection.

We've had a history of broken firmware emulation of misaligned accesses 
wreaking havoc.  We don't run into concrete bugs there because we avoid 
misaligned accesses as much as possible in the kernel, but I'd be 
worried that we'd trigger a lot of these when probing for misaligned 
accesses.

> There is also an long discussion about misaligned accesses
> for loooongarch.
>
> Basically if you want to run a common kernel (and userspace)
> you have to default to compiling everything with -mno-stict-align
> so that the compiler generates byte accesses for anything
> marked 'packed' (etc).
>
> Run-time tests can optimise some hot-spots.
>
> In any case 'slow' is probably pointless - unless the accesses
> take more than 1 or 2 extra cycles.

[Also below.]

> Oh, and you really never, ever want to emulate them.

Unfortunately we're kind of stuck with this one: the specs used to 
require that misaligned accesses were supported and thus there's a bunch 
of firmwares that emulate them (and various misaligned accesses spread 
around, though they're kind of a mess).  The specs no longer require 
this support, but just dropping it from firmware will break binaries.

There's been some vague plans to dig out of this, but it'd require some 
sort of firmware interface additions in order to turn off the emulation 
and that's going to take a while.  As it stands we've got a bunch of 
users that just want to know when they can emit misaligned accesses.

> Technically misaligned reads on (some) x86-64 cpu are slower
> than aligned ones, but the difference is marginal.
> I've measured two 64bit misaligned reads every clock.
> But it is consistently slower by much less than one clock
> per cache line.

The "fast" case is explicitly written to catch that flavor of 
implementation.

The "slow" one is a bit vaguer, but the general idea is to catch 
implementations that end up with some sort of pipeline flush on 
misaligned accesses.  We've got a lot of very small in-order processors 
in RISC-V land, and while I haven't gotten around to benchmarking them 
all my guess is that the spec requirement for support ended up with some 
simple implementations.

FWIW: I checked the c906 RTL and it's setting some exception-related 
info on misaligned accesses, but I'd need to actually benchmark on to 
know for sure and they're kind of a headache to deal with.

>
> 	David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
  

On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
> From: Palmer Dabbelt <palmer@rivosinc.com>
> 
> This key allows device trees to specify the performance of misaligned
> accesses to main memory regions from each CPU in the system.
> 
> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> Signed-off-by: Evan Green <evan@rivosinc.com>
> ---
> 
> (no changes since v1)
> 
>  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml b/Documentation/devicetree/bindings/riscv/cpus.yaml
> index c6720764e765..2c09bd6f2927 100644
> --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
> +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
> @@ -85,6 +85,21 @@ properties:
>      $ref: "/schemas/types.yaml#/definitions/string"
>      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
>  
> +  riscv,misaligned-access-performance:
> +    description:
> +      Identifies the performance of misaligned memory accesses to main memory
> +      regions.  There are three flavors of unaligned access performance: "emulated"

Is the performance: emulated the source of the dt_binding_check issues?
And the fix is as simple as:
-    description:
+    description: |
?

> +      means that misaligned accesses are emulated via software and thus
> +      extremely slow, "slow" means that misaligned accesses are supported by
> +      hardware but still slower that aligned accesses sequences, and "fast"
> +      means that misaligned accesses are as fast or faster than the
> +      cooresponding aligned accesses sequences.
> +    $ref: "/schemas/types.yaml#/definitions/string"
> +    enum:
> +      - emulated
> +      - slow
> +      - fast
> +
>    # RISC-V requires 'timebase-frequency' in /cpus, so disallow it here
>    timebase-frequency: false
>  
> -- 
> 2.25.1
>
  

On Tue, Feb 14, 2023 at 1:26 PM Conor Dooley <conor@kernel.org> wrote:
>
> On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
> > From: Palmer Dabbelt <palmer@rivosinc.com>
> >
> > This key allows device trees to specify the performance of misaligned
> > accesses to main memory regions from each CPU in the system.
> >
> > Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> > Signed-off-by: Evan Green <evan@rivosinc.com>
> > ---
> >
> > (no changes since v1)
> >
> >  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > index c6720764e765..2c09bd6f2927 100644
> > --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
> > +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > @@ -85,6 +85,21 @@ properties:
> >      $ref: "/schemas/types.yaml#/definitions/string"
> >      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
> >
> > +  riscv,misaligned-access-performance:
> > +    description:
> > +      Identifies the performance of misaligned memory accesses to main memory
> > +      regions.  There are three flavors of unaligned access performance: "emulated"
>
> Is the performance: emulated the source of the dt_binding_check issues?
> And the fix is as simple as:
> -    description:
> +    description: |
> ?

Yep, I can pass cleanly with that change. Thanks!
  

On Thu, Feb 09, 2023 at 08:51:22AM -0800, Palmer Dabbelt wrote:
> On Wed, 08 Feb 2023 04:45:10 PST (-0800), David.Laight@ACULAB.COM wrote:
> > From: Rob Herring
> > > Sent: 07 February 2023 17:06
> > > 
> > > On Mon, Feb 06, 2023 at 12:14:53PM -0800, Evan Green wrote:
> > > > From: Palmer Dabbelt <palmer@rivosinc.com>
> > > >
> > > > This key allows device trees to specify the performance of misaligned
> > > > accesses to main memory regions from each CPU in the system.
> > > >
> > > > Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
> > > > Signed-off-by: Evan Green <evan@rivosinc.com>
> > > > ---
> > > >
> > > > (no changes since v1)
> > > >
> > > >  Documentation/devicetree/bindings/riscv/cpus.yaml | 15 +++++++++++++++
> > > >  1 file changed, 15 insertions(+)
> > > >
> > > > diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml
> > > b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > > > index c6720764e765..2c09bd6f2927 100644
> > > > --- a/Documentation/devicetree/bindings/riscv/cpus.yaml
> > > > +++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
> > > > @@ -85,6 +85,21 @@ properties:
> > > >      $ref: "/schemas/types.yaml#/definitions/string"
> > > >      pattern: ^rv(?:64|32)imaf?d?q?c?b?v?k?h?(?:_[hsxz](?:[a-z])+)*$
> > > >
> > > > +  riscv,misaligned-access-performance:
> > > > +    description:
> > > > +      Identifies the performance of misaligned memory accesses to main memory
> > > > +      regions.  There are three flavors of unaligned access performance: "emulated"
> > > > +      means that misaligned accesses are emulated via software and thus
> > > > +      extremely slow, "slow" means that misaligned accesses are supported by
> > > > +      hardware but still slower that aligned accesses sequences, and "fast"
> > > > +      means that misaligned accesses are as fast or faster than the
> > > > +      cooresponding aligned accesses sequences.
> > > > +    $ref: "/schemas/types.yaml#/definitions/string"
> > > > +    enum:
> > > > +      - emulated
> > > > +      - slow
> > > > +      - fast
> > > 
> > > I don't think this belongs in DT. (I'm not sure about a userspace
> > > interface either.)
> 
> [Kind of answered below.]
> 
> > > Can't this be tested and determined at runtime? Do misaligned accesses
> > > and compare the performance. We already do this for things like memcpy
> > > or crypto implementation selection.
> 
> We've had a history of broken firmware emulation of misaligned accesses
> wreaking havoc.  We don't run into concrete bugs there because we avoid
> misaligned accesses as much as possible in the kernel, but I'd be worried
> that we'd trigger a lot of these when probing for misaligned accesses.

Then how do you distinguish between emulated and working vs. emulated 
and broken? Sounds like the kernel running things would motivate fixing 
firmware. :) If not, then broken platforms can disable the check with a 
kernel command line flag. 

> 
> > There is also an long discussion about misaligned accesses
> > for loooongarch.
> > 
> > Basically if you want to run a common kernel (and userspace)
> > you have to default to compiling everything with -mno-stict-align
> > so that the compiler generates byte accesses for anything
> > marked 'packed' (etc).
> > 
> > Run-time tests can optimise some hot-spots.
> > 
> > In any case 'slow' is probably pointless - unless the accesses
> > take more than 1 or 2 extra cycles.
> 
> [Also below.]
> 
> > Oh, and you really never, ever want to emulate them.
> 
> Unfortunately we're kind of stuck with this one: the specs used to require
> that misaligned accesses were supported and thus there's a bunch of
> firmwares that emulate them (and various misaligned accesses spread around,
> though they're kind of a mess).  The specs no longer require this support,
> but just dropping it from firmware will break binaries.
> 
> There's been some vague plans to dig out of this, but it'd require some sort
> of firmware interface additions in order to turn off the emulation and
> that's going to take a while.  As it stands we've got a bunch of users that
> just want to know when they can emit misaligned accesses.
> 
> > Technically misaligned reads on (some) x86-64 cpu are slower
> > than aligned ones, but the difference is marginal.
> > I've measured two 64bit misaligned reads every clock.
> > But it is consistently slower by much less than one clock
> > per cache line.
> 
> The "fast" case is explicitly written to catch that flavor of
> implementation.
> 
> The "slow" one is a bit vaguer, but the general idea is to catch
> implementations that end up with some sort of pipeline flush on misaligned
> accesses.  We've got a lot of very small in-order processors in RISC-V land,
> and while I haven't gotten around to benchmarking them all my guess is that
> the spec requirement for support ended up with some simple implementations.

If userspace wants to get into microarchitecture level optimizations, it 
should just look at the CPU model. IOW, use the CPU compatible to infer 
things rather than continuously adding properties in an adhoc manor 
trying to parameterize everything.

Rob