[v9,16/42] mm: Add guard pages around a shadow stack.

  The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.

The architecture of shadow stack constrains the ability of userspace to
move the shadow stack pointer (SSP) in order to prevent corrupting or
switching to other shadow stacks. The RSTORSSP instruction can move the
SSP to different shadow stacks, but it requires a specially placed token
in order to do this. However, the architecture does not prevent
incrementing the stack pointer to wander onto an adjacent shadow stack. To
prevent this in software, enforce guard pages at the beginning of shadow
stack VMAs, such that there will always be a gap between adjacent shadow
stacks.

Make the gap big enough so that no userspace SSP changing operations
(besides RSTORSSP), can move the SSP from one stack to the next. The
SSP can be incremented or decremented by CALL, RET  and INCSSP. CALL and
RET can move the SSP by a maximum of 8 bytes, at which point the shadow
stack would be accessed.

The INCSSP instruction can also increment the shadow stack pointer. It
is the shadow stack analog of an instruction like:

        addq    $0x80, %rsp

However, there is one important difference between an ADD on %rsp and
INCSSP. In addition to modifying SSP, INCSSP also reads from the memory
of the first and last elements that were "popped". It can be thought of
as acting like this:

READ_ONCE(ssp);       // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8);     // read+discard last popped stack element

The maximum distance INCSSP can move the SSP is 2040 bytes, before it
would read the memory. Therefore, a single page gap will be enough to
prevent any operation from shifting the SSP to an adjacent stack, since
it would have to land in the gap at least once, causing a fault.

This could be accomplished by using VM_GROWSDOWN, but this has a
downside. The behavior would allow shadow stacks to grow, which is
unneeded and adds a strange difference to how most regular stacks work.

In the maple tree code, there is some logic for retrying the unmapped
area search if a guard gap is violated. This retry should happen for
shadow stack guard gap violations as well. This logic currently only
checks for VM_GROWSDOWN for start gaps. Since shadow stacks also have
a start gap as well, create an new define VM_STARTGAP_FLAGS to hold
all the VM flag bits that have start gaps, and make mmap use it.

Co-developed-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Tested-by: Pengfei Xu <pengfei.xu@intel.com>
Tested-by: John Allen <john.allen@amd.com>
Tested-by: Kees Cook <keescook@chromium.org>
---
v9:
 - Add logic needed to still have guard gaps with maple tree.
---
 include/linux/mm.h | 54 ++++++++++++++++++++++++++++++++++++++++------
 mm/mmap.c          |  4 ++--
 2 files changed, 50 insertions(+), 8 deletions(-)
  

On Mon, Jun 12, 2023 at 05:10:42PM -0700, Rick Edgecombe wrote:
> The x86 Control-flow Enforcement Technology (CET) feature includes a new
> type of memory called shadow stack. This shadow stack memory has some
> unusual properties, which requires some core mm changes to function
> properly.

Reviewed-by: Mark Brown <broonie@kernel.org>
  

On Mon, Jun 12, 2023 at 05:10:42PM -0700, Rick Edgecombe wrote:
> +++ b/include/linux/mm.h
> @@ -342,7 +342,36 @@ extern unsigned int kobjsize(const void *objp);
>  #endif /* CONFIG_ARCH_HAS_PKEYS */
>  
>  #ifdef CONFIG_X86_USER_SHADOW_STACK
> -# define VM_SHADOW_STACK	VM_HIGH_ARCH_5 /* Should not be set with VM_SHARED */
> +/*
> + * This flag should not be set with VM_SHARED because of lack of support
> + * core mm. It will also get a guard page. This helps userspace protect
> + * itself from attacks. The reasoning is as follows:
> + *
> + * The shadow stack pointer(SSP) is moved by CALL, RET, and INCSSPQ. The
> + * INCSSP instruction can increment the shadow stack pointer. It is the
> + * shadow stack analog of an instruction like:
> + *
> + *   addq $0x80, %rsp
> + *
> + * However, there is one important difference between an ADD on %rsp
> + * and INCSSP. In addition to modifying SSP, INCSSP also reads from the
> + * memory of the first and last elements that were "popped". It can be
> + * thought of as acting like this:
> + *
> + * READ_ONCE(ssp);       // read+discard top element on stack
> + * ssp += nr_to_pop * 8; // move the shadow stack
> + * READ_ONCE(ssp-8);     // read+discard last popped stack element
> + *
> + * The maximum distance INCSSP can move the SSP is 2040 bytes, before
> + * it would read the memory. Therefore a single page gap will be enough
> + * to prevent any operation from shifting the SSP to an adjacent stack,
> + * since it would have to land in the gap at least once, causing a
> + * fault.
> + *
> + * Prevent using INCSSP to move the SSP between shadow stacks by
> + * having a PAGE_SIZE guard gap.
> + */
> +# define VM_SHADOW_STACK	VM_HIGH_ARCH_5
>  #else
>  # define VM_SHADOW_STACK	VM_NONE
>  #endif

This is a lot of very x86-specific language in a generic header file.
I'm sure there's a better place for all this text.
  

On Thu, 2023-06-22 at 19:21 +0100, Matthew Wilcox wrote:
> On Mon, Jun 12, 2023 at 05:10:42PM -0700, Rick Edgecombe wrote:
> > +++ b/include/linux/mm.h
> > @@ -342,7 +342,36 @@ extern unsigned int kobjsize(const void
> > *objp);
> >   #endif /* CONFIG_ARCH_HAS_PKEYS */
> >   
> >   #ifdef CONFIG_X86_USER_SHADOW_STACK
> > -# define VM_SHADOW_STACK       VM_HIGH_ARCH_5 /* Should not be set
> > with VM_SHARED */
> > +/*
> > + * This flag should not be set with VM_SHARED because of lack of
> > support
> > + * core mm. It will also get a guard page. This helps userspace
> > protect
> > + * itself from attacks. The reasoning is as follows:
> > + *
> > + * The shadow stack pointer(SSP) is moved by CALL, RET, and
> > INCSSPQ. The
> > + * INCSSP instruction can increment the shadow stack pointer. It
> > is the
> > + * shadow stack analog of an instruction like:
> > + *
> > + *   addq $0x80, %rsp
> > + *
> > + * However, there is one important difference between an ADD on
> > %rsp
> > + * and INCSSP. In addition to modifying SSP, INCSSP also reads
> > from the
> > + * memory of the first and last elements that were "popped". It
> > can be
> > + * thought of as acting like this:
> > + *
> > + * READ_ONCE(ssp);       // read+discard top element on stack
> > + * ssp += nr_to_pop * 8; // move the shadow stack
> > + * READ_ONCE(ssp-8);     // read+discard last popped stack element
> > + *
> > + * The maximum distance INCSSP can move the SSP is 2040 bytes,
> > before
> > + * it would read the memory. Therefore a single page gap will be
> > enough
> > + * to prevent any operation from shifting the SSP to an adjacent
> > stack,
> > + * since it would have to land in the gap at least once, causing a
> > + * fault.
> > + *
> > + * Prevent using INCSSP to move the SSP between shadow stacks by
> > + * having a PAGE_SIZE guard gap.
> > + */
> > +# define VM_SHADOW_STACK       VM_HIGH_ARCH_5
> >   #else
> >   # define VM_SHADOW_STACK      VM_NONE
> >   #endif
> 
> This is a lot of very x86-specific language in a generic header file.
> I'm sure there's a better place for all this text.

Yes, I couldn't find another place for it. This was the reasoning:
https://lore.kernel.org/lkml/07deaffc10b1b68721bbbce370e145d8fec2a494.camel@intel.com/

Did you have any particular place in mind?
  

On Thu, Jun 22, 2023 at 06:27:40PM +0000, Edgecombe, Rick P wrote:
> On Thu, 2023-06-22 at 19:21 +0100, Matthew Wilcox wrote:
> > On Mon, Jun 12, 2023 at 05:10:42PM -0700, Rick Edgecombe wrote:
> > > +++ b/include/linux/mm.h
> > > @@ -342,7 +342,36 @@ extern unsigned int kobjsize(const void
> > > *objp);
> > >   #endif /* CONFIG_ARCH_HAS_PKEYS */
> > >   
> > >   #ifdef CONFIG_X86_USER_SHADOW_STACK
> > > -# define VM_SHADOW_STACK       VM_HIGH_ARCH_5 /* Should not be set
> > > with VM_SHARED */
> > > +/*
> > > + * This flag should not be set with VM_SHARED because of lack of
> > > support
> > > + * core mm. It will also get a guard page. This helps userspace
> > > protect
> > > + * itself from attacks. The reasoning is as follows:
> > > + *
> > > + * The shadow stack pointer(SSP) is moved by CALL, RET, and
> > > INCSSPQ. The
> > > + * INCSSP instruction can increment the shadow stack pointer. It
> > > is the
> > > + * shadow stack analog of an instruction like:
> > > + *
> > > + *   addq $0x80, %rsp
> > > + *
> > > + * However, there is one important difference between an ADD on
> > > %rsp
> > > + * and INCSSP. In addition to modifying SSP, INCSSP also reads
> > > from the
> > > + * memory of the first and last elements that were "popped". It
> > > can be
> > > + * thought of as acting like this:
> > > + *
> > > + * READ_ONCE(ssp);       // read+discard top element on stack
> > > + * ssp += nr_to_pop * 8; // move the shadow stack
> > > + * READ_ONCE(ssp-8);     // read+discard last popped stack element
> > > + *
> > > + * The maximum distance INCSSP can move the SSP is 2040 bytes,
> > > before
> > > + * it would read the memory. Therefore a single page gap will be
> > > enough
> > > + * to prevent any operation from shifting the SSP to an adjacent
> > > stack,
> > > + * since it would have to land in the gap at least once, causing a
> > > + * fault.
> > > + *
> > > + * Prevent using INCSSP to move the SSP between shadow stacks by
> > > + * having a PAGE_SIZE guard gap.
> > > + */
> > > +# define VM_SHADOW_STACK       VM_HIGH_ARCH_5
> > >   #else
> > >   # define VM_SHADOW_STACK      VM_NONE
> > >   #endif
> > 
> > This is a lot of very x86-specific language in a generic header file.
> > I'm sure there's a better place for all this text.
> 
> Yes, I couldn't find another place for it. This was the reasoning:
> https://lore.kernel.org/lkml/07deaffc10b1b68721bbbce370e145d8fec2a494.camel@intel.com/
> 
> Did you have any particular place in mind?

Since it's near CONFIG_X86_USER_SHADOW_STACK the comment in mm.h could be 

/*
 * VMA is used for shadow stack and implies guard pages.
 * See arch/x86/kernel/shstk.c for details
 */

and the long reasoning comment can be moved near alloc_shstk in
arch/x86/kernel/shstk.h
  

On Fri, Jun 23, 2023 at 10:40:00AM +0300, Mike Rapoport wrote:
> On Thu, Jun 22, 2023 at 06:27:40PM +0000, Edgecombe, Rick P wrote:

> > Yes, I couldn't find another place for it. This was the reasoning:
> > https://lore.kernel.org/lkml/07deaffc10b1b68721bbbce370e145d8fec2a494.camel@intel.com/

> > Did you have any particular place in mind?

> Since it's near CONFIG_X86_USER_SHADOW_STACK the comment in mm.h could be 

> /*
>  * VMA is used for shadow stack and implies guard pages.
>  * See arch/x86/kernel/shstk.c for details
>  */

> and the long reasoning comment can be moved near alloc_shstk in
> arch/x86/kernel/shstk.h

This isn't an x86 specific concept, arm64 has a very similar extension
called Guarded Control Stack (which I should be publishing changes for
in the not too distant future) and riscv also has something.  For arm64
I'm using the generic mm changes wholesale, we have a similar need for
guard pages around the GCS and while the mechanics of accessing are
different the requirement ends up being the same.  Perhaps we could just
rewrite the comment to say that guard pages prevent over/underflow of
the stack by userspace and that a single page is sufficient for all
current architectures, with the details of the working for x86 put in
some x86 specific place?
  

On Fri, 2023-06-23 at 13:17 +0100, Mark Brown wrote:
> On Fri, Jun 23, 2023 at 10:40:00AM +0300, Mike Rapoport wrote:
> > On Thu, Jun 22, 2023 at 06:27:40PM +0000, Edgecombe, Rick P wrote:
> 
> > > Yes, I couldn't find another place for it. This was the
> > > reasoning:
> > > 
> https://lore.kernel.org/lkml/07deaffc10b1b68721bbbce370e145d8fec2a494.camel@intel.com/
> 
> > > Did you have any particular place in mind?
> 
> > Since it's near CONFIG_X86_USER_SHADOW_STACK the comment in mm.h
> could be 
> 
> > /*
> >   * VMA is used for shadow stack and implies guard pages.
> >   * See arch/x86/kernel/shstk.c for details
> >   */
> 
> > and the long reasoning comment can be moved near alloc_shstk in
> > arch/x86/kernel/shstk.h

Makes sense. Not sure why I didn't think of this earlier.

> 
> This isn't an x86 specific concept, arm64 has a very similar
> extension
> called Guarded Control Stack (which I should be publishing changes
> for
> in the not too distant future) and riscv also has something.  For
> arm64
> I'm using the generic mm changes wholesale, we have a similar need
> for
> guard pages around the GCS and while the mechanics of accessing are
> different the requirement ends up being the same.  Perhaps we could
> just
> rewrite the comment to say that guard pages prevent over/underflow of
> the stack by userspace and that a single page is sufficient for all
> current architectures, with the details of the working for x86 put in
> some x86 specific place?

Something sort of similar came up in regards to the riscv series, about
adding something like an is_shadow_stack_vma() helper. The plan was to
not make too many assumptions about the final details of the other
shadow stack features and leave that for refactoring. I think some kind
of generic comment like you suggest makes sense, but I don't want to
try to assert any arch specifics for features that are not upstream. It
should be very easy to tweak the comment when the time comes.

The points about x86 details not belonging in non-arch headers and
having some arch generic explanation in the file are well taken though.
  

On Sun, Jun 25, 2023 at 04:44:32PM +0000, Edgecombe, Rick P wrote:
> On Fri, 2023-06-23 at 13:17 +0100, Mark Brown wrote:

> > This isn't an x86 specific concept, arm64 has a very similar
> > extension
> > called Guarded Control Stack (which I should be publishing changes
> > for
> > in the not too distant future) and riscv also has something.  For
> > arm64
> > I'm using the generic mm changes wholesale, we have a similar need
> > for
> > guard pages around the GCS and while the mechanics of accessing are
> > different the requirement ends up being the same.  Perhaps we could
> > just
> > rewrite the comment to say that guard pages prevent over/underflow of
> > the stack by userspace and that a single page is sufficient for all
> > current architectures, with the details of the working for x86 put in
> > some x86 specific place?

> Something sort of similar came up in regards to the riscv series, about
> adding something like an is_shadow_stack_vma() helper. The plan was to
> not make too many assumptions about the final details of the other
> shadow stack features and leave that for refactoring. I think some kind
> of generic comment like you suggest makes sense, but I don't want to
> try to assert any arch specifics for features that are not upstream. It
> should be very easy to tweak the comment when the time comes.

> The points about x86 details not belonging in non-arch headers and
> having some arch generic explanation in the file are well taken though.

I think a statement to the effect that "this works for currently
supported architectures" is fine, if something comes along with
additional requirements then the comment can be adjusted as part of
merging the new thing.