[v4,10/39] x86/mm: Introduce _PAGE_COW

  Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.

The reason it's lightly used is that Dirty=1 is normally set by HW
_before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.

So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
in places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other
words, whenever Linux needs to create Write=0,Dirty=1, it instead creates
Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1.
Further differentiated by VMA flags, these PTE bit combinations would be
set as follows for various types of memory:

(Write=0,Cow=1,Dirty=0):
 - A modified, copy-on-write (COW) page. Previously when a typical
   anonymous writable mapping was made COW via fork(), the kernel would
   mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This
   happens in copy_present_pte().
 - A R/O page that has been COW'ed. The user page is in a R/O VMA,
   and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault
   handler creates a copy of the page and sets the new copy's PTE as
   Write=0 and Cow=1.
 - A shared shadow stack PTE. When a shadow stack page is being shared
   among processes (this happens at fork()), its PTE is made Dirty=0, so
   the next shadow stack access causes a fault, and the page is
   duplicated and Dirty=1 is set again. This is the COW equivalent for
   shadow stack pages, even though it's copy-on-access rather than
   copy-on-write.

(Write=0,Cow=0,Dirty=1):
 - A shadow stack PTE.
 - A Cow PTE created when a processor without shadow stack support set
   Dirty=1.

There are six bits left available to software in the 64-bit PTE after
consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels
because shadow stacks are not enabled there.

This is a prepratory patch. Changes to actually start marking _PAGE_COW
will follow once other pieces are in place.

Tested-by: Pengfei Xu <pengfei.xu@intel.com>
Tested-by: John Allen <john.allen@amd.com>
Co-developed-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---

v4:
 - Teach pte_flags_need_flush() about _PAGE_COW bit
 - Break apart patch for better bisectability

v3:
 - Add comment around _PAGE_TABLE in response to comment
   from (Andrew Cooper)
 - Check for PSE in pmd_shstk (Andrew Cooper)
 - Get to the point quicker in commit log (Andrew Cooper)
 - Clarify and reorder commit log for why the PTE bit examples have
   multiple entries. Apply same changes for comment. (peterz)
 - Fix comment that implied dirty bit for COW was a specific x86 thing
   (peterz)
 - Fix swapping of Write/Dirty (PeterZ)

v2:
 - Update commit log with comments (Dave Hansen)
 - Add comments in code to explain pte modification code better (Dave)
 - Clarify info on the meaning of various Write,Cow,Dirty combinations

 arch/x86/include/asm/pgtable.h       | 78 ++++++++++++++++++++++++++++
 arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++--
 arch/x86/include/asm/tlbflush.h      |  3 +-
 3 files changed, 134 insertions(+), 6 deletions(-)
  

On Fri, Dec 02, 2022 at 04:35:37PM -0800, Rick Edgecombe wrote:
> Some OSes have a greater dependence on software available bits in PTEs than
> Linux. That left the hardware architects looking for a way to represent a
> new memory type (shadow stack) within the existing bits. They chose to
> repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
> shadow stack memory, Linux should avoid creating memory with this PTE bit
> combination unless it intends for it to be shadow stack.
> 
> The reason it's lightly used is that Dirty=1 is normally set by HW
> _before_ a write. A write with a Write=0 PTE would typically only generate
> a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
> generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
> supports shadow stacks will no longer exhibit this oddity.
> 
> So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
> in places where Linux normally creates Write=0,Dirty=1, it can use the
> software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other
> words, whenever Linux needs to create Write=0,Dirty=1, it instead creates
> Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1.
> Further differentiated by VMA flags, these PTE bit combinations would be
> set as follows for various types of memory:
> 
> (Write=0,Cow=1,Dirty=0):
>  - A modified, copy-on-write (COW) page. Previously when a typical
>    anonymous writable mapping was made COW via fork(), the kernel would
>    mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This
>    happens in copy_present_pte().
>  - A R/O page that has been COW'ed. The user page is in a R/O VMA,
>    and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault
>    handler creates a copy of the page and sets the new copy's PTE as
>    Write=0 and Cow=1.
>  - A shared shadow stack PTE. When a shadow stack page is being shared
>    among processes (this happens at fork()), its PTE is made Dirty=0, so
>    the next shadow stack access causes a fault, and the page is
>    duplicated and Dirty=1 is set again. This is the COW equivalent for
>    shadow stack pages, even though it's copy-on-access rather than
>    copy-on-write.
> 
> (Write=0,Cow=0,Dirty=1):
>  - A shadow stack PTE.
>  - A Cow PTE created when a processor without shadow stack support set
>    Dirty=1.
> 
> There are six bits left available to software in the 64-bit PTE after
> consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels
> because shadow stacks are not enabled there.
> 
> This is a prepratory patch. Changes to actually start marking _PAGE_COW
> will follow once other pieces are in place.
> 
> Tested-by: Pengfei Xu <pengfei.xu@intel.com>
> Tested-by: John Allen <john.allen@amd.com>
> Co-developed-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
> Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com>

Reviewed-by: Kees Cook <keescook@chromium.org>
  

On Fri, Dec 02, 2022 at 04:35:37PM -0800, Rick Edgecombe wrote:
> There are six bits left available to software in the 64-bit PTE after
> consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels
> because shadow stacks are not enabled there.
> 
> This is a prepratory patch. Changes to actually start marking _PAGE_COW

Unknown word [prepratory] in commit message.
Suggestions: ['preparatory',

> will follow once other pieces are in place.

And regardless, you don't really need this sentence at all, AFAICT.

...

> +/*
> + * Normally COW memory can result in Dirty=1,Write=0 PTs. But in the case
							^^^

PTEs.

> + * of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the
> + * Dirty=1,Write=0 will result in the memory being treated as shaodw stack
> + * by the HW. So when creating COW memory, a software bit is used
> + * _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow()
> + * take a PTE marked conventially COW (Dirty=1) and transition it to the

Unknown word [conventially] in comment.
Suggestions: ['conventionally', ...

> + * shadow stack compatible version of COW (Cow=1).
> + */
> +

^ Superfluous newline.

> +static inline pte_t pte_mkcow(pte_t pte)
> +{
> +	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
> +		return pte;
> +
> +	pte = pte_clear_flags(pte, _PAGE_DIRTY);
> +	return pte_set_flags(pte, _PAGE_COW);
> +}
> +
> +static inline pte_t pte_clear_cow(pte_t pte)
> +{
> +	/*
> +	 * _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK kernels.

I'm guessing this "unnecessary" is supposed to mean that on kernels not
supporting shadow stack, a COW page uses the old bit flags?

I.e., Dirty=1,Write=0?

Might as well write it this way to be perfectly clear.

> +	 * See the _PAGE_COW definition for more details.
> +	 */
> +	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
> +		return pte;
> +
> +	/*
> +	 * PTE is getting copied-on-write, so it will be dirtied
> +	 * if writable, or made shadow stack if shadow stack and
> +	 * being copied on access. Set they dirty bit for both

"Set the dirty bit.."

> +	 * cases.
> +	 */
> +	pte = pte_set_flags(pte, _PAGE_DIRTY);
> +	return pte_clear_flags(pte, _PAGE_COW);
> +}

Rest looks ok.

Thx.
  

On Tue, 2022-12-20 at 22:29 +0100, Borislav Petkov wrote:
> On Fri, Dec 02, 2022 at 04:35:37PM -0800, Rick Edgecombe wrote:
> > There are six bits left available to software in the 64-bit PTE
> > after
> > consuming a bit for _PAGE_COW. No space is consumed in 32-bit
> > kernels
> > because shadow stacks are not enabled there.
> > 
> > This is a prepratory patch. Changes to actually start marking
> > _PAGE_COW
> 
> Unknown word [prepratory] in commit message.
> Suggestions: ['preparatory',

Sorry about all these spelling errors.

> 
> > will follow once other pieces are in place.
> 
> And regardless, you don't really need this sentence at all, AFAICT.
> 
> ...

Ok.

> 
> > +/*
> > + * Normally COW memory can result in Dirty=1,Write=0 PTs. But in
> > the case
>                                                         ^^^
> 
> PTEs.

Thanks.

> 
> > + * of X86_FEATURE_USER_SHSTK, the software COW bit is used, since
> > the
> > + * Dirty=1,Write=0 will result in the memory being treated as
> > shaodw stack
> > + * by the HW. So when creating COW memory, a software bit is used
> > + * _PAGE_BIT_COW. The following functions pte_mkcow() and
> > pte_clear_cow()
> > + * take a PTE marked conventially COW (Dirty=1) and transition it
> > to the
> 
> Unknown word [conventially] in comment.
> Suggestions: ['conventionally', ...
> 
> > + * shadow stack compatible version of COW (Cow=1).
> > + */
> > +
> 
> ^ Superfluous newline.

Thanks.

> 
> > +static inline pte_t pte_mkcow(pte_t pte)
> > +{
> > +       if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
> > +               return pte;
> > +
> > +       pte = pte_clear_flags(pte, _PAGE_DIRTY);
> > +       return pte_set_flags(pte, _PAGE_COW);
> > +}
> > +
> > +static inline pte_t pte_clear_cow(pte_t pte)
> > +{
> > +       /*
> > +        * _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK
> > kernels.
> 
> I'm guessing this "unnecessary" is supposed to mean that on kernels
> not
> supporting shadow stack, a COW page uses the old bit flags?
> 
> I.e., Dirty=1,Write=0?
> 
> Might as well write it this way to be perfectly clear.

Right, I can clarify.

> 
> > +        * See the _PAGE_COW definition for more details.
> > +        */
> > +       if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
> > +               return pte;
> > +
> > +       /*
> > +        * PTE is getting copied-on-write, so it will be dirtied
> > +        * if writable, or made shadow stack if shadow stack and
> > +        * being copied on access. Set they dirty bit for both
> 
> "Set the dirty bit.."

Oof.

> 
> > +        * cases.
> > +        */
> > +       pte = pte_set_flags(pte, _PAGE_DIRTY);
> > +       return pte_clear_flags(pte, _PAGE_COW);
> > +}
> 
> Rest looks ok.

Thanks.

> 
> Thx.
>