[RFC,V1,00/11] riscv: Introduce 64K base page

  Some existing architectures like ARM supports base page larger than 4K
as their MMU supports more page sizes. Thus, besides hugetlb page and
transparent huge page, there is another way for these architectures to
enjoy the benefits of fewer TLB misses without worrying about cost of
splitting and merging huge pages. However, on architectures with only
4K MMU, larger base page is unavailable now.

This patch series attempts to break through the limitation of MMU and
supports larger base page on RISC-V, which only supports 4K page size
now.

The key idea to implement larger base page based on 4K MMU is to
decouple the MMU page from the base page in view of kernel mm, which we
denote as software page. In contrary to software page, we denote the MMU
page as hardware page. Below is the difference between these two kinds
of pages.

1. Kernel memory management module manages, allocates and maps memory at
a granularity of software page, which should not be restricted by
MMU and can be larger than hardware page.

2. Architecture page table operations should be carried out from MMU's
perspective and page table entries are encoded at a granularity of
hardware page, which is 4K on RISC-V MMU now.

The main work to decouple these two kinds of pages lies in architecture
code. For example, we turn the pte_t struct to an array of page table
entries to match it with software page which can be larger than hardware
page, and adapt the page table operations accordingly. For 64K software
base page, the pte_t struct now contains 16 contiguous page table
entries which point to 16 contiguous 4K hardware pages.

To achieve the benefits of large base page, we applies Svnapot for each
base page's mapping. The Svnapot extension on RISC-V is like contiguous
PTE on ARM64. It allows ptes of a naturally aligned power-of 2 size
memory range be encoded in the same format to save the TLB space.

This patch series is the first version and is based on v6.7-rc1. This
version supports both bare metal and virtualization scenarios.

In the next versions, we will continue on the following works:

1. Reduce the memory usage of page table page as it only uses 4K space
while costs a whole base page.

2. When IMSIC interrupt file is smaller than 64K, extra isolation
measures for the interrupt file are needed. (S)PMP and IOPMP may be good
choices.

3. More consideration is needed to make this patch series collaborate
with folios better.

4. Support 64K base page on IOMMU.

5. The performance test is on schedule to verify the actual performance
improvement and the decrease in TLB miss rate.

Thanks in advance for comments.

Xu Lu (11):
  mm: Fix misused APIs on huge pte
  riscv: Introduce concept of hardware base page
  riscv: Adapt pte struct to gap between hw page and sw page
  riscv: Adapt pte operations to gap between hw page and sw page
  riscv: Decouple pmd operations and pte operations
  riscv: Distinguish pmd huge pte and napot huge pte
  riscv: Adapt satp operations to gap between hw page and sw page
  riscv: Apply Svnapot for base page mapping
  riscv: Adjust fix_btmap slots number to match variable page size
  riscv: kvm: Adapt kvm to gap between hw page and sw page
  riscv: Introduce 64K page size

 arch/Kconfig                        |   1 +
 arch/riscv/Kconfig                  |  28 +++
 arch/riscv/include/asm/fixmap.h     |   3 +-
 arch/riscv/include/asm/hugetlb.h    |  71 ++++++-
 arch/riscv/include/asm/page.h       |  16 +-
 arch/riscv/include/asm/pgalloc.h    |  21 ++-
 arch/riscv/include/asm/pgtable-32.h |   2 +-
 arch/riscv/include/asm/pgtable-64.h |  45 +++--
 arch/riscv/include/asm/pgtable.h    | 282 +++++++++++++++++++++++-----
 arch/riscv/kernel/efi.c             |   2 +-
 arch/riscv/kernel/head.S            |   4 +-
 arch/riscv/kernel/hibernate.c       |   3 +-
 arch/riscv/kvm/mmu.c                | 198 +++++++++++++------
 arch/riscv/mm/context.c             |   7 +-
 arch/riscv/mm/fault.c               |   1 +
 arch/riscv/mm/hugetlbpage.c         |  42 +++--
 arch/riscv/mm/init.c                |  25 +--
 arch/riscv/mm/kasan_init.c          |   7 +-
 arch/riscv/mm/pageattr.c            |   2 +-
 fs/proc/task_mmu.c                  |   2 +-
 include/asm-generic/hugetlb.h       |   7 +
 include/asm-generic/pgtable-nopmd.h |   1 +
 include/linux/pgtable.h             |   6 +
 mm/hugetlb.c                        |   2 +-
 mm/migrate.c                        |   5 +-
 mm/mprotect.c                       |   2 +-
 mm/rmap.c                           |  10 +-
 mm/vmalloc.c                        |   3 +-
 28 files changed, 616 insertions(+), 182 deletions(-)
  

On Thu, Nov 23, 2023, at 07:56, Xu Lu wrote:
> Some existing architectures like ARM supports base page larger than 4K
> as their MMU supports more page sizes. Thus, besides hugetlb page and
> transparent huge page, there is another way for these architectures to
> enjoy the benefits of fewer TLB misses without worrying about cost of
> splitting and merging huge pages. However, on architectures with only
> 4K MMU, larger base page is unavailable now.
>
> This patch series attempts to break through the limitation of MMU and
> supports larger base page on RISC-V, which only supports 4K page size
> now.
>
> The key idea to implement larger base page based on 4K MMU is to
> decouple the MMU page from the base page in view of kernel mm, which we
> denote as software page. In contrary to software page, we denote the MMU
> page as hardware page. Below is the difference between these two kinds
> of pages.

We have played with this on arm32, but the conclusion is that it's
almost never worth the memory overhead, as most workloads end up
using several times the amount of physical RAM after each small
file in the page cache and any sparse populated anonymous memory
area explodes to up to 16 times the size.

On ppc64, using 64KB pages was way to get around limitations in
their hashed MMU design, which had a much bigger performance impact
because any page table access ends up being a cache miss. On arm64,
there are some CPUs like the Fujitsu A64FX that are really bad at
4KB pages and don't support 16KB pages, so this is the only real
option.

You will see a notable performance benefit in synthetic benchmarks
like speccpu with 64KB pages, or on specific computational
workloads that have large densely packed memory chunks, but for
real workloads, the usual answer is to just use transparent
hugepages for larger mappings and a page size of no more than
16KB for the page cache.

With the work going into using folios in the kernel (see e.g.
https://lwn.net/Articles/932386/), even the workloads that
benefit from 64KB base pages should be better off with 4KB
pages and just using the TLB hints for large folios.

     Arnd
  

Thanks a lot for your reply! And sorry for replying so late.

On Thu, Nov 23, 2023 at 5:30 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Thu, Nov 23, 2023, at 07:56, Xu Lu wrote:
> > Some existing architectures like ARM supports base page larger than 4K
> > as their MMU supports more page sizes. Thus, besides hugetlb page and
> > transparent huge page, there is another way for these architectures to
> > enjoy the benefits of fewer TLB misses without worrying about cost of
> > splitting and merging huge pages. However, on architectures with only
> > 4K MMU, larger base page is unavailable now.
> >
> > This patch series attempts to break through the limitation of MMU and
> > supports larger base page on RISC-V, which only supports 4K page size
> > now.
> >
> > The key idea to implement larger base page based on 4K MMU is to
> > decouple the MMU page from the base page in view of kernel mm, which we
> > denote as software page. In contrary to software page, we denote the MMU
> > page as hardware page. Below is the difference between these two kinds
> > of pages.
>
> We have played with this on arm32, but the conclusion is that it's
> almost never worth the memory overhead, as most workloads end up
> using several times the amount of physical RAM after each small
> file in the page cache and any sparse populated anonymous memory
> area explodes to up to 16 times the size.
>
> On ppc64, using 64KB pages was way to get around limitations in
> their hashed MMU design, which had a much bigger performance impact
> because any page table access ends up being a cache miss. On arm64,
> there are some CPUs like the Fujitsu A64FX that are really bad at
> 4KB pages and don't support 16KB pages, so this is the only real
> option.
>
> You will see a notable performance benefit in synthetic benchmarks
> like speccpu with 64KB pages, or on specific computational
> workloads that have large densely packed memory chunks, but for
> real workloads, the usual answer is to just use transparent
> hugepages for larger mappings and a page size of no more than
> 16KB for the page cache.

Actually we did find actual performance benefits brought by 64K page
size in real business scenarios.
On the Ampere ARM server, when applying 64K base page size, we saw an
improvement of 2.5x for both qps and latency on redis, a performance
improvement of 10~20% on our own newsql database and 50% on object
storage.
For mysql, the qps increases about 14%, 17.5% and 20% for read-only,
write-only and random read/write workloads respectively. And the
latency reduces about 13.7%, 15.8% and 14.5% on average.
This is also why we chose to implement a similar feature on RISC-V in
the beginning.

>
> With the work going into using folios in the kernel (see e.g.
> https://lwn.net/Articles/932386/), even the workloads that
> benefit from 64KB base pages should be better off with 4KB
> pages and just using the TLB hints for large folios.

Maybe 64K page size combined with large folios can achieve more benefits.
As is mentioned in this patch[1], a 64K page size kernel combined with
large folios and THPs via cont pte can achieve speedup of 10.5x on
some memory-intensive workloads on arm64 SBSA server.

[1] https://lore.kernel.org/all/c507308d-bdd4-5f9e-d4ff-e96e4520be85@nvidia.com/

>
>      Arnd
  

A gentle ping.

On Thu, Nov 23, 2023 at 2:57 PM Xu Lu <luxu.kernel@bytedance.com> wrote:
>
> Some existing architectures like ARM supports base page larger than 4K
> as their MMU supports more page sizes. Thus, besides hugetlb page and
> transparent huge page, there is another way for these architectures to
> enjoy the benefits of fewer TLB misses without worrying about cost of
> splitting and merging huge pages. However, on architectures with only
> 4K MMU, larger base page is unavailable now.
>
> This patch series attempts to break through the limitation of MMU and
> supports larger base page on RISC-V, which only supports 4K page size
> now.
>
> The key idea to implement larger base page based on 4K MMU is to
> decouple the MMU page from the base page in view of kernel mm, which we
> denote as software page. In contrary to software page, we denote the MMU
> page as hardware page. Below is the difference between these two kinds
> of pages.
>
> 1. Kernel memory management module manages, allocates and maps memory at
> a granularity of software page, which should not be restricted by
> MMU and can be larger than hardware page.
>
> 2. Architecture page table operations should be carried out from MMU's
> perspective and page table entries are encoded at a granularity of
> hardware page, which is 4K on RISC-V MMU now.
>
> The main work to decouple these two kinds of pages lies in architecture
> code. For example, we turn the pte_t struct to an array of page table
> entries to match it with software page which can be larger than hardware
> page, and adapt the page table operations accordingly. For 64K software
> base page, the pte_t struct now contains 16 contiguous page table
> entries which point to 16 contiguous 4K hardware pages.
>
> To achieve the benefits of large base page, we applies Svnapot for each
> base page's mapping. The Svnapot extension on RISC-V is like contiguous
> PTE on ARM64. It allows ptes of a naturally aligned power-of 2 size
> memory range be encoded in the same format to save the TLB space.
>
> This patch series is the first version and is based on v6.7-rc1. This
> version supports both bare metal and virtualization scenarios.
>
> In the next versions, we will continue on the following works:
>
> 1. Reduce the memory usage of page table page as it only uses 4K space
> while costs a whole base page.
>
> 2. When IMSIC interrupt file is smaller than 64K, extra isolation
> measures for the interrupt file are needed. (S)PMP and IOPMP may be good
> choices.
>
> 3. More consideration is needed to make this patch series collaborate
> with folios better.
>
> 4. Support 64K base page on IOMMU.
>
> 5. The performance test is on schedule to verify the actual performance
> improvement and the decrease in TLB miss rate.
>
> Thanks in advance for comments.
>
> Xu Lu (11):
>   mm: Fix misused APIs on huge pte
>   riscv: Introduce concept of hardware base page
>   riscv: Adapt pte struct to gap between hw page and sw page
>   riscv: Adapt pte operations to gap between hw page and sw page
>   riscv: Decouple pmd operations and pte operations
>   riscv: Distinguish pmd huge pte and napot huge pte
>   riscv: Adapt satp operations to gap between hw page and sw page
>   riscv: Apply Svnapot for base page mapping
>   riscv: Adjust fix_btmap slots number to match variable page size
>   riscv: kvm: Adapt kvm to gap between hw page and sw page
>   riscv: Introduce 64K page size
>
>  arch/Kconfig                        |   1 +
>  arch/riscv/Kconfig                  |  28 +++
>  arch/riscv/include/asm/fixmap.h     |   3 +-
>  arch/riscv/include/asm/hugetlb.h    |  71 ++++++-
>  arch/riscv/include/asm/page.h       |  16 +-
>  arch/riscv/include/asm/pgalloc.h    |  21 ++-
>  arch/riscv/include/asm/pgtable-32.h |   2 +-
>  arch/riscv/include/asm/pgtable-64.h |  45 +++--
>  arch/riscv/include/asm/pgtable.h    | 282 +++++++++++++++++++++++-----
>  arch/riscv/kernel/efi.c             |   2 +-
>  arch/riscv/kernel/head.S            |   4 +-
>  arch/riscv/kernel/hibernate.c       |   3 +-
>  arch/riscv/kvm/mmu.c                | 198 +++++++++++++------
>  arch/riscv/mm/context.c             |   7 +-
>  arch/riscv/mm/fault.c               |   1 +
>  arch/riscv/mm/hugetlbpage.c         |  42 +++--
>  arch/riscv/mm/init.c                |  25 +--
>  arch/riscv/mm/kasan_init.c          |   7 +-
>  arch/riscv/mm/pageattr.c            |   2 +-
>  fs/proc/task_mmu.c                  |   2 +-
>  include/asm-generic/hugetlb.h       |   7 +
>  include/asm-generic/pgtable-nopmd.h |   1 +
>  include/linux/pgtable.h             |   6 +
>  mm/hugetlb.c                        |   2 +-
>  mm/migrate.c                        |   5 +-
>  mm/mprotect.c                       |   2 +-
>  mm/rmap.c                           |  10 +-
>  mm/vmalloc.c                        |   3 +-
>  28 files changed, 616 insertions(+), 182 deletions(-)
>
> --
> 2.20.1
>