[RFC,00/26] mm: reliable huge page allocator

  As memory capacity continues to grow, 4k TLB coverage has not been
able to keep up. On Meta's 64G webservers, close to 20% of execution
cycles are observed to be handling TLB misses when using 4k pages
only. Huge pages are shifting from being a nice-to-have optimization
for HPC workloads to becoming a necessity for common applications.

However, while trying to deploy THP more universally, we observe a
fragmentation problem in the page allocator that often prevents larger
requests from being met quickly, or met at all, at runtime. Since we
have to provision hardware capacity for worst case performance,
unreliable huge page coverage isn't of much help.

Drilling into the allocator, we find that existing defrag efforts,
such as mobility grouping and watermark boosting, help, but are
insufficient by themselves. We still observe a high number of blocks
being routinely shared by allocations of different migratetypes. This
in turn results in inefficient or ineffective reclaim/compaction runs.

In a broad sample of Meta servers, we find that unmovable allocations
make up less than 7% of total memory on average, yet occupy 34% of the
2M blocks in the system. We also found that this effect isn't
correlated with high uptimes, and that servers can get heavily
fragmented within the first hour of running a workload.

The following experiment shows that only 20min of build load under
moderate memory pressure already results in a significant number of
typemixed blocks (block analysis run after system is back to idle):

vanilla:
unmovable 50
movable 701
reclaimable 149
unmovable blocks with slab/lru pages: 13 ({'slab': 17, 'lru': 19} pages)
movable blocks with non-LRU pages: 77 ({'slab': 4257, 'kmem': 77, 'other': 2} pages)
reclaimable blocks with non-slab pages: 16 ({'lru': 37, 'kmem': 311, 'other': 26} pages)

patched:
unmovable 65
movable 457
reclaimable 159
free 219
unmovable blocks with slab/lru pages: 22 ({'slab': 0, 'lru': 38} pages)
movable blocks with non-LRU pages: 0 ({'slab': 0, 'kmem': 0, 'other': 0} pages)
reclaimable blocks with non-slab pages: 3 ({'lru': 36, 'kmem': 0, 'other': 23} pages)

[ The remaining "mixed blocks" in the patched kernel are false
  positives: LRU pages without migrate callbacks (empty_aops), and
  i915 shmem that is pinned until reclaimed through shrinkers. ]

Root causes

One of the behaviors that sabotage the page allocator's mobility
grouping is the fact that requests of one migratetype are allowed to
fall back into blocks of another type before reclaim and compaction
occur. This is a design decision to prioritize memory utilization over
block fragmentation - especially considering the history of lumpy
reclaim and its tendency to overreclaim. However, with compaction
available, these two goals are no longer in conflict: the scratch
space of free pages for compaction to work is only twice the size of
the allocation request; in most cases, only small amounts of
proactive, coordinated reclaim and compaction is required to prevent a
fallback which may fragment a pageblock indefinitely.

Another problem lies in how the page allocator drives reclaim and
compaction when it does invoke it. While the page allocator targets
migratetype grouping at the pageblock level, it calls reclaim and
compaction with the order of the allocation request. As most requests
are smaller than a pageblock, this results in partial block freeing
and subsequent fallbacks and type mixing.

Note that in combination, these two design decisions have a
self-reinforcing effect on fragmentation: 1. Partially used unmovable
blocks are filled up with fallback movable pages. 2. A subsequent
unmovable allocation, instead of grouping up, will then need to enter
reclaim, which most likely results in a partially freed movable block
that it falls back into. Over time, unmovable allocations are sparsely
scattered throughout the address space and poison many pageblocks.

Note that block fragmentation is driven by lower-order requests. It is
not reliably mitigated by the mere presence of higher-order requests.

Proposal

This series proposes to make THP allocations reliable by enforcing
pageblock hygiene, and aligning the allocator, reclaim and compaction
on the pageblock as the base unit for managing free memory. All orders
up to and including the pageblock are made first-class requests that
(outside of OOM situations) are expected to succeed without
exceptional investment by the allocating thread.

A neutral pageblock type is introduced, MIGRATE_FREE. The first
allocation to be placed into such a block claims it exclusively for
the allocation's migratetype. Fallbacks from a different type are no
longer allowed, and the block is "kept open" for more allocations of
the same type to ensure tight grouping. A pageblock becomes neutral
again only once all its pages have been freed.

Reclaim and compaction are changed from partial block reclaim to
producing whole neutral page blocks. The watermark logic is adjusted
to apply to neutral blocks, ensuring that background and direct
reclaim always maintain a readily-available reserve of them.

The defragmentation effort changes from reactive to proactive. In
turn, this makes defragmentation actually more efficient: compaction
only has to scan movable blocks and can skip other blocks entirely;
since movable blocks aren't poisoned by unmovable pages, the chances
of successful compaction in each block are greatly improved as well.

Defragmentation becomes an ongoing responsibility of all allocations,
rather than being the burden of only higher-order asks. This prevents
sub-block allocations - which cause block fragmentation in the first
place - from starving the increasingly important larger requests.

There is a slight increase in worst-case memory overhead by requiring
the watermarks to be met against neutral blocks even when there might
be free pages in typed blocks. However, the high watermarks are less
than 1% of the zone, so the increase is relatively small.

These changes only apply to CONFIG_COMPACTION kernels. Without
compaction, fallbacks and partial block reclaim remain the best
trade-off between memory utilization and fragmentation.

Initial Test Results

The following is purely an allocation reliability test. Achieving full
THP benefits in practice is tied to other pending changes, such as the
THP shrinker to avoid memory pressure from excessive internal
fragmentation, and tweaks to the kernel's THP allocation strategy.

The test is a kernel build under moderate-to-high memory pressure,
with a concurrent process trying to repeatedly fault THPs (madvise):

                                              HUGEALLOC-VANILLA       HUGEALLOC-PATCHED
Real time                                   265.04 (    +0.00%)     268.12 (    +1.16%)
User time                                  1131.05 (    +0.00%)    1131.13 (    +0.01%)
System time                                 474.66 (    +0.00%)     478.97 (    +0.91%)
THP fault alloc                           17913.24 (    +0.00%)   19647.50 (    +9.68%)
THP fault fallback                         1947.12 (    +0.00%)     223.40 (   -88.48%)
THP fault fail rate %                         9.80 (    +0.00%)       1.12 (   -80.34%)
Direct compact stall                        282.44 (    +0.00%)     543.90 (   +92.25%)
Direct compact fail                         262.44 (    +0.00%)     239.90 (    -8.56%)
Direct compact success                       20.00 (    +0.00%)     304.00 ( +1352.38%)
Direct compact success rate %                 7.15 (    +0.00%)      57.10 (  +612.90%)
Compact daemon scanned migrate            21643.80 (    +0.00%)  387479.80 ( +1690.18%)
Compact daemon scanned free              188462.36 (    +0.00%) 2842824.10 ( +1408.42%)
Compact direct scanned migrate          1601294.84 (    +0.00%)  275670.70 (   -82.78%)
Compact direct scanned free             4476155.60 (    +0.00%) 2438835.00 (   -45.51%)
Compact migrate scanned daemon %              1.32 (    +0.00%)      59.18 ( +2499.00%)
Compact free scanned daemon %                 3.95 (    +0.00%)      54.31 ( +1018.20%)
Alloc stall                                2425.00 (    +0.00%)     992.00 (   -59.07%)
Pages kswapd scanned                     586756.68 (    +0.00%)  975390.20 (   +66.23%)
Pages kswapd reclaimed                   385468.20 (    +0.00%)  437767.50 (   +13.57%)
Pages direct scanned                     335199.56 (    +0.00%)  501824.20 (   +49.71%)
Pages direct reclaimed                   127953.72 (    +0.00%)  151880.70 (   +18.70%)
Pages scanned kswapd %                       64.43 (    +0.00%)      66.39 (    +2.99%)
Swap out                                  14083.88 (    +0.00%)   45034.60 (  +219.74%)
Swap in                                    3395.08 (    +0.00%)    7767.50 (  +128.75%)
File refaults                             93546.68 (    +0.00%)  129648.30 (   +38.59%)

The THP fault success rate is drastically improved. A bigger share of
the work is done by the background threads, as they now proactively
maintain MIGRATE_FREE block reserves. The increase in memory pressure
is shown by the uptick in swap activity.

Status

Initial test results look promising, but production testing has been
lagging behind the effort to generalize this code for upstream, and
putting all the pieces together to make THP work. I'll follow up as I
gather more data.

Sending this out now as an RFC to get input on the overall direction.

The patches are based on v6.2.

 Documentation/admin-guide/sysctl/vm.rst |  21 -
 block/bdev.c                            |   2 +-
 include/linux/compaction.h              | 100 +---
 include/linux/gfp.h                     |   2 -
 include/linux/mm.h                      |   1 -
 include/linux/mmzone.h                  |  30 +-
 include/linux/page-isolation.h          |  28 +-
 include/linux/pageblock-flags.h         |   4 +-
 include/linux/vmstat.h                  |   8 -
 include/trace/events/mmflags.h          |   4 +-
 kernel/sysctl.c                         |   8 -
 mm/compaction.c                         | 242 +++-----
 mm/internal.h                           |  14 +-
 mm/memory_hotplug.c                     |   4 +-
 mm/page_alloc.c                         | 930 +++++++++++++-----------------
 mm/page_isolation.c                     |  42 +-
 mm/vmscan.c                             | 251 ++------
 mm/vmstat.c                             |   6 +-
 18 files changed, 629 insertions(+), 1068 deletions(-)
  

On Tue, Apr 18, 2023 at 03:12:47PM -0400, Johannes Weiner wrote:
> As memory capacity continues to grow, 4k TLB coverage has not been
> able to keep up. On Meta's 64G webservers, close to 20% of execution
> cycles are observed to be handling TLB misses when using 4k pages
> only. Huge pages are shifting from being a nice-to-have optimization
> for HPC workloads to becoming a necessity for common applications.
> 
> However, while trying to deploy THP more universally, we observe a
> fragmentation problem in the page allocator that often prevents larger
> requests from being met quickly, or met at all, at runtime. Since we
> have to provision hardware capacity for worst case performance,
> unreliable huge page coverage isn't of much help.
> 
> Drilling into the allocator, we find that existing defrag efforts,
> such as mobility grouping and watermark boosting, help, but are
> insufficient by themselves. We still observe a high number of blocks
> being routinely shared by allocations of different migratetypes. This
> in turn results in inefficient or ineffective reclaim/compaction runs.
> 
> In a broad sample of Meta servers, we find that unmovable allocations
> make up less than 7% of total memory on average, yet occupy 34% of the
> 2M blocks in the system. We also found that this effect isn't
> correlated with high uptimes, and that servers can get heavily
> fragmented within the first hour of running a workload.
> 
> The following experiment shows that only 20min of build load under
> moderate memory pressure already results in a significant number of
> typemixed blocks (block analysis run after system is back to idle):
> 
> vanilla:
> unmovable 50
> movable 701
> reclaimable 149
> unmovable blocks with slab/lru pages: 13 ({'slab': 17, 'lru': 19} pages)
> movable blocks with non-LRU pages: 77 ({'slab': 4257, 'kmem': 77, 'other': 2} pages)
> reclaimable blocks with non-slab pages: 16 ({'lru': 37, 'kmem': 311, 'other': 26} pages)
> 
> patched:
> unmovable 65
> movable 457
> reclaimable 159
> free 219
> unmovable blocks with slab/lru pages: 22 ({'slab': 0, 'lru': 38} pages)
> movable blocks with non-LRU pages: 0 ({'slab': 0, 'kmem': 0, 'other': 0} pages)
> reclaimable blocks with non-slab pages: 3 ({'lru': 36, 'kmem': 0, 'other': 23} pages)
> 
> [ The remaining "mixed blocks" in the patched kernel are false
>   positives: LRU pages without migrate callbacks (empty_aops), and
>   i915 shmem that is pinned until reclaimed through shrinkers. ]
> 
> Root causes
> 
> One of the behaviors that sabotage the page allocator's mobility
> grouping is the fact that requests of one migratetype are allowed to
> fall back into blocks of another type before reclaim and compaction
> occur. This is a design decision to prioritize memory utilization over
> block fragmentation - especially considering the history of lumpy
> reclaim and its tendency to overreclaim. However, with compaction
> available, these two goals are no longer in conflict: the scratch
> space of free pages for compaction to work is only twice the size of
> the allocation request; in most cases, only small amounts of
> proactive, coordinated reclaim and compaction is required to prevent a
> fallback which may fragment a pageblock indefinitely.
> 
> Another problem lies in how the page allocator drives reclaim and
> compaction when it does invoke it. While the page allocator targets
> migratetype grouping at the pageblock level, it calls reclaim and
> compaction with the order of the allocation request. As most requests
> are smaller than a pageblock, this results in partial block freeing
> and subsequent fallbacks and type mixing.
> 
> Note that in combination, these two design decisions have a
> self-reinforcing effect on fragmentation: 1. Partially used unmovable
> blocks are filled up with fallback movable pages. 2. A subsequent
> unmovable allocation, instead of grouping up, will then need to enter
> reclaim, which most likely results in a partially freed movable block
> that it falls back into. Over time, unmovable allocations are sparsely
> scattered throughout the address space and poison many pageblocks.
> 
> Note that block fragmentation is driven by lower-order requests. It is
> not reliably mitigated by the mere presence of higher-order requests.
> 
> Proposal
> 
> This series proposes to make THP allocations reliable by enforcing
> pageblock hygiene, and aligning the allocator, reclaim and compaction
> on the pageblock as the base unit for managing free memory. All orders
> up to and including the pageblock are made first-class requests that
> (outside of OOM situations) are expected to succeed without
> exceptional investment by the allocating thread.
> 
> A neutral pageblock type is introduced, MIGRATE_FREE. The first
> allocation to be placed into such a block claims it exclusively for
> the allocation's migratetype. Fallbacks from a different type are no
> longer allowed, and the block is "kept open" for more allocations of
> the same type to ensure tight grouping. A pageblock becomes neutral
> again only once all its pages have been freed.

Sounds like this will cause earlier OOM, no?

I guess with 2M pageblock on 64G server it shouldn't matter much. But how
about smaller machines?

> Reclaim and compaction are changed from partial block reclaim to
> producing whole neutral page blocks.

How does it affect allocation latencies? I see direct compact stall grew
substantially. Hm?

> The watermark logic is adjusted
> to apply to neutral blocks, ensuring that background and direct
> reclaim always maintain a readily-available reserve of them.
> 
> The defragmentation effort changes from reactive to proactive. In
> turn, this makes defragmentation actually more efficient: compaction
> only has to scan movable blocks and can skip other blocks entirely;
> since movable blocks aren't poisoned by unmovable pages, the chances
> of successful compaction in each block are greatly improved as well.
> 
> Defragmentation becomes an ongoing responsibility of all allocations,
> rather than being the burden of only higher-order asks. This prevents
> sub-block allocations - which cause block fragmentation in the first
> place - from starving the increasingly important larger requests.
> 
> There is a slight increase in worst-case memory overhead by requiring
> the watermarks to be met against neutral blocks even when there might
> be free pages in typed blocks. However, the high watermarks are less
> than 1% of the zone, so the increase is relatively small.
> 
> These changes only apply to CONFIG_COMPACTION kernels. Without
> compaction, fallbacks and partial block reclaim remain the best
> trade-off between memory utilization and fragmentation.
> 
> Initial Test Results
> 
> The following is purely an allocation reliability test. Achieving full
> THP benefits in practice is tied to other pending changes, such as the
> THP shrinker to avoid memory pressure from excessive internal
> fragmentation, and tweaks to the kernel's THP allocation strategy.
> 
> The test is a kernel build under moderate-to-high memory pressure,
> with a concurrent process trying to repeatedly fault THPs (madvise):
> 
>                                               HUGEALLOC-VANILLA       HUGEALLOC-PATCHED
> Real time                                   265.04 (    +0.00%)     268.12 (    +1.16%)
> User time                                  1131.05 (    +0.00%)    1131.13 (    +0.01%)
> System time                                 474.66 (    +0.00%)     478.97 (    +0.91%)
> THP fault alloc                           17913.24 (    +0.00%)   19647.50 (    +9.68%)
> THP fault fallback                         1947.12 (    +0.00%)     223.40 (   -88.48%)
> THP fault fail rate %                         9.80 (    +0.00%)       1.12 (   -80.34%)
> Direct compact stall                        282.44 (    +0.00%)     543.90 (   +92.25%)
> Direct compact fail                         262.44 (    +0.00%)     239.90 (    -8.56%)
> Direct compact success                       20.00 (    +0.00%)     304.00 ( +1352.38%)
> Direct compact success rate %                 7.15 (    +0.00%)      57.10 (  +612.90%)
> Compact daemon scanned migrate            21643.80 (    +0.00%)  387479.80 ( +1690.18%)
> Compact daemon scanned free              188462.36 (    +0.00%) 2842824.10 ( +1408.42%)
> Compact direct scanned migrate          1601294.84 (    +0.00%)  275670.70 (   -82.78%)
> Compact direct scanned free             4476155.60 (    +0.00%) 2438835.00 (   -45.51%)
> Compact migrate scanned daemon %              1.32 (    +0.00%)      59.18 ( +2499.00%)
> Compact free scanned daemon %                 3.95 (    +0.00%)      54.31 ( +1018.20%)
> Alloc stall                                2425.00 (    +0.00%)     992.00 (   -59.07%)
> Pages kswapd scanned                     586756.68 (    +0.00%)  975390.20 (   +66.23%)
> Pages kswapd reclaimed                   385468.20 (    +0.00%)  437767.50 (   +13.57%)
> Pages direct scanned                     335199.56 (    +0.00%)  501824.20 (   +49.71%)
> Pages direct reclaimed                   127953.72 (    +0.00%)  151880.70 (   +18.70%)
> Pages scanned kswapd %                       64.43 (    +0.00%)      66.39 (    +2.99%)
> Swap out                                  14083.88 (    +0.00%)   45034.60 (  +219.74%)
> Swap in                                    3395.08 (    +0.00%)    7767.50 (  +128.75%)
> File refaults                             93546.68 (    +0.00%)  129648.30 (   +38.59%)
> 
> The THP fault success rate is drastically improved. A bigger share of
> the work is done by the background threads, as they now proactively
> maintain MIGRATE_FREE block reserves. The increase in memory pressure
> is shown by the uptick in swap activity.
> 
> Status
> 
> Initial test results look promising, but production testing has been
> lagging behind the effort to generalize this code for upstream, and
> putting all the pieces together to make THP work. I'll follow up as I
> gather more data.
> 
> Sending this out now as an RFC to get input on the overall direction.
> 
> The patches are based on v6.2.
> 
>  Documentation/admin-guide/sysctl/vm.rst |  21 -
>  block/bdev.c                            |   2 +-
>  include/linux/compaction.h              | 100 +---
>  include/linux/gfp.h                     |   2 -
>  include/linux/mm.h                      |   1 -
>  include/linux/mmzone.h                  |  30 +-
>  include/linux/page-isolation.h          |  28 +-
>  include/linux/pageblock-flags.h         |   4 +-
>  include/linux/vmstat.h                  |   8 -
>  include/trace/events/mmflags.h          |   4 +-
>  kernel/sysctl.c                         |   8 -
>  mm/compaction.c                         | 242 +++-----
>  mm/internal.h                           |  14 +-
>  mm/memory_hotplug.c                     |   4 +-
>  mm/page_alloc.c                         | 930 +++++++++++++-----------------
>  mm/page_isolation.c                     |  42 +-
>  mm/vmscan.c                             | 251 ++------
>  mm/vmstat.c                             |   6 +-
>  18 files changed, 629 insertions(+), 1068 deletions(-)
> 
>
  

Hi Kirill, thanks for taking a look so quickly.

On Wed, Apr 19, 2023 at 02:54:02AM +0300, Kirill A. Shutemov wrote:
> On Tue, Apr 18, 2023 at 03:12:47PM -0400, Johannes Weiner wrote:
> > This series proposes to make THP allocations reliable by enforcing
> > pageblock hygiene, and aligning the allocator, reclaim and compaction
> > on the pageblock as the base unit for managing free memory. All orders
> > up to and including the pageblock are made first-class requests that
> > (outside of OOM situations) are expected to succeed without
> > exceptional investment by the allocating thread.
> > 
> > A neutral pageblock type is introduced, MIGRATE_FREE. The first
> > allocation to be placed into such a block claims it exclusively for
> > the allocation's migratetype. Fallbacks from a different type are no
> > longer allowed, and the block is "kept open" for more allocations of
> > the same type to ensure tight grouping. A pageblock becomes neutral
> > again only once all its pages have been freed.
> 
> Sounds like this will cause earlier OOM, no?
> 
> I guess with 2M pageblock on 64G server it shouldn't matter much. But how
> about smaller machines?

Yes, it's a tradeoff.

It's not really possible to reduce external fragmentation and increase
contiguity, without also increasing the risk of internal fragmentation
to some extent. The tradeoff is slighly less but overall faster memory.

A 2M block size *seems* reasonable for most current setups. It's
actually still somewhat on the lower side, if you consider that we had
4k blocks when memory was a few megabytes. (4k pages for 4M RAM is the
same ratio as 2M pages for 2G RAM. My phone has 8G and my desktop 32G.
64G is unusually small for a datacenter server.)

I wouldn't be opposed to sticking this behind a separate config option
if there are setups that WOULD want to keep the current best-effort
compaction without the block hygiene. But obviously, from a
maintenance POV life would be much easier if we didn't have to.

FWIF, I have been doing tests in an environment constrained to 2G and
haven't had any issues with premature OOMs. But I'm happy to test
other situations and workloads that might be of interest to people.

> > Reclaim and compaction are changed from partial block reclaim to
> > producing whole neutral page blocks.
> 
> How does it affect allocation latencies? I see direct compact stall grew
> substantially. Hm?

Good question.

There are 260 more compact stalls but also 1,734 more successful THP
allocations. And 1,433 fewer allocation stalls. There seems to be much
less direct work performed per successful allocation.

But of course, that's not the whole story. Let me trace the actual
latencies.

Thanks for your thoughts!
Johannes
  

On Tue, Apr 18, 2023 at 03:12:47PM -0400, Johannes Weiner wrote:
> This series proposes to make THP allocations reliable by enforcing
> pageblock hygiene, and aligning the allocator, reclaim and compaction
> on the pageblock as the base unit for managing free memory. All orders
> up to and including the pageblock are made first-class requests that
> (outside of OOM situations) are expected to succeed without
> exceptional investment by the allocating thread.
> 
> A neutral pageblock type is introduced, MIGRATE_FREE. The first
> allocation to be placed into such a block claims it exclusively for
> the allocation's migratetype. Fallbacks from a different type are no
> longer allowed, and the block is "kept open" for more allocations of
> the same type to ensure tight grouping. A pageblock becomes neutral
> again only once all its pages have been freed.

YES!  This is exactly what I've been thinking is the right solution
for some time.  Thank you for doing it.
  

On 4/19/23 04:08, Johannes Weiner wrote:
> Hi Kirill, thanks for taking a look so quickly.
> 
> On Wed, Apr 19, 2023 at 02:54:02AM +0300, Kirill A. Shutemov wrote:
>> On Tue, Apr 18, 2023 at 03:12:47PM -0400, Johannes Weiner wrote:
>> > This series proposes to make THP allocations reliable by enforcing
>> > pageblock hygiene, and aligning the allocator, reclaim and compaction
>> > on the pageblock as the base unit for managing free memory. All orders
>> > up to and including the pageblock are made first-class requests that
>> > (outside of OOM situations) are expected to succeed without
>> > exceptional investment by the allocating thread.
>> > 
>> > A neutral pageblock type is introduced, MIGRATE_FREE. The first
>> > allocation to be placed into such a block claims it exclusively for
>> > the allocation's migratetype. Fallbacks from a different type are no
>> > longer allowed, and the block is "kept open" for more allocations of
>> > the same type to ensure tight grouping. A pageblock becomes neutral
>> > again only once all its pages have been freed.
>> 
>> Sounds like this will cause earlier OOM, no?
>> 
>> I guess with 2M pageblock on 64G server it shouldn't matter much. But how
>> about smaller machines?
> 
> Yes, it's a tradeoff.
> 
> It's not really possible to reduce external fragmentation and increase
> contiguity, without also increasing the risk of internal fragmentation
> to some extent. The tradeoff is slighly less but overall faster memory.
> 
> A 2M block size *seems* reasonable for most current setups. It's
> actually still somewhat on the lower side, if you consider that we had
> 4k blocks when memory was a few megabytes. (4k pages for 4M RAM is the
> same ratio as 2M pages for 2G RAM. My phone has 8G and my desktop 32G.
> 64G is unusually small for a datacenter server.)
> 
> I wouldn't be opposed to sticking this behind a separate config option
> if there are setups that WOULD want to keep the current best-effort
> compaction without the block hygiene. But obviously, from a
> maintenance POV life would be much easier if we didn't have to.

As much as tunables are frowned upon in general, this could make sense to me
 even as a runtime tunable (maybe with defaults based on how large the
system is), because a datacenter server and a phone is after all not the
same thing. But of course it would be preferrable to find out it works
reasonably well even for the smaller systems. For example we already do
completely disable mobility grouping if there's too little RAM for it to
make sense, which is somewhat similar (but not completely identical) decision.

> FWIF, I have been doing tests in an environment constrained to 2G and
> haven't had any issues with premature OOMs. But I'm happy to test
> other situations and workloads that might be of interest to people.
> 
>> > Reclaim and compaction are changed from partial block reclaim to
>> > producing whole neutral page blocks.
>> 
>> How does it affect allocation latencies? I see direct compact stall grew
>> substantially. Hm?
> 
> Good question.
> 
> There are 260 more compact stalls but also 1,734 more successful THP
> allocations. And 1,433 fewer allocation stalls. There seems to be much
> less direct work performed per successful allocation.

Yeah if there's a workload that uses THP madvise to indicate it prefers the
compaction stalls to base page fallbacks, and compaction is more sucessful,
it won't defer further attempts so as a result there will be more stalls.
What we should watch out for are rather latencies of allocations that don't
prefer the stalls, but might now be forced to clean up new MIGRATE_FREE
pageblocks for their order-0 allocation that would previously just fallback,
etc.

> But of course, that's not the whole story. Let me trace the actual
> latencies.
> 
> Thanks for your thoughts!
> Johannes
  

On Wed, Apr 19, 2023 at 05:11:45AM +0100, Matthew Wilcox wrote:
> On Tue, Apr 18, 2023 at 03:12:47PM -0400, Johannes Weiner wrote:
> > This series proposes to make THP allocations reliable by enforcing
> > pageblock hygiene, and aligning the allocator, reclaim and compaction
> > on the pageblock as the base unit for managing free memory. All orders
> > up to and including the pageblock are made first-class requests that
> > (outside of OOM situations) are expected to succeed without
> > exceptional investment by the allocating thread.
> > 
> > A neutral pageblock type is introduced, MIGRATE_FREE. The first
> > allocation to be placed into such a block claims it exclusively for
> > the allocation's migratetype. Fallbacks from a different type are no
> > longer allowed, and the block is "kept open" for more allocations of
> > the same type to ensure tight grouping. A pageblock becomes neutral
> > again only once all its pages have been freed.
> 
> YES!  This is exactly what I've been thinking is the right solution
> for some time.  Thank you for doing it.
> 

It was considered once upon a time and comes up every so often as variants
of a "sticky" pageblock pageblock bit that prevents mixing. The risks was
ending up in a context where memory within a suitable pageblock cannot
be freed and all of the available MOVABLE pageblocks have at least one
pinned page that cannot migrate from the allocating context. It can also
potentially hit a case where the majority of memory is UNMOVABLE pageblocks,
each of which has a single pagetable page that cannot be freed without an
OOM kill. Variants of issues like this would manifestas an OOM kill with
plenty of memory free bug or excessive CPu usage on reclaim or compaction.

It doesn't kill the idea of the series at all but it puts a lot of emphasis
in splitting the series by low-risk and high-risk. Maybe to the extent where
the absolute protection against mixing can be broken in OOM situations,
kernel command line or sysctl.
  

On Fri, Apr 21, 2023 at 05:11:56PM +0100, Mel Gorman wrote:
> It was considered once upon a time and comes up every so often as variants
> of a "sticky" pageblock pageblock bit that prevents mixing. The risks was
> ending up in a context where memory within a suitable pageblock cannot
> be freed and all of the available MOVABLE pageblocks have at least one
> pinned page that cannot migrate from the allocating context. It can also
> potentially hit a case where the majority of memory is UNMOVABLE pageblocks,
> each of which has a single pagetable page that cannot be freed without an
> OOM kill. Variants of issues like this would manifestas an OOM kill with
> plenty of memory free bug or excessive CPu usage on reclaim or compaction.
> 
> It doesn't kill the idea of the series at all but it puts a lot of emphasis
> in splitting the series by low-risk and high-risk. Maybe to the extent where
> the absolute protection against mixing can be broken in OOM situations,
> kernel command line or sysctl.

Has a variant been previously considered where MOVABLE allocations are
allowed to come from UNMOVABLE blocks?  After all, MOVABLE allocations
are generally, well, movable.  So an UNMOVABLE allocation could try to
migrate pages from a MIXED pageblock in order to turn the MIXED pageblock
back into an UNMOVABLE pageblock.

This might work better in practice because GFP_NOFS allocations tend
to also be MOVABLE, so allowing them to take up some of the UNMOVABLE
space temporarily feels like a get-out-of-OOM card.

(I've resisted talking about plans to make page table pages movable
because I don't think that's your point; that's just an example of a
currently-unmovable allocation, right?)

I mention this in part because on my laptop, ZONE_DMA is almost unused:

Node 0, zone      DMA      0      0      0      0      0      0      0      0      1      2      2
Node 0, zone    DMA32   1685   1345   1152    554    424    212    104     40      2      0      0
Node 0, zone   Normal   6959   3530   1893   1862    629    483    107     10      0      0      0

That's 2 order-10 (=8MB), 2 order-9 (=4MB) and 1 order8 (=1MB) for a
total of 13MB of memory.  That's insignificant to a 16GB laptop, but on
smaller machines, it might be worth allowing MOVABLE allocations to come
from ZONE_DMA on the grounds that they can be easily freed if anybody
ever allocated from ZONE_DMA.
  

On 21.04.23 19:14, Matthew Wilcox wrote:
> On Fri, Apr 21, 2023 at 05:11:56PM +0100, Mel Gorman wrote:
>> It was considered once upon a time and comes up every so often as variants
>> of a "sticky" pageblock pageblock bit that prevents mixing. The risks was
>> ending up in a context where memory within a suitable pageblock cannot
>> be freed and all of the available MOVABLE pageblocks have at least one
>> pinned page that cannot migrate from the allocating context. It can also
>> potentially hit a case where the majority of memory is UNMOVABLE pageblocks,
>> each of which has a single pagetable page that cannot be freed without an
>> OOM kill. Variants of issues like this would manifestas an OOM kill with
>> plenty of memory free bug or excessive CPu usage on reclaim or compaction.
>>
>> It doesn't kill the idea of the series at all but it puts a lot of emphasis
>> in splitting the series by low-risk and high-risk. Maybe to the extent where
>> the absolute protection against mixing can be broken in OOM situations,
>> kernel command line or sysctl.
> 
> Has a variant been previously considered where MOVABLE allocations are
> allowed to come from UNMOVABLE blocks?  After all, MOVABLE allocations
> are generally, well, movable.  So an UNMOVABLE allocation could try to
> migrate pages from a MIXED pageblock in order to turn the MIXED pageblock
> back into an UNMOVABLE pageblock.

I might be completely off, but my understanding was that movable 
allocations can be happily placed into unmovable blocks if required already?

IIRC, it's primarily the zone fallback rules that prevent e.g., ZONE_DMA 
to get filled immediately with movable data in your example. I might eb 
wrong, though.

I guess what you mean is serving movable allocations much earlier from 
these other zones.

Having memory hotunplug in mind ( as always ;) ), I'd expect that such 
fragmentation must be allowed to happen to guarantee that memory (esp. 
ZONE_MOVABLE) can be properly evacuated even if there are not sufficient 
MOVABLE pageblocks around to hold all that (movable) data.