mm/thp: add CONFIG_TRANSPARENT_HUGEPAGE_NEVER option

  Add an option to disable transparent hugepages by default, in line with
the existing transparent_hugepage=never command line setting.

Rationale: khugepaged has its own non-negligible memory cost even if it
is not used by any applications, since it bumps up vm.min_free_kbytes to
its own required minimum in set_recommended_min_free_kbytes(). For
example, on a machine with 4GB RAM, with 3 mm zones and pageblock_order
== MAX_ORDER, starting khugepaged causes vm.min_free_kbytes increase
from 8MB to 132MB.

So if we use THP on machines with e.g. >=8GB of memory for better
performance, but avoid using it on lower-memory machines to avoid its
memory overhead, then for the same reason we also want to avoid even
starting khugepaged on those <8GB machines. So with
CONFIG_TRANSPARENT_HUGEPAGE_NEVER we can use the same kernel image on
both >=8GB and <8GB machines, with THP support enabled but khugepaged
not started by default. The userspace can then decide to enable THP
(i.e. start khugepaged) via sysfs if needed, based on the total amount
of memory.

This could also be achieved with the existing transparent_hugepage=never
setting in the kernel command line instead. But it seems cleaner to
avoid tweaking the command line for such a basic setting.

P.S. I see that CONFIG_TRANSPARENT_HUGEPAGE_NEVER was already proposed
in the past [1] but without an explanation of the purpose.

[1] https://lore.kernel.org/all/202211301651462590168@zte.com.cn/

Signed-off-by: Dmytro Maluka <dmaluka@chromium.org>
---
 mm/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)
  

On Mon,  4 Dec 2023 17:32:54 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:

> Add an option to disable transparent hugepages by default, in line with
> the existing transparent_hugepage=never command line setting.
> 
> Rationale: khugepaged has its own non-negligible memory cost even if it
> is not used by any applications, since it bumps up vm.min_free_kbytes to
> its own required minimum in set_recommended_min_free_kbytes(). For
> example, on a machine with 4GB RAM, with 3 mm zones and pageblock_order
> == MAX_ORDER, starting khugepaged causes vm.min_free_kbytes increase
> from 8MB to 132MB.
> 
> So if we use THP on machines with e.g. >=8GB of memory for better
> performance, but avoid using it on lower-memory machines to avoid its
> memory overhead, then for the same reason we also want to avoid even
> starting khugepaged on those <8GB machines. So with
> CONFIG_TRANSPARENT_HUGEPAGE_NEVER we can use the same kernel image on
> both >=8GB and <8GB machines, with THP support enabled but khugepaged
> not started by default. The userspace can then decide to enable THP
> (i.e. start khugepaged) via sysfs if needed, based on the total amount
> of memory.
> 
> This could also be achieved with the existing transparent_hugepage=never
> setting in the kernel command line instead. But it seems cleaner to
> avoid tweaking the command line for such a basic setting.
> 
> P.S. I see that CONFIG_TRANSPARENT_HUGEPAGE_NEVER was already proposed
> in the past [1] but without an explanation of the purpose.
> 
> ...
>
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -859,6 +859,12 @@ choice
>  	  madvise(MADV_HUGEPAGE) but it won't risk to increase the
>  	  memory footprint of applications without a guaranteed
>  	  benefit.
> +
> +	config TRANSPARENT_HUGEPAGE_NEVER
> +		bool "never"
> +	help
> +	  Disabling Transparent Hugepage by default. It can still be

s/Disabling/Disable/

> +	  enabled at runtime via sysfs.
>  endchoice

The patch adds the config option but doesn't use it?
  

On Mon, Dec 04, 2023 at 11:13:01AM -0800, Andrew Morton wrote:
> On Mon,  4 Dec 2023 17:32:54 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:
> 
> > Add an option to disable transparent hugepages by default, in line with
> > the existing transparent_hugepage=never command line setting.
> > 
> > Rationale: khugepaged has its own non-negligible memory cost even if it
> > is not used by any applications, since it bumps up vm.min_free_kbytes to
> > its own required minimum in set_recommended_min_free_kbytes(). For
> > example, on a machine with 4GB RAM, with 3 mm zones and pageblock_order
> > == MAX_ORDER, starting khugepaged causes vm.min_free_kbytes increase
> > from 8MB to 132MB.
> > 
> > So if we use THP on machines with e.g. >=8GB of memory for better
> > performance, but avoid using it on lower-memory machines to avoid its
> > memory overhead, then for the same reason we also want to avoid even
> > starting khugepaged on those <8GB machines. So with
> > CONFIG_TRANSPARENT_HUGEPAGE_NEVER we can use the same kernel image on
> > both >=8GB and <8GB machines, with THP support enabled but khugepaged
> > not started by default. The userspace can then decide to enable THP
> > (i.e. start khugepaged) via sysfs if needed, based on the total amount
> > of memory.
> > 
> > This could also be achieved with the existing transparent_hugepage=never
> > setting in the kernel command line instead. But it seems cleaner to
> > avoid tweaking the command line for such a basic setting.
> > 
> > P.S. I see that CONFIG_TRANSPARENT_HUGEPAGE_NEVER was already proposed
> > in the past [1] but without an explanation of the purpose.
> > 
> > ...
> >
> > --- a/mm/Kconfig
> > +++ b/mm/Kconfig
> > @@ -859,6 +859,12 @@ choice
> >  	  madvise(MADV_HUGEPAGE) but it won't risk to increase the
> >  	  memory footprint of applications without a guaranteed
> >  	  benefit.
> > +
> > +	config TRANSPARENT_HUGEPAGE_NEVER
> > +		bool "never"
> > +	help
> > +	  Disabling Transparent Hugepage by default. It can still be
> 
> s/Disabling/Disable/

It is in line with the descriptions of TRANSPARENT_HUGEPAGE_ALWAYS and
TRANSPARENT_HUGEPAGE_MADVISE: "Enabling Transparent Hugepage ..."

> > +	  enabled at runtime via sysfs.
> >  endchoice
> 
> The patch adds the config option but doesn't use it?

I should have been more precise: it is not a new option but a new choice
for CONFIG_TRANSPARENT_HUGEPAGE, in addition to the existing ALWAYS and
MADVISE choices. In mm/huge_memory.c in the declaration of the
transparent_hugepage_flags variable, if either ALWAYS or MADVISE is
chosen, transparent_hugepage_flags is initialized with such a value
that makes khugepaged being started by default during bootup. This patch
allows enabling CONFIG_TRANSPARENT_HUGEPAGE without setting either
ALWAYS or MADVISE, so that transparent_hugepage_flags is initialized
with such a value that khugepaged is not started by default.
  

On Mon, 4 Dec 2023 20:57:33 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:

> On Mon, Dec 04, 2023 at 11:13:01AM -0800, Andrew Morton wrote:
> > On Mon,  4 Dec 2023 17:32:54 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:
> > 
> > > Add an option to disable transparent hugepages by default, in line with
> > > the existing transparent_hugepage=never command line setting.
> > > 
> > > Rationale: khugepaged has its own non-negligible memory cost even if it
> > > is not used by any applications, since it bumps up vm.min_free_kbytes to
> > > its own required minimum in set_recommended_min_free_kbytes(). For
> > > example, on a machine with 4GB RAM, with 3 mm zones and pageblock_order
> > > == MAX_ORDER, starting khugepaged causes vm.min_free_kbytes increase
> > > from 8MB to 132MB.
> > > 
> > > So if we use THP on machines with e.g. >=8GB of memory for better
> > > performance, but avoid using it on lower-memory machines to avoid its
> > > memory overhead, then for the same reason we also want to avoid even
> > > starting khugepaged on those <8GB machines. So with
> > > CONFIG_TRANSPARENT_HUGEPAGE_NEVER we can use the same kernel image on
> > > both >=8GB and <8GB machines, with THP support enabled but khugepaged
> > > not started by default. The userspace can then decide to enable THP
> > > (i.e. start khugepaged) via sysfs if needed, based on the total amount
> > > of memory.
> > > 
> > > This could also be achieved with the existing transparent_hugepage=never
> > > setting in the kernel command line instead. But it seems cleaner to
> > > avoid tweaking the command line for such a basic setting.
> > > 
> > > P.S. I see that CONFIG_TRANSPARENT_HUGEPAGE_NEVER was already proposed
> > > in the past [1] but without an explanation of the purpose.
> > > 
> > > ...
> > >
> > > --- a/mm/Kconfig
> > > +++ b/mm/Kconfig
> > > @@ -859,6 +859,12 @@ choice
> > >  	  madvise(MADV_HUGEPAGE) but it won't risk to increase the
> > >  	  memory footprint of applications without a guaranteed
> > >  	  benefit.
> > > +
> > > +	config TRANSPARENT_HUGEPAGE_NEVER
> > > +		bool "never"
> > > +	help
> > > +	  Disabling Transparent Hugepage by default. It can still be
> > 
> > s/Disabling/Disable/
> 
> It is in line with the descriptions of TRANSPARENT_HUGEPAGE_ALWAYS and
> TRANSPARENT_HUGEPAGE_MADVISE: "Enabling Transparent Hugepage ..."

Those are incorrect also.

> > > +	  enabled at runtime via sysfs.
> > >  endchoice
> > 
> > The patch adds the config option but doesn't use it?
> 
> I should have been more precise: it is not a new option but a new choice
> for CONFIG_TRANSPARENT_HUGEPAGE, in addition to the existing ALWAYS and
> MADVISE choices. In mm/huge_memory.c in the declaration of the
> transparent_hugepage_flags variable, if either ALWAYS or MADVISE is
> chosen, transparent_hugepage_flags is initialized with such a value
> that makes khugepaged being started by default during bootup. This patch
> allows enabling CONFIG_TRANSPARENT_HUGEPAGE without setting either
> ALWAYS or MADVISE, so that transparent_hugepage_flags is initialized
> with such a value that khugepaged is not started by default.

OK, thanks.
  

On Mon, Dec 04, 2023 at 12:15:24PM -0800, Andrew Morton wrote:
> On Mon, 4 Dec 2023 20:57:33 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:
> 
> > On Mon, Dec 04, 2023 at 11:13:01AM -0800, Andrew Morton wrote:
> > > On Mon,  4 Dec 2023 17:32:54 +0100 Dmytro Maluka <dmaluka@chromium.org> wrote:
> > > 
> > > > Add an option to disable transparent hugepages by default, in line with
> > > > the existing transparent_hugepage=never command line setting.
> > > > 
> > > > Rationale: khugepaged has its own non-negligible memory cost even if it
> > > > is not used by any applications, since it bumps up vm.min_free_kbytes to
> > > > its own required minimum in set_recommended_min_free_kbytes(). For
> > > > example, on a machine with 4GB RAM, with 3 mm zones and pageblock_order
> > > > == MAX_ORDER, starting khugepaged causes vm.min_free_kbytes increase
> > > > from 8MB to 132MB.
> > > > 
> > > > So if we use THP on machines with e.g. >=8GB of memory for better
> > > > performance, but avoid using it on lower-memory machines to avoid its
> > > > memory overhead, then for the same reason we also want to avoid even
> > > > starting khugepaged on those <8GB machines. So with
> > > > CONFIG_TRANSPARENT_HUGEPAGE_NEVER we can use the same kernel image on
> > > > both >=8GB and <8GB machines, with THP support enabled but khugepaged
> > > > not started by default. The userspace can then decide to enable THP
> > > > (i.e. start khugepaged) via sysfs if needed, based on the total amount
> > > > of memory.
> > > > 
> > > > This could also be achieved with the existing transparent_hugepage=never
> > > > setting in the kernel command line instead. But it seems cleaner to
> > > > avoid tweaking the command line for such a basic setting.
> > > > 
> > > > P.S. I see that CONFIG_TRANSPARENT_HUGEPAGE_NEVER was already proposed
> > > > in the past [1] but without an explanation of the purpose.
> > > > 
> > > > ...
> > > >
> > > > --- a/mm/Kconfig
> > > > +++ b/mm/Kconfig
> > > > @@ -859,6 +859,12 @@ choice
> > > >  	  madvise(MADV_HUGEPAGE) but it won't risk to increase the
> > > >  	  memory footprint of applications without a guaranteed
> > > >  	  benefit.
> > > > +
> > > > +	config TRANSPARENT_HUGEPAGE_NEVER
> > > > +		bool "never"
> > > > +	help
> > > > +	  Disabling Transparent Hugepage by default. It can still be
> > > 
> > > s/Disabling/Disable/
> > 
> > It is in line with the descriptions of TRANSPARENT_HUGEPAGE_ALWAYS and
> > TRANSPARENT_HUGEPAGE_MADVISE: "Enabling Transparent Hugepage ..."
> 
> Those are incorrect also.

Ok, corrected in v2. Also clarified the changelog wrt your 2nd question.

> > > > +	  enabled at runtime via sysfs.
> > > >  endchoice
> > > 
> > > The patch adds the config option but doesn't use it?
> > 
> > I should have been more precise: it is not a new option but a new choice
> > for CONFIG_TRANSPARENT_HUGEPAGE, in addition to the existing ALWAYS and
> > MADVISE choices. In mm/huge_memory.c in the declaration of the
> > transparent_hugepage_flags variable, if either ALWAYS or MADVISE is
> > chosen, transparent_hugepage_flags is initialized with such a value
> > that makes khugepaged being started by default during bootup. This patch
> > allows enabling CONFIG_TRANSPARENT_HUGEPAGE without setting either
> > ALWAYS or MADVISE, so that transparent_hugepage_flags is initialized
> > with such a value that khugepaged is not started by default.
> 
> OK, thanks.