[v2,2/2] mm: memcg: introduce new event to trace shrink_memcg

  The shrink_memcg flow plays a crucial role in memcg reclamation.
Currently, it is not possible to trace this point from non-direct
reclaim paths. However, direct reclaim has its own tracepoint, so there
is no issue there. In certain cases, when debugging memcg pressure,
developers may need to identify all potential requests for memcg
reclamation including kswapd(). The patchset introduces the tracepoints
mm_vmscan_memcg_shrink_{begin|end}() to address this problem.

Example of output in the kswapd context (non-direct reclaim):
    kswapd0-39      [001] .....   240.356378: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.356396: mm_vmscan_memcg_shrink_end: nr_reclaimed=0 memcg=test
    kswapd0-39      [001] .....   240.356420: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.356454: mm_vmscan_memcg_shrink_end: nr_reclaimed=1 memcg=test
    kswapd0-39      [001] .....   240.356479: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.356506: mm_vmscan_memcg_shrink_end: nr_reclaimed=4 memcg=test
    kswapd0-39      [001] .....   240.356525: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.356593: mm_vmscan_memcg_shrink_end: nr_reclaimed=11 memcg=test
    kswapd0-39      [001] .....   240.356614: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.356738: mm_vmscan_memcg_shrink_end: nr_reclaimed=25 memcg=test
    kswapd0-39      [001] .....   240.356790: mm_vmscan_memcg_shrink_begin: order=0 gfp_flags=GFP_KERNEL memcg=test
    kswapd0-39      [001] .....   240.357125: mm_vmscan_memcg_shrink_end: nr_reclaimed=53 memcg=test

Signed-off-by: Dmitry Rokosov <ddrokosov@salutedevices.com>
---
 include/trace/events/vmscan.h | 14 ++++++++++++++
 mm/vmscan.c                   | 11 +++++++++++
 2 files changed, 25 insertions(+)
  

On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote:
> The shrink_memcg flow plays a crucial role in memcg reclamation.
> Currently, it is not possible to trace this point from non-direct
> reclaim paths.

Is this really true? AFAICS we have
mm_vmscan_lru_isolate
mm_vmscan_lru_shrink_active
mm_vmscan_lru_shrink_inactive

which are in the vry core of the memory reclaim. Sure post processing
those is some work.

[...]
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 45780952f4b5..6d89b39d9a91 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -6461,6 +6461,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
>  		 */
>  		cond_resched();
>  
> +#ifdef CONFIG_MEMCG
> +		trace_mm_vmscan_memcg_shrink_begin(sc->order,
> +						   sc->gfp_mask,
> +						   memcg);
> +#endif

this is a common code path for node and direct reclaim which means that
we will have multiple begin/end tracepoints covering similar operations.
To me that sounds excessive. If you are missing a cumulative kswapd
alternative to 
mm_vmscan_direct_reclaim_begin
mm_vmscan_direct_reclaim_end
mm_vmscan_memcg_reclaim_begin
mm_vmscan_memcg_reclaim_end
mm_vmscan_memcg_softlimit_reclaim_begin
mm_vmscan_memcg_softlimit_reclaim_end
mm_vmscan_node_reclaim_begin
mm_vmscan_node_reclaim_end

then place it into kswapd path. But it would be really great to
elaborate some more why this is really needed. Cannot you simply
aggregate stats for kswapd from existing tracepoints?
  

Hello Michal,

Thank you for the quick review!

On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote:
> On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote:
> > The shrink_memcg flow plays a crucial role in memcg reclamation.
> > Currently, it is not possible to trace this point from non-direct
> > reclaim paths.
> 
> Is this really true? AFAICS we have
> mm_vmscan_lru_isolate
> mm_vmscan_lru_shrink_active
> mm_vmscan_lru_shrink_inactive
> 
> which are in the vry core of the memory reclaim. Sure post processing
> those is some work.

Sure, you are absolutely right. In the usual scenario, the memcg
shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the
tracepoints you mentioned and analyze them. However, there is one
potential issue. Enabling these tracepoints will trigger the reclaim
events show for all pages. Although we can filter them per pid, we
cannot filter them per cgroup. Nevertheless, there are times when it
would be extremely beneficial to comprehend the effectiveness of the
reclaim process within the relevant cgroup. For this reason, I am adding
the cgroup name to the memcg tracepoints and implementing a cumulative
tracepoint for memcg shrink (LRU + slab)."

> 
> [...]
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 45780952f4b5..6d89b39d9a91 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -6461,6 +6461,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
> >  		 */
> >  		cond_resched();
> >  
> > +#ifdef CONFIG_MEMCG
> > +		trace_mm_vmscan_memcg_shrink_begin(sc->order,
> > +						   sc->gfp_mask,
> > +						   memcg);
> > +#endif
> 
> this is a common code path for node and direct reclaim which means that
> we will have multiple begin/end tracepoints covering similar operations.
> To me that sounds excessive. If you are missing a cumulative kswapd
> alternative to 
> mm_vmscan_direct_reclaim_begin
> mm_vmscan_direct_reclaim_end
> mm_vmscan_memcg_reclaim_begin
> mm_vmscan_memcg_reclaim_end
> mm_vmscan_memcg_softlimit_reclaim_begin
> mm_vmscan_memcg_softlimit_reclaim_end
> mm_vmscan_node_reclaim_begin
> mm_vmscan_node_reclaim_end
> 
> then place it into kswapd path. But it would be really great to
> elaborate some more why this is really needed. Cannot you simply
> aggregate stats for kswapd from existing tracepoints?
> 
> -- 
> Michal Hocko
> SUSE Labs
  

On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote:
> Hello Michal,
> 
> Thank you for the quick review!
> 
> On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote:
> > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote:
> > > The shrink_memcg flow plays a crucial role in memcg reclamation.
> > > Currently, it is not possible to trace this point from non-direct
> > > reclaim paths.
> > 
> > Is this really true? AFAICS we have
> > mm_vmscan_lru_isolate
> > mm_vmscan_lru_shrink_active
> > mm_vmscan_lru_shrink_inactive
> > 
> > which are in the vry core of the memory reclaim. Sure post processing
> > those is some work.
> 
> Sure, you are absolutely right. In the usual scenario, the memcg
> shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the
> tracepoints you mentioned and analyze them. However, there is one
> potential issue. Enabling these tracepoints will trigger the reclaim
> events show for all pages. Although we can filter them per pid, we
> cannot filter them per cgroup. Nevertheless, there are times when it
> would be extremely beneficial to comprehend the effectiveness of the
> reclaim process within the relevant cgroup. For this reason, I am adding
> the cgroup name to the memcg tracepoints and implementing a cumulative
> tracepoint for memcg shrink (LRU + slab)."

I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it
easier to postprocess per memcg reclaim. But you could do that just by
adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly
does this matter for kswapd and other global reclaim contexts?
  

On Wed, Nov 22, 2023 at 02:24:59PM +0100, Michal Hocko wrote:
> On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote:
> > Hello Michal,
> > 
> > Thank you for the quick review!
> > 
> > On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote:
> > > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote:
> > > > The shrink_memcg flow plays a crucial role in memcg reclamation.
> > > > Currently, it is not possible to trace this point from non-direct
> > > > reclaim paths.
> > > 
> > > Is this really true? AFAICS we have
> > > mm_vmscan_lru_isolate
> > > mm_vmscan_lru_shrink_active
> > > mm_vmscan_lru_shrink_inactive
> > > 
> > > which are in the vry core of the memory reclaim. Sure post processing
> > > those is some work.
> > 
> > Sure, you are absolutely right. In the usual scenario, the memcg
> > shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the
> > tracepoints you mentioned and analyze them. However, there is one
> > potential issue. Enabling these tracepoints will trigger the reclaim
> > events show for all pages. Although we can filter them per pid, we
> > cannot filter them per cgroup. Nevertheless, there are times when it
> > would be extremely beneficial to comprehend the effectiveness of the
> > reclaim process within the relevant cgroup. For this reason, I am adding
> > the cgroup name to the memcg tracepoints and implementing a cumulative
> > tracepoint for memcg shrink (LRU + slab)."
> 
> I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it
> easier to postprocess per memcg reclaim. But you could do that just by
> adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly
> does this matter for kswapd and other global reclaim contexts? 

From my point of view, kswapd and other non-direct reclaim paths are
important for memcg analysis because they also influence the memcg
reclaim statistics.

The tracepoint mm_vmscan_memcg_reclaim_{begin, end} is called from the
direct memcg reclaim flow, such as:
    - a direct write to the 'reclaim' node
    - changing 'max' and 'high' thresholds
    - raising the 'force_empty' mechanism
    - the charge path
    - etc.

However, it doesn't cover global reclaim contexts, so it doesn't provide
us with the full memcg reclaim statistics.
  

Michal, Shakeel,

Sorry for pinging you here, but I don't quite understand your decision
on this patchset.

Is it a NAK or not? If it's not, should I consider redesigning
something? For instance, introducing stub functions to
remove ifdefs from shrink_node_memcgs().

Thank you for taking the time to look into this!

On Wed, Nov 22, 2023 at 09:57:27PM +0300, Dmitry Rokosov wrote:
> On Wed, Nov 22, 2023 at 02:24:59PM +0100, Michal Hocko wrote:
> > On Wed 22-11-23 13:58:36, Dmitry Rokosov wrote:
> > > Hello Michal,
> > > 
> > > Thank you for the quick review!
> > > 
> > > On Wed, Nov 22, 2023 at 11:23:24AM +0100, Michal Hocko wrote:
> > > > On Wed 22-11-23 13:01:56, Dmitry Rokosov wrote:
> > > > > The shrink_memcg flow plays a crucial role in memcg reclamation.
> > > > > Currently, it is not possible to trace this point from non-direct
> > > > > reclaim paths.
> > > > 
> > > > Is this really true? AFAICS we have
> > > > mm_vmscan_lru_isolate
> > > > mm_vmscan_lru_shrink_active
> > > > mm_vmscan_lru_shrink_inactive
> > > > 
> > > > which are in the vry core of the memory reclaim. Sure post processing
> > > > those is some work.
> > > 
> > > Sure, you are absolutely right. In the usual scenario, the memcg
> > > shrinker utilizes two sub-shrinkers: slab and LRU. We can enable the
> > > tracepoints you mentioned and analyze them. However, there is one
> > > potential issue. Enabling these tracepoints will trigger the reclaim
> > > events show for all pages. Although we can filter them per pid, we
> > > cannot filter them per cgroup. Nevertheless, there are times when it
> > > would be extremely beneficial to comprehend the effectiveness of the
> > > reclaim process within the relevant cgroup. For this reason, I am adding
> > > the cgroup name to the memcg tracepoints and implementing a cumulative
> > > tracepoint for memcg shrink (LRU + slab)."
> > 
> > I can see how printing memcg in mm_vmscan_memcg_reclaim_begin makes it
> > easier to postprocess per memcg reclaim. But you could do that just by
> > adding that to mm_vmscan_memcg_reclaim_{begin, end}, no? Why exactly
> > does this matter for kswapd and other global reclaim contexts? 
> 
> From my point of view, kswapd and other non-direct reclaim paths are
> important for memcg analysis because they also influence the memcg
> reclaim statistics.
> 
> The tracepoint mm_vmscan_memcg_reclaim_{begin, end} is called from the
> direct memcg reclaim flow, such as:
>     - a direct write to the 'reclaim' node
>     - changing 'max' and 'high' thresholds
>     - raising the 'force_empty' mechanism
>     - the charge path
>     - etc.
> 
> However, it doesn't cover global reclaim contexts, so it doesn't provide
> us with the full memcg reclaim statistics.
> 
> -- 
> Thank you,
> Dmitry
  

On Thu 23-11-23 14:26:29, Dmitry Rokosov wrote:
> Michal, Shakeel,
> 
> Sorry for pinging you here, but I don't quite understand your decision
> on this patchset.
> 
> Is it a NAK or not? If it's not, should I consider redesigning
> something? For instance, introducing stub functions to
> remove ifdefs from shrink_node_memcgs().
> 
> Thank you for taking the time to look into this!

Sorry for a late reply. I have noticed you have posted a new version.
Let me have a look and comment there.
  

Michal,

On Mon, Nov 27, 2023 at 10:25:12AM +0100, Michal Hocko wrote:
> On Thu 23-11-23 14:26:29, Dmitry Rokosov wrote:
> > Michal, Shakeel,
> > 
> > Sorry for pinging you here, but I don't quite understand your decision
> > on this patchset.
> > 
> > Is it a NAK or not? If it's not, should I consider redesigning
> > something? For instance, introducing stub functions to
> > remove ifdefs from shrink_node_memcgs().
> > 
> > Thank you for taking the time to look into this!
> 
> Sorry for a late reply. I have noticed you have posted a new version.
> Let me have a look and comment there.

No problem! Thanks a lot for your time and attention! Let's continue in
the next version thread.