[2/2] x86/resctrl: Add tracepoint for llc_occupancy tracking

  In our production environment, after removing monitor groups, those unused
RMIDs get stuck in the limbo list forever because their llc_occupancy are
always larger than the threshold. But the unused RMIDs can be successfully
freed by turning up the threshold.

In order to know how much the threshold should be, perf can be used to acquire
the llc_occupancy of RMIDs in each rdt domain.

Instead of using perf tool to track llc_occupancy and filter the log manually,
it is more convenient for users to use tracepoint to do this work. So add a new
tracepoint that shows the llc_occupancy of busy RMIDs when scanning the limbo
list.

Signed-off-by: Haifeng Xu <haifeng.xu@shopee.com>
Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
---
 arch/x86/kernel/cpu/resctrl/monitor.c |  8 ++++++++
 arch/x86/kernel/cpu/resctrl/trace.h   | 15 +++++++++++++++
 2 files changed, 23 insertions(+)
  

Hello!

On 29/02/2024 07:11, Haifeng Xu wrote:
> In our production environment, after removing monitor groups, those unused
> RMIDs get stuck in the limbo list forever because their llc_occupancy are
> always larger than the threshold. But the unused RMIDs can be successfully
> freed by turning up the threshold.
> 
> In order to know how much the threshold should be, perf can be used to acquire
> the llc_occupancy of RMIDs in each rdt domain.
> 
> Instead of using perf tool to track llc_occupancy and filter the log manually,
> it is more convenient for users to use tracepoint to do this work. So add a new
> tracepoint that shows the llc_occupancy of busy RMIDs when scanning the limbo
> list.

> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index c34a35ec0f03..ada392ca75b2 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -362,6 +363,13 @@ void __check_limbo(struct rdt_domain *d, bool force_free)

[expanded the diff hunk for better context]
>               entry = __rmid_entry(idx);
>               if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
>                                          QOS_L3_OCCUP_EVENT_ID, &val,
>                                          arch_mon_ctx)) {
>                       rmid_dirty = true;
>               } else {
>                       rmid_dirty = (val >= resctrl_rmid_realloc_threshold);
>               }
>
>               if (force_free || !rmid_dirty) {
>                       clear_bit(idx, d->rmid_busy_llc);
>                       if (!--entry->busy)
>  				limbo_release_entry(entry);
>  		}

>  		cur_idx = idx + 1;

Ideally this would be the last line in the loop, its how the iterator advances to the next
bit in the bitmap.


> +		/* x86's CLOSID and RMID are independent numbers, so the entry's
> +		 * closid is a invalid CLOSID. But on arm64, the RMID value isn't
> +		 * a unique number for each CLOSID. It's necessary to track both
> +		 * CLOSID and RMID because there may be dependencies between each
> +		 * other on some architectures */
> +		trace_mon_llc_occupancy_limbo(entry->closid, entry->rmid, d->id, val);

I agree outputting both these values is the right thing to do.

resctrl_arch_rmid_read() could return an error, in which case val here is either
uninitialised, or the value of another RMID.
Could you put the tracepoint in the 'else' section of the if/else after
resctrl_arch_rmid_read()? This way it will only output a value to user-space if it is correct.


> diff --git a/arch/x86/kernel/cpu/resctrl/trace.h b/arch/x86/kernel/cpu/resctrl/trace.h
> index 495fb90c8572..35149a75c951 100644
> --- a/arch/x86/kernel/cpu/resctrl/trace.h
> +++ b/arch/x86/kernel/cpu/resctrl/trace.h
> @@ -35,6 +35,21 @@ TRACE_EVENT(pseudo_lock_l3,
>  	    TP_printk("hits=%llu miss=%llu",
>  		      __entry->l3_hits, __entry->l3_miss));
>  
> +TRACE_EVENT(mon_llc_occupancy_limbo,
> +	    TP_PROTO(u32 ctrl_hw_id, u32 mon_hw_id, int id, u64 occupancy),
> +	    TP_ARGS(ctrl_hw_id, mon_hw_id, id, occupancy),
> +	    TP_STRUCT__entry(__field(u32, ctrl_hw_id)
> +			     __field(u32, mon_hw_id)

> +			     __field(int, id)

Nit: Could we call this 'domain_id'? We've got two other ids already, so we should be
clear what each one is!


> +			     __field(u64, occupancy)),

Nit: 'occupancy_bytes'? Just to avoid user-space thinking it needs to convert from the
hardware unit in Intel's SDM ... and just in case we ever have to add another parameter
that is sort of occupancy too.


> +	    TP_fast_assign(__entry->ctrl_hw_id = ctrl_hw_id;
> +			   __entry->mon_hw_id = mon_hw_id;
> +			   __entry->id = id;
> +			   __entry->occupancy = occupancy;),
> +	    TP_printk("ctrl_hw_id=%u mon_hw_id=%u domain=%d llc_occupancy=%llu",
> +		      __entry->ctrl_hw_id, __entry->mon_hw_id, __entry->id, __entry->occupancy)
> +	   );
> +

Tracepoints always expose some implicit details of the kernel internals which can make
supporting them a headache. In this case - I've had some discussion with folk about future
work to defer the limbo work as late as possible - until a new control or monitor group is
allocated. This would be to avoid interrupting user-space tasks to update the limbo list
when the result isn't needed until alloc time.

In this case the tracepoint wouldn't be hit unless user-space made a mkdir() syscall to
force an update - I think this can just be a documentation problem.

I also don't think we should commit to this outputting values for all dirty RMID - which
it does today. If group creation were to fail because there weren't any free RMID you'd
get all the values, I think this is the only case where user-space would care.

Could we document the properties of the the trace-point that can be relied on in
Documentation/arch/x86/resctrl.rst ?

Something like:
| This tracepoint gives you the precise occupancy values for a subset of RMID that are not
| immediately available for allocation. This can't be relied on to produce output every
| second, it may be necessary to attempt to create an empty monitor group to force an
| update. Output may only be produced if creation of a control or monitor group fails.

I think we'll always walk a list of dirty RMID before failing to allocate an RMID, so this
should be future proof.


With the val and documentation bits:
Reviewed-by: James Morse <james.morse@arm.com>


Thanks,

James
  

On 2024/3/2 01:47, James Morse wrote:
> Hello!
> 
> On 29/02/2024 07:11, Haifeng Xu wrote:
>> In our production environment, after removing monitor groups, those unused
>> RMIDs get stuck in the limbo list forever because their llc_occupancy are
>> always larger than the threshold. But the unused RMIDs can be successfully
>> freed by turning up the threshold.
>>
>> In order to know how much the threshold should be, perf can be used to acquire
>> the llc_occupancy of RMIDs in each rdt domain.
>>
>> Instead of using perf tool to track llc_occupancy and filter the log manually,
>> it is more convenient for users to use tracepoint to do this work. So add a new
>> tracepoint that shows the llc_occupancy of busy RMIDs when scanning the limbo
>> list.
> 
>> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
>> index c34a35ec0f03..ada392ca75b2 100644
>> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
>> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
>> @@ -362,6 +363,13 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
> 
> [expanded the diff hunk for better context]
>>               entry = __rmid_entry(idx);
>>               if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
>>                                          QOS_L3_OCCUP_EVENT_ID, &val,
>>                                          arch_mon_ctx)) {
>>                       rmid_dirty = true;
>>               } else {
>>                       rmid_dirty = (val >= resctrl_rmid_realloc_threshold);
>>               }
>>
>>               if (force_free || !rmid_dirty) {
>>                       clear_bit(idx, d->rmid_busy_llc);
>>                       if (!--entry->busy)
>>  				limbo_release_entry(entry);
>>  		}
> 
>>  		cur_idx = idx + 1;
> 
> Ideally this would be the last line in the loop, its how the iterator advances to the next
> bit in the bitmap.
> 
> 
>> +		/* x86's CLOSID and RMID are independent numbers, so the entry's
>> +		 * closid is a invalid CLOSID. But on arm64, the RMID value isn't
>> +		 * a unique number for each CLOSID. It's necessary to track both
>> +		 * CLOSID and RMID because there may be dependencies between each
>> +		 * other on some architectures */
>> +		trace_mon_llc_occupancy_limbo(entry->closid, entry->rmid, d->id, val);
> 
> I agree outputting both these values is the right thing to do.
> 
> resctrl_arch_rmid_read() could return an error, in which case val here is either
> uninitialised, or the value of another RMID.
> Could you put the tracepoint in the 'else' section of the if/else after
> resctrl_arch_rmid_read()? This way it will only output a value to user-space if it is correct.
> 
> 
>> diff --git a/arch/x86/kernel/cpu/resctrl/trace.h b/arch/x86/kernel/cpu/resctrl/trace.h
>> index 495fb90c8572..35149a75c951 100644
>> --- a/arch/x86/kernel/cpu/resctrl/trace.h
>> +++ b/arch/x86/kernel/cpu/resctrl/trace.h
>> @@ -35,6 +35,21 @@ TRACE_EVENT(pseudo_lock_l3,
>>  	    TP_printk("hits=%llu miss=%llu",
>>  		      __entry->l3_hits, __entry->l3_miss));
>>  
>> +TRACE_EVENT(mon_llc_occupancy_limbo,
>> +	    TP_PROTO(u32 ctrl_hw_id, u32 mon_hw_id, int id, u64 occupancy),
>> +	    TP_ARGS(ctrl_hw_id, mon_hw_id, id, occupancy),
>> +	    TP_STRUCT__entry(__field(u32, ctrl_hw_id)
>> +			     __field(u32, mon_hw_id)
> 
>> +			     __field(int, id)
> 
> Nit: Could we call this 'domain_id'? We've got two other ids already, so we should be
> clear what each one is!
> 
> 
>> +			     __field(u64, occupancy)),
> 
> Nit: 'occupancy_bytes'? Just to avoid user-space thinking it needs to convert from the
> hardware unit in Intel's SDM ... and just in case we ever have to add another parameter
> that is sort of occupancy too.
> 
> 
>> +	    TP_fast_assign(__entry->ctrl_hw_id = ctrl_hw_id;
>> +			   __entry->mon_hw_id = mon_hw_id;
>> +			   __entry->id = id;
>> +			   __entry->occupancy = occupancy;),
>> +	    TP_printk("ctrl_hw_id=%u mon_hw_id=%u domain=%d llc_occupancy=%llu",
>> +		      __entry->ctrl_hw_id, __entry->mon_hw_id, __entry->id, __entry->occupancy)
>> +	   );
>> +
> 
> Tracepoints always expose some implicit details of the kernel internals which can make
> supporting them a headache. In this case - I've had some discussion with folk about future
> work to defer the limbo work as late as possible - until a new control or monitor group is
> allocated. This would be to avoid interrupting user-space tasks to update the limbo list
> when the result isn't needed until alloc time.
> 
> In this case the tracepoint wouldn't be hit unless user-space made a mkdir() syscall to
> force an update - I think this can just be a documentation problem.
> 
> I also don't think we should commit to this outputting values for all dirty RMID - which
> it does today. If group creation were to fail because there weren't any free RMID you'd
> get all the values, I think this is the only case where user-space would care.

Yes. 

> 
> Could we document the properties of the the trace-point that can be relied on in
> Documentation/arch/x86/resctrl.rst ?
> 
> Something like:
> | This tracepoint gives you the precise occupancy values for a subset of RMID that are not
> | immediately available for allocation. This can't be relied on to produce output every
> | second, it may be necessary to attempt to create an empty monitor group to force an
> | update. Output may only be produced if creation of a control or monitor group fails.
> 
> I think we'll always walk a list of dirty RMID before failing to allocate an RMID, so this
> should be future proof.
> 
> 
> With the val and documentation bits:
> Reviewed-by: James Morse <james.morse@arm.com>
> 
> 
> Thanks,
> 
> James

Thanks for your suggestions.