[v3] sched/core: introduce sched_core_idle_cpu()

  As core scheduling introduced, a new state of idle is defined as
force idle, running idle task but nr_running greater than zero.

If a cpu is in force idle state, idle_cpu() will return zero. This
result makes sense in some scenarios, e.g., load balance,
showacpu when dumping, and judge the RCU boost kthread is starving.

But this will cause error in other scenarios, e.g., tick_irq_exit():
When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
not become 1, which became 0 in tick_nohz_irq_enter().
ts->idle_sleeptime won't update in function update_ts_time_stats(), if
ts->idle_active is 0, which should be 1. And this bug will result that
ts->idle_sleeptime is less than the actual value, and finally will
result that the idle time in /proc/stat is less than the actual value.

To solve this problem, we introduce sched_core_idle_cpu(), which
returns 1 when force idle. We audit all users of idle_cpu(), and
change idle_cpu() into sched_core_idle_cpu() in function
tick_irq_exit().

v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
function tick_irq_exit(). And modify the corresponding commit log.

Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
Reviewed-by: Peter Zijlstra <peterz@infradead.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Joel Fernandes <joel@joelfernandes.org>
Link: https://lore.kernel.org/lkml/1687631295-126383-1-git-send-email-CruzZhao@linux.alibaba.com
---
 include/linux/sched.h |  2 ++
 kernel/sched/core.c   | 13 +++++++++++++
 kernel/softirq.c      |  2 +-
 3 files changed, 16 insertions(+), 1 deletion(-)
  

On Thu, Jun 29, 2023 at 12:02:04PM +0800, Cruz Zhao wrote:
> As core scheduling introduced, a new state of idle is defined as
> force idle, running idle task but nr_running greater than zero.
> 
> If a cpu is in force idle state, idle_cpu() will return zero. This
> result makes sense in some scenarios, e.g., load balance,
> showacpu when dumping, and judge the RCU boost kthread is starving.
> 
> But this will cause error in other scenarios, e.g., tick_irq_exit():
> When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
> that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
> is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
> not become 1, which became 0 in tick_nohz_irq_enter().
> ts->idle_sleeptime won't update in function update_ts_time_stats(), if
> ts->idle_active is 0, which should be 1. And this bug will result that
> ts->idle_sleeptime is less than the actual value, and finally will
> result that the idle time in /proc/stat is less than the actual value.
> 
> To solve this problem, we introduce sched_core_idle_cpu(), which
> returns 1 when force idle. We audit all users of idle_cpu(), and
> change idle_cpu() into sched_core_idle_cpu() in function
> tick_irq_exit().
> 
> v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
> function tick_irq_exit(). And modify the corresponding commit log.
> 
> Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
> Reviewed-by: Peter Zijlstra <peterz@infradead.org>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> Reviewed-by: Joel Fernandes <joel@joelfernandes.org>

Please wait for people to actually provide you with their Reviewed-by: tags
before writing it.

Aside of that, the patch looks good so you can put this one:

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
  

On 6/29/2023 12:02 AM, Cruz Zhao wrote:
> As core scheduling introduced, a new state of idle is defined as
> force idle, running idle task but nr_running greater than zero.
> 
> If a cpu is in force idle state, idle_cpu() will return zero. This
> result makes sense in some scenarios, e.g., load balance,
> showacpu when dumping, and judge the RCU boost kthread is starving.
> 
> But this will cause error in other scenarios, e.g., tick_irq_exit():
> When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
> that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
> is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
> not become 1, which became 0 in tick_nohz_irq_enter().
> ts->idle_sleeptime won't update in function update_ts_time_stats(), if
> ts->idle_active is 0, which should be 1. And this bug will result that
> ts->idle_sleeptime is less than the actual value, and finally will
> result that the idle time in /proc/stat is less than the actual value.
> 
> To solve this problem, we introduce sched_core_idle_cpu(), which
> returns 1 when force idle. We audit all users of idle_cpu(), and
> change idle_cpu() into sched_core_idle_cpu() in function
> tick_irq_exit().

Reviewed-by: Joel Fernandes <joel@joelfernandes.org>

 - Joel


> 
> v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
> function tick_irq_exit(). And modify the corresponding commit log.
> 
> Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
> Reviewed-by: Peter Zijlstra <peterz@infradead.org>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> Reviewed-by: Joel Fernandes <joel@joelfernandes.org>
> Link: https://lore.kernel.org/lkml/1687631295-126383-1-git-send-email-CruzZhao@linux.alibaba.com
> ---
>  include/linux/sched.h |  2 ++
>  kernel/sched/core.c   | 13 +++++++++++++
>  kernel/softirq.c      |  2 +-
>  3 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index b09a83bfad8b..73e61c0f10a7 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);
>  extern void sched_core_fork(struct task_struct *p);
>  extern int sched_core_share_pid(unsigned int cmd, pid_t pid, enum pid_type type,
>  				unsigned long uaddr);
> +extern int sched_core_idle_cpu(int cpu);
>  #else
>  static inline void sched_core_free(struct task_struct *tsk) { }
>  static inline void sched_core_fork(struct task_struct *p) { }
> +static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }
>  #endif
>  
>  extern void sched_set_stop_task(int cpu, struct task_struct *stop);
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 71c1a0f232b4..c80088956987 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)
>  	return cpu_rq(cpu)->idle;
>  }
>  
> +#ifdef CONFIG_SCHED_CORE
> +int sched_core_idle_cpu(int cpu)
> +{
> +	struct rq *rq = cpu_rq(cpu);
> +
> +	if (sched_core_enabled(rq) && rq->curr == rq->idle)
> +		return 1;
> +
> +	return idle_cpu(cpu);
> +}
> +
> +#endif
> +
>  #ifdef CONFIG_SMP
>  /*
>   * This function computes an effective utilization for the given CPU, to be
> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index c8a6913c067d..98b98991ce45 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)
>  	int cpu = smp_processor_id();
>  
>  	/* Make sure that timer wheel updates are propagated */
> -	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
> +	if ((sched_core_idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
>  		if (!in_hardirq())
>  			tick_nohz_irq_exit();
>  	}
  

On Thu, Jun 29, 2023 at 12:02:04PM +0800, Cruz Zhao wrote:
> As core scheduling introduced, a new state of idle is defined as
> force idle, running idle task but nr_running greater than zero.
> 
> If a cpu is in force idle state, idle_cpu() will return zero. This
> result makes sense in some scenarios, e.g., load balance,
> showacpu when dumping, and judge the RCU boost kthread is starving.
> 
> But this will cause error in other scenarios, e.g., tick_irq_exit():
> When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
> that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
> is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
> not become 1, which became 0 in tick_nohz_irq_enter().
> ts->idle_sleeptime won't update in function update_ts_time_stats(), if
> ts->idle_active is 0, which should be 1. And this bug will result that
> ts->idle_sleeptime is less than the actual value, and finally will
> result that the idle time in /proc/stat is less than the actual value.
> 
> To solve this problem, we introduce sched_core_idle_cpu(), which
> returns 1 when force idle. We audit all users of idle_cpu(), and
> change idle_cpu() into sched_core_idle_cpu() in function
> tick_irq_exit().
> 
> v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
> function tick_irq_exit(). And modify the corresponding commit log.
> 
> Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
> Reviewed-by: Peter Zijlstra <peterz@infradead.org>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> Reviewed-by: Joel Fernandes <joel@joelfernandes.org>
> Link: https://lore.kernel.org/lkml/1687631295-126383-1-git-send-email-CruzZhao@linux.alibaba.com
> ---
>  include/linux/sched.h |  2 ++
>  kernel/sched/core.c   | 13 +++++++++++++
>  kernel/softirq.c      |  2 +-
>  3 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index b09a83bfad8b..73e61c0f10a7 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);
>  extern void sched_core_fork(struct task_struct *p);
>  extern int sched_core_share_pid(unsigned int cmd, pid_t pid, enum pid_type type,
>  				unsigned long uaddr);
> +extern int sched_core_idle_cpu(int cpu);
>  #else
>  static inline void sched_core_free(struct task_struct *tsk) { }
>  static inline void sched_core_fork(struct task_struct *p) { }
> +static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }
>  #endif
>  
>  extern void sched_set_stop_task(int cpu, struct task_struct *stop);
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 71c1a0f232b4..c80088956987 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)
>  	return cpu_rq(cpu)->idle;
>  }
>  
> +#ifdef CONFIG_SCHED_CORE
> +int sched_core_idle_cpu(int cpu)
> +{
> +	struct rq *rq = cpu_rq(cpu);
> +
> +	if (sched_core_enabled(rq) && rq->curr == rq->idle)
> +		return 1;

If the intention is to consider forced idle cpus as idle, then should
the above condition written as:

	if (sched_core_enabled(rq) && rq->core->core_forceidle_count)
		return 1;
?

Or as long as a single cookied task is running, all normal idle cpus are
regarded forced idle here and 1 is returned while previously, idle_cpu()
is called for those cpus and if they have wakeup task pending, they are
not regarded as idle so looks like a behaviour change.

Thanks,
Aaron

> +
> +	return idle_cpu(cpu);
> +}
> +
> +#endif
> +
>  #ifdef CONFIG_SMP
>  /*
>   * This function computes an effective utilization for the given CPU, to be
> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index c8a6913c067d..98b98991ce45 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)
>  	int cpu = smp_processor_id();
>  
>  	/* Make sure that timer wheel updates are propagated */
> -	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
> +	if ((sched_core_idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
>  		if (!in_hardirq())
>  			tick_nohz_irq_exit();
>  	}
> -- 
> 2.27.0
>
  

On Tue, Jul 4, 2023 at 1:40 AM Aaron Lu <aaron.lu@intel.com> wrote:
>
> On Thu, Jun 29, 2023 at 12:02:04PM +0800, Cruz Zhao wrote:
> > As core scheduling introduced, a new state of idle is defined as
> > force idle, running idle task but nr_running greater than zero.
> >
> > If a cpu is in force idle state, idle_cpu() will return zero. This
> > result makes sense in some scenarios, e.g., load balance,
> > showacpu when dumping, and judge the RCU boost kthread is starving.
> >
> > But this will cause error in other scenarios, e.g., tick_irq_exit():
> > When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
> > that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
> > is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
> > not become 1, which became 0 in tick_nohz_irq_enter().
> > ts->idle_sleeptime won't update in function update_ts_time_stats(), if
> > ts->idle_active is 0, which should be 1. And this bug will result that
> > ts->idle_sleeptime is less than the actual value, and finally will
> > result that the idle time in /proc/stat is less than the actual value.
> >
> > To solve this problem, we introduce sched_core_idle_cpu(), which
> > returns 1 when force idle. We audit all users of idle_cpu(), and
> > change idle_cpu() into sched_core_idle_cpu() in function
> > tick_irq_exit().
> >
> > v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
> > function tick_irq_exit(). And modify the corresponding commit log.
> >
> > Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
> > Reviewed-by: Peter Zijlstra <peterz@infradead.org>
> > Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> > Reviewed-by: Joel Fernandes <joel@joelfernandes.org>
> > Link: https://lore.kernel.org/lkml/1687631295-126383-1-git-send-email-CruzZhao@linux.alibaba.com
> > ---
> >  include/linux/sched.h |  2 ++
> >  kernel/sched/core.c   | 13 +++++++++++++
> >  kernel/softirq.c      |  2 +-
> >  3 files changed, 16 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index b09a83bfad8b..73e61c0f10a7 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);
> >  extern void sched_core_fork(struct task_struct *p);
> >  extern int sched_core_share_pid(unsigned int cmd, pid_t pid, enum pid_type type,
> >                               unsigned long uaddr);
> > +extern int sched_core_idle_cpu(int cpu);
> >  #else
> >  static inline void sched_core_free(struct task_struct *tsk) { }
> >  static inline void sched_core_fork(struct task_struct *p) { }
> > +static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }
> >  #endif
> >
> >  extern void sched_set_stop_task(int cpu, struct task_struct *stop);
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 71c1a0f232b4..c80088956987 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)
> >       return cpu_rq(cpu)->idle;
> >  }
> >
> > +#ifdef CONFIG_SCHED_CORE
> > +int sched_core_idle_cpu(int cpu)
> > +{
> > +     struct rq *rq = cpu_rq(cpu);
> > +
> > +     if (sched_core_enabled(rq) && rq->curr == rq->idle)
> > +             return 1;
>
> If the intention is to consider forced idle cpus as idle, then should
> the above condition written as:
>
>         if (sched_core_enabled(rq) && rq->core->core_forceidle_count)
>                 return 1;
> ?
>
> Or as long as a single cookied task is running, all normal idle cpus are
> regarded forced idle here and 1 is returned while previously, idle_cpu()
> is called for those cpus and if they have wakeup task pending, they are
> not regarded as idle so looks like a behaviour change.
>

Ah you're right, great insight. _sigh_ I should not have missed that
during review. It will change idle_cpu() behavior if core sched is
enabled...

 - Joel
  

在 2023/7/4 下午1:39, Aaron Lu 写道:
>> +#ifdef CONFIG_SCHED_CORE
>> +int sched_core_idle_cpu(int cpu)
>> +{
>> +	struct rq *rq = cpu_rq(cpu);
>> +
>> +	if (sched_core_enabled(rq) && rq->curr == rq->idle)
>> +		return 1;
> 
> If the intention is to consider forced idle cpus as idle, then should
> the above condition written as:
> 
> 	if (sched_core_enabled(rq) && rq->core->core_forceidle_count)
> 		return 1;
> ?
> 
> Or as long as a single cookied task is running, all normal idle cpus are
> regarded forced idle here and 1 is returned while previously, idle_cpu()
> is called for those cpus and if they have wakeup task pending, they are
> not regarded as idle so looks like a behaviour change.
> 
> Thanks,
> Aaron
> 

I'll fix this problem in the next version.

Best,
Cruz Zhao

>> +
>> +	return idle_cpu(cpu);
>> +}
>> +
>> +#endif
>> +
>>  #ifdef CONFIG_SMP
>>  /*
>>   * This function computes an effective utilization for the given CPU, to be
>> diff --git a/kernel/softirq.c b/kernel/softirq.c
>> index c8a6913c067d..98b98991ce45 100644
>> --- a/kernel/softirq.c
>> +++ b/kernel/softirq.c
>> @@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)
>>  	int cpu = smp_processor_id();
>>  
>>  	/* Make sure that timer wheel updates are propagated */
>> -	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
>> +	if ((sched_core_idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
>>  		if (!in_hardirq())
>>  			tick_nohz_irq_exit();
>>  	}
>> -- 
>> 2.27.0
>>