[3/3] tick/nohz: Don't shutdown the lowres tick from itself

  In lowres dynticks mode, just like in highres dynticks mode, when there
is no tick to program in the future, the tick eventually gets
deactivated either:

* From the idle loop if in idle mode.
* From the IRQ exit if in full dynticks mode.

Therefore there is no need to deactivate it from the tick itself. This
just just brings more overhead in the idle tick path for no reason.

Fixes: 62c1256d5447 ("timers/nohz: Switch to ONESHOT_STOPPED in the low-res handler when the tick is stopped")
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
 kernel/time/tick-sched.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
  

On 7/14/23 08:08, Frederic Weisbecker wrote:
> In lowres dynticks mode, just like in highres dynticks mode, when there
> is no tick to program in the future, the tick eventually gets
> deactivated either:
> 
> * From the idle loop if in idle mode.
> * From the IRQ exit if in full dynticks mode.
> 
> Therefore there is no need to deactivate it from the tick itself. This
> just just brings more overhead in the idle tick path for no reason.
> 
> Fixes: 62c1256d5447 ("timers/nohz: Switch to ONESHOT_STOPPED in the low-res handler when the tick is stopped")
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> ---
>   kernel/time/tick-sched.c | 24 +++++++++++++-----------
>   1 file changed, 13 insertions(+), 11 deletions(-)
> 
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index dd25da8531f4..101251e103be 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -1392,18 +1392,16 @@ static void tick_lowres_handler(struct clock_event_device *dev)
>   	tick_sched_do_timer(ts, now);
>   	tick_sched_handle(ts, regs);
>   
> -	if (unlikely(ts->tick_stopped)) {
> -		/*
> -		 * The clockevent device is not reprogrammed, so change the
> -		 * clock event device to ONESHOT_STOPPED to avoid spurious
> -		 * interrupts on devices which might not be truly one shot.
> -		 */
> -		tick_program_event(KTIME_MAX, 1);
> -		return;
> +	/*
> +	 * In dynticks mode, tick reprogram is deferred:
> +	 * - to the idle task if in dynticks-idle
> +	 * - to IRQ exit if in full-dynticks.
> +	 */
> +	if (likely(!ts->tick_stopped)) {
> +		hrtimer_forward(&ts->sched_timer, now, TICK_NSEC);
> +		tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);
>   	}
>   
> -	hrtimer_forward(&ts->sched_timer, now, TICK_NSEC);
> -	tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);

One slight concern here though, where in the idle loop is the removed statement 
"tick_program_event(KTIME_MAX, 1);" happening if the tick was already stopped 
before? If it is happening in tick_nohz_stop_tick(), don't we early return from 
there and avoid doing that "tick_program_event(KTIME_MAX, 1);" altogether, if 
the tick was already stopped and the next event has not changed?

         /* Skip reprogram of event if its not changed */
         if (ts->tick_stopped && (expires == ts->next_tick)) {
                 /* Sanity check: make sure clockevent is actually programmed */
                 if (tick == KTIME_MAX || ts->next_tick ==  [...]
                         return;
		[...]
	}

Also just a nit, here you can remove indent by doing:

if (unlikely(ts->tick_stopped))
     return;
hrtimer_forward(&ts->sched_timer, now, TICK_NSEC);
tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);

Which is pretty much the original code except for the tick_program_event().

thanks,

  - Joel


>   }
>   
>   static inline void tick_nohz_activate(struct tick_sched *ts, int mode)
> @@ -1508,7 +1506,11 @@ static enum hrtimer_restart tick_highres_handler(struct hrtimer *timer)
>   	else
>   		ts->next_tick = 0;
>   
> -	/* No need to reprogram if we are in idle or full dynticks mode */
> +	/*
> +	 * In dynticks mode, tick reprogram is deferred:
> +	 * - to the idle task if in dynticks-idle
> +	 * - to IRQ exit if in full-dynticks.
> +	 */
>   	if (unlikely(ts->tick_stopped))
>   		return HRTIMER_NORESTART;
>
  

On Fri, Jul 14, 2023 at 02:44:49PM -0400, Joel Fernandes wrote:
> On 7/14/23 08:08, Frederic Weisbecker wrote:
> One slight concern here though, where in the idle loop is the removed
> statement "tick_program_event(KTIME_MAX, 1);" happening if the tick was
> already stopped before? If it is happening in tick_nohz_stop_tick(), don't
> we early return from there and avoid doing that
> "tick_program_event(KTIME_MAX, 1);" altogether, if the tick was already
> stopped and the next event has not changed?
> 
>         /* Skip reprogram of event if its not changed */
>         if (ts->tick_stopped && (expires == ts->next_tick)) {
>                 /* Sanity check: make sure clockevent is actually programmed */
>                 if (tick == KTIME_MAX || ts->next_tick ==  [...]
>                         return;
> 		[...]
> 	}

Sure, if tick_program_event(KTIME_MAX, 1) was already called in the
previous idle loop iteration, then there is no need to call that again.

Or am I missing something else?

> 
> Also just a nit, here you can remove indent by doing:
> 
> if (unlikely(ts->tick_stopped))
>     return;
> hrtimer_forward(&ts->sched_timer, now, TICK_NSEC);
> tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);
> 
> Which is pretty much the original code except for the tick_program_event().

Either I remove an indent or I remove a statement. I guess it's a matter of
personal taste. I don't mind either way :-)

Thanks.
  

On Fri, Jul 14, 2023 at 8:01 PM Frederic Weisbecker <frederic@kernel.org> wrote:
>
> On Fri, Jul 14, 2023 at 02:44:49PM -0400, Joel Fernandes wrote:
> > On 7/14/23 08:08, Frederic Weisbecker wrote:
> > One slight concern here though, where in the idle loop is the removed
> > statement "tick_program_event(KTIME_MAX, 1);" happening if the tick was
> > already stopped before? If it is happening in tick_nohz_stop_tick(), don't
> > we early return from there and avoid doing that
> > "tick_program_event(KTIME_MAX, 1);" altogether, if the tick was already
> > stopped and the next event has not changed?
> >
> >         /* Skip reprogram of event if its not changed */
> >         if (ts->tick_stopped && (expires == ts->next_tick)) {
> >                 /* Sanity check: make sure clockevent is actually programmed */
> >                 if (tick == KTIME_MAX || ts->next_tick ==  [...]
> >                         return;
> >               [...]
> >       }
>
> Sure, if tick_program_event(KTIME_MAX, 1) was already called in the
> previous idle loop iteration, then there is no need to call that again.
>
> Or am I missing something else?

Just take it with a grain of salt but I think you need to still call
tick_program_event(KTIME_MAX, 1) here for the case where the tick was
previously stopped, and then when the next tick fires (say after a
long time T), but that tick is a one-off and does not result in
restarting the tick -- then there is no one to call
"tick_program_event(KTIME_MAX, 1)".

I think that's the concern Nick was addressing in [1] so that it
resets the tick device correctly?

[1] https://lore.kernel.org/lkml/165089105607.4207.3022534114716811208.tip-bot2@tip-bot2/

> >
> > Also just a nit, here you can remove indent by doing:
> >
> > if (unlikely(ts->tick_stopped))
> >     return;
> > hrtimer_forward(&ts->sched_timer, now, TICK_NSEC);
> > tick_program_event(hrtimer_get_expires(&ts->sched_timer), 1);
> >
> > Which is pretty much the original code except for the tick_program_event().
>
> Either I remove an indent or I remove a statement. I guess it's a matter of
> personal taste. I don't mind either way :-)

Ah true, in defense of the "less indent" way, the original code was
also using that style. ;-) But I am also Ok with either way. :-)

thanks,

 - Joel
  

On Fri, Jul 14, 2023 at 09:02:43PM -0400, Joel Fernandes wrote:
> On Fri, Jul 14, 2023 at 8:01 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> >
> > On Fri, Jul 14, 2023 at 02:44:49PM -0400, Joel Fernandes wrote:
> > > On 7/14/23 08:08, Frederic Weisbecker wrote:
> > > One slight concern here though, where in the idle loop is the removed
> > > statement "tick_program_event(KTIME_MAX, 1);" happening if the tick was
> > > already stopped before? If it is happening in tick_nohz_stop_tick(), don't
> > > we early return from there and avoid doing that
> > > "tick_program_event(KTIME_MAX, 1);" altogether, if the tick was already
> > > stopped and the next event has not changed?
> > >
> > >         /* Skip reprogram of event if its not changed */
> > >         if (ts->tick_stopped && (expires == ts->next_tick)) {
> > >                 /* Sanity check: make sure clockevent is actually programmed */
> > >                 if (tick == KTIME_MAX || ts->next_tick ==  [...]
> > >                         return;
> > >               [...]
> > >       }
> >
> > Sure, if tick_program_event(KTIME_MAX, 1) was already called in the
> > previous idle loop iteration, then there is no need to call that again.
> >
> > Or am I missing something else?
> 
> Just take it with a grain of salt but I think you need to still call
> tick_program_event(KTIME_MAX, 1) here for the case where the tick was
> previously stopped, and then when the next tick fires (say after a
> long time T), but that tick is a one-off and does not result in
> restarting the tick -- then there is no one to call
> "tick_program_event(KTIME_MAX, 1)".

I'm a bit confused about that one-off thing. What can trigger that timer
interrupt if it has been stopped?

One thing can happen though: a pending timer IRQ while we are stopping the
tick (IRQs are disabled in that idle loop portion). But then that pending timer
interrupt is not going to reprogram another one. So it remains stopped.

Thanks.
  

> On Jul 15, 2023, at 2:19 PM, Frederic Weisbecker <frederic@kernel.org> wrote:
> 
> On Fri, Jul 14, 2023 at 09:02:43PM -0400, Joel Fernandes wrote:
>>> On Fri, Jul 14, 2023 at 8:01 PM Frederic Weisbecker <frederic@kernel.org> wrote:
>>> 
>>> On Fri, Jul 14, 2023 at 02:44:49PM -0400, Joel Fernandes wrote:
>>>> On 7/14/23 08:08, Frederic Weisbecker wrote:
>>>> One slight concern here though, where in the idle loop is the removed
>>>> statement "tick_program_event(KTIME_MAX, 1);" happening if the tick was
>>>> already stopped before? If it is happening in tick_nohz_stop_tick(), don't
>>>> we early return from there and avoid doing that
>>>> "tick_program_event(KTIME_MAX, 1);" altogether, if the tick was already
>>>> stopped and the next event has not changed?
>>>> 
>>>>        /* Skip reprogram of event if its not changed */
>>>>        if (ts->tick_stopped && (expires == ts->next_tick)) {
>>>>                /* Sanity check: make sure clockevent is actually programmed */
>>>>                if (tick == KTIME_MAX || ts->next_tick ==  [...]
>>>>                        return;
>>>>              [...]
>>>>      }
>>> 
>>> Sure, if tick_program_event(KTIME_MAX, 1) was already called in the
>>> previous idle loop iteration, then there is no need to call that again.
>>> 
>>> Or am I missing something else?
>> 
>> Just take it with a grain of salt but I think you need to still call
>> tick_program_event(KTIME_MAX, 1) here for the case where the tick was
>> previously stopped, and then when the next tick fires (say after a
>> long time T), but that tick is a one-off and does not result in
>> restarting the tick -- then there is no one to call
>> "tick_program_event(KTIME_MAX, 1)".
> 
> I'm a bit confused about that one-off thing. What can trigger that timer
> interrupt if it has been stopped?

It is the tick that has been stopped in this scenario.
The timer could still be armed to serve a future hrtimer?

I think the naming in the code for is a bit confusing for tick vs timer event,
so I could be the confused one.

Thanks,

 - Joel


> One thing can happen though: a pending timer IRQ while we are stopping the
> tick (IRQs are disabled in that idle loop portion). But then that pending timer
> interrupt is not going to reprogram another one. So it remains stopped.
> 
> Thanks.
  

On Sat, Jul 15, 2023 at 08:18:57PM +0200, Frederic Weisbecker wrote:
> On Fri, Jul 14, 2023 at 09:02:43PM -0400, Joel Fernandes wrote:
> > On Fri, Jul 14, 2023 at 8:01 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> > >
> > > On Fri, Jul 14, 2023 at 02:44:49PM -0400, Joel Fernandes wrote:
> > > > On 7/14/23 08:08, Frederic Weisbecker wrote:
> > > > One slight concern here though, where in the idle loop is the removed
> > > > statement "tick_program_event(KTIME_MAX, 1);" happening if the tick was
> > > > already stopped before? If it is happening in tick_nohz_stop_tick(), don't
> > > > we early return from there and avoid doing that
> > > > "tick_program_event(KTIME_MAX, 1);" altogether, if the tick was already
> > > > stopped and the next event has not changed?
> > > >
> > > >         /* Skip reprogram of event if its not changed */
> > > >         if (ts->tick_stopped && (expires == ts->next_tick)) {
> > > >                 /* Sanity check: make sure clockevent is actually programmed */
> > > >                 if (tick == KTIME_MAX || ts->next_tick ==  [...]
> > > >                         return;
> > > >               [...]
> > > >       }
> > >
> > > Sure, if tick_program_event(KTIME_MAX, 1) was already called in the
> > > previous idle loop iteration, then there is no need to call that again.
> > >
> > > Or am I missing something else?
> > 
> > Just take it with a grain of salt but I think you need to still call
> > tick_program_event(KTIME_MAX, 1) here for the case where the tick was
> > previously stopped, and then when the next tick fires (say after a
> > long time T), but that tick is a one-off and does not result in
> > restarting the tick -- then there is no one to call
> > "tick_program_event(KTIME_MAX, 1)".
> 
> I'm a bit confused about that one-off thing. What can trigger that timer
> interrupt if it has been stopped?
> 
> One thing can happen though: a pending timer IRQ while we are stopping the
> tick (IRQs are disabled in that idle loop portion). But then that pending timer
> interrupt is not going to reprogram another one. So it remains stopped.

I think I see what you mean now. Maybe I wrongly assumed the above 'Skip
reprogram of event' code could early return and skip over
"tick_program_event(KTIME_MAX, 1);", but I think it cannot because of the
"expires != ts->next_tick" check.

 Maybe the "tick_program_event(KTIME_MAX, 1)" bit in tick_nohz_handler() is
 supposed to handle buggy hardware where an unexpected timer event came
 through? In such a situation, the idle loop will not write
 "tick_program_event(KTIME_MAX, 1);" again because it already did so the
 previous time, as you pointed.

Adding Vineeth who is also looking into this code.

thanks,

 - Joel
  

On Mon, Jul 17, 2023 at 05:30:49PM +0000, Joel Fernandes wrote:
> I think I see what you mean now. Maybe I wrongly assumed the above 'Skip
> reprogram of event' code could early return and skip over
> "tick_program_event(KTIME_MAX, 1);", but I think it cannot because of the
> "expires != ts->next_tick" check.
> 
>  Maybe the "tick_program_event(KTIME_MAX, 1)" bit in tick_nohz_handler() is
>  supposed to handle buggy hardware where an unexpected timer event came
>  through? In such a situation, the idle loop will not write
>  "tick_program_event(KTIME_MAX, 1);" again because it already did so the
>  previous time, as you pointed.

Well at least if the double write was put there intentionally in order to
fix buggy hardware, this was neither mentionned nor commented anywhere AFAICT.

Thanks.

> 
> Adding Vineeth who is also looking into this code.
> 
> thanks,
> 
>  - Joel
>