[RFC,3/3] locktorture: Make the rt_boost factor a tunable

  The rt boosting in locktorture has a factor variable large enough that
boosting only happens once every minute or so. Add a tunable to educe
the factor so that boosting happens more often, to test paths and arrive
at failure modes earlier. With this change, I can set the factor to
like 50 and have the boosting happens every 10 seconds or so.

Tested with boot parameters:
locktorture.torture_type=mutex_lock
locktorture.onoff_interval=1
locktorture.nwriters_stress=8
locktorture.stutter=0
locktorture.rt_boost=1
locktorture.rt_boost_factor=50
locktorture.nlocks=3

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/locking/locktorture.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
  

On Wed, Nov 23, 2022 at 01:21:04AM +0000, Joel Fernandes (Google) wrote:
> The rt boosting in locktorture has a factor variable large enough that
> boosting only happens once every minute or so. Add a tunable to educe
> the factor so that boosting happens more often, to test paths and arrive
> at failure modes earlier. With this change, I can set the factor to
> like 50 and have the boosting happens every 10 seconds or so.
> 
> Tested with boot parameters:
> locktorture.torture_type=mutex_lock
> locktorture.onoff_interval=1
> locktorture.nwriters_stress=8
> locktorture.stutter=0
> locktorture.rt_boost=1
> locktorture.rt_boost_factor=50
> locktorture.nlocks=3
> 
> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>

This looks good, and is what I should have done to start with.  But
it depends on the first commit, so I will hold off for the moment.

						Thanx, Paul

> ---
>  kernel/locking/locktorture.c | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> index 5a388ac96a9b..e4529c2166e9 100644
> --- a/kernel/locking/locktorture.c
> +++ b/kernel/locking/locktorture.c
> @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
>  	     "Number of seconds between stats printk()s");
>  torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
>  torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
>  torture_param(int, verbose, 1,
>  	     "Enable verbose debugging printk()s");
>  torture_param(int, nlocks, 1,
> @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
>  
>  static void torture_rt_boost(struct torture_random_state *trsp)
>  {
> -	const unsigned int factor = 50000; /* yes, quite arbitrary */
> +	const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
>  
>  	if (!rt_boost)
>  		return;
>  
>  	if (!rt_task(current)) {
>  		/*
> -		 * Boost priority once every ~50k operations. When the
> -		 * task tries to take the lock, the rtmutex it will account
> +		 * Boost priority once every rt_boost_factor operations. When
> +		 * the task tries to take the lock, the rtmutex it will account
>  		 * for the new priority, and do any corresponding pi-dance.
>  		 */
>  		if (trsp && !(torture_random(trsp) %
> @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
>  			return;
>  	} else {
>  		/*
> -		 * The task will remain boosted for another ~500k operations,
> -		 * then restored back to its original prio, and so forth.
> +		 * The task will remain boosted for another 10*rt_boost_factor
> +		 * operations, then restored back to its original prio, and so
> +		 * forth.
>  		 *
>  		 * When @trsp is nil, we want to force-reset the task for
>  		 * stopping the kthread.
> -- 
> 2.38.1.584.g0f3c55d4c2-goog
>
  

On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> The rt boosting in locktorture has a factor variable large enough that
> boosting only happens once every minute or so. Add a tunable to educe
> the factor so that boosting happens more often, to test paths and arrive
> at failure modes earlier. With this change, I can set the factor to
> like 50 and have the boosting happens every 10 seconds or so.
> 
> Tested with boot parameters:
> locktorture.torture_type=mutex_lock
> locktorture.onoff_interval=1
> locktorture.nwriters_stress=8
> locktorture.stutter=0
> locktorture.rt_boost=1
> locktorture.rt_boost_factor=50
> locktorture.nlocks=3
> 
> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> ---
>  kernel/locking/locktorture.c | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> index 5a388ac96a9b..e4529c2166e9 100644
> --- a/kernel/locking/locktorture.c
> +++ b/kernel/locking/locktorture.c
> @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
>  	     "Number of seconds between stats printk()s");
>  torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
>  torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
>  torture_param(int, verbose, 1,
>  	     "Enable verbose debugging printk()s");
>  torture_param(int, nlocks, 1,
> @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
>  
>  static void torture_rt_boost(struct torture_random_state *trsp)
>  {
> -	const unsigned int factor = 50000; /* yes, quite arbitrary */
> +	const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
>  
>  	if (!rt_boost)
>  		return;
>  
>  	if (!rt_task(current)) {
>  		/*
> -		 * Boost priority once every ~50k operations. When the
> -		 * task tries to take the lock, the rtmutex it will account
> +		 * Boost priority once every rt_boost_factor operations. When
> +		 * the task tries to take the lock, the rtmutex it will account
>  		 * for the new priority, and do any corresponding pi-dance.
>  		 */
>  		if (trsp && !(torture_random(trsp) %
> @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
>  			return;
>  	} else {
>  		/*
> -		 * The task will remain boosted for another ~500k operations,
> -		 * then restored back to its original prio, and so forth.
> +		 * The task will remain boosted for another 10*rt_boost_factor
Maybe I understand incorrectly, the code is
cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?
May I know where the 10 comes from?

thanks,
Chenyu
  

On Wed, Dec 21, 2022 at 12:28:39PM +0800, Chen Yu wrote:
> On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> > The rt boosting in locktorture has a factor variable large enough that
> > boosting only happens once every minute or so. Add a tunable to educe
> > the factor so that boosting happens more often, to test paths and arrive
> > at failure modes earlier. With this change, I can set the factor to
> > like 50 and have the boosting happens every 10 seconds or so.
> > 
> > Tested with boot parameters:
> > locktorture.torture_type=mutex_lock
> > locktorture.onoff_interval=1
> > locktorture.nwriters_stress=8
> > locktorture.stutter=0
> > locktorture.rt_boost=1
> > locktorture.rt_boost_factor=50
> > locktorture.nlocks=3
> > 
> > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > ---
> >  kernel/locking/locktorture.c | 12 +++++++-----
> >  1 file changed, 7 insertions(+), 5 deletions(-)
> > 
> > diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> > index 5a388ac96a9b..e4529c2166e9 100644
> > --- a/kernel/locking/locktorture.c
> > +++ b/kernel/locking/locktorture.c
> > @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> >  	     "Number of seconds between stats printk()s");
> >  torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> >  torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> > +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> >  torture_param(int, verbose, 1,
> >  	     "Enable verbose debugging printk()s");
> >  torture_param(int, nlocks, 1,
> > @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
> >  
> >  static void torture_rt_boost(struct torture_random_state *trsp)
> >  {
> > -	const unsigned int factor = 50000; /* yes, quite arbitrary */
> > +	const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
> >  
> >  	if (!rt_boost)
> >  		return;
> >  
> >  	if (!rt_task(current)) {
> >  		/*
> > -		 * Boost priority once every ~50k operations. When the
> > -		 * task tries to take the lock, the rtmutex it will account
> > +		 * Boost priority once every rt_boost_factor operations. When
> > +		 * the task tries to take the lock, the rtmutex it will account
> >  		 * for the new priority, and do any corresponding pi-dance.
> >  		 */
> >  		if (trsp && !(torture_random(trsp) %
> > @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> >  			return;
> >  	} else {
> >  		/*
> > -		 * The task will remain boosted for another ~500k operations,
> > -		 * then restored back to its original prio, and so forth.
> > +		 * The task will remain boosted for another 10*rt_boost_factor
> Maybe I understand incorrectly, the code is
> cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?

It looks that way to me, but I might be missing something.  Joel?

							Thanx, Paul

> May I know where the 10 comes from?
> 
> thanks,
> Chenyu
  

On Thu, Jan 05, 2023 at 10:27:18AM -0800, Paul E. McKenney wrote:
> On Wed, Dec 21, 2022 at 12:28:39PM +0800, Chen Yu wrote:
> > On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> > > The rt boosting in locktorture has a factor variable large enough that
> > > boosting only happens once every minute or so. Add a tunable to educe
> > > the factor so that boosting happens more often, to test paths and arrive
> > > at failure modes earlier. With this change, I can set the factor to
> > > like 50 and have the boosting happens every 10 seconds or so.
> > > 
> > > Tested with boot parameters:
> > > locktorture.torture_type=mutex_lock
> > > locktorture.onoff_interval=1
> > > locktorture.nwriters_stress=8
> > > locktorture.stutter=0
> > > locktorture.rt_boost=1
> > > locktorture.rt_boost_factor=50
> > > locktorture.nlocks=3
> > > 
> > > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > > ---
> > >  kernel/locking/locktorture.c | 12 +++++++-----
> > >  1 file changed, 7 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> > > index 5a388ac96a9b..e4529c2166e9 100644
> > > --- a/kernel/locking/locktorture.c
> > > +++ b/kernel/locking/locktorture.c
> > > @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> > >  	     "Number of seconds between stats printk()s");
> > >  torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> > >  torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> > > +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> > >  torture_param(int, verbose, 1,
> > >  	     "Enable verbose debugging printk()s");
> > >  torture_param(int, nlocks, 1,
> > > @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
> > >  
> > >  static void torture_rt_boost(struct torture_random_state *trsp)
> > >  {
> > > -	const unsigned int factor = 50000; /* yes, quite arbitrary */
> > > +	const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
> > >  
> > >  	if (!rt_boost)
> > >  		return;
> > >  
> > >  	if (!rt_task(current)) {
> > >  		/*
> > > -		 * Boost priority once every ~50k operations. When the
> > > -		 * task tries to take the lock, the rtmutex it will account
> > > +		 * Boost priority once every rt_boost_factor operations. When
> > > +		 * the task tries to take the lock, the rtmutex it will account
> > >  		 * for the new priority, and do any corresponding pi-dance.
> > >  		 */
> > >  		if (trsp && !(torture_random(trsp) %
> > > @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> > >  			return;
> > >  	} else {
> > >  		/*
> > > -		 * The task will remain boosted for another ~500k operations,
> > > -		 * then restored back to its original prio, and so forth.
> > > +		 * The task will remain boosted for another 10*rt_boost_factor
> > Maybe I understand incorrectly, the code is
> > cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?
> 
> It looks that way to me, but I might be missing something.  Joel?
> > May I know where the 10 comes from?

The comment in existing code was 500k ops.

Yes, Chen is right, the comment can be improved to mention the actual
equation. I was just going by the initial comment of ~500K ops. Since factor
now defaults to 50k, this translates to 500k (10 times the factor) ops which
it does for a 4-5 CPU system.

But I am Ok with the comment changing to what Chen suggested though!

thanks,

 - Joel