[v2,1/2] net: make driver settling time configurable

  During IP auto configuration, some drivers apparently need to wait a
certain length of time to settle; as this is not true for all drivers,
make this length of time configurable.

Signed-off-by: David Ventura <david@davidv.dev>
---
 .../admin-guide/kernel-parameters.txt         |  4 ++++
 Documentation/admin-guide/nfs/nfsroot.rst     |  3 +++
 net/ipv4/ipconfig.c                           | 23 ++++++++++++++++---
 3 files changed, 27 insertions(+), 3 deletions(-)
  

On 2/8/24 12:52, David Ventura wrote:
> During IP auto configuration, some drivers apparently need to wait a
> certain length of time to settle; as this is not true for all drivers,
> make this length of time configurable.
> 
> Signed-off-by: David Ventura <david@davidv.dev>
> ---
>  .../admin-guide/kernel-parameters.txt         |  4 ++++
>  Documentation/admin-guide/nfs/nfsroot.rst     |  3 +++
>  net/ipv4/ipconfig.c                           | 23 ++++++++++++++++---
>  3 files changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index b47940577c10..b07a035642fa 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -2291,6 +2291,10 @@
>  
>  	ip=		[IP_PNP]
>  			See Documentation/admin-guide/nfs/nfsroot.rst.
> +	ip.dev_wait_ms=
> +			[IP_PNP]
> +			See Documentation/admin-guide/nfs/nfsroot.rst.
> +
>  
>  	ipcmni_extend	[KNL,EARLY] Extend the maximum number of unique System V
>  			IPC identifiers from 32,768 to 16,777,216.
> diff --git a/Documentation/admin-guide/nfs/nfsroot.rst b/Documentation/admin-guide/nfs/nfsroot.rst
> index 135218f33394..f26f7a342af6 100644
> --- a/Documentation/admin-guide/nfs/nfsroot.rst
> +++ b/Documentation/admin-guide/nfs/nfsroot.rst
> @@ -223,6 +223,9 @@ ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns
>    /proc/net/ipconfig/ntp_servers to an NTP client before mounting the real
>    root filesystem if it is on NFS).
>  
> +ip.dev_wait_ms=<value>
> +  Set the number of milliseconds to delay after opening the network device
> +  which will be autoconfigured. Defaults to 10 milliseconds.
>  
>  nfsrootdebug
>    This parameter enables debugging messages to appear in the kernel
> diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
> index c56b6fe6f0d7..cbf35163b973 100644
> --- a/net/ipv4/ipconfig.c
> +++ b/net/ipv4/ipconfig.c
> @@ -82,8 +82,6 @@
>  #define IPCONFIG_DYNAMIC
>  #endif
>  
> -/* Define the friendly delay before and after opening net devices */
> -#define CONF_POST_OPEN		10	/* After opening: 10 msecs */
>  
>  /* Define the timeout for waiting for a DHCP/BOOTP/RARP reply */
>  #define CONF_OPEN_RETRIES 	2	/* (Re)open devices twice */
> @@ -101,6 +99,7 @@
>  
>  /* Wait for carrier timeout default in seconds */
>  static unsigned int carrier_timeout = 120;
> +static unsigned int dev_wait_ms = 10;
>  
>  /*
>   * Public IP configuration
> @@ -1516,7 +1515,8 @@ static int __init ip_auto_config(void)
>  		return err;
>  
>  	/* Give drivers a chance to settle */
> -	msleep(CONF_POST_OPEN);
> +	if(dev_wait_ms > 0)
> +		msleep(dev_wait_ms);

What's the point to wait more than CONF_POST_OPEN with the change?

>  
>  	/*
>  	 * If the config information is insufficient (e.g., our IP address or
> @@ -1849,3 +1849,20 @@ static int __init set_carrier_timeout(char *str)
>  	return 1;
>  }
>  __setup("carrier_timeout=", set_carrier_timeout);
> +
> +
> +static int __init set_dev_wait_ms(char *str)
> +{
> +	ssize_t ret;
> +
> +	if (!str)
> +		return 0;
> +
> +	ret = kstrtouint(str, 0, &dev_wait_ms);
> +	if (ret)
> +		return 0;
> +
> +	return 1;
> +}
> +
> +__setup("ip.dev_wait_ms=", set_dev_wait_ms);
  

On 2/8/24 11:26, Denis Kirjanov wrote:
>
> On 2/8/24 12:52, David Ventura wrote:
>> During IP auto configuration, some drivers apparently need to wait a
>> certain length of time to settle; as this is not true for all drivers,
>> make this length of time configurable.
>>
>> Signed-off-by: David Ventura <david@davidv.dev>
>> ---
>>   .../admin-guide/kernel-parameters.txt         |  4 ++++
>>   Documentation/admin-guide/nfs/nfsroot.rst     |  3 +++
>>   net/ipv4/ipconfig.c                           | 23 ++++++++++++++++---
>>   3 files changed, 27 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>> index b47940577c10..b07a035642fa 100644
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -2291,6 +2291,10 @@
>>   
>>   	ip=		[IP_PNP]
>>   			See Documentation/admin-guide/nfs/nfsroot.rst.
>> +	ip.dev_wait_ms=
>> +			[IP_PNP]
>> +			See Documentation/admin-guide/nfs/nfsroot.rst.
>> +
>>   
>>   	ipcmni_extend	[KNL,EARLY] Extend the maximum number of unique System V
>>   			IPC identifiers from 32,768 to 16,777,216.
>> diff --git a/Documentation/admin-guide/nfs/nfsroot.rst b/Documentation/admin-guide/nfs/nfsroot.rst
>> index 135218f33394..f26f7a342af6 100644
>> --- a/Documentation/admin-guide/nfs/nfsroot.rst
>> +++ b/Documentation/admin-guide/nfs/nfsroot.rst
>> @@ -223,6 +223,9 @@ ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns
>>     /proc/net/ipconfig/ntp_servers to an NTP client before mounting the real
>>     root filesystem if it is on NFS).
>>   
>> +ip.dev_wait_ms=<value>
>> +  Set the number of milliseconds to delay after opening the network device
>> +  which will be autoconfigured. Defaults to 10 milliseconds.
>>   
>>   nfsrootdebug
>>     This parameter enables debugging messages to appear in the kernel
>> diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
>> index c56b6fe6f0d7..cbf35163b973 100644
>> --- a/net/ipv4/ipconfig.c
>> +++ b/net/ipv4/ipconfig.c
>> @@ -82,8 +82,6 @@
>>   #define IPCONFIG_DYNAMIC
>>   #endif
>>   
>> -/* Define the friendly delay before and after opening net devices */
>> -#define CONF_POST_OPEN		10	/* After opening: 10 msecs */
>>   
>>   /* Define the timeout for waiting for a DHCP/BOOTP/RARP reply */
>>   #define CONF_OPEN_RETRIES 	2	/* (Re)open devices twice */
>> @@ -101,6 +99,7 @@
>>   
>>   /* Wait for carrier timeout default in seconds */
>>   static unsigned int carrier_timeout = 120;
>> +static unsigned int dev_wait_ms = 10;
>>   
>>   /*
>>    * Public IP configuration
>> @@ -1516,7 +1515,8 @@ static int __init ip_auto_config(void)
>>   		return err;
>>   
>>   	/* Give drivers a chance to settle */
>> -	msleep(CONF_POST_OPEN);
>> +	if(dev_wait_ms > 0)
>> +		msleep(dev_wait_ms);
> What's the point to wait more than CONF_POST_OPEN with the change?

I didn't have anything specificin mind - shall I change this

to only enable/disable the 10ms wait instead of allowing a configurable 
value?

>>   
>>   	/*
>>   	 * If the config information is insufficient (e.g., our IP address or
>> @@ -1849,3 +1849,20 @@ static int __init set_carrier_timeout(char *str)
>>   	return 1;
>>   }
>>   __setup("carrier_timeout=", set_carrier_timeout);
>> +
>> +
>> +static int __init set_dev_wait_ms(char *str)
>> +{
>> +	ssize_t ret;
>> +
>> +	if (!str)
>> +		return 0;
>> +
>> +	ret = kstrtouint(str, 0, &dev_wait_ms);
>> +	if (ret)
>> +		return 0;
>> +
>> +	return 1;
>> +}
>> +
>> +__setup("ip.dev_wait_ms=", set_dev_wait_ms);
  

On Thu,  8 Feb 2024 10:52:29 +0100 David Ventura wrote:
> During IP auto configuration, some drivers apparently need to wait a
> certain length of time to settle; as this is not true for all drivers,
> make this length of time configurable.

Please CC folks who gave you feedback, Andrew's is missing.

Andrew, what do you think about just removing the wait?
Or decreasing it to 1ms?
It feels a little wasteful to be adding uAPI for something
which as you said is likely papering over ancient bugs. We'll 
fix the bugs which are still around and the uAPI will stay 
forever :(
  

On Fri, Feb 09, 2024 at 01:59:44PM -0800, Jakub Kicinski wrote:
> On Thu,  8 Feb 2024 10:52:29 +0100 David Ventura wrote:
> > During IP auto configuration, some drivers apparently need to wait a
> > certain length of time to settle; as this is not true for all drivers,
> > make this length of time configurable.
> 
> Please CC folks who gave you feedback, Andrew's is missing.
> 
> Andrew, what do you think about just removing the wait?
> Or decreasing it to 1ms?
> It feels a little wasteful to be adding uAPI for something
> which as you said is likely papering over ancient bugs. We'll 
> fix the bugs which are still around and the uAPI will stay 
> forever :(

My guess is, the broken drivers are doing setup stuff after they call
netdev_register().

Reducing it to 1ms will probably continue to hide such bugs. So we
could just go with that, and probably not see any regressions. Or we
can decide we really do want to know about broken drivers, and just
remove the delay.

Either way, we don't need a new uAPI.

David, is 1ms too long for you? If we do take the delay out, you are
going to receive some of the flack from regression reports.

      Andrew
  

On 2/9/24 23:11, Andrew Lunn wrote:
> On Fri, Feb 09, 2024 at 01:59:44PM -0800, Jakub Kicinski wrote:
>> On Thu,  8 Feb 2024 10:52:29 +0100 David Ventura wrote:
>>> During IP auto configuration, some drivers apparently need to wait a
>>> certain length of time to settle; as this is not true for all drivers,
>>> make this length of time configurable.
>> Please CC folks who gave you feedback, Andrew's is missing.
Thanks for the feedback, still learning this workflow
>>
>> Andrew, what do you think about just removing the wait?
>> Or decreasing it to 1ms?
>> It feels a little wasteful to be adding uAPI for something
>> which as you said is likely papering over ancient bugs. We'll
>> fix the bugs which are still around and the uAPI will stay
>> forever :(
> My guess is, the broken drivers are doing setup stuff after they call
> netdev_register().
>
> Reducing it to 1ms will probably continue to hide such bugs. So we
> could just go with that, and probably not see any regressions. Or we
> can decide we really do want to know about broken drivers, and just
> remove the delay.
>
> Either way, we don't need a new uAPI.

Would it make sense to move this to a build-time configuration flag?

I do not have a gut-feeling for which behaviors should be configurable

at build vs run time.

> David, is 1ms too long for you? If we do take the delay out, you are
> going to receive some of the flack from regression reports.

I've used this patch to experiment with different values, and the sleep time

behaves as described in Documentation/timers/timers-howto.rst, that is, a

call of `msleep(1)` usually delays boot time by 12~13 ms in my tests.

On top of this, I'm running this specific example on a no-smp systemm, 
where I

do not believe the `msleep` achieves anything (but I'm a kernel newbie 
so please

correct me if I'm wrong).

>
>        Andrew
>
  

> Would it make sense to move this to a build-time configuration flag?
> 
> I do not have a gut-feeling for which behaviors should be configurable
> 
> at build vs run time.

If it is build time, it becomes the distribution problem to pick a
value.

Might be best to just bite the bullet, set it to 0, and fixup whatever
breaks.

	Andrew