diff mbox series

[v3,6/6] PCI: hv: Use async probing to reduce boot time

Message ID 20230420024037.5921-7-decui@microsoft.com
State New
Headers
Series pci-hyper: Fix race condition bugs for fast device hotplug |

Commit Message

Dexuan Cui April 20, 2023, 2:40 a.m. UTC 
  Commit 414428c5da1c ("PCI: hv: Lock PCI bus on device eject") added
pci_lock_rescan_remove() and pci_unlock_rescan_remove() in
create_root_hv_pci_bus() and in hv_eject_device_work() to address the
race between create_root_hv_pci_bus() and hv_eject_device_work(), but it
turns that grabing the pci_rescan_remove_lock mutex is not enough:
refer to the earlier fix "PCI: hv: Add a per-bus mutex state_lock".

Now with hbus->state_lock and other fixes, the race is resolved, so
remove pci_{lock,unlock}_rescan_remove() in create_root_hv_pci_bus():
this removes the serialization in hv_pci_probe() and hence allows
async-probing (PROBE_PREFER_ASYNCHRONOUS) to work.

Add the async-probing flag to hv_pci_drv.

pci_{lock,unlock}_rescan_remove() in hv_eject_device_work() and in
hv_pci_remove() are still kept: according to the comment before
drivers/pci/probe.c: static DEFINE_MUTEX(pci_rescan_remove_lock),
"PCI device removal routines should always be executed under this mutex".

Signed-off-by: Dexuan Cui <decui@microsoft.com>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
Reviewed-by: Long Li <longli@microsoft.com>
Cc: stable@vger.kernel.org
---

v2:
  No change to the patch body.
  Improved the commit message [Michael Kelley]
  Added Cc:stable

v3:
  Added Michael's and Long Li's Reviewed-by.
  Fixed a typo in the commit message: grubing -> grabing [Thanks, Michael!]

 drivers/pci/controller/pci-hyperv.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

Comments

Simon Horman April 23, 2023, 7:11 p.m. UTC | #1 
On Wed, Apr 19, 2023 at 07:40:37PM -0700, Dexuan Cui wrote:
> Commit 414428c5da1c ("PCI: hv: Lock PCI bus on device eject") added
> pci_lock_rescan_remove() and pci_unlock_rescan_remove() in
> create_root_hv_pci_bus() and in hv_eject_device_work() to address the
> race between create_root_hv_pci_bus() and hv_eject_device_work(), but it
> turns that grabing the pci_rescan_remove_lock mutex is not enough:

nit: s/grabing/grabbing/g

> refer to the earlier fix "PCI: hv: Add a per-bus mutex state_lock".

...
Dexuan Cui April 24, 2023, 8:50 p.m. UTC | #2 
> From: Simon Horman <simon.horman@corigine.com>
> Sent: Sunday, April 23, 2023 12:11 PM
> To: Dexuan Cui <decui@microsoft.com>
> ...
> On Wed, Apr 19, 2023 at 07:40:37PM -0700, Dexuan Cui wrote:
> > Commit 414428c5da1c ("PCI: hv: Lock PCI bus on device eject") added
> > pci_lock_rescan_remove() and pci_unlock_rescan_remove() in
> > create_root_hv_pci_bus() and in hv_eject_device_work() to address the
> > race between create_root_hv_pci_bus() and hv_eject_device_work(), but it
> > turns that grabing the pci_rescan_remove_lock mutex is not enough:
> 
> nit: s/grabing/grabbing/g
Thanks for spotting this! I suppose the maintainer(s) would help fix this.
Lorenzo Pieralisi May 10, 2023, 8:23 a.m. UTC | #3 
On Wed, Apr 19, 2023 at 07:40:37PM -0700, Dexuan Cui wrote:
> Commit 414428c5da1c ("PCI: hv: Lock PCI bus on device eject") added
> pci_lock_rescan_remove() and pci_unlock_rescan_remove() in
> create_root_hv_pci_bus() and in hv_eject_device_work() to address the
> race between create_root_hv_pci_bus() and hv_eject_device_work(), but it
> turns that grabing the pci_rescan_remove_lock mutex is not enough:
> refer to the earlier fix "PCI: hv: Add a per-bus mutex state_lock".

This is meaningless for a commit log reader, there is nothing to
refer to.

> Now with hbus->state_lock and other fixes, the race is resolved, so

"other fixes" is meaningless too.

Explain the problem and how you fix it (this patch should be split
because the Subject does not represent what you are doing precisely,
see below).

> remove pci_{lock,unlock}_rescan_remove() in create_root_hv_pci_bus():
> this removes the serialization in hv_pci_probe() and hence allows
> async-probing (PROBE_PREFER_ASYNCHRONOUS) to work.
> 
> Add the async-probing flag to hv_pci_drv.

Adding the asynchronous probing should be a separate patch and
I don't think you should send it to stable kernels straight away
because a) it is not a fix b) it can trigger further regressions.

> pci_{lock,unlock}_rescan_remove() in hv_eject_device_work() and in
> hv_pci_remove() are still kept: according to the comment before
> drivers/pci/probe.c: static DEFINE_MUTEX(pci_rescan_remove_lock),
> "PCI device removal routines should always be executed under this mutex".

This patch should be split, first thing is to fix and document what
you are changing for pci_{lock,unlock}_rescan_remove() then add
asynchronous probing.

Lorenzo

> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Reviewed-by: Michael Kelley <mikelley@microsoft.com>
> Reviewed-by: Long Li <longli@microsoft.com>
> Cc: stable@vger.kernel.org
> ---
> 
> v2:
>   No change to the patch body.
>   Improved the commit message [Michael Kelley]
>   Added Cc:stable
> 
> v3:
>   Added Michael's and Long Li's Reviewed-by.
>   Fixed a typo in the commit message: grubing -> grabing [Thanks, Michael!]
> 
>  drivers/pci/controller/pci-hyperv.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index 3ae2f99dea8c2..2ea2b1b8a4c9a 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -2312,12 +2312,16 @@ static int create_root_hv_pci_bus(struct hv_pcibus_device *hbus)
>  	if (error)
>  		return error;
>  
> -	pci_lock_rescan_remove();
> +	/*
> +	 * pci_lock_rescan_remove() and pci_unlock_rescan_remove() are
> +	 * unnecessary here, because we hold the hbus->state_lock, meaning
> +	 * hv_eject_device_work() and pci_devices_present_work() can't race
> +	 * with create_root_hv_pci_bus().
> +	 */
>  	hv_pci_assign_numa_node(hbus);
>  	pci_bus_assign_resources(bridge->bus);
>  	hv_pci_assign_slots(hbus);
>  	pci_bus_add_devices(bridge->bus);
> -	pci_unlock_rescan_remove();
>  	hbus->state = hv_pcibus_installed;
>  	return 0;
>  }
> @@ -4003,6 +4007,9 @@ static struct hv_driver hv_pci_drv = {
>  	.remove		= hv_pci_remove,
>  	.suspend	= hv_pci_suspend,
>  	.resume		= hv_pci_resume,
> +	.driver = {
> +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> +	},
>  };
>  
>  static void __exit exit_hv_pci_drv(void)
> -- 
> 2.25.1
>
Dexuan Cui May 10, 2023, 5:12 p.m. UTC | #4 
> From: Lorenzo Pieralisi <lpieralisi@kernel.org>
> Sent: Wednesday, May 10, 2023 1:23 AM
> To: Dexuan Cui <decui@microsoft.com>
> ...
> On Wed, Apr 19, 2023 at 07:40:37PM -0700, Dexuan Cui wrote:
> > Commit 414428c5da1c ("PCI: hv: Lock PCI bus on device eject") added
> > pci_lock_rescan_remove() and pci_unlock_rescan_remove() in
> > create_root_hv_pci_bus() and in hv_eject_device_work() to address the
> > race between create_root_hv_pci_bus() and hv_eject_device_work(), but it
> > turns that grabing the pci_rescan_remove_lock mutex is not enough:
> > refer to the earlier fix "PCI: hv: Add a per-bus mutex state_lock".
> 
> This is meaningless for a commit log reader, there is nothing to
> refer to.
Correct. Because patch 5
[PATCH v3 5/6] PCI: hv: Add a per-bus mutex state_lock
has not been in any upstream tree, so I don't have a commit id yet.
 
> > Now with hbus->state_lock and other fixes, the race is resolved, so
> 
> "other fixes" is meaningless too.
Ditto. 
 
> Explain the problem and how you fix it (this patch should be split
> because the Subject does not represent what you are doing precisely,
> see below).
Ok, I will better explain the boot time issue.

> > remove pci_{lock,unlock}_rescan_remove() in create_root_hv_pci_bus():
> > this removes the serialization in hv_pci_probe() and hence allows
> > async-probing (PROBE_PREFER_ASYNCHRONOUS) to work.
> >
> > Add the async-probing flag to hv_pci_drv.
> 
> Adding the asynchronous probing should be a separate patch and
> I don't think you should send it to stable kernels straight away
> because a) it is not a fix b) it can trigger further regressions.
Agreed. I'll remove the line "Cc: stable".

> > pci_{lock,unlock}_rescan_remove() in hv_eject_device_work() and in
> > hv_pci_remove() are still kept: according to the comment before
> > drivers/pci/probe.c: static DEFINE_MUTEX(pci_rescan_remove_lock),
> > "PCI device removal routines should always be executed under this mutex".
> 
> This patch should be split, first thing is to fix and document what
> you are changing for pci_{lock,unlock}_rescan_remove() then add
> asynchronous probing.
> 
> Lorenzo
Ok, I'll split this patch into two.

Thanks for reviewing the patch. 
Can you please give an "Acked-by" or "Reviewed-by" to patch 1~5 
if they look good to you? The first 5 patches have been there for a
while, and they already got Michael's Reviewed-by. 

I hope the first 5 patches can go through the hyperv-fixes branch in
the hyperv tree
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/log/?h=hyperv-fixes
since they are specific to Hyper-V.

After the first 5 patches are in, I can refer to the commit IDs, and I
will split this patch (patch 6).

Thanks,
Dexuan
Dexuan Cui May 17, 2023, 12:02 a.m. UTC | #5 
> From: Dexuan Cui
> Sent: Wednesday, May 10, 2023 10:12 AM
> To: Lorenzo Pieralisi <lpieralisi@kernel.org>
> > ...
> > This patch should be split, first thing is to fix and document what
> > you are changing for pci_{lock,unlock}_rescan_remove() then add
> > asynchronous probing.
> >
> > Lorenzo
> Ok, I'll split this patch into two.
> 
> Thanks for reviewing the patch.
> Can you please give an "Acked-by" or "Reviewed-by" to patch 1~5
> if they look good to you? The first 5 patches have been there for a
> while, and they already got Michael's Reviewed-by.

Hi Lorenzo, Bjorn and all,
Ping -- it would be great to have your Acked-by or Reviewed-by for
patch 1 to 5.
 
> I hope the first 5 patches can go through the hyperv-fixes branch in
> the hyperv tree
> https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/log/?h=hyp
> erv-fixes
> since they are specific to Hyper-V.
> 
> After the first 5 patches are in, I can refer to the commit IDs, and I
> will split this patch (patch 6).
> 
> Thanks,
> Dexuan
Dexuan Cui May 23, 2023, 7:30 p.m. UTC | #6 
> From: Dexuan Cui <decui@microsoft.com>
> Sent: Tuesday, May 16, 2023 5:03 PM
>  ...
> > From: Dexuan Cui
> > Sent: Wednesday, May 10, 2023 10:12 AM
> > To: Lorenzo Pieralisi <lpieralisi@kernel.org>
> > > ...
> > > This patch should be split, first thing is to fix and document what
> > > you are changing for pci_{lock,unlock}_rescan_remove() then add
> > > asynchronous probing.
> > >
> > > Lorenzo
> > Ok, I'll split this patch into two.
> >
> > Thanks for reviewing the patch.
> > Can you please give an "Acked-by" or "Reviewed-by" to patch 1~5
> > if they look good to you? The first 5 patches have been there for a
> > while, and they already got Michael's Reviewed-by.
>
> Hi Lorenzo, Bjorn and all,
> Ping -- it would be great to have your Acked-by or Reviewed-by for
> patch 1 to 5.

Gentle ping .

> > I hope the first 5 patches can go through the hyperv-fixes branch in
> > the hyperv tree
> >
> https://git.ke/
> rnel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fhyperv%2Flinux.git%2F
> log%2F%3Fh%3Dhyp&data=05%7C01%7Cdecui%40microsoft.com%7C65c86f
> fe8d8542dbae0708db566a1607%7C72f988bf86f141af91ab2d7cd011db47%
> 7C1%7C0%7C638198785892993948%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7C%7C%7C&sdata=h9vxjnGo6oYzare%2FqqcXndg2NZZ0Ap%2BH33q0i
> Mtf7D4%3D&reserved=0
> > erv-fixes
> > since they are specific to Hyper-V.
> >
> > After the first 5 patches are in, I can refer to the commit IDs, and I
> > will split this patch (patch 6).
> >
> > Thanks,
> > Dexuan
diff mbox series

Patch

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index 3ae2f99dea8c2..2ea2b1b8a4c9a 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -2312,12 +2312,16 @@  static int create_root_hv_pci_bus(struct hv_pcibus_device *hbus)
 	if (error)
 		return error;
 
-	pci_lock_rescan_remove();
+	/*
+	 * pci_lock_rescan_remove() and pci_unlock_rescan_remove() are
+	 * unnecessary here, because we hold the hbus->state_lock, meaning
+	 * hv_eject_device_work() and pci_devices_present_work() can't race
+	 * with create_root_hv_pci_bus().
+	 */
 	hv_pci_assign_numa_node(hbus);
 	pci_bus_assign_resources(bridge->bus);
 	hv_pci_assign_slots(hbus);
 	pci_bus_add_devices(bridge->bus);
-	pci_unlock_rescan_remove();
 	hbus->state = hv_pcibus_installed;
 	return 0;
 }
@@ -4003,6 +4007,9 @@  static struct hv_driver hv_pci_drv = {
 	.remove		= hv_pci_remove,
 	.suspend	= hv_pci_suspend,
 	.resume		= hv_pci_resume,
+	.driver = {
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
 };
 
 static void __exit exit_hv_pci_drv(void)