[RFC,v10,4/5] iommu/vt-d: don't issue ATS Invalidation request when device is disconnected
Message ID | 20231228170504.720794-2-haifeng.zhao@linux.intel.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp2119421dyb; Thu, 28 Dec 2023 09:09:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFQ1V2FSl6ji6Z0VJRIE+XSk+lgFwMfmT3A+9xF0UGr8iQax0ZFf8ta+RpuvKiXFAB8I+99 X-Received: by 2002:a17:90b:f07:b0:28c:2678:928e with SMTP id br7-20020a17090b0f0700b0028c2678928emr10337604pjb.25.1703783369425; Thu, 28 Dec 2023 09:09:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703783369; cv=none; d=google.com; s=arc-20160816; b=jCAd0cN7MZs8KKrjwzcxNma6yuoROxRu7/KyNa0Eh6oC4pl03Sfvk+Eu/BwcvDhP+b eu8ycE3/+4ImIHNRMuamO5C+DG+c4lyY3F+yPhqnrgfTqIPB2wVOozTqoU4eDirclmUo b+vSVuwCVhB2H3WHuMyoIf8+cw9RU45RjzRHQYacCmcSoaiV2shxIthEQVEFBIHWwf1H fMY6Pf3mhN+w5FLA4M6GyPmyiRGTirotOJDJQCVwnCZZBU429V79PK5F6g4i4pu0b/O3 gpGHxCAmHGKQDQIRh+OHKy20HuweJN2vCicqPLeqEDEWO9oOWRP40XkC5GRej8/2TOoa N3VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=0KbdgSt+1Ns2g+dlgMfVT6kEYwmbpTwTEJjlZ1qPv84=; fh=dlWkbAla6c7cXCen1OD7/w45BfQUKF45lw88dDQPBUc=; b=OEYV9bo5WXm9K61LhIBCyRHlSxhvzT3x+ghask8KcWB7G9GJZUWWHWFB7uw45p8vpy 8OSV0BIlE9uzqRDN0vchfjUFEU9xA/xSZhHYl0WkiYLXQlKpx5IbY8FJYE6G6/i0xHDx agpIfB6t6krWZclfU18EpN6EOfXy2NPhD7xOAGbl3keVf6eQus1c/FbxXwaEKjrgdlJK ElNRlxOhZU8d5XIEQXWC2o68QqFssWPAIhO2XI+EDcadm9ayAx7kxv2MYOu6++j2VWTK E9jLuu6gOHLhwZ+f8Ik3o33ZiQ1L59j1mXdQwQcMntahlb5ioUJFwhCM+Pcdq3+I3ryH SZtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=hftXHG8b; spf=pass (google.com: domain of linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id t11-20020a17090abc4b00b0028bf7ab5a92si12396547pjv.147.2023.12.28.09.09.28 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Dec 2023 09:09:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=hftXHG8b; spf=pass (google.com: domain of linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-12802-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id BFBB2B23E0C for <ouuuleilei@gmail.com>; Thu, 28 Dec 2023 17:05:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9792410799; Thu, 28 Dec 2023 17:05:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="hftXHG8b" X-Original-To: linux-kernel@vger.kernel.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0F9CF9D7; Thu, 28 Dec 2023 17:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1703783114; x=1735319114; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jlJA6l9/LxpjxQJJdNKaR0breVRQ2CVmXJiAZTX9pxk=; b=hftXHG8bJF8GVA/joG5fRsoTyXfHX6WCq1ckjlhodRXeobv9s4MKT+jm sHrFeAj3Ho+dfdn0cYWcpTaVLEfBhxLXQVov+mg3jRP3IUzsZ0Z0XXf1J HyYq7if6XR8GblY/lS1ZYrLoQ7oSDVJ9QlC/I95UZ3mHvWEGeCF6ijyDY Z2j2mCRy2aTqk/xF2ws6UhfcXF2jO1OYwiLM5MXA/KMbhyXOdPBDfRnEx +lAgNLb+jLzXDeOfzmGlatExGfMMVZnllM79FFQh8LMTPd46c0oe6H8zK 5H4YX5BMbEr5RrbGIf/ygQBV329Ts+NoW1PSFJXREyC+3c5Tih5AxGI4k Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10937"; a="10123461" X-IronPort-AV: E=Sophos;i="6.04,312,1695711600"; d="scan'208";a="10123461" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Dec 2023 09:05:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10937"; a="848992471" X-IronPort-AV: E=Sophos;i="6.04,312,1695711600"; d="scan'208";a="848992471" Received: from ply01-vm-store.bj.intel.com ([10.238.153.201]) by fmsmga004.fm.intel.com with ESMTP; 28 Dec 2023 09:05:10 -0800 From: Ethan Zhao <haifeng.zhao@linux.intel.com> To: kevin.tian@intel.com, bhelgaas@google.com, baolu.lu@linux.intel.com, dwmw2@infradead.org, will@kernel.org, robin.murphy@arm.com, lukas@wunner.de Cc: linux-pci@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [RFC PATCH v10 4/5] iommu/vt-d: don't issue ATS Invalidation request when device is disconnected Date: Thu, 28 Dec 2023 12:05:03 -0500 Message-Id: <20231228170504.720794-2-haifeng.zhao@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231228170504.720794-1-haifeng.zhao@linux.intel.com> References: <20231228170504.720794-1-haifeng.zhao@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: <linux-kernel.vger.kernel.org> List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org> List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786546350555999786 X-GMAIL-MSGID: 1786546350555999786 |
Series |
fix vt-d hard lockup when hotplug ATS capable device
|
|
Commit Message
Ethan Zhao
Dec. 28, 2023, 5:05 p.m. UTC
Except those aggressive hotplug cases - surprise remove a hotplug device while its safe removal is requested and handled in process by: 1. pull it out directly. 2. turn off its power. 3. bring the link down. 4. just died there that moment. etc, in a word, 'gone' or 'disconnected'. Mostly are regular normal safe removal and surprise removal unplug. these hot unplug handling process could be optimized for fix the ATS Invalidation hang issue by calling pci_dev_is_disconnected() in function devtlb_invalidation_with_pasid() to check target device state to avoid sending meaningless ATS Invalidation request to iommu when device is gone. (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1) For safe removal, device wouldn't be removed untill the whole software handling process is done, it wouldn't trigger the hard lock up issue caused by too long ATS Invalidation timeout wait. In safe removal path, device state isn't set to pci_channel_io_perm_failure in pciehp_unconfigure_device() by checking 'presence' parameter, calling pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will return false there, wouldn't break the function. For surprise removal, device state is set to pci_channel_io_perm_failure in pciehp_unconfigure_device(), means device is already gone (disconnected) call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will return true to break the function not to send ATS Invalidation request to the disconnected device blindly, thus avoid the further long time waiting triggers the hard lockup. safe removal & surprise removal pciehp_ist() pciehp_handle_presence_or_link_change() pciehp_disable_slot() remove_board() pciehp_unconfigure_device(presence) Tested-by: Haorong Ye <yehaorong@bytedance.com> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com> --- drivers/iommu/intel/pasid.c | 2 ++ 1 file changed, 2 insertions(+)
Comments
On 12/29/23 1:05 AM, Ethan Zhao wrote: > Except those aggressive hotplug cases - surprise remove a hotplug device > while its safe removal is requested and handled in process by: > > 1. pull it out directly. > 2. turn off its power. > 3. bring the link down. > 4. just died there that moment. > > etc, in a word, 'gone' or 'disconnected'. > > Mostly are regular normal safe removal and surprise removal unplug. > these hot unplug handling process could be optimized for fix the ATS > Invalidation hang issue by calling pci_dev_is_disconnected() in function > devtlb_invalidation_with_pasid() to check target device state to avoid > sending meaningless ATS Invalidation request to iommu when device is gone. > (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1) > > For safe removal, device wouldn't be removed untill the whole software > handling process is done, it wouldn't trigger the hard lock up issue > caused by too long ATS Invalidation timeout wait. In safe removal path, > device state isn't set to pci_channel_io_perm_failure in > pciehp_unconfigure_device() by checking 'presence' parameter, calling > pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will return > false there, wouldn't break the function. > > For surprise removal, device state is set to pci_channel_io_perm_failure in > pciehp_unconfigure_device(), means device is already gone (disconnected) > call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will > return true to break the function not to send ATS Invalidation request to > the disconnected device blindly, thus avoid the further long time waiting > triggers the hard lockup. > > safe removal & surprise removal > > pciehp_ist() > pciehp_handle_presence_or_link_change() > pciehp_disable_slot() > remove_board() > pciehp_unconfigure_device(presence) > > Tested-by: Haorong Ye <yehaorong@bytedance.com> > Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com> > --- > drivers/iommu/intel/pasid.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c > index 715943531091..3d5ed27f39ef 100644 > --- a/drivers/iommu/intel/pasid.c > +++ b/drivers/iommu/intel/pasid.c > @@ -480,6 +480,8 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu, > if (!info || !info->ats_enabled) > return; > > + if (pci_dev_is_disconnected(to_pci_dev(dev))) > + return; Why do you need the above after changes in PATCH 2/5? It's unnecessary and not complete. We have other places where device TLB invalidation is issued, right? > /* > * When PASID 0 is used, it indicates RID2PASID(DMA request w/o PASID), > * devTLB flush w/o PASID should be used. For non-zero PASID under Best regards, baolu
On 1/10/2024 1:24 PM, Baolu Lu wrote: > On 12/29/23 1:05 AM, Ethan Zhao wrote: >> Except those aggressive hotplug cases - surprise remove a hotplug device >> while its safe removal is requested and handled in process by: >> >> 1. pull it out directly. >> 2. turn off its power. >> 3. bring the link down. >> 4. just died there that moment. >> >> etc, in a word, 'gone' or 'disconnected'. >> >> Mostly are regular normal safe removal and surprise removal unplug. >> these hot unplug handling process could be optimized for fix the ATS >> Invalidation hang issue by calling pci_dev_is_disconnected() in function >> devtlb_invalidation_with_pasid() to check target device state to avoid >> sending meaningless ATS Invalidation request to iommu when device is >> gone. >> (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1) >> >> For safe removal, device wouldn't be removed untill the whole software >> handling process is done, it wouldn't trigger the hard lock up issue >> caused by too long ATS Invalidation timeout wait. In safe removal path, >> device state isn't set to pci_channel_io_perm_failure in >> pciehp_unconfigure_device() by checking 'presence' parameter, calling >> pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will >> return >> false there, wouldn't break the function. >> >> For surprise removal, device state is set to >> pci_channel_io_perm_failure in >> pciehp_unconfigure_device(), means device is already gone (disconnected) >> call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will >> return true to break the function not to send ATS Invalidation >> request to >> the disconnected device blindly, thus avoid the further long time >> waiting >> triggers the hard lockup. >> >> safe removal & surprise removal >> >> pciehp_ist() >> pciehp_handle_presence_or_link_change() >> pciehp_disable_slot() >> remove_board() >> pciehp_unconfigure_device(presence) >> >> Tested-by: Haorong Ye <yehaorong@bytedance.com> >> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com> >> --- >> drivers/iommu/intel/pasid.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c >> index 715943531091..3d5ed27f39ef 100644 >> --- a/drivers/iommu/intel/pasid.c >> +++ b/drivers/iommu/intel/pasid.c >> @@ -480,6 +480,8 @@ devtlb_invalidation_with_pasid(struct intel_iommu >> *iommu, >> if (!info || !info->ats_enabled) >> return; >> + if (pci_dev_is_disconnected(to_pci_dev(dev))) >> + return; > > Why do you need the above after changes in PATCH 2/5? It's unnecessary > and not complete. We have other places where device TLB invalidation is > issued, right? This one could be regarded as optimization, no need to trapped into rabbit hole if we could predict the result. because the bad thing is we don't know what response to us in the rabbit hole from third party switch (bridges will feedback timeout to requester as PCIe spec mentioned if the endpoint is gone). Thanks, Ethan > >> /* >> * When PASID 0 is used, it indicates RID2PASID(DMA request w/o >> PASID), >> * devTLB flush w/o PASID should be used. For non-zero PASID under > > Best regards, > baolu
On 1/10/24 4:37 PM, Ethan Zhao wrote: > > On 1/10/2024 1:24 PM, Baolu Lu wrote: >> On 12/29/23 1:05 AM, Ethan Zhao wrote: >>> Except those aggressive hotplug cases - surprise remove a hotplug device >>> while its safe removal is requested and handled in process by: >>> >>> 1. pull it out directly. >>> 2. turn off its power. >>> 3. bring the link down. >>> 4. just died there that moment. >>> >>> etc, in a word, 'gone' or 'disconnected'. >>> >>> Mostly are regular normal safe removal and surprise removal unplug. >>> these hot unplug handling process could be optimized for fix the ATS >>> Invalidation hang issue by calling pci_dev_is_disconnected() in function >>> devtlb_invalidation_with_pasid() to check target device state to avoid >>> sending meaningless ATS Invalidation request to iommu when device is >>> gone. >>> (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1) >>> >>> For safe removal, device wouldn't be removed untill the whole software >>> handling process is done, it wouldn't trigger the hard lock up issue >>> caused by too long ATS Invalidation timeout wait. In safe removal path, >>> device state isn't set to pci_channel_io_perm_failure in >>> pciehp_unconfigure_device() by checking 'presence' parameter, calling >>> pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will >>> return >>> false there, wouldn't break the function. >>> >>> For surprise removal, device state is set to >>> pci_channel_io_perm_failure in >>> pciehp_unconfigure_device(), means device is already gone (disconnected) >>> call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will >>> return true to break the function not to send ATS Invalidation >>> request to >>> the disconnected device blindly, thus avoid the further long time >>> waiting >>> triggers the hard lockup. >>> >>> safe removal & surprise removal >>> >>> pciehp_ist() >>> pciehp_handle_presence_or_link_change() >>> pciehp_disable_slot() >>> remove_board() >>> pciehp_unconfigure_device(presence) >>> >>> Tested-by: Haorong Ye <yehaorong@bytedance.com> >>> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com> >>> --- >>> drivers/iommu/intel/pasid.c | 2 ++ >>> 1 file changed, 2 insertions(+) >>> >>> diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c >>> index 715943531091..3d5ed27f39ef 100644 >>> --- a/drivers/iommu/intel/pasid.c >>> +++ b/drivers/iommu/intel/pasid.c >>> @@ -480,6 +480,8 @@ devtlb_invalidation_with_pasid(struct intel_iommu >>> *iommu, >>> if (!info || !info->ats_enabled) >>> return; >>> + if (pci_dev_is_disconnected(to_pci_dev(dev))) >>> + return; >> >> Why do you need the above after changes in PATCH 2/5? It's unnecessary >> and not complete. We have other places where device TLB invalidation is >> issued, right? > > This one could be regarded as optimization, no need to trapped into rabbit > > hole if we could predict the result. because the bad thing is we don't know > > what response to us in the rabbit hole from third party switch (bridges > will > > feedback timeout to requester as PCIe spec mentioned if the endpoint is > > gone). The IOMMU hardware has its own timeout mechanism. This timeout might happen if: 1) The link to the endpoint is broken, so the invalidation completion message is lost on the way. 2) The device has a longer timeout value, so the device is still busy with handling the cache invalidation when IOMMU's timeout is triggered. Here, we are doing the following: For Case 1, we return -ETIMEDOUT directly. For Case 2, we attempt to retry. Best regards, baolu
On 1/11/2024 10:24 AM, Baolu Lu wrote: > On 1/10/24 4:37 PM, Ethan Zhao wrote: >> >> On 1/10/2024 1:24 PM, Baolu Lu wrote: >>> On 12/29/23 1:05 AM, Ethan Zhao wrote: >>>> Except those aggressive hotplug cases - surprise remove a hotplug >>>> device >>>> while its safe removal is requested and handled in process by: >>>> >>>> 1. pull it out directly. >>>> 2. turn off its power. >>>> 3. bring the link down. >>>> 4. just died there that moment. >>>> >>>> etc, in a word, 'gone' or 'disconnected'. >>>> >>>> Mostly are regular normal safe removal and surprise removal unplug. >>>> these hot unplug handling process could be optimized for fix the ATS >>>> Invalidation hang issue by calling pci_dev_is_disconnected() in >>>> function >>>> devtlb_invalidation_with_pasid() to check target device state to avoid >>>> sending meaningless ATS Invalidation request to iommu when device >>>> is gone. >>>> (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1) >>>> >>>> For safe removal, device wouldn't be removed untill the whole software >>>> handling process is done, it wouldn't trigger the hard lock up issue >>>> caused by too long ATS Invalidation timeout wait. In safe removal >>>> path, >>>> device state isn't set to pci_channel_io_perm_failure in >>>> pciehp_unconfigure_device() by checking 'presence' parameter, calling >>>> pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will >>>> return >>>> false there, wouldn't break the function. >>>> >>>> For surprise removal, device state is set to >>>> pci_channel_io_perm_failure in >>>> pciehp_unconfigure_device(), means device is already gone >>>> (disconnected) >>>> call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() >>>> will >>>> return true to break the function not to send ATS Invalidation >>>> request to >>>> the disconnected device blindly, thus avoid the further long time >>>> waiting >>>> triggers the hard lockup. >>>> >>>> safe removal & surprise removal >>>> >>>> pciehp_ist() >>>> pciehp_handle_presence_or_link_change() >>>> pciehp_disable_slot() >>>> remove_board() >>>> pciehp_unconfigure_device(presence) >>>> >>>> Tested-by: Haorong Ye <yehaorong@bytedance.com> >>>> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com> >>>> --- >>>> drivers/iommu/intel/pasid.c | 2 ++ >>>> 1 file changed, 2 insertions(+) >>>> >>>> diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c >>>> index 715943531091..3d5ed27f39ef 100644 >>>> --- a/drivers/iommu/intel/pasid.c >>>> +++ b/drivers/iommu/intel/pasid.c >>>> @@ -480,6 +480,8 @@ devtlb_invalidation_with_pasid(struct >>>> intel_iommu *iommu, >>>> if (!info || !info->ats_enabled) >>>> return; >>>> + if (pci_dev_is_disconnected(to_pci_dev(dev))) >>>> + return; >>> >>> Why do you need the above after changes in PATCH 2/5? It's unnecessary >>> and not complete. We have other places where device TLB invalidation is >>> issued, right? >> >> This one could be regarded as optimization, no need to trapped into >> rabbit >> >> hole if we could predict the result. because the bad thing is we >> don't know >> >> what response to us in the rabbit hole from third party switch >> (bridges will >> >> feedback timeout to requester as PCIe spec mentioned if the endpoint is >> >> gone). > > The IOMMU hardware has its own timeout mechanism. This timeout might > happen if: > > 1) The link to the endpoint is broken, so the invalidation completion > message is lost on the way. > 2) The device has a longer timeout value, so the device is still busy > with handling the cache invalidation when IOMMU's timeout is > triggered. > > Here, we are doing the following: > > For Case 1, we return -ETIMEDOUT directly. For Case 2, we attempt to > retry. Yes, Intel VT-d will setup a hardware timer if devtlb invalidation issued and wait descripton submitted, that hardware timer is limited resource, will tick till gets the timeout if the endpoint is dead/broken etc. even we bail out in qi_submit_sync() for case #1, the hardware timer still ticks there, if many of such request issued, the iommu will run out of hardware resouce. so we should avoid such case as possible as we could. though the Intel VT-d says the timeout value will not more than " PCIe read timeout", but in fact, we got more than 12 seconds before get ITE. for case #2, the retry has pre-conditon as I know, there is fault, cleared. So I call it "rabbit hole". To run into that rabbit hole is last choice, not best. Thanks, Ethan > > Best regards, > baolu
diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c index 715943531091..3d5ed27f39ef 100644 --- a/drivers/iommu/intel/pasid.c +++ b/drivers/iommu/intel/pasid.c @@ -480,6 +480,8 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu, if (!info || !info->ats_enabled) return; + if (pci_dev_is_disconnected(to_pci_dev(dev))) + return; /* * When PASID 0 is used, it indicates RID2PASID(DMA request w/o PASID), * devTLB flush w/o PASID should be used. For non-zero PASID under