From patchwork Sun Jan 28 23:32:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "David E. Box" X-Patchwork-Id: 193218 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp314551dyb; Sun, 28 Jan 2024 18:02:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IHtjeopucwIXafGS7oWM9ExWnDLKdGLQZMyuhNoV7AVoI2IWdWx4mHuJujT6SKAQJDP3TL3 X-Received: by 2002:a05:6808:16a3:b0:3be:260:6278 with SMTP id bb35-20020a05680816a300b003be02606278mr6096818oib.42.1706493746966; Sun, 28 Jan 2024 18:02:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706493746; cv=pass; d=google.com; s=arc-20160816; b=J+JOniA0I8SL9jpcCVCRJ1kpJx1TjdD4abDlW8tpVSsMT/RQ+N18G/DPLEz1FeINS6 tTLtnCT6Unv84AB5no02iO0TdZ7nSprGUbVbCdSlIVeQIwXd2YvqrXIm4NynPvOXzP9f IteJ0l8Q7vxREkPpVQt76H41VoXmOsfq8iwHM9t7JXKDPxBNsdHZEONgViVbpFabMbvU pHsA6ORzQj0i+pIOCo+bzH9DrZ9qwI3QwPVoR4+G0fDMWNe7ASoIWecudJrLhUXwimVR aL4jjO/gM1YgyCWqdLMs0Nmmthm8tEx9K/7GRv93PiNsFPY+Cl1VIPIfWuFZj9wp3ZpY iEyQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=88plBD05MJRxY2uhHwXRNdzrEUoiglmRM/i+xDyPhK8=; fh=4cLLCZOZ1uAixzSG12P6cLh/7sn/90zAnCRwDJZB7tA=; b=MByUrNCcfXHphoz41aFFSstoqRgBNiH5RRSJyLoMKjZnKkalOBoaLXzO1sRHNA9IXt 1JOkZK130fxQWSJyRjlCOYJrobu4AFkBl3odBc390PWQ7Dhz8yUBcqnGu1bogBcpl/+L 4Jzt3CEp6DdZmsBTZ4rUw0UZFSDRCXO+mCoeSXZYaUGyGcdRsCRbItim3tOZ6R9UwUyf 0I3M5++MZSKHs+PnZxwx0KjQ0Pg2udh1dvLYW3VzEwZpnMSfFaVXlx5BoIZX7U6Oiyc5 TK54ectDjSTqoE8XJi1RJSG3gzUC25JwwLhaIeULt/yjfk+XCTSUl7x78m00N/NW6rjK Pxqg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=VS6fBcMp; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42011-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42011-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id bv124-20020a632e82000000b005d84e5bb276si4116909pgb.885.2024.01.28.18.02.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 18:02:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42011-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=VS6fBcMp; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42011-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42011-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 34819285062 for ; Sun, 28 Jan 2024 23:33:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C40124D599; Sun, 28 Jan 2024 23:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VS6fBcMp" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDD9E2D044; Sun, 28 Jan 2024 23:32:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.31 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484736; cv=none; b=F5UwbU1ghKD04eQKx6JBOMEXZLrNA7jU1RMuSE/eOZh4MaYM4+e0V/q+aN5zl/uV+FlAlVzv+IpKC54ufGxiV0ZhN1KmS6OOU3Dd7tnnJn0DmqLuyKkKSRQJVtXVOvHNPPdXTWRhFgYK/CSs6d6vlkN3ks3M+aAz4QH5zWvqya4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484736; c=relaxed/simple; bh=Gj88eSiw6qEvWoHcGlqiDmWPTw9OEWFwUx0OlE6ZLgE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gmRPOaSc7dYFXUD75wDTg7kLgGZ/A4QS2K0xZvmMKmDv7J7i60qCZkvIGX3bcETuqSEbQblqm+jBzBHEMIS77clcRGFZYoo17Pbllfg0un82L95R4BSJMBIc1s+oyD1nm3pfdCoR1KppamfD9CncF8L85kBYTQn6h6mFYpZB5V4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=VS6fBcMp; arc=none smtp.client-ip=134.134.136.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706484734; x=1738020734; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Gj88eSiw6qEvWoHcGlqiDmWPTw9OEWFwUx0OlE6ZLgE=; b=VS6fBcMpr1XeU3N3/x2th5I2jRtwkquxP265t7ON/HoSp5RC2fQef5FQ ZQ6m4BV4ZXvZfdltwlvUbzKH6dOlrLhLcb2YuzA1ccTQc/9MkEk6+Ddev Tp0rea29tIBFpH7Q2/Xo5RcQjKAwnZrF+i/yTMFhzEOaasb9YaLCjtr+Z ncGP5OIMQS1NPzJ1eUtJPnGxybOzu87SsSpK7LMQ/344+CknMBtc96oIq bOa9cpbS9rwmdyE1lwC3+Q7MwF4cS9SifLFzknlC56LTIpCMVKk7/MQjc xNrGRvBOTGgwJfwxwhl4YDMsYGOqytaK6x7gRLswhURiIiI8pQIpEjg7u w==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="467081093" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="467081093" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="930913770" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="930913770" Received: from linux.intel.com ([10.54.29.200]) by fmsmga001.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:13 -0800 Received: from debox1-desk4.intel.com (sdutta-mobl2.amr.corp.intel.com [10.209.55.52]) by linux.intel.com (Postfix) with ESMTP id 8923558047E; Sun, 28 Jan 2024 15:32:12 -0800 (PST) From: "David E. Box" To: mika.westerberg@linux.intel.com, david.e.box@linux.intel.com, ilpo.jarvinen@linux.intel.com, bhelgaas@google.com, rjw@rjwysocki.net Cc: tasev.stefanoska@skynet.be, enriquezmark36@gmail.com, kernel@witt.link, wse@tuxedocomputers.com, vidyas@nvidia.com, kai.heng.feng@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, ricky_wu@realtek.com, mario.limonciello@amd.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/5] PCI: Always build aspm.c Date: Sun, 28 Jan 2024 15:32:08 -0800 Message-Id: <20240128233212.1139663-2-david.e.box@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240128233212.1139663-1-david.e.box@linux.intel.com> References: <20240128233212.1139663-1-david.e.box@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789388387196852050 X-GMAIL-MSGID: 1789388387196852050 Some ASPM related tasks, such as save and restore of LTR and L1SS capabilities, still need to be performed when CONFIG_PCIASPM=n. To prepare for these changes, wrap the current code in aspm.c with an ifdef and always build. Also move pci_configure_ltr() and pci_bridge_reconfigure_ltr() into aspm.c since they only build when CONFIG_PCIEASPM is set. Suggested-by: Bjorn Helgaas Signed-off-by: David E. Box --- drivers/pci/pci.c | 18 --------- drivers/pci/pci.h | 5 ++- drivers/pci/pcie/Makefile | 2 +- drivers/pci/pcie/aspm.c | 78 +++++++++++++++++++++++++++++++++++++++ drivers/pci/probe.c | 61 ------------------------------ 5 files changed, 83 insertions(+), 81 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index bdbf8a94b4d0..71229ec39e88 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1563,24 +1563,6 @@ static int pci_save_pcie_state(struct pci_dev *dev) return 0; } -void pci_bridge_reconfigure_ltr(struct pci_dev *dev) -{ -#ifdef CONFIG_PCIEASPM - struct pci_dev *bridge; - u32 ctl; - - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) { - pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl); - if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) { - pci_dbg(bridge, "re-enabling LTR\n"); - pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - } - } -#endif -} - static void pci_restore_pcie_state(struct pci_dev *dev) { int i = 0; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index f43873049d52..6771862de921 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -97,7 +97,6 @@ void pci_msi_init(struct pci_dev *dev); void pci_msix_init(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); -void pci_bridge_reconfigure_ltr(struct pci_dev *dev); int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type); static inline void pci_wakeup_event(struct pci_dev *dev) @@ -571,11 +570,15 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); void pcie_aspm_pm_state_change(struct pci_dev *pdev); void pcie_aspm_powersave_config_link(struct pci_dev *pdev); +void pci_configure_ltr(struct pci_dev *pdev); +void pci_bridge_reconfigure_ltr(struct pci_dev *pdev); #else static inline void pcie_aspm_init_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_exit_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_pm_state_change(struct pci_dev *pdev) { } static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { } +static inline void pci_configure_ltr(struct pci_dev *pdev) { } +static inline void pci_bridge_reconfigure_ltr(struct pci_dev *pdev) { } #endif #ifdef CONFIG_PCIE_ECRC diff --git a/drivers/pci/pcie/Makefile b/drivers/pci/pcie/Makefile index 8de4ed5f98f1..6461aa93fe76 100644 --- a/drivers/pci/pcie/Makefile +++ b/drivers/pci/pcie/Makefile @@ -6,7 +6,7 @@ pcieportdrv-y := portdrv.o rcec.o obj-$(CONFIG_PCIEPORTBUS) += pcieportdrv.o -obj-$(CONFIG_PCIEASPM) += aspm.o +obj-y += aspm.o obj-$(CONFIG_PCIEAER) += aer.o err.o obj-$(CONFIG_PCIEAER_INJECT) += aer_inject.o obj-$(CONFIG_PCIE_PME) += pme.o diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 060f4b3c8698..6d077e237a65 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -24,6 +24,8 @@ #include "../pci.h" +#ifdef CONFIG_PCIEASPM + #ifdef MODULE_PARAM_PREFIX #undef MODULE_PARAM_PREFIX #endif @@ -943,6 +945,81 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) up_read(&pci_bus_sem); } +void pci_bridge_reconfigure_ltr(struct pci_dev *pdev) +{ + struct pci_dev *bridge; + u32 ctl; + + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) { + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl); + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) { + pci_dbg(bridge, "re-enabling LTR\n"); + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + } + } +} + +void pci_configure_ltr(struct pci_dev *pdev) +{ + struct pci_host_bridge *host = pci_find_host_bridge(pdev->bus); + struct pci_dev *bridge; + u32 cap, ctl; + + if (!pci_is_pcie(pdev)) + return; + + /* Read L1 PM substate capabilities */ + pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); + + pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &cap); + if (!(cap & PCI_EXP_DEVCAP2_LTR)) + return; + + pcie_capability_read_dword(pdev, PCI_EXP_DEVCTL2, &ctl); + if (ctl & PCI_EXP_DEVCTL2_LTR_EN) { + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT) { + pdev->ltr_path = 1; + return; + } + + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) + pdev->ltr_path = 1; + + return; + } + + if (!host->native_ltr) + return; + + /* + * Software must not enable LTR in an Endpoint unless the Root + * Complex and all intermediate Switches indicate support for LTR. + * PCIe r4.0, sec 6.18. + */ + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT) { + pcie_capability_set_word(pdev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + pdev->ltr_path = 1; + return; + } + + /* + * If we're configuring a hot-added device, LTR was likely + * disabled in the upstream bridge, so re-enable it before enabling + * it in the new device. + */ + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) { + pci_bridge_reconfigure_ltr(pdev); + pcie_capability_set_word(pdev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + pdev->ltr_path = 1; + } +} + /* Recheck latencies and update aspm_capable for links under the root */ static void pcie_update_aspm_capable(struct pcie_link_state *root) { @@ -1447,3 +1524,4 @@ bool pcie_aspm_support_enabled(void) { return aspm_support_enabled; } +#endif /* CONFIG_PCIEASPM */ diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index ed6b7f48736a..0b8c2c9cc9dd 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2159,67 +2159,6 @@ static void pci_configure_relaxed_ordering(struct pci_dev *dev) } } -static void pci_configure_ltr(struct pci_dev *dev) -{ -#ifdef CONFIG_PCIEASPM - struct pci_host_bridge *host = pci_find_host_bridge(dev->bus); - struct pci_dev *bridge; - u32 cap, ctl; - - if (!pci_is_pcie(dev)) - return; - - /* Read L1 PM substate capabilities */ - dev->l1ss = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_L1SS); - - pcie_capability_read_dword(dev, PCI_EXP_DEVCAP2, &cap); - if (!(cap & PCI_EXP_DEVCAP2_LTR)) - return; - - pcie_capability_read_dword(dev, PCI_EXP_DEVCTL2, &ctl); - if (ctl & PCI_EXP_DEVCTL2_LTR_EN) { - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) { - dev->ltr_path = 1; - return; - } - - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) - dev->ltr_path = 1; - - return; - } - - if (!host->native_ltr) - return; - - /* - * Software must not enable LTR in an Endpoint unless the Root - * Complex and all intermediate Switches indicate support for LTR. - * PCIe r4.0, sec 6.18. - */ - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) { - pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - dev->ltr_path = 1; - return; - } - - /* - * If we're configuring a hot-added device, LTR was likely - * disabled in the upstream bridge, so re-enable it before enabling - * it in the new device. - */ - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) { - pci_bridge_reconfigure_ltr(dev); - pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - dev->ltr_path = 1; - } -#endif -} - static void pci_configure_eetlp_prefix(struct pci_dev *dev) { #ifdef CONFIG_PCI_PASID From patchwork Sun Jan 28 23:32:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "David E. Box" X-Patchwork-Id: 193249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp373560dyb; Sun, 28 Jan 2024 21:31:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IE7iO8ZSdjcMMMi6aD4SeD4QwoXi3wBeJd1AZlVucr6nymeTHf+o+s+ygxCWeevy/Bi3mYy X-Received: by 2002:a17:906:2e8b:b0:a31:91a8:799c with SMTP id o11-20020a1709062e8b00b00a3191a8799cmr4067982eji.20.1706506291831; Sun, 28 Jan 2024 21:31:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706506291; cv=pass; d=google.com; s=arc-20160816; b=kGgCnvVekMcUWRv0hdS+HMs90362P3aVC1kk/O4p3zhNtPrn1U+RubN9pVkXQ8Hiyp vGGN6CwrtNvZWaoD4NWv73FTFe6XHQUTfKgaDv7uIq3Jyuk7ayV/ji1pGvvIcGv2QjEn yZlNSq8TtuQd3qUVs+li+QiwcN2M9Mj2SpMMmNM7J1Hx5zA8C1+6MDPIytyX0bs8TuZX d1aCYDj879+IltkyUpZuSmlHbzCRiObyVIRUonrFgN5sCPfDo+Mjp8eCe6FmGLoLQp0v wEjPEuzMwQtYSO5lm4ZDzMFQxlSxX4DP3eHpboJeCxA35YT4UjvRc8z3AAqNKQvAZChX XK5A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=lvoOTiNCMbPWwvxEjIOiGEAmf+tyvfyMi1g4VCLUtMo=; fh=4cLLCZOZ1uAixzSG12P6cLh/7sn/90zAnCRwDJZB7tA=; b=k/fcVHWD1nCmY3Aj6Y5XVFv4pXb4qi7N7F1AzzydCswBqwIfiRAqBP5h/Fq37WWOuT jFt5Phf+QJfPEmwlWbZ27JNjPuR4g7Jqzyy6CgNc9TLrUnizy1njGhmIMHVkbDjbfKld cLnJ66O98JW988tsgLIPY1WlWsC5x4dhDf/hTbgqty5hM3bq+Vs7f0Yrdu2NNEdA862t OzwLvrwqrZl+pX3VWSG5mqS7fwIW/F2ZqAXFljkAmjYe2lcghGPDRr+YVxQrlNTualUz bfzWw7qSsfeMmzKxGpzk2DfDGqulvY8je1kMznEwSkTQJtLhoYh2q16FXaW6fhQWenco buCw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nDgcT85+; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42013-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42013-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id q24-20020a170906a09800b00a35c869725fsi446723ejy.486.2024.01.28.21.31.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 21:31:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42013-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nDgcT85+; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42013-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42013-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C193C1F23E9D for ; Sun, 28 Jan 2024 23:33:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 56DA54EB2B; Sun, 28 Jan 2024 23:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nDgcT85+" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87758446C7; Sun, 28 Jan 2024 23:32:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.65 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; cv=none; b=ak7U3V7+jvTL1QItB8JoptN7NzgEK86fMPPZovZUVNME3G0MMD5gSuAjycuhI2272QEEQkTIDICH7L4B06wyE32mUdLMFE2HaFsFKnvgus9gQJRvtrq/QUZZFmQCesodMwedhWwKi9/73qt1Hi+HxRYeXKH7VpwTwjahJCwZhm8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; c=relaxed/simple; bh=A9rffJbajff9awr7vGDw2RsxSnFOjMip9EPd4+FSp9U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ug0hs+I0S5XcsUUmBeQw4fwzPlicBX546OOdJRNm8NFLNAQijFVLGnmlnhth4ImZGfqW52u5wOe9082+LxRdHi3tyxDcuCZiuFi0Qf4gsMqojf9RWKMHlZlKbfye5oSSYR3Z3V0zfapExLYy8RCFnubrJEYm0qrMgrcT8hfeQxg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nDgcT85+; arc=none smtp.client-ip=134.134.136.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706484736; x=1738020736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=A9rffJbajff9awr7vGDw2RsxSnFOjMip9EPd4+FSp9U=; b=nDgcT85+pYgo8Vcfs4KuceeVSCrN1r0FKk/ii+QDhI7H9lIyLfUR/NHM tIs4gM6YApshqUZ/N7BahCsd+cp8emRbpGk9WQBWVCMruxnBTHbpkAgDu OCtnr+4x/kLU+rj1KR45Z2P4EhstuxOUDJOgrgplsi48RwRbeSBibeYGp 7hsI2cBKgpB8SyaQhV8qVu2uw0L7+ptFNl8LpVkVnQVQckQSnmLGSot1l F+4irKAYg/KquirDBghimzuNNuzQgvdtBJyQVLXnyltvVs2rrraFsqw+U UZ2xOqy6hHC2G6zNl79UZSaSvqFPAYFTA1vWGDFL6rEIM523UTkWNAkT1 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="406535989" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="406535989" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="857921255" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="857921255" Received: from linux.intel.com ([10.54.29.200]) by fmsmga004.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:13 -0800 Received: from debox1-desk4.intel.com (sdutta-mobl2.amr.corp.intel.com [10.209.55.52]) by linux.intel.com (Postfix) with ESMTP id E6860580DD4; Sun, 28 Jan 2024 15:32:12 -0800 (PST) From: "David E. Box" To: mika.westerberg@linux.intel.com, david.e.box@linux.intel.com, ilpo.jarvinen@linux.intel.com, bhelgaas@google.com, rjw@rjwysocki.net Cc: tasev.stefanoska@skynet.be, enriquezmark36@gmail.com, kernel@witt.link, wse@tuxedocomputers.com, vidyas@nvidia.com, kai.heng.feng@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, ricky_wu@realtek.com, mario.limonciello@amd.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/5] PCI: Create function to save L1SS offset Date: Sun, 28 Jan 2024 15:32:09 -0800 Message-Id: <20240128233212.1139663-3-david.e.box@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240128233212.1139663-1-david.e.box@linux.intel.com> References: <20240128233212.1139663-1-david.e.box@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789401541487286011 X-GMAIL-MSGID: 1789401541487286011 The offset for the L1 Substate Capability register is not saved in pci_dev until pci_configure_ltr() which only builds with CONFIG_PCIEASPM. Instead, create a separate function to retrieve the offset so that it is always available. This offset will be used to save and restore the L1SS registers even when PCIEASPM=n. Signed-off-by: David E. Box --- drivers/pci/pci.h | 1 + drivers/pci/pcie/aspm.c | 9 ++++++--- drivers/pci/probe.c | 1 + include/linux/pci.h | 4 ++-- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 6771862de921..b48e8e4f360f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -97,6 +97,7 @@ void pci_msi_init(struct pci_dev *dev); void pci_msix_init(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); +void pci_aspm_get_l1ss(struct pci_dev *pdev); int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type); static inline void pci_wakeup_event(struct pci_dev *dev) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 6d077e237a65..93718b733af3 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -24,6 +24,12 @@ #include "../pci.h" +void pci_aspm_get_l1ss(struct pci_dev *pdev) +{ + /* Read L1 PM substate capabilities */ + pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); +} + #ifdef CONFIG_PCIEASPM #ifdef MODULE_PARAM_PREFIX @@ -970,9 +976,6 @@ void pci_configure_ltr(struct pci_dev *pdev) if (!pci_is_pcie(pdev)) return; - /* Read L1 PM substate capabilities */ - pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); - pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &cap); if (!(cap & PCI_EXP_DEVCAP2_LTR)) return; diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 0b8c2c9cc9dd..e39ad912ce8c 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2208,6 +2208,7 @@ static void pci_configure_device(struct pci_dev *dev) pci_configure_mps(dev); pci_configure_extended_tags(dev, NULL); pci_configure_relaxed_ordering(dev); + pci_aspm_get_l1ss(dev); pci_configure_ltr(dev); pci_configure_eetlp_prefix(dev); pci_configure_serr(dev); diff --git a/include/linux/pci.h b/include/linux/pci.h index dea043bc1e38..dfc4b525c7a1 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -390,9 +390,9 @@ struct pci_dev { unsigned int d3hot_delay; /* D3hot->D0 transition time in ms */ unsigned int d3cold_delay; /* D3cold->D0 transition time in ms */ -#ifdef CONFIG_PCIEASPM - struct pcie_link_state *link_state; /* ASPM link state */ u16 l1ss; /* L1SS Capability pointer */ +#ifdef CONFIG_PCIEASPM + struct pcie_link_state *link_state; /* ASPM link state */ unsigned int ltr_path:1; /* Latency Tolerance Reporting supported from root to here */ #endif From patchwork Sun Jan 28 23:32:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "David E. Box" X-Patchwork-Id: 193193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp293108dyb; Sun, 28 Jan 2024 16:50:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IGVeVLgkd0i6bjYHi8qoAe6MtwTSyU6qpDnM+3YSKFWkAwJ7jqPedRDmtba3AxTI0Z/tL9U X-Received: by 2002:a81:b187:0:b0:602:acba:ece1 with SMTP id p129-20020a81b187000000b00602acbaece1mr2813816ywh.15.1706489407838; Sun, 28 Jan 2024 16:50:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706489407; cv=pass; d=google.com; s=arc-20160816; b=PqMnPwTDw8ne/F/HY4fCTdLvL23DKbywjaNbUm936kG5PKKSpeHi3TVNXcdHZjZtH+ 1QA4aCFiV+XuV90OoGB2FdajFRD/zFsge6UgBOoGrV62wubBXUZIHNh5u7f3MF02hgHA HQFyAt5NWuFiFN/HKp9Uovfmw1/Y0Q0uNlEReNSPqX0yfphQr2aEixlZgmOx0xjmj/uH As7eVMBrul9m0gn2OFNjfwNY+MF2fcO6GzFuHgXo18Fom3fWPezgWHzaQ3vlAHjZhy4q 7xumZVOzS7ecY5oMrXZsHSKtHaT5ggTCpClDdYR7HzTGPVDYxDVpHscp44I3ikiZqk35 p/kA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=RGIZpd9Kt6N3GCjLvRDJEXxoMS2A6m0F3bIKgdtzmgg=; fh=4cLLCZOZ1uAixzSG12P6cLh/7sn/90zAnCRwDJZB7tA=; b=LcIWw/vaHpzE60kB5OE77ROKNN/qVX7QjQsCa+4ycY0LukPzV9ZZ6PbGTU8ORDwXV+ crYE/LzPZZLZFSApV+0tVSepXLGXAG8mufCzWQ7ka11mCaLCVNZIquGCSwLWlabNCwU7 jOAIPbZHC8sN4JhI0Hgd8wZe2uNB2pLwpe23fOfrzGciwgLWq2CYuj7YfOGz5vUcqWGW kWNo2f7k4Zg7ph5UMQXHS8XUlkBUvN+kvrpU6fDbYOMHHMRK9LS78llU4YlxZJ8LbsMK OAGf5ch82MkBj7tYA1wFfDoNgXdzGpni1L+s/gmI4VXzxc4n4WhipXgZ0OmUSq/GnWZN /h1A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KbowVwyy; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42012-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42012-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id d2-20020a05621421c200b0068c427612cbsi3510802qvh.553.2024.01.28.16.50.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 16:50:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42012-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KbowVwyy; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42012-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42012-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 979A81C230AD for ; Sun, 28 Jan 2024 23:33:42 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 77F8A4F1ED; Sun, 28 Jan 2024 23:32:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KbowVwyy" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 025FD44375; Sun, 28 Jan 2024 23:32:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.31 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484737; cv=none; b=JRNAwe6656LfaaUJGUPwO+MJc/6QqD9WmtoGOOUDO2Z6u/7XiTcTlyJL/iQtNN7b/cnaYqU6c1nf9Ub6S5tAqh46r0utnEXI+W4sOCK7ZVhcGJmGmXVT/kj96G1eyDCKH0anAxkA2yKvEaeIJBiJZrIWzXZViRLCs3RMWLM6xf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484737; c=relaxed/simple; bh=zbvixNuPPU6eR4jeUQSZ0UqwT+uwHo8lM70mfjwnmpw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cJRAT+TKzKfRBCQfAAk1B3OinIXWCTtCN2W1/kXggVUzXVhIZpObWY/d3sInG3ph122RM6DbX5ZyLOfKr7WmT48pA0eBD9e88n65FzpdNceo4ovvTpq1wnxdf857C8skMrX52SjLMIfrFXCBY7k3sC2O+uNWVeTC756XXlY/A64= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KbowVwyy; arc=none smtp.client-ip=134.134.136.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706484735; x=1738020735; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zbvixNuPPU6eR4jeUQSZ0UqwT+uwHo8lM70mfjwnmpw=; b=KbowVwyyN8cOzKiX42Xa3LbOkjSHuh1VWIKFan+IBBRNkViaA7g/EZK8 0bPk/D3BBqq79y/bUguzm4YFTMsG1bQ3VFc3SOBKThwuYgwUbxgYKtQEu +Hc2B3OGIkKTZ7akqqsE9wkQ8/rWv+WqrlU0DIW0NS8Yre7yYjEu2JQuP NB2oOgT+mhNeqXcvk5FE/C4GzbyKlMalCNSezvqpW4nNNxCnvbJnCWiFo mtYiCNJO63gqt+Tlitc4MaVjkOU9uJ1wmEBp4xQ3zM6y0g1gxrpDT9nA4 dBoBdicykOFx6GAP5m2u6YQwNGWjtLAcjYix6M9H+c3mgfmwvHQSUNWOg A==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="467081105" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="467081105" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="930913778" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="930913778" Received: from linux.intel.com ([10.54.29.200]) by fmsmga001.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 Received: from debox1-desk4.intel.com (sdutta-mobl2.amr.corp.intel.com [10.209.55.52]) by linux.intel.com (Postfix) with ESMTP id 4F7FC580DE6; Sun, 28 Jan 2024 15:32:13 -0800 (PST) From: "David E. Box" To: mika.westerberg@linux.intel.com, david.e.box@linux.intel.com, ilpo.jarvinen@linux.intel.com, bhelgaas@google.com, rjw@rjwysocki.net Cc: tasev.stefanoska@skynet.be, enriquezmark36@gmail.com, kernel@witt.link, wse@tuxedocomputers.com, vidyas@nvidia.com, kai.heng.feng@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, ricky_wu@realtek.com, mario.limonciello@amd.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/5] PCI/ASPM: Add back L1 PM Substate save and restore Date: Sun, 28 Jan 2024 15:32:10 -0800 Message-Id: <20240128233212.1139663-4-david.e.box@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240128233212.1139663-1-david.e.box@linux.intel.com> References: <20240128233212.1139663-1-david.e.box@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789383837295325266 X-GMAIL-MSGID: 1789383837295325266 Commit 4ff116d0d5fd ("PCI/ASPM: Save L1 PM Substates Capability for suspend/resume") was reverted due to a regression that caused resume from suspend to fail on certain systems. However, lack of this feature is now causing systems to fail to enter low power CPU states, drawing more power from the battery. The original revert mentioned that we restore L1 PM substate configuration even though ASPM L1 may already be enabled. This is due the fact that the pci_restore_aspm_l1ss_state() was called before pci_restore_pcie_state(). Try to enable this functionality again following PCIe r6.0.1, sec 5.5.4 more closely by: 1) Do not restore ASPM configuration in pci_restore_pcie_state() but do that after PCIe capability is restored in pci_restore_aspm_state() following PCIe r6.0, sec 5.5.4. 2) If BIOS reenables L1SS on us, particularly L1.2, we need to clear the enables in the right order, downstream before upstream. Defer restoring the L1SS config until we are at the downstream component. Then update the config for both ends of the link in the prescribed order. 3) Program ASPM L1 PM substate configuration before L1 enables. 4) Program ASPM L1 PM substate enables last after rest of the fields in the capability are programmed. Reported-by: Koba Ko Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217321 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216782 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216877 Cc: Tasev Nikola Cc: Mark Enriquez Cc: Thomas Witt Cc: Werner Sembach Cc: Vidya Sagar Co-developed-by: Mika Westerberg Signed-off-by: Mika Westerberg Co-developed-by: David E. Box Signed-off-by: David E. Box --- V1 - Move aspm save/restore calls to pci_save/restore_pcie_state() - Move aspm save/restore functions outside of CONFIG_PCIEASPM so that L1SS state is always managed. - pcie_link_state is now unavailable when CONFIG_PCIEASPM=n, so use pcie_downstream_port() to test for the link during restore. - Comment and code cleanup suggested by Bjorn and Mika Previous history before new series: v5: https://lore.kernel.org/linux-pci/20231221011250.191599-1-david.e.box@linux.intel.com/ v4: https://lore.kernel.org/linux-pci/20231002070044.2299644-1-mika.westerberg@linux.intel.com/ v3: https://lore.kernel.org/linux-pci/20230925074636.2893747-1-mika.westerberg@linux.intel.com/ v2: https://lore.kernel.org/linux-pci/20230911073352.3472918-1-mika.westerberg@linux.intel.com/ v1: https://lore.kernel.org/linux-pci/20230627062442.54008-1-mika.westerberg@linux.intel.com/ drivers/pci/pci.c | 19 +++++- drivers/pci/pci.h | 2 + drivers/pci/pcie/aspm.c | 136 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 144 insertions(+), 13 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 71229ec39e88..0a8613e77dab 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1560,6 +1560,8 @@ static int pci_save_pcie_state(struct pci_dev *dev) pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &cap[i++]); pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &cap[i++]); + pci_save_aspm_state(dev); + return 0; } @@ -1567,7 +1569,7 @@ static void pci_restore_pcie_state(struct pci_dev *dev) { int i = 0; struct pci_cap_saved_state *save_state; - u16 *cap; + u16 *cap, val; save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); if (!save_state) @@ -1582,12 +1584,20 @@ static void pci_restore_pcie_state(struct pci_dev *dev) cap = (u16 *)&save_state->cap.data[0]; pcie_capability_write_word(dev, PCI_EXP_DEVCTL, cap[i++]); - pcie_capability_write_word(dev, PCI_EXP_LNKCTL, cap[i++]); + + /* + * Restore only the LNKCTL register with the ASPM control field + * clear. ASPM will be restored in pci_restore_aspm_state(). + */ + val = cap[i++] & ~PCI_EXP_LNKCTL_ASPMC; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL, val); pcie_capability_write_word(dev, PCI_EXP_SLTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_RTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_DEVCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_SLTCTL2, cap[i++]); + + pci_restore_aspm_state(dev); } static int pci_save_pcix_state(struct pci_dev *dev) @@ -3497,6 +3507,11 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev) if (error) pci_err(dev, "unable to allocate suspend buffer for LTR\n"); + error = pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_L1SS, + 2 * sizeof(u32)); + if (error) + pci_err(dev, "unable to allocate suspend buffer for ASPM-L1SS\n"); + pci_allocate_vc_save_buffers(dev); } diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index b48e8e4f360f..7b14cdbe2e69 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -98,6 +98,8 @@ void pci_msix_init(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); void pci_aspm_get_l1ss(struct pci_dev *pdev); +void pci_save_aspm_state(struct pci_dev *pdev); +void pci_restore_aspm_state(struct pci_dev *pdev); int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type); static inline void pci_wakeup_event(struct pci_dev *dev) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 93718b733af3..3185058e9c41 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -30,6 +30,131 @@ void pci_aspm_get_l1ss(struct pci_dev *pdev) pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); } +static void pci_clear_and_set_dword(struct pci_dev *pdev, int pos, + u32 clear, u32 set) +{ + u32 val; + + pci_read_config_dword(pdev, pos, &val); + val &= ~clear; + val |= set; + pci_write_config_dword(pdev, pos, val); +} + +void pci_save_aspm_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *save_state; + u16 l1ss = pdev->l1ss; + u32 *cap; + + /* + * Save L1 substate configuration. The ASPM L0s/L1 configuration + * is already saved in pci_save_pcie_state(). + */ + if (!l1ss) + return; + + save_state = pci_find_saved_ext_cap(pdev, PCI_EXT_CAP_ID_L1SS); + if (!save_state) + return; + + cap = &save_state->cap.data[0]; + pci_read_config_dword(pdev, l1ss + PCI_L1SS_CTL2, cap++); + pci_read_config_dword(pdev, l1ss + PCI_L1SS_CTL1, cap++); +} + +static void pcie_restore_aspm_l1ss(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *pl_save_state, *cl_save_state; + struct pci_dev *parent = pdev->bus->self; + u32 *cap, pl_ctl1, pl_ctl2, pl_l1_2_enable; + u32 cl_ctl1, cl_ctl2, cl_l1_2_enable; + + /* + * In case BIOS enabled L1.2 after resume, we need to disable it first + * on the downstream component before the upstream. So, don't attempt to + * restore either until we are at the downstream component. + */ + if (pcie_downstream_port(pdev) || !parent) + return; + + if (!pdev->l1ss || !parent->l1ss) + return; + + cl_save_state = pci_find_saved_ext_cap(pdev, PCI_EXT_CAP_ID_L1SS); + pl_save_state = pci_find_saved_ext_cap(parent, PCI_EXT_CAP_ID_L1SS); + if (!cl_save_state || !pl_save_state) + return; + + cap = &cl_save_state->cap.data[0]; + cl_ctl2 = *cap++; + cl_ctl1 = *cap; + cap = &pl_save_state->cap.data[0]; + pl_ctl2 = *cap++; + pl_ctl1 = *cap; + + + /* + * Disable L1.2 on this downstream endpoint device first, followed + * by the upstream + */ + pci_clear_and_set_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, + PCI_L1SS_CTL1_L1_2_MASK, 0); + pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, + PCI_L1SS_CTL1_L1_2_MASK, 0); + + /* + * In addition, Common_Mode_Restore_Time and LTR_L1.2_THRESHOLD + * in PCI_L1SS_CTL1 must be programmed *before* setting the L1.2 + * enable bits, even though they're all in PCI_L1SS_CTL1. + */ + pl_l1_2_enable = pl_ctl1 & PCI_L1SS_CTL1_L1_2_MASK; + pl_ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK; + cl_l1_2_enable = cl_ctl1 & PCI_L1SS_CTL1_L1_2_MASK; + cl_ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK; + + /* Write back without enables first (above we cleared them in ctl1) */ + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL2, pl_ctl2); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL2, cl_ctl2); + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL1, pl_ctl1); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, cl_ctl1); + + + /* Then write back the enables */ + if (pl_l1_2_enable || cl_l1_2_enable) { + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL1, + pl_ctl1 | pl_l1_2_enable); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, + cl_ctl1 | cl_l1_2_enable); + } +} + +void pci_restore_aspm_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *save_state; + u16 *cap, val; + + save_state = pci_find_saved_cap(pdev, PCI_CAP_ID_EXP); + + if (!save_state) + return; + + cap = (u16 *)&save_state->cap.data[0]; + /* Must match the ordering in pci_save/restore_pcie_state() */ + val = cap[1] & PCI_EXP_LNKCTL_ASPMC; + if (!val) + return; + + /* + * We restore L1 substate configuration first before enabling L1 + * as the PCIe spec 6.0 sec 5.5.4 suggests. + */ + pcie_restore_aspm_l1ss(pdev); + + /* Re-enable L0s/L1 */ + pcie_capability_set_word(pdev, PCI_EXP_LNKCTL, val); +} + #ifdef CONFIG_PCIEASPM #ifdef MODULE_PARAM_PREFIX @@ -434,17 +559,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) } } -static void pci_clear_and_set_dword(struct pci_dev *pdev, int pos, - u32 clear, u32 set) -{ - u32 val; - - pci_read_config_dword(pdev, pos, &val); - val &= ~clear; - val |= set; - pci_write_config_dword(pdev, pos, val); -} - /* Calculate L1.2 PM substate timing parameters */ static void aspm_calc_l12_info(struct pcie_link_state *link, u32 parent_l1ss_cap, u32 child_l1ss_cap) From patchwork Sun Jan 28 23:32:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "David E. Box" X-Patchwork-Id: 193217 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp313765dyb; Sun, 28 Jan 2024 18:00:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEbVRnznysELfbpSNTkX+wgIiHrtxRDiEUCi6W5RQLuDdTrZd++G0uYL02l6Es031ata85 X-Received: by 2002:ac8:5941:0:b0:429:fe73:f352 with SMTP id 1-20020ac85941000000b00429fe73f352mr4678423qtz.134.1706493631129; Sun, 28 Jan 2024 18:00:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706493631; cv=pass; d=google.com; s=arc-20160816; b=p6xok6qlX8kTPytMEQio9rppiNA4pgocohmH1oHK4QWkOuLjSWT3MXy9DwfxJAHuw3 /Tfhsljv9Cvx4N34LcyiatiqtgyMTW0e44RK+FheeZ293gbRQ/ZoBMsTuCD1dCPMAIOF nAzXJRcyVMXmpP3D3W4Lo02RUPFn5D040bU/51+6QtH/YTaefxcXJf+spnV4B9oyB62s bPrFbsvSo97N3OPZRXiMubZqm2AhQgHgqlrlDXCLGSE9cr9wTFx+enwhBCrR+RvLewDf rWiDMUd3naMyaMmc4YRZSTQnc56iR0sjwbCRhrAgwpS9kFv7lVC8OQJw6yx9X9fNI78i ofmw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=yCrk2S0VnSw8aF4Ab64oIY+6XLPZYhtBoEMlXHxI+Rw=; fh=4cLLCZOZ1uAixzSG12P6cLh/7sn/90zAnCRwDJZB7tA=; b=WGLiu7Cb32sak2FsvUXxMV99C+i/S+raPexhhasPg5q2BcOcSppVbiV3W9WZ+72GN9 /oRkoO986esG9SaPS5uFmxXW9MKmLg7fJvZwTLsRVedmU0Kab2PVtuOqkInm5kNaOKWw 6FkW/lj9X+c+BqBo7vpgZk7eobLGN6Q3f9Qn8DPT/09oSaLB7qHSDX71HYbYRHfeTRXP fftZqztuC3VixvLZl3kmMs5P8HF4eiUPV43NYPEIN8P4z8qH1DTBKFDEMKALg9VIzo6M 14mXNDSreG8V74VGKDjEMezyiBxffqtNANgyP/QzOFy4ASAuIaxJDk1zX1qznefLu2iv 3M5w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AWWsE4pW; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42014-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42014-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id x17-20020ac85f11000000b0042a9547696asi3700347qta.702.2024.01.28.18.00.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 18:00:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42014-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AWWsE4pW; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42014-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42014-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id DAA281C22F34 for ; Sun, 28 Jan 2024 23:33:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C583A4EB54; Sun, 28 Jan 2024 23:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AWWsE4pW" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D117C446DC; Sun, 28 Jan 2024 23:32:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.31 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; cv=none; b=gsiZulFQVTP/M/fm1GfqKmt2B7ax71uwYfYZINURu8laMP885KGQeT/aW4fzcNJXO7EK9ZS2MW5x9vg6dyhmKTUli29d0lMPzLjdjToSsbqsPkno5wMp9jDclvaIn/PpPZtcTg7ORNtNIkv2UJaRCkVdqkBdqdRNpPuk5lHplMI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; c=relaxed/simple; bh=rro8bua4GahrpLh+0Y+aOJSQnICNHshw7fWOe3S8s9w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cWCRBLxp+lFZUXTrLg/v1h8UBuJy1r9nhxPCxWuHlzAI7HnhXwj35YXrtwn2Sk/XUD6cZXWHM4VPqko0Pv6T2TMSe6aAaN+00MwlO9F/LZFwOgjhY3tXJEE/gczsEIrAJAXIRgqtby0R12dou+TL4+t26dHaeSqFbXeCvwWUQyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AWWsE4pW; arc=none smtp.client-ip=134.134.136.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706484736; x=1738020736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rro8bua4GahrpLh+0Y+aOJSQnICNHshw7fWOe3S8s9w=; b=AWWsE4pWRIgm56TwixDWvb5csfBr/CCcCJ63zC/UuZcHXLG2VJ52LOd/ Q289JrlMWG/JycZWOVxlqzm4PbA1p74Od2J0nrG46ZT/wLPUkPekt5A8Y PgPzyQAx81SvoheSF0qQdQfWRP0WGx1gGN7zKovamhRN5YQQfvZdZgZ9C dcly75jyCEHfanuHNpYHEDn3v2tLxbYQFq9X99W7Bn03h8hTvq5niiKch pmno7ntw8wbhgppVNBPD9rQK/2z4VYf13zvj8qmP2SQLv5C3yUYO943y+ PNC+rz/FYeAAWKDuMo8i8HLYmtXk4iFpylSH+KBqVnzPxfLc5Lj7KYTrv w==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="467081116" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="467081116" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="930913782" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="930913782" Received: from linux.intel.com ([10.54.29.200]) by fmsmga001.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 Received: from debox1-desk4.intel.com (sdutta-mobl2.amr.corp.intel.com [10.209.55.52]) by linux.intel.com (Postfix) with ESMTP id AAAD0580D78; Sun, 28 Jan 2024 15:32:13 -0800 (PST) From: "David E. Box" To: mika.westerberg@linux.intel.com, david.e.box@linux.intel.com, ilpo.jarvinen@linux.intel.com, bhelgaas@google.com, rjw@rjwysocki.net Cc: tasev.stefanoska@skynet.be, enriquezmark36@gmail.com, kernel@witt.link, wse@tuxedocomputers.com, vidyas@nvidia.com, kai.heng.feng@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, ricky_wu@realtek.com, mario.limonciello@amd.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/5] PCI: Move pci_save/restore_ltr_state() to aspm.c Date: Sun, 28 Jan 2024 15:32:11 -0800 Message-Id: <20240128233212.1139663-5-david.e.box@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240128233212.1139663-1-david.e.box@linux.intel.com> References: <20240128233212.1139663-1-david.e.box@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789388265837780168 X-GMAIL-MSGID: 1789388265837780168 Since the LTR Capability is linked with ASPM and only enabled when CONFIG_PCIEASPM is set, move the save/restore code to aspm.c Suggested-by: Bjorn Helgaas Signed-off-by: David E. Box --- drivers/pci/pci.c | 40 ---------------------------------------- drivers/pci/pci.h | 2 ++ drivers/pci/pcie/aspm.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 40 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 0a8613e77dab..61e56e040510 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1636,46 +1636,6 @@ static void pci_restore_pcix_state(struct pci_dev *dev) pci_write_config_word(dev, pos + PCI_X_CMD, cap[i++]); } -static void pci_save_ltr_state(struct pci_dev *dev) -{ - int ltr; - struct pci_cap_saved_state *save_state; - u32 *cap; - - if (!pci_is_pcie(dev)) - return; - - ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); - if (!ltr) - return; - - save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); - if (!save_state) { - pci_err(dev, "no suspend buffer for LTR; ASPM issues possible after resume\n"); - return; - } - - /* Some broken devices only support dword access to LTR */ - cap = &save_state->cap.data[0]; - pci_read_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, cap); -} - -static void pci_restore_ltr_state(struct pci_dev *dev) -{ - struct pci_cap_saved_state *save_state; - int ltr; - u32 *cap; - - save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); - ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); - if (!save_state || !ltr) - return; - - /* Some broken devices only support dword access to LTR */ - cap = &save_state->cap.data[0]; - pci_write_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, *cap); -} - /** * pci_save_state - save the PCI configuration space of a device before * suspending diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 7b14cdbe2e69..98f54b48f013 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -100,6 +100,8 @@ void pci_bridge_d3_update(struct pci_dev *dev); void pci_aspm_get_l1ss(struct pci_dev *pdev); void pci_save_aspm_state(struct pci_dev *pdev); void pci_restore_aspm_state(struct pci_dev *pdev); +void pci_save_ltr_state(struct pci_dev *dev); +void pci_restore_ltr_state(struct pci_dev *dev); int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type); static inline void pci_wakeup_event(struct pci_dev *dev) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 3185058e9c41..f7712d8453a4 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -155,6 +155,46 @@ void pci_restore_aspm_state(struct pci_dev *pdev) pcie_capability_set_word(pdev, PCI_EXP_LNKCTL, val); } +void pci_save_ltr_state(struct pci_dev *dev) +{ + int ltr; + struct pci_cap_saved_state *save_state; + u32 *cap; + + if (!pci_is_pcie(dev)) + return; + + ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); + if (!ltr) + return; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); + if (!save_state) { + pci_err(dev, "no suspend buffer for LTR; ASPM issues possible after resume\n"); + return; + } + + /* Some broken devices only support dword access to LTR */ + cap = &save_state->cap.data[0]; + pci_read_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, cap); +} + +void pci_restore_ltr_state(struct pci_dev *dev) +{ + struct pci_cap_saved_state *save_state; + int ltr; + u32 *cap; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); + ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); + if (!save_state || !ltr) + return; + + /* Some broken devices only support dword access to LTR */ + cap = &save_state->cap.data[0]; + pci_write_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, *cap); +} + #ifdef CONFIG_PCIEASPM #ifdef MODULE_PARAM_PREFIX From patchwork Sun Jan 28 23:32:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "David E. Box" X-Patchwork-Id: 193194 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp293160dyb; Sun, 28 Jan 2024 16:50:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IFIXWpAW42CPT32Q+f3uBD7i4KW+7HhtQlGfq1vcoeivCrjnFdW0nRy/sh8//eMHBx0vQyh X-Received: by 2002:a05:620a:b02:b0:783:ab42:faf4 with SMTP id t2-20020a05620a0b0200b00783ab42faf4mr4713249qkg.127.1706489418520; Sun, 28 Jan 2024 16:50:18 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706489418; cv=pass; d=google.com; s=arc-20160816; b=n8r0pRlbUgzwBW4XcGXfQc96TlK+emvY4M5GOi2fn6QNCMmzTuPKw8F/bKBNuBprTO GqKP2Fw9aQzqEyB4EQOjhZqK7I0kupkMrSnlfE9RrveFPxGQoXR9PpRxKqTevawy2L4x z7Y4jMWntQ8AJCz8l8REf+Z8AdEyjwUuwpXpc1Pg9CuGRF+DuaIOg4iS+RGi1YMqOJ6T wyednBQV7auLz0yfAnMC4uPoo7pjtN30AnYx0W8LYMd9imQn/m9mHPjaLE9CAyrCf9VB JNQXJ680zZRF/vFDt0cdAvhfTTTtfDkcUY54iJ/tIh6SLhX+KHRvRRRI2VHfeoAnBWsw CHiA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=oxVC7xrvL1Q5bMg54y7zXPT0xUkucqZ05XOmAM7q4aw=; fh=4cLLCZOZ1uAixzSG12P6cLh/7sn/90zAnCRwDJZB7tA=; b=RRQ/VYDncuWe2oI5+FaDt6JpOk/7uXx+2T+17TXX6pQe55QMlZhSXMiRAreVWqx3sz b8hHMiutGeII23L5FEiNJX7F9poQCFgxk38TpKELAq+gxQaec4TgLU43YuB+lpN7reWP JH7/ObpePL2QfG9nkop5ZLCNrFOqY5QRMSDVVtC6gSXqQ3d4/84FsRjyF6fBJTRJdfc9 nlg7Vr3ABwcmsJ/nC6CtJOCc8Xz+gC3TKvDVr2L3UaJiOq1BS2ztVYV8GpDws7GXBDbB tN7E2wSgohreOKz1S0RCZOEfqST+/vTyqRvjBivmFFfdu9bAWN8V/MOfsm77Wsw9tpfs EGJA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KB42YIRH; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42015-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42015-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id b22-20020a05620a127600b00783b616fd50si2375566qkl.329.2024.01.28.16.50.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 16:50:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-42015-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KB42YIRH; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-42015-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-42015-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 871991C22F0C for ; Sun, 28 Jan 2024 23:33:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A54A94EB49; Sun, 28 Jan 2024 23:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KB42YIRH" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43DA64595D; Sun, 28 Jan 2024 23:32:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.31 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; cv=none; b=Fsp4gZWasuFbtiVrVkGpI1ni0ouziL67rn8xRY/KjAnMBlNAsY8xOm9LUNCo8FLGCfY4GgcAQz6b3DtmvG5FSW2ugXSzjlFfYMJlXiGnJaOwaLoSAt0MqsTXYJOu3DjzhMHAlHM/dh1565yGZU/GrjZUkGL/s03D930ITxyg+R8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706484738; c=relaxed/simple; bh=bx8duAkTrynztEMQULCsS8d2N4dVOXeKVQtbu5S9ZNg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Dsoa4hUlwB7B5G3MLZIDuhd1QoT/045iaruvjZ9X4SiE7IHvDk95Do5qQ16YhIVBQuBJJJYWMF2cwwk/4FfRMKiS0TNxiKUp2vPBdTt9G0wzV2rQH7Rg+NTO1cWT8ehPMCvBNcoxM+RVlNoAFV6NTARljp39TEtkdtD8EN6rtuU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KB42YIRH; arc=none smtp.client-ip=134.134.136.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706484737; x=1738020737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bx8duAkTrynztEMQULCsS8d2N4dVOXeKVQtbu5S9ZNg=; b=KB42YIRH/UaRoyB0x2L/QGye/5MNpgpWjmrYB25jR7nuQemssAC+QkI4 7M+1z11B0VYLKWWlylIHFy65OupVMwfyC8rbhLVNcgOg8tOSdi23tr1VH V3K08mrGTp30+gHYgD2EAM3Y19tz6nubc+wcx+Q6QmiuAR0G6quBAC5Cm IeREyRD/6WplW3HuE+ozu1Sia3n4tOuliamr4Pr3L9bn5qROspXJSRRLI gvCjTrDW6GiV7hkfCYmCieySZVoxDeg9Hy46XPUaru/gfLn1dQJYx8Fnz iSVWKgwEscPX0p02A/TMo/hy6te6LiSbrLxyi27onwxyU+Ie/LWqIyRlq w==; X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="467081131" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="467081131" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10967"; a="930913784" X-IronPort-AV: E=Sophos;i="6.05,226,1701158400"; d="scan'208";a="930913784" Received: from linux.intel.com ([10.54.29.200]) by fmsmga001.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2024 15:32:14 -0800 Received: from debox1-desk4.intel.com (sdutta-mobl2.amr.corp.intel.com [10.209.55.52]) by linux.intel.com (Postfix) with ESMTP id 12C9F580DDD; Sun, 28 Jan 2024 15:32:14 -0800 (PST) From: "David E. Box" To: mika.westerberg@linux.intel.com, david.e.box@linux.intel.com, ilpo.jarvinen@linux.intel.com, bhelgaas@google.com, rjw@rjwysocki.net Cc: tasev.stefanoska@skynet.be, enriquezmark36@gmail.com, kernel@witt.link, wse@tuxedocomputers.com, vidyas@nvidia.com, kai.heng.feng@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, ricky_wu@realtek.com, mario.limonciello@amd.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/5] PCI: Save and restore LTR state from pci_save/restore_pcie_state() Date: Sun, 28 Jan 2024 15:32:12 -0800 Message-Id: <20240128233212.1139663-6-david.e.box@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240128233212.1139663-1-david.e.box@linux.intel.com> References: <20240128233212.1139663-1-david.e.box@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789383848705915747 X-GMAIL-MSGID: 1789383848705915747 ASPM state is saved and restored from pci_save/restore_pcie_state(). Since the LTR Capability is linked with ASPM, move the LTR save and restore calls there as well. Suggested-by: Bjorn Helgaas Signed-off-by: David E. Box --- drivers/pci/pci.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 61e56e040510..78c3c9d82b3b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1561,6 +1561,7 @@ static int pci_save_pcie_state(struct pci_dev *dev) pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &cap[i++]); pci_save_aspm_state(dev); + pci_save_ltr_state(dev); return 0; } @@ -1571,6 +1572,12 @@ static void pci_restore_pcie_state(struct pci_dev *dev) struct pci_cap_saved_state *save_state; u16 *cap, val; + /* + * Restore max latencies (in the LTR capability) before enabling + * LTR itself (in the PCIe capability). + */ + pci_restore_ltr_state(dev); + save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); if (!save_state) return; @@ -1660,7 +1667,6 @@ int pci_save_state(struct pci_dev *dev) if (i != 0) return i; - pci_save_ltr_state(dev); pci_save_dpc_state(dev); pci_save_aer_state(dev); pci_save_ptm_state(dev); @@ -1761,12 +1767,6 @@ void pci_restore_state(struct pci_dev *dev) if (!dev->state_saved) return; - /* - * Restore max latencies (in the LTR capability) before enabling - * LTR itself (in the PCIe capability). - */ - pci_restore_ltr_state(dev); - pci_restore_pcie_state(dev); pci_restore_pasid_state(dev); pci_restore_pri_state(dev);