From patchwork Thu May 11 13:14:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ilpo_J=C3=A4rvinen?= X-Patchwork-Id: 92570 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4367492vqo; Thu, 11 May 2023 06:22:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4utDiaizI3RTXLVmbnBvJCkJYKj2GGHs+JTsLxg2JtSMebk33WQDe3FSu5SvMN253q55tu X-Received: by 2002:a05:6a20:3d8b:b0:104:2d89:8f75 with SMTP id s11-20020a056a203d8b00b001042d898f75mr634678pzi.24.1683811338149; Thu, 11 May 2023 06:22:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683811338; cv=none; d=google.com; s=arc-20160816; b=E4hDyFJVMz+JvlQ91zQ944EtHVfjH+UStDkS+Sd7tnTxkOGLz8NBIK018RnNNlgUp/ g7RYMka8BwWbuutb4X1n170qwhXr1ngbPMDy2YxSm68924ygAQcmGKJ29S03KiROtoLD Ggv0Ocq8EZJxUi2a7FHUB9dT9niWhxdR5OMI0UHI3C+X7j6Q3fPNvchg2KsvZupapFkq H4pIubFLYvhoiij6m2rq5gKpU9odg+coVb/p4cy3FerZXwcm/4mVj8FWNtOeLV9Z3u3Z Hvq7MeB5rQ+Qm9i9LkdWdtE87M1Ns8OeEq6Cozkeb1dO/mXXPAQ/7v1sJMHtwYN3r/5B rDVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FKaUiGopWq/MYBCE5I+366N4ihonKYV49DZm1TWPDZw=; b=mA52sfW0Rjr+dVP7x9CJx9OM7Z5C4S7krML1hhGdE7gwN12v+oUgQonlK16wKkW/Sw K1vDs5s81LHmtKm0iokHwFCyaE2BIsei47EpeH9RDEIlqTNUsw67qLSJnGCy3rPmonb5 CfkanKbwVFgbT1jDYSu/QjYbiBlrr+l1l5qDBHuCCZwCtWlZvW2jCpI4ieHMa4oBP2P6 b9Nhknws4u1SqKOWDELo/lOgV7uH5lUJdNyxITQKD253ynL7gqA1goC3FhTwzy/gckbJ z2D25kfKfRjGgXz/j/RdBkwyWQg1bq+/5rrLjqjlvxxBjZfDQiSQRjz45FjygiqNIjKa 1Ysg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CXASp47I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h14-20020a056a00000e00b00640d9c06df3si3141221pfk.329.2023.05.11.06.21.36; Thu, 11 May 2023 06:22:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CXASp47I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238161AbjEKNQ1 (ORCPT + 99 others); Thu, 11 May 2023 09:16:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238068AbjEKNPu (ORCPT ); Thu, 11 May 2023 09:15:50 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06B70DD88; Thu, 11 May 2023 06:15:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683810916; x=1715346916; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jtsJhk981/YB6xXduLi5eO9mCj6c1o6wGiSwZ35uutw=; b=CXASp47IaIqvG2kLnB9KTA6ERZskvH4QvPyUbkQ95xqY7h65UFtFkW4D zuMz5OCAmoq+r0NJq5NgdyKE9F7lCph1CO5+aFxBoSNPwbF9RNqCJ2dha iCvUcm7K7LEWnk6a+hYNbR5FMmW9qluq3H7fNhXmo0+/7RVtOUvgBXFr/ 2d+4FvsK8fXQ29Qi6fOfBtmI4yQaACkGdA29qofBJxwfoUMxiL3fWYZ9w xCAbjY79bLdSxoS0MjnPGGLZs4trNdptUgZ/wjbd2qVoiDog2NNKYHp0E qftfe6GlzlDxCcCNyWoIht9mRwiaukW+LPK7LrtBFASSQl0UOKY/efAg/ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10706"; a="378619428" X-IronPort-AV: E=Sophos;i="5.99,266,1677571200"; d="scan'208";a="378619428" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2023 06:15:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10706"; a="650169666" X-IronPort-AV: E=Sophos;i="5.99,266,1677571200"; d="scan'208";a="650169666" Received: from jsanche3-mobl1.ger.corp.intel.com (HELO ijarvine-MOBL2.ger.corp.intel.com) ([10.252.39.112]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2023 06:15:11 -0700 From: =?utf-8?q?Ilpo_J=C3=A4rvinen?= To: linux-pci@vger.kernel.org, Bjorn Helgaas , Rob Herring , Lorenzo Pieralisi , =?utf-8?q?Krzysztof_Wilczy=C5=84ski?= , Lukas Wunner , Dennis Dalessandro , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: =?utf-8?q?Ilpo_J=C3=A4rvinen?= Subject: [PATCH 06/17] IB/hfi1: Use pcie_lnkctl{,2}_clear_and_set() for changing LNKCTL{,2} Date: Thu, 11 May 2023 16:14:30 +0300 Message-Id: <20230511131441.45704-7-ilpo.jarvinen@linux.intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230511131441.45704-1-ilpo.jarvinen@linux.intel.com> References: <20230511131441.45704-1-ilpo.jarvinen@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765604157769887220?= X-GMAIL-MSGID: =?utf-8?q?1765604157769887220?= Don't assume that only the driver would be accessing LNKCTL/LNKCTL2. ASPM policy changes can trigger write to LNKCTL outside of driver's control. And in the case of upstream (parent), the driver does not even own the device it's changing the registers for. Use pcie_lnkctl_clear_and_set() and pcie_lnkctl2_clear_and_set() which do proper locking to avoid losing concurrent updates to the register value. Suggested-by: Lukas Wunner Signed-off-by: Ilpo Järvinen --- drivers/infiniband/hw/hfi1/aspm.c | 16 ++++++---------- drivers/infiniband/hw/hfi1/pcie.c | 28 ++++++---------------------- 2 files changed, 12 insertions(+), 32 deletions(-) diff --git a/drivers/infiniband/hw/hfi1/aspm.c b/drivers/infiniband/hw/hfi1/aspm.c index a3c53be4072c..d3f3b7e9b833 100644 --- a/drivers/infiniband/hw/hfi1/aspm.c +++ b/drivers/infiniband/hw/hfi1/aspm.c @@ -66,12 +66,10 @@ static void aspm_hw_enable_l1(struct hfi1_devdata *dd) return; /* Enable ASPM L1 first in upstream component and then downstream */ - pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL, - PCI_EXP_LNKCTL_ASPMC, - PCI_EXP_LNKCTL_ASPM_L1); - pcie_capability_clear_and_set_word(dd->pcidev, PCI_EXP_LNKCTL, - PCI_EXP_LNKCTL_ASPMC, - PCI_EXP_LNKCTL_ASPM_L1); + pcie_lnkctl_clear_and_set(parent, PCI_EXP_LNKCTL_ASPMC, + PCI_EXP_LNKCTL_ASPM_L1); + pcie_lnkctl_clear_and_set(dd->pcidev, PCI_EXP_LNKCTL_ASPMC, + PCI_EXP_LNKCTL_ASPM_L1); } void aspm_hw_disable_l1(struct hfi1_devdata *dd) @@ -79,11 +77,9 @@ void aspm_hw_disable_l1(struct hfi1_devdata *dd) struct pci_dev *parent = dd->pcidev->bus->self; /* Disable ASPM L1 first in downstream component and then upstream */ - pcie_capability_clear_and_set_word(dd->pcidev, PCI_EXP_LNKCTL, - PCI_EXP_LNKCTL_ASPMC, 0x0); + pcie_lnkctl_clear_and_set(dd->pcidev, PCI_EXP_LNKCTL_ASPMC, 0); if (parent) - pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL, - PCI_EXP_LNKCTL_ASPMC, 0x0); + pcie_lnkctl_clear_and_set(parent, PCI_EXP_LNKCTL_ASPMC, 0); } static void aspm_enable(struct hfi1_devdata *dd) diff --git a/drivers/infiniband/hw/hfi1/pcie.c b/drivers/infiniband/hw/hfi1/pcie.c index 08732e1ac966..fe7324d38d64 100644 --- a/drivers/infiniband/hw/hfi1/pcie.c +++ b/drivers/infiniband/hw/hfi1/pcie.c @@ -1212,14 +1212,10 @@ int do_pcie_gen3_transition(struct hfi1_devdata *dd) (u32)lnkctl2); /* only write to parent if target is not as high as ours */ if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) < target_vector) { - lnkctl2 &= ~PCI_EXP_LNKCTL2_TLS; - lnkctl2 |= target_vector; - dd_dev_info(dd, "%s: ..new link control2: 0x%x\n", __func__, - (u32)lnkctl2); - ret = pcie_capability_write_word(parent, - PCI_EXP_LNKCTL2, lnkctl2); + pcie_lnkctl2_clear_and_set(parent, PCI_EXP_LNKCTL2_TLS, + target_vector); if (ret) { - dd_dev_err(dd, "Unable to write to PCI config\n"); + dd_dev_err(dd, "Unable to change PCI target speed\n"); return_error = 1; goto done; } @@ -1228,22 +1224,10 @@ int do_pcie_gen3_transition(struct hfi1_devdata *dd) } dd_dev_info(dd, "%s: setting target link speed\n", __func__); - ret = pcie_capability_read_word(dd->pcidev, PCI_EXP_LNKCTL2, &lnkctl2); + ret = pcie_lnkctl2_clear_and_set(dd->pcidev, PCI_EXP_LNKCTL2_TLS, + target_vector); if (ret) { - dd_dev_err(dd, "Unable to read from PCI config\n"); - return_error = 1; - goto done; - } - - dd_dev_info(dd, "%s: ..old link control2: 0x%x\n", __func__, - (u32)lnkctl2); - lnkctl2 &= ~PCI_EXP_LNKCTL2_TLS; - lnkctl2 |= target_vector; - dd_dev_info(dd, "%s: ..new link control2: 0x%x\n", __func__, - (u32)lnkctl2); - ret = pcie_capability_write_word(dd->pcidev, PCI_EXP_LNKCTL2, lnkctl2); - if (ret) { - dd_dev_err(dd, "Unable to write to PCI config\n"); + dd_dev_err(dd, "Unable to change PCI link control2\n"); return_error = 1; goto done; }