From patchwork Thu Nov 24 14:25:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johan Hovold X-Patchwork-Id: 25584 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3436080wrr; Thu, 24 Nov 2022 06:40:16 -0800 (PST) X-Google-Smtp-Source: AA0mqf5WC7uwksD0bZJChAMi0Cbt5fiiEvyXoGnS7OV6+ZOrgcp5EMyAB9OmUDuvGamydf9Ir7ub X-Received: by 2002:a17:90a:ae0f:b0:20d:b124:33b1 with SMTP id t15-20020a17090aae0f00b0020db12433b1mr35140774pjq.202.1669300816487; Thu, 24 Nov 2022 06:40:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669300816; cv=none; d=google.com; s=arc-20160816; b=kt6GbdTZkJjO51BurxgCIR+QVsTJ9ZWOC0bfyV2s+MyHGesJDjOV6HR4+3Zgx4yBP8 Tcgw7K+eRQni/GJLCE22DWrp7SBEQgyzJkzAu0wtI5ZJ2Cs8g94UTnjWVacAl8Klw+wK MlyZy0KYfAXZiZK1CBnLL5err5m2q4JRF0LGLxc/WhpZ3cwirffQSxHA1i92hN5ZZ4zw X0m3tr5vm9ywEjd32jyUeinkTyN/RzJRrybfiQwuNyJ4/s/MDxYMvbk90nbE/pVM5YYz H+Gt89MR+wtVzM6ST8KaZG+KaATXnjPiikjgSGx7FLUVFDXE0qHvHqU/wOMceqjZunu9 x8FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=ZM/uBL8+A/tt1id2NlfL0Vh1wniBY29MzsWMlVvAyFw=; b=J531B3liU5z0xF1aL1eKGVPHbw0maW5Z0ZPWeE3P39Xf9sGqvymAuRMuRKc9JTcov1 A6/+NjhdEruLAc9A2gf2bjHYzrcZjsZB2cBQn+rIbL4DnYCsiRYrvdK9V5TufPih1Bl/ lnThM64ZUJrEtgx5gHKW1SSfp7nNHKmgcqII4YleMSGXjaYPRJhHnktHNOU4EkMloMVF FCkdYg/P6BAJfqMa4YrIeOsM38tyqKT7WVI5i8zrfdvj+axDnpX81f4Jgo2kPEBudn/6 EenmKCiYPG1TppWrv7Aw0U6qjkqdBmMYG2rbAoln+04wAEQ03RanHuP+obNv2atEiBCy YQRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MbShXb8E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ju19-20020a170903429300b00179faf5c34asi1115658plb.379.2022.11.24.06.40.02; Thu, 24 Nov 2022 06:40:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MbShXb8E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229681AbiKXOaB (ORCPT + 99 others); Thu, 24 Nov 2022 09:30:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229379AbiKXOaA (ORCPT ); Thu, 24 Nov 2022 09:30:00 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76E9F7213E; Thu, 24 Nov 2022 06:29:59 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1E42AB82835; Thu, 24 Nov 2022 14:29:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9307C433D6; Thu, 24 Nov 2022 14:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669300196; bh=hIeGLPE7dl6RAVsZMh0WHZl4Pd+eFlTWe3fogJ7tYp8=; h=From:To:Cc:Subject:Date:From; b=MbShXb8EdixsZ3fJ0JShJ3FyadhLA0/Uo4HMvt9mdlnloJbIBTG0pIzZG50manRt7 SNLhQBN0bxsF/UVdDL2uQOKniSaiMGutZyosktOPjQQTq3ZVfZGcjOf6e/tjj/ZGaI uMTQ2sOAKuBqujLw4DP8P98ChJUAZ8cwkWx+/nyck1zuhiPM46y0fuEcQD7WZkjwlT i447f/Ho97jimg+Sl9FS8MbHifLK4Ws0bQ0GqZEabHYswZVI0hCZ3t5U94LHjxBHn8 yYb1nC68rz3qiKDH9bnRK8h/BLci+K5RnvWm0cf0jAAL92N8jH+tEnv3jQejzUX4TF GO0vhyHrpFymA== Received: from johan by xi.lan with local (Exim 4.94.2) (envelope-from ) id 1oyDEH-0007fo-H2; Thu, 24 Nov 2022 15:29:30 +0100 From: Johan Hovold To: Bjorn Andersson Cc: Andy Gross , Konrad Dybcio , Rob Herring , Krzysztof Kozlowski , Manivannan Sadhasivam , Will Deacon , Robin Murphy , Christoph Hellwig , Ard Biesheuvel , Catalin Marinas , linux-arm-msm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Johan Hovold Subject: [PATCH] arm64: dts: qcom: sc8280xp: fix PCIe DMA coherency Date: Thu, 24 Nov 2022 15:25:01 +0100 Message-Id: <20221124142501.29314-1-johan+linaro@kernel.org> X-Mailer: git-send-email 2.37.4 MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750388773230220969?= X-GMAIL-MSGID: =?utf-8?q?1750388773230220969?= The devices on the SC8280XP PCIe buses are cache coherent and must be marked as such to avoid data corruption. A coherent device can, for example, end up snooping stale data from the caches instead of using data written by the CPU through the non-cacheable mapping which is used for consistent DMA buffers for non-coherent devices. Note that this is much more likely to happen since commit c44094eee32f ("arm64: dma: Drop cache invalidation from arch_dma_prep_coherent()") that was added in 6.1 and which removed the cache invalidation when setting up the non-cacheable mapping. Marking the PCIe devices as coherent specifically fixes the intermittent NVMe probe failures observed on the Thinkpad X13s, which was due to corruption of the submission and completion queues. This was typically observed as corruption of the admin submission queue (with well-formed completion): could not locate request for tag 0x0 nvme nvme0: invalid id 0 completed on queue 0 or corruption of the admin or I/O completion queues (malformed completion): could not locate request for tag 0x45f nvme nvme0: invalid id 25695 completed on queue 25965 presumably as these queues are small enough to not be allocated using CMA which in turn make them more likely to be cached (e.g. due to accesses to nearby pages through the cacheable linear map). Increasing the buffer sizes to two pages to force CMA allocation also appears to make the problem go away. Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes") Signed-off-by: Johan Hovold Reviewed-by: Konrad Dybcio Reviewed-by: Manivannan Sadhasivam --- arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi index 27f5c2f82338..7748cd29276d 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi +++ b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi @@ -854,6 +854,8 @@ pcie4: pcie@1c00000 { <0x02000000 0x0 0x30300000 0x0 0x30300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <6>; num-lanes = <1>; @@ -951,6 +953,8 @@ pcie3b: pcie@1c08000 { <0x02000000 0x0 0x32300000 0x0 0x32300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <5>; num-lanes = <2>; @@ -1046,6 +1050,8 @@ pcie3a: pcie@1c10000 { <0x02000000 0x0 0x34300000 0x0 0x34300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <4>; num-lanes = <4>; @@ -1144,6 +1150,8 @@ pcie2b: pcie@1c18000 { <0x02000000 0x0 0x38300000 0x0 0x38300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <3>; num-lanes = <2>; @@ -1239,6 +1247,8 @@ pcie2a: pcie@1c20000 { <0x02000000 0x0 0x3c300000 0x0 0x3c300000 0x0 0x1d00000>; bus-range = <0x00 0xff>; + dma-coherent; + linux,pci-domain = <2>; num-lanes = <4>;