From patchwork Fri Nov 18 23:09:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 22636 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp460062wrr; Fri, 18 Nov 2022 15:23:11 -0800 (PST) X-Google-Smtp-Source: AA0mqf4duJd6IQolLArbO9cg+tT04K6ksMNOLmsVJaeEhwySaGzNZRTbuA0IChRvd5D9PniYUwEm X-Received: by 2002:a17:906:6d88:b0:7ad:b86b:3ff with SMTP id h8-20020a1709066d8800b007adb86b03ffmr8056999ejt.448.1668813791676; Fri, 18 Nov 2022 15:23:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668813791; cv=none; d=google.com; s=arc-20160816; b=VJZ6wYH482k4VyEK+sBZ6QqevFTdIvVnxv8U1b9gi8qn0XuAwWcUfmSrntHJW8iCYY NsTU30aImA6scmabuxhxqCdcP7r7basI9mgPVR+Pzbu7bupF7u4Z8woky37YUSaeSomp gz3UjGBHXmm0qzC7rLjjDseGaAXP5gWwoR7NKlJ4qm8vdGvrCck81pqGQ78Ls3hpvBXL bEd1CIH8E84IyFggaurIm7iSVrKDLtnm4WA6L5c4OZlYFbISzG8cHWnj9xqS4WxOHlAy ue8Mbn+09IUa/3P4429F7s4a4o44jyGVPQgBODp4b5pypVKvJgOi5MCNn0Mol4P/XXsa LU/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:subject:cc:to:from:date:dkim-signature; bh=fqxYlPjStId8INOMKlUIJ9udLA0AzXzEw7XWCP07sCA=; b=TfIKew4Td35zfWWNgvdHeHygjaEhZdgsBYcWiCbpkV0cN+3zeyK03Jd8yNA5yC8mht EPptf+5KT8uUSiwNDTAp08rF4+nPav2z6Z50hCeUCCR037T6qK7eeqjc0r69ibCo9Dig /PpVlYYyDecpm+OcEd7xXD75KR607cB9tuDVcxsvVhFcLlcgX9PYv72F1U7EQFRcPP8H ZVwh8tebwQ/G/RRXquPIrk+xFqJlZld9pb17jobnu7SvIRm/ueMW8KvpF9bcPrCCqjYN qngaPEFdyuV/CXFAyUXmFZcIEUkKGasCpJDSvieDI6LHu67qMUwmp3BavMZbK6AaggeH AaUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShB2nT5h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u1-20020a1709064ac100b0078c3197bf86si3385928ejt.533.2022.11.18.15.22.45; Fri, 18 Nov 2022 15:23:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShB2nT5h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229719AbiKRXVR (ORCPT + 99 others); Fri, 18 Nov 2022 18:21:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231790AbiKRXU5 (ORCPT ); Fri, 18 Nov 2022 18:20:57 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71E2F7C6AF for ; Fri, 18 Nov 2022 15:09:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668812961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fqxYlPjStId8INOMKlUIJ9udLA0AzXzEw7XWCP07sCA=; b=ShB2nT5hZ9TmyBLg70rRqZJNTRlu0vl02lTtMRH2+0j8qi4E3gyKzf6dpHkdC3gvyI5V5J EYsZxmrROWx7cyioIf9p91Bhw5StM/A4w0aP1Gop2wdnAeshK4epd+mlxPGwCwlQUakZRk g0W3Be7bdZlFfWHVP8G1JWDbxitc8rk= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-347-zZhdaS2ZMiOzxxwsoT0uFg-1; Fri, 18 Nov 2022 18:09:20 -0500 X-MC-Unique: zZhdaS2ZMiOzxxwsoT0uFg-1 Received: by mail-io1-f72.google.com with SMTP id y5-20020a056602120500b006cf628c14ddso3358089iot.15 for ; Fri, 18 Nov 2022 15:09:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fqxYlPjStId8INOMKlUIJ9udLA0AzXzEw7XWCP07sCA=; b=mbO232NvvydxzENQtxJBrcC5AUSQ3iTV25+93xIDzwpSSNaKm9F3/1EqSU0JCm9swf AE5sKdt184dRRMaGMJCuXugiPVobpB2qQiS0XXJiYz6R0U9UrdgJkPhbRcgqdd192SV0 rwt3qblI326brhRuoEbsiQm2svV5Xgt5ocoItVT4zdTL24eA0ovVUDAQQcvedEAt5/H7 y+ziNIhjUfiRhYBYvl7wPU2qSuBQuqWbdg8k1wHm+LIFfeyPEB16M6JWVGV64BFOAS2V IQKEint//0ZQmwkJh29pehV31Yo234SmTenKUNmFfFpLZPkJL+StSe18N4EzNVEPGpf5 mLxw== X-Gm-Message-State: ANoB5pnjSp/+zPy9ldbm29ZH+3v/X6aaN7vEYbhxuW9vJV08ddTg/Gem Wc6LHDV7OX1n2XipRm66IVhvP+KMcCQw9ERgQzqQkkgIy7o6E7cpewcBNr97f9i1joAXfdgG7gu F9GGcaMAMX0PjDJp0+Szgz+PN X-Received: by 2002:a05:6638:e8e:b0:365:ca83:bafb with SMTP id p14-20020a0566380e8e00b00365ca83bafbmr4266499jas.272.1668812959485; Fri, 18 Nov 2022 15:09:19 -0800 (PST) X-Received: by 2002:a05:6638:e8e:b0:365:ca83:bafb with SMTP id p14-20020a0566380e8e00b00365ca83bafbmr4266495jas.272.1668812959204; Fri, 18 Nov 2022 15:09:19 -0800 (PST) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id g17-20020a056e02131100b0030249f369f7sm1631332ilr.82.2022.11.18.15.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 15:09:18 -0800 (PST) Date: Fri, 18 Nov 2022 16:09:16 -0700 From: Alex Williamson To: "linux-pci@vger.kernel.org" Cc: , christian.koenig@amd.com Subject: [RFC] Resizable BARs vs bridges with BARs Message-ID: <20221118160916.7e165306.alex.williamson@redhat.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.34; x86_64-redhat-linux-gnu) MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749878090209813142?= X-GMAIL-MSGID: =?utf-8?q?1749878090209813142?= Hi, I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR: pci_bus 0000:5d: root bus resource [io 0x8000-0x9fff window] pci_bus 0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window] pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<< pci_bus 0000:5d: root bus resource [bus 5d-7f] But resizing fails with -ENOSPC. The topology looks like this: +-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380] \-04.0-[61]----00.0 Intel Corporation Device 4f92 The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture: 5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode]) Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M] Kernel driver in use: pcieport The trouble comes on the upstream PCIe switch port: 5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode]) >>> Region 0: Memory at b010000000 (64-bit, prefetchable) Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Kernel driver in use: pcieport Note region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU: 60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller]) Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M] Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M] Expansion ROM at [disabled] Capabilities: [420 v1] Physical Resizable BAR BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB It's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device. In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a pci_request_region() or equivalent by looking for the IORESOURCE_BUSY flag, but I also suspect this is potentially racy. The patch below works for me, allowing the new resourceN_resize sysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks, Alex diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index b4096598dbcb..8c332a08174d 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -2137,13 +2137,19 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type) next = bridge; do { bridge = next; - for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; + for (i = PCI_STD_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; i++) { struct resource *res = &bridge->resource[i]; if ((res->flags ^ type) & PCI_RES_TYPE_MASK) continue; + if (i < PCI_STD_NUM_BARS) { + if (!(res->flags & IORESOURCE_BUSY)) + pci_release_resource(bridge, i); + continue; + } + /* Ignore BARs which are still in use */ if (res->child) continue;