Message ID | 20230306151014.60913-2-schnelle@linux.ibm.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1905915wrd; Mon, 6 Mar 2023 07:31:50 -0800 (PST) X-Google-Smtp-Source: AK7set9Znp2Tuf/5Gn/zStTbLgpmcVpk1HtT3YbrXqpNZz5iV/2/OS1PTeKmOuN3B1Ro9o2e8biF X-Received: by 2002:a17:906:b791:b0:8e3:8543:8e71 with SMTP id dt17-20020a170906b79100b008e385438e71mr10838597ejb.40.1678116709929; Mon, 06 Mar 2023 07:31:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678116709; cv=none; d=google.com; s=arc-20160816; b=VUy5wd2l+0P3Lj2hVDdV2R3nJivmhR1qgQlUSEzJSFapbzlgk3SVWWY6MJ0KkU1bz7 jJ8N2yW/yBnbtoo53oJ+G6BpeUYmDrnIGXmqSWzXtFzEAkDIAV8uBl+WKVbQmrLJ1ZQq CkKwGbTXv6EdyyfGOuNTUUeVC5CvLqwWweaQTtl279m44SVM4hOMIqKPAMMqVrEE9C7b 6THkuMJx4l6MS17wjMkqZtVugAxzCEaywSOBlWbYE3dhsOM1DjPYqUxKmBQ3MHtqIlAI 0ZePa8hLvvUciwE8Aul6ZhS9MxiE7B7cwuoVPF240L3vW/RrWzLKEpwCIL5DzLVKNiRg sIPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TwW+Or2S2PLq3VbE/F7b+vYT6TztXbk2jXK5TMJEAR4=; b=dtSDi1FvaY5FT729b/r2H9IT2/25o1itDdsse4Js98QEvuq573ftn/hzu9dqyTFh6s ZAj9FWdzurH0kMtxWmZVmkEz3fEPJrV1BOjXtvZLg1s4XsR/Nulyt4oe8nlpTABz4NmL owMloYJdnv8+ljN0pRTYmUoIMwrZSRL2j2V0w+F284uExWmWwe2IZNk7mLWqurVd13cm yy5qJzE7e8O1P1A9OyLUK59eB2b4bXZpAPI0z1sU5aRrSaqB+Jt1GnV7hw+pqgPTrtpl Gg9rBh3wf6iRMHF53sbTzsCjY55xBu5DCGmQCLGS7fbIeB7SVQ9csbA5kR9TXvsSc7F8 kzQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=T9cTylv2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i9-20020a50fc09000000b004bbae097f61si10570498edr.210.2023.03.06.07.31.26; Mon, 06 Mar 2023 07:31:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=T9cTylv2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231151AbjCFPKd (ORCPT <rfc822;toshivichauhan@gmail.com> + 99 others); Mon, 6 Mar 2023 10:10:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229706AbjCFPK2 (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Mon, 6 Mar 2023 10:10:28 -0500 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D44E121971; Mon, 6 Mar 2023 07:10:22 -0800 (PST) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 326E7VMB001475; Mon, 6 Mar 2023 15:10:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=TwW+Or2S2PLq3VbE/F7b+vYT6TztXbk2jXK5TMJEAR4=; b=T9cTylv2n+qzkXU8eI5bRUvfKIMWMH/aDoZlrGZj9wzXNGiTR/dDzKLX/hYxoZVJXZEL iMNNOtK9A6flJuI2K9q+ROHRRqmQ/vZnEO4rK6A3lp8veNTdiMUdDoT2jSaYmEkdNUNo yosVz8qrs4Hc8C8B1U3/8wnSFmWtOiPzfzJCjTH3Rt8IJ7f8tPaW6uHOaG/Ruhnj0Iar rTebyKmEj1ujxGE8B/fGdAgpQRu8nskjWWM8moMsqvU9EzuB/pMb64hHpcefkhf/4cPP yXjfb6m3tl2vajkD8BYCYcClFurJuFGBSkvfAfVtTH+UafYLFU8e5BruWTgcYu9DyVql Ag== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3p507ny1e2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 06 Mar 2023 15:10:21 +0000 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 326F621S006753; Mon, 6 Mar 2023 15:10:21 GMT Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3p507ny1d7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 06 Mar 2023 15:10:21 +0000 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3265OEpj008153; Mon, 6 Mar 2023 15:10:19 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma02fra.de.ibm.com (PPS) with ESMTPS id 3p419ka9q9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 06 Mar 2023 15:10:19 +0000 Received: from smtpav05.fra02v.mail.ibm.com (smtpav05.fra02v.mail.ibm.com [10.20.54.104]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 326FAFlA50594162 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 6 Mar 2023 15:10:15 GMT Received: from smtpav05.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ABF5720040; Mon, 6 Mar 2023 15:10:15 +0000 (GMT) Received: from smtpav05.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70B822004B; Mon, 6 Mar 2023 15:10:15 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by smtpav05.fra02v.mail.ibm.com (Postfix) with ESMTP; Mon, 6 Mar 2023 15:10:15 +0000 (GMT) From: Niklas Schnelle <schnelle@linux.ibm.com> To: Bjorn Helgaas <bhelgaas@google.com>, Lukas Wunner <lukas@wunner.de> Cc: Gerd Bayer <gbayer@linux.ibm.com>, Matthew Rosato <mjrosato@linux.ibm.com>, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v2 1/4] PCI: s390: Fix use-after-free of PCI resources with per-function hotplug Date: Mon, 6 Mar 2023 16:10:11 +0100 Message-Id: <20230306151014.60913-2-schnelle@linux.ibm.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230306151014.60913-1-schnelle@linux.ibm.com> References: <20230306151014.60913-1-schnelle@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: IZKwZl4NiPoFDH7b7oi1MVyHgfUIlKUc X-Proofpoint-ORIG-GUID: bW0WNXHmHQrET1E7Hu05aAw3XC7y81H8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-06_08,2023-03-06_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 adultscore=0 phishscore=0 mlxlogscore=762 bulkscore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 impostorscore=0 mlxscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2303060133 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759632907496839121?= X-GMAIL-MSGID: =?utf-8?q?1759632907496839121?= |
Series |
PCI: s390: Fix user-after-free and clean up
|
|
Commit Message
Niklas Schnelle
March 6, 2023, 3:10 p.m. UTC
On s390 PCI functions may be hotplugged individually even when they
belong to a multi-function device. In particular on an SR-IOV device VFs
may be removed and later re-added.
In commit a50297cf8235 ("s390/pci: separate zbus creation from
scanning") it was missed however that struct pci_bus and struct
zpci_bus's resource list retained a reference to the PCI functions MMIO
resources even though those resources are released and freed on
hot-unplug. These stale resources may subsequently be claimed when the
PCI function re-appears resulting in use-after-free.
One idea of fixing this use-after-free in s390 specific code that was
investigated was to simply keep resources around from the moment a PCI
function first appeared until the whole virtual PCI bus created for
a multi-function device disappears. The problem with this however is
that due to the requirement of artificial MMIO addreesses (address
cookies) extra logic is then needed to keep the address cookies
compatible on re-plug. At the same time the MMIO resources semantically
belong to the PCI function so tying their lifecycle to the function
seems more logical.
Instead a simpler approach is to remove the resources of an individually
hot-unplugged PCI function from the PCI bus's resource list while
keeping the resources of other PCI functions on the PCI bus untouched.
This is done by introducing pci_bus_remove_resource() to remove an
individual resource. Similarly the resource also needs to be removed
from the struct zpci_bus's resource list. It turns out however, that
there is really no need to add the MMIO resources to the struct
zpci_bus's resource list at all and instead we can simply use the
zpci_bar_struct's resource pointer directly.
Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning")
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
v1 -> v2:
- Remove return at the end of function returning void
arch/s390/pci/pci.c | 16 ++++++++++------
arch/s390/pci/pci_bus.c | 12 +++++-------
arch/s390/pci/pci_bus.h | 3 +--
drivers/pci/bus.c | 21 +++++++++++++++++++++
include/linux/pci.h | 1 +
5 files changed, 38 insertions(+), 15 deletions(-)
Comments
On Mon, Mar 06, 2023 at 04:10:11PM +0100, Niklas Schnelle wrote: > On s390 PCI functions may be hotplugged individually even when they > belong to a multi-function device. In particular on an SR-IOV device VFs > may be removed and later re-added. > > In commit a50297cf8235 ("s390/pci: separate zbus creation from > scanning") it was missed however that struct pci_bus and struct > zpci_bus's resource list retained a reference to the PCI functions MMIO > resources even though those resources are released and freed on > hot-unplug. These stale resources may subsequently be claimed when the > PCI function re-appears resulting in use-after-free. > > One idea of fixing this use-after-free in s390 specific code that was > investigated was to simply keep resources around from the moment a PCI > function first appeared until the whole virtual PCI bus created for > a multi-function device disappears. The problem with this however is > that due to the requirement of artificial MMIO addreesses (address > cookies) extra logic is then needed to keep the address cookies > compatible on re-plug. At the same time the MMIO resources semantically > belong to the PCI function so tying their lifecycle to the function > seems more logical. > > Instead a simpler approach is to remove the resources of an individually > hot-unplugged PCI function from the PCI bus's resource list while > keeping the resources of other PCI functions on the PCI bus untouched. > > This is done by introducing pci_bus_remove_resource() to remove an > individual resource. Similarly the resource also needs to be removed > from the struct zpci_bus's resource list. It turns out however, that > there is really no need to add the MMIO resources to the struct > zpci_bus's resource list at all and instead we can simply use the > zpci_bar_struct's resource pointer directly. > > Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning") > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com> Acked-by: Bjorn Helgaas <bhelgaas@google.com> The meat of this is mostly in s390, so I think it makes more sense to merge via that tree. But let me know if you'd rather that I take it. > --- > v1 -> v2: > - Remove return at the end of function returning void > > arch/s390/pci/pci.c | 16 ++++++++++------ > arch/s390/pci/pci_bus.c | 12 +++++------- > arch/s390/pci/pci_bus.h | 3 +-- > drivers/pci/bus.c | 21 +++++++++++++++++++++ > include/linux/pci.h | 1 + > 5 files changed, 38 insertions(+), 15 deletions(-) > > diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c > index ef38b1514c77..e16afacc8fd1 100644 > --- a/arch/s390/pci/pci.c > +++ b/arch/s390/pci/pci.c > @@ -544,8 +544,7 @@ static struct resource *__alloc_res(struct zpci_dev *zdev, unsigned long start, > return r; > } > > -int zpci_setup_bus_resources(struct zpci_dev *zdev, > - struct list_head *resources) > +int zpci_setup_bus_resources(struct zpci_dev *zdev) > { > unsigned long addr, size, flags; > struct resource *res; > @@ -581,7 +580,6 @@ int zpci_setup_bus_resources(struct zpci_dev *zdev, > return -ENOMEM; > } > zdev->bars[i].res = res; > - pci_add_resource(resources, res); > } > zdev->has_resources = 1; > > @@ -590,17 +588,23 @@ int zpci_setup_bus_resources(struct zpci_dev *zdev, > > static void zpci_cleanup_bus_resources(struct zpci_dev *zdev) > { > + struct resource *res; > int i; > > + pci_lock_rescan_remove(); > for (i = 0; i < PCI_STD_NUM_BARS; i++) { > - if (!zdev->bars[i].size || !zdev->bars[i].res) > + res = zdev->bars[i].res; > + if (!res) > continue; > > + release_resource(res); > + pci_bus_remove_resource(zdev->zbus->bus, res); > zpci_free_iomap(zdev, zdev->bars[i].map_idx); > - release_resource(zdev->bars[i].res); > - kfree(zdev->bars[i].res); > + zdev->bars[i].res = NULL; > + kfree(res); > } > zdev->has_resources = 0; > + pci_unlock_rescan_remove(); > } > > int pcibios_device_add(struct pci_dev *pdev) > diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c > index 6a8da1b742ae..a99926af2b69 100644 > --- a/arch/s390/pci/pci_bus.c > +++ b/arch/s390/pci/pci_bus.c > @@ -41,9 +41,7 @@ static int zpci_nb_devices; > */ > static int zpci_bus_prepare_device(struct zpci_dev *zdev) > { > - struct resource_entry *window, *n; > - struct resource *res; > - int rc; > + int rc, i; > > if (!zdev_enabled(zdev)) { > rc = zpci_enable_device(zdev); > @@ -57,10 +55,10 @@ static int zpci_bus_prepare_device(struct zpci_dev *zdev) > } > > if (!zdev->has_resources) { > - zpci_setup_bus_resources(zdev, &zdev->zbus->resources); > - resource_list_for_each_entry_safe(window, n, &zdev->zbus->resources) { > - res = window->res; > - pci_bus_add_resource(zdev->zbus->bus, res, 0); > + zpci_setup_bus_resources(zdev); > + for (i = 0; i < PCI_STD_NUM_BARS; i++) { > + if (zdev->bars[i].res) > + pci_bus_add_resource(zdev->zbus->bus, zdev->bars[i].res, 0); > } > } > > diff --git a/arch/s390/pci/pci_bus.h b/arch/s390/pci/pci_bus.h > index e96c9860e064..af9f0ac79a1b 100644 > --- a/arch/s390/pci/pci_bus.h > +++ b/arch/s390/pci/pci_bus.h > @@ -30,8 +30,7 @@ static inline void zpci_zdev_get(struct zpci_dev *zdev) > > int zpci_alloc_domain(int domain); > void zpci_free_domain(int domain); > -int zpci_setup_bus_resources(struct zpci_dev *zdev, > - struct list_head *resources); > +int zpci_setup_bus_resources(struct zpci_dev *zdev); > > static inline struct zpci_dev *zdev_from_bus(struct pci_bus *bus, > unsigned int devfn) > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c > index 83ae838ceb5f..549c4bd5caec 100644 > --- a/drivers/pci/bus.c > +++ b/drivers/pci/bus.c > @@ -76,6 +76,27 @@ struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n) > } > EXPORT_SYMBOL_GPL(pci_bus_resource_n); > > +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res) > +{ > + struct pci_bus_resource *bus_res, *tmp; > + int i; > + > + for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) { > + if (bus->resource[i] == res) { > + bus->resource[i] = NULL; > + return; > + } > + } > + > + list_for_each_entry_safe(bus_res, tmp, &bus->resources, list) { > + if (bus_res->res == res) { > + list_del(&bus_res->list); > + kfree(bus_res); > + return; > + } > + } > +} > + > void pci_bus_remove_resources(struct pci_bus *bus) > { > int i; > diff --git a/include/linux/pci.h b/include/linux/pci.h > index fafd8020c6d7..b50e5c79f7e3 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -1438,6 +1438,7 @@ void pci_bus_add_resource(struct pci_bus *bus, struct resource *res, > unsigned int flags); > struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n); > void pci_bus_remove_resources(struct pci_bus *bus); > +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res); > int devm_request_pci_bus_resources(struct device *dev, > struct list_head *resources); > > -- > 2.37.2 >
On Wed, 2023-03-08 at 17:14 -0600, Bjorn Helgaas wrote: > On Mon, Mar 06, 2023 at 04:10:11PM +0100, Niklas Schnelle wrote: > > On s390 PCI functions may be hotplugged individually even when they > > belong to a multi-function device. In particular on an SR-IOV device VFs > > may be removed and later re-added. > > > > In commit a50297cf8235 ("s390/pci: separate zbus creation from > > scanning") it was missed however that struct pci_bus and struct > > zpci_bus's resource list retained a reference to the PCI functions MMIO > > resources even though those resources are released and freed on > > hot-unplug. These stale resources may subsequently be claimed when the > > PCI function re-appears resulting in use-after-free. > > > > One idea of fixing this use-after-free in s390 specific code that was > > investigated was to simply keep resources around from the moment a PCI > > function first appeared until the whole virtual PCI bus created for > > a multi-function device disappears. The problem with this however is > > that due to the requirement of artificial MMIO addreesses (address > > cookies) extra logic is then needed to keep the address cookies > > compatible on re-plug. At the same time the MMIO resources semantically > > belong to the PCI function so tying their lifecycle to the function > > seems more logical. > > > > Instead a simpler approach is to remove the resources of an individually > > hot-unplugged PCI function from the PCI bus's resource list while > > keeping the resources of other PCI functions on the PCI bus untouched. > > > > This is done by introducing pci_bus_remove_resource() to remove an > > individual resource. Similarly the resource also needs to be removed > > from the struct zpci_bus's resource list. It turns out however, that > > there is really no need to add the MMIO resources to the struct > > zpci_bus's resource list at all and instead we can simply use the > > zpci_bar_struct's resource pointer directly. > > > > Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning") > > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com> > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > > The meat of this is mostly in s390, so I think it makes more sense to > merge via that tree. But let me know if you'd rather that I take it. > > Thanks for taking a look and the valuable suggestions. I'll coordinate with Vasily to take this via the s390 tree. As for the locking I agree it is out of scope for this series. Meant more that the resource handling might be a good place to start splitting up the pci_rescan_remove_lock and that I might take a look at that if I find the time which of course we're all lacking. Regards, Niklas
On 3/6/23 10:10 AM, Niklas Schnelle wrote: > On s390 PCI functions may be hotplugged individually even when they > belong to a multi-function device. In particular on an SR-IOV device VFs > may be removed and later re-added. > > In commit a50297cf8235 ("s390/pci: separate zbus creation from > scanning") it was missed however that struct pci_bus and struct > zpci_bus's resource list retained a reference to the PCI functions MMIO > resources even though those resources are released and freed on > hot-unplug. These stale resources may subsequently be claimed when the > PCI function re-appears resulting in use-after-free. > > One idea of fixing this use-after-free in s390 specific code that was > investigated was to simply keep resources around from the moment a PCI > function first appeared until the whole virtual PCI bus created for > a multi-function device disappears. The problem with this however is > that due to the requirement of artificial MMIO addreesses (address > cookies) extra logic is then needed to keep the address cookies > compatible on re-plug. At the same time the MMIO resources semantically > belong to the PCI function so tying their lifecycle to the function > seems more logical. > > Instead a simpler approach is to remove the resources of an individually > hot-unplugged PCI function from the PCI bus's resource list while > keeping the resources of other PCI functions on the PCI bus untouched. > > This is done by introducing pci_bus_remove_resource() to remove an > individual resource. Similarly the resource also needs to be removed > from the struct zpci_bus's resource list. It turns out however, that > there is really no need to add the MMIO resources to the struct > zpci_bus's resource list at all and instead we can simply use the > zpci_bar_struct's resource pointer directly. > > Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning") > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com> Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
On Wed, Mar 08, 2023 at 05:14:49PM -0600, Bjorn Helgaas wrote: > On Mon, Mar 06, 2023 at 04:10:11PM +0100, Niklas Schnelle wrote: > > On s390 PCI functions may be hotplugged individually even when they > > belong to a multi-function device. In particular on an SR-IOV device VFs > > may be removed and later re-added. > > > > In commit a50297cf8235 ("s390/pci: separate zbus creation from > > scanning") it was missed however that struct pci_bus and struct > > zpci_bus's resource list retained a reference to the PCI functions MMIO > > resources even though those resources are released and freed on > > hot-unplug. These stale resources may subsequently be claimed when the > > PCI function re-appears resulting in use-after-free. > > > > One idea of fixing this use-after-free in s390 specific code that was > > investigated was to simply keep resources around from the moment a PCI > > function first appeared until the whole virtual PCI bus created for > > a multi-function device disappears. The problem with this however is > > that due to the requirement of artificial MMIO addreesses (address > > cookies) extra logic is then needed to keep the address cookies > > compatible on re-plug. At the same time the MMIO resources semantically > > belong to the PCI function so tying their lifecycle to the function > > seems more logical. > > > > Instead a simpler approach is to remove the resources of an individually > > hot-unplugged PCI function from the PCI bus's resource list while > > keeping the resources of other PCI functions on the PCI bus untouched. > > > > This is done by introducing pci_bus_remove_resource() to remove an > > individual resource. Similarly the resource also needs to be removed > > from the struct zpci_bus's resource list. It turns out however, that > > there is really no need to add the MMIO resources to the struct > > zpci_bus's resource list at all and instead we can simply use the > > zpci_bar_struct's resource pointer directly. > > > > Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning") > > Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com> > > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > > The meat of this is mostly in s390, so I think it makes more sense to > merge via that tree. But let me know if you'd rather that I take it. I'll take it via s390 tree. Thanks
On Mon, Mar 06, 2023 at 04:10:11PM +0100, Niklas Schnelle wrote: > --- a/drivers/pci/bus.c > +++ b/drivers/pci/bus.c > @@ -76,6 +76,27 @@ struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n) > } > EXPORT_SYMBOL_GPL(pci_bus_resource_n); > > +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res) > +{ > + struct pci_bus_resource *bus_res, *tmp; > + int i; > + > + for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) { > + if (bus->resource[i] == res) { > + bus->resource[i] = NULL; > + return; > + } > + } > + > + list_for_each_entry_safe(bus_res, tmp, &bus->resources, list) { > + if (bus_res->res == res) { > + list_del(&bus_res->list); > + kfree(bus_res); > + return; > + } > + } > +} I realize this has already been applied so s390.git/master, but nevertheless would like to point out there's a handy pci_bus_for_each_resource() helper which could have been used here instead of the for-loop. Sorry for chiming in late, I just spotted this while flushing out my inbox. Adding Andy to cc: who's been active in this area recently. Thanks, Lukas
On Mon, Apr 17, 2023 at 10:46 AM Lukas Wunner <lukas@wunner.de> wrote: > On Mon, Mar 06, 2023 at 04:10:11PM +0100, Niklas Schnelle wrote: ... > > +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res) > > +{ > > + struct pci_bus_resource *bus_res, *tmp; > > + int i; > > + > > + for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) { > > + if (bus->resource[i] == res) { > > + bus->resource[i] = NULL; > > + return; > > + } > > + } > > + > > + list_for_each_entry_safe(bus_res, tmp, &bus->resources, list) { > > + if (bus_res->res == res) { > > + list_del(&bus_res->list); > > + kfree(bus_res); > > + return; > > + } > > + } > > +} > > I realize this has already been applied so s390.git/master, > but nevertheless would like to point out there's a handy > pci_bus_for_each_resource() helper which could have been > used here instead of the for-loop. Actually in this case it's not possible. The above code nullifies the matched resource or removes it from the list. We don't have any good iterator inside pci_bus_for_each_resource() to do the latter. There might be other places in the kernel code where something like above is used and we can split a separate helper exactly for matching cases, but with the above context I dunno we need to do anything right now. P.S. Thanks for Cc'ing me.
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index ef38b1514c77..e16afacc8fd1 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -544,8 +544,7 @@ static struct resource *__alloc_res(struct zpci_dev *zdev, unsigned long start, return r; } -int zpci_setup_bus_resources(struct zpci_dev *zdev, - struct list_head *resources) +int zpci_setup_bus_resources(struct zpci_dev *zdev) { unsigned long addr, size, flags; struct resource *res; @@ -581,7 +580,6 @@ int zpci_setup_bus_resources(struct zpci_dev *zdev, return -ENOMEM; } zdev->bars[i].res = res; - pci_add_resource(resources, res); } zdev->has_resources = 1; @@ -590,17 +588,23 @@ int zpci_setup_bus_resources(struct zpci_dev *zdev, static void zpci_cleanup_bus_resources(struct zpci_dev *zdev) { + struct resource *res; int i; + pci_lock_rescan_remove(); for (i = 0; i < PCI_STD_NUM_BARS; i++) { - if (!zdev->bars[i].size || !zdev->bars[i].res) + res = zdev->bars[i].res; + if (!res) continue; + release_resource(res); + pci_bus_remove_resource(zdev->zbus->bus, res); zpci_free_iomap(zdev, zdev->bars[i].map_idx); - release_resource(zdev->bars[i].res); - kfree(zdev->bars[i].res); + zdev->bars[i].res = NULL; + kfree(res); } zdev->has_resources = 0; + pci_unlock_rescan_remove(); } int pcibios_device_add(struct pci_dev *pdev) diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c index 6a8da1b742ae..a99926af2b69 100644 --- a/arch/s390/pci/pci_bus.c +++ b/arch/s390/pci/pci_bus.c @@ -41,9 +41,7 @@ static int zpci_nb_devices; */ static int zpci_bus_prepare_device(struct zpci_dev *zdev) { - struct resource_entry *window, *n; - struct resource *res; - int rc; + int rc, i; if (!zdev_enabled(zdev)) { rc = zpci_enable_device(zdev); @@ -57,10 +55,10 @@ static int zpci_bus_prepare_device(struct zpci_dev *zdev) } if (!zdev->has_resources) { - zpci_setup_bus_resources(zdev, &zdev->zbus->resources); - resource_list_for_each_entry_safe(window, n, &zdev->zbus->resources) { - res = window->res; - pci_bus_add_resource(zdev->zbus->bus, res, 0); + zpci_setup_bus_resources(zdev); + for (i = 0; i < PCI_STD_NUM_BARS; i++) { + if (zdev->bars[i].res) + pci_bus_add_resource(zdev->zbus->bus, zdev->bars[i].res, 0); } } diff --git a/arch/s390/pci/pci_bus.h b/arch/s390/pci/pci_bus.h index e96c9860e064..af9f0ac79a1b 100644 --- a/arch/s390/pci/pci_bus.h +++ b/arch/s390/pci/pci_bus.h @@ -30,8 +30,7 @@ static inline void zpci_zdev_get(struct zpci_dev *zdev) int zpci_alloc_domain(int domain); void zpci_free_domain(int domain); -int zpci_setup_bus_resources(struct zpci_dev *zdev, - struct list_head *resources); +int zpci_setup_bus_resources(struct zpci_dev *zdev); static inline struct zpci_dev *zdev_from_bus(struct pci_bus *bus, unsigned int devfn) diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c index 83ae838ceb5f..549c4bd5caec 100644 --- a/drivers/pci/bus.c +++ b/drivers/pci/bus.c @@ -76,6 +76,27 @@ struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n) } EXPORT_SYMBOL_GPL(pci_bus_resource_n); +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res) +{ + struct pci_bus_resource *bus_res, *tmp; + int i; + + for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) { + if (bus->resource[i] == res) { + bus->resource[i] = NULL; + return; + } + } + + list_for_each_entry_safe(bus_res, tmp, &bus->resources, list) { + if (bus_res->res == res) { + list_del(&bus_res->list); + kfree(bus_res); + return; + } + } +} + void pci_bus_remove_resources(struct pci_bus *bus) { int i; diff --git a/include/linux/pci.h b/include/linux/pci.h index fafd8020c6d7..b50e5c79f7e3 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1438,6 +1438,7 @@ void pci_bus_add_resource(struct pci_bus *bus, struct resource *res, unsigned int flags); struct resource *pci_bus_resource_n(const struct pci_bus *bus, int n); void pci_bus_remove_resources(struct pci_bus *bus); +void pci_bus_remove_resource(struct pci_bus *bus, struct resource *res); int devm_request_pci_bus_resources(struct device *dev, struct list_head *resources);