From patchwork Mon Jan 15 18:54:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188309 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887156dyc; Mon, 15 Jan 2024 10:56:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IHJd6R8lm17WXs3b3R3XuAXnS+gJ8NAPRrJQnrqqTdoQQGGrwCBS8nv29aWhGc9tPX36cBL X-Received: by 2002:a50:9fcb:0:b0:559:6f91:4c6d with SMTP id c69-20020a509fcb000000b005596f914c6dmr434172edf.26.1705344968851; Mon, 15 Jan 2024 10:56:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705344968; cv=none; d=google.com; s=arc-20160816; b=quF0Du0Kwdd70OCZAzT9ESqsF8UJGoqyY17QihV74kvBjnjaJG9EhbX7rkFYCLqGAw 8sa9ju84bI7uo0M7wlNDCAQ+L3s0Ec+swjGrUXg+wgqrs8tzTgxXmZjJR7UFeLJ1OLmi 00GGj0fEA2h7k88nnMFvAmVQDwuv6aDDIsdHVN8sqZZdvlmNu9O5gMSXpKZGR/wCiizM A1X5HMr+axtrvrdorlug1PaRwhXjDsqI7J3u9r7YBNZJtTPC3Ylbv7qjlvfrda6XbRAe r8IfWP5ViT8Ht1XsB7YKMzH8HyQKb9wozeSDQidaA8Bl18It9Ywp2aYxaqdq1xbnkKB3 rIvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=cdc69wRJvUREg0pyMQwpjT0CLYOu6Qa/NnRT4JAlqmc=; fh=H6lVIw+KZeomf8cbJ+jDGydQlHrKW9Bp+9pD33yGxs8=; b=X3HE7VUpDQoD7seFc+dMRGv9zFCTP7lrc0vO8vdnIGa1BhdW+cfRifsf0hy7bzyRne 6c981gtP96vhoj87xP4afEFrY2zYRZJIb1Fmp1C6PspLdjYx30Z8+9mrWjRMlWsSsuib 5Km+k8sPs9Ap0UDhF4E/Uqzs7iCJBQi5YI4fM37vHG51vGZu5jpMCjc/na5Aayj4LBwr P8TU1oGnieX3iFKUN4dfZoqoOOV78eIwe05daFw674nx2rNmxIJv+kqSY4UEOumlBEdA Y0WE7IwEdAzbddd9a5NgeHHBibTBt04ElUXO2KG9ttGTUK0+zF0SDsABSzPz/rtPHlWO TcQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=RLhwY6SJ; spf=pass (google.com: domain of linux-kernel+bounces-26414-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26414-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id em3-20020a056402364300b00558da305541si3087907edb.189.2024.01.15.10.56.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:56:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26414-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=RLhwY6SJ; spf=pass (google.com: domain of linux-kernel+bounces-26414-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26414-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 320A41F21A3C for ; Mon, 15 Jan 2024 18:56:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0A0F219473; Mon, 15 Jan 2024 18:54:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="RLhwY6SJ" Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7252918658; Mon, 15 Jan 2024 18:54:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FISggo016509; Mon, 15 Jan 2024 18:54:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=cdc69wRJvUREg0pyMQwpjT0CLYOu6Qa/NnRT4JAlqmc=; b=RLhwY6SJ07I9fkqzkeIETArkHZWDKpRdSKgRAQ5TPHrDZadRwj/+JaKhiSy6nHOKhd79 3g7qTG/4+9E+GVbHM5NvAFoINAFHhxl6vPZWusplUAE9QtKewKhlBILtxYbXW7kcCc3x 6OG1z1AyH0RvlKIfChA+u/QjO5zL+tk6Ei3PHW29Y0of9JRgoj31iWxK4hNGXd6y+M0E RQKjAtrQMicFrKf6oFjguBOCqZvUTBAUVCg4RFJSCY9r+nLfb8mFE9Hg4IQWdBYXey4p 6b0sx0d7a85vk8DzMXEIHnRq8DwRp8OeXy6NDEtqs0VwQTwA4FT/oWpLaJbYnOCBp0Y6 ng== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd0r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:46 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FIm1nQ029222; Mon, 15 Jan 2024 18:54:46 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd0g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:46 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FImDec026494; Mon, 15 Jan 2024 18:54:45 GMT Received: from smtprelay04.wdc07v.mail.ibm.com ([172.16.1.71]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3vm72jsnjj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:45 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay04.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIshcs40632920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:44 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D3C855805A; Mon, 15 Jan 2024 18:54:43 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D25C258052; Mon, 15 Jan 2024 18:54:42 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:42 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , stable@vger.kernel.org Subject: [PATCH v4 1/6] s390/vfio-ap: always filter entire AP matrix Date: Mon, 15 Jan 2024 13:54:31 -0500 Message-ID: <20240115185441.31526-2-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: yP1PaFeRexguCfiwqnjj1-q439TCwhpQ X-Proofpoint-ORIG-GUID: 0NqZTIsj5sSwZXvO7_D_8A00LU6P00fW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 priorityscore=1501 impostorscore=0 mlxscore=0 spamscore=0 malwarescore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 clxscore=1015 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183806395143193 X-GMAIL-MSGID: 1788183806395143193 From: Tony Krowiak The vfio_ap_mdev_filter_matrix function is called whenever a new adapter or domain is assigned to the mdev. The purpose of the function is to update the guest's AP configuration by filtering the matrix of adapters and domains assigned to the mdev. When an adapter or domain is assigned, only the APQNs associated with the APID of the new adapter or APQI of the new domain are inspected. If an APQN does not reference a queue device bound to the vfio_ap device driver, then it's APID will be filtered from the mdev's matrix when updating the guest's AP configuration. Inspecting only the APID of the new adapter or APQI of the new domain will result in passing AP queues through to a guest that are not bound to the vfio_ap device driver under certain circumstances. Consider the following: guest's AP configuration (all also assigned to the mdev's matrix): 14.0004 14.0005 14.0006 16.0004 16.0005 16.0006 unassign domain 4 unbind queue 16.0005 assign domain 4 When domain 4 is re-assigned, since only domain 4 will be inspected, the APQNs that will be examined will be: 14.0004 16.0004 Since both of those APQNs reference queue devices that are bound to the vfio_ap device driver, nothing will get filtered from the mdev's matrix when updating the guest's AP configuration. Consequently, queue 16.0005 will get passed through despite not being bound to the driver. This violates the linux device model requirement that a guest shall only be given access to devices bound to the device driver facilitating their pass-through. To resolve this problem, every adapter and domain assigned to the mdev will be inspected when filtering the mdev's matrix. Signed-off-by: Tony Krowiak Acked-by: Halil Pasic Fixes: 48cae940c31d ("s390/vfio-ap: refresh guest's APCB by filtering AP resources assigned to mdev") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 57 +++++++++---------------------- 1 file changed, 17 insertions(+), 40 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index acb710d3d7bc..1f7a6c106786 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -674,8 +674,7 @@ static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev) * Return: a boolean value indicating whether the KVM guest's APCB was changed * by the filtering or not. */ -static bool vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm, - struct ap_matrix_mdev *matrix_mdev) +static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev) { unsigned long apid, apqi, apqn; DECLARE_BITMAP(prev_shadow_apm, AP_DEVICES); @@ -696,8 +695,8 @@ static bool vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm, bitmap_and(matrix_mdev->shadow_apcb.aqm, matrix_mdev->matrix.aqm, (unsigned long *)matrix_dev->info.aqm, AP_DOMAINS); - for_each_set_bit_inv(apid, apm, AP_DEVICES) { - for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) { + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { /* * If the APQN is not bound to the vfio_ap device * driver, then we can't assign it to the guest's @@ -962,7 +961,6 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; - DECLARE_BITMAP(apm_delta, AP_DEVICES); struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev); mutex_lock(&ap_perms_mutex); @@ -991,11 +989,8 @@ static ssize_t assign_adapter_store(struct device *dev, } vfio_ap_mdev_link_adapter(matrix_mdev, apid); - memset(apm_delta, 0, sizeof(apm_delta)); - set_bit_inv(apid, apm_delta); - if (vfio_ap_mdev_filter_matrix(apm_delta, - matrix_mdev->matrix.aqm, matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev)) vfio_ap_mdev_update_guest_apcb(matrix_mdev); ret = count; @@ -1171,7 +1166,6 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; - DECLARE_BITMAP(aqm_delta, AP_DOMAINS); struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev); mutex_lock(&ap_perms_mutex); @@ -1200,11 +1194,8 @@ static ssize_t assign_domain_store(struct device *dev, } vfio_ap_mdev_link_domain(matrix_mdev, apqi); - memset(aqm_delta, 0, sizeof(aqm_delta)); - set_bit_inv(apqi, aqm_delta); - if (vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, aqm_delta, - matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev)) vfio_ap_mdev_update_guest_apcb(matrix_mdev); ret = count; @@ -2109,9 +2100,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) if (matrix_mdev) { vfio_ap_mdev_link_queue(matrix_mdev, q); - if (vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, - matrix_mdev->matrix.aqm, - matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev)) vfio_ap_mdev_update_guest_apcb(matrix_mdev); } dev_set_drvdata(&apdev->device, q); @@ -2461,34 +2450,22 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info, static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) { - bool do_hotplug = false; - int filter_domains = 0; - int filter_adapters = 0; - DECLARE_BITMAP(apm, AP_DEVICES); - DECLARE_BITMAP(aqm, AP_DOMAINS); + bool filter_domains, filter_adapters, filter_cdoms, do_hotplug = false; mutex_lock(&matrix_mdev->kvm->lock); mutex_lock(&matrix_dev->mdevs_lock); - filter_adapters = bitmap_and(apm, matrix_mdev->matrix.apm, - matrix_mdev->apm_add, AP_DEVICES); - filter_domains = bitmap_and(aqm, matrix_mdev->matrix.aqm, - matrix_mdev->aqm_add, AP_DOMAINS); - - if (filter_adapters && filter_domains) - do_hotplug |= vfio_ap_mdev_filter_matrix(apm, aqm, matrix_mdev); - else if (filter_adapters) - do_hotplug |= - vfio_ap_mdev_filter_matrix(apm, - matrix_mdev->shadow_apcb.aqm, - matrix_mdev); - else - do_hotplug |= - vfio_ap_mdev_filter_matrix(matrix_mdev->shadow_apcb.apm, - aqm, matrix_mdev); + filter_adapters = bitmap_intersects(matrix_mdev->matrix.apm, + matrix_mdev->apm_add, AP_DEVICES); + filter_domains = bitmap_intersects(matrix_mdev->matrix.aqm, + matrix_mdev->aqm_add, AP_DOMAINS); + filter_cdoms = bitmap_intersects(matrix_mdev->matrix.adm, + matrix_mdev->adm_add, AP_DOMAINS); + + if (filter_adapters || filter_domains) + do_hotplug = vfio_ap_mdev_filter_matrix(matrix_mdev); - if (bitmap_intersects(matrix_mdev->matrix.adm, matrix_mdev->adm_add, - AP_DOMAINS)) + if (filter_cdoms) do_hotplug |= vfio_ap_mdev_filter_cdoms(matrix_mdev); if (do_hotplug) From patchwork Mon Jan 15 18:54:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188310 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887169dyc; Mon, 15 Jan 2024 10:56:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IHUyBJxE6h/pKlP/meU0X1+MDvr3GLMAsK+ci770OTqqPHBr5M9JccOa4W0QWBJCmaiHRX0 X-Received: by 2002:a05:6214:29cd:b0:681:68fd:772f with SMTP id gh13-20020a05621429cd00b0068168fd772fmr593302qvb.97.1705344970340; Mon, 15 Jan 2024 10:56:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705344970; cv=none; d=google.com; s=arc-20160816; b=Wy3OvJTK/5DmEFWp+CU2+oiBRRJBOOXQJIMGBzI//siOpI4hHO5tJE+9SBniQ1Ouk6 V1cKTuHQ7ih2c/ONFqa5E7NsHcN3LmohfKGsBvIFJAg7sS1oIoYGXs9TVokjxTZ9ART5 qBln5+6rgqxxVkJu7Cpxbkt0AkDaFKsbGzkytBnLtHUDs4yCZ1QyCLo1efjd1s6Fvahx 4jKzIthu9gLl7Eo9fQD/1Go5OrfqhVIYMjhXanUwq4+elvbVHd9bht0hkQMTr7PHJNK3 j8iQkKAv9hJ/2iDZntJrUJlObJ2f794rlHNqYd/FaH38vuqvsX44qjCUB19bivwZ82Z6 PQ/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=bKRG8CRUSEfAt7y8e4in2UO7wdpopq3ttdLN1uPxr5c=; fh=H6lVIw+KZeomf8cbJ+jDGydQlHrKW9Bp+9pD33yGxs8=; b=xKhoZT7JaygexzMO1cOdZ9HosvCrd87Bo+Yt91/dScfCuWtth+1IeQWyIfdlTqA3pU a6nUpo+hBIiAXOID4fUoqrUFNFlVN8lI0JoOM4U9AAq2sbpzzIRI2Eeuw+Sli7BKT1v6 yfG71PvUlDDeMym/5vN0gmWG3xBXfcb966bRjD1LcKwoCnCrvCI07/I6m04KSrokOntB 1imDz7RkHtqC+gNHCvLmh74aq3phoVP+9s0ZbF1GruXC4TNRWJuv+halhYgDHDWbTNXD V/x9tF6zlCZAPUzDgAMo5J29FY65lEQoLuE341HoBnMjQnJnJgmAYVN4R31ZO5YAlWdu S0Sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ANJfvFEy; spf=pass (google.com: domain of linux-kernel+bounces-26415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26415-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id u8-20020a0c8dc8000000b00680b1539bdcsi8572426qvb.551.2024.01.15.10.56.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:56:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ANJfvFEy; spf=pass (google.com: domain of linux-kernel+bounces-26415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26415-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 258431C20E6C for ; Mon, 15 Jan 2024 18:56:10 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 428E819479; Mon, 15 Jan 2024 18:54:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ANJfvFEy" Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3E3518B09; Mon, 15 Jan 2024 18:54:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FISfnc016495; Mon, 15 Jan 2024 18:54:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=bKRG8CRUSEfAt7y8e4in2UO7wdpopq3ttdLN1uPxr5c=; b=ANJfvFEyFIzotR4H5ozUTu7vs+uEKlE5GhBbLmSzqqtHV3Ud8vrUa2LAFHGw6QQXTeC6 xA4vYoKokQN4asUbN4zow5wHi0nDKMLrZLiuXr7tPR/UgQGVIpqgB9BOQFOe+fjVpISv 89jC8mUnHtcL+h7ZjVtsGFtmdceNEIKafiqU3C7zTAfb7uVXRgStaykp9DX9zst9fUBo oeJcHqsVrSmQM6C6vyUXV/sc++5Eyx8ibjcEqw6ONbAgZjFWFwg0yxJzWPFHxncGXNIT dSqCrYuLGWAR17zjLrEzxWGt320LSuR6K7kjBZq311qbrH0Dxe8BiHDDl2Rr30oRBj2L Ww== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd19-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:48 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FIpOfx005086; Mon, 15 Jan 2024 18:54:47 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd0q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:47 +0000 Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FIMsDC008633; Mon, 15 Jan 2024 18:54:46 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([172.16.1.73]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3vm6bk9vpv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:46 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIsjAd16319082 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:45 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 00C235805A; Mon, 15 Jan 2024 18:54:45 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EFAED58052; Mon, 15 Jan 2024 18:54:43 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:43 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , stable@vger.kernel.org Subject: [PATCH v4 2/6] s390/vfio-ap: loop over the shadow APCB when filtering guest's AP configuration Date: Mon, 15 Jan 2024 13:54:32 -0500 Message-ID: <20240115185441.31526-3-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: tAYHrTb_G1jvAKch1bsxRdOOkifmlZsf X-Proofpoint-ORIG-GUID: s31kwEDGqjhWMsJgXZQhXTzRbOGa8l-w X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 priorityscore=1501 impostorscore=0 mlxscore=0 spamscore=0 malwarescore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 clxscore=1015 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183807846836497 X-GMAIL-MSGID: 1788183807846836497 From: Tony Krowiak While filtering the mdev matrix, it doesn't make sense - and will have unexpected results - to filter an APID from the matrix if the APID or one of the associated APQIs is not in the host's AP configuration. There are two reasons for this: 1. An adapter or domain that is not in the host's AP configuration can be assigned to the matrix; this is known as over-provisioning. Queue devices, however, are only created for adapters and domains in the host's AP configuration, so there will be no queues associated with an over-provisioned adapter or domain to filter. 2. The adapter or domain may have been externally removed from the host's configuration via an SE or HMC attached to a DPM enabled LPAR. In this case, the vfio_ap device driver would have been notified by the AP bus via the on_config_changed callback and the adapter or domain would have already been filtered. Since the matrix_mdev->shadow_apcb.apm and matrix_mdev->shadow_apcb.aqm are copied from the mdev matrix sans the APIDs and APQIs not in the host's AP configuration, let's loop over those bitmaps instead of those assigned to the matrix. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic Fixes: 48cae940c31d ("s390/vfio-ap: refresh guest's APCB by filtering AP resources assigned to mdev") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 1f7a6c106786..e825e13847fe 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -695,8 +695,9 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev) bitmap_and(matrix_mdev->shadow_apcb.aqm, matrix_mdev->matrix.aqm, (unsigned long *)matrix_dev->info.aqm, AP_DOMAINS); - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { + for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, AP_DEVICES) { + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, + AP_DOMAINS) { /* * If the APQN is not bound to the vfio_ap device * driver, then we can't assign it to the guest's From patchwork Mon Jan 15 18:54:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188311 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887241dyc; Mon, 15 Jan 2024 10:56:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNbdXgw4RRl7dv9P/O+iZ1elgdPyT6mTbxVslF4YA5lpwdc/uaKq8+PtGNyP3rbO28AAfd X-Received: by 2002:a81:9a07:0:b0:5ff:4b02:73b9 with SMTP id r7-20020a819a07000000b005ff4b0273b9mr130219ywg.21.1705344981723; Mon, 15 Jan 2024 10:56:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705344981; cv=none; d=google.com; s=arc-20160816; b=c2lAReDv7ICs7hZROoERFCJybMHfk4kxVPaIbbRVFQIs+x8y17ThP5JEJJO5ZCCE+x XULl6aID0YnvjZTtE/sqsVZItUBRFZH6E+BP67tJX8948/MDirf/I2toBo9EPSSXlS28 hr66vTlm+x/gm+1r6y+RD6Sr0loiYPpotyEV9Tvu89TDkoJHC2VzmH7BPddNcHU8vIKl WafR22KdAG+mxad+buNak5U1jKlnWnCsNIu3mvRrq1Z4/jZAPEFuuG5km9b/zz8bmTXM KmT+p1FRuF2HYyKTidKcotWv4EvvDtJKJQpBzUPfBvFGdT9DG0VibtiUwqvObMg/+Q2n roSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=4WTuNJU7dggzhMxg+LKkTN4UTqO72KHYK9oF7xPUUCg=; fh=M0oRxUbO5PmdBwbCvdqALEcb6RJBVEJ1jCZynSfqY4Y=; b=f2uuQMNi/S+0Okpqj7B5NTVqYSqVp5T1kA0tZs8UDZ38jRkmr6J7KGSbCMfroyFOXJ rb2OSCqHA2wrcSUmdRJOOS4j0E1jtC7az3iH/88gQST8NSNNcG9DMtFKXyNko1zkY0ry UFBeXNS6VrDSnwX/0SLE+ASWs0zWhO7dpU4IQ+W3tHs0MTP354HAmLYDHWvEMESkoqXu N+G9fR4KHaN5iXx/fPw/QEy/L0LmzJbaYu4q298y4bnFlm5o7zmuUiy58lZcZF4ko+UK +YNazitg4YOihrFql7vnSHloFJHZzPli6VzOiyHvzT2OmShJXav5/xW33TJyoMVymv7G XkkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=DRACd0HI; spf=pass (google.com: domain of linux-kernel+bounces-26416-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26416-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id y18-20020a0cf152000000b0067a360ada6esi8554023qvl.255.2024.01.15.10.56.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:56:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26416-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=DRACd0HI; spf=pass (google.com: domain of linux-kernel+bounces-26416-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26416-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 71E691C206A6 for ; Mon, 15 Jan 2024 18:56:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 08BE5199AF; Mon, 15 Jan 2024 18:54:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="DRACd0HI" Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CB0818C2F; Mon, 15 Jan 2024 18:54:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FI1Ea2016044; Mon, 15 Jan 2024 18:54:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=4WTuNJU7dggzhMxg+LKkTN4UTqO72KHYK9oF7xPUUCg=; b=DRACd0HImjfPxGQR4oUs+OGXhN4G01JPZhrm3PYu2AtBunpDWA8/onRpDiocSgY/PV57 GmAYdsPCaPd/8VKBDBPzDiRFF8c1ZxaI6H7RQKiTxudgaDLi0HRmu9t6+1MkJOzU89OM ZMnVh69Q5Pn4Wa2Pv4frHC3vRxqciwPiJuQ3/FU6B/wuNUAW/o/+KDbUElCc2saY6p68 jxSD+8BkyCh76tAnm1D1Zh563/D3HKXuP2yvnO/xYY4CrEpJ0jOvTGrlvclzYSOKoiPS 7Id+cSec7WIhuc0gQ2KA6Tdsmdglyk550sZ9JvCUSeDXSbghFXdvrjZJAK+4WL2qFZVI 7w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn94a1fje-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:49 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FInRZW007475; Mon, 15 Jan 2024 18:54:48 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn94a1fhy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:48 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHeEi6018627; Mon, 15 Jan 2024 18:54:47 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([172.16.1.74]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3vm5una043-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:47 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIskGv27984620 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:46 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 16C0B5805A; Mon, 15 Jan 2024 18:54:46 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1EA2A58052; Mon, 15 Jan 2024 18:54:45 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:45 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v4 3/6] s390/vfio-ap: let 'on_scan_complete' callback filter matrix and update guest's APCB Date: Mon, 15 Jan 2024 13:54:33 -0500 Message-ID: <20240115185441.31526-4-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 6ePKe-RCQwp5ce62VQYtqc7CVwyKQB_A X-Proofpoint-GUID: jEjOyPbi8AgNUqHonmDNRZSBd4RiJDMc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 malwarescore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 impostorscore=0 mlxscore=0 clxscore=1015 priorityscore=1501 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183819648931131 X-GMAIL-MSGID: 1788183819648931131 From: Tony Krowiak When adapters and/or domains are added to the host's AP configuration, this may result in multiple queue devices getting created and probed by the vfio_ap device driver. For each queue device probed, the matrix of adapters and domains assigned to a matrix mdev will be filtered to update the guest's APCB. If any adapters or domains get added to or removed from the APCB, the guest's AP configuration will be dynamically updated (i.e., hot plug/unplug). To dynamically update the guest's configuration, its VCPUs must be taken out of SIE for the period of time it takes to make the update. This is disruptive to the guest's operation and if there are many queues probed due to a change in the host's AP configuration, this could be troublesome. The problem is exacerbated by the fact that the 'on_scan_complete' callback also filters the mdev's matrix and updates the guest's AP configuration. In order to reduce the potential amount of disruption to the guest that may result from a change to the host's AP configuration, let's bypass the filtering of the matrix and updating of the guest's AP configuration in the probe callback - if due to a host config change - and defer it until the 'on_scan_complete' callback is invoked after the AP bus finishes its device scan operation. This way the filtering and updating will be performed only once regardless of the number of queues added. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- drivers/s390/crypto/vfio_ap_ops.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e825e13847fe..e45d0bf7b126 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -2101,9 +2101,22 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) if (matrix_mdev) { vfio_ap_mdev_link_queue(matrix_mdev, q); + /* + * If we're in the process of handling the adding of adapters or + * domains to the host's AP configuration, then let the + * vfio_ap device driver's on_scan_complete callback filter the + * matrix and update the guest's AP configuration after all of + * the new queue devices are probed. + */ + if (!bitmap_empty(matrix_mdev->apm_add, AP_DEVICES) || + !bitmap_empty(matrix_mdev->aqm_add, AP_DOMAINS)) + goto done; + if (vfio_ap_mdev_filter_matrix(matrix_mdev)) vfio_ap_mdev_update_guest_apcb(matrix_mdev); } + +done: dev_set_drvdata(&apdev->device, q); release_update_locks_for_mdev(matrix_mdev); From patchwork Mon Jan 15 18:54:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188315 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887878dyc; Mon, 15 Jan 2024 10:57:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IE7WLRRs2G5eA/omcJQMFdBMOR8uY5up3Q/D+PyVbytzhhaWuXMj7Vj6aWdjVyiV5oCBYLO X-Received: by 2002:a05:600c:310d:b0:40c:2c40:8c with SMTP id g13-20020a05600c310d00b0040c2c40008cmr3144817wmo.154.1705345051382; Mon, 15 Jan 2024 10:57:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705345051; cv=none; d=google.com; s=arc-20160816; b=qBOeqz5MCT7W4uDz2qqFaK6NmYQvku6Jj1hS1Z5stiJ6bckurZSdgwI2FxPUgxXD3/ Qv6f3pSUY4hTPZFK7SbifPdWxjdNv54Ito2vucAog6maCp/Ihb04LXEf+UqG/Rg2n3xa Nn6APuZyQxPqBeixQMwDAQv8wXhOxbFMILOXf4NHQOLULPDqX99yOly9FdkHL2SPlbS+ hFX7NsN5hBpsC7vHOOB7mvL+gCMev7fa2Qt7vZ2NjrCjlm+UnLMFCOn2khhdzbycCPkM ZpCAy3Z4huzukVC5LSBVf8eZ3EZuMh/FawFS+HuK0dBKrPviOQENLeZMMZr4RKkqxq0M e5Ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=1QHxEDZTej+pCxUllhidJ3z189FRFE7/J4eGVFu6S/A=; fh=H6lVIw+KZeomf8cbJ+jDGydQlHrKW9Bp+9pD33yGxs8=; b=U1qnh27KHN9veJ+VJolAWpruywlFpkWBP8pkp2BPjJyDJPnPXvt18osk4uUqnSCNxZ stuor461PV990NQ/4MpfRz7fVdt3TlAf3HpYeQKp5oA0wwrJSlw1KIUdnFsDCOmX+8Pc Bu0U1S218RFkCNGxK0DB5N2Pk4Q8SeSRRlZ/cXpp+FzcovpBrUzJtS7V7gjRfgrRW+j/ FwjUlGX5aqhc17TlNK60uJI7uVuCkie91ePqUIjO/NYgjIgo5+ZD7NOL1CjIzuRJotNf lcZYKq8AxIuwVTABxD9rvaONr0IN/GGhA9OeFcm9Cl+IVwmwKdPH4WUNtXp/7uU7rMKy sdcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b="FF/CwNF1"; spf=pass (google.com: domain of linux-kernel+bounces-26419-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26419-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id n5-20020a056402434500b00558b1989dadsi3859110edc.193.2024.01.15.10.57.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:57:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26419-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b="FF/CwNF1"; spf=pass (google.com: domain of linux-kernel+bounces-26419-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26419-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id BB90B1F22B3D for ; Mon, 15 Jan 2024 18:57:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6EB6F1A59E; Mon, 15 Jan 2024 18:55:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="FF/CwNF1" Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD38F19440; Mon, 15 Jan 2024 18:54:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353724.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHUW4w006782; Mon, 15 Jan 2024 18:54:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=1QHxEDZTej+pCxUllhidJ3z189FRFE7/J4eGVFu6S/A=; b=FF/CwNF16KWNRRl93xKX1TyzXj3+I7e8Rk+A0zK1KyAY9HKZz8W1Tiwcm9C6YWGcMFKE eBuo74sTiJ+AFgJEvsAjyU+rE1PlOi9FDhXL9JX6tPP1ONLIyL0R/3hBBx0AfEBavDN5 PpYIrgqJ7tNlgQALUnC3I5AqzlAMPL4EhVYJvGbm1tcGdFSqUfU4CJ3whBAVcZ/YSDNN Pd6jIa/FBSIIbgIMe7XPDSsjjmprx13ggcOOhCXGZ3p4JKEQb6giGh6Aw5vq+I0J0GfF 7UMhWvMIY4tX9ny7NFQA9geq1BWA1HZt4Fzl3IK9bk1NT9qEfXyPZ/pY6y6emrFuikc/ 7w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn7x8u3ja-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:50 +0000 Received: from m0353724.ppops.net (m0353724.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FIUga3003778; Mon, 15 Jan 2024 18:54:49 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn7x8u3hy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:49 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHD4aX008775; Mon, 15 Jan 2024 18:54:49 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([172.16.1.4]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3vm57ya59h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:48 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIslbl29360858 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:47 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3DBC45805E; Mon, 15 Jan 2024 18:54:47 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 329B858052; Mon, 15 Jan 2024 18:54:46 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:46 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , stable@vger.kernel.org Subject: [PATCH v4 4/6] s390/vfio-ap: reset queues filtered from the guest's AP config Date: Mon, 15 Jan 2024 13:54:34 -0500 Message-ID: <20240115185441.31526-5-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: taLB3-CvCPsmAm3j6yQyN2-ODSqanm5I X-Proofpoint-ORIG-GUID: TNqeuyDF8gmAGwnB_c08hI9tnYg_wtKy X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 malwarescore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 adultscore=0 suspectscore=0 phishscore=0 priorityscore=1501 mlxlogscore=999 spamscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183892691549429 X-GMAIL-MSGID: 1788183892691549429 From: Tony Krowiak When filtering the adapters from the configuration profile for a guest to create or update a guest's AP configuration, if the APID of an adapter and the APQI of a domain identify a queue device that is not bound to the vfio_ap device driver, the APID of the adapter will be filtered because an individual APQN can not be filtered due to the fact the APQNs are assigned to an AP configuration as a matrix of APIDs and APQIs. Consequently, a guest will not have access to all of the queues associated with the filtered adapter. If the queues are subsequently made available again to the guest, they should re-appear in a reset state; so, let's make sure all queues associated with an adapter unplugged from the guest are reset. In order to identify the set of queues that need to be reset, let's allow a vfio_ap_queue object to be simultaneously stored in both a hashtable and a list: A hashtable used to store all of the queues assigned to a matrix mdev; and/or, a list used to store a subset of the queues that need to be reset. For example, when an adapter is hot unplugged from a guest, all guest queues associated with that adapter must be reset. Since that may be a subset of those assigned to the matrix mdev, they can be stored in a list that can be passed to the vfio_ap_mdev_reset_queues function. Signed-off-by: Tony Krowiak Acked-by: Halil Pasic Fixes: 48cae940c31d ("s390/vfio-ap: refresh guest's APCB by filtering AP resources assigned to mdev") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 171 +++++++++++++++++++------- drivers/s390/crypto/vfio_ap_private.h | 11 +- 2 files changed, 133 insertions(+), 49 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e45d0bf7b126..44cd29aace8e 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -32,7 +32,8 @@ #define AP_RESET_INTERVAL 20 /* Reset sleep interval (20ms) */ -static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable); +static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev); +static int vfio_ap_mdev_reset_qlist(struct list_head *qlist); static struct vfio_ap_queue *vfio_ap_find_queue(int apqn); static const struct vfio_device_ops vfio_ap_matrix_dev_ops; static void vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q); @@ -665,16 +666,23 @@ static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev) * device driver. * * @matrix_mdev: the matrix mdev whose matrix is to be filtered. + * @apm_filtered: a 256-bit bitmap for storing the APIDs filtered from the + * guest's AP configuration that are still in the host's AP + * configuration. * * Note: If an APQN referencing a queue device that is not bound to the vfio_ap * driver, its APID will be filtered from the guest's APCB. The matrix * structure precludes filtering an individual APQN, so its APID will be - * filtered. + * filtered. Consequently, all queues associated with the adapter that + * are in the host's AP configuration must be reset. If queues are + * subsequently made available again to the guest, they should re-appear + * in a reset state * * Return: a boolean value indicating whether the KVM guest's APCB was changed * by the filtering or not. */ -static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev) +static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_filtered) { unsigned long apid, apqi, apqn; DECLARE_BITMAP(prev_shadow_apm, AP_DEVICES); @@ -684,6 +692,7 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev) bitmap_copy(prev_shadow_apm, matrix_mdev->shadow_apcb.apm, AP_DEVICES); bitmap_copy(prev_shadow_aqm, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS); vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb); + bitmap_clear(apm_filtered, 0, AP_DEVICES); /* * Copy the adapters, domains and control domains to the shadow_apcb @@ -709,8 +718,16 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev) apqn = AP_MKQID(apid, apqi); q = vfio_ap_mdev_get_queue(matrix_mdev, apqn); if (!q || q->reset_status.response_code) { - clear_bit_inv(apid, - matrix_mdev->shadow_apcb.apm); + clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + + /* + * If the adapter was previously plugged into + * the guest, let's let the caller know that + * the APID was filtered. + */ + if (test_bit_inv(apid, prev_shadow_apm)) + set_bit_inv(apid, apm_filtered); + break; } } @@ -812,7 +829,7 @@ static void vfio_ap_mdev_remove(struct mdev_device *mdev) mutex_lock(&matrix_dev->guests_lock); mutex_lock(&matrix_dev->mdevs_lock); - vfio_ap_mdev_reset_queues(&matrix_mdev->qtable); + vfio_ap_mdev_reset_queues(matrix_mdev); vfio_ap_mdev_unlink_fr_queues(matrix_mdev); list_del(&matrix_mdev->node); mutex_unlock(&matrix_dev->mdevs_lock); @@ -922,6 +939,47 @@ static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, AP_MKQID(apid, apqi)); } +static int reset_queues_for_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_reset) +{ + struct vfio_ap_queue *q, *tmpq; + struct list_head qlist; + unsigned long apid, apqi; + int apqn, ret = 0; + + if (bitmap_empty(apm_reset, AP_DEVICES)) + return 0; + + INIT_LIST_HEAD(&qlist); + + for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) { + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, + AP_DOMAINS) { + /* + * If the domain is not in the host's AP configuration, + * then resetting it will fail with response code 01 + * (APQN not valid). + */ + if (!test_bit_inv(apqi, + (unsigned long *)matrix_dev->info.aqm)) + continue; + + apqn = AP_MKQID(apid, apqi); + q = vfio_ap_mdev_get_queue(matrix_mdev, apqn); + + if (q) + list_add_tail(&q->reset_qnode, &qlist); + } + } + + ret = vfio_ap_mdev_reset_qlist(&qlist); + + list_for_each_entry_safe(q, tmpq, &qlist, reset_qnode) + list_del(&q->reset_qnode); + + return ret; +} + /** * assign_adapter_store - parses the APID from @buf and sets the * corresponding bit in the mediated matrix device's APM @@ -962,6 +1020,7 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; + DECLARE_BITMAP(apm_filtered, AP_DEVICES); struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev); mutex_lock(&ap_perms_mutex); @@ -991,8 +1050,10 @@ static ssize_t assign_adapter_store(struct device *dev, vfio_ap_mdev_link_adapter(matrix_mdev, apid); - if (vfio_ap_mdev_filter_matrix(matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev, apm_filtered)) { vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apids(matrix_mdev, apm_filtered); + } ret = count; done: @@ -1023,11 +1084,12 @@ static struct vfio_ap_queue * adapter was assigned. * @matrix_mdev: the matrix mediated device to which the adapter was assigned. * @apid: the APID of the unassigned adapter. - * @qtable: table for storing queues associated with unassigned adapter. + * @qlist: list for storing queues associated with unassigned adapter that + * need to be reset. */ static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev, unsigned long apid, - struct ap_queue_table *qtable) + struct list_head *qlist) { unsigned long apqi; struct vfio_ap_queue *q; @@ -1035,11 +1097,10 @@ static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { q = vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi); - if (q && qtable) { + if (q && qlist) { if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) - hash_add(qtable->queues, &q->mdev_qnode, - q->apqn); + list_add_tail(&q->reset_qnode, qlist); } } } @@ -1047,26 +1108,23 @@ static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev, static void vfio_ap_mdev_hot_unplug_adapter(struct ap_matrix_mdev *matrix_mdev, unsigned long apid) { - int loop_cursor; - struct vfio_ap_queue *q; - struct ap_queue_table *qtable = kzalloc(sizeof(*qtable), GFP_KERNEL); + struct vfio_ap_queue *q, *tmpq; + struct list_head qlist; - hash_init(qtable->queues); - vfio_ap_mdev_unlink_adapter(matrix_mdev, apid, qtable); + INIT_LIST_HEAD(&qlist); + vfio_ap_mdev_unlink_adapter(matrix_mdev, apid, &qlist); if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) { clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); vfio_ap_mdev_update_guest_apcb(matrix_mdev); } - vfio_ap_mdev_reset_queues(qtable); + vfio_ap_mdev_reset_qlist(&qlist); - hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) { + list_for_each_entry_safe(q, tmpq, &qlist, reset_qnode) { vfio_ap_unlink_mdev_fr_queue(q); - hash_del(&q->mdev_qnode); + list_del(&q->reset_qnode); } - - kfree(qtable); } /** @@ -1167,6 +1225,7 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; + DECLARE_BITMAP(apm_filtered, AP_DEVICES); struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev); mutex_lock(&ap_perms_mutex); @@ -1196,8 +1255,10 @@ static ssize_t assign_domain_store(struct device *dev, vfio_ap_mdev_link_domain(matrix_mdev, apqi); - if (vfio_ap_mdev_filter_matrix(matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev, apm_filtered)) { vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apids(matrix_mdev, apm_filtered); + } ret = count; done: @@ -1210,7 +1271,7 @@ static DEVICE_ATTR_WO(assign_domain); static void vfio_ap_mdev_unlink_domain(struct ap_matrix_mdev *matrix_mdev, unsigned long apqi, - struct ap_queue_table *qtable) + struct list_head *qlist) { unsigned long apid; struct vfio_ap_queue *q; @@ -1218,11 +1279,10 @@ static void vfio_ap_mdev_unlink_domain(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { q = vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi); - if (q && qtable) { + if (q && qlist) { if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) - hash_add(qtable->queues, &q->mdev_qnode, - q->apqn); + list_add_tail(&q->reset_qnode, qlist); } } } @@ -1230,26 +1290,23 @@ static void vfio_ap_mdev_unlink_domain(struct ap_matrix_mdev *matrix_mdev, static void vfio_ap_mdev_hot_unplug_domain(struct ap_matrix_mdev *matrix_mdev, unsigned long apqi) { - int loop_cursor; - struct vfio_ap_queue *q; - struct ap_queue_table *qtable = kzalloc(sizeof(*qtable), GFP_KERNEL); + struct vfio_ap_queue *q, *tmpq; + struct list_head qlist; - hash_init(qtable->queues); - vfio_ap_mdev_unlink_domain(matrix_mdev, apqi, qtable); + INIT_LIST_HEAD(&qlist); + vfio_ap_mdev_unlink_domain(matrix_mdev, apqi, &qlist); if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); vfio_ap_mdev_update_guest_apcb(matrix_mdev); } - vfio_ap_mdev_reset_queues(qtable); + vfio_ap_mdev_reset_qlist(&qlist); - hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) { + list_for_each_entry_safe(q, tmpq, &qlist, reset_qnode) { vfio_ap_unlink_mdev_fr_queue(q); - hash_del(&q->mdev_qnode); + list_del(&q->reset_qnode); } - - kfree(qtable); } /** @@ -1604,7 +1661,7 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev) get_update_locks_for_kvm(kvm); kvm_arch_crypto_clear_masks(kvm); - vfio_ap_mdev_reset_queues(&matrix_mdev->qtable); + vfio_ap_mdev_reset_queues(matrix_mdev); kvm_put_kvm(kvm); matrix_mdev->kvm = NULL; @@ -1740,15 +1797,33 @@ static void vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q) } } -static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable) +static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev) { int ret = 0, loop_cursor; struct vfio_ap_queue *q; - hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) + hash_for_each(matrix_mdev->qtable.queues, loop_cursor, q, mdev_qnode) vfio_ap_mdev_reset_queue(q); - hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) { + hash_for_each(matrix_mdev->qtable.queues, loop_cursor, q, mdev_qnode) { + flush_work(&q->reset_work); + + if (q->reset_status.response_code) + ret = -EIO; + } + + return ret; +} + +static int vfio_ap_mdev_reset_qlist(struct list_head *qlist) +{ + int ret = 0; + struct vfio_ap_queue *q; + + list_for_each_entry(q, qlist, reset_qnode) + vfio_ap_mdev_reset_queue(q); + + list_for_each_entry(q, qlist, reset_qnode) { flush_work(&q->reset_work); if (q->reset_status.response_code) @@ -1934,7 +2009,7 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev, ret = vfio_ap_mdev_get_device_info(arg); break; case VFIO_DEVICE_RESET: - ret = vfio_ap_mdev_reset_queues(&matrix_mdev->qtable); + ret = vfio_ap_mdev_reset_queues(matrix_mdev); break; case VFIO_DEVICE_GET_IRQ_INFO: ret = vfio_ap_get_irq_info(arg); @@ -2080,6 +2155,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) { int ret; struct vfio_ap_queue *q; + DECLARE_BITMAP(apm_filtered, AP_DEVICES); struct ap_matrix_mdev *matrix_mdev; ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group); @@ -2112,15 +2188,17 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) !bitmap_empty(matrix_mdev->aqm_add, AP_DOMAINS)) goto done; - if (vfio_ap_mdev_filter_matrix(matrix_mdev)) + if (vfio_ap_mdev_filter_matrix(matrix_mdev, apm_filtered)) { vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apids(matrix_mdev, apm_filtered); + } } done: dev_set_drvdata(&apdev->device, q); release_update_locks_for_mdev(matrix_mdev); - return 0; + return ret; err_remove_group: sysfs_remove_group(&apdev->device.kobj, &vfio_queue_attr_group); @@ -2464,6 +2542,7 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info, static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) { + DECLARE_BITMAP(apm_filtered, AP_DEVICES); bool filter_domains, filter_adapters, filter_cdoms, do_hotplug = false; mutex_lock(&matrix_mdev->kvm->lock); @@ -2477,7 +2556,7 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) matrix_mdev->adm_add, AP_DOMAINS); if (filter_adapters || filter_domains) - do_hotplug = vfio_ap_mdev_filter_matrix(matrix_mdev); + do_hotplug = vfio_ap_mdev_filter_matrix(matrix_mdev, apm_filtered); if (filter_cdoms) do_hotplug |= vfio_ap_mdev_filter_cdoms(matrix_mdev); @@ -2485,6 +2564,8 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) if (do_hotplug) vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apids(matrix_mdev, apm_filtered); + mutex_unlock(&matrix_dev->mdevs_lock); mutex_unlock(&matrix_mdev->kvm->lock); } diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 88aff8b81f2f..20eac8b0f0b9 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -83,10 +83,10 @@ struct ap_matrix { }; /** - * struct ap_queue_table - a table of queue objects. - * - * @queues: a hashtable of queues (struct vfio_ap_queue). - */ + * struct ap_queue_table - a table of queue objects. + * + * @queues: a hashtable of queues (struct vfio_ap_queue). + */ struct ap_queue_table { DECLARE_HASHTABLE(queues, 8); }; @@ -133,6 +133,8 @@ struct ap_matrix_mdev { * @apqn: the APQN of the AP queue device * @saved_isc: the guest ISC registered with the GIB interface * @mdev_qnode: allows the vfio_ap_queue struct to be added to a hashtable + * @reset_qnode: allows the vfio_ap_queue struct to be added to a list of queues + * that need to be reset * @reset_status: the status from the last reset of the queue * @reset_work: work to wait for queue reset to complete */ @@ -143,6 +145,7 @@ struct vfio_ap_queue { #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; struct hlist_node mdev_qnode; + struct list_head reset_qnode; struct ap_queue_status reset_status; struct work_struct reset_work; }; From patchwork Mon Jan 15 18:54:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188314 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887862dyc; Mon, 15 Jan 2024 10:57:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IEA6Ke4nTsaW1rYfKFN0wLN5rH22me3bsXmT7gxxpztIGpQNDdwruDauO6zsXYqWfZU4SlG X-Received: by 2002:a05:6a20:ea0:b0:19a:3e20:372a with SMTP id fk32-20020a056a200ea000b0019a3e20372amr4510040pzb.119.1705345049730; Mon, 15 Jan 2024 10:57:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705345049; cv=none; d=google.com; s=arc-20160816; b=mB3Fpj1g0WA9AjMP55yVRQjE1n+gz0vHiese3SefWHIoyz4wiHP/9A+NoZ0syqVuQI zsHTfBYdnB4vHR/83viXcE6VEgxqoJGYS5o4QL+500F6SaUpFj3FYm7ky4JVFGLqqth6 dxIpSaKF3pssikaKlHoYhIwKCpQdc2C2Xysl3hpTLjWCo/x5sS5TcJJ/Jz60DZcJd9fe Yfm+4Qa58E4eknps9AAbwb5FWlP9WOeav93estPdHL62Zsb3HGGlVJgelTP6ZAfYqxGr SNA9cg16eSwKWkPDFHdziRx4lSirw4Rb9gwAMJ7YkaskmcNUOaBxzv3G/F3gcGXUUXUq Al9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=9glprldnvKen03f9RgQnxU92EqPtX6fzMlc1KWpzOtE=; fh=H6lVIw+KZeomf8cbJ+jDGydQlHrKW9Bp+9pD33yGxs8=; b=cxHxKaE/RTwpZK96wK9ItKy1TH/yGUczGtUj6bLI9rqhdZCJmzJf/mLoGYbYY8yYQs DR1cTg0i+D2z59pQWOdrKDKFKAYfwfEvD3BlPixwjWD1GsF5YRtR+i7Yo+CtTDoW6nnI Rf6yqy4N1MmQN8U2mxr5Fjs/1igLcQeX3jy6txukpI+ZaX0z4GwIjvifncLTGsktmi86 wQbjUMmssxCsLkL1O17mlQAFBqa51Ug6njzATPaqEky9vPybtRps4R//wC6iOg9WtrRN iG7PVpFcxmx1QpkYPi0chlOp7gWpavpujvc8N8hX0Q/yazAlBCPpFsyDN3JAzz1/jymE xLjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=X5ItclKI; spf=pass (google.com: domain of linux-kernel+bounces-26417-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26417-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id a24-20020a056a001d1800b006da4d96cb0asi9392892pfx.191.2024.01.15.10.57.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:57:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26417-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=X5ItclKI; spf=pass (google.com: domain of linux-kernel+bounces-26417-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26417-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 8D629B21E8A for ; Mon, 15 Jan 2024 18:57:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0984A1A585; Mon, 15 Jan 2024 18:54:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="X5ItclKI" Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DA8519455; Mon, 15 Jan 2024 18:54:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FISthM016670; Mon, 15 Jan 2024 18:54:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9glprldnvKen03f9RgQnxU92EqPtX6fzMlc1KWpzOtE=; b=X5ItclKI4B+CVG7cjWFDAhTgkZbaNBLJYte9nZJDyep+xccnkqClYUNoYTabGEIkFWO9 aUhbbiH1nNmK1BqCJ/J3vVNxs1dAKsiUfxHyQZmDMyaSl7D0psKqIls513XMDqIdTmrC sXPhIDDGQGofSkyhfXHmcZrVwP3lXvafxusBauhtdkwrT6UO7ea+N6T7cSvjehhKQjgb 3yz6G6iiF3dtXUKWMFM8MoBcT2BKee8P0scXrQz7ERigkvvooKMxxPhIi52hIiEfbG7E +P9VH6uwhKI4yWHVp+AypEllFpf7eKZCGYIvci9YwZ1g18csKfhx/BHz9ai1ShTfUGu6 8Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd29-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:51 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FIqejn007966; Mon, 15 Jan 2024 18:54:51 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vna12gd1u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:51 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHoTEh018768; Mon, 15 Jan 2024 18:54:50 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3vm5una047-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:49 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIsmOK16909004 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:48 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5DE705805E; Mon, 15 Jan 2024 18:54:48 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5977058052; Mon, 15 Jan 2024 18:54:47 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:47 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , stable@vger.kernel.org Subject: [PATCH v4 5/6] s390/vfio-ap: reset queues associated with adapter for queue unbound from driver Date: Mon, 15 Jan 2024 13:54:35 -0500 Message-ID: <20240115185441.31526-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: euRUpawGHTPo3E2EaCWfC-qBjWzqgvrx X-Proofpoint-ORIG-GUID: INkAJv9bjqlbbiFMXfosKaWzCEHA9dSQ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 priorityscore=1501 impostorscore=0 mlxscore=0 spamscore=0 malwarescore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 clxscore=1015 mlxlogscore=930 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183891188763528 X-GMAIL-MSGID: 1788183891188763528 From: Tony Krowiak When a queue is unbound from the vfio_ap device driver, if that queue is assigned to a guest's AP configuration, its associated adapter is removed because queues are defined to a guest via a matrix of adapters and domains; so, it is not possible to remove a single queue. If an adapter is removed from the guest's AP configuration, all associated queues must be reset to prevent leaking crypto data should any of them be assigned to a different guest or device driver. The one caveat is that if the queue is being removed because the adapter or domain has been removed from the host's AP configuration, then an attempt to reset the queue will fail with response code 01, AP-queue number not valid; so resetting these queues should be skipped. Acked-by: Halil Pasic Signed-off-by: Tony Krowiak Fixes: 09d31ff78793 ("s390/vfio-ap: hot plug/unplug of AP devices when probed/removed") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 76 +++++++++++++++++-------------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 44cd29aace8e..550c936c413d 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -939,45 +939,45 @@ static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, AP_MKQID(apid, apqi)); } +static void collect_queues_to_reset(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, + struct list_head *qlist) +{ + struct vfio_ap_queue *q; + unsigned long apqi; + + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS) { + q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); + if (q) + list_add_tail(&q->reset_qnode, qlist); + } +} + +static void reset_queues_for_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + struct list_head qlist; + + INIT_LIST_HEAD(&qlist); + collect_queues_to_reset(matrix_mdev, apid, &qlist); + vfio_ap_mdev_reset_qlist(&qlist); +} + static int reset_queues_for_apids(struct ap_matrix_mdev *matrix_mdev, unsigned long *apm_reset) { - struct vfio_ap_queue *q, *tmpq; struct list_head qlist; - unsigned long apid, apqi; - int apqn, ret = 0; + unsigned long apid; if (bitmap_empty(apm_reset, AP_DEVICES)) return 0; INIT_LIST_HEAD(&qlist); - for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) { - for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, - AP_DOMAINS) { - /* - * If the domain is not in the host's AP configuration, - * then resetting it will fail with response code 01 - * (APQN not valid). - */ - if (!test_bit_inv(apqi, - (unsigned long *)matrix_dev->info.aqm)) - continue; - - apqn = AP_MKQID(apid, apqi); - q = vfio_ap_mdev_get_queue(matrix_mdev, apqn); - - if (q) - list_add_tail(&q->reset_qnode, &qlist); - } - } + for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) + collect_queues_to_reset(matrix_mdev, apid, &qlist); - ret = vfio_ap_mdev_reset_qlist(&qlist); - - list_for_each_entry_safe(q, tmpq, &qlist, reset_qnode) - list_del(&q->reset_qnode); - - return ret; + return vfio_ap_mdev_reset_qlist(&qlist); } /** @@ -2217,24 +2217,30 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) matrix_mdev = q->matrix_mdev; if (matrix_mdev) { - vfio_ap_unlink_queue_fr_mdev(q); - apid = AP_QID_CARD(q->apqn); apqi = AP_QID_QUEUE(q->apqn); - - /* - * If the queue is assigned to the guest's APCB, then remove - * the adapter's APID from the APCB and hot it into the guest. - */ + /* If the queue is assigned to the guest's AP configuration */ if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { + /* + * Since the queues are defined via a matrix of adapters + * and domains, it is not possible to hot unplug a + * single queue; so, let's unplug the adapter. + */ clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apid(matrix_mdev, apid); + goto done; } } vfio_ap_mdev_reset_queue(q); flush_work(&q->reset_work); + +done: + if (matrix_mdev) + vfio_ap_unlink_queue_fr_mdev(q); + dev_set_drvdata(&apdev->device, NULL); kfree(q); release_update_locks_for_mdev(matrix_mdev); From patchwork Mon Jan 15 18:54:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 188313 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2614:b0:101:6a76:bbe3 with SMTP id mm20csp1887843dyc; Mon, 15 Jan 2024 10:57:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5VX3WGf8c1uCanbCuKUBziV6zin6G6ddFBN4/tBXQzmy5d1/pCpXxL/nkYkZAbheV8k7o X-Received: by 2002:a05:6358:471d:b0:175:c293:6b45 with SMTP id j29-20020a056358471d00b00175c2936b45mr3496195rwn.64.1705345048491; Mon, 15 Jan 2024 10:57:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705345048; cv=none; d=google.com; s=arc-20160816; b=E6PPfd0GbLYl/LxhZoA0T98ZiTWG0caJe4O+vR0yrEd+6POMudlomml9UC20i2nwdF YvpPOGpsQCpGXGMQ9ZMWGMyEyg9fT6spE3C/AHr9cq/FmITPE+Jy7Hg7DawZTgniVOvw Hf8eouo/eZKOLR8t20hKL9ONgZXzvLmyIDhIr6pgn70reeHse0AT6+sBRuazKUaxxKXB q3Yc0qEbvjuAz/O84+3nr25u19L4zyGCZHjWiZCoxUt2N94z3dJCTLLPxi2jt7epRbz5 F8a3YEgGitx1ForNlkXsJBJTOogqH23Zf2IpDFecLJLzYsVJgRkNYmNHM7H5K9p3muir RfOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=QiVXsK0Tbzx56s4STPzPtbC2hmLhKdcNXHSamhEQq70=; fh=H6lVIw+KZeomf8cbJ+jDGydQlHrKW9Bp+9pD33yGxs8=; b=C13Yq0t/OvGtsxsEe0F1/I3w5VUPlhM1SvXBFrzeCz/aznQh8gC1WMBKBwBEUldo3n 5JhdbHmkvIm0AKFsYRJFALwKhSsY+Li8HlL6aqt/UGAxL5TbwdxjvB8KlB8bcg9mbxRS 5P/sPLSiYgiG5UdYXVhhY0yPFJ+Ygy1giCYThPrxAbpNm4SH+QTcvNW29DKk7exVctUc E+pUi5PBiW/OwcEhV08M+J3xLB+G0oi64dXB2Tda6a4yTruqg5lmEbFdQrzbg0patvQo Le16iQEpm40pPx+vp3P1VRtG57vfug+wlZgwGUPm4beLy6bv62f25SNRaoFo0zWUIUWx 4NQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ZPj8HwLa; spf=pass (google.com: domain of linux-kernel+bounces-26418-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26418-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id e20-20020a656794000000b005cd811e2ffesi9553966pgr.18.2024.01.15.10.57.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jan 2024 10:57:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-26418-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=ZPj8HwLa; spf=pass (google.com: domain of linux-kernel+bounces-26418-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-26418-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 4E9B0B21E66 for ; Mon, 15 Jan 2024 18:57:16 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E16801A582; Mon, 15 Jan 2024 18:54:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ZPj8HwLa" Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9B5919469; Mon, 15 Jan 2024 18:54:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353724.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHx79q018410; Mon, 15 Jan 2024 18:54:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=QiVXsK0Tbzx56s4STPzPtbC2hmLhKdcNXHSamhEQq70=; b=ZPj8HwLa3VoR0c/sCGeTbjvwAqDpd3uCEcwcp8xNXMliBAEBi/erMyS4nqk4R0eBVFfu C1iZrZeGJN/Qt/kXACRvizuEhJOLZo/4tEuGxEhAAYOuwE3+xOmGFv1qKzkK92+nb0zv AftTsJDlb1NN06upxt7bP9Xst4S06tJDqUK7LDnvri4muUgCTkPF8dQvgWoMAVmGgGas aEJL+PGYEu9YEYummBP1Fmpwm7Yea2/9lWwCGArcFFxAsMt9IXhdAyxGR2ChbBlVJs9y /WOjf3KUQMhCs2tlWXtGvFHod75CdzKnba4/tCi+8FhCwIMDshAN2C2v8fFP8CSMEpjt 5w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn7x8u3js-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:52 +0000 Received: from m0353724.ppops.net (m0353724.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40FIqGRP028552; Mon, 15 Jan 2024 18:54:51 GMT Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vn7x8u3jh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:51 +0000 Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40FHFEws014779; Mon, 15 Jan 2024 18:54:50 GMT Received: from smtprelay04.dal12v.mail.ibm.com ([172.16.1.6]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 3vm4usj7rh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 15 Jan 2024 18:54:50 +0000 Received: from smtpav04.dal12v.mail.ibm.com (smtpav04.dal12v.mail.ibm.com [10.241.53.103]) by smtprelay04.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40FIsnIU21758594 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 15 Jan 2024 18:54:49 GMT Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7E52A58056; Mon, 15 Jan 2024 18:54:49 +0000 (GMT) Received: from smtpav04.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 791D35805A; Mon, 15 Jan 2024 18:54:48 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.164.202]) by smtpav04.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 15 Jan 2024 18:54:48 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, agordeev@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , stable@vger.kernel.org Subject: [PATCH v4 6/6] s390/vfio-ap: do not reset queue removed from host config Date: Mon, 15 Jan 2024 13:54:36 -0500 Message-ID: <20240115185441.31526-7-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240115185441.31526-1-akrowiak@linux.ibm.com> References: <20240115185441.31526-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: SG7HXdSgvtvmNDZ9uJ8dl18B9P1je-BJ X-Proofpoint-ORIG-GUID: RBacV-aimrM4LLgK1zIi9ncj-JgXn7_Z X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-15_13,2024-01-15_03,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 malwarescore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 adultscore=0 suspectscore=0 phishscore=0 priorityscore=1501 mlxlogscore=999 spamscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401150139 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788183889123841080 X-GMAIL-MSGID: 1788183889123841080 From: Tony Krowiak When a queue is unbound from the vfio_ap device driver, it is reset to ensure its crypto data is not leaked when it is bound to another device driver. If the queue is unbound due to the fact that the adapter or domain was removed from the host's AP configuration, then attempting to reset it will fail with response code 01 (APID not valid) getting returned from the reset command. Let's ensure that the queue is assigned to the host's configuration before resetting it. Signed-off-by: Tony Krowiak Reviewed-by: Jason J. Herne Reviewed-by: Halil Pasic Fixes: eeb386aeb5b7 ("s390/vfio-ap: handle config changed and scan complete notification") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 550c936c413d..983b3b16196c 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -2215,10 +2215,10 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) q = dev_get_drvdata(&apdev->device); get_update_locks_for_queue(q); matrix_mdev = q->matrix_mdev; + apid = AP_QID_CARD(q->apqn); + apqi = AP_QID_QUEUE(q->apqn); if (matrix_mdev) { - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); /* If the queue is assigned to the guest's AP configuration */ if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { @@ -2234,8 +2234,16 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) } } - vfio_ap_mdev_reset_queue(q); - flush_work(&q->reset_work); + /* + * If the queue is not in the host's AP configuration, then resetting + * it will fail with response code 01, (APQN not valid); so, let's make + * sure it is in the host's config. + */ + if (test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm) && + test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) { + vfio_ap_mdev_reset_queue(q); + flush_work(&q->reset_work); + } done: if (matrix_mdev)