From patchwork Thu Jan 11 14:38:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 187410 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1494396dyi; Thu, 11 Jan 2024 06:43:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IEQR+7J/WUnbKjDZe82Fcp2hnWV/w06BhLfcEmnziAVo2gzaQM1QJyuVAjYVTQZ3OKqnh7D X-Received: by 2002:a17:902:e747:b0:1d5:9ad4:1c87 with SMTP id p7-20020a170902e74700b001d59ad41c87mr395721plf.13.1704984218254; Thu, 11 Jan 2024 06:43:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704984218; cv=none; d=google.com; s=arc-20160816; b=JO3kRaXpLS6sgoHH+cl8YOWlXzxVLv0SgIxJaTyWfmUUjzTc6cIIt+qYKSEEJgTLhS 0QDKlZCGZjBOaaRbpB4MxPTf0dZRUk/GD6lI6FVqnSn+QldOtH/F+KNQ+iqGfFXOOvDj PUM8pz0DzdT7SHzIfKattaVDQKi+gQN4weXa8xouZaDAYro7tRiCLr0xyNmJWJqMCSwQ 4T9WScGtpBZwoPTctx8q5aXH+tO7vYlXa66+aIr1u7ybDk0VA3KOBcIbuLqTjjKf8qU0 iZIYFXHYvoBfqJloqnb0+O15FfwZ9ZTkLZfW6CARY/VDW2puiVGgavRJ9TvpLqwonRwx CSDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=+K+MciumVhATDK/ceqJmYklCACUu6513662urtlgic8=; fh=89dtS/PwPJUSahQCrBwvdQ9UYtztJf8P1g5jTeyUoxY=; b=dmbFb641fbSam5saWUPei43QmPYyjDPDuZl+CzWCvegd4ZBX5nSESCNjJ5k2HA8B6A E7c6YMXR4YG4HuZH7TFvKjH9aUNmI2HL6fZ1J365nifj+CIuJlnr8iIWQEGQNOuMdCsA D205t8peokvSg7LJxTjaXjbSpscpN7TD0J2tVFVPnq7G945FT2Chbr9Bk5TCNN9ST3Zg WyNfssGcadYYEwA+sEKxmwrgf9EWYBWmQu8xZsWB2OjqwP+jBzv4OXj+rJay5T1xFyAW XoI3GBN+0z/PFfPzqkKXJ0q4BvZLIUAnB65dtXGBSjghN1jnsNiopECuVGB1rfp9RjaQ Lymw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=XXTWlEnw; spf=pass (google.com: domain of linux-kernel+bounces-23722-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23722-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id kv3-20020a17090328c300b001d573476f54si1153093plb.241.2024.01.11.06.43.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 06:43:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23722-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=XXTWlEnw; spf=pass (google.com: domain of linux-kernel+bounces-23722-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23722-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 2A9FFB262B7 for ; Thu, 11 Jan 2024 14:41:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A78FF524AE; Thu, 11 Jan 2024 14:39:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="XXTWlEnw" Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95DEF50243; Thu, 11 Jan 2024 14:39:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Received: from pps.filterd (m0353724.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 40BEXkCK029981; Thu, 11 Jan 2024 14:38:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=+K+MciumVhATDK/ceqJmYklCACUu6513662urtlgic8=; b=XXTWlEnwnDJgUTN2JaT7yVrAybOTMThGzkVlBuZx2bAnDGxiHZUOgcFaQxerDRgCqNKN PBxM2nb+NxoKQwU2Kn3I8hnNGA1KtyRKlqsMMu/8OZwN3Pc0/W4yP0pQqfUESuuOg663 5uV2dpyBQv8Oh7ESiUpcc0MdWMnDUG9BZfBR+k0Cbws0iLAe875nSnCl0nxKlfrxME+/ JOQJsEbyHlTrzDZ7LT663kyIgvQdESijpbPX3p7mEL9N4lVG9I0lmhmbOxmzsrZmcd6i cMlnGz+MqJGboDDRnTrK0gH5/fQkF0J35j4git1vthfsbBzmgrPSKdbcgU7mlEP7ZV0C Ng== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vjj70045y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 11 Jan 2024 14:38:57 +0000 Received: from m0353724.ppops.net (m0353724.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 40BEZa2v003293; Thu, 11 Jan 2024 14:38:56 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3vjj700456-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 11 Jan 2024 14:38:56 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 40BDKasK001339; Thu, 11 Jan 2024 14:38:55 GMT Received: from smtprelay04.wdc07v.mail.ibm.com ([172.16.1.71]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3vfkdkkmnf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 11 Jan 2024 14:38:55 +0000 Received: from smtpav06.dal12v.mail.ibm.com (smtpav06.dal12v.mail.ibm.com [10.241.53.105]) by smtprelay04.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 40BEcr6336831994 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jan 2024 14:38:54 GMT Received: from smtpav06.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C48AA58065; Thu, 11 Jan 2024 14:38:53 +0000 (GMT) Received: from smtpav06.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D686A5805D; Thu, 11 Jan 2024 14:38:52 +0000 (GMT) Received: from li-2c1e724c-2c76-11b2-a85c-ae42eaf3cb3d.ibm.com.com (unknown [9.61.174.181]) by smtpav06.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 11 Jan 2024 14:38:52 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, pasic@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, stable@vger.kernel.org Subject: [PATCH v3 5/6] s390/vfio-ap: reset queues associated with adapter for queue unbound from driver Date: Thu, 11 Jan 2024 09:38:39 -0500 Message-ID: <20240111143846.8801-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240111143846.8801-1-akrowiak@linux.ibm.com> References: <20240111143846.8801-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: eKJjeIF8PziRydl92JKs28J4AoUgmg1W X-Proofpoint-ORIG-GUID: Bv86QjINgbY1UHKA1eoZcMQjrrxyKu2v X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-11_07,2024-01-11_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 lowpriorityscore=0 priorityscore=1501 phishscore=0 adultscore=0 bulkscore=0 mlxlogscore=916 spamscore=0 mlxscore=0 suspectscore=0 impostorscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2401110116 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787805531507367006 X-GMAIL-MSGID: 1787805531507367006 When a queue is unbound from the vfio_ap device driver, if that queue is assigned to a guest's AP configuration, its associated adapter is removed because queues are defined to a guest via a matrix of adapters and domains; so, it is not possible to remove a single queue. If an adapter is removed from the guest's AP configuration, all associated queues must be reset to prevent leaking crypto data should any of them be assigned to a different guest or device driver. The one caveat is that if the queue is being removed because the adapter or domain has been removed from the host's AP configuration, then an attempt to reset the queue will fail with response code 01, AP-queue number not valid; so resetting these queues should be skipped. Acked-by: Halil Pasic Signed-off-by: Tony Krowiak Fixes: 09d31ff78793 ("s390/vfio-ap: hot plug/unplug of AP devices when probed/removed") Cc: --- drivers/s390/crypto/vfio_ap_ops.c | 78 ++++++++++++++++--------------- 1 file changed, 41 insertions(+), 37 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 11f8f0bcc7ed..e014108067dc 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -935,45 +935,45 @@ static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, AP_MKQID(apid, apqi)); } +static void collect_queues_to_reset(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, + struct list_head *qlist) +{ + struct vfio_ap_queue *q; + unsigned long apqi; + + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS) { + q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); + if (q) + list_add_tail(&q->reset_qnode, qlist); + } +} + +static void reset_queues_for_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + struct list_head qlist; + + INIT_LIST_HEAD(&qlist); + collect_queues_to_reset(matrix_mdev, apid, &qlist); + vfio_ap_mdev_reset_qlist(&qlist); +} + static int reset_queues_for_apids(struct ap_matrix_mdev *matrix_mdev, unsigned long *apm_reset) { - struct vfio_ap_queue *q, *tmpq; struct list_head qlist; - unsigned long apid, apqi; - int apqn, ret = 0; + unsigned long apid; if (bitmap_empty(apm_reset, AP_DEVICES)) return 0; INIT_LIST_HEAD(&qlist); - for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) { - for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, - AP_DOMAINS) { - /* - * If the domain is not in the host's AP configuration, - * then resetting it will fail with response code 01 - * (APQN not valid). - */ - if (!test_bit_inv(apqi, - (unsigned long *)matrix_dev->info.aqm)) - continue; - - apqn = AP_MKQID(apid, apqi); - q = vfio_ap_mdev_get_queue(matrix_mdev, apqn); - - if (q) - list_add_tail(&q->reset_qnode, &qlist); - } - } + for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) + collect_queues_to_reset(matrix_mdev, apid, &qlist); - ret = vfio_ap_mdev_reset_qlist(&qlist); - - list_for_each_entry_safe(q, tmpq, &qlist, reset_qnode) - list_del(&q->reset_qnode); - - return ret; + return vfio_ap_mdev_reset_qlist(&qlist); } /** @@ -2199,24 +2199,28 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) matrix_mdev = q->matrix_mdev; if (matrix_mdev) { - vfio_ap_unlink_queue_fr_mdev(q); - - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); - - /* - * If the queue is assigned to the guest's APCB, then remove - * the adapter's APID from the APCB and hot it into the guest. - */ + /* If the queue is assigned to the guest's AP configuration */ if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { + /* + * Since the queues are defined via a matrix of adapters + * and domains, it is not possible to hot unplug a + * single queue; so, let's unplug the adapter. + */ clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); vfio_ap_mdev_update_guest_apcb(matrix_mdev); + reset_queues_for_apid(matrix_mdev, apid); + goto done; } } vfio_ap_mdev_reset_queue(q); flush_work(&q->reset_work); + +done: + if (matrix_mdev) + vfio_ap_unlink_queue_fr_mdev(q); + dev_set_drvdata(&apdev->device, NULL); kfree(q); release_update_locks_for_mdev(matrix_mdev);