From patchwork Wed Jan 4 12:05:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Schnelle X-Patchwork-Id: 38885 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp5107683wrt; Wed, 4 Jan 2023 04:08:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXuPeGAS/fBZ5xg/9cvL9JHGIIi4qkIFU0ftCw2q1//R6fIQa+2Z6I/sAx4H/WI4mBrYeI5W X-Received: by 2002:a17:906:dfc2:b0:78d:f454:387a with SMTP id jt2-20020a170906dfc200b0078df454387amr40508257ejc.55.1672834096422; Wed, 04 Jan 2023 04:08:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672834096; cv=none; d=google.com; s=arc-20160816; b=tJleW+vor0GM1vQSgKwSpbVVeBmqQccZNA32vbGbJhnZ59ijC9y2QHI5VBYusXhq05 B8iZ983K5ydVGjFbQ7BabZIl/vcLURDRZoYQkxKI7XRms1nMQUHQEjcqEK7CisYG0BWC VqtLGxPzJBXIKCkXLSdVrSdgwXme95Zyr0hcd7wzkrKdbIxf+87IJplf/eVYOJB+FNVj XwlCTPzBT/XQqcahd7SlN2WV6DzpNFIaJAw5DmfAFs/Et/k0tqHed/dnyT1UlMMIuwTI AOcX8DroT5VwHCj6Fy6I6hMjT+BXTLkfibCQX7JxY1TgjazqBhbn/pfEuwxMToYtZAjZ ahJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TTigsHQA4u4PDHj55LcJsSJIzqOkQBYUHhLkRxRiFT8=; b=OiWcZAjN3Aaj3EhrE+phvtpsq6/SmHKtU6xBkqGEGRRGxr+1IUdLdbdGdruj6oCifU ME3a98WjX8PBf1KdHI+O0yB5othGEclCPy6Bo1IivSFquLI8ABtBG0kTJW1A85EthtTm R4heUjUgZWaOz6oVwBe9ct9kk4JKh5mwZU1tapa+O1NMjcG6TMg3DnfYDl0xeL4BhhJR 9Yxx0AOHdKdi3mHGlB/VMavoUbR0PzG3o7hebkRzc6WmzDIqmDfIdMBfBdaG+pUBlmxC quR5d0VQZYLBNKBuQlay0dDLGhDQgd82LNMhKETnUc2YMP4uisA9d+ZoorGIDh8qZMbZ qAVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=Bp94XPcK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gb5-20020a170907960500b007c0c0d7c4c8si30257380ejc.44.2023.01.04.04.07.48; Wed, 04 Jan 2023 04:08:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=Bp94XPcK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234961AbjADMGe (ORCPT + 99 others); Wed, 4 Jan 2023 07:06:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230250AbjADMGL (ORCPT ); Wed, 4 Jan 2023 07:06:11 -0500 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B07B321B1; Wed, 4 Jan 2023 04:06:09 -0800 (PST) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 304BU8k1024914; Wed, 4 Jan 2023 12:05:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=TTigsHQA4u4PDHj55LcJsSJIzqOkQBYUHhLkRxRiFT8=; b=Bp94XPcKkobpZyQk2tHPk1XLey3XwFQNbwltTgvqZt5YtCxjqxNvc/gfWHQ1rN1nbQWD rui4VAncWENszcUrzDhGmpro29Fw29nF/d6TM3qz1MsaFG9Evd7S3DPJ4ORfebg5+IaB 9yJqCKraqwvLFYh/WbatLX/ivRskIi/+4FyJOo1eZaRhBlaP5i6eVuM+1bF14Xt0wN1/ EF1i5qbowLGO6y7MGbbEmVm+V7wjNQf0Du9BOlnRylOON7RpMkZoytuotfgAQl7XOy9l U0haKovdYwSNnXvCim7xlesa11N1E+vLrKZr2pxULFFRho9+biBsW2QYaFCpNyeLYaLV rw== Received: from ppma06fra.de.ibm.com (48.49.7a9f.ip4.static.sl-reverse.com [159.122.73.72]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3mw8mp8pnd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jan 2023 12:05:52 +0000 Received: from pps.filterd (ppma06fra.de.ibm.com [127.0.0.1]) by ppma06fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 303Lfvxo028038; Wed, 4 Jan 2023 12:05:50 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma06fra.de.ibm.com (PPS) with ESMTPS id 3mtcbfkuur-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jan 2023 12:05:50 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 304C5lDx49807848 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 4 Jan 2023 12:05:47 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E4F502004B; Wed, 4 Jan 2023 12:05:46 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 831A42004F; Wed, 4 Jan 2023 12:05:46 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 4 Jan 2023 12:05:46 +0000 (GMT) From: Niklas Schnelle To: Joerg Roedel , Will Deacon , Robin Murphy , Jason Gunthorpe , Wenjia Zhang Cc: Matthew Rosato , Gerd Bayer , Pierre Morel , iommu@lists.linux.dev, linux-s390@vger.kernel.org, borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com, svens@linux.ibm.com, linux-kernel@vger.kernel.org, Julian Ruess Subject: [PATCH v4 6/7] iommu/dma: Enable variable queue size and use larger single queue Date: Wed, 4 Jan 2023 13:05:42 +0100 Message-Id: <20230104120543.308933-7-schnelle@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230104120543.308933-1-schnelle@linux.ibm.com> References: <20230104120543.308933-1-schnelle@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 8ai6dUzSu1pjZbn7lFCuhiwjCRT41aL5 X-Proofpoint-GUID: 8ai6dUzSu1pjZbn7lFCuhiwjCRT41aL5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2023-01-04_06,2023-01-04_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 malwarescore=0 adultscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0 priorityscore=1501 mlxlogscore=999 mlxscore=0 impostorscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301040102 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754093685406826294?= X-GMAIL-MSGID: =?utf-8?q?1754093685406826294?= Flush queues currently use a fixed compile time size of 256 entries. This being a power of 2 allows the compiler to use shifts and mask instead of more expensive modulo operations. With per-CPU flush queues larger queue sizes would hit per-CPU allocation limits, with a single flush queue these limits do not apply however. As single flush queue mode is intended for environments with epensive IOTLB flushes it then makes sense to use a larger queue size and timeout. To this end re-order struct iova_fq so we can use a dynamic array and make the flush queue size and timeout variable. So as not to lose the shift and mask optimization, check that the variable length is a power of 2 and use explicit shift and mask instead of letting the compiler optimize this. For now use a large fixed queue size and timeout for single flush queues that brings its performance on s390 paged memory guests on par with the previous s390 specific DMA API implementation. In the future the flush queue size can then be turned into a config option or kernel parameter. Signed-off-by: Niklas Schnelle --- drivers/iommu/dma-iommu.c | 60 ++++++++++++++++++++++++++------------- 1 file changed, 41 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ca27e8ba91a2..2c848145bc97 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -61,6 +61,8 @@ struct iommu_dma_cookie { struct timer_list fq_timer; /* 1 when timer is active, 0 when not */ atomic_t fq_timer_on; + /* timeout in ms */ + unsigned long fq_timer_timeout; }; /* Trivial linear page allocator for IOMMU_DMA_MSI_COOKIE */ dma_addr_t msi_iova; @@ -86,10 +88,16 @@ static int __init iommu_dma_forcedac_setup(char *str) early_param("iommu.forcedac", iommu_dma_forcedac_setup); /* Number of entries per flush queue */ -#define IOVA_FQ_SIZE 256 +#define IOVA_DEFAULT_FQ_SIZE 256 + +/* Number of entries for a single queue */ +#define IOVA_SINGLE_FQ_SIZE 32768 /* Timeout (in ms) after which entries are flushed from the queue */ -#define IOVA_FQ_TIMEOUT 10 +#define IOVA_DEFAULT_FQ_TIMEOUT 10 + +/* Timeout (in ms) for a single queue */ +#define IOVA_SINGLE_FQ_TIMEOUT 1000 /* Flush queue entry for deferred flushing */ struct iova_fq_entry { @@ -101,18 +109,19 @@ struct iova_fq_entry { /* Per-CPU flush queue structure */ struct iova_fq { - struct iova_fq_entry entries[IOVA_FQ_SIZE]; - unsigned int head, tail; spinlock_t lock; + unsigned int head, tail; + unsigned int mod_mask; + struct iova_fq_entry entries[]; }; #define fq_ring_for_each(i, fq) \ - for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_FQ_SIZE) + for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) & (fq)->mod_mask) static inline bool fq_full(struct iova_fq *fq) { assert_spin_locked(&fq->lock); - return (((fq->tail + 1) % IOVA_FQ_SIZE) == fq->head); + return (((fq->tail + 1) & fq->mod_mask) == fq->head); } static inline unsigned int fq_ring_add(struct iova_fq *fq) @@ -121,7 +130,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq) assert_spin_locked(&fq->lock); - fq->tail = (idx + 1) % IOVA_FQ_SIZE; + fq->tail = (idx + 1) & fq->mod_mask; return idx; } @@ -143,7 +152,7 @@ static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq) fq->entries[idx].iova_pfn, fq->entries[idx].pages); - fq->head = (fq->head + 1) % IOVA_FQ_SIZE; + fq->head = (fq->head + 1) & fq->mod_mask; } } @@ -241,7 +250,7 @@ static void queue_iova(struct iommu_dma_cookie *cookie, if (!atomic_read(&cookie->fq_timer_on) && !atomic_xchg(&cookie->fq_timer_on, 1)) mod_timer(&cookie->fq_timer, - jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT)); + jiffies + msecs_to_jiffies(cookie->fq_timer_timeout)); } static void iommu_dma_free_fq_single(struct iova_fq *fq) @@ -283,43 +292,45 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie) } -static void iommu_dma_init_one_fq(struct iova_fq *fq) +static void iommu_dma_init_one_fq(struct iova_fq *fq, unsigned int fq_size) { int i; fq->head = 0; fq->tail = 0; + fq->mod_mask = fq_size - 1; spin_lock_init(&fq->lock); - for (i = 0; i < IOVA_FQ_SIZE; i++) + for (i = 0; i < fq_size; i++) INIT_LIST_HEAD(&fq->entries[i].freelist); } -static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie) +static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie, unsigned int fq_size) { struct iova_fq *queue; - queue = vzalloc(sizeof(*queue)); + queue = vzalloc(struct_size(queue, entries, fq_size)); if (!queue) return -ENOMEM; - iommu_dma_init_one_fq(queue); + iommu_dma_init_one_fq(queue, fq_size); cookie->single_fq = queue; return 0; } -static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie) +static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie, unsigned int fq_size) { struct iova_fq __percpu *queue; int cpu; - queue = alloc_percpu(struct iova_fq); + queue = __alloc_percpu(struct_size(queue, entries, fq_size), + __alignof__(*queue)); if (!queue) return -ENOMEM; for_each_possible_cpu(cpu) - iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu)); + iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu), fq_size); cookie->percpu_fq = queue; return 0; } @@ -328,24 +339,35 @@ static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie) int iommu_dma_init_fq(struct iommu_domain *domain) { struct iommu_dma_cookie *cookie = domain->iova_cookie; + unsigned int fq_size = IOVA_DEFAULT_FQ_SIZE; int rc; if (cookie->fq_domain) return 0; + if (domain->type == IOMMU_DOMAIN_DMA_SQ) + fq_size = IOVA_SINGLE_FQ_SIZE; + + if (!is_power_of_2(fq_size)) { + pr_err("FQ size must be a power of 2\n"); + return -EINVAL; + } + atomic64_set(&cookie->fq_flush_start_cnt, 0); atomic64_set(&cookie->fq_flush_finish_cnt, 0); if (domain->type == IOMMU_DOMAIN_DMA_FQ) - rc = iommu_dma_init_fq_percpu(cookie); + rc = iommu_dma_init_fq_percpu(cookie, fq_size); else - rc = iommu_dma_init_fq_single(cookie); + rc = iommu_dma_init_fq_single(cookie, fq_size); if (rc) { pr_warn("iova flush queue initialization failed\n"); return rc; } + cookie->fq_timer_timeout = (domain->type == IOMMU_DOMAIN_DMA_SQ) ? + IOVA_SINGLE_FQ_TIMEOUT : IOVA_DEFAULT_FQ_TIMEOUT; timer_setup(&cookie->fq_timer, fq_flush_timeout, 0); atomic_set(&cookie->fq_timer_on, 0); /*