From patchwork Tue Sep 12 16:28:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 138309 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp536361vqx; Tue, 12 Sep 2023 09:41:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGoNnDXC7jLoWL+DEF1ViZH7rOvGJao9CZrNDvzEedzuGjXENqCnPLhvflHiTWK4yRiAfZI X-Received: by 2002:a17:90a:ee90:b0:267:f1d0:ca70 with SMTP id i16-20020a17090aee9000b00267f1d0ca70mr12612473pjz.47.1694536908513; Tue, 12 Sep 2023 09:41:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694536908; cv=none; d=google.com; s=arc-20160816; b=nYcsM/W9lKHkQA7JvRdDLAzRTJ4od+wMOQffLSMZ24C8uskJ0sXOCL3AJGKKW8uaRC 4vDE7FttX1R8ZLNgKfJt1Lp72ndD42lEt+Y0hIKtHFlnpM9UfdZjpJsO8dOgDPguwlpi ea6fDnHmyQyYSc8OtEpGVWOwcke/5uVPC4n9gvk6KeASeE57z5k/10STEu+pMlmbHFln Yg0y6uO4wii35c+l6Ha5oC4xVDNxQOp4xyrslUTp3sRQ5g5+PHqAk17JD6AtnosjuWj9 VAEw9wz4GXQ8iIKYyqlL6HzP+6v/CedHtBb4AKQa9rhMk+rpskPcut6gCo2gUAOMqDy3 W/CA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=zubbuhETN4tQk0MD70BS3tBYOpZL4ON/1q0KtkFdKKQ=; fh=qweYnFDea2BVIoXVvRaB4VgC1vetVFKO2/YhgHGLeVQ=; b=MUQG26EDl94kFC91C/q66ESUpCaj9kjgITRodojN2j68Wze4z2UvVE4hcKIarc+EJr +MYKFIdERTMlbsWZrBGqBOktgDyXb7ciIJF5jFQ/VhSavTLT4C4TLeBWWnRCqDKCsCZC E59VOtxIpwnI8exUqfv79xbv8lQQfPxPYIA7hcrgWGr0im5D4V/81a9cdC7s4IHF/MBq yUj8z10qMBuyQsHfP0BajuTUHvsgYhHBIVhB/XXT+GlmLM08akv/3m+E3gpXdNFcoon5 54G7A8iKAp1Zi2iYJeG4SCE5+E9FAjvALNV3etxX/xxIy5TDDbdypv5cQNRMhhu/Gd+Q HNbA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id d2-20020a63d642000000b0056fa96b3df2si8026963pgj.208.2023.09.12.09.41.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 09:41:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id B70FA8024DE2; Tue, 12 Sep 2023 09:28:47 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236718AbjILQ2W (ORCPT + 37 others); Tue, 12 Sep 2023 12:28:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236772AbjILQ2V (ORCPT ); Tue, 12 Sep 2023 12:28:21 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5D674170B for ; Tue, 12 Sep 2023 09:28:17 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E170D75; Tue, 12 Sep 2023 09:28:54 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E5D763F738; Tue, 12 Sep 2023 09:28:15 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org Cc: will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, zhangzekun11@huawei.com, john.g.garry@oracle.com, dheerajkumar.srivastava@amd.com, jsnitsel@redhat.com Subject: [PATCH v3 1/2] iommu/iova: Make the rcache depot scale better Date: Tue, 12 Sep 2023 17:28:05 +0100 Message-Id: X-Mailer: git-send-email 2.39.2.101.g768bb238c484.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Sep 2023 09:28:47 -0700 (PDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776850733466095528 X-GMAIL-MSGID: 1776850733466095528 The algorithm in the original paper specifies the storage of full magazines in the depot as an unbounded list rather than a fixed-size array. It turns out to be pretty straightforward to do this in our implementation with no significant loss of efficiency. This allows the depot to scale up to the working set sizes of larger systems, while also potentially saving some memory on smaller ones too. Since this involves touching struct iova_magazine with the requisite care, we may as well reinforce the comment with a proper assertion too. Reviewed-by: John Garry Reviewed-by: Jerry Snitselaar Signed-off-by: Robin Murphy --- v3: No change drivers/iommu/iova.c | 65 ++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 30 deletions(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 10b964600948..dd2309e9a6c5 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -622,15 +622,19 @@ EXPORT_SYMBOL_GPL(reserve_iova); /* * As kmalloc's buffer size is fixed to power of 2, 127 is chosen to * assure size of 'iova_magazine' to be 1024 bytes, so that no memory - * will be wasted. + * will be wasted. Since only full magazines are inserted into the depot, + * we don't need to waste PFN capacity on a separate list head either. */ #define IOVA_MAG_SIZE 127 -#define MAX_GLOBAL_MAGS 32 /* magazines per bin */ struct iova_magazine { - unsigned long size; + union { + unsigned long size; + struct iova_magazine *next; + }; unsigned long pfns[IOVA_MAG_SIZE]; }; +static_assert(!(sizeof(struct iova_magazine) & (sizeof(struct iova_magazine) - 1))); struct iova_cpu_rcache { spinlock_t lock; @@ -640,8 +644,7 @@ struct iova_cpu_rcache { struct iova_rcache { spinlock_t lock; - unsigned long depot_size; - struct iova_magazine *depot[MAX_GLOBAL_MAGS]; + struct iova_magazine *depot; struct iova_cpu_rcache __percpu *cpu_rcaches; }; @@ -717,6 +720,21 @@ static void iova_magazine_push(struct iova_magazine *mag, unsigned long pfn) mag->pfns[mag->size++] = pfn; } +static struct iova_magazine *iova_depot_pop(struct iova_rcache *rcache) +{ + struct iova_magazine *mag = rcache->depot; + + rcache->depot = mag->next; + mag->size = IOVA_MAG_SIZE; + return mag; +} + +static void iova_depot_push(struct iova_rcache *rcache, struct iova_magazine *mag) +{ + mag->next = rcache->depot; + rcache->depot = mag; +} + int iova_domain_init_rcaches(struct iova_domain *iovad) { unsigned int cpu; @@ -734,7 +752,6 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) rcache = &iovad->rcaches[i]; spin_lock_init(&rcache->lock); - rcache->depot_size = 0; rcache->cpu_rcaches = __alloc_percpu(sizeof(*cpu_rcache), cache_line_size()); if (!rcache->cpu_rcaches) { @@ -776,7 +793,6 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, struct iova_rcache *rcache, unsigned long iova_pfn) { - struct iova_magazine *mag_to_free = NULL; struct iova_cpu_rcache *cpu_rcache; bool can_insert = false; unsigned long flags; @@ -794,12 +810,7 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, if (new_mag) { spin_lock(&rcache->lock); - if (rcache->depot_size < MAX_GLOBAL_MAGS) { - rcache->depot[rcache->depot_size++] = - cpu_rcache->loaded; - } else { - mag_to_free = cpu_rcache->loaded; - } + iova_depot_push(rcache, cpu_rcache->loaded); spin_unlock(&rcache->lock); cpu_rcache->loaded = new_mag; @@ -812,11 +823,6 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, spin_unlock_irqrestore(&cpu_rcache->lock, flags); - if (mag_to_free) { - iova_magazine_free_pfns(mag_to_free, iovad); - iova_magazine_free(mag_to_free); - } - return can_insert; } @@ -854,9 +860,9 @@ static unsigned long __iova_rcache_get(struct iova_rcache *rcache, has_pfn = true; } else { spin_lock(&rcache->lock); - if (rcache->depot_size > 0) { + if (rcache->depot) { iova_magazine_free(cpu_rcache->loaded); - cpu_rcache->loaded = rcache->depot[--rcache->depot_size]; + cpu_rcache->loaded = iova_depot_pop(rcache); has_pfn = true; } spin_unlock(&rcache->lock); @@ -895,9 +901,8 @@ static void free_iova_rcaches(struct iova_domain *iovad) struct iova_rcache *rcache; struct iova_cpu_rcache *cpu_rcache; unsigned int cpu; - int i, j; - for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { + for (int i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { rcache = &iovad->rcaches[i]; if (!rcache->cpu_rcaches) break; @@ -907,8 +912,8 @@ static void free_iova_rcaches(struct iova_domain *iovad) iova_magazine_free(cpu_rcache->prev); } free_percpu(rcache->cpu_rcaches); - for (j = 0; j < rcache->depot_size; ++j) - iova_magazine_free(rcache->depot[j]); + while (rcache->depot) + iova_magazine_free(iova_depot_pop(rcache)); } kfree(iovad->rcaches); @@ -942,16 +947,16 @@ static void free_global_cached_iovas(struct iova_domain *iovad) { struct iova_rcache *rcache; unsigned long flags; - int i, j; - for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { + for (int i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { rcache = &iovad->rcaches[i]; spin_lock_irqsave(&rcache->lock, flags); - for (j = 0; j < rcache->depot_size; ++j) { - iova_magazine_free_pfns(rcache->depot[j], iovad); - iova_magazine_free(rcache->depot[j]); + while (rcache->depot) { + struct iova_magazine *mag = iova_depot_pop(rcache); + + iova_magazine_free_pfns(mag, iovad); + iova_magazine_free(mag); } - rcache->depot_size = 0; spin_unlock_irqrestore(&rcache->lock, flags); } } From patchwork Tue Sep 12 16:28:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 138306 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp532635vqx; Tue, 12 Sep 2023 09:35:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE9wbGXG1xIIdjUZZoJTrFdOBbetyAxqKwsNfcb38pyNGlJli8nvBWRnLJlxCwWdQZmGHvQ X-Received: by 2002:a05:6a00:124b:b0:68f:cb69:8e7b with SMTP id u11-20020a056a00124b00b0068fcb698e7bmr328095pfi.2.1694536544078; Tue, 12 Sep 2023 09:35:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694536544; cv=none; d=google.com; s=arc-20160816; b=VKErggUdPkfMfM9SYMdMb+oXNvy3Gs5CsdMLDLgkuN8DAzm33k4H3blE7YBw3I3Jm9 ffvEEEvu0n3AcpfbglzAoqHrvFV2maGX7GtnlQqhKbJnx7++FtYZvQv2r6qNeNFeBLJX t8PEo8B5WjGCMFjtd+cG20kC/h/ub7URUxikLch+6pJwF9SG0oeldhDSSjlT8SPlGw1Q 0Xm2uNSEPhF7aBULff58skQX/zg9s3tKsFDZBn3c8kPlOutkn6D8KjEt8o3N6WBIJ2TV LuvpObrdKsI9HrT0sMjobFRG30b9OS7M3h4bKsBX7MZkq8ihDqyRAe+OcwxDjejiuORH C+xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=1VcyqQ2ULqlHLp/Yo7O3OcwQVBLlkPrWk6cyKFc4FJk=; fh=qweYnFDea2BVIoXVvRaB4VgC1vetVFKO2/YhgHGLeVQ=; b=e2Xg/nqm1/pIVon04H30XEA3caU01A949UMrMurL+s9K5duFVMIHxBgk2sYVk900Ft B0GlfAZ6vPosW+NfmLfAtp7lg8Q3HS1UKdpRNhH6xkgxg4s5TaUvOcRR9kIsKbo7Gvix 4EKpOn3OYL83FMMmZCSdDYFeydmLOP0vIwQFMzlHeKf6FHTa+azO7U4/eCh72cohdQXw IrdBdCEPXFLuzUKfY0ADnGczfDkcWHi6VUWG3rpbsW4g8SpQctSkrOac6Cua46BEroyy LEmSh2kPsv2EhOfk1ZyaQUYT+d6DUb0jIv+/DQihZcPU+PQsdxeiweBHHAUCXamztzcb muEQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id s13-20020a056a00194d00b0068a65b26fc6si5812240pfk.43.2023.09.12.09.35.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 09:35:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id C192F8232678; Tue, 12 Sep 2023 09:28:42 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236780AbjILQ2Y (ORCPT + 37 others); Tue, 12 Sep 2023 12:28:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236769AbjILQ2X (ORCPT ); Tue, 12 Sep 2023 12:28:23 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9850A1703 for ; Tue, 12 Sep 2023 09:28:19 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 89091C15; Tue, 12 Sep 2023 09:28:56 -0700 (PDT) Received: from e121345-lin.cambridge.arm.com (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1CBE73F738; Tue, 12 Sep 2023 09:28:17 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org Cc: will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, zhangzekun11@huawei.com, john.g.garry@oracle.com, dheerajkumar.srivastava@amd.com, jsnitsel@redhat.com Subject: [PATCH v3 2/2] iommu/iova: Manage the depot list size Date: Tue, 12 Sep 2023 17:28:06 +0100 Message-Id: <03170665c56d89c6ce6081246b47f68d4e483308.1694535580.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.39.2.101.g768bb238c484.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 12 Sep 2023 09:28:42 -0700 (PDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776850351220778534 X-GMAIL-MSGID: 1776850351220778534 Automatically scaling the depot up to suit the peak capacity of a workload is all well and good, but it would be nice to have a way to scale it back down again if the workload changes. To that end, add backround reclaim that will gradually free surplus magazines if the depot size remains above a reasonable threshold for long enough. Reviewed-by: Jerry Snitselaar Signed-off-by: Robin Murphy --- v3: Make sure iova_depot_work_func() locking is IRQ-safe drivers/iommu/iova.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index dd2309e9a6c5..d30e453d0fb4 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -11,6 +11,7 @@ #include #include #include +#include /* The anchor node sits above the top of the usable address space */ #define IOVA_ANCHOR ~0UL @@ -627,6 +628,8 @@ EXPORT_SYMBOL_GPL(reserve_iova); */ #define IOVA_MAG_SIZE 127 +#define IOVA_DEPOT_DELAY msecs_to_jiffies(100) + struct iova_magazine { union { unsigned long size; @@ -644,8 +647,11 @@ struct iova_cpu_rcache { struct iova_rcache { spinlock_t lock; + unsigned int depot_size; struct iova_magazine *depot; struct iova_cpu_rcache __percpu *cpu_rcaches; + struct iova_domain *iovad; + struct delayed_work work; }; static struct iova_magazine *iova_magazine_alloc(gfp_t flags) @@ -726,6 +732,7 @@ static struct iova_magazine *iova_depot_pop(struct iova_rcache *rcache) rcache->depot = mag->next; mag->size = IOVA_MAG_SIZE; + rcache->depot_size--; return mag; } @@ -733,6 +740,25 @@ static void iova_depot_push(struct iova_rcache *rcache, struct iova_magazine *ma { mag->next = rcache->depot; rcache->depot = mag; + rcache->depot_size++; +} + +static void iova_depot_work_func(struct work_struct *work) +{ + struct iova_rcache *rcache = container_of(work, typeof(*rcache), work.work); + struct iova_magazine *mag = NULL; + unsigned long flags; + + spin_lock_irqsave(&rcache->lock, flags); + if (rcache->depot_size > num_online_cpus()) + mag = iova_depot_pop(rcache); + spin_unlock_irqrestore(&rcache->lock, flags); + + if (mag) { + iova_magazine_free_pfns(mag, rcache->iovad); + iova_magazine_free(mag); + schedule_delayed_work(&rcache->work, IOVA_DEPOT_DELAY); + } } int iova_domain_init_rcaches(struct iova_domain *iovad) @@ -752,6 +778,8 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) rcache = &iovad->rcaches[i]; spin_lock_init(&rcache->lock); + rcache->iovad = iovad; + INIT_DELAYED_WORK(&rcache->work, iova_depot_work_func); rcache->cpu_rcaches = __alloc_percpu(sizeof(*cpu_rcache), cache_line_size()); if (!rcache->cpu_rcaches) { @@ -812,6 +840,7 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, spin_lock(&rcache->lock); iova_depot_push(rcache, cpu_rcache->loaded); spin_unlock(&rcache->lock); + schedule_delayed_work(&rcache->work, IOVA_DEPOT_DELAY); cpu_rcache->loaded = new_mag; can_insert = true; @@ -912,6 +941,7 @@ static void free_iova_rcaches(struct iova_domain *iovad) iova_magazine_free(cpu_rcache->prev); } free_percpu(rcache->cpu_rcaches); + cancel_delayed_work_sync(&rcache->work); while (rcache->depot) iova_magazine_free(iova_depot_pop(rcache)); }