From patchwork Fri Aug 11 13:02:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "zhangzekun (A)" X-Patchwork-Id: 134527 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1119791vqi; Fri, 11 Aug 2023 07:06:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE5SIPZmufOVBo1OjDXHV/CmUPgsAgrpNQlvgk5Grzxyz6+Ek86bOQzAGnaxiiu8lwFDn3n X-Received: by 2002:a05:6a00:9a3:b0:67e:bf65:ae68 with SMTP id u35-20020a056a0009a300b0067ebf65ae68mr2353808pfg.3.1691762765923; Fri, 11 Aug 2023 07:06:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691762765; cv=none; d=google.com; s=arc-20160816; b=Yf75v37yu1vxUOWE2nsHu5kozF/hSAgQZXcujevx+poz6TGSw+tKftMm8AckGdIu0e MdbwH3YYP+Amh6WJvaedafUyseUjLUJ56XRU0pGm1NAbFs5JtssocggCiw7RiUymnPMw nw7sEcJo0IL04I2EVUCF2XiSDs9ZwgTmoAOXuPv2ztH+ORIQJZSGiz/TXNJa3W72Bb9R ULfAlrQpo5Iw0cXfuQppuM4K+kDIZInHly/7IYlLGwFua2lKhwclX8O4xNaoB4pDG6vl xYBVtuD/WA0WMkxBIV5M9sokZIf43WW24vAWaYiWg5xSOKnJKUqNr7o+IyEclBy+TcF+ /yHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=9/U+pD5oRHnqpzvvvJZGZULfZp9IgaQsblFITN+uEmE=; fh=j+d8Qji3Kd78Y5wikuIe56W4rxsMj0I3YfL77K4ftaQ=; b=swc6XhiFlxqcC0dfQ5mLfbiHuw0wkPoYlQ6byzbCRWPYayjNllxOIc7GoZyldaj0+d +4FAE2u1dlkrM2wRSrSO/sE8aAJWjz2ZS7sysANpZmDtlyfK7IkPsdJKGM5lHf3+xHAE /T/LxaG6zIFna7Gqjz5YOjCbZINKZ2oj6Izz6oi3Xmi6aTPPYqw3lbLxrsQP9JkTUKN+ +nUTva/9lI07LnWg5tkVtifT0vFRLXUcNdx3qW6yask0mpYziY4FaoZwEBco0zJNJJmY ru0jNyt9yEbfCWuIcBt/THVCXjbzqpa1K3y8Ahpg+ylFxVU4NYvxVd3koh3QOeJYEcEp gLCQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f20-20020a056a0022d400b006875170380asi3563923pfj.121.2023.08.11.07.05.49; Fri, 11 Aug 2023 07:06:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234059AbjHKNLP (ORCPT + 99 others); Fri, 11 Aug 2023 09:11:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229994AbjHKNLL (ORCPT ); Fri, 11 Aug 2023 09:11:11 -0400 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC84518B for ; Fri, 11 Aug 2023 06:11:09 -0700 (PDT) Received: from kwepemi500019.china.huawei.com (unknown [172.30.72.54]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4RMkcb0k0TzqSg0; Fri, 11 Aug 2023 21:08:15 +0800 (CST) Received: from huawei.com (10.175.112.208) by kwepemi500019.china.huawei.com (7.221.188.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 11 Aug 2023 21:11:06 +0800 From: Zhang Zekun To: , , CC: , , , , , , Subject: [RESEND PATCH 1/2] iommu/iova: Add check for cpu_rcache in free_iova_rcaches Date: Fri, 11 Aug 2023 21:02:45 +0800 Message-ID: <20230811130246.42719-2-zhangzekun11@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230811130246.42719-1-zhangzekun11@huawei.com> References: <20230811130246.42719-1-zhangzekun11@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.208] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To kwepemi500019.china.huawei.com (7.221.188.117) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773941834442183914 X-GMAIL-MSGID: 1773941834442183914 free_iova_rcaches() needs to check if cpu_rcache->loaded and cpu_rcache->prev is NULL before freeing them. Because iova_domain_init_rcaches() may fail to alloc magazine for cpu_rcache->loaded and cpu_rcache->prev, but they will be freed for all cpus. Fixes: 32e92d9f6f87 ("iommu/iova: Separate out rcache init") Signed-off-by: Zhang Zekun --- drivers/iommu/iova.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 10b964600948..3c784a28e9ed 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -746,8 +746,12 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) spin_lock_init(&cpu_rcache->lock); cpu_rcache->loaded = iova_magazine_alloc(GFP_KERNEL); + if (!cpu_rcache->loaded) { + ret = -ENOMEM; + goto out_err; + } cpu_rcache->prev = iova_magazine_alloc(GFP_KERNEL); - if (!cpu_rcache->loaded || !cpu_rcache->prev) { + if (!cpu_rcache->prev) { ret = -ENOMEM; goto out_err; } @@ -903,7 +907,11 @@ static void free_iova_rcaches(struct iova_domain *iovad) break; for_each_possible_cpu(cpu) { cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); + if (!cpu_rcache->loaded) + break; iova_magazine_free(cpu_rcache->loaded); + if (!cpu_rcache->prev) + break; iova_magazine_free(cpu_rcache->prev); } free_percpu(rcache->cpu_rcaches); From patchwork Fri Aug 11 13:02:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "zhangzekun (A)" X-Patchwork-Id: 134505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp1103137vqi; Fri, 11 Aug 2023 06:41:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHRPYS6HdThmDrc7NxUC4/UELvqiVh6b/gp9vKDYygu5BZcMySDW7Pa0btx1FajeyUK/pwi X-Received: by 2002:a17:902:e891:b0:1b9:e091:8037 with SMTP id w17-20020a170902e89100b001b9e0918037mr2370062plg.30.1691761270829; Fri, 11 Aug 2023 06:41:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691761270; cv=none; d=google.com; s=arc-20160816; b=HIIOwYGY9kwttFgAxoOeR+IJFmQJsXNojwq1cxTP92rdDlLCliZqBG8qO7MHoRwwMY qTjF6eu0wNCJzOHoaRVXMaeiI7zjOigB2sRoNIXhdrhxgIsdTCbUdaWeUQuq4F3ArFv+ GhrKpFEP5ko3ynDCn2i7Wo2FvW2E72QvpHiNGWoqnNlV3160301fFZs9UsOnx7OQm5AI IcQ8FZRfIavsoVdJCAu30+U/HCJWBMDLzymnBLU2dY5YpcQeWZRre9pAFreo+EtDOwZ5 +80jNzlT8KeWy/2D/dyxHLs0DVPnCp816W5lBU21XjGSJiAlS2Rq9sjIMXaZTsrZQwU2 evjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=Gd13WaQl8CA5eHBzKRkgAFmdrcJSk/orVJWVanWI6LM=; fh=j+d8Qji3Kd78Y5wikuIe56W4rxsMj0I3YfL77K4ftaQ=; b=bypl0ndtN9qzPikeCExk8mEFY+oNdVLhvLzYScTZ9+7XkKICf5shlefU3JpiRZ/zz5 CeHv9INamSOmOOkhNLPuUeevpogbSX73nV/iSseS5kt27Nu/mT0o+UjQdhasogYG6psc Xx9BLiQmnBOORvcmdXAWSUptib9dqFr3WMMQ/riWOJiFRQezTI4XWaJXVCrMP/tHjuRA AoxlnMLDfeWZSU1jerWvk3qSoC9BgHUF3Z1Oo3pxzLHoRaCuNAA64oVNX/Uj4pklC8Kx /4Jju4Lls3sdSPUYFGUdWIpHDAcLHmRu6JPyjyryLA6ClG+2Ma26utU5f4tEhBS34EMu z3IQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n9-20020a170902d0c900b001b22f31cce9si3303323pln.179.2023.08.11.06.40.56; Fri, 11 Aug 2023 06:41:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233180AbjHKNLT (ORCPT + 99 others); Fri, 11 Aug 2023 09:11:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233890AbjHKNLM (ORCPT ); Fri, 11 Aug 2023 09:11:12 -0400 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EF6930E3 for ; Fri, 11 Aug 2023 06:11:10 -0700 (PDT) Received: from kwepemi500019.china.huawei.com (unknown [172.30.72.57]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4RMkcb69drzqSfv; Fri, 11 Aug 2023 21:08:15 +0800 (CST) Received: from huawei.com (10.175.112.208) by kwepemi500019.china.huawei.com (7.221.188.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 11 Aug 2023 21:11:07 +0800 From: Zhang Zekun To: , , CC: , , , , , , Subject: [RESEND PATCH 2/2] iommu/iova: allocate iova_rcache->depot dynamicly Date: Fri, 11 Aug 2023 21:02:46 +0800 Message-ID: <20230811130246.42719-3-zhangzekun11@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230811130246.42719-1-zhangzekun11@huawei.com> References: <20230811130246.42719-1-zhangzekun11@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.208] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To kwepemi500019.china.huawei.com (7.221.188.117) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773940266633121970 X-GMAIL-MSGID: 1773940266633121970 In fio test with 4k,read,and allowed cpus to 0-255, we observe a performance decrease of IOPS. The normal IOPS can reach up to 1980k, but we can only get about 1600k. abormal IOPS: Jobs: 12 (f=12): [R(12)][99.3%][r=6220MiB/s][r=1592k IOPS][eta 00m:12s] Jobs: 12 (f=12): [R(12)][99.4%][r=6215MiB/s][r=1591k IOPS][eta 00m:11s] Jobs: 12 (f=12): [R(12)][99.4%][r=6335MiB/s][r=1622k IOPS][eta 00m:10s] Jobs: 12 (f=12): [R(12)][99.5%][r=6194MiB/s][r=1586k IOPS][eta 00m:09s] Jobs: 12 (f=12): [R(12)][99.6%][r=6173MiB/s][r=1580k IOPS][eta 00m:08s] Jobs: 12 (f=12): [R(12)][99.6%][r=5984MiB/s][r=1532k IOPS][eta 00m:07s] Jobs: 12 (f=12): [R(12)][99.7%][r=6374MiB/s][r=1632k IOPS][eta 00m:06s] Jobs: 12 (f=12): [R(12)][99.7%][r=6343MiB/s][r=1624k IOPS][eta 00m:05s] normal IOPS: Jobs: 12 (f=12): [R(12)][99.3%][r=7736MiB/s][r=1980k IOPS][eta 00m:12s] Jobs: 12 (f=12): [R(12)][99.4%][r=7744MiB/s][r=1982k IOPS][eta 00m:11s] Jobs: 12 (f=12): [R(12)][99.4%][r=7737MiB/s][r=1981k IOPS][eta 00m:10s] Jobs: 12 (f=12): [R(12)][99.5%][r=7735MiB/s][r=1980k IOPS][eta 00m:09s] Jobs: 12 (f=12): [R(12)][99.6%][r=7741MiB/s][r=1982k IOPS][eta 00m:08s] Jobs: 12 (f=12): [R(12)][99.6%][r=7740MiB/s][r=1982k IOPS][eta 00m:07s] Jobs: 12 (f=12): [R(12)][99.7%][r=7736MiB/s][r=1981k IOPS][eta 00m:06s] Jobs: 12 (f=12): [R(12)][99.7%][r=7736MiB/s][r=1980k IOPS][eta 00m:05s] The current struct of iova_rcache will have iova_cpu_rcache for every cpu, and these iova_cpu_rcaches use a common buffer iova_rcache->depot to exchange iovas among iova_cpu_rcaches. A machine with 256 cpus will have 256 iova_cpu_rcaches and 1 iova_rcache->depot per iova_domain. However, the max size of iova_rcache->depot is fixed to MAX_GLOBAL_MAGS which equals to 32, and can't grow with the number of cpus, and this can cause problem in some condition. Some drivers will only free iovas in their irq call back function. For the driver in this case it has 16 thread irqs to free iova, but these irq call back function will only free iovas on 16 certain cpus(cpu{0,16, 32...,240}). Thread irq which smp affinity is 0-15, will only free iova on cpu 0. However, the driver will alloc iova on all cpus(cpu{0-255}), cpus without free iova in local cpu_rcache need to get free iovas from iova_rcache->depot. The current size of iova_rcache->depot max size is 32, and it seems to be too small for 256 users (16 cpus will put iovas to iova_rcache->depot and 240 cpus will try to get iova from it). Set iova_rcache->depot grow with the num of possible cpus, and the decrease of IOPS disappear. Signed-off-by: Zhang Zekun --- drivers/iommu/iova.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 3c784a28e9ed..df37a4501e98 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -238,6 +238,7 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad, static struct kmem_cache *iova_cache; static unsigned int iova_cache_users; +static unsigned int max_global_mags; static DEFINE_MUTEX(iova_cache_mutex); static struct iova *alloc_iova_mem(void) @@ -625,7 +626,6 @@ EXPORT_SYMBOL_GPL(reserve_iova); * will be wasted. */ #define IOVA_MAG_SIZE 127 -#define MAX_GLOBAL_MAGS 32 /* magazines per bin */ struct iova_magazine { unsigned long size; @@ -641,7 +641,7 @@ struct iova_cpu_rcache { struct iova_rcache { spinlock_t lock; unsigned long depot_size; - struct iova_magazine *depot[MAX_GLOBAL_MAGS]; + struct iova_magazine **depot; struct iova_cpu_rcache __percpu *cpu_rcaches; }; @@ -722,6 +722,13 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) unsigned int cpu; int i, ret; + /* + * the size of max global mags should growth with the num of + * cpus + */ + if (!max_global_mags) + max_global_mags = max_t(unsigned int, 32, num_possible_cpus()); + iovad->rcaches = kcalloc(IOVA_RANGE_CACHE_MAX_SIZE, sizeof(struct iova_rcache), GFP_KERNEL); @@ -733,6 +740,12 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) struct iova_rcache *rcache; rcache = &iovad->rcaches[i]; + rcache->depot = kcalloc(max_global_mags, sizeof(struct iova_magazine *), + GFP_KERNEL); + if (!rcache->depot) { + ret = -ENOMEM; + goto out_err; + } spin_lock_init(&rcache->lock); rcache->depot_size = 0; rcache->cpu_rcaches = __alloc_percpu(sizeof(*cpu_rcache), @@ -798,7 +811,7 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, if (new_mag) { spin_lock(&rcache->lock); - if (rcache->depot_size < MAX_GLOBAL_MAGS) { + if (rcache->depot_size < max_global_mags) { rcache->depot[rcache->depot_size++] = cpu_rcache->loaded; } else { @@ -903,8 +916,12 @@ static void free_iova_rcaches(struct iova_domain *iovad) for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { rcache = &iovad->rcaches[i]; - if (!rcache->cpu_rcaches) + if (!rcache->depot) + break; + if (!rcache->cpu_rcaches) { + kfree(rcache->depot); break; + } for_each_possible_cpu(cpu) { cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); if (!cpu_rcache->loaded) @@ -917,6 +934,7 @@ static void free_iova_rcaches(struct iova_domain *iovad) free_percpu(rcache->cpu_rcaches); for (j = 0; j < rcache->depot_size; ++j) iova_magazine_free(rcache->depot[j]); + kfree(rcache->depot); } kfree(iovad->rcaches);