From patchwork Mon Dec 5 14:59:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29707 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2302588wrr; Mon, 5 Dec 2022 07:01:15 -0800 (PST) X-Google-Smtp-Source: AA0mqf4VREfdP8m8I3NqUScufuwzk7ipKjR9s4L4OkYuQmdBVl9CYtv7mh9++Gh0THpBgY3G73au X-Received: by 2002:a05:6e02:cae:b0:302:edb1:6f9 with SMTP id 14-20020a056e020cae00b00302edb106f9mr11184794ilg.23.1670252475039; Mon, 05 Dec 2022 07:01:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670252475; cv=none; d=google.com; s=arc-20160816; b=JjyldhlxE4p/Q4n9Tb37ou7tEV8oWyEyxc6mmhKLZt9b99inUZ5miBZaPpbvZdn8O9 WzvV1cR88QOUCtM7hgMQC+3iIVdqA0M9LNhY6+OWetc6dGxygjWzK1A765C0dAt0urMg dvE6S5UuKeE+CYw7h/wQuD83lGgh8ZxVbB8yB5zCYjHR3EM00sYfgN5DTGspfREChTuG /1Ggzo6Ezu6stK5D1nhVjjxxYedZhVvJtm4HsNhDOgYrB58MPU0wcvOY1uHIzBiusLEp yzH8SzH7ADn+dG5YY7fGkzpUjYm9d7hzbdKTdIzg5CkXHCQ3JxuWV840EXDuoMrt4qfY Axfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jW6N9lNs9aHxnq0Gv9Edz1amHHS3PbY2z2d5+0qT2t4=; b=dVnlU+XMzAX3Iwvh9SfZL+NNgC1UayF08m/rUqMeIlQlFCMz6tVluhaDpnZUOi/eBN unAWs7a6lxzGXtT58DO+6RKJRxV5NDXCeJ5klPEtdSmmAhjIHkNLJrd652I6vr/iuezc Al5PL3QLPikIVKBTj390H0sPhUYMAUctQtpCxIiRtdvreYG0PsGcZ4S/Ye0rkSEilSkW k0LYhhThA2OUPSjReSPJazK91rTs36LKwebpc1UUd1mSgRuLVTHU18rmxMYtKOCNfdAG eMMMCWp4Bb0H4ECeiaO5eLh/Ejl0AvaIHoP9BBFh94qaluLVQ+9cmZolsPYQv4rJQbaJ TFiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ZDve+bap; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y21-20020aa79435000000b0057731e4f610si71487pfo.84.2022.12.05.07.00.58; Mon, 05 Dec 2022 07:01:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ZDve+bap; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231845AbiLEPA1 (ORCPT + 99 others); Mon, 5 Dec 2022 10:00:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231599AbiLEPAZ (ORCPT ); Mon, 5 Dec 2022 10:00:25 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB206BC85 for ; Mon, 5 Dec 2022 07:00:24 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5EZsYo010692 for ; Mon, 5 Dec 2022 07:00:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=jW6N9lNs9aHxnq0Gv9Edz1amHHS3PbY2z2d5+0qT2t4=; b=ZDve+bapQAu7yDP8M4eZ++3X7Atj7Oi/MPdhOX8GOyN0o96Th+aa0aFtIrmp5gNK3eZS HzF3xxhFebuIGv3tqNRl5XfJNX9K6+4lD6lgY1dexHfkteTL8RPOFw97DAomCtSxxIsr 8acc7jar+0AVMiwkV8Vx32lnj+RfkqLMkgaenN7XvgNfN1U+/1N4HHpWpGQ3o1urOyRK 3Ti0b6tZ4n5xjP0Q2j7bZESHDlrEai4sXBg3oCxGH4HeWr2AeTRCouymw4bcvB/XmFO4 MvgXj2mCpC2mM6M7tvXpT6At2VZAC++PdbSJSZkV5CO26DjSDOIEXr/LZODVH26ZmerN qQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntux7x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:00:23 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:00:22 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id A52D2C6156DD; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 01/11] dmapool: add alloc/free performance test Date: Mon, 5 Dec 2022 06:59:27 -0800 Message-ID: <20221205145937.54367-2-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 0P1HgfABaooqcOIgzK216V-MDpPTcDvG X-Proofpoint-GUID: 0P1HgfABaooqcOIgzK216V-MDpPTcDvG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751386659375243849?= X-GMAIL-MSGID: =?utf-8?q?1751386659375243849?= From: Keith Busch Provide a module that allocates and frees many blocks of various sizes and report how long it takes. This is intended to provide a consistent way to measure how changes to the dma_pool_alloc/free routines affect timing. Signed-off-by: Keith Busch --- mm/Kconfig | 9 +++ mm/Makefile | 1 + mm/dmapool_test.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+) create mode 100644 mm/dmapool_test.c diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b5052..12a78d64eeb72 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1044,6 +1044,15 @@ config GUP_TEST comment "GUP_TEST needs to have DEBUG_FS enabled" depends on !GUP_TEST && !DEBUG_FS +config DMAPOOL_TEST + tristate "Enable a module to run time tests on dma_pool" + depends on HAS_DMA + help + Provides a module that will allocate and free many blocks of various + sizes and report how long it takes. This is intended to provide a + consistent way to measure how changes to the dma_pool_alloc/free + routines affect performance. + config GUP_GET_PTE_LOW_HIGH bool diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e293..3a08f5d7b1782 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -103,6 +103,7 @@ obj-$(CONFIG_MEMCG) += swap_cgroup.o endif obj-$(CONFIG_CGROUP_HUGETLB) += hugetlb_cgroup.o obj-$(CONFIG_GUP_TEST) += gup_test.o +obj-$(CONFIG_DMAPOOL_TEST) += dmapool_test.o obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o diff --git a/mm/dmapool_test.c b/mm/dmapool_test.c new file mode 100644 index 0000000000000..2873d91f509b4 --- /dev/null +++ b/mm/dmapool_test.c @@ -0,0 +1,138 @@ +#include +#include +#include +#include +#include +#include + +#define NR_TESTS (100) + +struct dma_pool_pair { + dma_addr_t dma; + void *v; +}; + +static const int block_sizes[] = { + 16, + 64, + 256, + 1024, + 4096 +}; + +static struct dma_pool *pool; +struct device test_dev; +u64 dma_mask; + +static inline int nr_blocks(int size) +{ + return clamp_t(int, (PAGE_SIZE / size) * 512, 1024, 8192); +} + +static int dmapool_test_alloc(struct dma_pool_pair *p, int blocks) +{ + int i; + + for (i = 0; i < blocks; i++) { + p[i].v = dma_pool_alloc(pool, GFP_KERNEL, + &p[i].dma); + if (!p[i].v) + goto pool_fail; + } + + for (i = 0; i < blocks; i++) + dma_pool_free(pool, p[i].v, p[i].dma); + + return 0; + +pool_fail: + for (--i; i >= 0; i--) + dma_pool_free(pool, p[i].v, p[i].dma); + return -ENOMEM; +} + +static int dmapool_test_block(int block_size) +{ + int blocks = nr_blocks(block_size); + ktime_t start_time, end_time; + struct dma_pool_pair *p; + int i, ret; + + p = kcalloc(blocks, sizeof(*p), GFP_KERNEL); + if (!p) + return -ENOMEM; + + pool = dma_pool_create("test pool", &test_dev, block_size, + block_size, 0); + if (!pool) { + ret = -ENOMEM; + goto free_pairs; + } + + start_time = ktime_get(); + for (i = 0; i < NR_TESTS; i++) { + ret = dmapool_test_alloc(p, blocks); + if (ret) + goto free_pool; + if (need_resched()) + cond_resched(); + } + end_time = ktime_get(); + + printk("dmapool test: size:%-4d blocks:%-6d time:%llu\n", block_size, + blocks, ktime_us_delta(end_time, start_time)); + +free_pool: + dma_pool_destroy(pool); +free_pairs: + kfree(p); + return ret; +} + +static void dmapool_test_release(struct device *dev) +{ +} + +static int dmapool_checks(void) +{ + int i, ret; + + ret = dev_set_name(&test_dev, "dmapool-test"); + if (ret) + return ret; + + ret = device_register(&test_dev); + if (ret) { + printk("%s: register failed:%d\n", __func__, ret); + goto put_device; + } + + test_dev.release = dmapool_test_release; + test_dev.dma_ops = NULL; + test_dev.dma_mask = &dma_mask; + ret = dma_set_mask_and_coherent(&test_dev, DMA_BIT_MASK(64)); + if (ret) { + printk("%s: mask failed:%d\n", __func__, ret); + goto del_device; + } + + for (i = 0; i < ARRAY_SIZE(block_sizes); i++) { + ret = dmapool_test_block(block_sizes[i]); + if (ret) + break; + } + +del_device: + device_del(&test_dev); +put_device: + put_device(&test_dev); + return ret; +} + +static void dmapool_exit(void) +{ +} + +module_init(dmapool_checks); +module_exit(dmapool_exit); +MODULE_LICENSE("GPL"); From patchwork Mon Dec 5 14:59:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29708 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2302643wrr; Mon, 5 Dec 2022 07:01:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf6UkeVXuwf4shjuAZhoA4GGWSqZKCjOB1Xs9w3yjhxlB69liotdNxE7zWzJztEIaWeusih0 X-Received: by 2002:a63:dd13:0:b0:46e:ccbd:b136 with SMTP id t19-20020a63dd13000000b0046eccbdb136mr61108905pgg.515.1670252479478; Mon, 05 Dec 2022 07:01:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670252479; cv=none; d=google.com; s=arc-20160816; b=vdHY45z+TXKFGil/yZMoXw8FjsC2r720RITLH3Xlz707Ign0vDypLO05wxYWXhofnK TyrRHe2/hdpYnkmOGTdf2LYjAX6p9ARjCtm36Y06nOLv2utFbYRyyxelESABbn0a3KSt r4HfuNRcdmmE5Qp3evxr/wiy442V11mJKqOy+3L4hDTwjFoatRop0Skr++HzuHQ2V8Pk J43P15u+VWqJxMX6+IXwmxCGe934+4TCTA7ruTPuIG2BBpSf/g8wzZR6MDBVxRIxQA8g rV2Rs+/oRf5b86+N0EhMTDQywB1WNkkWHJ1V+j0IlWY/pkGpNWrr6qJFrgAxn8QLkXyB H9NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YU1RvihRnvDJ0EI7aPZRW/ndJTmtKABPXXQKd2XE/QM=; b=d8kLI2vbDOlnc3nWelx0aDXLk9kkLO6Xic8zzLBM0Hx2yhJ1MoIAX2AfuWLVsNVXLR KJMl424YkSu7AmYpOEzfcXBtOq256t4QGAKLUOtuqkMTjdL8+oKp0nPUQxqbawroecsi gDYWO/7+3g1lvjh0UpXOMDp8rTTmSllC8uPSUxs7bU/MuShsAdwRGqOSskqyeT/x56Ek QDissqrZ1TpRisAE0M3WIi9HTbExOHO0J43BLCsWEdETyOv9UsFImeDeZNz8w91hgJgj Bm1lQVKl7P9WfgWhUysqn6w/IMkacczY8p7bOxxitNxzJJZ+0/PVkl1xxqS73nttYBCu QIKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=h72aB84u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y10-20020aa79aea000000b00576d944dd65si4063392pfp.54.2022.12.05.07.01.03; Mon, 05 Dec 2022 07:01:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=h72aB84u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232020AbiLEPAa (ORCPT + 99 others); Mon, 5 Dec 2022 10:00:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231923AbiLEPA2 (ORCPT ); Mon, 5 Dec 2022 10:00:28 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA071DFCE for ; Mon, 5 Dec 2022 07:00:25 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2B5C0SIX000740 for ; Mon, 5 Dec 2022 07:00:25 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=YU1RvihRnvDJ0EI7aPZRW/ndJTmtKABPXXQKd2XE/QM=; b=h72aB84uF535vhqcSF/upzVkRKlcy1O4YHQ33llTxkXXqGNEv07PJm/9UT0GamvqFm69 letw054sf96K2yDECmaBk5enLnO0+h+gBqXZpn3F9n3gVsFu4VqjzsvjyQUHNSWvgrsH ZYetGQPpLXxAeBwNCLvHfg+F6A153f/jl5EwJ17EI7tbQ5br8Zz0ZbAHA16VCXqzRSKp yMbMqL1mylZiV7qWL1Zmk1MRN/59JJuWSaNU8zWZiPNEAKZ11tvspWwFc9X4MX/uo989 +XykDO8PcS9Hz6d+i8hEd+kGzILJyBRYQoaU9DJOf749WEbwODmDaNso2QkV4OFWjpcB vA== Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3m9g8c1kks-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:00:24 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:00:22 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id C5CE2C6156E0; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 02/11] dmapool: remove checks for dev == NULL Date: Mon, 5 Dec 2022 06:59:28 -0800 Message-ID: <20221205145937.54367-3-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: SIAsFnmQprUPdP9asPoy9-ng49KiGAL9 X-Proofpoint-ORIG-GUID: SIAsFnmQprUPdP9asPoy9-ng49KiGAL9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751386663779761111?= X-GMAIL-MSGID: =?utf-8?q?1751386663779761111?= From: Tony Battersby dmapool originally tried to support pools without a device because dma_alloc_coherent() supports allocations without a device. But nobody ended up using dma pools without a device, and trying to do so will result in an oops. So remove the checks for pool->dev == NULL since they are unneeded bloat. Signed-off-by: Tony Battersby [added check for null dev on create] Signed-off-by: Keith Busch --- mm/dmapool.c | 45 ++++++++++++++------------------------------- 1 file changed, 14 insertions(+), 31 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index a7eb5d0eb2da7..559207e1c3339 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -134,6 +134,9 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, size_t allocation; bool empty = false; + if (!dev) + return NULL; + if (align == 0) align = 1; else if (align & (align - 1)) @@ -275,7 +278,7 @@ void dma_pool_destroy(struct dma_pool *pool) mutex_lock(&pools_reg_lock); mutex_lock(&pools_lock); list_del(&pool->pools); - if (pool->dev && list_empty(&pool->dev->dma_pools)) + if (list_empty(&pool->dev->dma_pools)) empty = true; mutex_unlock(&pools_lock); if (empty) @@ -284,12 +287,8 @@ void dma_pool_destroy(struct dma_pool *pool) list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { if (is_page_busy(page)) { - if (pool->dev) - dev_err(pool->dev, "%s %s, %p busy\n", __func__, - pool->name, page->vaddr); - else - pr_err("%s %s, %p busy\n", __func__, - pool->name, page->vaddr); + dev_err(pool->dev, "%s %s, %p busy\n", __func__, + pool->name, page->vaddr); /* leak the still-in-use consistent memory */ list_del(&page->page_list); kfree(page); @@ -351,12 +350,8 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, for (i = sizeof(page->offset); i < pool->size; i++) { if (data[i] == POOL_POISON_FREED) continue; - if (pool->dev) - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); - else - pr_err("%s %s, %p (corrupted)\n", - __func__, pool->name, retval); + dev_err(pool->dev, "%s %s, %p (corrupted)\n", + __func__, pool->name, retval); /* * Dump the first 4 bytes even if they are not @@ -411,12 +406,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) page = pool_find_page(pool, dma); if (!page) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); - else - pr_err("%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); + dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", + __func__, pool->name, vaddr, &dma); return; } @@ -426,12 +417,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) #ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); - else - pr_err("%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); + dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + __func__, pool->name, vaddr, &dma); return; } { @@ -442,12 +429,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) continue; } spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, dma %pad already free\n", - __func__, pool->name, &dma); - else - pr_err("%s %s, dma %pad already free\n", - __func__, pool->name, &dma); + dev_err(pool->dev, "%s %s, dma %pad already free\n", + __func__, pool->name, &dma); return; } } From patchwork Mon Dec 5 14:59:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29718 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2314493wrr; Mon, 5 Dec 2022 07:19:42 -0800 (PST) X-Google-Smtp-Source: AA0mqf5YKW2/FoBpylKyU2AcFAMadSowyuTw233y1m+95bFgsXVRnlvMobnx64cvCYm5TRxeYH2U X-Received: by 2002:a05:6402:3892:b0:454:cbef:c161 with SMTP id fd18-20020a056402389200b00454cbefc161mr57497407edb.365.1670253582273; Mon, 05 Dec 2022 07:19:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253582; cv=none; d=google.com; s=arc-20160816; b=jRDCp2o5EMRHwZDyfnR5i5fSKOMGslM1MZxGqfJoRJUEnZsN/LKgYeRphHyP6MDe6M 2Ek63m7meSwBHhdSJyAwxPpU1MIhRDYD4+/RgstPqB7alor/7Mzjp62g0DRN+MvrUWL0 h5doeabL3RfGkfzuRGFnT/2m5eaf27SYF3dnY/+MTaKcACTMO5clLhMmyvChG9QBbYtS N3T+5JoexRjYepfAFLYbJCH9jP+hJpXeDUnHQKhAZ2rjd1c3atZxW9GrfIERNtUZwMLy N12MrswCKiE+IAvcR13pxFDlkGDVMJj92rPpTgFx+SXBRvNAwrlSvnswH80JIj9Nq1xs Enog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MLWHDUS20C0jmHJ39hBO1H02jIUsWm5bZEXT2SOrcPk=; b=mw0qLkxA1ZGNojmgife2doRynw3yxi8XDOGi4ArLhUdcPTPhHliuGuZJMchQpp58yV Ig/Al/gYBEChTMXSFWJ8/ZBy48fbENJK4ELCiLo9oRs6raXWT7f850EYRsh1sVTGfqJD QB7lgiQ/wGbvjpzeeXA58gQfvpM9gatktcFmefUWgalvD/V9Q8JMiur+aeNYWdsaW70p XcAdMhMIbkzqe/byJuEHNq0Ph2XnC7z14A+/kcFcGFrAbzsMD+kpaka2P7CouzzzOOjN Fq3GU7smNPYbKv0qBqx4hJjLQyWG69b+ih8Bg4hg3GXhUEI9eWrnYLfkn71+kxtb7agT kC7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ejYF75fU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds15-20020a170907724f00b007ae165c79c8si12772838ejc.940.2022.12.05.07.19.17; Mon, 05 Dec 2022 07:19:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ejYF75fU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231829AbiLEPOR (ORCPT + 99 others); Mon, 5 Dec 2022 10:14:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230522AbiLEPNe (ORCPT ); Mon, 5 Dec 2022 10:13:34 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8340D5F84 for ; Mon, 5 Dec 2022 07:13:27 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B51d4Sq028659 for ; Mon, 5 Dec 2022 07:13:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=MLWHDUS20C0jmHJ39hBO1H02jIUsWm5bZEXT2SOrcPk=; b=ejYF75fU7LKM8wl5jpImHC6ob7OmFqntqgSVGc7q7fBRtNTo2YbmogV9/hYldchrM5wM G0+tjYboDIyd5JbXawla0H4uvIOtKIYTIurU5zdZBP0kxApMAZGRIOvXKAig2QNU3KMf eIiafVB8VOpgv6FqOoo28oMSYTlMz1YLZA6rOqxa7tMTjf+eHGdJuwl4dbcwngVrLne1 l16/SnAwj5orN2eNpyh6zDEi0moXxB76qmVzGa0EU66PzkXkQAohAew/QcGDljqEA6kz WusGmDZ0R0+69WEmS1e817ytYI09qRBuM4V9W9N5PPvhKdasT7p0qd7KndjZCpcdkU7Z yw== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv1f1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:13:26 -0800 Received: from twshared0551.06.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:13:24 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6027CC6156E3; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 03/11] dmapool: use sysfs_emit() instead of scnprintf() Date: Mon, 5 Dec 2022 06:59:29 -0800 Message-ID: <20221205145937.54367-4-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: gIKqeh2PxnGikkroHP_deWhRD57Zjpuo X-Proofpoint-GUID: gIKqeh2PxnGikkroHP_deWhRD57Zjpuo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387820076889007?= X-GMAIL-MSGID: =?utf-8?q?1751387820076889007?= From: Tony Battersby Use sysfs_emit instead of scnprintf, snprintf or sprintf. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 559207e1c3339..20616b760bb9c 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -64,18 +64,11 @@ static DEFINE_MUTEX(pools_reg_lock); static ssize_t pools_show(struct device *dev, struct device_attribute *attr, char *buf) { - unsigned temp; - unsigned size; - char *next; + int size; struct dma_page *page; struct dma_pool *pool; - next = buf; - size = PAGE_SIZE; - - temp = scnprintf(next, size, "poolinfo - 0.1\n"); - size -= temp; - next += temp; + size = sysfs_emit(buf, "poolinfo - 0.1\n"); mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { @@ -90,16 +83,14 @@ static ssize_t pools_show(struct device *dev, struct device_attribute *attr, cha spin_unlock_irq(&pool->lock); /* per-pool info, no real statistics yet */ - temp = scnprintf(next, size, "%-16s %4u %4zu %4zu %2u\n", - pool->name, blocks, - pages * (pool->allocation / pool->size), - pool->size, pages); - size -= temp; - next += temp; + size += sysfs_emit_at(buf, size, "%-16s %4u %4zu %4zu %2u\n", + pool->name, blocks, + pages * (pool->allocation / pool->size), + pool->size, pages); } mutex_unlock(&pools_lock); - return PAGE_SIZE - size; + return size; } static DEVICE_ATTR_RO(pools); From patchwork Mon Dec 5 14:59:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29713 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2310592wrr; Mon, 5 Dec 2022 07:12:56 -0800 (PST) X-Google-Smtp-Source: AA0mqf4uv9IKYuRmSp3XQSAAZiDwpQkBsJSYw5TLLnX9PXiqKHhw2UH5RFNMejyflX/jMHpCYAa3 X-Received: by 2002:a17:906:4997:b0:7ae:ec5c:a99e with SMTP id p23-20020a170906499700b007aeec5ca99emr52204109eju.219.1670253176543; Mon, 05 Dec 2022 07:12:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253176; cv=none; d=google.com; s=arc-20160816; b=CoNYk3oAA52/iIyT+jeu6DosHKlwZtmRPbP+PTqpb4U+xihAAreQzulPVQif4kMgD5 yMlEmx/gmMVC6NfIw47tZIitNgDdPvY5Oy0RehWtqJ3xQrDM3bWY7jr9JCJrSc/QLqV/ NFz4Vv9hyB4NLZbRWdK3aF1qJZAR4ZGKexFFsa7bt5JdqY901MHHQTk1R2APEOLvXh6P b2Y/5cSAYwbPWC8d9WyWPv7yLrrT3p0oLHGVmakSIqi+ckqY1P4BPC0WisXuN0//Xsff DD2ZVYvQ6NgvwkGF9BLDyzlrPEJZ+OuY9ouLJid73h7vYy2QX8vY07VkPjLCCcXh6dBD pcVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TsEKLROJI6bk+d/af34BtH3nHi+XT+AwCEfmZNUH3/Y=; b=GmM2Q0AVX2a8K4VMWwk6sX6AWVXwJAY8aypI1FnYfLZwUyk8Q4YCNbIWKgX7xCgqjY 8eoJgV/GL6S+Vuck1Bmh67NthTVLvJmAnX6Ie1YxPOeFC+9ltkDA+FoexDP/SM95u3I/ /rbLjiuLnNGFDrTPXswQ3+X5+7SG2y8tP3rvslmWZEPP/zaDYYSSDO5H6BSQJ8e8001o fDM4PD7v+3nGqsI7RaPE5o3J9qr40Zbv2tGPtvzFoX2wbmLNWwPp9znuiagX8GBm2wdD K+FaBfjzKDVYyQeiYLvd6IpNEJ3XbyIkUD4Nt8l1/2gBxOzaEx5GYvZ3KujLjJ9SIErQ 69CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=n2s7vjs8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nb36-20020a1709071ca400b007c10638840asi239719ejc.75.2022.12.05.07.12.31; Mon, 05 Dec 2022 07:12:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=n2s7vjs8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232282AbiLEPD1 (ORCPT + 99 others); Mon, 5 Dec 2022 10:03:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232287AbiLEPDX (ORCPT ); Mon, 5 Dec 2022 10:03:23 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4C411CFD0 for ; Mon, 5 Dec 2022 07:03:22 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5EI7CO031279 for ; Mon, 5 Dec 2022 07:03:22 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=TsEKLROJI6bk+d/af34BtH3nHi+XT+AwCEfmZNUH3/Y=; b=n2s7vjs8KN+BGlwCWmAM3FI6+I2qNWWEDl7YsCA2k46CZ/7DMyFdU9VzlF7oyomPrQjJ CfuFPPABnbZaJqnY6LBHCNxBnnudgd+ny2xYQa5twOoeOH8PiiuNDG/PIhj35Y/pRqRl lCw1f4Qjwi3OvNjhgPhE0s9C8tClj/2iR4oNOYZSI8yk25zwioYHnbSdkkDEY1aNveT8 CppP79V42qM4gc+WFQPxq+0/6I1dVHulvlndcRSiUFUUvpvO7f0FA4964+xLgjz0vrJr Fq1NxJLX8Tjy4iuD1KQgJFp47CgphSlIs3EzpDKgmkhuI5qtOzIA9UNG8Z5D7xLrFqwV zA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561asg3-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:03:21 -0800 Received: from twshared26225.38.frc1.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:03:20 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id D6291C6156EA; Mon, 5 Dec 2022 07:00:10 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 04/11] dmapool: cleanup integer types Date: Mon, 5 Dec 2022 06:59:30 -0800 Message-ID: <20221205145937.54367-5-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: CJwgNXB_5qrWdRXOstgq8w2xv39hTJQF X-Proofpoint-ORIG-GUID: CJwgNXB_5qrWdRXOstgq8w2xv39hTJQF X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387395016444324?= X-GMAIL-MSGID: =?utf-8?q?1751387395016444324?= From: Tony Battersby To represent the size of a single allocation, dmapool currently uses 'unsigned int' in some places and 'size_t' in other places. Standardize on 'unsigned int' to reduce overhead, but use 'size_t' when counting all the blocks in the entire pool. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 20616b760bb9c..ee993bb59fc27 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -43,10 +43,10 @@ struct dma_pool { /* the pool */ struct list_head page_list; spinlock_t lock; - size_t size; struct device *dev; - size_t allocation; - size_t boundary; + unsigned int size; + unsigned int allocation; + unsigned int boundary; char name[32]; struct list_head pools; }; @@ -73,7 +73,7 @@ static ssize_t pools_show(struct device *dev, struct device_attribute *attr, cha mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { unsigned pages = 0; - unsigned blocks = 0; + size_t blocks = 0; spin_lock_irq(&pool->lock); list_for_each_entry(page, &pool->page_list, page_list) { @@ -83,9 +83,10 @@ static ssize_t pools_show(struct device *dev, struct device_attribute *attr, cha spin_unlock_irq(&pool->lock); /* per-pool info, no real statistics yet */ - size += sysfs_emit_at(buf, size, "%-16s %4u %4zu %4zu %2u\n", + size += sysfs_emit_at(buf, size, "%-16s %4zu %4zu %4u %2u\n", pool->name, blocks, - pages * (pool->allocation / pool->size), + (size_t) pages * + (pool->allocation / pool->size), pool->size, pages); } mutex_unlock(&pools_lock); @@ -133,7 +134,7 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, else if (align & (align - 1)) return NULL; - if (size == 0) + if (size == 0 || size > INT_MAX) return NULL; else if (size < 4) size = 4; @@ -146,6 +147,8 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, else if ((boundary < size) || (boundary & (boundary - 1))) return NULL; + boundary = min(boundary, allocation); + retval = kmalloc(sizeof(*retval), GFP_KERNEL); if (!retval) return retval; @@ -306,7 +309,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, { unsigned long flags; struct dma_page *page; - size_t offset; + unsigned int offset; void *retval; might_alloc(mem_flags); From patchwork Mon Dec 5 14:59:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29729 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2316345wrr; Mon, 5 Dec 2022 07:22:43 -0800 (PST) X-Google-Smtp-Source: AA0mqf66O73GUGMxqTGY479WLvZMd2AO5V+0VyBsOT728/LOXJ2MUMP34fT0HiNqUPuTjF53MVUT X-Received: by 2002:a05:6402:104b:b0:461:7d1f:1a7b with SMTP id e11-20020a056402104b00b004617d1f1a7bmr55493775edu.400.1670253762505; Mon, 05 Dec 2022 07:22:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253762; cv=none; d=google.com; s=arc-20160816; b=y/4OaDyCyFt5pEltWNZAd/IMXiNjWB8UL4Jn3+b+Id958jSYIR7t5Eqbwp6tAdEMYc vCjSig/hmBkBOOrMcErfeNdL4TA5Bwo9B2wtGBxiCP8NsN03phYKPrIjnMGaLXjvg4+T tO7uWOCzTNhgUHOTifMC5xeJR7PcRhqaug90Z/2Q+Mz75D+/kXomlC2tQgZk6WkITVpU yLkYY06h+miezEv7S0MrNuKao31MSrZ3lNwixIGY/3AYM/qlEuaqKVNONytFqtd3kZFs i+vDnM//LDAO9gjxl1a8cbzw+MyLR1y4MA/FsfAWUjcfdgcKeH4pjfboxIXKwn5cD1Kn o+zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P/KRpfmdlQUKxAAPrLlGtIIYCnA6qEIJS0aGmE3w/OM=; b=fN6DOeua4cltxjQAcnoLK0pEGLBiQV/WClYMgZZ0FbA0YRjcNlgoDDjH9OyLfMuJ/r Qb6HpILLHVjSbHS7oqHB7GggZnmdtvyotGjbhjbrnZy6l/Kpk3AtnIx19JdIBh9i/W01 jYN6ziLVIamuqQNBXzen7QU4pw6ogAfzEj6vnMHLtJ9EZMu8MrTlVyLD+FwE4psUv+DT e/0RGX3hKyCU7ebfa7RYMeGnWlsRjfWgv8E3ux6PjvhkjA45tl/1g2sHuBMzedjXyXPl nN4gv1V8MU5mK8fpB0IgX7IKoLFv7hFDEYWs1fIxTLHnr2hcMh2GDavSVa5Nneqg9fpt +YmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=jtG+mgTh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q31-20020a056402249f00b0046bf2cb7a3asi9323917eda.578.2022.12.05.07.22.18; Mon, 05 Dec 2022 07:22:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=jtG+mgTh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232709AbiLEPUx (ORCPT + 99 others); Mon, 5 Dec 2022 10:20:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232530AbiLEPTc (ORCPT ); Mon, 5 Dec 2022 10:19:32 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F0871A229 for ; Mon, 5 Dec 2022 07:19:30 -0800 (PST) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B50UkFr019563 for ; Mon, 5 Dec 2022 07:19:30 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=P/KRpfmdlQUKxAAPrLlGtIIYCnA6qEIJS0aGmE3w/OM=; b=jtG+mgTh7Z80eRamWInP2j8Uv8btmTRrxRZYCGOm49bRo9tMfiR3ycsoe+/u5rgu88g1 6kQimJqXC5ofllDbbygKkFozEB4Q7dQGFA+kDt13DPL7oEo7qneol1qQwWdO5bFa6zDS adSeduIBU4TheBfJhIuKOujGMpOAJsP6Uz9gX1fYHTn0yUTywwBCzXvXVfnDxJvqTAiB /Eix5ptf30dLZBp5uO/U18JJB0Nipzn/VCIwePnENXY5CiVs92DfRKgAcF3V2fD3CXdN cnLIyCxpwRm6BYUk9V2c0WKQda8vOCZwnXdz0aDdjCpVsuudjMBn2wFjkhbpinTTVvYg KQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m83cskxvd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:30 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 7A815C6156F4; Mon, 5 Dec 2022 07:00:10 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 05/11] dmapool: speedup DMAPOOL_DEBUG with init_on_alloc Date: Mon, 5 Dec 2022 06:59:31 -0800 Message-ID: <20221205145937.54367-6-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: o7lCOkJUmQNJNock67g4mqJzwFPvpAl5 X-Proofpoint-ORIG-GUID: o7lCOkJUmQNJNock67g4mqJzwFPvpAl5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751388008859771168?= X-GMAIL-MSGID: =?utf-8?q?1751388008859771168?= From: Tony Battersby Avoid double-memset of the same allocated memory in dma_pool_alloc() when both DMAPOOL_DEBUG is enabled and init_on_alloc=1. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index ee993bb59fc27..eaed3ffb42aa8 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -356,7 +356,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, break; } } - if (!(mem_flags & __GFP_ZERO)) + if (!want_init_on_alloc(mem_flags)) memset(retval, POOL_POISON_ALLOCATED, pool->size); #endif spin_unlock_irqrestore(&pool->lock, flags); From patchwork Mon Dec 5 14:59:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29712 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2308582wrr; Mon, 5 Dec 2022 07:09:40 -0800 (PST) X-Google-Smtp-Source: AA0mqf5VRmJLSa0luaptfxvv73FJa/psBwCir3Fkda+XXydaFgZtbSnxG8EEW8g4pUlAuB5/w/Hw X-Received: by 2002:a17:906:bf2:b0:7ac:a2b7:6c97 with SMTP id z18-20020a1709060bf200b007aca2b76c97mr67593013ejg.497.1670252979869; Mon, 05 Dec 2022 07:09:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670252979; cv=none; d=google.com; s=arc-20160816; b=tvS6k+gkiMSGpbl8TpWy/CtxWEaXo1rWX8psdkakitqNQU7KIOJYg1aSVaARI4kCj2 qgdHk6p/kKRhGBmD//LSN8/sQwVGf1LD3pLNv+8/m7Q2PuKm3NF2ZNqH2NdYL7ezVAog 7Itc/f4HV0okkqmiJLJt2+xf08kiEuP5rLIeLzq5iHcYybf08hb1qpBiR1D6N3N9B4j2 /b0vx5DBT5wtUA/bp+anZXFiA4ZWoQUIje4X7vgRjc+Xp+VLW8P9c4Qi/dyKnO8NBOVj eatwgT2GBrLIIw8ZEsDNJVK95k8SKMOB4EOMiCMXa0BEky/vEkf08xknulWk6f0KAYEi LxPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TZNkpjr8/3ibxb7IcC5QH+0+yx1Gj6xL4H95QeGVkb8=; b=JWqo4pbXXwDcmJaUiAfgzFv3hL78ZkfpWI3qoy2MUm9JHQCK12Q6yYd4unMKCsUWZ+ p3YX60q7125HfoFUbIk7Z9NNXlqA/mnbohMK7Dev2f22Q88WmDNqTkMCUCo2Zwp0fwWd +dh5sOFWyNuVS5tUv1GvD4B4hIT9vdNFWjIU6vuWoALOj1omUL353wv+ZR869aXCF6VX vNclMrqOhgvhzV+oV0GQpuUSr7RhmK/kLKWYxpcu4WPGIfEXMrX9RNpGZV5LXZx59CC/ rAc09sJRX1b8jlVYFaACmiFHCkj7KC5YBBbjrc92PKwTo00slPdPVaA2WrqA47X689Ge H5dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=WRuVaZkA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eo5-20020a056402530500b0046b5d89b71dsi10845890edb.612.2022.12.05.07.09.12; Mon, 05 Dec 2022 07:09:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=WRuVaZkA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232287AbiLEPDb (ORCPT + 99 others); Mon, 5 Dec 2022 10:03:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232239AbiLEPD2 (ORCPT ); Mon, 5 Dec 2022 10:03:28 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DF9C13D04 for ; Mon, 5 Dec 2022 07:03:27 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B4NuwBT029063 for ; Mon, 5 Dec 2022 07:03:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=TZNkpjr8/3ibxb7IcC5QH+0+yx1Gj6xL4H95QeGVkb8=; b=WRuVaZkAiQ0pCRZ7ZVydL+UdVVfjJf4lRIdFT/06c2sfdAzJUZ5JuBl4nHPnVQYVhF5z lEpFBpyQtbaOSKEIyWmfg/AHDFJyZOwAadrl++UGZHMkz7sj2GnmEpB/WAGdLAz0Pn1S C1+/kJcxNbH0tiqjl9O9FfIIHhwyDhrxkrMQFlY5+GTu9GFU7o/VcIQug2eVSvoQdNxx dgxLO3JjXsw5mFBb1F7SB/ZC4q6uaw3HHNAIfFAWDfmJtVGUTIgcFQQUCie9GzSKDmTt itseNgU6bmMtNzmOB474PNHt8DaAKRGy+1v7AmhXFzS3ZTkqzZ9UG6/QMkxTkeE0FVK/ mw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m84v5jwf3-16 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:03:26 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:03:21 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 29613C6156F6; Mon, 5 Dec 2022 07:00:11 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 06/11] dmapool: move debug code to own functions Date: Mon, 5 Dec 2022 06:59:32 -0800 Message-ID: <20221205145937.54367-7-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: e5kUmKCFh95jWDr0nKtOqt6HqEWBPCci X-Proofpoint-GUID: e5kUmKCFh95jWDr0nKtOqt6HqEWBPCci X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387188645542801?= X-GMAIL-MSGID: =?utf-8?q?1751387188645542801?= From: Keith Busch Clean up the normal path by moving the debug code outside it. Signed-off-by: Keith Busch --- mm/dmapool.c | 96 +++++++++++++++++++++++++++++----------------------- 1 file changed, 54 insertions(+), 42 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index eaed3ffb42aa8..8a7aa19e650a1 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -294,6 +294,32 @@ void dma_pool_destroy(struct dma_pool *pool) } EXPORT_SYMBOL(dma_pool_destroy); +static inline void pool_check_block(struct dma_pool *pool, void *retval, + unsigned int offset, gfp_t mem_flags) +{ +#ifdef DMAPOOL_DEBUG + int i; + u8 *data = retval; + /* page->offset is stored in first 4 bytes */ + for (i = sizeof(offset); i < pool->size; i++) { + if (data[i] == POOL_POISON_FREED) + continue; + dev_err(pool->dev, "%s %s, %p (corrupted)\n", + __func__, pool->name, retval); + + /* + * Dump the first 4 bytes even if they are not + * POOL_POISON_FREED + */ + print_hex_dump(KERN_ERR, "", DUMP_PREFIX_OFFSET, 16, 1, + data, pool->size, 1); + break; + } + if (!want_init_on_alloc(mem_flags)) + memset(retval, POOL_POISON_ALLOCATED, pool->size); +#endif +} + /** * dma_pool_alloc - get a block of consistent memory * @pool: dma pool that will produce the block @@ -336,29 +362,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, page->offset = *(int *)(page->vaddr + offset); retval = offset + page->vaddr; *handle = offset + page->dma; -#ifdef DMAPOOL_DEBUG - { - int i; - u8 *data = retval; - /* page->offset is stored in first 4 bytes */ - for (i = sizeof(page->offset); i < pool->size; i++) { - if (data[i] == POOL_POISON_FREED) - continue; - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); - - /* - * Dump the first 4 bytes even if they are not - * POOL_POISON_FREED - */ - print_hex_dump(KERN_ERR, "", DUMP_PREFIX_OFFSET, 16, 1, - data, pool->size, 1); - break; - } - } - if (!want_init_on_alloc(mem_flags)) - memset(retval, POOL_POISON_ALLOCATED, pool->size); -#endif + pool_check_block(pool, retval, offset, mem_flags); spin_unlock_irqrestore(&pool->lock, flags); if (want_init_on_alloc(mem_flags)) @@ -381,6 +385,32 @@ static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) return NULL; } +static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, + void *vaddr) +{ +#ifdef DMAPOOL_DEBUG + unsigned int chain = page->offset; + + if ((dma - page->dma) != offset) { + dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + __func__, pool->name, vaddr, &dma); + return true; + } + + while (chain < pool->allocation) { + if (chain != offset) { + chain = *(int *)(page->vaddr + chain); + continue; + } + dev_err(pool->dev, "%s %s, dma %pad already free\n", + __func__, pool->name, &dma); + return true; + } + memset(vaddr, POOL_POISON_FREED, pool->size); +#endif + return false; +} + /** * dma_pool_free - put block back into dma pool * @pool: the dma pool holding the block @@ -408,28 +438,10 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) offset = vaddr - page->vaddr; if (want_init_on_free()) memset(vaddr, 0, pool->size); -#ifdef DMAPOOL_DEBUG - if ((dma - page->dma) != offset) { + if (pool_page_err(pool, page, vaddr)) { spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); return; } - { - unsigned int chain = page->offset; - while (chain < pool->allocation) { - if (chain != offset) { - chain = *(int *)(page->vaddr + chain); - continue; - } - spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, dma %pad already free\n", - __func__, pool->name, &dma); - return; - } - } - memset(vaddr, POOL_POISON_FREED, pool->size); -#endif page->in_use--; *(int *)vaddr = page->offset; From patchwork Mon Dec 5 14:59:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29732 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2318461wrr; Mon, 5 Dec 2022 07:26:24 -0800 (PST) X-Google-Smtp-Source: AA0mqf7jW+o8CQ4fmLea5SmSxKnvUqpA9fdCICmsRPLah62Nccpff/TeR1PT65xwbL0GH5/Sy6t3 X-Received: by 2002:aa7:c38d:0:b0:46c:fa25:f03d with SMTP id k13-20020aa7c38d000000b0046cfa25f03dmr901102edq.291.1670253984000; Mon, 05 Dec 2022 07:26:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253983; cv=none; d=google.com; s=arc-20160816; b=L+3G9aFN4iLcQ7ZKXDRid/HAN01mve4fqs9uL+DI2z1Q8yhj3ZBeaOTkhyMb7ASFgU y94hi7X8h2s/NbIFU9v1mPBC/7VW783AGpsMYX/Vt80xcQ/Zu9+nmEPyhG5+z8ytzprg pAgQ8JKVpXiixQg0plXkz93FA66rDsW6rPqpc/gZO7yqrv0IXKIMo84VUAl4IE2+5QXq XLpYash2QVzRmDKA4pc0kyHRurEhOIFcjKq0Lu6Hp+HPgQTuQVE3wZSN5/IMtMRpKBwf g+hr8ZKE6I26VoPWAU4/qGu+kJ1S98P+zMawL9svo4+I1+qUoiFI96N5TdjZaGAdlZHw Sy7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ADvvFQUdwkTabb+mgOJxdznef25hLYACuBWcCNuRK/M=; b=vyZ3mEoQzBkGXxNq99lHLSs1bPTygE0d9mCDXZHNGPIi6KvD209mx0gceovF81z72t t+wbPY2UoLi4Tx9hI02qoCiKB/hXCybrPCBxrBNrtxYHDrkzZ5Vk5t3qodMRyeDzlNC2 ndqbw9IpKn/VW+ulnzbniaTJpUYokdMnkfGbuOGpSZhHa5BZd6a+dkCbdNfQaF6KaLxa 5KFN44vSouGGXmSx10Tu8hxQHySahnppU8lbjM/zRieFx7FIr6XARO60cd4Zx/89XyZL gA20TwWEsKobxGxcnsyxZ7U8Bqe9vAdX+2TeaNYgvNfuvmlgMlsw1lDZPWKOCUYrANs6 Z7oA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ELaivGWd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nd19-20020a170907629300b007af041e40b4si12295961ejc.129.2022.12.05.07.25.49; Mon, 05 Dec 2022 07:26:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ELaivGWd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232117AbiLEPVG (ORCPT + 99 others); Mon, 5 Dec 2022 10:21:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232621AbiLEPTx (ORCPT ); Mon, 5 Dec 2022 10:19:53 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E0691DF2B for ; Mon, 5 Dec 2022 07:19:34 -0800 (PST) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5CVba5012959 for ; Mon, 5 Dec 2022 07:19:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=ADvvFQUdwkTabb+mgOJxdznef25hLYACuBWcCNuRK/M=; b=ELaivGWdKmR+7P/NGl0TQf7pvFbhWJmnbb62RC3z369pUeY6UlSZuwWKDKnOMSokdVJs Arw3dzuIgZEFFiZKh7MI7upKp3GquRl9YtAp/b9OeNH9VpE3fc6BuIEppTY6lc31y+FK LaHIinB+i7JmHxo6VpvquK9mcBDYlPHu/VKMFVu0hGibcaFJHo9ZodKMSPrw5DipBopS uGmULh1f9rX4DcZCFpIYeeDEGjI873/8R0AuQD5HMA8ykYMx3wM2CDemmSaePK0SWjpn j4fDvG28OUbNt9QR1pEaZAc+xeP4vszHywKECkkB0CfBOhvspsI4pCJ29Qp/DKwpnGas FA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m9gqj9gct-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:33 -0800 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub103.TheFacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:30 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6E86FC615703; Mon, 5 Dec 2022 07:00:11 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 07/11] dmapool: rearrange page alloc failure handling Date: Mon, 5 Dec 2022 06:59:33 -0800 Message-ID: <20221205145937.54367-8-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: M-RugQYgPLFsi3Q5ETmSydZwIfhtxjAM X-Proofpoint-ORIG-GUID: M-RugQYgPLFsi3Q5ETmSydZwIfhtxjAM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751388241752477449?= X-GMAIL-MSGID: =?utf-8?q?1751388241752477449?= From: Keith Busch Handle the error in a condition so the good path can be in the normal flow. Signed-off-by: Keith Busch --- mm/dmapool.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 8a7aa19e650a1..f77e2af4941de 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -222,17 +222,17 @@ static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) return NULL; page->vaddr = dma_alloc_coherent(pool->dev, pool->allocation, &page->dma, mem_flags); - if (page->vaddr) { -#ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); -#endif - pool_initialise_page(pool, page); - page->in_use = 0; - page->offset = 0; - } else { + if (!page->vaddr) { kfree(page); - page = NULL; + return NULL; } +#ifdef DMAPOOL_DEBUG + memset(page->vaddr, POOL_POISON_FREED, pool->allocation); +#endif + pool_initialise_page(pool, page); + page->in_use = 0; + page->offset = 0; + return page; } From patchwork Mon Dec 5 14:59:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29717 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2312830wrr; Mon, 5 Dec 2022 07:16:39 -0800 (PST) X-Google-Smtp-Source: AA0mqf5WSgfPE80r7JQ+kMJRXqW4nZ9rFHm5Qu6CIGqehJkBKFIbjc0S6tzGPLMrQV2pv6FXoNAz X-Received: by 2002:a17:906:8e93:b0:7c0:c10e:3fa9 with SMTP id ru19-20020a1709068e9300b007c0c10e3fa9mr1305475ejc.385.1670253399196; Mon, 05 Dec 2022 07:16:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253399; cv=none; d=google.com; s=arc-20160816; b=m4wNAuEjJcIK3GRQ5ejEFSu2hZQ5zrZLU3kSmL+1JQ9RESCOMMf6uhjC37ARle9bvP gcX0ZzlxuRu/gnf4iqhNgS9sEODxNV+Tu4gcuJu4feYRvtxQGLO6w0feWeBdJ/Rs0b58 doSlQiRUEVWt3NMauTnUA+hYifsoI8EDV8/HW88ZSIy/2CaV+AglxcGErclTFcTamVuD +Ujy9F+ofa2NS1B/JGxhcMmy4wYWhdT9EcUDI8UgOeKFMnC6jMwOB/QKUvsDQLHA724F suP9tcOLgDCg+Nc2qIKJp9GTiL0xysDyUkz1vj1IWMn4VP0mEKq77PsnG2r4G/HIEPkG VB9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5sHDLhRqo9/RufklnGKFVoQ7JnNipMp+QhHHPmz6D+s=; b=yG7bjwlXobnIGL4J9EgQ+JSZOdyXxjtChB4drmasavk4W3ydiqujon27wQTY1npNCK C/30KlXNKHbaT9r8nbl05yEyWqJOTo/hbb4ry0PpsuIb6VZWC5U+75PrSjipY1ONejM9 EJMjY/vm9/YA43MzVfJV9lzm4d+rl0365dZD5scB2V7jkY6r0xlvStuK9tnMpZWt1dk9 5EIgSyQTGF3qSJeKyz6E90xpy2bgrQ1smpZlehFe4yWtXuL/Zq5bIKTKR0MNO92UGKQp N3ODIdQhYXa5wLix0AjZqZPw9TXM32weHovHz092NBDxq5QS7f4c9NJYjm0nvplejvMa kYvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ZZKdYNls; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eh9-20020a0564020f8900b00459195ae973si9971180edb.414.2022.12.05.07.16.13; Mon, 05 Dec 2022 07:16:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=ZZKdYNls; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231890AbiLEPH1 (ORCPT + 99 others); Mon, 5 Dec 2022 10:07:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232239AbiLEPHX (ORCPT ); Mon, 5 Dec 2022 10:07:23 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F0A41D333 for ; Mon, 5 Dec 2022 07:07:22 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5Er24M016570 for ; Mon, 5 Dec 2022 07:07:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=5sHDLhRqo9/RufklnGKFVoQ7JnNipMp+QhHHPmz6D+s=; b=ZZKdYNlsm54rW2y2lXptkIcf6E1liXqw1DNspW49oxKbTmP2vwjiCs4DRRXeems2hYR3 6qapFMudkYne5j5GHbcpsX1UVn2043+GI8tuykSaN824e9yky3kjr5XSyQhbXBShv3uo NQHPOFuGNTr8Ol3n3UahOu1iLedUpdBbIHN0UnVBPhBsHj2PfSxdwWA5qdtDph2S3d3f 3QKeKpPxulwP8jxrB5HSC1fW1vli3JuotxSJ5ZVo+77ZD0hoHMG8DSKDdlSIhOFB1cfS GB6t71Ixo6lHkqw4IkY1RGIVUR9WNGVVcuaR5fcvMul8+T9d7d5pgnQfyKg+Nz+8PnBN /g== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv08u-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:07:21 -0800 Received: from twshared13940.35.frc1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:07:20 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 8B08CC615706; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 08/11] dmapool: consolidate page initialization Date: Mon, 5 Dec 2022 06:59:34 -0800 Message-ID: <20221205145937.54367-9-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 7YLAVJ-68DODBvOn3Nk_Ms-XVOl_65d_ X-Proofpoint-GUID: 7YLAVJ-68DODBvOn3Nk_Ms-XVOl_65d_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387628291168185?= X-GMAIL-MSGID: =?utf-8?q?1751387628291168185?= From: Keith Busch Various fields of the dma pool are set in different places. Move it all to one function. Signed-off-by: Keith Busch --- mm/dmapool.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index f77e2af4941de..33d20ceff18c5 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -202,6 +202,8 @@ static void pool_initialise_page(struct dma_pool *pool, struct dma_page *page) unsigned int offset = 0; unsigned int next_boundary = pool->boundary; + page->in_use = 0; + page->offset = 0; do { unsigned int next = offset + pool->size; if (unlikely((next + pool->size) >= next_boundary)) { @@ -211,6 +213,8 @@ static void pool_initialise_page(struct dma_pool *pool, struct dma_page *page) *(int *)(page->vaddr + offset) = next; offset = next; } while (offset < pool->allocation); + + list_add(&page->page_list, &pool->page_list); } static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) @@ -229,10 +233,6 @@ static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) #ifdef DMAPOOL_DEBUG memset(page->vaddr, POOL_POISON_FREED, pool->allocation); #endif - pool_initialise_page(pool, page); - page->in_use = 0; - page->offset = 0; - return page; } @@ -354,8 +354,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, return NULL; spin_lock_irqsave(&pool->lock, flags); - - list_add(&page->page_list, &pool->page_list); + pool_initialise_page(pool, page); ready: page->in_use++; offset = page->offset; From patchwork Mon Dec 5 14:59:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29730 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2317014wrr; Mon, 5 Dec 2022 07:23:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf4je9Da9ZVk66pTeY3ZMeIkmc/3z/cxSoeBzQGaDAPHgG5+n5YvEHUBSNuJLhl7UL0FjSQb X-Received: by 2002:a17:906:524b:b0:7ad:a030:485c with SMTP id y11-20020a170906524b00b007ada030485cmr67708517ejm.605.1670253828239; Mon, 05 Dec 2022 07:23:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253828; cv=none; d=google.com; s=arc-20160816; b=FOiMJSlJktGwHFs2wndHPyJgvybdSkGx7u/fgzCQbn7v51Oz3DcBiMq4NgHUGXBvuJ qPSL78zZjY1IV0b1lvFNLXA+uY+QiUvT3cPmTBe20TSppm12zI4MSNVb5RpMT2ACICnI NPdLPn6yZkBtEyvkVlvs6eXMY3B8haGg2K3g3KCOW71RAWidcZyS6oGyU0iPYZoeEuNj iaMM5keJBS/9w083N6YKzmJtDZvXFCMCeFZUYR/1oGh/UII+vUoP/vCFsPNCk8U/GX3U /UKIc60tWSjEw4dJ2qXU501z/2SFSL+Xb71p0x7zAx/SvAqM3olbIHOqmGz8Y/zv5oKj FrCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVY03djUbGuw9RyCWd6Eexf1lAzlqKJXlcJv1cor+cc=; b=Hy3+UlPn/t+EDJ3OI68DWrk0G1EMoi54k4pOh5j/+wXkSmkF5MTj9QD1yPkGPDT1w3 pPlaTclaKd9itbAJ4RBSJUPbwmvU0oZyg0vQWfF82NNT8kYJcKbwoXOPDU/Tu/zOUmGK /7Vf3m67lg0y6wY1Y/8K7/flbZZDh32d5A/pDhLNQ9LM6Yf3BA3Mn3+1zfmvjBEID7eB vCV2CV1NhqQ/6+jed8sbf/ef9O4cBVdwt0ZgUQl29s5sQD18pOVvz1ErKe5gf6yWftT1 h0W1VabFPGyUoJ6d+afoSDe6yxTFH522FbKkNGmol7+xEnC++LSRzJP2aAkHNOJvo/Hl yptg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=QLN4GBhn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g8-20020a056402090800b0046b3638938bsi12161028edz.536.2022.12.05.07.23.21; Mon, 05 Dec 2022 07:23:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=QLN4GBhn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232574AbiLEPU4 (ORCPT + 99 others); Mon, 5 Dec 2022 10:20:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232553AbiLEPTe (ORCPT ); Mon, 5 Dec 2022 10:19:34 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EEE71A3AD for ; Mon, 5 Dec 2022 07:19:31 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B4LV1RP002801 for ; Mon, 5 Dec 2022 07:19:31 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=vVY03djUbGuw9RyCWd6Eexf1lAzlqKJXlcJv1cor+cc=; b=QLN4GBhnbA83aq6qOliJgBUpxscTILYzIYn4FBmCQUXsY0guqxzDTGzMbv1wiLpjeJd1 1kXhbx5PtNDp8z7ZTQ8fX09HUzubN2G3RIFZwNrb6q6/an9SdF20Ts13Qs2pjagC42Sm Juvu/nK1ng81PI8ZbYGbcNHF5ep+mp6hU1cA4DUFQyi3EOL0TJamGK9WLtGOlf098kqF YxeRX/gIKjuwc6oYotHgsf9+jhI3xzlBhUTYWLp0F5JSGlJkNiaalY/WlhzJwKKyrFy3 06/uLfeJU75Ff9urSFA88fxTXl+9jGCXavrFP6auX4NXkV9+GpQKEcIw8u7pTAeM7mef cQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561avw9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:31 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id BDE46C615708; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 09/11] dmapool: simplify freeing Date: Mon, 5 Dec 2022 06:59:35 -0800 Message-ID: <20221205145937.54367-10-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: PymH5d3YwBk1q4cny15SOprF69vKwoqV X-Proofpoint-ORIG-GUID: PymH5d3YwBk1q4cny15SOprF69vKwoqV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751388078470085384?= X-GMAIL-MSGID: =?utf-8?q?1751388078470085384?= From: Keith Busch The actions for busy and not busy are mostly the same, so combine these and remove the unnecessary function. Also, the pool is about to be freed so there's no need to poison the page data since we only check for poison on alloc, which can't be done on a freed pool. Signed-off-by: Keith Busch --- mm/dmapool.c | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 33d20ceff18c5..44622f2bf4641 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -241,18 +241,6 @@ static inline bool is_page_busy(struct dma_page *page) return page->in_use != 0; } -static void pool_free_page(struct dma_pool *pool, struct dma_page *page) -{ - dma_addr_t dma = page->dma; - -#ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); -#endif - dma_free_coherent(pool->dev, pool->allocation, page->vaddr, dma); - list_del(&page->page_list); - kfree(page); -} - /** * dma_pool_destroy - destroys a pool of dma memory blocks. * @pool: dma pool that will be destroyed @@ -280,14 +268,14 @@ void dma_pool_destroy(struct dma_pool *pool) mutex_unlock(&pools_reg_lock); list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { - if (is_page_busy(page)) { + if (!is_page_busy(page)) + dma_free_coherent(pool->dev, pool->allocation, + page->vaddr, page->dma); + else dev_err(pool->dev, "%s %s, %p busy\n", __func__, pool->name, page->vaddr); - /* leak the still-in-use consistent memory */ - list_del(&page->page_list); - kfree(page); - } else - pool_free_page(pool, page); + list_del(&page->page_list); + kfree(page); } kfree(pool); @@ -445,11 +433,6 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) page->in_use--; *(int *)vaddr = page->offset; page->offset = offset; - /* - * Resist a temptation to do - * if (!is_page_busy(page)) pool_free_page(pool, page); - * Better have a few empty pages hang around. - */ spin_unlock_irqrestore(&pool->lock, flags); } EXPORT_SYMBOL(dma_pool_free); From patchwork Mon Dec 5 14:59:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29714 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2311341wrr; Mon, 5 Dec 2022 07:14:07 -0800 (PST) X-Google-Smtp-Source: AA0mqf5poq299qXsMHaVWYCTEcEY9rka0t1sOdklCYnlAvcy31hkvRCfzb0y2fZoJ/QlncWczPLf X-Received: by 2002:a05:6402:12d4:b0:46c:a78c:4dcd with SMTP id k20-20020a05640212d400b0046ca78c4dcdmr5536802edx.216.1670253247417; Mon, 05 Dec 2022 07:14:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253247; cv=none; d=google.com; s=arc-20160816; b=1JdTcyYSFb1R8g9UbmfRFTomr06mbGpqY/q7vSX8ZSdzdRHsFLCNXS3KwtgbHIir4o Kn0WO/NOFHTYqPY8u7KEE/LsfpbLBwqt289TdjOwY/cfYasTyoADZuByYcCt/OX/56hc +L2Rey9U8uJrwOF9lwohtm0W10ox5uxa4IZGbQivB07iSIHd/B1ZLXLNIJHgM+No+Xdr PMmdZ6CppSoCiHxc1H3AdFmM/CCpKaOCSkheerRTtrjkCyoONqmWvyS7+MXsgV7lIqkP l+AcGwAilGOaYDWhfuImsRYZjmYPubMPGOo0nQm6ygfDSRMsrsPbleUak4pyiyCMwHxU kWmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=F5ndiXKSIYwdaNgVa7CtE+9m7dnA1GNZSuFoZd2plto=; b=z/AMdWr+sI9J84IfZuaJxN+u470fjz4OfHmfmUiVMmNEOLYENAJaW/WxuHkhHehiAQ dc37sHiWrz3P1UjnOKeuh+rh5CP7vFRiQkfn5K/8qiPvFcz2DDFacdEfO8m1S7tT0XFL kKfPm7XbE86frJ1F4zrD0MDjmSoGmnUkDPyL8oCxPDT7FCinTMhVoHm+2P3GFN6hktg0 7hjUNGYf7WdD22XhxS5FqXrhegrdEQ1/lqOOMateBoND1X815ek9nqaZl63O73/0+ikY OLowBmYQGKDmQAiH+11E6rreKeCZwRxeFN3cXeWR8R2g6eq0QVeIsysHB2IlNsUbPoxg MUpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=mchHDRAi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eo5-20020a056402530500b0046b5d89b71dsi10845890edb.612.2022.12.05.07.13.42; Mon, 05 Dec 2022 07:14:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=mchHDRAi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232181AbiLEPHZ (ORCPT + 99 others); Mon, 5 Dec 2022 10:07:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232070AbiLEPHW (ORCPT ); Mon, 5 Dec 2022 10:07:22 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88D7C1D32E for ; Mon, 5 Dec 2022 07:07:21 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5Er24L016570 for ; Mon, 5 Dec 2022 07:07:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=F5ndiXKSIYwdaNgVa7CtE+9m7dnA1GNZSuFoZd2plto=; b=mchHDRAi5Y0NSIA/d5ZQByFsbGCqJGxyw/MVRIdaNFXNvPVeaq988bd12h7IoioZs6EI SD3zJCyGVReY0pzNdSxOeJf0mpdvBksWs7eulu083Kaw7Nt4wVr0j2r9fPfgxbhGelIq TFbLZfXKIiP/0F9VtKtg0fo7M35YRx/T5YOxFFL93ljc4clxOAq4ILIpu71JEHGgKLHO u97Z05whH18sMXo18co/ua/e3GQksMUQjhtUyV8cbzMmJj1JJ3Rev4+6w9x9tjl5H/5s JahH1T9bKot/R3RA3a2HtRbL19m1hjz87pPdVDQAHQ84L6zHNbRHf+XwlaQzr3VkyNRL Xw== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv08u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:07:21 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:07:19 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 49143C615720; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 10/11] dmapool: don't memset on free twice Date: Mon, 5 Dec 2022 06:59:36 -0800 Message-ID: <20221205145937.54367-11-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 7zteIjRPGoiqQ6V4rJv-8VpjGRg8hYF0 X-Proofpoint-GUID: 7zteIjRPGoiqQ6V4rJv-8VpjGRg8hYF0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387468906497516?= X-GMAIL-MSGID: =?utf-8?q?1751387468906497516?= From: Keith Busch If debug is enabled, dmapool will poison the range, so no need to clear it to 0 immediately before writing over it. Signed-off-by: Keith Busch --- mm/dmapool.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 44622f2bf4641..f5b79c3268856 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -372,10 +372,10 @@ static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) return NULL; } +#ifdef DMAPOOL_DEBUG static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, void *vaddr) { -#ifdef DMAPOOL_DEBUG unsigned int chain = page->offset; if ((dma - page->dma) != offset) { @@ -394,9 +394,17 @@ static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, return true; } memset(vaddr, POOL_POISON_FREED, pool->size); -#endif return false; } +#else +static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, + void *vaddr) +{ + if (want_init_on_free()) + memset(vaddr, 0, pool->size); + return false; +} +#endif /** * dma_pool_free - put block back into dma pool @@ -423,8 +431,6 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) } offset = vaddr - page->vaddr; - if (want_init_on_free()) - memset(vaddr, 0, pool->size); if (pool_page_err(pool, page, vaddr)) { spin_unlock_irqrestore(&pool->lock, flags); return; From patchwork Mon Dec 5 14:59:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 29719 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2314520wrr; Mon, 5 Dec 2022 07:19:43 -0800 (PST) X-Google-Smtp-Source: AA0mqf781fnlZIKYSRmyVoOo9Tt9DPfzIk2s9US5ISMuuYAdeERyO+BHsX/uw8qg4/Y9Lcb49vU1 X-Received: by 2002:a17:906:94c1:b0:7c0:68c8:19db with SMTP id d1-20020a17090694c100b007c068c819dbmr29210179ejy.485.1670253583795; Mon, 05 Dec 2022 07:19:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670253583; cv=none; d=google.com; s=arc-20160816; b=Mc2MQk6/5DLZxL6Ms2cIZIEriKOrUFzID+wfxBxIT8CGMdDXAS8YCAvwy8x2b9VhHo cXKsxXvH2HnncKSfNOlp6lDKmRBV3dC3IMfrnHmn3QTYJDgcLhVgHmMGCB7F+dWMPoV4 SgeoYF944EOEiF9QRW4/ntdPWDkcTgVVBP2gdMEZh57GFZnGCQmbrEfw1DuntVnwI6Iv g082nXid8NcFCYT1k8659BEBvaqwwXk3OdaeY69xmEax77kHhPTGcXOorgH4UoWDHdIz zAoW65VroXK3GaR+uUwuFlrymiD+HMTPA+Vx/CVckY4/KHjsSLzq7Kub7zzAgDANvCFq 3bEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1ZsQ24c7xBxg+bQGPfjzx0xptWB/LenrbaJZ6LhGCHo=; b=Qd4BHF4esOVo40ADGlFWXgSuVXKcfjc9B66gEQmWUPDxsR9YZsZds5y9J/WD4iFw7k 93ix5ndW3tykLzkSlPqhaaxPt7e92uSLkdTKAbKzA4vIvvJmzmGRueEZEDaD42fdD1nH ymrFUYX3jJ5ithTLqcNG71lNVPxh6XDXpiuW8fnBZtg8nLTGHc0KHVANRWyMz32BHcOm Um+V0zf+f2ddjTHMu6whReEvBhAXzOOxTx99utCzDQH8ceARL+gdZX7Z7D6WVqvZPx/j SDHLPa21Q7NGQ9KRGKQuvM+iyQ6JwKKdTINFKsY4oOcbzFKPF+wzp8dzOS1TzpeYHdy5 mnbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=U69QM2oJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id rh16-20020a17090720f000b007c080b02137si9856847ejb.932.2022.12.05.07.19.19; Mon, 05 Dec 2022 07:19:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@meta.com header.s=s2048-2021-q4 header.b=U69QM2oJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=meta.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230246AbiLEPOW (ORCPT + 99 others); Mon, 5 Dec 2022 10:14:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231838AbiLEPNf (ORCPT ); Mon, 5 Dec 2022 10:13:35 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61B571010 for ; Mon, 5 Dec 2022 07:13:33 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5B8jGS025632 for ; Mon, 5 Dec 2022 07:13:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=1ZsQ24c7xBxg+bQGPfjzx0xptWB/LenrbaJZ6LhGCHo=; b=U69QM2oJlw0o5rHLUMeD4J7s5YpqYUxubHHE6brB0x+xvIWzZ1cCMB52Hc4FMFjYCbG0 FjRk3vG1H9SQ+Is+D1H6+E0h409TS1Ff1pZo1IuHIqvaeI36x5GIqZ2wozARJc1Bbkrp stMq0YLh45JigeZonkayo/DtZZgCktoO1ldw1RExXp29COnNW7q0p8kaGh4WmHVwJpmN dUyT1vIGkBAynotsuKsNeBEGsFLmYTtEQyN1tlE8b2OH9gXfeQduxTc4o8agNHV45r8o 1h/wUNZDm8vEy+lWxsSW8N1ttrrKJ/rlxBK0pJK5pAnXLVlQRKat0FcnevujYxzJfSJY Ig== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561auny-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:13:33 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:13:31 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6A86CC61572C; Mon, 5 Dec 2022 07:00:13 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 11/11] dmapool: link blocks across pages Date: Mon, 5 Dec 2022 06:59:37 -0800 Message-ID: <20221205145937.54367-12-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: q02K3_yzZF6hoTErihxaKG4rddpwEent X-Proofpoint-ORIG-GUID: q02K3_yzZF6hoTErihxaKG4rddpwEent X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751387821855496694?= X-GMAIL-MSGID: =?utf-8?q?1751387821855496694?= From: Keith Busch The allocated dmapool pages are never freed for the lifetime of the pool. There is no need for the two level list+stack lookup for finding a free block since nothing is ever removed from the list. Just use a simple stack, reducing time complexity to constant. The implementation inserts the stack linking elements and the dma handle of the block within itself when freed. This means the smallest possible dmapool block is increased to at most 16 bytes to accomodate these fields, but there are no exisiting users requesting a dma pool smaller than that anyway. Removing the list has a significant change in performance. Using the kernel's micro-benchmarking self test: Before: # modprobe dmapool_test dmapool test: size:16 blocks:8192 time:57282 dmapool test: size:64 blocks:8192 time:172562 dmapool test: size:256 blocks:8192 time:789247 dmapool test: size:1024 blocks:2048 time:371823 dmapool test: size:4096 blocks:1024 time:362237 After: # modprobe dmapool_test dmapool test: size:16 blocks:8192 time:24997 dmapool test: size:64 blocks:8192 time:26584 dmapool test: size:256 blocks:8192 time:33542 dmapool test: size:1024 blocks:2048 time:9022 dmapool test: size:4096 blocks:1024 time:6045 The module test allocates quite a few blocks that may not accurately represent how these pools are used in real life. For a more marco level benchmark, running fio high-depth + high-batched on nvme, this patch shows submission and completion latency reduced by ~100usec each, 1% IOPs improvement, and perf record's time spent in dma_pool_alloc/free were reduced by half. Signed-off-by: Keith Busch --- mm/dmapool.c | 213 ++++++++++++++++++++++++++------------------------- 1 file changed, 108 insertions(+), 105 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index f5b79c3268856..ca6cc5d3d9e53 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -40,13 +40,22 @@ #define DMAPOOL_DEBUG 1 #endif +struct dma_block { + struct dma_block *next_block; + dma_addr_t dma; +}; + struct dma_pool { /* the pool */ struct list_head page_list; spinlock_t lock; struct device *dev; + struct dma_block *next_block; unsigned int size; unsigned int allocation; unsigned int boundary; + unsigned int nr_blocks; + unsigned int nr_active; + unsigned int nr_pages; char name[32]; struct list_head pools; }; @@ -55,8 +64,6 @@ struct dma_page { /* cacheable header for 'allocation' bytes */ struct list_head page_list; void *vaddr; dma_addr_t dma; - unsigned int in_use; - unsigned int offset; }; static DEFINE_MUTEX(pools_lock); @@ -64,30 +71,18 @@ static DEFINE_MUTEX(pools_reg_lock); static ssize_t pools_show(struct device *dev, struct device_attribute *attr, char *buf) { - int size; - struct dma_page *page; struct dma_pool *pool; + unsigned size; size = sysfs_emit(buf, "poolinfo - 0.1\n"); mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { - unsigned pages = 0; - size_t blocks = 0; - - spin_lock_irq(&pool->lock); - list_for_each_entry(page, &pool->page_list, page_list) { - pages++; - blocks += page->in_use; - } - spin_unlock_irq(&pool->lock); - /* per-pool info, no real statistics yet */ - size += sysfs_emit_at(buf, size, "%-16s %4zu %4zu %4u %2u\n", - pool->name, blocks, - (size_t) pages * - (pool->allocation / pool->size), - pool->size, pages); + size += sysfs_emit_at(buf, size, "%-16s %4u %4u %4u %2u\n", + pool->name, pool->nr_active, + pool->nr_blocks, pool->size, + pool->nr_pages); } mutex_unlock(&pools_lock); @@ -96,6 +91,25 @@ static ssize_t pools_show(struct device *dev, struct device_attribute *attr, cha static DEVICE_ATTR_RO(pools); +static inline struct dma_block *pool_block_pop(struct dma_pool *pool) +{ + struct dma_block *block = pool->next_block; + + if (block) { + pool->next_block = block->next_block; + pool->nr_active++; + } + return block; +} + +static inline void pool_block_push(struct dma_pool *pool, struct dma_block *block, + dma_addr_t dma) +{ + block->dma = dma; + block->next_block = pool->next_block; + pool->next_block = block; +} + /** * dma_pool_create - Creates a pool of consistent memory blocks, for dma. * @name: name of pool, for diagnostics @@ -136,8 +150,8 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, if (size == 0 || size > INT_MAX) return NULL; - else if (size < 4) - size = 4; + if (size < sizeof(struct dma_block)) + size = sizeof(struct dma_block); size = ALIGN(size, align); allocation = max_t(size_t, size, PAGE_SIZE); @@ -162,6 +176,10 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, retval->size = size; retval->boundary = boundary; retval->allocation = allocation; + retval->nr_blocks = 0; + retval->nr_active = 0; + retval->nr_pages = 0; + retval->next_block = NULL; INIT_LIST_HEAD(&retval->pools); @@ -199,22 +217,24 @@ EXPORT_SYMBOL(dma_pool_create); static void pool_initialise_page(struct dma_pool *pool, struct dma_page *page) { - unsigned int offset = 0; - unsigned int next_boundary = pool->boundary; - - page->in_use = 0; - page->offset = 0; - do { - unsigned int next = offset + pool->size; - if (unlikely((next + pool->size) >= next_boundary)) { - next = next_boundary; + unsigned int next_boundary = pool->boundary, offset = 0; + struct dma_block *block; + + while (offset < pool->allocation) { + if (offset > next_boundary) { + offset = next_boundary; next_boundary += pool->boundary; + continue; } - *(int *)(page->vaddr + offset) = next; - offset = next; - } while (offset < pool->allocation); + + block = page->vaddr + offset; + pool_block_push(pool, block, page->dma + offset); + offset += pool->size; + pool->nr_blocks++; + } list_add(&page->page_list, &pool->page_list); + pool->nr_pages++; } static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) @@ -236,11 +256,6 @@ static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) return page; } -static inline bool is_page_busy(struct dma_page *page) -{ - return page->in_use != 0; -} - /** * dma_pool_destroy - destroys a pool of dma memory blocks. * @pool: dma pool that will be destroyed @@ -252,7 +267,7 @@ static inline bool is_page_busy(struct dma_page *page) void dma_pool_destroy(struct dma_pool *pool) { struct dma_page *page, *tmp; - bool empty = false; + bool empty = false, busy = false; if (unlikely(!pool)) return; @@ -267,13 +282,15 @@ void dma_pool_destroy(struct dma_pool *pool) device_remove_file(pool->dev, &dev_attr_pools); mutex_unlock(&pools_reg_lock); + if (pool->nr_active) { + dev_err(pool->dev, "%s %s busy\n", __func__, pool->name); + busy = true; + } + list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { - if (!is_page_busy(page)) + if (!busy) dma_free_coherent(pool->dev, pool->allocation, page->vaddr, page->dma); - else - dev_err(pool->dev, "%s %s, %p busy\n", __func__, - pool->name, page->vaddr); list_del(&page->page_list); kfree(page); } @@ -282,18 +299,18 @@ void dma_pool_destroy(struct dma_pool *pool) } EXPORT_SYMBOL(dma_pool_destroy); -static inline void pool_check_block(struct dma_pool *pool, void *retval, - unsigned int offset, gfp_t mem_flags) +static inline void pool_check_block(struct dma_pool *pool, struct dma_block *block, + gfp_t mem_flags) { -#ifdef DMAPOOL_DEBUG +#ifdef DMAPOOL_DEBUG + u8 *data = block; int i; - u8 *data = retval; - /* page->offset is stored in first 4 bytes */ - for (i = sizeof(offset); i < pool->size; i++) { + + for (i = sizeof(struct dma_block); i < pool->size; i++) { if (data[i] == POOL_POISON_FREED) continue; - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); + dev_err(pool->dev, "%s %s, %p (corrupted)\n", __func__, + pool->name, block); /* * Dump the first 4 bytes even if they are not @@ -303,8 +320,9 @@ static inline void pool_check_block(struct dma_pool *pool, void *retval, data, pool->size, 1); break; } + if (!want_init_on_alloc(mem_flags)) - memset(retval, POOL_POISON_ALLOCATED, pool->size); + memset(block, POOL_POISON_ALLOCATED, pool->size); #endif } @@ -321,44 +339,41 @@ static inline void pool_check_block(struct dma_pool *pool, void *retval, void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, dma_addr_t *handle) { - unsigned long flags; + struct dma_block *block; struct dma_page *page; - unsigned int offset; - void *retval; + unsigned long flags; might_alloc(mem_flags); spin_lock_irqsave(&pool->lock, flags); - list_for_each_entry(page, &pool->page_list, page_list) { - if (page->offset < pool->allocation) - goto ready; - } - - /* pool_alloc_page() might sleep, so temporarily drop &pool->lock */ - spin_unlock_irqrestore(&pool->lock, flags); + block = pool_block_pop(pool); + if (!block) { + /* + * pool_alloc_page() might sleep, so temporarily drop + * &pool->lock + */ + spin_unlock_irqrestore(&pool->lock, flags); - page = pool_alloc_page(pool, mem_flags & (~__GFP_ZERO)); - if (!page) - return NULL; + page = pool_alloc_page(pool, mem_flags & (~__GFP_ZERO)); + if (!page) + return NULL; - spin_lock_irqsave(&pool->lock, flags); - pool_initialise_page(pool, page); - ready: - page->in_use++; - offset = page->offset; - page->offset = *(int *)(page->vaddr + offset); - retval = offset + page->vaddr; - *handle = offset + page->dma; - pool_check_block(pool, retval, offset, mem_flags); + spin_lock_irqsave(&pool->lock, flags); + pool_initialise_page(pool, page); + block = pool_block_pop(pool); + } spin_unlock_irqrestore(&pool->lock, flags); + *handle = block->dma; + pool_check_block(pool, block, mem_flags); if (want_init_on_alloc(mem_flags)) - memset(retval, 0, pool->size); + memset(block, 0, pool->size); - return retval; + return block; } EXPORT_SYMBOL(dma_pool_alloc); +#ifdef DMAPOOL_DEBUG static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) { struct dma_page *page; @@ -372,33 +387,35 @@ static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) return NULL; } -#ifdef DMAPOOL_DEBUG -static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, - void *vaddr) +static inline bool pool_block_err(struct dma_pool *pool, void *vaddr, + dma_addr_t dma) { - unsigned int chain = page->offset; + struct dma_block *block = pool->next_block; + struct dma_page *page; - if ((dma - page->dma) != offset) { - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + page = pool_find_page(pool, dma); + if (!page) { + dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", __func__, pool->name, vaddr, &dma); return true; } - while (chain < pool->allocation) { - if (chain != offset) { - chain = *(int *)(page->vaddr + chain); + while (block) { + if (block != vaddr) { + block = block->next_block; continue; } dev_err(pool->dev, "%s %s, dma %pad already free\n", __func__, pool->name, &dma); return true; } + memset(vaddr, POOL_POISON_FREED, pool->size); return false; } #else -static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, - void *vaddr) +static inline bool pool_block_err(struct dma_pool *pool, void *vaddr, + dma_addr_t dma) { if (want_init_on_free()) memset(vaddr, 0, pool->size); @@ -417,28 +434,14 @@ static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *page, */ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) { - struct dma_page *page; + struct dma_block *block = vaddr; unsigned long flags; - unsigned int offset; spin_lock_irqsave(&pool->lock, flags); - page = pool_find_page(pool, dma); - if (!page) { - spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); - return; + if (!pool_block_err(pool, vaddr, dma)) { + pool_block_push(pool, block, dma); + pool->nr_active--; } - - offset = vaddr - page->vaddr; - if (pool_page_err(pool, page, vaddr)) { - spin_unlock_irqrestore(&pool->lock, flags); - return; - } - - page->in_use--; - *(int *)vaddr = page->offset; - page->offset = offset; spin_unlock_irqrestore(&pool->lock, flags); } EXPORT_SYMBOL(dma_pool_free);