From patchwork Tue Apr 4 12:39:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2995909vqo; Tue, 4 Apr 2023 05:43:46 -0700 (PDT) X-Google-Smtp-Source: AKy350bZWyw2Dv98kRkuM32eVAY5R42dU0ZMYkm+UAvv5902gJtwoK/ruUEuAB7QGewS4UqxJZ5b X-Received: by 2002:a17:90a:1a05:b0:23d:5485:b80e with SMTP id 5-20020a17090a1a0500b0023d5485b80emr2865962pjk.6.1680612225967; Tue, 04 Apr 2023 05:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612225; cv=none; d=google.com; s=arc-20160816; b=B19WsZTsPOhauMbfLX3rnVd1TQ9QVeozc8WcP0B6xxXfLXRYOrC8FoaKvfmhf+VfTK Uoh0eKHm1uVnA4MA+n5GmW9FOKF8hnBmuRkaoqL4khRPp8eDtujpxQO9avneNCf4mT8L JGQ3lm5/hgV8cwajuI5D/k18c8Jx9DuqvxF4BJTlSl7EHNEshe7xWPQzdw8QHGrFQAmA hTJpXvARLokwjz69XhoJ/gqyup2gm96dcNWB8m3Dqod1VCpjLWXfhp7wpwUu4gNM2lwf 7FTofQ4JDflUO/svV5kKo0hNnv9IgfmQf9vCeODxt0HSD9k546H8K7SN9ZyNhFhIpRGv Rmaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=MBZlopQlaa3q6pwmx/rtZaXmrUyLc41MivYhwz5LpLAwDK4FKvM0xg4wdTOX+cGfKM W4BJyBn3dpRgb3/rH2LrZr16FSHNhAffQeoHYAfmru8vaNOMfYVLhchD0MdgzJCrY/3g UWthYHY6925je24meUrSDveM4Z3F0Hd9Ta6GPFsDquW+KpfCzyErw7jflJCt0R3JDIQ4 +jrM7ckkACPtUVJdvjEDZwW28w033dsuG6BfGHB6kin/ZcH4v4iXwZHvkKMtbaLW6+2Y In+F81wcQdJ4e/jnaVvp9sljZw4jTz8XChLtESnsssL2+ikvR9LNRCxMgPbD+sPuhGSk Pxxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="CD/HtzQQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020a17090a784100b0022c5a147e9fsi1566583pjl.115.2023.04.04.05.43.32; Tue, 04 Apr 2023 05:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="CD/HtzQQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234623AbjDDMl2 (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234770AbjDDMk7 (ORCPT ); Tue, 4 Apr 2023 08:40:59 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A40D135AC; Tue, 4 Apr 2023 05:40:56 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id h8so130056090ede.8; Tue, 04 Apr 2023 05:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=CD/HtzQQLAjag/OQbDs6IryLQwHEpG9SXXXTqYvROafActLs1tDcaGlOMLU9KNHfjA lW2BTZI6EzPRmCopK0CB0uUDtOV4KbTRSyBMATh8KMVNYYmjPq2+Zspnl+40GCUDh1Lt 0xtuaTH6wWSEGVy3Hq9SGkn6DO6L61SqTOUkMcBwwt1o152GRY6PyHU+bygxcOU51X9K vkvSvbz8ocmBdquouI+tU5gNKowH8aBZ29PdRaKarjt1LR2rj1tLUC+77umQ8WE9oU2h Fn1D3gh4lJUB2ALhTwVL8kEBQRHMINtIIlnOMSycxTkJ3ylHh586DaEjFM64WnqIYnA3 1pKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=pFket0w3gF4CNf3iQ2K/Sntwob8nDNR5+D6ZdXaIaTQ245jhttpk/hS82r2FzwLMEc xx0c6EsLjzM5xtZ+pQHQ+krSSEYfjJvVsUjjacHlFJtb8Umf3UqSfO92n32O3jW+RNBd bLPExElJngz9Mq2Nrb1MajrT69OzGqeR3jvy69enTjZR+BJpvs7K9AAZhr6gm5dBc/YW ISotbmhpihbqQyklSOc09t2UEd3bJUKOLVvRkhbGfOgT6jZ70V3FyllkXZojH7X8ymYU G+s4vd7A+zKDKf9ekoXixfn01vAg9sYU/I8WXq+/X3TtX/yiRtKwyvI5KLt3zB4sRbsO bJbQ== X-Gm-Message-State: AAQBX9eQFIsnkWIdbU5y3yX4UDeNMM/0gCS0eNlwH9twu+oi3BMnM7YB xRuqB3Y9FKFrqCEqV7MXJpiVBFtKOfw= X-Received: by 2002:a17:907:988e:b0:947:bd54:cb8e with SMTP id ja14-20020a170907988e00b00947bd54cb8emr2478777ejc.10.1680612054982; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 08/13] io_uring/rsrc: optimise io_rsrc_put allocation Date: Tue, 4 Apr 2023 13:39:52 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249645026453377?= X-GMAIL-MSGID: =?utf-8?q?1762249645026453377?= Every io_rsrc_node keeps a list of items to put, and all entries are kmalloc()'ed. However, it's quite often to queue up only one entry per node, so let's add an inline entry there to avoid extra allocations. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 51 ++++++++++++++++++++++++++++++++----------------- io_uring/rsrc.h | 2 ++ 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 2378beecdc0a..9647c02be0dc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,26 +140,34 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } +static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, + struct io_rsrc_put *prsrc) +{ + struct io_ring_ctx *ctx = rsrc_data->ctx; + + if (prsrc->tag) { + if (ctx->flags & IORING_SETUP_IOPOLL) { + mutex_lock(&ctx->uring_lock); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + mutex_unlock(&ctx->uring_lock); + } else { + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + } + } + rsrc_data->do_put(ctx, prsrc); +} + static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; + if (ref_node->inline_items) + io_rsrc_put_work_one(rsrc_data, &ref_node->item); + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); - - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } - - rsrc_data->do_put(ctx, prsrc); + io_rsrc_put_work_one(rsrc_data, prsrc); kfree(prsrc); } @@ -251,6 +259,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; + ref_node->inline_items = 0; return ref_node; } @@ -729,15 +738,23 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, { u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; + bool inline_item = true; - prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); - if (!prsrc) - return -ENOMEM; + if (!node->inline_items) { + prsrc = &node->item; + node->inline_items++; + } else { + prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); + if (!prsrc) + return -ENOMEM; + inline_item = false; + } prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->item_list); + if (!inline_item) + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 509a5ea7eabf..11703082d125 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -49,6 +49,8 @@ struct io_rsrc_node { * came from the same table and so are of the same type. */ struct list_head item_list; + struct io_rsrc_put item; + int inline_items; }; struct io_mapped_ubuf {