From patchwork Tue Apr 4 12:39:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79070 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2997091vqo; Tue, 4 Apr 2023 05:46:05 -0700 (PDT) X-Google-Smtp-Source: AKy350Yu3zODh8/VEKXay9gjdHFRdoehtpKVujz6iOxyJ7gahbyYq6t/Zc13zJkj9QWOUfe8/BEc X-Received: by 2002:a17:902:ca05:b0:1a1:b440:3773 with SMTP id w5-20020a170902ca0500b001a1b4403773mr2118266pld.27.1680612364754; Tue, 04 Apr 2023 05:46:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612364; cv=none; d=google.com; s=arc-20160816; b=QrsLLr71+Hj+Vx+E6bUdX/TRXb+gIioRSj4vKgOXINid2vQ8cDvyENjPDh6E7hUlp+ 06AADLoVU0l9PkJ/r7PxW1cplI0btombs/fNnpSZNNzzkon0lI7I0PtqJVxmg2Ub0ERC bGc3mHI/sd0skSH87d+AYZzquxDHDFPM/pZC05XTWHGC2z0nWn+W05zB5Sd7w3s7pF+B 2VdHfe0O5cO2ZKzPmckw5qJgpppA1Rogj3o5g0oU7AxzDZntRe6qA5ZzKt66j0fIBB2p 6RdMP05MmUoWvmE5oTPTdQPAP6XQ7NtbzUholvmV/mc0nUPEGMg/oU4uwtPOsIRzNYZ8 SMbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=B4jUcK/aZoVr0sdbTUSCMWYGWimEni1x0jIOLPJhcKwGziNAlIzNymfmm0WqkwxdQH nzoHx623qIi5hXJIb1WmYV02UeLhEBqScGFJuZR9pIWcWhtOfAvnE9lmg/Z5MFkxM84c 5dhaDKBhz6IEorgx3eK61BEiX/chYHOuIQQiInmoX5fwO8zxjAiP1mjyoIYAIBrSNQcu nvd2JKwca3JfzhuzWVZnUUsUaRHMxfH5BdFvNKiDC2fcqcvUnM1OExxEyOFjU6K4ucA+ qZJzVNBOi0hV6z5ZRphbsV7rMEJISDbOCcRU//ZGWjG2tgsYdZe6OZQf/7Ka+U8SDqur 6fYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="X/J1llEl"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b15-20020a63e70f000000b0050bfc85d989si9756664pgi.154.2023.04.04.05.45.50; Tue, 04 Apr 2023 05:46:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="X/J1llEl"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234352AbjDDMln (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234944AbjDDMlB (ORCPT ); Tue, 4 Apr 2023 08:41:01 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BA941BC7; Tue, 4 Apr 2023 05:40:58 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id cn12so130060658edb.4; Tue, 04 Apr 2023 05:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=X/J1llEltWTp4lG2E3lTY7RTuXuKmHYGHhZGxSbzgjJt2spsoc45a6S/UqZmy9U95v 3AJyxDDnt6hnM3wGswTBwN1w1rruCJajzH0hS0det28+p4PqaXO8No707bkfYlIH4gT4 Ex2bhCgpaA6oAvTi8s4cYa4nscj59hZrpo1b3ng1sk6YDj3US4/VIvvsp4VvCSLg+Ujx h9EEbua/G4akmIB9ncr08dx49v8Qc70b1mVxMeD3YoySqg/Uc5IN/XwrJFru7RQb+Ybl izyZWc77RLGkl1EcLvSEk2Bp2iPDemOm0BTIkg8dWFmwopEODxnBjKwkbHz2tviJFjob IU8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=RORmEIuMWGFX33I16cb6QNsDJXHrR13ShHDaBgvBSPTZw50EamLC490Rc0GncfE4wv 7s4LmboPKH1ClcLrVM6NFGw1pveRK+udra3LEMXle55Xi3IatgYBgYFjqsFWDg3IdmCT c91zEkMf2ZPHoujLaLBrZ88qOc39yYzJ+Q9j0rKBqGhyd50Bi+CS6RkzPPKX5ldIyLMx I2cva+YBgZBnyA1SRsVpVKpJpDkW7eeN4/JMfK7FvVtaOl53GMbs8irKCnbEbs9Q53h7 Fxnvt/DuLQEeYghNBggxIEjaqnIiagmN67d96jFxms9BoRDMsPPu4gBblLICiYncnx60 K5aA== X-Gm-Message-State: AAQBX9eWMsB/9a17Mz/cGOp3xStxoDBFImnJ9V/r1fGIbbgE6OqWg8j8 AEjTDpEc+ADikCg72l+KVgJeBR4VWYw= X-Received: by 2002:a17:907:8c10:b0:8f0:143d:ee34 with SMTP id ta16-20020a1709078c1000b008f0143dee34mr2624104ejc.1.1680612057944; Tue, 04 Apr 2023 05:40:57 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:57 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 13/13] io_uring/rsrc: add custom limit for node caching Date: Tue, 4 Apr 2023 13:39:57 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249790617219164?= X-GMAIL-MSGID: =?utf-8?q?1762249790617219164?= The number of entries in the rsrc node cache is limited to 512, which still seems unnecessarily large. Add per cache thresholds and set to to 32 for the rsrc node cache. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/alloc_cache.h | 6 ++++-- io_uring/io_uring.c | 9 ++++++--- io_uring/rsrc.h | 2 ++ 4 files changed, 13 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5d772e36e7fc..4a6ce03a4903 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -190,6 +190,7 @@ struct io_ev_fd { struct io_alloc_cache { struct io_wq_work_node list; unsigned int nr_cached; + unsigned int max_cached; size_t elem_size; }; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 2fbecaa3a1ba..851a527afb5e 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -13,7 +13,7 @@ struct io_cache_entry { static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, struct io_cache_entry *entry) { - if (cache->nr_cached < IO_ALLOC_CACHE_MAX) { + if (cache->nr_cached < cache->max_cached) { cache->nr_cached++; wq_stack_add_head(&entry->node, &cache->list); /* KASAN poisons object */ @@ -38,10 +38,12 @@ static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *c return NULL; } -static inline void io_alloc_cache_init(struct io_alloc_cache *cache, size_t size) +static inline void io_alloc_cache_init(struct io_alloc_cache *cache, + unsigned max_nr, size_t size) { cache->list.next = NULL; cache->nr_cached = 0; + cache->max_cached = max_nr; cache->elem_size = size; } diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index da36fa1eeac9..ae90d2753e0d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,9 +310,12 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); - io_alloc_cache_init(&ctx->rsrc_node_cache, sizeof(struct io_rsrc_node)); - io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); - io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); + io_alloc_cache_init(&ctx->rsrc_node_cache, IO_NODE_ALLOC_CACHE_MAX, + sizeof(struct io_rsrc_node)); + io_alloc_cache_init(&ctx->apoll_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct async_poll)); + io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 7ab9b2b2e757..8729f2fee256 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -6,6 +6,8 @@ #include "alloc_cache.h" +#define IO_NODE_ALLOC_CACHE_MAX 32 + #define IO_RSRC_TAG_TABLE_SHIFT (PAGE_SHIFT - 3) #define IO_RSRC_TAG_TABLE_MAX (1U << IO_RSRC_TAG_TABLE_SHIFT) #define IO_RSRC_TAG_TABLE_MASK (IO_RSRC_TAG_TABLE_MAX - 1)