From patchwork Tue Apr 4 12:39:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79080 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3001277vqo; Tue, 4 Apr 2023 05:53:28 -0700 (PDT) X-Google-Smtp-Source: AKy350Zs1+IfTXZTn8z3xsC+Du+qpjnJyGNmCG5TUnn7PM8d2dldSUJrshgsjqB3QfLyCFbIp9HC X-Received: by 2002:a05:6402:5114:b0:4fb:223a:76e7 with SMTP id m20-20020a056402511400b004fb223a76e7mr20643242edd.18.1680612808361; Tue, 04 Apr 2023 05:53:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612808; cv=none; d=google.com; s=arc-20160816; b=gREaEz7P6jWjZPS/ALeqa81DMkBhYpvAHJCQqNX/EHNoXj7GJ4C0jsqWM6+wgYRRhv OLJ1gdNk3b2QXXCl29rRpSA6xARd0sWa1mUVgp4gr4jKeeD7TDNP06+11n8UTWL+uBCE 3r6Q4vUOD6o7GAfnd6yqt2Sann8Pr0Bi5Am0QRqkWLncaz8d5pCd3p8T13xkxtBL3gDN WV9mCZf5mNR9eAxzPMw7PqtkM6UUlxA6TlO++S29kFa9z7Rye4eEEUXFdLebYgF2v/sR GWhtJo+ma1mXdbs2VgqJ6PXYwDduneD8ZMZZq8OjqSTbiFB0qpEAL91bEP/oeCb3NzGU h9pA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ynSkjWiStdChDNmK6w/UU6qev6e/KMNN2NT7DgiXXHE=; b=bFoY+7aQwRZMxJkOwprptpsrIToFZsFP0K9ic77YjFxEANIOyvCfFOzEdnV0HgGNbK i7sNW5pRp0JEhnKnIhozO5dMsiTVuTMA/hA3F1iY24kzcnirH4ESw+tawGNXeqvV7EaB UDQgJfdZT1XwYhDRrDkA4wQfzuFCucqdpGIMJGNS5/LYN0STA5v6udOACCwapt8BfgHh t2ck63X/lYodJZpGF9hCvqRTiHL3T621gEgcM4vUbNPUdZP4TSJbVaL0qF0iuvLViacL ziBApT7ss+a8t4Tl1Mf0uCBmxrKpkq57voqaobIGyWhAOu39UOGIecT8NBGotUQ/TTV5 2OeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=aPtNfhhD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d8-20020aa7d688000000b004fb33bbfa18si4895802edr.130.2023.04.04.05.53.04; Tue, 04 Apr 2023 05:53:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=aPtNfhhD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234936AbjDDMlA (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234499AbjDDMk4 (ORCPT ); Tue, 4 Apr 2023 08:40:56 -0400 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B489AE4A; Tue, 4 Apr 2023 05:40:52 -0700 (PDT) Received: by mail-ed1-x52c.google.com with SMTP id fi11so6525271edb.10; Tue, 04 Apr 2023 05:40:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ynSkjWiStdChDNmK6w/UU6qev6e/KMNN2NT7DgiXXHE=; b=aPtNfhhD6HDyNQalImrT+8W73E2NbJu5FXJZWAiY0ChnnI3+bKrQz43eWEd9PNjY/y a7ynjbzXgZuY/TPKpj/LrlPgSsGh6SAUCFZoiM0jv+CQZGg3LhgkVN7GdIXgsuxFJ99i OfDsvIWKmojCZu2z+CQM83ChibNGTU81YOh5vuOIh5vC2Cma05hOIAvUzBoL/SFCNAf2 SGB4hZiESzRvM5sxGhix+BiKDV6ewfWwpCAot0L/CKY7dwvjg4mdde3l6hZXT6qr8vJS P583DMeH/Nxhe7EGxAqbP7n20NtLnHet0dB8E9spE94CthjK/CZYlIIIKhfmI83qlpYA WsYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ynSkjWiStdChDNmK6w/UU6qev6e/KMNN2NT7DgiXXHE=; b=YuZVd7Ik8M2dEhIjotqWIBODErKVIPZs4SC2QvEQQc+oZXa/MeA3GZeTJkBt4Pexoa Wry4rd/AfO2dGJx5jfJ8lj/aFEeLC6+coUAkezhAQYI2pXdrZWX9pI0WHQZLd8XR40/S l8oxer0ZxzF9MdUd7rIklNVqqv+ovkSHuZIXMqwfLbMyvq2jYZnegGOe41Zmp2GGBoQL rDGx+1kdkaplc0KcikLubA1njGzDnIk1rhyr/ENZEIjp8UHgXrx70FmRUQRi2atpVo5e /E0FiHPb3M8uE7ORzK9GgEuPNT/wu5sMkF/+moUJVqZddX3ePDZiR3J4DCnmAawFvHZI 5k8Q== X-Gm-Message-State: AAQBX9e4E17oD42AUWx3LhH0ROmFpaPHjZSXq4zPF5ieQbJ4xmVVxC9W azuYpv/Y//r8QjGWnlyaCZS3LYDMKnw= X-Received: by 2002:a17:907:2071:b0:921:5e7b:1c27 with SMTP id qp17-20020a170907207100b009215e7b1c27mr2121888ejb.24.1680612050967; Tue, 04 Apr 2023 05:40:50 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 01/13] io_uring/rsrc: use non-pcpu refcounts for nodes Date: Tue, 4 Apr 2023 13:39:45 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250256454816204?= X-GMAIL-MSGID: =?utf-8?q?1762250256454816204?= One problem with the current rsrc infra is that often updates will generates lots of rsrc nodes, each carry pcpu refs. That takes quite a lot of memory, especially if there is a stall, and takes lots of CPU cycles. Only pcpu allocations takes >50 of CPU with a naive benchmark updating files in a loop. Replace pcpu refs with normal refcounting. There is already a hot path avoiding atomics / refs, but following patches will further improve it. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 15 +++++---------- io_uring/rsrc.h | 6 ++++-- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index a5ed0ee7c160..1b9b7f98fb7e 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -155,7 +155,7 @@ void io_rsrc_refs_refill(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - percpu_ref_get_many(&ctx->rsrc_node->refs, IO_RSRC_REF_BATCH); + refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -220,13 +220,11 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) { - percpu_ref_exit(&ref_node->refs); kfree(ref_node); } -static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref) +__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) { - struct io_rsrc_node *node = container_of(ref, struct io_rsrc_node, refs); struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; bool first_add = false; @@ -269,11 +267,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - if (percpu_ref_init(&ref_node->refs, io_rsrc_node_ref_zero, - 0, GFP_KERNEL)) { - kfree(ref_node); - return NULL; - } + refcount_set(&ref_node->refs, 1); INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -298,7 +292,8 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); - percpu_ref_kill(&rsrc_node->refs); + /* put master ref */ + io_rsrc_put_node(rsrc_node, 1); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f27f4975217d..1467b31843bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,7 +37,7 @@ struct io_rsrc_data { }; struct io_rsrc_node { - struct percpu_ref refs; + refcount_t refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; @@ -54,6 +54,7 @@ struct io_mapped_ubuf { }; void io_rsrc_put_tw(struct callback_head *cb); +void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_rsrc_refs_refill(struct io_ring_ctx *ctx); void io_wait_rsrc_data(struct io_rsrc_data *data); @@ -109,7 +110,8 @@ int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) { - percpu_ref_put_many(&node->refs, nr); + if (refcount_sub_and_test(nr, &node->refs)) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc(struct io_kiocb *req) From patchwork Tue Apr 4 12:39:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2995550vqo; Tue, 4 Apr 2023 05:43:05 -0700 (PDT) X-Google-Smtp-Source: AKy350baaB/fNzVfTgee6hZxFpxb1WvfjnhiWSOqzm4eLHzo4lbt2V+6DddI9zxV2p1BarQwXyXW X-Received: by 2002:a17:90b:17c7:b0:23f:7d05:8765 with SMTP id me7-20020a17090b17c700b0023f7d058765mr2691072pjb.10.1680612185639; Tue, 04 Apr 2023 05:43:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612185; cv=none; d=google.com; s=arc-20160816; b=PhxDDDXFAtHvNfTeL2+zvAbVFVKTziHlePj8VEXwBJ+tMTncBNMTq5EmmVy/JPEMc0 F6x/jwZLwwhFaTgmPARjku4RdBL7Yq1tRSZ5xVWjGIrXs2sfP+rQbctpUxgr0Fe+NgqD PXebpoz0aVn6LIjh2un7PqspOeNp5Z1ftwVJoyHESI25hC+d6bMTimbVM74KeVbzxsI9 fyPXTAiv7JEvGt6NsANPv4eLk0xbw5QW0BxJop+ZZa+tsSbh3gMBiaybCvPYLtgNmOba PE9yQCE/0pJalxC/xAsTh58PIAiOCVYtUh5ZT0GGNsRbnVKZRsjYrQZdU6YYuPq2TbxY FPUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Gp4fQySUslStHdzI0fEQY4d4pTvLrEwKRJkhU50QDoY=; b=yioUazYhmTJMnYNkiwCd3yhxJ8Nbj+dmGmmGvi64nR69WwYiHoAnY+h8L3TmdEYyHZ tOuzF3zY7wqbOTcmubBh1WGxsMMOvkkJ1ZiIvndwhE0WkRqKX9i0pWKKc/c4+xN60ewn FspUfHkwyu07gkq4+JoeUXkhwIOuTJ8Ddoz7CsFB/LyoOgCOxmWPzsJne5BDwOP5Ep+z wreH3ZRb8KOpNU+cHmLqZ0OuyvjEdmBPP6erJuO+JOarG+WtnHS+EVt4pUS6qmMHg9xq X4obu7iOpRMhjvdylV2/DfIN1/6LV1oTxWlVpa+j1/+q2Eqpy47PEpXXbkl6ck+wdab5 0Qqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=pso0Tc26; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020a17090a784100b0022c5a147e9fsi1566583pjl.115.2023.04.04.05.42.52; Tue, 04 Apr 2023 05:43:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=pso0Tc26; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234977AbjDDMlD (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234497AbjDDMk4 (ORCPT ); Tue, 4 Apr 2023 08:40:56 -0400 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51DFA187; Tue, 4 Apr 2023 05:40:53 -0700 (PDT) Received: by mail-ed1-x536.google.com with SMTP id r11so130054997edd.5; Tue, 04 Apr 2023 05:40:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Gp4fQySUslStHdzI0fEQY4d4pTvLrEwKRJkhU50QDoY=; b=pso0Tc26NdvEP4utoF1qPtN5aQJgCmhgkdQZILTriOeyBFvXRLe+oWtQRva7Y7FNv4 tVAJ4eptJr5L9BwP3aAhaemLpeybTKSRtQSbRzDK7Lp0T2s4XGMisjjwRW6ako2b4MVk jBzix4Rm/49oXRZuUmqaUP/BXSXgHRmZKBgE+3sRRU3pQfAJWL91moNjTuUNmk3jd22E qIAvg9KaZ2EeiVb71dKnrc4e8g+1GGD8qUsS3juFamrJMpxw8uyRZuDUJN0y2GH+Pall ZVo6SW3PAMKY+GKNQiiu/MnGYcXAEnstjz+ugzW7vYQe5/DUqBj1TU7POye6MvfjhTmw JsIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Gp4fQySUslStHdzI0fEQY4d4pTvLrEwKRJkhU50QDoY=; b=Pytw9CMJfFJ6KBKuA4SYJ63ABfLhZBSzqqn0yFrO2pyVi0RlHt3zjzWIhOJPibP61s QQ0JCvxdTnfsg6hzwJXF+1yiIqZEFU7e1Fi+x/Oz+hM/Gshn9p06Hmec27KAtKOhkrSe ht/nLezYM6oid+g5qplh+ExxUHmw2mTxU5keihAIzNsC3yttAK9kF9VipCrPa/ciZNmb 3zCQi9wR88pO0Rqe41AljnyNWNOdV3R0y2NIny65lO2w7HpQuNM8pKEqZ+e/G7nYPaE6 8IFzf+IKs+DeKKpNaR5QUMZTTH9N6NobAKG3BeqaWJDwYAZdo+P0dgC4176xoajFUndB /dXw== X-Gm-Message-State: AAQBX9e2VcBRjopZkn5JGHwhwvf1OFasGHqwFfhqs10qotyqka8cgGeT CgHN592T9HfcN1s+luzDAsZlxvo2WXY= X-Received: by 2002:a17:906:1982:b0:878:72f7:bd87 with SMTP id g2-20020a170906198200b0087872f7bd87mr2110265ejd.6.1680612051666; Tue, 04 Apr 2023 05:40:51 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:51 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 02/13] io_uring/rsrc: keep cached refs per node Date: Tue, 4 Apr 2023 13:39:46 +0100 Message-Id: <9edc3669c1d71b06c2dca78b2b2b8bb9292738b9.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249603313700054?= X-GMAIL-MSGID: =?utf-8?q?1762249603313700054?= We cache refs of the current node (i.e. ctx->rsrc_node) in ctx->rsrc_cached_refs. We'll be moving away from atomics, so move the cached refs in struct io_rsrc_node for now. It's a prep patch and shouldn't change anything in practise. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 - io_uring/rsrc.c | 15 +++++++++------ io_uring/rsrc.h | 16 +++++++++------- 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 561fa421c453..a0a5b5964d3a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -240,7 +240,6 @@ struct io_ring_ctx { * uring_lock, and updated through io_uring_register(2) */ struct io_rsrc_node *rsrc_node; - int rsrc_cached_refs; atomic_t cancel_seq; struct io_file_table file_table; unsigned nr_user_files; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1b9b7f98fb7e..e9187d49d558 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -36,9 +36,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, void io_rsrc_refs_drop(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { - if (ctx->rsrc_cached_refs) { - io_rsrc_put_node(ctx->rsrc_node, ctx->rsrc_cached_refs); - ctx->rsrc_cached_refs = 0; + struct io_rsrc_node *node = ctx->rsrc_node; + + if (node && node->cached_refs) { + io_rsrc_put_node(node, node->cached_refs); + node->cached_refs = 0; } } @@ -151,11 +153,11 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx) +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) __must_hold(&ctx->uring_lock) { - ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); + node->cached_refs += IO_RSRC_REF_BATCH; + refcount_add(IO_RSRC_REF_BATCH, &node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -300,6 +302,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; + ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 1467b31843bc..950535e2b9f4 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -43,6 +43,7 @@ struct io_rsrc_node { struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; + int cached_refs; }; struct io_mapped_ubuf { @@ -56,7 +57,7 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx); +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); void io_rsrc_refs_drop(struct io_ring_ctx *ctx); @@ -128,17 +129,18 @@ static inline void io_req_put_rsrc_locked(struct io_kiocb *req, if (node) { if (node == ctx->rsrc_node) - ctx->rsrc_cached_refs++; + node->cached_refs++; else io_rsrc_put_node(node, 1); } } -static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx) +static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, + struct io_rsrc_node *node) { - ctx->rsrc_cached_refs--; - if (unlikely(ctx->rsrc_cached_refs < 0)) - io_rsrc_refs_refill(ctx); + node->cached_refs--; + if (unlikely(node->cached_refs < 0)) + io_rsrc_refs_refill(ctx, node); } static inline void io_req_set_rsrc_node(struct io_kiocb *req, @@ -151,7 +153,7 @@ static inline void io_req_set_rsrc_node(struct io_kiocb *req, lockdep_assert_held(&ctx->uring_lock); req->rsrc_node = ctx->rsrc_node; - io_charge_rsrc_node(ctx); + io_charge_rsrc_node(ctx, ctx->rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } From patchwork Tue Apr 4 12:39:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79088 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3004705vqo; Tue, 4 Apr 2023 06:00:06 -0700 (PDT) X-Google-Smtp-Source: AKy350bVoHE7RqDJRhznK3K+jl4sPWuT6d6U5PC8NkVqviVmB05f48cs2R/17bZfoPBNJy1n/NQP X-Received: by 2002:aa7:d1cc:0:b0:502:4862:d453 with SMTP id g12-20020aa7d1cc000000b005024862d453mr1974725edp.3.1680613206070; Tue, 04 Apr 2023 06:00:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680613206; cv=none; d=google.com; s=arc-20160816; b=svtb+etI4ikaAjwj7Ld8aYM8/dbgPtf5CEpP0u52vwSNGK9XeNEIIlyKyN+pDjdRm2 6I2FmS2FGb/qlZ1+Ognjj70pi4z8ktp/e6QHkH/ADFMi35E+5uffE0lkelEWReZFQRxj aGpqL800nRGjN+MlAX/+5CDhQIR/2r3dRyHj3Lq3Y28jNo72okKx5GUj9lsSo4mCAiQU zOYqkNDT6vGvitBQIm7ILE+jeFeTg1cWsvYaZaya7rqh3KIFMeIaUunFnEBBlucFyM+2 pDlRGGB9+foa0yFzvdbuADPWOXjPMyEm6QFx+0bgfmInlpj2QMZ7mCEbz0hEr23Vjzv/ cm+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7NWdR6MPB6XAokVEPIwltF2umB3WBqKQ7V2xA8hlx+8=; b=HtKcXZGFV5TXsYcrMrjAlWTA8KoZB5MQ+TSGJFu14FKR9UequNJFM4MXXgTH3dsX0y /wJj9LA9KF9Ex21pFBOrLMxG0LNZWIMbEYiV6jB3gRMixdmTY+FHLUFhYYIHnth7MZ4K v1by+LzzK6lbRKLU+/o360/buU8iDMsypic2jJM/dZCMZNo54cw5kBlYvlgNfV492yWo IUAzfJyzxtXgtugGMfFQbePhIKKEfZ9UbBFiaQzkn8zQJ9jqdvbaJ+wFjc8hDz1sD6kg EXLSZ/Ephf9UEPADm0TAVzs144k1NlB4AYvAAKaqkB5d+XnvhdFMKbmicIEtxCOfwmLL WA5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=A2Irdkos; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gz5-20020a170907a04500b0093a0ef38401si9823680ejc.94.2023.04.04.05.59.41; Tue, 04 Apr 2023 06:00:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=A2Irdkos; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235016AbjDDMlG (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234606AbjDDMk5 (ORCPT ); Tue, 4 Apr 2023 08:40:57 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D41D90; Tue, 4 Apr 2023 05:40:54 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id ew6so129934757edb.7; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7NWdR6MPB6XAokVEPIwltF2umB3WBqKQ7V2xA8hlx+8=; b=A2IrdkosuU5qJYIAvW+WrYNBZB12BQQqpN7XLYVE2WDD3CO0lAjDS+craxmRF4yOr8 VWgYmj6W6t6FVUYsBh2H+zX3+0PSiwwbio0FdfVaYa7wCbaFqxv9Kefl0oeY0DfR4Mkh 2r9l7gIGgn2N1btGcwHKWbfYxpyJPrO+dLFu69PTsOnZwLZprRHk9atL/e3c167J61Gw NM2yzEY3Km9KctkaAA/EG5BJfMpU5L6uO9y2MSyNIMXAKBfUFYSd4iZtavI9ioeyunj2 nYgOxitaCITguhG50Xf8SD83alXzdBbJmx2waWhFBGpZTPiWJNTmDWIRBtJABJ//oy4x ypug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7NWdR6MPB6XAokVEPIwltF2umB3WBqKQ7V2xA8hlx+8=; b=SFwvFDgng3m7R6e+hPyc+JeFlMEiRCPXiXAJY+rua//iu8u7ov7PR54yOsKZw2Bb5j SBdz8MGoIaBojCuBoUWPUUUVHVSXiVPBc5XwfXvKSsOUojnwYXgKy6I2SMebtJQmgYjy jSmEb5DP0Gc62DLVJeCeebw2UlkUBdAI1+U2dgB9oEYKbEkaFGmmmnNAlxUC1GUAxcXI pQkLho85XqWW8ejoe49kWFOyuPp5nSXgyINVN6ffgYr/9X2dyQOyhc1DBY/yyxUmY0xf B+aDmXTBg394P03SFnxOaS2q8wjpViBcCCeXh5HsW683c4IRG3dzClccqVU8rjMttPWq 5VWg== X-Gm-Message-State: AAQBX9fICUpX1kz/VehSRF0ZjBQ0lnuStQFuSYEIazHlxsQ6gY3ibOii VR5EcBHxpS2UXhEZAJ+KSpDUh5dAnkY= X-Received: by 2002:a17:906:1786:b0:948:6e9c:273e with SMTP id t6-20020a170906178600b009486e9c273emr2242990eje.62.1680612052400; Tue, 04 Apr 2023 05:40:52 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:52 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 03/13] io_uring: don't put nodes under spinlocks Date: Tue, 4 Apr 2023 13:39:47 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250673444656989?= X-GMAIL-MSGID: =?utf-8?q?1762250673444656989?= io_req_put_rsrc() doesn't need any locking, so move it out of a spinlock section in __io_req_complete_post() and adjust helpers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 +++++-- io_uring/rsrc.h | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a0b64831c455..596af20cddb4 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -970,6 +970,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 static void __io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; + struct io_rsrc_node *rsrc_node = NULL; io_cq_lock(ctx); if (!(req->flags & REQ_F_CQE_SKIP)) @@ -990,7 +991,7 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_put_kbuf_comp(req); io_dismantle_req(req); - io_req_put_rsrc(req); + rsrc_node = req->rsrc_node; /* * Selected buffer deallocation in io_clean_op() assumes that * we don't hold ->completion_lock. Clean them here to avoid @@ -1001,6 +1002,8 @@ static void __io_req_complete_post(struct io_kiocb *req) ctx->locked_free_nr++; } io_cq_unlock_post(ctx); + + io_put_rsrc_node(rsrc_node); } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1117,7 +1120,7 @@ __cold void io_free_req(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; - io_req_put_rsrc(req); + io_put_rsrc_node(req->rsrc_node); io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 950535e2b9f4..8164777279ba 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -115,10 +115,10 @@ static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) io_rsrc_node_ref_zero(node); } -static inline void io_req_put_rsrc(struct io_kiocb *req) +static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (req->rsrc_node) - io_rsrc_put_node(req->rsrc_node, 1); + if (node) + io_rsrc_put_node(node, 1); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, From patchwork Tue Apr 4 12:39:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79090 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3006755vqo; Tue, 4 Apr 2023 06:02:02 -0700 (PDT) X-Google-Smtp-Source: AKy350ZMOJdb3ByNB5zcDqYAkd524BfHGoUsEiCjUr4GH4j7Au3z2z7aW1HTjkkQq45IXC90UMi/ X-Received: by 2002:aa7:d955:0:b0:502:a700:dc8b with SMTP id l21-20020aa7d955000000b00502a700dc8bmr2168128eds.9.1680613322116; Tue, 04 Apr 2023 06:02:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680613322; cv=none; d=google.com; s=arc-20160816; b=yOwHauKaksGCFbazeD3MFJX0LMa5utchZHADNIQGrXTecx+0XDlvLf2Mn5k/ufYISK A+iX3XxYiVaavU3omBl7JBNhwnJBeiZJuu4JAL5SBsV32VIzfWWjivqHeJ5z0WfKCVs4 VhxW1bmFfg/5chKE85K93f5StQWnViAWKOsLpRoI5Q8OvrDgB+hAploVpfvVOGBE+G97 phbGNaDbYfmxYog9jYhmR7lLa4IHFFt36xqfcABgaspE43gTuTiSOtX+mOC7Noh/v5+n /n3idyKl8EQBa1nkqlYr+az3STf6vo0zAOtqjMZZ+IApznWxI+t6yKuclTNk1altVy8K 9Xqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=F32O348mTkggRcCcK2wkzqvHv1nRgFctuPngecPv3KY=; b=A46feWxTVcP71R80IJGPA6qmYJdh5MiyGf0GH9CLE9jd7aVSaihugX0hGEppxTJ9xN hW4GFrvAoEfaJ1rauXbJ2Mzaj3PQF4Yb8iIkZsTLh6UZswjDJ0geZ/EOcBvs8uU+xZ8V nOCYzSGC7VQI+3bazNGOifcWGqemEYffGuwYrIaSrYLgMDNf5o7gqIEswzr4Zz4P0bzH mF2VQDCkETqNcCOC7hkrLvpWD5EPlPBToKsYp3JPSgYUqlUkNSUDTTiLNr6iN5b3hN6i r2yYSsK87hNkHLDrjG4ulOlMDBraIWR2ve+zmUUQZo/HivoKLaDyPcaQLRjV4NPhCERy WdEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=dTK7+Kye; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b3-20020aa7c6c3000000b004faf6a08e25si926215eds.246.2023.04.04.06.01.33; Tue, 04 Apr 2023 06:02:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=dTK7+Kye; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234709AbjDDMlK (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234000AbjDDMk6 (ORCPT ); Tue, 4 Apr 2023 08:40:58 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76E1AD8; Tue, 4 Apr 2023 05:40:54 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id cn12so130059676edb.4; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F32O348mTkggRcCcK2wkzqvHv1nRgFctuPngecPv3KY=; b=dTK7+Kye78JXOJfhcNt8QivolTT8rVP53t8GXXvuGVNhpH8N24pc2EoeSFwmyWZEWs AGbEG0jOIjwCTd04Zps0w3m1P+vKhsMcbIpQaYx/Zrk2ki5Hj/jKinjGlb1UgxyYawvr p3AWcONqwd/fk4GXL7mvTxJe6ReZNXMoLzDO3C83my3sO4oe2HmM7929WWWspr6KYt1e bD4mCFWZHL3XeeRns0BB4yADWqE0M/2s/FfGVoVwH1LoiyW3zyETnRwzGfaBvkXcViLr 1Ek01KizZI53MjFIa1c0kS/6VfHXob/40V0x06HjYltnfmo1ZN9nfcGBtNkB8oT8HY6x 58WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F32O348mTkggRcCcK2wkzqvHv1nRgFctuPngecPv3KY=; b=xclh7kU/hzPIeiounBe9iiuXTAFCrDhOE10PH1YsKrriqfq0uSqv/nX6eO2fQaEokW Mi+MyRaxE+VHFyMsYVddUPHO38VwZZeqTOeEd0EkowxbBIOgDQialkKYbAMMQYFlw6wU k0zJISF92lpAnl5SGTn9jaBAdQYEoV0Ley5k7dHQPcMju5t7O3prWA7EbD51qbXpL3p7 PxPsuJ2tR06WZBaaMZWKrcDteY+b+GagwhZZYLqU95972zIssm8+dwu6mlX5uJQQbGtG hN27Q6UzXx487bAzJIe/uMNkkuA8+32QKA5j9iG0gQNRjl2Xh1DW31ZO7swFjOXivaSQ rsPQ== X-Gm-Message-State: AAQBX9fS03bpcRVXYBuuMpO2Y+vkw1vuQCIrrzqsFhgRHIiBLphsT3po rlLCU0KPpV184ZVAUxA70HxM0T44sv8= X-Received: by 2002:a17:906:ee1:b0:862:c1d5:ea1b with SMTP id x1-20020a1709060ee100b00862c1d5ea1bmr2064833eji.8.1680612052843; Tue, 04 Apr 2023 05:40:52 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:52 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 04/13] io_uring: io_free_req() via tw Date: Tue, 4 Apr 2023 13:39:48 +0100 Message-Id: <3a92fe80bb068757e51aaa0b105cfbe8f5dfee9e.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250794845899142?= X-GMAIL-MSGID: =?utf-8?q?1762250794845899142?= io_free_req() is not often used but nevertheless problematic as there is no way to know the current context, it may be used from the submission path or even by an irq handler. Push it to a fresh context using task_work. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 596af20cddb4..98320f4b0bca 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1116,7 +1116,7 @@ static inline void io_dismantle_req(struct io_kiocb *req) io_put_file(req->file); } -__cold void io_free_req(struct io_kiocb *req) +static __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; @@ -1130,6 +1130,12 @@ __cold void io_free_req(struct io_kiocb *req) spin_unlock(&ctx->completion_lock); } +__cold void io_free_req(struct io_kiocb *req) +{ + req->io_task_work.func = io_free_req_tw; + io_req_task_work_add(req); +} + static void __io_req_find_next_prep(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; From patchwork Tue Apr 4 12:39:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79075 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3000000vqo; Tue, 4 Apr 2023 05:51:14 -0700 (PDT) X-Google-Smtp-Source: AKy350ZfmkjFAM4op7g5SXqAGofEKfb0wMlkGpjU6tWjCjIrgmMb+e5cJST5+2axhPGXpbcnTKr4 X-Received: by 2002:a17:906:7c53:b0:92a:11be:1a40 with SMTP id g19-20020a1709067c5300b0092a11be1a40mr20540900ejp.11.1680612673812; Tue, 04 Apr 2023 05:51:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612673; cv=none; d=google.com; s=arc-20160816; b=SZWD3ZI0q/tdxxImq2Ipmh2+m/bYOOuvtfsVzlPBE/XOzRhTn7Vzx/JWnmawD4E+tI 2C8TKM4YvlayTb7omwoqlCqgDUh+plZilnXTmE9JOqPMUmRab10im86S80Knt6yvn6DR 6FnEEci8Pu8hZgv8SbMBWp72bT+eT2jStUXgRcm9cXNaLthyqAl8Ha3R+urD5stNhlTZ 86d+xWP+HUJjw7BPmMcEq2qHCkclP/6M+iHxdwir+s7oQRR9a00b8jvNRQrP9jtwfFpJ vUChxGdxl3jEHAY4N/Tplz9jqGgSG3S8/Rxxo3SXVL6OZQlzeYtBCFnHlyPTz/0/c9ns y7AA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xiac71Z/ZNmEtbhG51KDumqDuoRCwK/TNCs/88XmD5g=; b=KUOHy09UZae5+ppkm45Jx7qlF9J7W6rzdopjs/bN5cIzul37JDS497qqsf8cCoDP5/ viAMaPIkAyqzFkbQQwsH+O4Vwfip9YeMLhm+3QwQ6nBOD5a1j6SAYuylo5M2w1gPW4zw RPinB4iVnSinwNSPMOJ5kWpN+3aP+2qbrFg8BbprZ1U3wt3SGCf32yKo4tL6tPhfeddM DZzZhhgfRE/fa+xOGqKBl7spYZ6JUwbCanoyP2YuVBvbrnHIoe9koA0mVhuiY2+rB1S3 tEk1OJrpDCrkCk1BW3MNLe80XHYU857wnj/OQrYmCQ3N9YQKTKmd3pw/6U9oJcUkcF74 8zcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=U8OeirzI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bx14-20020a170906a1ce00b0093defbd6293si1850742ejb.1050.2023.04.04.05.50.49; Tue, 04 Apr 2023 05:51:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=U8OeirzI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234971AbjDDMlZ (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234701AbjDDMk7 (ORCPT ); Tue, 4 Apr 2023 08:40:59 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB0B910D5; Tue, 4 Apr 2023 05:40:54 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id ew6so129934902edb.7; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xiac71Z/ZNmEtbhG51KDumqDuoRCwK/TNCs/88XmD5g=; b=U8OeirzID3x78bkm6uM+W+9QCYE12gVuYh81wBNSQwgCJJLGxwcL7ywn9tFsztqHoT GIzO/RQ6ILUyY2mcydQI1pEE6Iwpmy2J1996VbH2YIzI+9QdtEitqVjiM06o0RCOm6gv 7k+qms1j7uV/HWmKcWn20z+9SdnImxuiQ4q6M+xg4yEjPJ8yCxYc1B9v4uLvs4h+Duwm C43c+4kjrRCeQJtd6BzoS5DpHNA4xbAxlWeCENEMs7Md3bcRRmzEgkBUpKbxD75YHaEh r1HArHEoleAI5HANib7wu/Me+6RV74k+cj0HgCxmqRiCrcKaGQegNCRQ+WSbfRjD3etP C7Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xiac71Z/ZNmEtbhG51KDumqDuoRCwK/TNCs/88XmD5g=; b=N6O38GAHXhia43wJt9bRRkmu+CPPw+U+PYn0KlN++6WZwLvYoD3kp/8D0CMawHYOZ3 VpQeSmgCxB6/UMAtsRbmhSE/i0M+hED84pSoS5mXnO+RyJUObeZMNUuEeoP1YxKdpL78 ckBROjlWN5I1wtdDMYx86bCXLQgWK+hus/n3mjx/CJzEbWQSIWtgVAhUFhdrfh+ngo3x fAWaP7lJIN+6pYPa26nGhwqxPf91GFG/QxIgXr5jFEORmj+7OIgqjBvlko36XT+STNlw nh6x+OGV50fnVORDgh7HJre2ZttDPgUMNErUtwfQDThTRapO18fAUWnrWZtM7iIEn4sa XmHQ== X-Gm-Message-State: AAQBX9cggxnhX5inkqjCzVPU+TJxHehZ8tnwQdovVnTZBURifSfkBXBm e5NOdx/JFiNuHKMwXO63LBE7dawWsqw= X-Received: by 2002:a17:906:bc4c:b0:947:d3f0:8328 with SMTP id s12-20020a170906bc4c00b00947d3f08328mr11696047ejv.1.1680612053390; Tue, 04 Apr 2023 05:40:53 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:53 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 05/13] io_uring/rsrc: protect node refs with uring_lock Date: Tue, 4 Apr 2023 13:39:49 +0100 Message-Id: <25b142feed7d831008257d90c8b17c0115d4fc15.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250115294943469?= X-GMAIL-MSGID: =?utf-8?q?1762250115294943469?= Currently, for nodes we have an atomic counter and some cached (non-atomic) refs protected by uring_lock. Let's put all ref manipulations under uring_lock and get rid of the atomic part. It's free as in all cases we care about we already hold the lock. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 18 ++++++++++++------ io_uring/rsrc.c | 30 ++++-------------------------- io_uring/rsrc.h | 29 +++++------------------------ 3 files changed, 21 insertions(+), 56 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 98320f4b0bca..36a76c7b34f0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -967,7 +967,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 return true; } -static void __io_req_complete_post(struct io_kiocb *req) +static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *rsrc_node = NULL; @@ -1003,7 +1003,11 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_cq_unlock_post(ctx); - io_put_rsrc_node(rsrc_node); + if (rsrc_node) { + io_ring_submit_lock(ctx, issue_flags); + io_put_rsrc_node(rsrc_node); + io_ring_submit_unlock(ctx, issue_flags); + } } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1013,12 +1017,12 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || !(req->ctx->flags & IORING_SETUP_IOPOLL)) { - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags); } else { struct io_ring_ctx *ctx = req->ctx; mutex_lock(&ctx->uring_lock); - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); mutex_unlock(&ctx->uring_lock); } } @@ -1120,7 +1124,10 @@ static __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; - io_put_rsrc_node(req->rsrc_node); + if (req->rsrc_node) { + io_tw_lock(ctx, ts); + io_put_rsrc_node(req->rsrc_node); + } io_dismantle_req(req); io_put_task_remote(req->task, 1); @@ -2790,7 +2797,6 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); - io_rsrc_refs_drop(ctx); /* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ io_wait_rsrc_data(ctx->buf_data); io_wait_rsrc_data(ctx->file_data); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index e9187d49d558..89e43e59b490 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,23 +27,10 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage); -#define IO_RSRC_REF_BATCH 100 - /* only define max */ #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) -void io_rsrc_refs_drop(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_rsrc_node *node = ctx->rsrc_node; - - if (node && node->cached_refs) { - io_rsrc_put_node(node, node->cached_refs); - node->cached_refs = 0; - } -} - int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -153,13 +140,6 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) - __must_hold(&ctx->uring_lock) -{ - node->cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &node->refs); -} - static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; @@ -225,7 +205,8 @@ void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) kfree(ref_node); } -__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) +void io_rsrc_node_ref_zero(struct io_rsrc_node *node) + __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; @@ -269,7 +250,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - refcount_set(&ref_node->refs, 1); + ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -283,8 +264,6 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, WARN_ON_ONCE(!ctx->rsrc_backup_node); WARN_ON_ONCE(data_to_kill && !ctx->rsrc_node); - io_rsrc_refs_drop(ctx); - if (data_to_kill) { struct io_rsrc_node *rsrc_node = ctx->rsrc_node; @@ -295,14 +274,13 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_rsrc_put_node(rsrc_node, 1); + io_put_rsrc_node(rsrc_node); ctx->rsrc_node = NULL; } if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; - ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8164777279ba..a96103095f0f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,13 +37,12 @@ struct io_rsrc_data { }; struct io_rsrc_node { - refcount_t refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; + int refs; bool done; - int cached_refs; }; struct io_mapped_ubuf { @@ -57,10 +56,8 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); -void io_rsrc_refs_drop(struct io_ring_ctx *ctx); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); @@ -109,38 +106,22 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) -{ - if (refcount_sub_and_test(nr, &node->refs)) - io_rsrc_node_ref_zero(node); -} - static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (node) - io_rsrc_put_node(node, 1); + if (node && !--node->refs) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) { - struct io_rsrc_node *node = req->rsrc_node; - - if (node) { - if (node == ctx->rsrc_node) - node->cached_refs++; - else - io_rsrc_put_node(node, 1); - } + io_put_rsrc_node(req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - node->cached_refs--; - if (unlikely(node->cached_refs < 0)) - io_rsrc_refs_refill(ctx, node); + node->refs++; } static inline void io_req_set_rsrc_node(struct io_kiocb *req, From patchwork Tue Apr 4 12:39:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2996070vqo; Tue, 4 Apr 2023 05:44:10 -0700 (PDT) X-Google-Smtp-Source: AKy350aucew4uEmK6etmsoqsuT+w4GHu1Tnei+PjtU2uMYfcvuRJAZR6gPE8EI8uXwEK1HzDRA48 X-Received: by 2002:a17:90b:3b4f:b0:237:161e:33bc with SMTP id ot15-20020a17090b3b4f00b00237161e33bcmr2757878pjb.16.1680612250630; Tue, 04 Apr 2023 05:44:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612250; cv=none; d=google.com; s=arc-20160816; b=WnjFA8Vwx6uXqBpmimVYFKlMcQ3WjcwA6mLTIVbADXQ/1lVWUmh9MMJ6EEA7/BLHo/ iZCplSRt+Svcl86zTQdmuLn6KsQC8DGG7L1LYqc0/GedAiAf1xqtJBd1sYS6AkhLhS1A u1kSXp3JP4vXXPhdvK/t2WxIiF4GuKuWwCHIyhNOFqb/9+IYKv21TDSfr6GOm/GsLKvt brTgRjqiNuBslHX4Hawlbg5OMKDqU6vMUioQ38eDoiijmL36WiOlOaP6fVAW0rfriziJ hKVglfYdjhu/K3tKVv1lFmqSYjAMlk/kdcaLp66wgxmWRrdYuUFyhROu/tZhNEmYHQ+6 n70w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DjQOCF2m5FEa29JFzTi+p7tq3jGoDrTcPjGyMCsupCY=; b=dLWBRddF4g9q0Oe1KQuBmvTK01XRHJ6ekKXH8BIjutFL4fsfGq+9nNxnI2EwEejE0/ EBXk6ZYp1ofceWUfhEkuBHXpfTbKKDIcrrcyV1LMGsPLdOADcG9gqTXs5Jm0Qe1qUhfl IaUcqTwthhHC+VTXYR3uFBDfRZiw0YfJpnt34NJHJ9yJy/+XQIki3cdo6dj6DallaPqa LSMokmIQPmkq2atEsntFSZCXBKgTvBDTre890berV6kUBcAuMXiOFF5FX3svgHPV17Ye DBhHoXrmFT1qy4zsdcwDpbYLFw60NYV9QK7WGGol4M50aEcGzdau+SbgLOTngr7l2/eK gnWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=EBSA+3FS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gc24-20020a17090b311800b0023f064f20c7si9867168pjb.168.2023.04.04.05.43.58; Tue, 04 Apr 2023 05:44:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=EBSA+3FS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234780AbjDDMlN (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234615AbjDDMk6 (ORCPT ); Tue, 4 Apr 2023 08:40:58 -0400 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B4C1195; Tue, 4 Apr 2023 05:40:54 -0700 (PDT) Received: by mail-ed1-x532.google.com with SMTP id er13so89026783edb.9; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DjQOCF2m5FEa29JFzTi+p7tq3jGoDrTcPjGyMCsupCY=; b=EBSA+3FSAf2l0JlcdciCq2q1iDCL5Fj9iJsuAytH19HB+lDwtMeCWQJbjez84ctk8e cL8RFWEaRknf2qPJnJ3mS866NUVCdPzTF79WEzjQq9Bg2psWDV6P3vlXgftoMpRhrYJG Pym+m7uEycWCVDjvJ71mHste7SR/zMPE2EZifzMYJMrq14ViFjn5ukgQgccGL68nAdgO tt5Z1TEjDuXxPnTvrjwDODfP3mGtv8vgGgn5FUlhy/Sh+HEcT7RO56VoTQRFxtpVSTfd 8gfcY8mlesRSTpb7EO501tUVnelH73/lRsss0nP3T14gUxmAQ3fJhOOZIM5qD4cAY0Ad tksA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DjQOCF2m5FEa29JFzTi+p7tq3jGoDrTcPjGyMCsupCY=; b=SZ5vQX7x0Vh9BMFKgmCuVJIqiPK97VhYl6zfxtkdVzG2iTzq5PFEQaiE/nqHp3zLKJ 6WRHKJK+GVO0LbowJ/T3XM73rcsQ50XNvl2ck7lgOECL/3k+TxWJnV8NQradUN6vW5Nu GA2XXijjS7O0WLchoXeUgq0C6arHuLIk0VM3rFCmA+ZhY5ZA+IicnD+bQFRvHd8YJ5YI LesJ/FS1k9YVIODhzhmO2cRRG6jJ4tPJyEKobI7lDbpi0V1VyiyhNs5pMRY8PrW5ASQY kz0OLGOiepueJqPez3vJkiSjiSWN8jesR/vxVLYaJTMopzmZxicL/0vlma9zn0eKtxxE 2obg== X-Gm-Message-State: AAQBX9elgQvDrkwyP4Tm8aH0FqB2aCHCCCnKvyWDqft94ZOrqmqcEfxX JMPniDmWmS735DKaC5qpkAP82Wle6Z8= X-Received: by 2002:a17:906:95ca:b0:931:4f2c:4e83 with SMTP id n10-20020a17090695ca00b009314f2c4e83mr2479643ejy.63.1680612053938; Tue, 04 Apr 2023 05:40:53 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:53 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 06/13] io_uring/rsrc: kill rsrc_ref_lock Date: Tue, 4 Apr 2023 13:39:50 +0100 Message-Id: <6b60af883c263551190b526a55ff2c9d5ae07141.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249670836416909?= X-GMAIL-MSGID: =?utf-8?q?1762249670836416909?= We use ->rsrc_ref_lock spinlock to protect ->rsrc_ref_list in io_rsrc_node_ref_zero(). Now we removed pcpu refcounting, which means io_rsrc_node_ref_zero() is not executed from the irq context as an RCU callback anymore, and we also put it under ->uring_lock. io_rsrc_node_switch(), which queues up nodes into the list, is also protected by ->uring_lock, so we can safely get rid of ->rsrc_ref_lock. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 1 - io_uring/rsrc.c | 5 ----- 3 files changed, 1 insertion(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index a0a5b5964d3a..9492889f00c0 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -333,8 +333,8 @@ struct io_ring_ctx { struct delayed_work rsrc_put_work; struct callback_head rsrc_put_tw; struct llist_head rsrc_put_llist; + /* protected by ->uring_lock */ struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 36a76c7b34f0..764df5694d73 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -325,7 +325,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->defer_list); INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); - spin_lock_init(&ctx->rsrc_ref_lock); INIT_LIST_HEAD(&ctx->rsrc_ref_list); INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 89e43e59b490..f3493b9d2bbb 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -209,11 +209,9 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - unsigned long flags; bool first_add = false; unsigned long delay = HZ; - spin_lock_irqsave(&ctx->rsrc_ref_lock, flags); node->done = true; /* if we are mid-quiesce then do not delay */ @@ -229,7 +227,6 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) list_del(&node->node); first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); } - spin_unlock_irqrestore(&ctx->rsrc_ref_lock, flags); if (!first_add) return; @@ -268,9 +265,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_node *rsrc_node = ctx->rsrc_node; rsrc_node->rsrc_data = data_to_kill; - spin_lock_irq(&ctx->rsrc_ref_lock); list_add_tail(&rsrc_node->node, &ctx->rsrc_ref_list); - spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); /* put master ref */ From patchwork Tue Apr 4 12:39:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79081 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3001607vqo; Tue, 4 Apr 2023 05:54:05 -0700 (PDT) X-Google-Smtp-Source: AKy350ax8QfxIUApLI5r4UXhm1i7hDH57eq4gUcs/QOJp13Vx/VW31ffVJYCq7uesBJGUlQZddW4 X-Received: by 2002:a17:906:cd18:b0:8b2:7567:9c30 with SMTP id oz24-20020a170906cd1800b008b275679c30mr2440980ejb.59.1680612845102; Tue, 04 Apr 2023 05:54:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612845; cv=none; d=google.com; s=arc-20160816; b=WQNjqk8D4q6CDr9hCpoRr8YcpjOx8rUlQvugfaGLecWYo7/JO3hIw7qqR1Llm8O+Ml TLpYpx8c0H2lZzOYOzCFYaWPge20iF54RkvLPuuKfNRXrYma2ZX1IFdc95VJlq3qjaia rEYMAxzh2xjtpVdeK5JQ+93au8pQUqdR1h7KmagSXn7ph/3zgD78CC4jNA/e7RTvoS+K 6ARq8bMf+FrcOEx35gSxofKv1eyq1qEQgEu+3UOCKhFtNkizBsgHmavJgaCfNgU4YPkp ExTpWZqLWHLg+oln8YCr5M2GKRW9vLYdqsWlbILCc3IUZn/bi9Qlj+sdHpqbbAH8eg/V cBEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=48iFWUfpdleMqEh/dEcvKEXwQRaih9b/03a5Bfe+CrM=; b=Yy2uJF0cS7ClRfXXkAgVY+Q4r5odtg6fLimFQvaafRoxwNOqdVRGwd9JVY4+DVHSW6 ctm5goJNjiVBAgtH5yXQqKzX2MgIjbEXtJHctI387rBLNXxTY8Dvn9vkG6aUIeNCnvxA On2e51AVhSkHnsVMGcC0c24p1ixFNZ2xzyHTSuY4Cx3jwJfCNj/vyiUD11cLH8kTuqfm Zf51eedzWFe/Y5ug2sJVSexaPu22tLAUV8w+VhkugM6/UrY2WkLdqoIykcuPkvM2fBVd FBxfpFtDU/Z4zlXrRfY0y0+9RY9p2j3Q8UrOjBfBeHlBLtYxvnNC8ytq6XZ/H9p2yoaB wfAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=gl2kV2u3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u7-20020a17090626c700b009316b784857si2804245ejc.352.2023.04.04.05.53.38; Tue, 04 Apr 2023 05:54:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=gl2kV2u3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233561AbjDDMlS (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234644AbjDDMk7 (ORCPT ); Tue, 4 Apr 2023 08:40:59 -0400 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1F6135A6; Tue, 4 Apr 2023 05:40:56 -0700 (PDT) Received: by mail-ed1-x52b.google.com with SMTP id w9so130034910edc.3; Tue, 04 Apr 2023 05:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=48iFWUfpdleMqEh/dEcvKEXwQRaih9b/03a5Bfe+CrM=; b=gl2kV2u30DiuF6LupyGJhc9fmdXteYGvxc6WMMxQ83owbOpufrLobHdjp7vJ76EHJe DFLCkr9ikzj6jB8Lx9bpBWtWijJTmPgRcduPEEBluNfaPgqUCYx3eXvNRIXw4Im4rrxp ZZldOEBkDIZjteD7LSlQiyopcCqnmHABeZ7kDpsqWuIw6MF5GaLBYTz5NGyCRXGUoxp7 ipMyvoluJLjtxjqftPEjuEVwPK351nasYVHb7iHl5CwuIT3SaQ65gC1IJEF5dF88mWgY FMCIGHgl4laJj+IdD/toCmbvnOTIPaACCFhKUeIpTcSdN8dso/qs6EGr97scyqWTi2bO GKdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=48iFWUfpdleMqEh/dEcvKEXwQRaih9b/03a5Bfe+CrM=; b=ukeex1gOJL/QN1OzwNURJZhog2vjwkjp4B35fRbiz4rThDu5QhNPBIim7FK4VXX4Z7 pZ8sqHqmUo5QVGe9jMeAA+/59KJHKm6V+hFuGB1AP4OvrDTqrwRkK7yvOFR7zwMAEo8V j/yGO9AxYQUgG4C8dLlfqYHdm0xjbopdsdna/ctF2DHQUrLy1nDRUnC9IX8RVaLf/aLX T8mcAwqWzRhSqti24draJDvKOM3pMvy2zXzZtNYwcnKRR/tIKtHTMvcTUlnrgbhthcqh g+lB9fECprHe40rbPSr52PMAy2SIoxIZcUFcnpoZS5lkvTCG+4FAdMjWr+2ZknDOjt+B NJow== X-Gm-Message-State: AAQBX9eEjmfrh/MUNWmxJlHoEkFMO5nDcfM0C+jeuiFm4EIb/HxVWiGc kF1EeMTFleCxEjlRw3lbH+FGniKL1aE= X-Received: by 2002:a17:907:bb87:b0:8aa:a9fe:a3fc with SMTP id xo7-20020a170907bb8700b008aaa9fea3fcmr2098510ejc.8.1680612054507; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 07/13] io_uring/rsrc: rename rsrc_list Date: Tue, 4 Apr 2023 13:39:51 +0100 Message-Id: <3e34d4dfc1fdbb6b520f904ee6187c2ccf680efe.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250295021674975?= X-GMAIL-MSGID: =?utf-8?q?1762250295021674975?= We have too many "rsrc" around which makes the name of struct io_rsrc_node::rsrc_list confusing. The field is responsible for keeping a list of files or buffers, so call it item_list and add comments around. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 6 +++--- io_uring/rsrc.h | 8 +++++++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f3493b9d2bbb..2378beecdc0a 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -146,7 +146,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; - list_for_each_entry_safe(prsrc, tmp, &ref_node->rsrc_list, list) { + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); if (prsrc->tag) { @@ -249,7 +249,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); - INIT_LIST_HEAD(&ref_node->rsrc_list); + INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; return ref_node; } @@ -737,7 +737,7 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->rsrc_list); + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index a96103095f0f..509a5ea7eabf 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -38,11 +38,17 @@ struct io_rsrc_data { struct io_rsrc_node { struct list_head node; - struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; int refs; bool done; + + /* + * Keeps a list of struct io_rsrc_put to be completed. Each entry + * represents one rsrc (e.g. file or buffer), but all of them should've + * came from the same table and so are of the same type. + */ + struct list_head item_list; }; struct io_mapped_ubuf { From patchwork Tue Apr 4 12:39:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2995909vqo; Tue, 4 Apr 2023 05:43:46 -0700 (PDT) X-Google-Smtp-Source: AKy350bZWyw2Dv98kRkuM32eVAY5R42dU0ZMYkm+UAvv5902gJtwoK/ruUEuAB7QGewS4UqxJZ5b X-Received: by 2002:a17:90a:1a05:b0:23d:5485:b80e with SMTP id 5-20020a17090a1a0500b0023d5485b80emr2865962pjk.6.1680612225967; Tue, 04 Apr 2023 05:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612225; cv=none; d=google.com; s=arc-20160816; b=B19WsZTsPOhauMbfLX3rnVd1TQ9QVeozc8WcP0B6xxXfLXRYOrC8FoaKvfmhf+VfTK Uoh0eKHm1uVnA4MA+n5GmW9FOKF8hnBmuRkaoqL4khRPp8eDtujpxQO9avneNCf4mT8L JGQ3lm5/hgV8cwajuI5D/k18c8Jx9DuqvxF4BJTlSl7EHNEshe7xWPQzdw8QHGrFQAmA hTJpXvARLokwjz69XhoJ/gqyup2gm96dcNWB8m3Dqod1VCpjLWXfhp7wpwUu4gNM2lwf 7FTofQ4JDflUO/svV5kKo0hNnv9IgfmQf9vCeODxt0HSD9k546H8K7SN9ZyNhFhIpRGv Rmaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=MBZlopQlaa3q6pwmx/rtZaXmrUyLc41MivYhwz5LpLAwDK4FKvM0xg4wdTOX+cGfKM W4BJyBn3dpRgb3/rH2LrZr16FSHNhAffQeoHYAfmru8vaNOMfYVLhchD0MdgzJCrY/3g UWthYHY6925je24meUrSDveM4Z3F0Hd9Ta6GPFsDquW+KpfCzyErw7jflJCt0R3JDIQ4 +jrM7ckkACPtUVJdvjEDZwW28w033dsuG6BfGHB6kin/ZcH4v4iXwZHvkKMtbaLW6+2Y In+F81wcQdJ4e/jnaVvp9sljZw4jTz8XChLtESnsssL2+ikvR9LNRCxMgPbD+sPuhGSk Pxxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="CD/HtzQQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020a17090a784100b0022c5a147e9fsi1566583pjl.115.2023.04.04.05.43.32; Tue, 04 Apr 2023 05:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="CD/HtzQQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234623AbjDDMl2 (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234770AbjDDMk7 (ORCPT ); Tue, 4 Apr 2023 08:40:59 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A40D135AC; Tue, 4 Apr 2023 05:40:56 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id h8so130056090ede.8; Tue, 04 Apr 2023 05:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=CD/HtzQQLAjag/OQbDs6IryLQwHEpG9SXXXTqYvROafActLs1tDcaGlOMLU9KNHfjA lW2BTZI6EzPRmCopK0CB0uUDtOV4KbTRSyBMATh8KMVNYYmjPq2+Zspnl+40GCUDh1Lt 0xtuaTH6wWSEGVy3Hq9SGkn6DO6L61SqTOUkMcBwwt1o152GRY6PyHU+bygxcOU51X9K vkvSvbz8ocmBdquouI+tU5gNKowH8aBZ29PdRaKarjt1LR2rj1tLUC+77umQ8WE9oU2h Fn1D3gh4lJUB2ALhTwVL8kEBQRHMINtIIlnOMSycxTkJ3ylHh586DaEjFM64WnqIYnA3 1pKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rgmgIweSfjINzgFXMjWSh8rgR6O7dvb7i4eqwTPbb1A=; b=pFket0w3gF4CNf3iQ2K/Sntwob8nDNR5+D6ZdXaIaTQ245jhttpk/hS82r2FzwLMEc xx0c6EsLjzM5xtZ+pQHQ+krSSEYfjJvVsUjjacHlFJtb8Umf3UqSfO92n32O3jW+RNBd bLPExElJngz9Mq2Nrb1MajrT69OzGqeR3jvy69enTjZR+BJpvs7K9AAZhr6gm5dBc/YW ISotbmhpihbqQyklSOc09t2UEd3bJUKOLVvRkhbGfOgT6jZ70V3FyllkXZojH7X8ymYU G+s4vd7A+zKDKf9ekoXixfn01vAg9sYU/I8WXq+/X3TtX/yiRtKwyvI5KLt3zB4sRbsO bJbQ== X-Gm-Message-State: AAQBX9eQFIsnkWIdbU5y3yX4UDeNMM/0gCS0eNlwH9twu+oi3BMnM7YB xRuqB3Y9FKFrqCEqV7MXJpiVBFtKOfw= X-Received: by 2002:a17:907:988e:b0:947:bd54:cb8e with SMTP id ja14-20020a170907988e00b00947bd54cb8emr2478777ejc.10.1680612054982; Tue, 04 Apr 2023 05:40:54 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 08/13] io_uring/rsrc: optimise io_rsrc_put allocation Date: Tue, 4 Apr 2023 13:39:52 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249645026453377?= X-GMAIL-MSGID: =?utf-8?q?1762249645026453377?= Every io_rsrc_node keeps a list of items to put, and all entries are kmalloc()'ed. However, it's quite often to queue up only one entry per node, so let's add an inline entry there to avoid extra allocations. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 51 ++++++++++++++++++++++++++++++++----------------- io_uring/rsrc.h | 2 ++ 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 2378beecdc0a..9647c02be0dc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,26 +140,34 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } +static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, + struct io_rsrc_put *prsrc) +{ + struct io_ring_ctx *ctx = rsrc_data->ctx; + + if (prsrc->tag) { + if (ctx->flags & IORING_SETUP_IOPOLL) { + mutex_lock(&ctx->uring_lock); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + mutex_unlock(&ctx->uring_lock); + } else { + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + } + } + rsrc_data->do_put(ctx, prsrc); +} + static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; + if (ref_node->inline_items) + io_rsrc_put_work_one(rsrc_data, &ref_node->item); + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); - - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } - - rsrc_data->do_put(ctx, prsrc); + io_rsrc_put_work_one(rsrc_data, prsrc); kfree(prsrc); } @@ -251,6 +259,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; + ref_node->inline_items = 0; return ref_node; } @@ -729,15 +738,23 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, { u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; + bool inline_item = true; - prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); - if (!prsrc) - return -ENOMEM; + if (!node->inline_items) { + prsrc = &node->item; + node->inline_items++; + } else { + prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); + if (!prsrc) + return -ENOMEM; + inline_item = false; + } prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->item_list); + if (!inline_item) + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 509a5ea7eabf..11703082d125 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -49,6 +49,8 @@ struct io_rsrc_node { * came from the same table and so are of the same type. */ struct list_head item_list; + struct io_rsrc_put item; + int inline_items; }; struct io_mapped_ubuf { From patchwork Tue Apr 4 12:39:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79087 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3004136vqo; Tue, 4 Apr 2023 05:58:54 -0700 (PDT) X-Google-Smtp-Source: AKy350YiPPjR3jlGn8A3t5/jWjkHaP8cpnrLnkEqCZ5H9eWBaxujtQ6ntsGckj2LeUssdMnZjQVL X-Received: by 2002:a17:906:c046:b0:87b:e105:5d9a with SMTP id bm6-20020a170906c04600b0087be1055d9amr1882690ejb.67.1680613133949; Tue, 04 Apr 2023 05:58:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680613133; cv=none; d=google.com; s=arc-20160816; b=oBbXmEFIB6KN1Xwh0ZEV4lZqYbvVkCTjaXqpfVrZemLz9h9raWoLO7PEyU7ILpaq+P Y88B7YGgsYmkgUKZQflcLZmfRWWCLCtcf+jPucuNV7VUq/m9VOEkin7sAMVgzfVxZmdk cOH+gQlRa7jVsYNYOrE5blzK2Q3CUbMSwma0X1zGqZJQfRh4bDMfH337K9wN11AABKIR NRv1rQ52us0ctvi/Fg5x0weRsy9KCfYQh2GQ8vhF2DZOj1JUj/lewrars2D8Y7RtxHpv 2tthSwlV5XjQfrA0hxHrIq4+6rx7Gi7fncbWiMsC/YoF4GVaNk+/L36RlljA9IQIIqTy BYow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b0oibOM/o2KUQ8VFakULVS2Dn6jFvS8HYN/Ntyf1ZUA=; b=FBcFgbhFJHpN3xnVCIsXA7ltG+TsdFSw0XZWPekLSIliHdSpuilhByGwSq5iQaFdHz N9boPQoqLgcuXL6HAajrlaMnCF/cbrwA7MEx4JMdw6IC7FzBm04u4svBpIpaGSdH4HS2 Wyxmxa1CHSbyDK3kJPpj5LCfIsZ/NfA9TPvvpKKLHeH1uzbGXk3yxc3Z6j982kcZH9ee QTIbcCekwwEId6OELMrG+SsE3gp0RJGR2zclspM97ePJFiRIBPxC1xbSaLeCeYMVRHLU nEgjtagq5iZP/4Upw/AwudbsYH7FydAGNkrLOwswgv9u/+eMtsdv6DBBVU2iDumn2Pg1 3hOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=GwmjIJ4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lr6-20020a170906fb8600b0092ff55fd0cfsi250968ejb.818.2023.04.04.05.58.29; Tue, 04 Apr 2023 05:58:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=GwmjIJ4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234341AbjDDMlc (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234878AbjDDMlA (ORCPT ); Tue, 4 Apr 2023 08:41:00 -0400 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 313731739; Tue, 4 Apr 2023 05:40:57 -0700 (PDT) Received: by mail-ed1-x532.google.com with SMTP id w9so130035052edc.3; Tue, 04 Apr 2023 05:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b0oibOM/o2KUQ8VFakULVS2Dn6jFvS8HYN/Ntyf1ZUA=; b=GwmjIJ4S5yZX09/3v4NT/IFIyTPeYrkm2Q05j90JZgJnwC1c3dxwX+NWIroO8dpNl2 h8F8cEay5gFQK9LLVgLiJVuGYE6i7rPlr9vZD7WWGMZl07kU3CU5OYbrGI0cn8njH7vN ZV2fWsdunDt0/PMfTzCZMAQP/NilA8N9BY5wN1seG3Nca+3T9X5RTxY6lVSAyB54ZO4P 8L8f2acB23xJt2TfDZRNRLwZ9YAZs0zdOy+Ra2tu/VvtwdQJUc2wjP+6aUWAYSzaqW5n kjhhl/tx5QVtBjwRbxmKHKH4TEwVEIkvzud54Abpwpla4nPce9+5e8RawpE/ARFOrmUh eYNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b0oibOM/o2KUQ8VFakULVS2Dn6jFvS8HYN/Ntyf1ZUA=; b=UnYw9SdUufSlVnNAWvuhV9d6Y/AlYsI2o8fqf5yt77ljdC6XddO0hPe/K5H4BS0KhT guyntw987filBujoVZg4IgSVFb7ltyEUF9oNgEqlzmtVLvYujyWBpN7S3+3sb/rSRm7Y uA1hRLPH6ma2UNpeTGNYvDp6ROFZMocW/QYP4vFUHkBy5zCaQtBTcgrsvPH31bm4/F3M Qg4y7niOheXFlFp8R4lJVcV44F6aR2PejGrm8Pr1fSlHqG57R7CPpVFxyBG7GF/ATUCB vHqSImzndAcyvKgY8RXsYpGiIYLviovR2wWQOsywq1U2Y2V9yl5S8yPFFDneNKApSVNZ h5yw== X-Gm-Message-State: AAQBX9e/TLvEpUhbjBq/nX5T+gL+4cS1iMQeklm2bxVkTgBzVm7r2jiI gHyCsQAAoG1+HXfTwrP2VJYS7VKLC6o= X-Received: by 2002:a17:906:3999:b0:93b:46f7:a716 with SMTP id h25-20020a170906399900b0093b46f7a716mr1763115eje.50.1680612055565; Tue, 04 Apr 2023 05:40:55 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:55 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 09/13] io_uring/rsrc: don't offload node free Date: Tue, 4 Apr 2023 13:39:53 +0100 Message-Id: <13fb1aac1e8d068ad8fd4a0c6d0d157ab61b90c0.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250597679786552?= X-GMAIL-MSGID: =?utf-8?q?1762250597679786552?= struct delayed_work rsrc_put_work was previously used to offload node freeing because io_rsrc_node_ref_zero() was previously called by RCU in the IRQ context. Now, as percpu refcounting is gone, we can do it eagerly at the spot without pushing it to a worker. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 -- io_uring/io_uring.c | 6 ---- io_uring/rsrc.c | 59 +++------------------------------- 3 files changed, 4 insertions(+), 64 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9492889f00c0..47496059e13a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -330,9 +330,6 @@ struct io_ring_ctx { struct io_rsrc_data *file_data; struct io_rsrc_data *buf_data; - struct delayed_work rsrc_put_work; - struct callback_head rsrc_put_tw; - struct llist_head rsrc_put_llist; /* protected by ->uring_lock */ struct list_head rsrc_ref_list; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 764df5694d73..d6a0025afc31 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -326,9 +326,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); INIT_LIST_HEAD(&ctx->rsrc_ref_list); - INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); - init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); - init_llist_head(&ctx->rsrc_put_llist); init_llist_head(&ctx->work_llist); INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; @@ -2821,11 +2818,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_rsrc_node_destroy(ctx->rsrc_node); if (ctx->rsrc_backup_node) io_rsrc_node_destroy(ctx->rsrc_backup_node); - flush_delayed_work(&ctx->rsrc_put_work); - flush_delayed_work(&ctx->fallback_work); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); - WARN_ON_ONCE(!llist_empty(&ctx->rsrc_put_llist)); #if defined(CONFIG_UNIX) if (ctx->ring_sock) { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 9647c02be0dc..77cb2f8cfd68 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -145,15 +145,8 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, { struct io_ring_ctx *ctx = rsrc_data->ctx; - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } + if (prsrc->tag) + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); rsrc_data->do_put(ctx, prsrc); } @@ -176,32 +169,6 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) complete(&rsrc_data->done); } -void io_rsrc_put_work(struct work_struct *work) -{ - struct io_ring_ctx *ctx; - struct llist_node *node; - - ctx = container_of(work, struct io_ring_ctx, rsrc_put_work.work); - node = llist_del_all(&ctx->rsrc_put_llist); - - while (node) { - struct io_rsrc_node *ref_node; - struct llist_node *next = node->next; - - ref_node = llist_entry(node, struct io_rsrc_node, llist); - __io_rsrc_put_work(ref_node); - node = next; - } -} - -void io_rsrc_put_tw(struct callback_head *cb) -{ - struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, - rsrc_put_tw); - - io_rsrc_put_work(&ctx->rsrc_put_work.work); -} - void io_wait_rsrc_data(struct io_rsrc_data *data) { if (data && !atomic_dec_and_test(&data->refs)) @@ -217,34 +184,18 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - bool first_add = false; - unsigned long delay = HZ; node->done = true; - - /* if we are mid-quiesce then do not delay */ - if (node->rsrc_data->quiesce) - delay = 0; - while (!list_empty(&ctx->rsrc_ref_list)) { node = list_first_entry(&ctx->rsrc_ref_list, struct io_rsrc_node, node); /* recycle ref nodes in order */ if (!node->done) break; - list_del(&node->node); - first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); - } - if (!first_add) - return; - - if (ctx->submitter_task) { - if (!task_work_add(ctx->submitter_task, &ctx->rsrc_put_tw, - ctx->notify_method)) - return; + list_del(&node->node); + __io_rsrc_put_work(node); } - mod_delayed_work(system_wq, &ctx->rsrc_put_work, delay); } static struct io_rsrc_node *io_rsrc_node_alloc(void) @@ -320,13 +271,11 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, if (ret < 0) { atomic_inc(&data->refs); /* wait for all works potentially completing data->done */ - flush_delayed_work(&ctx->rsrc_put_work); reinit_completion(&data->done); mutex_lock(&ctx->uring_lock); break; } - flush_delayed_work(&ctx->rsrc_put_work); ret = wait_for_completion_interruptible(&data->done); if (!ret) { mutex_lock(&ctx->uring_lock); From patchwork Tue Apr 4 12:39:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2996986vqo; Tue, 4 Apr 2023 05:45:54 -0700 (PDT) X-Google-Smtp-Source: AKy350Zxw59AVByZG3oHJQzbH6sgVoFdWbpQvRsF06Vb9aEiWLRz4n63MSeV9ur2njPwGKFah7JH X-Received: by 2002:a17:90a:bf18:b0:240:daf9:7ab6 with SMTP id c24-20020a17090abf1800b00240daf97ab6mr2565359pjs.40.1680612353732; Tue, 04 Apr 2023 05:45:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612353; cv=none; d=google.com; s=arc-20160816; b=mPBZpXh9n6IC4B3htBZQPr7sFi/30x/3VYBJMIo+RnV4pnwWCYGAfq1evfBU9vlHg5 oLkY9bOBcV3MlKkXrV+A2TQPRGB1CxLoRRKltBTO6fKdM8vLp6KfmOQ2/gjIJvtV9tB9 1VM79l+yrXVQI9qGyh7oTExPtKftCGjS/mN6Reik/nPBnPJe9OcCIaKcshq+u3LCENOd +B+2SO36+jPKOAi2hXPmv40hVa1rme13V/GzBwGNxwI2VbpaswW+XbtAK4TaCSwMnRTv KB9/1ZfMsVhyLcjA77+nq5bKzrcDgVrcOdMC47eiM3bsStwSI1mNJbiIjcC07Vrq5FP2 WDSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OosuePooWAYrbPI47Qhw79nJ97AML6lNG9IhcjMYqms=; b=KJg9zFndKtJR7Zl7aXKYIUvXrv/HQsh+ngNNhXKyZzH789ibvWbi1oNydI2yX/7/R6 kPCAvPH5mJZI7lZVwqHfX25IF9+6Z6G/iTBevW6juFJJbcRaJGC+rVesYv7U08RO6jKq opV2sCb+zz8GlsALJbkNJvaKhOJU34QsnM4iB7bXHnFhfmVXESmtv5jj5//KWJPijmj1 uWX5SkGYglgEGa7pbjn2Q2rUBJf0+k0iWw1zE4C4GFcvnaWPlqY6b5ZwfQSfpEnL449N HJyTOVpWjwaqkcAiPinX8/p2OzQjQfRTGSW1443kyhiGMhrZPBg6i70tW8c56AaL89+z cqfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="dSk/vKob"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gc24-20020a17090b311800b0023f064f20c7si9867168pjb.168.2023.04.04.05.45.40; Tue, 04 Apr 2023 05:45:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="dSk/vKob"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234998AbjDDMlh (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234903AbjDDMlA (ORCPT ); Tue, 4 Apr 2023 08:41:00 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5A4B2691; Tue, 4 Apr 2023 05:40:57 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id y4so130081362edo.2; Tue, 04 Apr 2023 05:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OosuePooWAYrbPI47Qhw79nJ97AML6lNG9IhcjMYqms=; b=dSk/vKob9igkkdeg8NRXjQ7AQa5psSBN7jeNLQ7wZKtCoGXz3xvA+iirvCDAbsMnu5 iblL8A2WF37DKGDHzOq+vUlxJzAILpV57JaEO6b3/qUCpm8bTXxx8q/+fe+FWv6ZGxKy b4jwFT9PPWHyPEQMbxMe/OiURYQcCrqA5i6km6aw7upGeNdxk2g3t6fQSv9MLjauIBWh dPLkt6Wd1KlvCd/VRa+TDrG+IGNHiHWX2wHVMR0wDLPTFuzClLh56+xclOFHG6vHPNMB mzpfE7ZE0u/w1pUag2Ku021PAw3oBN9cnDnTaCOWsVKlJ9SX7NeIovxSjavRnu/YvHnV djoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OosuePooWAYrbPI47Qhw79nJ97AML6lNG9IhcjMYqms=; b=8LqIO7Ie/DrmxV6hODoRl+hxwM0f1fpwzLw8KzuOAsK+w2wQoAqXdzupYPzdeLGV5H x863p7F5SzNmoRp4/b+IJaPLQmxPJwIRYpc8QQx3P/YiHhlm6p/os3fp6bMovaWdHSIy yqSM03xggIvT/D7nkp34STQLB+WuF+2FzNj+cu5fMedVbDA9Pk25DpwgWNPtwRK9ldZq kH7M3npyelSuizFWFEk4xXn/fkJBnScbbLNlXLipfO965awF3OfLpBJTum9qEIUwMrwx VhxEZ0hoBOINzJnggIui37RaK7jjPJhhGBqdHxltFmQgbCwnP2p8vEGBL+BtgLQZsUzz rW9A== X-Gm-Message-State: AAQBX9fRE73j1sBrqsHFCYQyCn4RVTzXpYhTdbO4vYn+C5+aii1cml06 3twVlan1lhmez7COCvjPgHyBfM2hTfo= X-Received: by 2002:a17:906:1dd4:b0:92a:8067:7637 with SMTP id v20-20020a1709061dd400b0092a80677637mr1939585ejh.61.1680612056176; Tue, 04 Apr 2023 05:40:56 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:56 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 10/13] io_uring/rsrc: cache struct io_rsrc_node Date: Tue, 4 Apr 2023 13:39:54 +0100 Message-Id: <252a9d9ef9654e6467af30fdc02f57c0118fb76e.1680576071.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249779653843615?= X-GMAIL-MSGID: =?utf-8?q?1762249779653843615?= Add allocation cache for struct io_rsrc_node, it's always allocated and put under ->uring_lock, so it doesn't need any extra synchronisation around caches. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 11 +++++++++-- io_uring/rsrc.c | 23 +++++++++++++++-------- io_uring/rsrc.h | 9 +++++++-- 4 files changed, 32 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 47496059e13a..5d772e36e7fc 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -332,6 +332,7 @@ struct io_ring_ctx { /* protected by ->uring_lock */ struct list_head rsrc_ref_list; + struct io_alloc_cache rsrc_node_cache; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d6a0025afc31..419d6f42935f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,6 +310,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); + io_alloc_cache_init(&ctx->rsrc_node_cache, sizeof(struct io_rsrc_node)); io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); @@ -2790,6 +2791,11 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) mutex_unlock(&ctx->uring_lock); } +static void io_rsrc_node_cache_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct io_rsrc_node, cache)); +} + static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); @@ -2815,9 +2821,9 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) - io_rsrc_node_destroy(ctx->rsrc_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_node); if (ctx->rsrc_backup_node) - io_rsrc_node_destroy(ctx->rsrc_backup_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_backup_node); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); @@ -2829,6 +2835,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) #endif WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); + io_alloc_cache_free(&ctx->rsrc_node_cache, io_rsrc_node_cache_free); if (ctx->mm_account) { mmdrop(ctx->mm_account); ctx->mm_account = NULL; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 77cb2f8cfd68..cbf563fcb053 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -164,7 +164,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) kfree(prsrc); } - io_rsrc_node_destroy(ref_node); + io_rsrc_node_destroy(rsrc_data->ctx, ref_node); if (atomic_dec_and_test(&rsrc_data->refs)) complete(&rsrc_data->done); } @@ -175,9 +175,10 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) wait_for_completion(&data->done); } -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - kfree(ref_node); + if (!io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache)) + kfree(node); } void io_rsrc_node_ref_zero(struct io_rsrc_node *node) @@ -198,13 +199,19 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) } } -static struct io_rsrc_node *io_rsrc_node_alloc(void) +static struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) { struct io_rsrc_node *ref_node; + struct io_cache_entry *entry; - ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); - if (!ref_node) - return NULL; + entry = io_alloc_cache_get(&ctx->rsrc_node_cache); + if (entry) { + ref_node = container_of(entry, struct io_rsrc_node, cache); + } else { + ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); + if (!ref_node) + return NULL; + } ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); @@ -243,7 +250,7 @@ int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) { if (ctx->rsrc_backup_node) return 0; - ctx->rsrc_backup_node = io_rsrc_node_alloc(); + ctx->rsrc_backup_node = io_rsrc_node_alloc(ctx); return ctx->rsrc_backup_node ? 0 : -ENOMEM; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 11703082d125..3b9f4c57c47c 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -4,6 +4,8 @@ #include +#include "alloc_cache.h" + #define IO_RSRC_TAG_TABLE_SHIFT (PAGE_SHIFT - 3) #define IO_RSRC_TAG_TABLE_MAX (1U << IO_RSRC_TAG_TABLE_SHIFT) #define IO_RSRC_TAG_TABLE_MASK (IO_RSRC_TAG_TABLE_MAX - 1) @@ -37,8 +39,11 @@ struct io_rsrc_data { }; struct io_rsrc_node { + union { + struct io_cache_entry cache; + struct io_rsrc_data *rsrc_data; + }; struct list_head node; - struct io_rsrc_data *rsrc_data; struct llist_node llist; int refs; bool done; @@ -65,7 +70,7 @@ void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_wait_rsrc_data(struct io_rsrc_data *data); -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); From patchwork Tue Apr 4 12:39:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2997349vqo; Tue, 4 Apr 2023 05:46:33 -0700 (PDT) X-Google-Smtp-Source: AKy350YXwNrK8mIoJsLNGSJqCGnmi2eb71nAL7CVmy+hM049rVl9lT9y8lZTE4hHgpKmJQ0/NIa0 X-Received: by 2002:a17:903:41c7:b0:19f:2dff:21a4 with SMTP id u7-20020a17090341c700b0019f2dff21a4mr3126457ple.16.1680612392948; Tue, 04 Apr 2023 05:46:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612392; cv=none; d=google.com; s=arc-20160816; b=0AJTLwr/mDpxrOhI/lGG2yjjzayNOKn/cUVLZzcY7ap4Nzfb4T4FvrlHBMVy2U49d4 0rBJZdPRcd4TqhbXUsA4mZ4xf6iWz28TrCFrbfXqCtryav8eBuS9y01ML4R/XDUjhvBq qc8xRQbzSc7fiZo8inUC8pZUIUHTmRTdcVUx10ha5qDzWYhlQnscPWbkXsK9DblGeujX gWCuDPFlYZ10VIExkv/PcjWgfru+LGp7M4jvRipaceBaXLrcEagcZpW/YXHdPyR7YI0X i+SBMnYunFvv2kOVTRwlNk0RsCBWJ3zclWq6YCVnXmZeaEznfThvYDuLK4l/97MrR9+2 kbGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/Cxnc6JJ4qGOse8EhEinOqNQd7wwXuaUvmBXkiroock=; b=OZpaWo4+PZ28qwcZHmnWSn3SSDUFKYsaj0ZM++DnodqkQ+4GULfVlI92JOLpa4zxAX fbLeOxIvll7QQr7zXVktnEJnlNLWf/GziLlQiQcK5Iti4AT1dQ12nxuAKiVDQgt+daIm rsjRo8dzKZqHJt2xF2CcaEGp0c5VnqMse0QxzV8FbcnLvfyFF05KTGPpEqk0b3PMpkkB OXWSEkcdmAG23QB6pOS/fQkVl/VNRvsiV4lpE1f1E/knOjr/8VH1yqH2g0AIjlNkfZWk mUbynn6oBUeW5tGHoKST23a5U4Rr2ia3XeIr1ZdeZ3DNj+CfbtQQK3rmfnsCWx2gNNTY EChg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=gFUqc+zK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r11-20020a632b0b000000b0050f66d4e070si11098890pgr.417.2023.04.04.05.46.20; Tue, 04 Apr 2023 05:46:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=gFUqc+zK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234676AbjDDMll (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234932AbjDDMlA (ORCPT ); Tue, 4 Apr 2023 08:41:00 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51A0ADC; Tue, 4 Apr 2023 05:40:58 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id h8so130056381ede.8; Tue, 04 Apr 2023 05:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Cxnc6JJ4qGOse8EhEinOqNQd7wwXuaUvmBXkiroock=; b=gFUqc+zKGga9k0qzAdqlH1gM+8CtYlpy+pGMckAxAwFPuqueTJ86Ciw5ZRqgLgqU2l sJAQXjA8vS2mlG1UyQYjtuA91TI9s8OPE95zwTJ7jy7x/B17xFADrqup42Q7+jCN1YRJ FDBKPxaFtwNPt9/Llb8vouwQD8tchPYs7zK0dommYY3w8fRPjeA2NhDQ7wgoNGpZnqn+ dF33eg4jfLs+Avp61orMDDdRwPyVx0htnPED3iN3Zxl4bU3MVsGBpZ1lvQRA2o5ao+Ly RcsGJIFsxP8iBp+I/TLQj2mja4v3csBKd+0JdQImLefKd0NdT6bo3xFGJMv+YGqoE5O/ tHjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Cxnc6JJ4qGOse8EhEinOqNQd7wwXuaUvmBXkiroock=; b=TIP5tfsgWtcPYzmQog6VZESsX7ckn0xKEj5wilmvaOpR47p+A9I9UgP8Mrw/h2u0Wg 2riOq4jVrGQq/X3rEHDr417+zfCevCcZ047TmVYHjTeQRk18hMLqnXGz+S9B50x4Lwze 754Ze+My6LWnzmdUYSHcUAfnINjB9KF8eWLCF4Ndpixwedg2MjTDw2kXJujf/okBPB8R 40uTMOYsw23L773p1T9twlenkXQUaWN3CyHtzDpHSzlXbCXf7BBwhK8kfbZyROa4k9fa LGk9NF0I4GZ3BQcf3i/hJZQtFW54GsCa7tIjk7Z7XCnN9n+jOSgj1XH3havYqhBHx0JO 9XoQ== X-Gm-Message-State: AAQBX9fjHuGdYy0MkDfEG01s/nxb1V1AVJAnlXUNrYvWgh+Zw5+bbxA9 xre0LO21O7K8tDlengKhl9qG0/hAal4= X-Received: by 2002:a17:906:3012:b0:92f:ccf3:f0ce with SMTP id 18-20020a170906301200b0092fccf3f0cemr2131169ejz.45.1680612056699; Tue, 04 Apr 2023 05:40:56 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:56 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 11/13] io_uring/rsrc: add lockdep sanity checks Date: Tue, 4 Apr 2023 13:39:55 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249820409559213?= X-GMAIL-MSGID: =?utf-8?q?1762249820409559213?= We should hold ->uring_lock while putting nodes with io_put_rsrc_node(), add a lockdep check for that. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ++-- io_uring/rsrc.c | 2 +- io_uring/rsrc.h | 6 ++++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 419d6f42935f..da36fa1eeac9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1002,7 +1002,7 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) if (rsrc_node) { io_ring_submit_lock(ctx, issue_flags); - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } @@ -1123,7 +1123,7 @@ static __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) if (req->rsrc_node) { io_tw_lock(ctx, ts); - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index cbf563fcb053..95edc5f73204 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -236,7 +236,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 3b9f4c57c47c..cf24c3fd701f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -119,8 +119,10 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_put_rsrc_node(struct io_rsrc_node *node) +static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { + lockdep_assert_held(&ctx->uring_lock); + if (node && !--node->refs) io_rsrc_node_ref_zero(node); } @@ -128,7 +130,7 @@ static inline void io_put_rsrc_node(struct io_rsrc_node *node) static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) { - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, From patchwork Tue Apr 4 12:39:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79078 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3000904vqo; Tue, 4 Apr 2023 05:52:47 -0700 (PDT) X-Google-Smtp-Source: AKy350ZrfuYjKRfzsj9Q4GmMYtN1DAT5N6I7Qrqvo3Fjd5qCClBLu7xoiJ6SoqUMgBBSvvZnKwuA X-Received: by 2002:a17:906:2e8f:b0:93e:3127:fc28 with SMTP id o15-20020a1709062e8f00b0093e3127fc28mr2103504eji.39.1680612767408; Tue, 04 Apr 2023 05:52:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612767; cv=none; d=google.com; s=arc-20160816; b=xGZMyp2Jx4qMStdy5IUic2X2hDXgT5kOQC5AR1WIhI1e0k8ghj3LPR2Wfx5tSEaRmH IkyppL3wtBBtsk7JtD0Vup7NAEB00pHe76JoZC5dpN6m1YIBhB7UFdRN1HaxWYAZNwF/ uHyNbCgfM3srfX5DjWZYNOIR7WO4T9cf4Fj47uE5nrzrq2+ZRU8lSs5sE6uquz/tUh7W KjPpqYauFpGqFQRCR5Jqwi5z016knYUcGjvHiN/vraveDZHStB1wa8Mizo/tA4AHecLg zj3VhogXL/wjw4BoGnFxXFPv1JBZIfAyhZIJEvpY/9KITr/31DLRb1mPluTjXU26X56w 7W0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n5A3CaykmQoi1hwmp2XQWbZMbgV4kSoJXYsNmB3sL+Q=; b=XpjOSW5POKpAP/3s3VFg8sd3J6KKZtnamK4LhRr9BYzA6hDbt8+BOt6XwHxsXow60C xaOPjNW1JhD9GSwOk+17zzxtNyEXbgWy9n/1n93ZyDygivCIPZ3QYPJVZo79zFStgv4i EwKNkZKLVjL4ku8k/R/URTDS6PvDzLBEzsm2aECK9O9Z3iI2JKErFmD4W5KyJmKxD5xC U5D5eBhP3avMtvry/j8qg+9jn6n42qqIN2cI40xMI4c32w3Jvv9qjNXneJ6NwQ0Eqwhh WfZmGPlTwFUazIguKU7YTGFkI3F5vLjrSnYl+egsmiaN7nb2N91f061CPKUp+oTQPXkk whZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Va0857XB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x26-20020a1709064bda00b0092fbbe5348bsi310229ejv.918.2023.04.04.05.52.23; Tue, 04 Apr 2023 05:52:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Va0857XB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234587AbjDDMlr (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234496AbjDDMlB (ORCPT ); Tue, 4 Apr 2023 08:41:01 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BBC31BCB; Tue, 4 Apr 2023 05:40:59 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id eh3so129968552edb.11; Tue, 04 Apr 2023 05:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612057; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=n5A3CaykmQoi1hwmp2XQWbZMbgV4kSoJXYsNmB3sL+Q=; b=Va0857XBhN3i/RceyFvKAGG9NyTjeRMoHgviPepMFyhspMvR5xQoNdjq6JcHUjpf28 WQmrGfLNDfk5aNTyPmbQ1sLVZGhlvo+ADJpX+iIE3O6ehr5J2yNyQU5893OVV4DQi8hF rFcfE41F0gzqJAhU996/e1mjAjhRr1n5g2vx1+BxZ4ujdJ/KY53hgLHA4lDJ+VL7Ut6G 0xNd/CH48gL4nDYivWdMqoWtnvL177k+quPFndCTV+D7B4aLnueFijUrhaWLex01WNHw QeiLlRrJqZvD9ZSL2WPV5S6hxIFY2prUFEruWMLIb927Pk0jR+YrLAfjwha2BdGLplnL TgCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612057; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n5A3CaykmQoi1hwmp2XQWbZMbgV4kSoJXYsNmB3sL+Q=; b=CaMCg7fn7xofEyxK+zgKP0FW7wE7KyqVLW79LFK/Jn2Hw0Peh6yO9tT2LUqq0LbeIr 2wPlxK83pbtK14PGrAyi0gCuiDDFteWLmQzAyS0Bgyori5ZpC5n+ztmJm+Pp4InYvfdY qTxqCPP8J54ilkTVve3Yu/6d3jRaiL6LJBEEFdVLq1bhXbvbOS6XExQeamCkdEgUKk7l yzGmlCdnJm9Sh/LpWihYnXN4O6DtDD6Vkpug/SznGCwjyVRtytu426aKLyBvgvZTJyOw hDp3s0a3+loT8odxw2kogYPyka6Be6qiGNLjEAXbOda0A9fSIiRit194rJ7parH2sCqY RnUQ== X-Gm-Message-State: AAQBX9dXIjzBvhawCEyesThhj7ytYAZzPrm0xt9/GYWyhHXMZb2X+TFx l75rEibnUxlvu4x1g7RwvFc4iKZ6vyE= X-Received: by 2002:a17:907:a04e:b0:931:f132:5c61 with SMTP id gz14-20020a170907a04e00b00931f1325c61mr2130933ejc.47.1680612057264; Tue, 04 Apr 2023 05:40:57 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:56 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 12/13] io_uring/rsrc: optimise io_rsrc_data refcounting Date: Tue, 4 Apr 2023 13:39:56 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762250213499693438?= X-GMAIL-MSGID: =?utf-8?q?1762250213499693438?= Every struct io_rsrc_node takes a struct io_rsrc_data reference, which means all rsrc updates do 2 extra atomics. Replace atomics refcounting with a int as it's all done under ->uring_lock. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 30 ++++++++++++++++++------------ io_uring/rsrc.h | 2 +- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 95edc5f73204..74e13230fa0c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -31,6 +31,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) +static inline bool io_put_rsrc_data_ref(struct io_rsrc_data *rsrc_data) +{ + return !--rsrc_data->refs; +} + int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -165,13 +170,13 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) } io_rsrc_node_destroy(rsrc_data->ctx, ref_node); - if (atomic_dec_and_test(&rsrc_data->refs)) + if (io_put_rsrc_data_ref(rsrc_data)) complete(&rsrc_data->done); } void io_wait_rsrc_data(struct io_rsrc_data *data) { - if (data && !atomic_dec_and_test(&data->refs)) + if (data && !io_put_rsrc_data_ref(data)) wait_for_completion(&data->done); } @@ -234,7 +239,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, rsrc_node->rsrc_data = data_to_kill; list_add_tail(&rsrc_node->node, &ctx->rsrc_ref_list); - atomic_inc(&data_to_kill->refs); + data_to_kill->refs++; /* put master ref */ io_put_rsrc_node(ctx, rsrc_node); ctx->rsrc_node = NULL; @@ -267,8 +272,8 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, return ret; io_rsrc_node_switch(ctx, data); - /* kill initial ref, already quiesced if zero */ - if (atomic_dec_and_test(&data->refs)) + /* kill initial ref */ + if (io_put_rsrc_data_ref(data)) return 0; data->quiesce = true; @@ -276,17 +281,19 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, do { ret = io_run_task_work_sig(ctx); if (ret < 0) { - atomic_inc(&data->refs); - /* wait for all works potentially completing data->done */ - reinit_completion(&data->done); mutex_lock(&ctx->uring_lock); + if (!data->refs) { + ret = 0; + } else { + /* restore the master reference */ + data->refs++; + } break; } - ret = wait_for_completion_interruptible(&data->done); if (!ret) { mutex_lock(&ctx->uring_lock); - if (atomic_read(&data->refs) <= 0) + if (!data->refs) break; /* * it has been revived by another thread while @@ -361,6 +368,7 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, data->nr = nr; data->ctx = ctx; data->do_put = do_put; + data->refs = 1; if (utags) { ret = -EFAULT; for (i = 0; i < nr; i++) { @@ -371,8 +379,6 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, goto fail; } } - - atomic_set(&data->refs, 1); init_completion(&data->done); *pdata = data; return 0; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index cf24c3fd701f..7ab9b2b2e757 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -33,8 +33,8 @@ struct io_rsrc_data { u64 **tags; unsigned int nr; rsrc_put_fn *do_put; - atomic_t refs; struct completion done; + int refs; bool quiesce; }; From patchwork Tue Apr 4 12:39:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 79070 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2997091vqo; Tue, 4 Apr 2023 05:46:05 -0700 (PDT) X-Google-Smtp-Source: AKy350Yu3zODh8/VEKXay9gjdHFRdoehtpKVujz6iOxyJ7gahbyYq6t/Zc13zJkj9QWOUfe8/BEc X-Received: by 2002:a17:902:ca05:b0:1a1:b440:3773 with SMTP id w5-20020a170902ca0500b001a1b4403773mr2118266pld.27.1680612364754; Tue, 04 Apr 2023 05:46:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680612364; cv=none; d=google.com; s=arc-20160816; b=QrsLLr71+Hj+Vx+E6bUdX/TRXb+gIioRSj4vKgOXINid2vQ8cDvyENjPDh6E7hUlp+ 06AADLoVU0l9PkJ/r7PxW1cplI0btombs/fNnpSZNNzzkon0lI7I0PtqJVxmg2Ub0ERC bGc3mHI/sd0skSH87d+AYZzquxDHDFPM/pZC05XTWHGC2z0nWn+W05zB5Sd7w3s7pF+B 2VdHfe0O5cO2ZKzPmckw5qJgpppA1Rogj3o5g0oU7AxzDZntRe6qA5ZzKt66j0fIBB2p 6RdMP05MmUoWvmE5oTPTdQPAP6XQ7NtbzUholvmV/mc0nUPEGMg/oU4uwtPOsIRzNYZ8 SMbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=B4jUcK/aZoVr0sdbTUSCMWYGWimEni1x0jIOLPJhcKwGziNAlIzNymfmm0WqkwxdQH nzoHx623qIi5hXJIb1WmYV02UeLhEBqScGFJuZR9pIWcWhtOfAvnE9lmg/Z5MFkxM84c 5dhaDKBhz6IEorgx3eK61BEiX/chYHOuIQQiInmoX5fwO8zxjAiP1mjyoIYAIBrSNQcu nvd2JKwca3JfzhuzWVZnUUsUaRHMxfH5BdFvNKiDC2fcqcvUnM1OExxEyOFjU6K4ucA+ qZJzVNBOi0hV6z5ZRphbsV7rMEJISDbOCcRU//ZGWjG2tgsYdZe6OZQf/7Ka+U8SDqur 6fYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="X/J1llEl"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b15-20020a63e70f000000b0050bfc85d989si9756664pgi.154.2023.04.04.05.45.50; Tue, 04 Apr 2023 05:46:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="X/J1llEl"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234352AbjDDMln (ORCPT + 99 others); Tue, 4 Apr 2023 08:41:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234944AbjDDMlB (ORCPT ); Tue, 4 Apr 2023 08:41:01 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BA941BC7; Tue, 4 Apr 2023 05:40:58 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id cn12so130060658edb.4; Tue, 04 Apr 2023 05:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680612058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=X/J1llEltWTp4lG2E3lTY7RTuXuKmHYGHhZGxSbzgjJt2spsoc45a6S/UqZmy9U95v 3AJyxDDnt6hnM3wGswTBwN1w1rruCJajzH0hS0det28+p4PqaXO8No707bkfYlIH4gT4 Ex2bhCgpaA6oAvTi8s4cYa4nscj59hZrpo1b3ng1sk6YDj3US4/VIvvsp4VvCSLg+Ujx h9EEbua/G4akmIB9ncr08dx49v8Qc70b1mVxMeD3YoySqg/Uc5IN/XwrJFru7RQb+Ybl izyZWc77RLGkl1EcLvSEk2Bp2iPDemOm0BTIkg8dWFmwopEODxnBjKwkbHz2tviJFjob IU8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680612058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1CwMuzyH2dEdSI/CO0b8pg+tuSUqIX+/sHQt7TiN5ZM=; b=RORmEIuMWGFX33I16cb6QNsDJXHrR13ShHDaBgvBSPTZw50EamLC490Rc0GncfE4wv 7s4LmboPKH1ClcLrVM6NFGw1pveRK+udra3LEMXle55Xi3IatgYBgYFjqsFWDg3IdmCT c91zEkMf2ZPHoujLaLBrZ88qOc39yYzJ+Q9j0rKBqGhyd50Bi+CS6RkzPPKX5ldIyLMx I2cva+YBgZBnyA1SRsVpVKpJpDkW7eeN4/JMfK7FvVtaOl53GMbs8irKCnbEbs9Q53h7 Fxnvt/DuLQEeYghNBggxIEjaqnIiagmN67d96jFxms9BoRDMsPPu4gBblLICiYncnx60 K5aA== X-Gm-Message-State: AAQBX9eWMsB/9a17Mz/cGOp3xStxoDBFImnJ9V/r1fGIbbgE6OqWg8j8 AEjTDpEc+ADikCg72l+KVgJeBR4VWYw= X-Received: by 2002:a17:907:8c10:b0:8f0:143d:ee34 with SMTP id ta16-20020a1709078c1000b008f0143dee34mr2624104ejc.1.1680612057944; Tue, 04 Apr 2023 05:40:57 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:2b22]) by smtp.gmail.com with ESMTPSA id g8-20020a170906394800b008cafeec917dsm5978851eje.101.2023.04.04.05.40.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 05:40:57 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 13/13] io_uring/rsrc: add custom limit for node caching Date: Tue, 4 Apr 2023 13:39:57 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762249790617219164?= X-GMAIL-MSGID: =?utf-8?q?1762249790617219164?= The number of entries in the rsrc node cache is limited to 512, which still seems unnecessarily large. Add per cache thresholds and set to to 32 for the rsrc node cache. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/alloc_cache.h | 6 ++++-- io_uring/io_uring.c | 9 ++++++--- io_uring/rsrc.h | 2 ++ 4 files changed, 13 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5d772e36e7fc..4a6ce03a4903 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -190,6 +190,7 @@ struct io_ev_fd { struct io_alloc_cache { struct io_wq_work_node list; unsigned int nr_cached; + unsigned int max_cached; size_t elem_size; }; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 2fbecaa3a1ba..851a527afb5e 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -13,7 +13,7 @@ struct io_cache_entry { static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, struct io_cache_entry *entry) { - if (cache->nr_cached < IO_ALLOC_CACHE_MAX) { + if (cache->nr_cached < cache->max_cached) { cache->nr_cached++; wq_stack_add_head(&entry->node, &cache->list); /* KASAN poisons object */ @@ -38,10 +38,12 @@ static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *c return NULL; } -static inline void io_alloc_cache_init(struct io_alloc_cache *cache, size_t size) +static inline void io_alloc_cache_init(struct io_alloc_cache *cache, + unsigned max_nr, size_t size) { cache->list.next = NULL; cache->nr_cached = 0; + cache->max_cached = max_nr; cache->elem_size = size; } diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index da36fa1eeac9..ae90d2753e0d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,9 +310,12 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); - io_alloc_cache_init(&ctx->rsrc_node_cache, sizeof(struct io_rsrc_node)); - io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); - io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); + io_alloc_cache_init(&ctx->rsrc_node_cache, IO_NODE_ALLOC_CACHE_MAX, + sizeof(struct io_rsrc_node)); + io_alloc_cache_init(&ctx->apoll_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct async_poll)); + io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 7ab9b2b2e757..8729f2fee256 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -6,6 +6,8 @@ #include "alloc_cache.h" +#define IO_NODE_ALLOC_CACHE_MAX 32 + #define IO_RSRC_TAG_TABLE_SHIFT (PAGE_SHIFT - 3) #define IO_RSRC_TAG_TABLE_MAX (1U << IO_RSRC_TAG_TABLE_SHIFT) #define IO_RSRC_TAG_TABLE_MASK (IO_RSRC_TAG_TABLE_MAX - 1)