From patchwork Thu Mar 30 14:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77223 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1185941vqo; Thu, 30 Mar 2023 07:55:49 -0700 (PDT) X-Google-Smtp-Source: AKy350bcVRJRtRapm0DjFJtIIwvr2bgwTZL87DyGPkQt1ZQSOVXQsNX4UBmgFylx76KN94WFbJ3m X-Received: by 2002:a17:907:8d8d:b0:947:5bd9:1110 with SMTP id tf13-20020a1709078d8d00b009475bd91110mr2493657ejc.45.1680188149323; Thu, 30 Mar 2023 07:55:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680188149; cv=none; d=google.com; s=arc-20160816; b=k8jO66ryXMwvC7z2AWNjIp/lIhTYvGVmQTC/RfiN5tKf5M9MB8uCoN2iSF0GsBlHE9 DRZsNrxvODxnUAmD1fWpoQ5AdWsEGN64nwc2qYkaMNzf4Z18aZGk8eZYefi4BgsGuPq/ ZW4ur0SR9JWR4tzttgKn0w5+Nm1jvOOH+zeE6sQJH9Er3baJdjhotssMW/m+j0fSgMLu d6kna5IItCj6OLHdTon/k1gofR9djtpaVPPeaEQTROTafEUBfhsHOziAv+YoF3dJwAI7 AWex5zVq1T4UoI2vWbVmF2CnIR6ZZR3h1/b1VODY15pBWkXY0M59YAHkN2GKUUfmxQr2 uLnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=og4wgHhZR2GI1KYFHJJde+4r5/spcbG2U9hgu2p3spE=; b=A6bE7tj53TNEhJUYJpoJhZnQ/5a0hGnTo5YVlaEgAJg3bObBEeB1XCG2qI7kBdYVfi KoTId7z+kqELXBlbhieebIMWFt9LBpsokd7nKQR/zMqad07cjPZnJIMyQentGqOjrfXA z2F/yEl67orlGaM6uVFHPYCUQmpdK6EWrxMRpBqF16rHjYTmoIG9vYAfU6KOxH/S1jvw xrmYQOfM7isZgMAsiMR07BpwCbz8cJm9ob860IBEKbED+x+x/NPfNFcifVPG1Kqvjh+r eCnp2ZOSwkjO7xjxuIG3aDBIX0g3f7aIuKd4r8aXsKaQ2mUU/Oya4WxmPW7crnX6m8ki Oqcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hRTo+Lt0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a13-20020a1709064a4d00b0092bc7e163e9si31261836ejv.658.2023.03.30.07.55.24; Thu, 30 Mar 2023 07:55:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hRTo+Lt0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232536AbjC3Oyk (ORCPT + 99 others); Thu, 30 Mar 2023 10:54:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232517AbjC3Oyf (ORCPT ); Thu, 30 Mar 2023 10:54:35 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47C37C0; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id q19so16311543wrc.5; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=og4wgHhZR2GI1KYFHJJde+4r5/spcbG2U9hgu2p3spE=; b=hRTo+Lt0jHxCwNBNqYPtSs/Fu3DRJIFkKl1hfQPK34dp4JAhtXHHi5Fzn0q/FXzjdO yWceZAoczbvHCjKwpfse9IW+0x6CFa5vppjOUpUnUE/UdW8LEscWvNv0fw/S/DsqiJri hZiRsl2bTtE/B4+R7sohDYRgKD+qjohZ/PqsdyZHV6Rd4qar5WpYCOJXtLHJUY+0L/B+ XrwsAJNeY8MyxkC1lrfH78DOiXq6t/CtrUF2qb3YjeExRNeWmk5gaOBpFf5JS8kihd7X +LMLFIquPGdR8/txghJ/ZQfP80sozyEY0RuNHdszUpycfk7gzeDTBIrSxILoYMi/wM9D RSVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=og4wgHhZR2GI1KYFHJJde+4r5/spcbG2U9hgu2p3spE=; b=j8/ltYu7Zifqfwwx7pcdixAAmKhKZqWekWCRfYQ3BfDSQyPZM98InL4VVwaRe+/n3H a0kDXq7y9MJr1rEaMhvZJY+7+SUO14FB6gsFzL+53C/YvTi3YZ9xmWWfF0F/ThYcpd2H /l8qqyv22WQo69nhPebkWHd1S88Poe3+Fvt1nGtLOJ4W9hRDJn57oELlyx4dd0+hBDYf tolfv1M9ECGAeWASj4GevwuUkC71IMeDo94+LBh5tcqB289dbFG62rQjBeQvXaCTEiMN 1f75XkkXlZWH+wNvjRIQsbWg6VlNt5pXb0lOcR5XT3PRJkqnFN8yCMtA190w0okHshXu G8lg== X-Gm-Message-State: AAQBX9forATOKaNK5FaXrsLManEuQuCIDtodYcc7CEfn3cuw0jZuQYo9 hJcnBEIDYoIpTfS6aWZ6nIbYHGzGBic= X-Received: by 2002:adf:dd10:0:b0:2d8:e6ba:99e8 with SMTP id a16-20020adfdd10000000b002d8e6ba99e8mr17987366wrm.33.1680188072431; Thu, 30 Mar 2023 07:54:32 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:32 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 01/11] io_uring/rsrc: use non-pcpu refcounts for nodes Date: Thu, 30 Mar 2023 15:53:19 +0100 Message-Id: <243504691227d611f27a0f642ec8982db6ef4ead.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761804968883202878?= X-GMAIL-MSGID: =?utf-8?q?1761804968883202878?= One problem with the current rsrc infra is that often updates will generates lots of rsrc nodes, each carry pcpu refs. That takes quite a lot of memory, especially if there is a stall, and takes lots of CPU cycles. Only pcpu allocations takes >50 of CPU with a naive benchmark updating files in a loop. Replace pcpu refs with normal refcounting. There is already a hot path avoiding atomics / refs, but following patches will further improve it. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 15 +++++---------- io_uring/rsrc.h | 6 ++++-- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7a43aed8e395..f2da9e251e3f 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -155,7 +155,7 @@ void io_rsrc_refs_refill(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - percpu_ref_get_many(&ctx->rsrc_node->refs, IO_RSRC_REF_BATCH); + refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -220,13 +220,11 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) { - percpu_ref_exit(&ref_node->refs); kfree(ref_node); } -static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref) +__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) { - struct io_rsrc_node *node = container_of(ref, struct io_rsrc_node, refs); struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; bool first_add = false; @@ -269,11 +267,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - if (percpu_ref_init(&ref_node->refs, io_rsrc_node_ref_zero, - 0, GFP_KERNEL)) { - kfree(ref_node); - return NULL; - } + refcount_set(&ref_node->refs, 1); INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -298,7 +292,8 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); - percpu_ref_kill(&rsrc_node->refs); + /* put master ref */ + io_rsrc_put_node(rsrc_node, 1); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f27f4975217d..1467b31843bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,7 +37,7 @@ struct io_rsrc_data { }; struct io_rsrc_node { - struct percpu_ref refs; + refcount_t refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; @@ -54,6 +54,7 @@ struct io_mapped_ubuf { }; void io_rsrc_put_tw(struct callback_head *cb); +void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_rsrc_refs_refill(struct io_ring_ctx *ctx); void io_wait_rsrc_data(struct io_rsrc_data *data); @@ -109,7 +110,8 @@ int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) { - percpu_ref_put_many(&node->refs, nr); + if (refcount_sub_and_test(nr, &node->refs)) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc(struct io_kiocb *req) From patchwork Thu Mar 30 14:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77252 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1239970vqo; Thu, 30 Mar 2023 09:09:43 -0700 (PDT) X-Google-Smtp-Source: AKy350bTsMWpmwUiRpeapjbQTbQ6MtmcVuWkVeG44am7+lO3XicN7ItJxlEsCdayPU2mnXeM2oeQ X-Received: by 2002:a62:6346:0:b0:62b:47fc:a970 with SMTP id x67-20020a626346000000b0062b47fca970mr21764566pfb.9.1680192582883; Thu, 30 Mar 2023 09:09:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192582; cv=none; d=google.com; s=arc-20160816; b=WrHwVxQxgJYJDC8W17KUDP60ePbaKIVmZSaNzUvD8CuQsjEDIsDyvNaNe/zcvXLxFI JjmW4vr491/IpSxMJVKc4GYVovfAkKR9oh2AMmY7By8Xz+JkBd6b8wGns7oni6Z0MqLH OKtVyXaOodEYvhRNijP9P/eo7x7vF+71hJG97pIQnYYOGm6XzejqvaDqXOAB4Fp4KEV+ AnxteQ1gNRZEgMv/QoKmCJoJhTuro00XC/BYbpS1vW4zeGxqcIGuw7Gya0GF9q8CCCjW MmK9bgb6gt1pYwFOqFATqeDTD5yS19O4Cy6Dj5fBlMBVBoJtkpwm8zYk5wfI20xrnR6U EGUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QZd7Q0w7/z/xq1l37RPu+7/1znEwBdl9Vpp14WIchi8=; b=bGcIyK0zDfnVFdPrVnQPPvYZ7NhKQsy3EF7SoMQkezH6REyCcbFMPat8d7EB1uB2B+ 5Yu2duovQtjlogrAO3UJyj3tjletzhp9T+JMnm3E4ggjdaLGKcXpXEaCuHJZkwc8vvlT qQJ6l/2I1L0vFh+okdC9c3/brIJZAGB+RC38YCrj03Z4tmNTa23O4Xlv4AtOwoOUVWPw w6/ls62xUjfm8nnOymKpbTYX7I1IGIgZteH4ZTnmvjjuqSWYWl9S7sKILB2Y9O1k8P2/ 5p5438FMMrRNkg+XLyI6irO/IT2pwIFt6+Syi2/xEydutXIPqsYdjbZobIGhj8MfcKI5 NjQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="oF/E6TAr"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r18-20020aa79ed2000000b0062d7d3f1192si119861pfq.166.2023.03.30.09.09.28; Thu, 30 Mar 2023 09:09:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="oF/E6TAr"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232553AbjC3Oyn (ORCPT + 99 others); Thu, 30 Mar 2023 10:54:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232508AbjC3Oyg (ORCPT ); Thu, 30 Mar 2023 10:54:36 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F043186; Thu, 30 Mar 2023 07:54:35 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id v1so19392294wrv.1; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QZd7Q0w7/z/xq1l37RPu+7/1znEwBdl9Vpp14WIchi8=; b=oF/E6TAr4lrfZC3fmGeL29xj5v2LilnNvXGMYN+gdNTEYSTobcTwHL6LMKkTnvTilM guWV31Ph4qQ12vMVFlMB9ziYdjPcyyFyUsGiPQlTbFQTPIuT/h/y7KoNi5T3NvGWxZXS btEX8m6FwtdOP1vj2Xk0DudDnm0Z4lJxtRd75g9EBhQIW8PmRDGrJ5M+2yv3yevg0Z0Y z3EOIPlf1W67j7MJZTP7tTv5DGVjSaCwuRN4D+6umji+8us0pcx4vt1FPWIsSwP2/BBK 2Qj0AS1PinRa6wnfLaukPvvitAD67UH7whoWpVytPV9clOKYcEZr+Qw4xNomkHHYSsSK t4hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QZd7Q0w7/z/xq1l37RPu+7/1znEwBdl9Vpp14WIchi8=; b=riWvFsM7E8fx0B2T40iZEvXFzW74SBenDk3odP9lvs2gxuzZvqUPPyzL4GQu9XISMO JhQ/lq/7zA+HPmJa751y/iHlkPgwGSPmZJQrfx5e58AlZouDBw1oSmLqHhBPru5X8yCs 2LJNdVwzC/NBGE1xsEii0Q3mq8QOJP+ejeQKm/nJtNrnwYTwU7/r3mKXy5lP3cOHHAN8 gNDU1LkBKQYe/TPig90n6G2SZf0ULitcWltKqUTYnfJditkxYPPxl8Iysd72v9pNZzEl tlhF6TLvUDoof8nM2C3kLGTc5z6ngzp0XX3f/zC+irM0e6tt/p53oaU3YxcicgGrKuC4 9t7w== X-Gm-Message-State: AAQBX9chM8VXRmM6OVk5RdOJroJc5yuh3e3tNb0LKXmJclXucaTbTyaz w2AhoamhpYp79ueTocEUWBAPXDlCPrw= X-Received: by 2002:a05:6000:12c3:b0:2ce:9f35:b645 with SMTP id l3-20020a05600012c300b002ce9f35b645mr19081107wrx.20.1680188074358; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 02/11] io_uring/rsrc: keep cached refs per node Date: Thu, 30 Mar 2023 15:53:20 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809617795839955?= X-GMAIL-MSGID: =?utf-8?q?1761809617795839955?= We cache refs of the current node (i.e. ctx->rsrc_node) in ctx->rsrc_cached_refs. We'll be moving away from atomics, so move the cached refs in struct io_rsrc_node for now. It's a prep patch and shouldn't change anything in practise. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 - io_uring/rsrc.c | 15 +++++++++------ io_uring/rsrc.h | 16 +++++++++------- 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 561fa421c453..a0a5b5964d3a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -240,7 +240,6 @@ struct io_ring_ctx { * uring_lock, and updated through io_uring_register(2) */ struct io_rsrc_node *rsrc_node; - int rsrc_cached_refs; atomic_t cancel_seq; struct io_file_table file_table; unsigned nr_user_files; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f2da9e251e3f..1e7c960737fd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -36,9 +36,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, void io_rsrc_refs_drop(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { - if (ctx->rsrc_cached_refs) { - io_rsrc_put_node(ctx->rsrc_node, ctx->rsrc_cached_refs); - ctx->rsrc_cached_refs = 0; + struct io_rsrc_node *node = ctx->rsrc_node; + + if (node && node->cached_refs) { + io_rsrc_put_node(node, node->cached_refs); + node->cached_refs = 0; } } @@ -151,11 +153,11 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx) +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) __must_hold(&ctx->uring_lock) { - ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); + node->cached_refs += IO_RSRC_REF_BATCH; + refcount_add(IO_RSRC_REF_BATCH, &node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -300,6 +302,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; + ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 1467b31843bc..950535e2b9f4 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -43,6 +43,7 @@ struct io_rsrc_node { struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; + int cached_refs; }; struct io_mapped_ubuf { @@ -56,7 +57,7 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx); +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); void io_rsrc_refs_drop(struct io_ring_ctx *ctx); @@ -128,17 +129,18 @@ static inline void io_req_put_rsrc_locked(struct io_kiocb *req, if (node) { if (node == ctx->rsrc_node) - ctx->rsrc_cached_refs++; + node->cached_refs++; else io_rsrc_put_node(node, 1); } } -static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx) +static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, + struct io_rsrc_node *node) { - ctx->rsrc_cached_refs--; - if (unlikely(ctx->rsrc_cached_refs < 0)) - io_rsrc_refs_refill(ctx); + node->cached_refs--; + if (unlikely(node->cached_refs < 0)) + io_rsrc_refs_refill(ctx, node); } static inline void io_req_set_rsrc_node(struct io_kiocb *req, @@ -151,7 +153,7 @@ static inline void io_req_set_rsrc_node(struct io_kiocb *req, lockdep_assert_held(&ctx->uring_lock); req->rsrc_node = ctx->rsrc_node; - io_charge_rsrc_node(ctx); + io_charge_rsrc_node(ctx, ctx->rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } From patchwork Thu Mar 30 14:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77224 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1186070vqo; Thu, 30 Mar 2023 07:56:02 -0700 (PDT) X-Google-Smtp-Source: AKy350Z4TiWTvku39FprF2rYkDZ/9QrhKuMJlyz9ongTdOrVAqxDwAaq1obcDoXVDPeek0gEcnNm X-Received: by 2002:a05:6402:415:b0:500:46f2:e7db with SMTP id q21-20020a056402041500b0050046f2e7dbmr23378452edv.15.1680188162073; Thu, 30 Mar 2023 07:56:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680188162; cv=none; d=google.com; s=arc-20160816; b=E8J8qCNmxTBd8cv1dBQJ8znoNeDbWRedVvkD1QGcCWL6YNqi/QJDljJHzaJdW9aew/ VLZFby8IAi1tsR4OKw2fRP0ynSQys6COjr69nNF9Gnk7m9RoVz+EBy5ntGUTPrL9j6ce CmNYlGRRYf+roguheG3n0SmbZMoGl7bNTQHuzHw+vDCQCt1BK8bfQ/4SbSaSOmpF/430 VgNBZY6581tB7gNBTWiuEzMso3pe1gPXKA3HUrwW9QwvuI2nd+V1gRkY+0C3yL+up9I5 0uyhw4jHDiFyMiUcDJSf5QdZF0lVInwPG2jZyqizpTzAWZr5L8L36J8iNFP1aQu/ilpZ EOOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WWAczytuFz9lGZfCu/Rbuh88s55sTzs57HoEPIGs0GU=; b=FlR2AIp56V6ZOZiOqVh+9c8GMDbbmB8c5G8RiCauRzsDOXSWJaN4cZ8goK1lF0Jis2 waTTmXpTS7gJWG96mrJ6R3ifDDRr6IeWAJ27DP0ijCrQtDCPTeZYb5Jn0Q/wpWnaEk+U Wtv0GapRIVTUlErIHgyxOCyIaa1gUnJctMljzXDnIrxS02j7A9d8Z2UC02m6C8UnVQtk sklHac3OOgSyVlJNMpoi4tSksK0YO2mrnmhp3yX8kqGAgPYQ5nVupcBEhz9mVzElhCDc DLJ80lZwwoCUfFufvTODThpxVdV/NFE2y6Gtv/Ukn2m2C+6uCbOQ6rTWRSGADa420AQv NR/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZPQgYFuW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b17-20020aa7df91000000b004ac8358be51si32776552edy.464.2023.03.30.07.55.37; Thu, 30 Mar 2023 07:56:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZPQgYFuW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232570AbjC3Oyt (ORCPT + 99 others); Thu, 30 Mar 2023 10:54:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232539AbjC3Oyi (ORCPT ); Thu, 30 Mar 2023 10:54:38 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A01BA5D8; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id m6-20020a05600c3b0600b003ee6e324b19so11974552wms.1; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WWAczytuFz9lGZfCu/Rbuh88s55sTzs57HoEPIGs0GU=; b=ZPQgYFuWcz1SvJblTrxSJFLx5OAkM/KL/BYFd0RSfa/H2AiXu8SDOQ5j5v3iJprcop Jx9t48yuY9un4JFMIBZ0Tzqb/jVGOO6oTYOOE87IYBLtZrXfvYaxbD9Fa+ErOFN9ROHZ uEMnKIRmPXJoSZRA4y1uHnQeIGVXv7ebHVolEBgDmcM91gxa9mS4vxV+dLlkfDHiGEUq hrmrqXUi5RXmuK+b/ctz8UsjGEg9kZ6N18KJ9wFAb3LF5mxKTohvByLBmHIrGkRq3Oq7 BDvCISHhLESl1mCXuFCwFrNwydXnxpXpXJZ5vvnRAl9xhNkSijs3wYq93FgJUOh+MifF ecaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WWAczytuFz9lGZfCu/Rbuh88s55sTzs57HoEPIGs0GU=; b=zBUikyicM/zeIOYvvcF0FJG1rvDA7Xk2osdcEii+KEYCigA6MaeqrJ55SP3+R/sHuS JRgte4vX/mIgB4QkfZaoVwlpX2BhH7z9srWx9J0bIuoD7lkDZ/ItccSEkCA/twjw96hy Nx2nCOvchcjaoA2Bi2X6GRITHdNd9DsXk8UuhDXQmOaVJ+LdbMhpA5U26Dbev3I2PA0F Eo3na+FQgsM52MBQsQ2nRqtwzBrXMpsaJjLmlKhAoixN/dHhdBK5K01Y0FdFM2ELZeH4 LDC57UooIJ3DvKlFzYtJj0zTs5rvpFFNuaaZJW2p11lg+OePSEok2g6YQOjOyOOBboiW 9PvQ== X-Gm-Message-State: AO0yUKXOSbJ1vZxlm6P6SCWtO+UsJeOQpk0t69Gm6YqaoZyy0h7MdHwA Rausc7zmykOLp+pRRrpM2z0YpAu/DBo= X-Received: by 2002:a1c:7702:0:b0:3ed:4f7d:f6ee with SMTP id t2-20020a1c7702000000b003ed4f7df6eemr18359942wmi.14.1680188075498; Thu, 30 Mar 2023 07:54:35 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:35 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 03/11] io_uring: don't put nodes under spinlocks Date: Thu, 30 Mar 2023 15:53:21 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761804982438914943?= X-GMAIL-MSGID: =?utf-8?q?1761804982438914943?= io_req_put_rsrc() doesn't need any locking, so move it out of a spinlock section in __io_req_complete_post() and adjust helpers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 +++++-- io_uring/rsrc.h | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 536940675c67..c944833099a6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -970,6 +970,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 static void __io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; + struct io_rsrc_node *rsrc_node = NULL; io_cq_lock(ctx); if (!(req->flags & REQ_F_CQE_SKIP)) @@ -990,7 +991,7 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_put_kbuf_comp(req); io_dismantle_req(req); - io_req_put_rsrc(req); + rsrc_node = req->rsrc_node; /* * Selected buffer deallocation in io_clean_op() assumes that * we don't hold ->completion_lock. Clean them here to avoid @@ -1001,6 +1002,8 @@ static void __io_req_complete_post(struct io_kiocb *req) ctx->locked_free_nr++; } io_cq_unlock_post(ctx); + + io_put_rsrc_node(rsrc_node); } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1117,7 +1120,7 @@ __cold void io_free_req(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; - io_req_put_rsrc(req); + io_put_rsrc_node(req->rsrc_node); io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 950535e2b9f4..8164777279ba 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -115,10 +115,10 @@ static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) io_rsrc_node_ref_zero(node); } -static inline void io_req_put_rsrc(struct io_kiocb *req) +static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (req->rsrc_node) - io_rsrc_put_node(req->rsrc_node, 1); + if (node) + io_rsrc_put_node(node, 1); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, From patchwork Thu Mar 30 14:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77248 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1238598vqo; Thu, 30 Mar 2023 09:08:04 -0700 (PDT) X-Google-Smtp-Source: AKy350aePfxvLYGohUMdPCHb+7p8RLCYO92QTPgOCLygdk7eglCC1VF+AWws2gNIjhY232bIkWBs X-Received: by 2002:a17:902:e80c:b0:19b:afb:b92e with SMTP id u12-20020a170902e80c00b0019b0afbb92emr31262322plg.40.1680192484449; Thu, 30 Mar 2023 09:08:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192484; cv=none; d=google.com; s=arc-20160816; b=as786KqISYWZQxov1sj0CBHKrGisNb2vjdZh9jIghnZyZ9RjaunEMA3k2RzDunwxXx 3yIv8nAJ4XsBkJ0bnUYQSgLaJCqJ/f5cPJORY91F3CclcMTFd5J/jIEWronrx+IZcudd 2Ovmp/iA7xki+FqEP+9yMJzwtqx8Uy7HuB2CpjxL7QnUanJbhkkkS5TQwNTAL8p5zjB1 LTK60rdHCc4mO39a3JJxzBXbcV46aSD0Axgxp5tn0jqk3lucJNxfkpUohFookbTL98qK BIUgbIgNm70YgxgbNYKf9LewI8/0RD1oh41kpABBrOtIDZrMwryik4FtZbTMZqyOsAtQ jFyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eKfMO97Puxbxg9b5V2zEzriSKSgWDMsPXMYWqPT9qnU=; b=prQKd/pdFg1ygMFDKIr5wtpYjeQ50blNOmmiU7RpOJS8es+igK/VUPacYiQOTp1Mhq UQxmz+a7EOG6PR2Vatc3AjOhT2qJNE3iTq2Ycfq8HJv07RNDMxePVJrZjLZEBCYl214+ HckQLRNnkJB1DVo6DCn5s+JCWc8/6K4eDGfAKAr0auvwceNOA4FJ1WjahiJXv7/sWnkJ onmUc74mV0XiQPt7yPK8p5ZRciEbn5ih7TfqO94wUliyTnuB2q8jJATabDL5TuMsiznw G87LzegV95IyoVFwBIV+Km65aZmWD6ZPBr3KeqdlQr8QZUe+N3Kfuvp1Ofc8iD/WFRt8 N+NA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=NY7Rm6tR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ix7-20020a170902f80700b0019e3ffa255asi33565224plb.184.2023.03.30.09.07.37; Thu, 30 Mar 2023 09:08:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=NY7Rm6tR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232613AbjC3Oyw (ORCPT + 99 others); Thu, 30 Mar 2023 10:54:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232517AbjC3Oyl (ORCPT ); Thu, 30 Mar 2023 10:54:41 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43F5E469E; Thu, 30 Mar 2023 07:54:39 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id l12so19362276wrm.10; Thu, 30 Mar 2023 07:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eKfMO97Puxbxg9b5V2zEzriSKSgWDMsPXMYWqPT9qnU=; b=NY7Rm6tRHFZ76SKQjHQdrDnj9n2jWzPmsbsXRr5sKH2eH3gRHQlQaR4Xoy5egkkK3c EGB/hzWp1i/3AzduUWZ0JfogkFiG/lN3KpUiFOWNOj0HuAABld2CwxHAYYR91+cifYln lLkPPpPcxJcB/9hItQ8keDXeLXji7HIwIybnBFNggrZtCIS5IwaZYLyw+ykua7UJmNwc jumPip94qNG8lZOkxUUzU8YTEBlEIkRpRgMXlhTiDkvZUBuZ4Vy/AsMWi4vfnLGJn7+Y of2tdDBRpfALUs8LZOZoKBAM22ckJDR5ThRkc3GrCWEjvxS0GdqOQzIw72A6U6d2B1cS xdQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eKfMO97Puxbxg9b5V2zEzriSKSgWDMsPXMYWqPT9qnU=; b=rnOztK7g0qXOxoferby0TlKXPJE0sR8f3Vf9pWOMbMgoaM7GAZOt+m5j2vRU+ImoeH Rgl0wKkBN7/HHitMojaD1+MbZ9QXNaDvRBaaEsTYETz0vmpynM3glevcuFRAWAqFxB4a NaesG+542U+xeRDtIYY3pBSttkQNW+WegNEL2kfloxkzkBiP533cPTtMdKN9DGE9+dck QQ0mPuUHRiV7vDU3gIhDqSKQ+G87Lm5FPyLIDNVQht36M15bb9PEFKSo2HpO64x7Y0j4 5qSwOMBsl25mTIEbscG9MEEYJZGqT5hI37a6y+fvo3Ty0PwzZCyMPNAzRV8XENJRk4Uq bYjA== X-Gm-Message-State: AAQBX9ecNOoehB6Rh6UIDnDjU+HRWCUqGXI8TyLDIF/0sy51yFCz4qib tmFSvpk0OrpqLIFEn5ztbxevWUxComM= X-Received: by 2002:a5d:500a:0:b0:2cb:29eb:a35e with SMTP id e10-20020a5d500a000000b002cb29eba35emr5127532wrt.11.1680188077486; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 04/11] io_uring: io_free_req() via tw Date: Thu, 30 Mar 2023 15:53:22 +0100 Message-Id: <6b0c640860a6250abf883840b160ccd50fd5f311.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809514578998246?= X-GMAIL-MSGID: =?utf-8?q?1761809514578998246?= io_free_req() is not often used but nevertheless problematic as there is no way to know the current context, it may be used from the submission path or even by an irq handler. Push it to a fresh context using task_work. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c944833099a6..52a88da65f57 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1116,7 +1116,7 @@ static inline void io_dismantle_req(struct io_kiocb *req) io_put_file(req->file); } -__cold void io_free_req(struct io_kiocb *req) +__cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; @@ -1130,6 +1130,12 @@ __cold void io_free_req(struct io_kiocb *req) spin_unlock(&ctx->completion_lock); } +__cold void io_free_req(struct io_kiocb *req) +{ + req->io_task_work.func = io_free_req_tw; + io_req_task_work_add(req); +} + static void __io_req_find_next_prep(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; From patchwork Thu Mar 30 14:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77246 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1234503vqo; Thu, 30 Mar 2023 09:03:55 -0700 (PDT) X-Google-Smtp-Source: AKy350ZVUqCFRUv7dKDKf446S5ZrG0uGbvn6jZVye2cm/zVIWiYsTiGRVyA+CE9HPeCG4Lo2VV1Y X-Received: by 2002:a17:906:5a62:b0:932:5f7d:db33 with SMTP id my34-20020a1709065a6200b009325f7ddb33mr22758395ejc.34.1680192235504; Thu, 30 Mar 2023 09:03:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192235; cv=none; d=google.com; s=arc-20160816; b=ngc554iep5DnOPk3EVKrfyszuhaU7rykKSkiZEqyVCNFTZ8p9xM7UfiiSNNREmtodb b+2TTKM4yvOYsqllgP/DJIy8ipNgWPz7liX0Mz4tyTWMe2mkL5InGPqDl5JgWnoQxuhA LBCFiUsxBOFUa+dDIt8RTdSAtOBAXZr1H9bz1LNKfhyLJw/SXNoQQPZ1rms8jdmUZAQk wcXEicICmfASw+RnbkMiUPBraO2xBgzZn5caU8VwEGlAXDZC29tQ+0hiTa0jtJAf9oAE X15hkEt4jSmehU35DQuK/qbQil3HWPWDHqqHaGJHrD5QMpduTQQYLovbaktCSAKWenHO ZlUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0i8xnepg2BvrVcYo9lxg1ps1ZVOTUo6D6qHGZbwYEt0=; b=UNRO2TMezb93u2o8NdiK+T4Nmc5cVTxCDgfFAZihs3vKAUlQsAlA0CpYIr97KI7N38 p4/n1UG+8VY2eWmWzEweRziQ08Xm/3dWZKsrC9GtwC3qP5dmSPqX4tEYW51mtp+KWfUr B84c6Owy7pgyRL6ax5QeRgp7E0sYTSeVzXC+4IxjmOw5kF1U7RsxbGCfkmbzUre+L8Ko lYGjg/zJTM7sNtqm1jmFnrzxdR1/v5YH+xqcaoP/kWRsP9GcXI0nqVzufsB85tzJQwFq UO9l90HGr7fCxIQO/0ut5Y/OFCHlc3Pvb4jNvW4D10dstMPla8Z4jAf6gUKv2V0mR4Og atdw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=cl2ADzpk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u19-20020a170906109300b0092ce291dd12si135879eju.395.2023.03.30.09.02.50; Thu, 30 Mar 2023 09:03:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=cl2ADzpk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232582AbjC3OzA (ORCPT + 99 others); Thu, 30 Mar 2023 10:55:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbjC3Oyt (ORCPT ); Thu, 30 Mar 2023 10:54:49 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C665FA273; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id j24so19424969wrd.0; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0i8xnepg2BvrVcYo9lxg1ps1ZVOTUo6D6qHGZbwYEt0=; b=cl2ADzpkTY/LFYdmyRhsDqzayDOvTYOGD2jnCQGmZx/Kgyw4F9oopSiAn7OUyNZTJ7 USRzQoUySTxFUnfOZROxaZQku9kXxtLAoJZiF4XFTdiCm7k9xkhfV0Wa/thfKM3o86UG ee1EQqQSWJNTdPcBxtCxl+yfeTZeG0OXhETx/dSmZd80H3IaFb/Cy7TfFvCj14JBY4OK hdcoQ4dllmyuqg86bTKJ3D5DfneriZ03NKkt3XaCbtt0GmgFV7LtZXrkPzSI/W0DSrPK whCN5fxCyTMxC6FKcmgsxf0ejVKAnbEl3S6PNgARe6JSPpaQbSpE5Uaj/BjAxRPjLjAK jxxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0i8xnepg2BvrVcYo9lxg1ps1ZVOTUo6D6qHGZbwYEt0=; b=V8IPxSGx3e/yb8N2rupvqa5G53NttYJ1XY19vZePRNWgY8hdcOZDg46Y3DeRhrvDd8 e7p8HG3/hmSokIwAIag3N5tOAUIKmeSFlxYVB8oP30fjOaZAu8AxSAfoXxSSAZO0my71 fqDMBubaLQ02cGIZ2ASt0tqTQYEmrgn/CAHGvSca6K4j9RLAZIu/wpKQcTCywlt5qgsJ yr2iNtGjl3uWGCh5Pkr8IuXtrfEGw5H3vqmkS5dTTQYvB8PJlKRfYtEqj+9CvlboO35O 8OrdzCzFQzKs4D3o5CykDHMD4OjFBS3huWuxudF1tRyPA7UCSkDfoeh75c0gvIAjUuc/ GfgA== X-Gm-Message-State: AAQBX9dC1EKf7DzNjW9lwD+W33oUQ6Ys07FVpcsYN7nvWmSvOTAh/JdS 8xpT/wgmUnYO+8t0xbF1xYq+GU99qRo= X-Received: by 2002:adf:f185:0:b0:2e4:b4f8:896c with SMTP id h5-20020adff185000000b002e4b4f8896cmr2517301wro.48.1680188080053; Thu, 30 Mar 2023 07:54:40 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 05/11] io_uring/rsrc: protect node refs with uring_lock Date: Thu, 30 Mar 2023 15:53:23 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809253497281352?= X-GMAIL-MSGID: =?utf-8?q?1761809253497281352?= Currently, for nodes we have an atomic counter and some cached (non-atomic) refs protected by uring_lock. Let's put all ref manipulations under uring_lock and get rid of the atomic part. It's free as in all cases we care about we already hold the lock. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 18 ++++++++++++------ io_uring/rsrc.c | 30 ++++-------------------------- io_uring/rsrc.h | 29 +++++------------------------ 3 files changed, 21 insertions(+), 56 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 52a88da65f57..e55c48d91209 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -967,7 +967,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 return true; } -static void __io_req_complete_post(struct io_kiocb *req) +static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *rsrc_node = NULL; @@ -1003,7 +1003,11 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_cq_unlock_post(ctx); - io_put_rsrc_node(rsrc_node); + if (rsrc_node) { + io_ring_submit_lock(ctx, issue_flags); + io_put_rsrc_node(rsrc_node); + io_ring_submit_unlock(ctx, issue_flags); + } } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1013,12 +1017,12 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || !(req->ctx->flags & IORING_SETUP_IOPOLL)) { - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags); } else { struct io_ring_ctx *ctx = req->ctx; mutex_lock(&ctx->uring_lock); - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); mutex_unlock(&ctx->uring_lock); } } @@ -1120,7 +1124,10 @@ __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; - io_put_rsrc_node(req->rsrc_node); + if (req->rsrc_node) { + io_tw_lock(ctx, ts); + io_put_rsrc_node(req->rsrc_node); + } io_dismantle_req(req); io_put_task_remote(req->task, 1); @@ -2791,7 +2798,6 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); - io_rsrc_refs_drop(ctx); /* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ io_wait_rsrc_data(ctx->buf_data); io_wait_rsrc_data(ctx->file_data); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1e7c960737fd..1237fc77c250 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,23 +27,10 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage); -#define IO_RSRC_REF_BATCH 100 - /* only define max */ #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) -void io_rsrc_refs_drop(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_rsrc_node *node = ctx->rsrc_node; - - if (node && node->cached_refs) { - io_rsrc_put_node(node, node->cached_refs); - node->cached_refs = 0; - } -} - int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -153,13 +140,6 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) - __must_hold(&ctx->uring_lock) -{ - node->cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &node->refs); -} - static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; @@ -225,7 +205,8 @@ void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) kfree(ref_node); } -__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) +void io_rsrc_node_ref_zero(struct io_rsrc_node *node) + __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; @@ -269,7 +250,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - refcount_set(&ref_node->refs, 1); + ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -283,8 +264,6 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, WARN_ON_ONCE(!ctx->rsrc_backup_node); WARN_ON_ONCE(data_to_kill && !ctx->rsrc_node); - io_rsrc_refs_drop(ctx); - if (data_to_kill) { struct io_rsrc_node *rsrc_node = ctx->rsrc_node; @@ -295,14 +274,13 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_rsrc_put_node(rsrc_node, 1); + io_put_rsrc_node(rsrc_node); ctx->rsrc_node = NULL; } if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; - ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8164777279ba..fd3bdd30a993 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,13 +37,12 @@ struct io_rsrc_data { }; struct io_rsrc_node { - refcount_t refs; + int refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; - int cached_refs; }; struct io_mapped_ubuf { @@ -57,10 +56,8 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); -void io_rsrc_refs_drop(struct io_ring_ctx *ctx); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); @@ -109,38 +106,22 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) -{ - if (refcount_sub_and_test(nr, &node->refs)) - io_rsrc_node_ref_zero(node); -} - static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (node) - io_rsrc_put_node(node, 1); + if (node && !--node->refs) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) { - struct io_rsrc_node *node = req->rsrc_node; - - if (node) { - if (node == ctx->rsrc_node) - node->cached_refs++; - else - io_rsrc_put_node(node, 1); - } + io_put_rsrc_node(req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - node->cached_refs--; - if (unlikely(node->cached_refs < 0)) - io_rsrc_refs_refill(ctx, node); + node->refs++; } static inline void io_req_set_rsrc_node(struct io_kiocb *req, From patchwork Thu Mar 30 14:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77255 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1240934vqo; Thu, 30 Mar 2023 09:10:46 -0700 (PDT) X-Google-Smtp-Source: AKy350bZf35a7+y/bh15xac4ac+wdMr9OJStlJOK1kTr6db7uMH6JKK+NdcKSsvX3jLCXT1grIAp X-Received: by 2002:a17:90a:4fc2:b0:23b:5155:30a9 with SMTP id q60-20020a17090a4fc200b0023b515530a9mr25593231pjh.40.1680192646531; Thu, 30 Mar 2023 09:10:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192646; cv=none; d=google.com; s=arc-20160816; b=lIXREbT7jV2x1q4D8bXCkiRGUTThTVs7BlWJNIffApIEQY0TRbK3s11dBzVv450YK/ wDaxVEwIfNhiAZa6O/rqUeIWqPRt1++0qQd2/MewDu41wjMaODeECaDjhtLqSPw+QpK8 dD+G9sr8VPmOAHZORSEyHWbq6MH8fTHyIGxjGt2u5uUzBp7eEE89Lkd/tuknXkkz9Sd5 GEVa9fzx5Hwt8tNyDjvMsvnxYbep1ofGPtkCPFyATwX9bPpWYfcNR436XRb7pbkEL1Lk nf2/fY63CpUwoKd/wG226ycCuB/KhlaD+wt5v8eP13d+5i27NcQGT6AjynVeqBF9B1oo iFUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MBkviYTvQX5do4saoK2KG74OjTaM9gN+C198bVZ0XcA=; b=u1rjovGOw0p5F9cuCYcOpcyN7YyxJzN4DuwBOOjc9zE+sceltbnS6fxpYAJXOMH62A G20G36YbqWeDDtAwse/aVmumO0I9Lnwn259iWpP3c6dKvzEtRf5/7LkJ2BOajZUV+i6S kK4k+q+BCzUv2V0h+7Q9uzDV+QzIE5aIZ/9iqQhyze2ZUN0jRCPCDKhJB9Mk1+KTPMpz 76uVLvGPKSFkU1jM8jrC/dUNWllrddPyZ5uYLhiutPjyN6zbmwCwaJnTVgIy4gECtU4+ aWEkfORfM5yW4+7/A4xgXWq25DoexPMLsGu+0+j8Pl1OFjXMx+h561a6zzQFzfmuO+nA hAfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=WSAv013m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b1-20020a17090a6e0100b00233e76fe27fsi4259187pjk.183.2023.03.30.09.10.32; Thu, 30 Mar 2023 09:10:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=WSAv013m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232649AbjC3O4z (ORCPT + 99 others); Thu, 30 Mar 2023 10:56:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232558AbjC3Oyt (ORCPT ); Thu, 30 Mar 2023 10:54:49 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C73B0BB9C; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id v1so19392652wrv.1; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MBkviYTvQX5do4saoK2KG74OjTaM9gN+C198bVZ0XcA=; b=WSAv013ms1/pzIyLkw4lGdHYzlytBUueBGnxq6JYDswxuiARTBXPhRIzJsx4xnCirS BrpknhFxgFsE0H934q/yLcThuaSk/4aeQzNY0hAhrsb2ypDepJWq5GY+/wzLnL1ClZm0 0xFdYpHaUR4wliz/kwKKHYVIG1rSwxeqLZvMqALYupXguyvhu7mG6JzhaKKWMhqndtPV 2PjRLKYSW7SUYwbQYioG+TZykNnTM1BEnf3KXpDF5iCULV/gnj/NZRIljXHOGXYM3J2q Mh8OX2Yzg1pQOLdanyhPELplfN7+8S6v48lq4AjvHAN5dP75Fgw5ObTHUStaekbadlOS RZ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MBkviYTvQX5do4saoK2KG74OjTaM9gN+C198bVZ0XcA=; b=e5hA1kSikkRTU+dQFFvvmAOO2Dzk11V2EJXhHSx27SgODyqsE9VAJ9UQFUuNM5/ONc 5EPn51WvvgA6mLlbeaiCiNkf5ygubBMZK3fE1l+jL2Bj08R8BC79pyBg2X4olKY4ldz1 JOVv/YOZ7qhwpG4niat+OhW5a08RympMJGQny06na+Odxm9hpWFdg+SgxNrSV2VM2+TV LCgULuqQUGsyxVDCv5xt9KMKLijdWSzrODT0kGG+iTrSRwP/DuSVaUrVRMGfCisF8L0C z1w9TPy3R3dFkorIlTizsmWXv9IdV4EvAbDmLiQG9jKAx1P0ubWrROS8MrcR/spWJ54+ oMwA== X-Gm-Message-State: AAQBX9elmaqalEb+XrZ9VAL83ekD989AqtMwEOsUMCKQjttCzco3WMLQ W8kDLJv6trRTp7IIQlS/K5zSYbsgFBA= X-Received: by 2002:a5d:404c:0:b0:2e5:1ee3:df77 with SMTP id w12-20020a5d404c000000b002e51ee3df77mr1142541wrp.46.1680188081160; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 06/11] io_uring/rsrc: kill rsrc_ref_lock Date: Thu, 30 Mar 2023 15:53:24 +0100 Message-Id: <80e76722fb1317d5315528b8b57de559ca5aea33.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809684607488001?= X-GMAIL-MSGID: =?utf-8?q?1761809684607488001?= We use ->rsrc_ref_lock spinlock to protect ->rsrc_ref_list in io_rsrc_node_ref_zero(). Now we removed pcpu refcounting, which means io_rsrc_node_ref_zero() is not executed from the irq context as an RCU callback anymore, and we also put it under ->uring_lock. io_rsrc_node_switch(), which queues up nodes into the list, is also protected by ->uring_lock, so we can safely get rid of ->rsrc_ref_lock. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 1 - io_uring/rsrc.c | 5 ----- 3 files changed, 1 insertion(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index a0a5b5964d3a..9492889f00c0 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -333,8 +333,8 @@ struct io_ring_ctx { struct delayed_work rsrc_put_work; struct callback_head rsrc_put_tw; struct llist_head rsrc_put_llist; + /* protected by ->uring_lock */ struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e55c48d91209..e94780c0a024 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -325,7 +325,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->defer_list); INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); - spin_lock_init(&ctx->rsrc_ref_lock); INIT_LIST_HEAD(&ctx->rsrc_ref_list); INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1237fc77c250..e122b6e5f9c5 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -209,11 +209,9 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - unsigned long flags; bool first_add = false; unsigned long delay = HZ; - spin_lock_irqsave(&ctx->rsrc_ref_lock, flags); node->done = true; /* if we are mid-quiesce then do not delay */ @@ -229,7 +227,6 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) list_del(&node->node); first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); } - spin_unlock_irqrestore(&ctx->rsrc_ref_lock, flags); if (!first_add) return; @@ -268,9 +265,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_node *rsrc_node = ctx->rsrc_node; rsrc_node->rsrc_data = data_to_kill; - spin_lock_irq(&ctx->rsrc_ref_lock); list_add_tail(&rsrc_node->node, &ctx->rsrc_ref_list); - spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); /* put master ref */ From patchwork Thu Mar 30 14:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1186432vqo; Thu, 30 Mar 2023 07:56:40 -0700 (PDT) X-Google-Smtp-Source: AKy350apnJTJN4nD4TWkUEMePZJf5wzggdA+G4e9CzHVs1MZcAgaU2L4M41bzZGDurLJ77QLDr5I X-Received: by 2002:a17:907:7b9e:b0:8b1:fc:b06d with SMTP id ne30-20020a1709077b9e00b008b100fcb06dmr27318364ejc.77.1680188200536; Thu, 30 Mar 2023 07:56:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680188200; cv=none; d=google.com; s=arc-20160816; b=MojBMiZYq2OuKU92FOaxVRZaMVddgJyzC0GkWom13+EpOtuxmI/s79e5PCJ2Z1IME+ QkxQn549Et1QYJuz5n5h3wTsljP2peYazzo0IMyhaB+UF2WKzh3WwkqEAWlmri+q/LFl gNAYdLk22zrCkeXbyiC/hS8mc0L0g0SXVz0M9yaFquJG4RfLEqIcdune49lfDXZVisll vHaTagd174EGprViE7vpRrjPv3+Iw49Gr9ITz7VkEaXKaLA2VMya/+uvyWYF0yrbFGEN tczRhQUkIJ1Z9geVmhvlJBe0qIJo0U5Kv+thmURssTI0fkuH9aZU9vr9zBs02AMqEPIZ NTlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mFbJzcJB80Re1Q+sgkx/0FejbOmENveKGMdPIGHHusQ=; b=LfXt1GLxxrEs3AHoZcPRhOyvvT2F5E8hQu7c6MN1+Z5GqtSNXAjGHCtXr3q5O/hjNj 2AAAgTXLnF2xOAlcxkKe4f0TdIXAZhl0EJXyyzvpkY6rDzeMf0wyi6Hk7oH7b+tg4hd2 eTyTOk6c5RvuOx+hge0NTgUbFkINkJHmJb4ZHuZ9aT5Px4ckOxrM8O7DEifw6Q4iOGua b2EfvmViB4UmRcSkLOlXFwTop0i2utBSYDEc9KR8Ir8zjR+Z6S5Yu1l1w/RlKIP9KopE RHeMl5ODm/IpuJQXPI4FtKsCo6V5mxiW7w/xiN5Px8tmgFzLEAckw0q+U8XX3Wv7gMV0 NxIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=fAaanC13; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i20-20020a170906115400b008b17cc2e38asi295226eja.582.2023.03.30.07.56.16; Thu, 30 Mar 2023 07:56:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=fAaanC13; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232556AbjC3Oy5 (ORCPT + 99 others); Thu, 30 Mar 2023 10:54:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232431AbjC3Oyu (ORCPT ); Thu, 30 Mar 2023 10:54:50 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3054DBDDC; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id r11so19373800wrr.12; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mFbJzcJB80Re1Q+sgkx/0FejbOmENveKGMdPIGHHusQ=; b=fAaanC13ZIcJPOqlgTHTTR8OQ622D5DOKgBDq8AivQ6OZol52ff75ivXCbEWtrKlC5 58sKNg8ei9TLxFzneeQcX+weXD5pVTkx5Una8QbUQ0N0zBSz4/9zQLBi2nTYeg17d/07 BDhvnc5atjth6fl/hQ5ySvlGOQ4T/zeoOqc1+3smISDFKVYbnru8vA3kiUons8RW7x/9 9BBOsInbG1nNwAtXQR4YCUNxAWpzV6a4ttCV3W5sudKKKy/FayqZQN9Pt/AsjOb5bby8 VnWKU1P6pGUsEfhtkSI0HXYIv9yMTfVnkL7D6zLr5YFZHIhgMR7qyMLLX2+qBWF9JQkJ jaZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mFbJzcJB80Re1Q+sgkx/0FejbOmENveKGMdPIGHHusQ=; b=7pdXaNG7xaXlQoYQxmzQdHSSeilQEnirizGCc5mSCujOHfKjUwniSq1Pf/gjuNOS5L GDW63J88pzh0TvN2etLkbFWD5qU52vkrANFpkX2kJLrkjH7PmMKNQaGxIl02ccwgpeaS KDsXSsAUc2P+L0vDfG1RKUFNK5VB/pXiDDvXFNNB9+PYCv1Kdi2iKo360wWzU8uscMhl NVT3gV/h3hsxSTzpywLrx5+6U/CyjURJ1nTLsq0Imrdteaqed5I2vdJt+6rv3Rs/53Av FtODsPjjD1qNaFahqIzWIYiI7L8al3AJrH5SKisVQbUGF4baF3qrfJhSEkfL+UlirgmN MTkQ== X-Gm-Message-State: AAQBX9e/w8p5SKdUhVIRKpspyG4DExUS/oCHWPpPxUm3gQs13yMuz3Lo uGlkt6hsqkpGzexBLc1pO3wzXX5gXiY= X-Received: by 2002:a5d:53c8:0:b0:2dd:2a04:b73f with SMTP id a8-20020a5d53c8000000b002dd2a04b73fmr15772746wrw.49.1680188082258; Thu, 30 Mar 2023 07:54:42 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 07/11] io_uring/rsrc: rename rsrc_list Date: Thu, 30 Mar 2023 15:53:25 +0100 Message-Id: <00684614f29297882e2c03456351532cf027e167.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761805022892311906?= X-GMAIL-MSGID: =?utf-8?q?1761805022892311906?= We have too many "rsrc" around which makes the name of struct io_rsrc_node::rsrc_list confusing. The field is responsible for keeping a list of files or buffers, so call it item_list and add comments around. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 6 +++--- io_uring/rsrc.h | 8 +++++++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index e122b6e5f9c5..10006fb169d2 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -146,7 +146,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; - list_for_each_entry_safe(prsrc, tmp, &ref_node->rsrc_list, list) { + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); if (prsrc->tag) { @@ -249,7 +249,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); - INIT_LIST_HEAD(&ref_node->rsrc_list); + INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; return ref_node; } @@ -737,7 +737,7 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->rsrc_list); + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index fd3bdd30a993..c68846de031f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -39,10 +39,16 @@ struct io_rsrc_data { struct io_rsrc_node { int refs; struct list_head node; - struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; + + /* + * Keeps a list of struct io_rsrc_put to be completed. Each entry + * represents one rsrc (e.g. file or buffer), but all of them should've + * came from the same table and so are of the same type. + */ + struct list_head item_list; }; struct io_mapped_ubuf { From patchwork Thu Mar 30 14:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77240 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1231157vqo; Thu, 30 Mar 2023 09:00:40 -0700 (PDT) X-Google-Smtp-Source: AKy350aGwsMtNHSwwIvTQV19IfdHmJuhLO6u5WN3HYzJQlqwSsJrAw5daT1VVGPDP49q0ok6almH X-Received: by 2002:aa7:d44c:0:b0:502:52dd:57a2 with SMTP id q12-20020aa7d44c000000b0050252dd57a2mr8769981edr.24.1680192040673; Thu, 30 Mar 2023 09:00:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192040; cv=none; d=google.com; s=arc-20160816; b=NRbyWk92/YjuK6PMDphNyOgNDV/27vezttEpdUj4mMg+xFljnowzRv+2Oqo2e0sUyf 5J3AVDqq5xf2ywCLN6lXMNnI4AxMwxmC7Rmc8dlY69/InoV+ZCUGeVre9pMsx+pucncf A+r3YqDHZZl7OlfIkFCbPWxLvYI+D9H98c5rqIakWlII3W4fzJQZQtAhICKm6ghMMrZV TGw9KZk63uzXlyH2TKxeoxOYWTgNqNuni+4VlnQsWNTVAtnoDW2i+HL5ULFBdJc1QZbQ /8iKqXFukX0bNeZa6Rwx02ShHMtp62tRX7G8BtV47vgmaky1KmBuAP9swUrjtwvrXbVo UUxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=h5UqEuOCJJfrll2Bn9RfMOzzWqUksKYu3hj36bqt65g=; b=yxdBMm8NDMvtkZMz/sq2UJM89Rh7NnAIiYZQ7jXhxHrm8iNzzXLLn/APg2Z8raWM/s Opo7sewoGjkUbLzByZfUQTbAm1udts0RaXn2vbEdfEE+LRf/0aNjXCbViSgbnwVhoBSN Yj7/U4cpQ8a+kIU4a1TvZc/TaTXAEcZ1BEXAVT2UMHAuwiyWlC2TSGbNmwfQhOTI9DwB vkVTJU3AGIZjDMj7UBFXTjtyrcQAQgqppJVGDspfhxSeSW2ad/dbvqftbhsjDZqCI+4J iq96fHmaQ7VfMi7dAUmIsIu25vUhLojNEW2vrzS3FY7GOdanzZxUI/xUWVZsDL1bNwI2 36bQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZV3O1gZv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ba29-20020a0564021add00b004fd2a8b4962si113227edb.132.2023.03.30.09.00.15; Thu, 30 Mar 2023 09:00:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZV3O1gZv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232599AbjC3OzD (ORCPT + 99 others); Thu, 30 Mar 2023 10:55:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232598AbjC3Oyv (ORCPT ); Thu, 30 Mar 2023 10:54:51 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE967CA33; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id v1so19392779wrv.1; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h5UqEuOCJJfrll2Bn9RfMOzzWqUksKYu3hj36bqt65g=; b=ZV3O1gZvzbcSV9N01nRjxEXlT1bE5bVrzuzGPkbJEMK3lyPmu8mkDZhDpCBqhkg0cR zdmioPHKXXP7kioj47w4vHRX96j5rdpk16up0OUTOvY2RCZjdHkaNzEWWIJF6ARNg82i YAw6u1a14YutSOwyhHtBqZSgX5fxgNcY31F1AZTVVPObhcJrRAzpw37Dd2s4JL05NlDM KQiSUc2kTw/LAP69o2xXedgkAQzeYVDRC4BfCcGi5+n8bmzrTT4OI+Xf/nZp1Rw6RCXh YrMKkwiuOU7BCd/MHcqtn1iT6Yq7hGSHRF0clJJhQ47vxc5rwx7aVSJMqEFlanWQg0JX r+YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h5UqEuOCJJfrll2Bn9RfMOzzWqUksKYu3hj36bqt65g=; b=rDVHcY8pTewPfeYwCp0x4IY7n+AXmSK2LfiqBiZmA4R7IOVjE+IjUp2JXnLP+F1ocD LiRFjZnxHt9ZOcw6qGgZyt4RoHgQzwlOIY79AbrW6HFu5gmPumIG5qrwUhyZXJSqKkPc Q40On2B6Ry6ytikQkMQy8zglir2EJxZVdDj/MQEy73hB+Thg8USPFUzgRUrIkvb4Ecup gY0xBhsXLmUdArcNi8vVQu8lVRuWxdZV0krI03a+Ms2f4O4Bi4lIJuZM4C7ZX5JZBtYz NFAlgpQ9qxSnxli8Un26aoTT+y9Y/558qrgN2GUVePf+WkPIJtczYuPHSflySEQIWnD0 +xiA== X-Gm-Message-State: AAQBX9dIkC2bNbGZkZoq5bOyyp7etffrO5rR+/FyQ5xa0rvGu0SG2vQW IKYjf8H0Bu9wims8vUJ6eHtMsABP7mQ= X-Received: by 2002:adf:e552:0:b0:2d6:8d2d:5a7c with SMTP id z18-20020adfe552000000b002d68d2d5a7cmr18407231wrm.57.1680188083411; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 08/11] io_uring/rsrc: optimise io_rsrc_put allocation Date: Thu, 30 Mar 2023 15:53:26 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809049299344848?= X-GMAIL-MSGID: =?utf-8?q?1761809049299344848?= Every io_rsrc_node keeps a list of items to put, and all entries are kmalloc()'ed. However, it's quite often to queue up only one entry per node, so let's add an inline entry there to avoid extra allocations. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 51 ++++++++++++++++++++++++++++++++----------------- io_uring/rsrc.h | 2 ++ 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 10006fb169d2..95e71300bb35 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,26 +140,34 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } +static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, + struct io_rsrc_put *prsrc) +{ + struct io_ring_ctx *ctx = rsrc_data->ctx; + + if (prsrc->tag) { + if (ctx->flags & IORING_SETUP_IOPOLL) { + mutex_lock(&ctx->uring_lock); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + mutex_unlock(&ctx->uring_lock); + } else { + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + } + } + rsrc_data->do_put(ctx, prsrc); +} + static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; + if (ref_node->inline_items) + io_rsrc_put_work_one(rsrc_data, &ref_node->item); + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); - - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } - - rsrc_data->do_put(ctx, prsrc); + io_rsrc_put_work_one(rsrc_data, prsrc); kfree(prsrc); } @@ -251,6 +259,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; + ref_node->inline_items = 0; return ref_node; } @@ -729,15 +738,23 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, { u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; + bool inline_item = true; - prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); - if (!prsrc) - return -ENOMEM; + if (!node->inline_items) { + prsrc = &node->item; + node->inline_items++; + } else { + prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); + if (!prsrc) + return -ENOMEM; + inline_item = false; + } prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->item_list); + if (!inline_item) + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c68846de031f..17293ab90f64 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -49,6 +49,8 @@ struct io_rsrc_node { * came from the same table and so are of the same type. */ struct list_head item_list; + struct io_rsrc_put item; + int inline_items; }; struct io_mapped_ubuf { From patchwork Thu Mar 30 14:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77263 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1242925vqo; Thu, 30 Mar 2023 09:13:09 -0700 (PDT) X-Google-Smtp-Source: AKy350ZYZesgZrcACWCyUM/1X1ACGeV0jtxhSy3+NzVwccKo214NuUm6kbTkU2KCxZxUU9V0+44t X-Received: by 2002:a17:902:fb8b:b0:1a1:7899:f001 with SMTP id lg11-20020a170902fb8b00b001a17899f001mr21958138plb.42.1680192789684; Thu, 30 Mar 2023 09:13:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192789; cv=none; d=google.com; s=arc-20160816; b=VBkU23sxjkZFcxdeo2n+c5hXSZeVYtP/Qs7v5i4IJSBMxBYUHjjr/CKRIjENF/qXPl CAIngkdrVz278b+v/fpgj/ncMIsopfRYCf1eXYLwwn0yvzuhf3U5rEYJaaG0fK0TAa7o a64218XYp1WMwf4VwebHCEaUVvTK00gqFrgg/OSI6ZrRRnXqFKLwDJStbMoZRVvCMB7A UKmoIGpV/MngjjvgOyHqqo//6yETFq2pWSMo+uc9hU5V3VtMfbnAa7sz/mYckxDbHzZ6 wjO9ljWA79TXQHxQdqgDOyWZ/VNLEuAMccdgmiQVzd0dbdGC9cjClLU5smpX5ap+Hckf +Guw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TsovVhZknI6qeDE8kTU6QPdUJtFC7EMxs3/MNa3hVfA=; b=XgoAh66MFqTZd+7/SSPLqUkVHtEnJay+Q/DhnoR3SW9WWt5mC7liDroIZqwtboNKcy LeERafOhspkRUcwWeYbqwWDIkohTaDSBQO9DshF8y/ZZ5173X1yzMB40Qnuypp2MgVrf mTDaBxYjlAsVJtJtQOlUF22EpOVOsCAUN89Kwfs15wnsC9z5XM9xIue4IDHcXfeSG8mc 9rY/R5d44CL+KmVPtatP5S7qoZeGfFwaczw8kk8C8zjIrUURAjh8Fw+0xfVD7rRX9NAe 5+9Fa8OOoAKxaSKUaY0Ut/d9AfbV84X6hWVoRVDwO/4NHuAZHU8kOMWlewOj67bedSh2 iP/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=kfwYRcon; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z64-20020a636543000000b00513955deec2si23442pgb.248.2023.03.30.09.12.55; Thu, 30 Mar 2023 09:13:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=kfwYRcon; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232625AbjC3OzO (ORCPT + 99 others); Thu, 30 Mar 2023 10:55:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232611AbjC3Oyw (ORCPT ); Thu, 30 Mar 2023 10:54:52 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEC9ACC06; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id q19so16312196wrc.5; Thu, 30 Mar 2023 07:54:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TsovVhZknI6qeDE8kTU6QPdUJtFC7EMxs3/MNa3hVfA=; b=kfwYRcon7F1ORzR/RlIjWpwfgQKmtyIOPTNAWE95Z/LyNWD0toDNGfhLyrWbWRxQXb EZt3Tr2IBWMTLdJpnV4WJ0Bl2sgYaZq2nvo9N8/yB3lRkojyRpIii51pcNtLKGcS51M9 Xkh9fnMl5JKbqOEA+CA6TYVa6SjK5R2E57t0IOgkE8WPgFmMBt8EYZlkQzBho2D7nzHF 1/G1J+H/W93aiCzFTM9hcH7YOoB/KF7ELA7O+LVOn8iwZv0m0ztL1/b3SkrjQql8lyK9 3CNR/OuOZJOioEqSVeaNuvxAgridGwr9zeUt6qX5rncc6zQVeRooNNHnpoDJ33iWH19L Xrzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TsovVhZknI6qeDE8kTU6QPdUJtFC7EMxs3/MNa3hVfA=; b=fy+UFcNTj/gT4mDvXZ+I8aqWUTsfhiqsZ1FbN9NjWa8tSyUYg3V9b0W9nhRZgCBdiq Ceuyd4uAh2Y5HWHqWCQTc37TgyfLzDaOqylh+lqAo05zR8vcbe8OTr2P9M07fgZlSPRA RGnKSRY0K4/GP5UxglSg7I2qo+HwdNFEwvZqnD7CnIQnAIik5rTP2osIc98+wdzGsV2+ 9+Nqj+ZmuJwa/d01K8B17rq/oJLbyvGgT/ksZJeT6L5N/6XLhxKUbyi9Fy7fmYUKBBrY JLKVdKZsaphVEcgsnWolENqSECtMLubCTTJbeCamJyS0pD9ml7YZP1Zfxa0c6jQq/U6q dikA== X-Gm-Message-State: AAQBX9fCAlGAmJ01M/RlP8vOT9es2ay+MtHRwZT6J1BinXpf3rd5RWGi NZ33AQqNSxKYFtf1BXp8KOJlYsrRB7A= X-Received: by 2002:a5d:404a:0:b0:2d1:947:318b with SMTP id w10-20020a5d404a000000b002d10947318bmr18567845wrp.13.1680188084304; Thu, 30 Mar 2023 07:54:44 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 09/11] io_uring/rsrc: don't offload node free Date: Thu, 30 Mar 2023 15:53:27 +0100 Message-Id: <8083bdb49f57a968104137f1f256af8a5d46da64.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809834760833678?= X-GMAIL-MSGID: =?utf-8?q?1761809834760833678?= struct delayed_work rsrc_put_work was previously used to offload node freeing because io_rsrc_node_ref_zero() was previously called by RCU in the IRQ context. Now, as percpu refcounting is gone, we can do it eagerly at the spot without pushing it to a worker. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 -- io_uring/io_uring.c | 6 ---- io_uring/rsrc.c | 59 +++------------------------------- 3 files changed, 4 insertions(+), 64 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9492889f00c0..47496059e13a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -330,9 +330,6 @@ struct io_ring_ctx { struct io_rsrc_data *file_data; struct io_rsrc_data *buf_data; - struct delayed_work rsrc_put_work; - struct callback_head rsrc_put_tw; - struct llist_head rsrc_put_llist; /* protected by ->uring_lock */ struct list_head rsrc_ref_list; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e94780c0a024..8c3886a4ca96 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -326,9 +326,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); INIT_LIST_HEAD(&ctx->rsrc_ref_list); - INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); - init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); - init_llist_head(&ctx->rsrc_put_llist); init_llist_head(&ctx->work_llist); INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; @@ -2822,11 +2819,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_rsrc_node_destroy(ctx->rsrc_node); if (ctx->rsrc_backup_node) io_rsrc_node_destroy(ctx->rsrc_backup_node); - flush_delayed_work(&ctx->rsrc_put_work); - flush_delayed_work(&ctx->fallback_work); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); - WARN_ON_ONCE(!llist_empty(&ctx->rsrc_put_llist)); #if defined(CONFIG_UNIX) if (ctx->ring_sock) { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 95e71300bb35..0f4e245dee1b 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -145,15 +145,8 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, { struct io_ring_ctx *ctx = rsrc_data->ctx; - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } + if (prsrc->tag) + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); rsrc_data->do_put(ctx, prsrc); } @@ -176,32 +169,6 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) complete(&rsrc_data->done); } -void io_rsrc_put_work(struct work_struct *work) -{ - struct io_ring_ctx *ctx; - struct llist_node *node; - - ctx = container_of(work, struct io_ring_ctx, rsrc_put_work.work); - node = llist_del_all(&ctx->rsrc_put_llist); - - while (node) { - struct io_rsrc_node *ref_node; - struct llist_node *next = node->next; - - ref_node = llist_entry(node, struct io_rsrc_node, llist); - __io_rsrc_put_work(ref_node); - node = next; - } -} - -void io_rsrc_put_tw(struct callback_head *cb) -{ - struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, - rsrc_put_tw); - - io_rsrc_put_work(&ctx->rsrc_put_work.work); -} - void io_wait_rsrc_data(struct io_rsrc_data *data) { if (data && !atomic_dec_and_test(&data->refs)) @@ -217,34 +184,18 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - bool first_add = false; - unsigned long delay = HZ; node->done = true; - - /* if we are mid-quiesce then do not delay */ - if (node->rsrc_data->quiesce) - delay = 0; - while (!list_empty(&ctx->rsrc_ref_list)) { node = list_first_entry(&ctx->rsrc_ref_list, struct io_rsrc_node, node); /* recycle ref nodes in order */ if (!node->done) break; - list_del(&node->node); - first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); - } - if (!first_add) - return; - - if (ctx->submitter_task) { - if (!task_work_add(ctx->submitter_task, &ctx->rsrc_put_tw, - ctx->notify_method)) - return; + list_del(&node->node); + __io_rsrc_put_work(node); } - mod_delayed_work(system_wq, &ctx->rsrc_put_work, delay); } static struct io_rsrc_node *io_rsrc_node_alloc(void) @@ -320,13 +271,11 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, if (ret < 0) { atomic_inc(&data->refs); /* wait for all works potentially completing data->done */ - flush_delayed_work(&ctx->rsrc_put_work); reinit_completion(&data->done); mutex_lock(&ctx->uring_lock); break; } - flush_delayed_work(&ctx->rsrc_put_work); ret = wait_for_completion_interruptible(&data->done); if (!ret) { mutex_lock(&ctx->uring_lock); From patchwork Thu Mar 30 14:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77242 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1231804vqo; Thu, 30 Mar 2023 09:01:16 -0700 (PDT) X-Google-Smtp-Source: AKy350a2gnqFRL9G48zUcfYq0bd0PuJQCk81nETPeA8asmMQJg9NgERKbdp6iGVTXOPh6dZSID8r X-Received: by 2002:a17:906:2b46:b0:931:bc4a:ce08 with SMTP id b6-20020a1709062b4600b00931bc4ace08mr26207492ejg.36.1680192076599; Thu, 30 Mar 2023 09:01:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680192076; cv=none; d=google.com; s=arc-20160816; b=y/MNJZL1MK1zgQF8qbgP4i9RogBfbh9xjK5PxxGIFsbDXEXdjkk/oYwDTam/xJ7yZg IEoam7nlrwvujEy7aq7xJR17djZ1qYlat58j99QeTkN1Q0nwyY2cb0lSSu6fJEBH/Xx9 tztloLbvTe84pjlTutgdlzAuckVlwMStz8dpoGPnRtd5twBVFYatO4XULwmZ82IZBUP9 K3Y3bgoGsyMeCQEtWwknoSZ8cdlKPVkpGFSFK9F/LGnKBYVbPdscXci2fbnNlOFA+cJt AAIdXDJEj4gl2OkU90urvSZlFn5y0Uba30pspXGr/W9oGDCFEHxt6FisX777IFg1jQtk keAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5L1thsz1ijtI+bii8vpLvzOuK72ES0dN5LpyxAEJB5c=; b=rskx3JvZNUPGxwhcdB1leKKjNanQ92kItjN/JPbts8G/Rnw4j9oxKzdYzc67vWecCH 1YUxkCL/8ycg2GYKtiOvs+IFfGTD302FnL0A8AEFyIBWuvYKoWRM65hE1aNCeSkD76ED tvfrN13jo/7JMD3woh4qJUG/utzOO6SqHMIVAJy7qGuj3MLopYjNYB0AAypV3ZuUL428 ZS7J235HrCwfp92RuOMC3HG4iMJssYuDfgpdvY4ZKyg79ryIPhMVTKO+RbZKJzHIMpHt 8r9eJleKX/98MeChlT4URiYocxClXx32VyeTjhtYHFa+e00uAYz4uTpS2lAWegzXzknf 6HzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=S8QHHi5z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v14-20020a1709063bce00b00945552ee129si8702737ejf.616.2023.03.30.09.00.39; Thu, 30 Mar 2023 09:01:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=S8QHHi5z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232678AbjC3OzS (ORCPT + 99 others); Thu, 30 Mar 2023 10:55:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232647AbjC3Oyz (ORCPT ); Thu, 30 Mar 2023 10:54:55 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85756CC0B; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id y14so19396356wrq.4; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5L1thsz1ijtI+bii8vpLvzOuK72ES0dN5LpyxAEJB5c=; b=S8QHHi5z9LQOXa3uaWYMZ73OR1vDlNr3ze+rGxV67KXFNl+cuHXiqn5xbDswvDcMbt kdzkDFjmn2YYLGAt2MvCosiHyv0RaOltvUfqrEHfPO9pcC7c67zsfsQ6p9B6xrCr9TPw XLAxYueWYIYjma7qkjufO78bsMj3LhAjKmV32/78VXV0hIIfm7uZ/bjraokrdb+rnPuk CEV7/JHzJ3Dp5s5nGyMrzE3gJfrWX6m2go530EO+HSUcIN7nrLwni/bqJRAzqo9nLpQU 66tK2GGCYaq/M/oAqOdbQ6UD0qAM5CT9N8nS/Ic26HZIIJXGZMJ3MCzFNgfZdgq/92Gs Ss6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5L1thsz1ijtI+bii8vpLvzOuK72ES0dN5LpyxAEJB5c=; b=6kKQu9BAYX881ImrUsv0BvDOP7sxNGeEIOIXFIeXEE6lFPnun9Zd27RWuVkTM22FFG q+enWFsyAQpQRxuKMRJlkyOtFwoTRJhQx9uMfojsjRdXqCxLHShOH0vms6YXH5uhxXNh FAPKNAhvD3wmF4C67HKn6PUYxlU/ivOF93jx23VcVMK1+RDJopyCVa38t/u+B0kjIqqW VowGOzliUSRg60tDcLVIBL8Le4xzzN3vaYTrcB6EZ/KnM0XA8cvdwTgVNRvLul4sGF/l 0VPoZoa6D5YpHt84leF4l/rA1LvbMEJCC7IPybIbcBXyYz/24DgYvCHE1qFNGlWXGa1u 5K6g== X-Gm-Message-State: AAQBX9e7/reayQqiCBKACW2L+p+LLRgtrtrwGPjLwMjO/dZk6mHiG5oZ WIeT3BROepx3xkh9lrrOHcceo6xC8mA= X-Received: by 2002:a5d:5601:0:b0:2d2:ac99:a72 with SMTP id l1-20020a5d5601000000b002d2ac990a72mr20884439wrv.46.1680188086134; Thu, 30 Mar 2023 07:54:46 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 10/11] io_uring/rsrc: cache struct io_rsrc_node Date: Thu, 30 Mar 2023 15:53:28 +0100 Message-Id: <7f5eb1b89e8dcf93739607c79bbf7aec1784cbbe.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761809086805446226?= X-GMAIL-MSGID: =?utf-8?q?1761809086805446226?= Add allocation cache for struct io_rsrc_node, it's always allocated and put under ->uring_lock, so it doesn't need any extra synchronisation around caches. Signed-off-by: Pavel Begunkov Reviewed-by: Gabriel Krisman Bertazi --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 11 +++++++++-- io_uring/rsrc.c | 23 +++++++++++++++-------- io_uring/rsrc.h | 5 ++++- 4 files changed, 29 insertions(+), 11 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 47496059e13a..5d772e36e7fc 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -332,6 +332,7 @@ struct io_ring_ctx { /* protected by ->uring_lock */ struct list_head rsrc_ref_list; + struct io_alloc_cache rsrc_node_cache; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8c3886a4ca96..beedaf403284 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,6 +310,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); + io_alloc_cache_init(&ctx->rsrc_node_cache, sizeof(struct io_rsrc_node)); io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); @@ -2791,6 +2792,11 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) mutex_unlock(&ctx->uring_lock); } +void io_rsrc_node_cache_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct io_rsrc_node, cache)); +} + static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); @@ -2816,9 +2822,9 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) - io_rsrc_node_destroy(ctx->rsrc_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_node); if (ctx->rsrc_backup_node) - io_rsrc_node_destroy(ctx->rsrc_backup_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_backup_node); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); @@ -2830,6 +2836,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) #endif WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); + io_alloc_cache_free(&ctx->rsrc_node_cache, io_rsrc_node_cache_free); if (ctx->mm_account) { mmdrop(ctx->mm_account); ctx->mm_account = NULL; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 0f4e245dee1b..345631091d80 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -164,7 +164,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) kfree(prsrc); } - io_rsrc_node_destroy(ref_node); + io_rsrc_node_destroy(rsrc_data->ctx, ref_node); if (atomic_dec_and_test(&rsrc_data->refs)) complete(&rsrc_data->done); } @@ -175,9 +175,10 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) wait_for_completion(&data->done); } -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - kfree(ref_node); + if (!io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache)) + kfree(node); } void io_rsrc_node_ref_zero(struct io_rsrc_node *node) @@ -198,13 +199,19 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) } } -static struct io_rsrc_node *io_rsrc_node_alloc(void) +static struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) { struct io_rsrc_node *ref_node; + struct io_cache_entry *entry; - ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); - if (!ref_node) - return NULL; + entry = io_alloc_cache_get(&ctx->rsrc_node_cache); + if (entry) { + ref_node = container_of(entry, struct io_rsrc_node, cache); + } else { + ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); + if (!ref_node) + return NULL; + } ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); @@ -243,7 +250,7 @@ int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) { if (ctx->rsrc_backup_node) return 0; - ctx->rsrc_backup_node = io_rsrc_node_alloc(); + ctx->rsrc_backup_node = io_rsrc_node_alloc(ctx); return ctx->rsrc_backup_node ? 0 : -ENOMEM; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 17293ab90f64..d1555eaae81a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -4,6 +4,8 @@ #include +#include "alloc_cache.h" + #define IO_RSRC_TAG_TABLE_SHIFT (PAGE_SHIFT - 3) #define IO_RSRC_TAG_TABLE_MAX (1U << IO_RSRC_TAG_TABLE_SHIFT) #define IO_RSRC_TAG_TABLE_MASK (IO_RSRC_TAG_TABLE_MAX - 1) @@ -37,6 +39,7 @@ struct io_rsrc_data { }; struct io_rsrc_node { + struct io_cache_entry cache; int refs; struct list_head node; struct io_rsrc_data *rsrc_data; @@ -65,7 +68,7 @@ void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_wait_rsrc_data(struct io_rsrc_data *data); -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); From patchwork Thu Mar 30 14:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 77275 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1246686vqo; Thu, 30 Mar 2023 09:17:38 -0700 (PDT) X-Google-Smtp-Source: AKy350ZaDyYLDUtkBkzs0E+1qN7IyLjgakxWdQ3GtPZXFQzdovKOnbCuaAYNFI1KEs4bvYTHefTS X-Received: by 2002:a17:90b:3ecb:b0:23b:3641:cf16 with SMTP id rm11-20020a17090b3ecb00b0023b3641cf16mr7051252pjb.11.1680193058409; Thu, 30 Mar 2023 09:17:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680193058; cv=none; d=google.com; s=arc-20160816; b=Yg3+A1Ohq6CU5PkErJ23fiAww6XAzq7z507tvRza0U8+MdVaDKQIDgpydpH8ap3JC9 BG3KpU9fnHk7fyOYTgQ3sOc8tj4jaSApSKaP3/6B/OMYv3GCYjOiMrkok0HRzeI7PAua bEVOyaEaHa38xqoIBwaKhxYgmqX84abYUe+oeF2zOnhJhPmliOF5c/3oAV2uwbgmCySC UZiKt9WUxnNBbFN28gOD2wCCH6dRGxTuHeROgxhDioXS+AO9gtA5bxaVTjhHhcO4Erou 4MMXE6LVr2APdITIUQuk0yQW2470ltdla98nqjD8pNSqwCAQj+4lZcaLDUp+gQC7XASF e2hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Cs6esp50Jp8jC0Q9hhTRpI4nFHoCs+WBE2+fqKZb6bE=; b=T6T/jz26Ct/nFjTOBrNhtI8ffHURWWjzZ/jALXa/BVACvKoz63OX5STuyJomT9mrty LGmkblUX/U+ClCl6o3PQGzjKc3zvoLxJW6yqxCjRQ5RRT3aHOqxp0Az0eVJvXeblHBAB RE6Q0KH8i03j0Rnkz5KeOvHX7CGsPVpI0FbwWBlOGm+cIKLUT6JMX8cD5UJVPJEVJdRh FNp24ATO/s4CbRIwLiDsTUPlYlINfAK07xt5Il0DsmpT5OofyTF9L17jwNN/Rfl4w3H/ ILyGHZsRoy6j2yGDrbQGfc/dcBktZtjJ+1gyEb1W9LexXXjyATmb3n4mNyUUAXUV22jo Jo0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=lZAb3+CF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r11-20020a632b0b000000b0050f66d4e070si9429pgr.417.2023.03.30.09.17.20; Thu, 30 Mar 2023 09:17:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=lZAb3+CF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232702AbjC3OzZ (ORCPT + 99 others); Thu, 30 Mar 2023 10:55:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232587AbjC3OzA (ORCPT ); Thu, 30 Mar 2023 10:55:00 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4056FCC28; Thu, 30 Mar 2023 07:54:50 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id h17so19368928wrt.8; Thu, 30 Mar 2023 07:54:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cs6esp50Jp8jC0Q9hhTRpI4nFHoCs+WBE2+fqKZb6bE=; b=lZAb3+CF9ww4EB25PAj0+FL6j458WyQZqxNaCRdNWNkCqzpefmh+epQzj26e3OrL8n TawKc4VhwlrLo35FK5XVNnUUfAZzzv4mIacNOrsSiYkn478r4ECEfAjWEG9hH5lnNh7X BkdLNArI1Xv6UnoeTi6296UPd++TUI5Ui/T93an238DKnTulXzRnQ9eMD/A9zK5xxU0y BV7BSq2FjbBKojJdqQ5mCUbaWV+np5V2VdduZBuDPk9HLHiqt4djiYChWGDy8BcrZKU2 yE/Bq3prxOLlWMFVPZv/ZEH2jnZprFNE0fyHk08xrt3HwTz72C5Ep0TuaR1HuCM8VntC 89VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cs6esp50Jp8jC0Q9hhTRpI4nFHoCs+WBE2+fqKZb6bE=; b=ZcaqVxvXKLPbyQsDCZJK4zwNVk0roXyeFmTiJlDmtM4z6ackJp+1v9pz9YLG8RMhIG IK4mn75pz7jlSZ2frc+k20z0iYd5W3jHGWxEula/R0uC1GeednuhwCph13dm/X7z5qaH L+qGhO0N/Q3bY5Fp1Df6obk1CqXb79fQ2oi2HYj527liwBuQWhn2mNchMJiMbI//4nNy SVdb1txe0bljhZ4GhjaXEdpoU9bymdid+yUitCUeNvVDdqzJ8C7VRbS1QbZVxkw5aZod 6WY9CzkMM0jhDh7y63n/fSjTzEm8w8Mqxsg+uIVF5x04exBUBRyNh+ZxwDWESUuxw7bh ks3g== X-Gm-Message-State: AAQBX9dCAVnBmuJ8/4lw6WeQhc0BjZayrlTUEW/y/2UlUE/Pu28Im8W1 GmHK2YMPr21SAzooVdKN+oYDLA7uLGY= X-Received: by 2002:adf:f549:0:b0:2e5:ab4:22f0 with SMTP id j9-20020adff549000000b002e50ab422f0mr1582562wrp.60.1680188088366; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 11/11] io_uring/rsrc: add lockdep sanity checks Date: Thu, 30 Mar 2023 15:53:29 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761810116577574303?= X-GMAIL-MSGID: =?utf-8?q?1761810116577574303?= We should hold ->uring_lock while putting nodes with io_put_rsrc_node(), add a lockdep check for that. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ++-- io_uring/rsrc.c | 2 +- io_uring/rsrc.h | 6 ++++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index beedaf403284..a781b7243b97 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1002,7 +1002,7 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) if (rsrc_node) { io_ring_submit_lock(ctx, issue_flags); - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } @@ -1123,7 +1123,7 @@ __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) if (req->rsrc_node) { io_tw_lock(ctx, ts); - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 345631091d80..d4bca5e18434 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -236,7 +236,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index d1555eaae81a..99f2df4eafa1 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -117,8 +117,10 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_put_rsrc_node(struct io_rsrc_node *node) +static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { + lockdep_assert_held(&ctx->uring_lock); + if (node && !--node->refs) io_rsrc_node_ref_zero(node); } @@ -126,7 +128,7 @@ static inline void io_put_rsrc_node(struct io_rsrc_node *node) static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) { - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx,