From patchwork Thu Apr 6 13:20:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 80238 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1026581vqo; Thu, 6 Apr 2023 06:29:12 -0700 (PDT) X-Google-Smtp-Source: AKy350Z3T2yBYboD2chQN8T38BXSSLTa8ZbE4vOLS7RsfJvYRBJoH01zte6IN6lkcMmYPQfYAg/i X-Received: by 2002:a17:906:c458:b0:920:388c:5c60 with SMTP id ck24-20020a170906c45800b00920388c5c60mr6312531ejb.41.1680787752265; Thu, 06 Apr 2023 06:29:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680787752; cv=none; d=google.com; s=arc-20160816; b=Xhkc0fKq5EY5NqOCA8kTKXcZ/hly9oNu6ORgYEEoDi0zM0Clo7pRVFXHq3TfvQRjuQ 63RU1N6u7DNgkYXHhPWuMydZyoR+1jFVDYO029IekA8OHGYK2CHQyg1pq/2Gdr8clv/d gWzpgld/pGuw7Cf6i6YhikrtzNWaJuZV52oynz028ycDEU+QsrO+ajTBodbDbDL/7tyk REkUfM69E6AfljFrFnHy7EoZU6WlPVXZHRukqoNrdJvnG1vag6g1PZaPqnQKvp0ZFkI2 6dovkxbAn8P5fd5gywmmqvk59+z3XEPqf4l/jVRcpFyCZeWx6tiwVEu0XiTsVwAfYT2e 1uxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OFPFrQtthgmu2R5B+cqkgAt4SXW7neAKcH/973v1o60=; b=R0l9CzheyAyT+nLDly6E1dpDt3SCQOELzMcDCkKmKfuIYI+J//c1jHyqZhRlVRx1ka Ckrw3ddGL4Ek3oI3tJSwVpFm79qrEsN6O6zodnez16yoVlK00FFHJvaUFg00bz4irruF 95yLwiPK5Y92mW03CaynwD5ebCLQBxic0sNpbhp84D8o1A65bLx3zpL1jn8UaKvqIF+B SbGFcON0ZKRnIj6rNDp+oMrIXstjlPzaaHsK122xffd5XwSYq+NPw3ZW5kH1aDqgKf38 x5ho7JXtRbJ8ML1Gi56ESepoiXVKYgkVY+4TrHBKE5x2Z1RVXRWQUY40qQb5FIskXoh3 dqTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=jDHpciR8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ne3-20020a1709077b8300b00946c2651290si499319ejc.232.2023.04.06.06.28.47; Thu, 06 Apr 2023 06:29:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=jDHpciR8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238412AbjDFNVL (ORCPT + 99 others); Thu, 6 Apr 2023 09:21:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238141AbjDFNU5 (ORCPT ); Thu, 6 Apr 2023 09:20:57 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 619E9A246; Thu, 6 Apr 2023 06:20:26 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-50299ceefa4so1300953a12.2; Thu, 06 Apr 2023 06:20:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680787224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OFPFrQtthgmu2R5B+cqkgAt4SXW7neAKcH/973v1o60=; b=jDHpciR8zlv2xmb3+NF0BJkS/+wymZqezlIUfWqud0zV2V+oRkaFEMPNULVUHUNnNT 0Nj2KAh3ZJPzkgQQXQxB3owvsxSTEBJ4+RC5oWBRbDieJstz/KAWzFTJ2CB/eL9xEumO H20k5ZIoNSSmEBoLRPPcc37wkfXpi5Ikhm78V7ORHSoq6CbpNhkwIggCqTsW+DxI7IFt c+tCwUpoOorU/BR2AO9S3zm+YJMITMR6DSDkz3uE4FEShzsyOCfzsLE3qZHB+VBhBqt9 ym4Sa2CKWpsiNmtcRiDkM9NUjNsEEZl9Aa3BsG+fMcrPE/jI4JEX9pu8BghKfdj/Pbzq zUSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680787224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OFPFrQtthgmu2R5B+cqkgAt4SXW7neAKcH/973v1o60=; b=sAIk8A35l1TgOI+q9mAnYW3xz31+3bTZ+5wJKkq4dpAkznNzT1JCsHiyB1pQAgzNQz IUuKBeVV8JLTWk6QCqrOGtpfmE3jrXhp2UfuxoEu83oBV92Fm3FzRkFNFEBsld0eCrrt U3m+KN9K29biBCzphO9SmqnHGrdGKh6MUvZMCHrPZuGN5TjwAKqoiqHDHEvjnLMz8qkL Pr9wr9C7+PHSGBflXPQ61sk8lyrw/Iv7A5XWg1hHR8n4xthKo+8X/CFIhcYvItFTWdCQ 4YGwoxa1+wppD5fo9UBlCgFVuD2HZRSBWgfZJ+AV3kBUUlGybfUBqbqGd7hpMZKZrdXx eDzQ== X-Gm-Message-State: AAQBX9dbuKheAfJBvyMIqtyJKG5ZS0kDdx9jA9olXOhfUFalPMYZKRpz imDVwWCc5jJkVUu636eDVESDY5g6EeA= X-Received: by 2002:aa7:ce15:0:b0:502:2148:2980 with SMTP id d21-20020aa7ce15000000b0050221482980mr5078208edv.30.1680787224359; Thu, 06 Apr 2023 06:20:24 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:a638]) by smtp.gmail.com with ESMTPSA id m20-20020a509994000000b0050470aa444fsm312732edb.51.2023.04.06.06.20.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Apr 2023 06:20:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/8] io_uring: optimie local tw add ctx pinning Date: Thu, 6 Apr 2023 14:20:08 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762433697858921696?= X-GMAIL-MSGID: =?utf-8?q?1762433697858921696?= We currently pin the ctx for io_req_local_work_add() with percpu_ref_get/put, which imply two rcu_read_lock/unlock pairs and some extra overhead on top in the fast path. Replace it with a pure rcu read and let io_ring_exit_work() synchronise against it. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 29a0516ee5ce..fb7215b543cd 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1332,9 +1332,9 @@ void __io_req_task_work_add(struct io_kiocb *req, bool allow_local) struct io_ring_ctx *ctx = req->ctx; if (allow_local && ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - percpu_ref_get(&ctx->refs); + rcu_read_lock(); io_req_local_work_add(req); - percpu_ref_put(&ctx->refs); + rcu_read_unlock(); return; } @@ -3052,6 +3052,10 @@ static __cold void io_ring_exit_work(struct work_struct *work) spin_lock(&ctx->completion_lock); spin_unlock(&ctx->completion_lock); + /* pairs with RCU read section in io_req_local_work_add() */ + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + synchronize_rcu(); + io_ring_ctx_free(ctx); }