From patchwork Wed Dec 14 02:50:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 33025 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp499709wrn; Tue, 13 Dec 2022 18:54:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf6S/SUdL4lvN7bWjt3hTSwHq7YQKLEXv1XB2kwD/t5a75HQz3jOvQIO0n7CDbf6WpX4bILc X-Received: by 2002:a05:6a00:1696:b0:56e:dca8:ba71 with SMTP id k22-20020a056a00169600b0056edca8ba71mr28673936pfc.32.1670986484567; Tue, 13 Dec 2022 18:54:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670986484; cv=none; d=google.com; s=arc-20160816; b=jTKe05PY9OFCwKGg+7p1H2yFEQ+TdcJGPUO0nE36N591uMz0PId94RnjtVGsn7FeMH 6XuUQEoE/FSikcr180VDoDKKt92pMnCQjcfVPJbBfowryezYBABvi6ps2G0pxYY3R0Ky eZzokXrUaWzwdg00NLMc9JspLHQ1z/fgXh0q19T6GTC+p4gV38SXQVmPZjjJisMXLw46 wzYThFXA78o17Yr5gPmATuOXqIR+8VlCK6OuVjgl6Qgay5mjCcKuplEeKjHGuy6e43oU 0ybBD44MaNToEpCb93Go87/CNssyHxK7pS90Q3jpbVBfiwwmgleZbt75ALe+aXFoxWfv mLsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YvXYXT7+pZNjN3bsp9+0AEDOU4OMWFYmE3SAXAaEld0=; b=xnFoC62Ohl9GjUnKhctXEU4vIWkFWMTs5/Sq8JG4LE3xxBrI5womu6gxRzyb6U3fVC A8lv5GkWMXhqoqi7w+je0XA2nFoGpSJGgx86m/sLGtYj+fv8EN70PPf07Wz1emmh/MF0 tPwV5dZy7R5tY8odzZCg2lGpUk5UItrWV0G04PkhwzxG49IfIHE0iENs+itjFJwx5S7Q aIWM+IPj14CkBazZgEmuhZIGeq6nDzmbUA/zTHXAuCjaDDZBYUMWg73F74vo9qICa1es 5S1H0XRvkyDhtSHd7A24HkxrJciIPptrj4hQV8rZ6DIaRi2b81C403LkzJy6zHAKb9v2 PO5A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JbmO1LSz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z13-20020aa785cd000000b00561898445bdsi13548231pfn.273.2022.12.13.18.54.31; Tue, 13 Dec 2022 18:54:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JbmO1LSz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237159AbiLNCwY (ORCPT + 99 others); Tue, 13 Dec 2022 21:52:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236890AbiLNCwL (ORCPT ); Tue, 13 Dec 2022 21:52:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6F6217A8D for ; Tue, 13 Dec 2022 18:51:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YvXYXT7+pZNjN3bsp9+0AEDOU4OMWFYmE3SAXAaEld0=; b=JbmO1LSzu+tuhIo6GXcle8PSPwzWYI4jx7oGBCJWKpgLiv+0a1/L5R7FweGmLTJhsdJlpI OrYIV3LU+QbiwRY9nit1vw2kqUaN7WkfXGnut4H28ejtC8R/HB9mNJRuNskvaxX8O0URkm NLeebrY0RbHh5pEOlpXmJ/JaWhvM69w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-290-pa7vWdvINQiKRuTjDpGF-w-1; Tue, 13 Dec 2022 21:51:22 -0500 X-MC-Unique: pa7vWdvINQiKRuTjDpGF-w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 327A0806002; Wed, 14 Dec 2022 02:51:22 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FA9A492B00; Wed, 14 Dec 2022 02:51:20 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 1/3] lib/percpu-refcount: support to exit refcount automatically during releasing Date: Wed, 14 Dec 2022 10:50:59 +0800 Message-Id: <20221214025101.1268437-2-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752156323693234365?= X-GMAIL-MSGID: =?utf-8?q?1752156323693234365?= We only have two users in which percpu_ref_exit() is called from ->release(). Add flag of PERCPU_REF_AUTO_EXIT for avoiding to call percpu_ref_exit() from ->release() directly since we need to drain ->release() in percpu_ref_exit() for fixing use-after-free. Signed-off-by: Ming Lei --- include/linux/percpu-refcount.h | 21 +++++++++++++++++++-- lib/percpu-refcount.c | 9 ++++++--- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h index d73a1c08c3e3..006c6aae261e 100644 --- a/include/linux/percpu-refcount.h +++ b/include/linux/percpu-refcount.h @@ -90,6 +90,11 @@ enum { * Allow switching from atomic mode to percpu mode. */ PERCPU_REF_ALLOW_REINIT = 1 << 2, + + /* + * call percpu_ref_exit() when releasing + */ + PERCPU_REF_AUTO_EXIT = 1 << 3, }; struct percpu_ref_data { @@ -98,6 +103,7 @@ struct percpu_ref_data { percpu_ref_func_t *confirm_switch; bool force_atomic:1; bool allow_reinit:1; + bool auto_exit:1; struct rcu_head rcu; struct percpu_ref *ref; }; @@ -331,8 +337,19 @@ static inline void percpu_ref_put_many(struct percpu_ref *ref, unsigned long nr) if (__ref_is_percpu(ref, &percpu_count)) this_cpu_sub(*percpu_count, nr); - else if (unlikely(atomic_long_sub_and_test(nr, &ref->data->count))) - ref->data->release(ref); + else { + struct percpu_ref_data *data = ref->data; + struct percpu_ref copy = *ref; + bool release = false; + + if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { + data->release(ref); + release = true; + } + + if (release && data->auto_exit) + percpu_ref_exit(©); + } rcu_read_unlock(); } diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index 668f6aa6a75d..c0cadf92948f 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -82,6 +82,7 @@ int percpu_ref_init(struct percpu_ref *ref, percpu_ref_func_t *release, data->force_atomic = flags & PERCPU_REF_INIT_ATOMIC; data->allow_reinit = flags & PERCPU_REF_ALLOW_REINIT; + data->auto_exit = flags & PERCPU_REF_AUTO_EXIT; if (flags & (PERCPU_REF_INIT_ATOMIC | PERCPU_REF_INIT_DEAD)) { ref->percpu_count_ptr |= __PERCPU_REF_ATOMIC; @@ -123,9 +124,11 @@ static void __percpu_ref_exit(struct percpu_ref *ref) * * This function exits @ref. The caller is responsible for ensuring that * @ref is no longer in active use. The usual places to invoke this - * function from are the @ref->release() callback or in init failure path - * where percpu_ref_init() succeeded but other parts of the initialization - * of the embedding object failed. + * function from are where the refcounter is confirmed as idle or in init + * failure path where percpu_ref_init() succeeded but other parts of the + * initialization of the embedding object failed. For caller which needs + * to call percpu_ref_exit() in ->release, please pass PERCPU_REF_AUTO_EXIT + * to percpu_ref_init(). */ void percpu_ref_exit(struct percpu_ref *ref) { From patchwork Wed Dec 14 02:51:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 33022 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp499476wrn; Tue, 13 Dec 2022 18:54:05 -0800 (PST) X-Google-Smtp-Source: AA0mqf6YNz/SNe2QlHus8hKvvDWVKpdJFQJB9EX/B+heOr+501ya4bAbbMqhKbBeo5LqKTkJtjpZ X-Received: by 2002:a05:6a00:1696:b0:56d:6a0:b6b2 with SMTP id k22-20020a056a00169600b0056d06a0b6b2mr27555887pfc.32.1670986445284; Tue, 13 Dec 2022 18:54:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670986445; cv=none; d=google.com; s=arc-20160816; b=yg5ISA3u29Bzc9VnEnkSiR/CENiALm/z+L2kf1YfRwPZaUPrzVCIsEnPd0l9Jh438i 82oCuobbdyugmS3voZUygwiFtNULeYhvedDNdIbH4gQ5B6uxPvavSmPXMlkNLE1c9ffl w1GdWtBQCzfkQlXZubDTpbSapTKKdAq0hSvsTLow+Cr9rXuLqwLXsO0Va3cpXs7XqLxi VUsVWbPtVEZzBnyM5E8l/7fOwD+tPcafxbZRHLOon2vHZhNiwfW/aPsiIgFm9op5HscV 2gNGSoE/ZPFIh+Ro4nTZNBUgPcBTyyD3otgxTtUyiKmj3FoTB3NQ3uVMDOj77sz4TIoU N+XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BYOjfeiBRlo83yXPXn1TZ8OQl1UZe5v7B1C1yln8/XY=; b=D3kF7+SLvfHQLSQiBuOPrJrbhFbe6wfX0fmDgwq6EGnVzyKcgP2tGjp2btRsRUg8Hj sVMx6f94CzQUXo60Vd39ldg4lxVtAuk7jGITgOSyEniVL3k3UJ79IHaXGpOIe1YzrE7F J5tI/TOOTqQqtwpYtzzQP3iKbbBh+gopVsqqwh9IBXoe/zxkobsj4fLdwKwIy5AlwJ6i /R/RlRFykU/9k8d07HBk6Ytg4UcnYdZMWeoEcVSnmQ/u/UUtvHTkM0AeTP3LGvL216eR B0JgySZmJluBPjeLX07DZlS4IX8z2sn+T5PMfLROVa3kY2vzGMaaRZmEHfsbLeTEz6b1 TEBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=W4GYjbyu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v21-20020a056a00149500b00572dc2ed1f7si3765035pfu.191.2022.12.13.18.53.10; Tue, 13 Dec 2022 18:54:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=W4GYjbyu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237077AbiLNCwQ (ORCPT + 99 others); Tue, 13 Dec 2022 21:52:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236731AbiLNCwK (ORCPT ); Tue, 13 Dec 2022 21:52:10 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29B5E22538 for ; Tue, 13 Dec 2022 18:51:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BYOjfeiBRlo83yXPXn1TZ8OQl1UZe5v7B1C1yln8/XY=; b=W4GYjbyu3o+FAp6/iO7b7tMJAVf3nLarRQVI5BwllOD7hByuMqz5Im3y8b3l/pdgaWUk6u 0xRS2ROvDVZI22cW66wWgMY8UkRUI46eoAhAUbwltNT69eYWq0A/FuB8JnD2WOX1qGPKA+ 38w4G+SQFrUoH6a/zaHBJ0jICDHt1sM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-TswRycF3MN-7n1xDGsI2kQ-1; Tue, 13 Dec 2022 21:51:27 -0500 X-MC-Unique: TswRycF3MN-7n1xDGsI2kQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 787843832781; Wed, 14 Dec 2022 02:51:26 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F6B51121314; Wed, 14 Dec 2022 02:51:25 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 2/3] lib/percpu-refcount: apply PERCPU_REF_AUTO_EXIT Date: Wed, 14 Dec 2022 10:51:00 +0800 Message-Id: <20221214025101.1268437-3-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752156282687928712?= X-GMAIL-MSGID: =?utf-8?q?1752156282687928712?= Apply the added new flag of PERCPU_REF_AUTO_EXIT, so that users needn't to call percpu_ref_exit() in ->release() any more. Signed-off-by: Ming Lei --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 4 ++-- mm/memcontrol.c | 5 ++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 22d7ba05e9fe..07d1af3fad28 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -122,7 +122,6 @@ static inline void rtrs_srv_inflight_ref_release(struct percpu_ref *ref) struct rtrs_srv_path, ids_inflight_ref); - percpu_ref_exit(&srv_path->ids_inflight_ref); complete(&srv_path->complete_done); } @@ -147,7 +146,8 @@ static int rtrs_srv_alloc_ops_ids(struct rtrs_srv_path *srv_path) } ret = percpu_ref_init(&srv_path->ids_inflight_ref, - rtrs_srv_inflight_ref_release, 0, GFP_KERNEL); + rtrs_srv_inflight_ref_release, + PERCPU_REF_AUTO_EXIT, GFP_KERNEL); if (ret) { pr_err("Percpu reference init failed\n"); goto err; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 266a1ab05434..eaca7d16b143 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -294,7 +294,6 @@ static void obj_cgroup_release(struct percpu_ref *ref) list_del(&objcg->list); spin_unlock_irqrestore(&objcg_lock, flags); - percpu_ref_exit(ref); kfree_rcu(objcg, rcu); } @@ -307,8 +306,8 @@ static struct obj_cgroup *obj_cgroup_alloc(void) if (!objcg) return NULL; - ret = percpu_ref_init(&objcg->refcnt, obj_cgroup_release, 0, - GFP_KERNEL); + ret = percpu_ref_init(&objcg->refcnt, obj_cgroup_release, + PERCPU_REF_AUTO_EXIT, GFP_KERNEL); if (ret) { kfree(objcg); return NULL; From patchwork Wed Dec 14 02:51:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 33023 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp499478wrn; Tue, 13 Dec 2022 18:54:05 -0800 (PST) X-Google-Smtp-Source: AA0mqf7+hHOCOWI+KKSORnTBa9MGG5ckIRy0mQqKwtQfKhCFJZaN3vPSndkhyg/jDicE+Kz+lrjp X-Received: by 2002:a05:6a20:6909:b0:a3:bdd3:8cb0 with SMTP id q9-20020a056a20690900b000a3bdd38cb0mr36609706pzj.56.1670986445321; Tue, 13 Dec 2022 18:54:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670986445; cv=none; d=google.com; s=arc-20160816; b=Tw5zQW32+ZiCNYvIiE547chAfwxl0by25mkrki6XREbbJ8ky4Mn47pjL01+oXYfLPq obMFcA2mRNReYMzzH4yPoa047mP8V+Mu34hMs0uQFH+HxJRGQTEhdXQLxFuDZsBz5IBy XlUaRt0SFlKy7/yCjQ/oz71MY3ZuOWsbJiJIYLUBX3ZveDbJpJSQyJLd3QVVd65RKzTE Sj4N0oUPCbfwI9OXLZ13i6X3wrGORSoOya2iDgqointRx6DafInJOQaj7VM4N5hnS+hl CpU38JqpmGMzfQM9OEHd8fb5LuYcHhRvrHi8meGyY+scLb24mlZanQgTHZrG3MohbFlh t5/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QJTsGV35CYOraEoRISqpKpr/2uwkfrZs9x/0odiJU0M=; b=I5A9ltQRbt4QGVli5aLuTjuNVApkfJ6lA1DypkSuWf1sMHvlU/8llF8PVeJ0y1kHko U1U43x66hmMWyzaDMRzwPYXjrwMT3ZFE5/is/S+DivGsmN7Ct480GkiNvmXYzjdOg5zi NIs25JSJhaYPcWqNLQhzFsJ9wUkz1nM5dGPjh3Sw+VSjGJzSdbv/N2EVUK3eqQPIhfx+ ySLypaKXx7Hz3VULDN6ju+QwoIl44r8Zzv/m2vAIAEiL5VM99GIymP4F5N1iW5Kti3gv dKeW5ffAHxoCKQDFYPoVYLzhTDq+XA0Ex635PUmF5dRrwk88zPUGgAJsREUq5tePkXxM 2G0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cfca+OB7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 20-20020a630314000000b0047882ff26basi2930185pgd.475.2022.12.13.18.53.46; Tue, 13 Dec 2022 18:54:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cfca+OB7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237054AbiLNCwf (ORCPT + 99 others); Tue, 13 Dec 2022 21:52:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236608AbiLNCwV (ORCPT ); Tue, 13 Dec 2022 21:52:21 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DADA17AB5 for ; Tue, 13 Dec 2022 18:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670986292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QJTsGV35CYOraEoRISqpKpr/2uwkfrZs9x/0odiJU0M=; b=cfca+OB7wUvQs+nnyPs5gOIEsfk3zUtlcqHM4lL17r//GII9ubIz5fkVI8epPrbtv94c0b QS3LwDYSuKwb7ZL1meRlHktYti/n4Jbkvr2pZDNSIek/0r0NL2AMFylT4q8XFf77UVfc8c MaggVteFr8nKv3IT2BzPxzUmWBQ/3qo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-279-J8NTIGfpNlG_jTgAlDFv2Q-1; Tue, 13 Dec 2022 21:51:31 -0500 X-MC-Unique: J8NTIGfpNlG_jTgAlDFv2Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C42023832786; Wed, 14 Dec 2022 02:51:30 +0000 (UTC) Received: from localhost (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id ECD2540ED784; Wed, 14 Dec 2022 02:51:29 +0000 (UTC) From: Ming Lei To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Zhong Jinghua , Yu Kuai , Dennis Zhou , Ming Lei Subject: [PATCH 3/3] lib/percpu-refcount: drain ->release() in perpcu_ref_exit() Date: Wed, 14 Dec 2022 10:51:01 +0800 Message-Id: <20221214025101.1268437-4-ming.lei@redhat.com> In-Reply-To: <20221214025101.1268437-1-ming.lei@redhat.com> References: <20221214025101.1268437-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752156282691933338?= X-GMAIL-MSGID: =?utf-8?q?1752156282691933338?= The pattern of wait_event(percpu_ref_is_zero()) has been used in several kernel components, and this way actually has the following risk: - percpu_ref_is_zero() can be returned just between atomic_long_sub_and_test() and ref->data->release(ref) - given the refcount is found as zero, percpu_ref_exit() could be called, and the host data structure is freed - then use-after-free is triggered in ->release() when the user host data structure is freed after percpu_ref_exit() returns Reported-by: Zhong Jinghua Fixes: 2b0d3d3e4fcf ("percpu_ref: reduce memory footprint of percpu_ref in fast path") Signed-off-by: Ming Lei --- include/linux/percpu-refcount.h | 41 ++++++++++++++++++++++----------- lib/percpu-refcount.c | 22 ++++++++++++++++++ 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h index 006c6aae261e..6ef29ebffd58 100644 --- a/include/linux/percpu-refcount.h +++ b/include/linux/percpu-refcount.h @@ -55,6 +55,7 @@ #include #include #include +#include struct percpu_ref; typedef void (percpu_ref_func_t)(struct percpu_ref *); @@ -104,6 +105,7 @@ struct percpu_ref_data { bool force_atomic:1; bool allow_reinit:1; bool auto_exit:1; + bool being_release:1; struct rcu_head rcu; struct percpu_ref *ref; }; @@ -137,6 +139,7 @@ void percpu_ref_kill_and_confirm(struct percpu_ref *ref, void percpu_ref_resurrect(struct percpu_ref *ref); void percpu_ref_reinit(struct percpu_ref *ref); bool percpu_ref_is_zero(struct percpu_ref *ref); +wait_queue_head_t *percpu_ref_get_switch_waitq(void); /** * percpu_ref_kill - drop the initial ref @@ -319,6 +322,29 @@ static inline bool percpu_ref_tryget_live(struct percpu_ref *ref) return ret; } +/* Internal helper, please do not call it outside */ +static inline void __percpu_ref_put_many(struct percpu_ref *ref, + unsigned long nr) +{ + struct percpu_ref_data *data = ref->data; + struct percpu_ref copy = *ref; + bool release = false; + + data->being_release = 1; + if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { + data->release(ref); + release = true; + } + data->being_release = 0; + + if (release) { + if (data->auto_exit) + percpu_ref_exit(©); + /* re-use switch waitq for ack the release done */ + wake_up_all(percpu_ref_get_switch_waitq()); + } +} + /** * percpu_ref_put_many - decrement a percpu refcount * @ref: percpu_ref to put @@ -337,19 +363,8 @@ static inline void percpu_ref_put_many(struct percpu_ref *ref, unsigned long nr) if (__ref_is_percpu(ref, &percpu_count)) this_cpu_sub(*percpu_count, nr); - else { - struct percpu_ref_data *data = ref->data; - struct percpu_ref copy = *ref; - bool release = false; - - if (unlikely(atomic_long_sub_and_test(nr, &data->count))) { - data->release(ref); - release = true; - } - - if (release && data->auto_exit) - percpu_ref_exit(©); - } + else + __percpu_ref_put_many(ref, nr); rcu_read_unlock(); } diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index c0cadf92948f..fd50eda233ed 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -140,6 +140,22 @@ void percpu_ref_exit(struct percpu_ref *ref) if (!data) return; + /* + * We may reach here because wait_event(percpu_ref_is_zero()) + * returns, and ->release() may not be completed or even started + * ye, then use-after-free is caused, so drain ->release() here + */ + if (!data->auto_exit) { + /* + * Order reading the atomic count in percpu_ref_is_zero + * and reading data->being_release. The counter pair is + * the one implied in atomic_long_sub_and_test() called + * from __percpu_ref_put_many(). + */ + smp_rmb(); + wait_event(percpu_ref_switch_waitq, !data->being_release); + } + spin_lock_irqsave(&percpu_ref_switch_lock, flags); ref->percpu_count_ptr |= atomic_long_read(&ref->data->count) << __PERCPU_REF_FLAG_BITS; @@ -480,3 +496,9 @@ void percpu_ref_resurrect(struct percpu_ref *ref) spin_unlock_irqrestore(&percpu_ref_switch_lock, flags); } EXPORT_SYMBOL_GPL(percpu_ref_resurrect); + +wait_queue_head_t *percpu_ref_get_switch_waitq() +{ + return &percpu_ref_switch_waitq; +} +EXPORT_SYMBOL_GPL(percpu_ref_get_switch_waitq);