From patchwork Fri Dec 8 00:52:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175494 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163179vqy; Thu, 7 Dec 2023 16:53:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IFKWeibv9sCOSxs5gxQ1LrDPORi+jBirGVz4nr9stJRmh8xXEH5zu7Z5pev/PQCoDizEQeU X-Received: by 2002:a05:6e02:1d81:b0:35d:5995:7994 with SMTP id h1-20020a056e021d8100b0035d59957994mr4455781ila.46.1701996797153; Thu, 07 Dec 2023 16:53:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996797; cv=none; d=google.com; s=arc-20160816; b=0zDmHHLMGPVntHs6buyvQ3sOLiL9atGt1U6j2fqpEEitJjEPCbvITsp+NqbiRSdn45 SXTepXfz6M0uFOx3TImY55G4dvzh1OEi+bz/EooRQEjiBNDAWdGVM1LGhXHV6kt2msTh uZdgpyl0+CiKzHk/yE90XIXkq1k92RXpYB6bkHDeD4xuqrEuZ6dBt09B8STekSuPSocl pqxgxEsSm9dZGxynnOZ+/gOq6XHfGAEDCiCKYyrhnp6289Icgp2E0sA6PHeIZV5nl5MU QZMrczN5XAGOKZtRTgputs7cnzIfWv/upjcXgO4l6j7FjQ86S8KJkgWa8K9/jzXcoVdg 1u9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=VK/mE0l6dF+pgEQ/a29RIXejus4h8m61K1NdvDJeNVZH2pgJVbvMul5JQQPgpqUP2F P/dDCAL8LNJ8OHtaTtS6AgJT3VMK1Cbu0IKo/cEKN+/Dg3X3uLWqVdly2QuZ5lD6saii grZMHw1KPzSimLNAkl0hrkeWoAhUSN5OrrjDhBjdY1JUqh9+jXsuJC1Iiyy0s9Aht3l5 7nzi0uQii3ON10Ziv2e25VJuasciXv+auuGPn514hjR5tCXgImi+an85YpMOe2TooMZF R5FUl1LrHHwhYmMee2jjyOs/3hwoe0o+NT5NVa8z4oCrBgGRQZttuVgC1tEgW1INV9h9 GNnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=04ViboTO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id l8-20020a6542c8000000b005b902b736f3si534347pgp.615.2023.12.07.16.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=04ViboTO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 1885080EE7F4; Thu, 7 Dec 2023 16:53:14 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444206AbjLHAw6 (ORCPT + 99 others); Thu, 7 Dec 2023 19:52:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444192AbjLHAwy (ORCPT ); Thu, 7 Dec 2023 19:52:54 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D32941737 for ; Thu, 7 Dec 2023 16:52:59 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d1b2153ba1so15605067b3.2 for ; Thu, 07 Dec 2023 16:52:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996779; x=1702601579; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; b=04ViboTOIUXyfwWbPzlEoZFF/dh03VXifu1JKxfqaSlKwEQiqXKJd4X8DHJrmkY2aL 97OtIgxuK3IVGuc49JAKnJVUG/knMMvGrPdklyJPCHl03IC35fZ9Wp4QvjLKs6Q/ukkK 7mMiLBwxGap9Pn5d/nedz+9YoYMlPvtgW6VmozGDNGBMMl1c/wW5m6DXQKIlVW2F+V4I K762D9kq+Wc0lPmU65Tt9pcEJXRpWJ181whoaXiw5AfB+NDam3uzB1a4nkznRdZ9PN/T OhhJKt1tQS2y8sqNIxWHXVWRUrv4JigxPx6ACvC4ccLeUpe3ndg7DksWqd0TcJDw43ey 53FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996779; x=1702601579; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; b=VZUJvS5EsoE4Ih1juNKa5bIGTsCoahaEo5eIQ8qOhqdIcfKWtqErGKSvV9bt0USnli FkXmWio7xao+WV2qAvD6eimRS3jXsfz3iCMuUBIXMQG0Y+vhbyXNS31REHfKABHAYiMq PCTo5qjZ62KUKMYzbhSzOHZrj8RGOK+OAeIOHbT+umAjv3UO0R4nEzusqFEmHu0fGfbP yTo54qz0clR72YLeap2TVHoxjPyLFJ72llEkMEhryY80yPfp7Ub4x/PPaucJtESu+Jtl iStXw+ZCgXwoLXPv5ovpOjGU4U+drIO0PF6pjoxgY9b16XJbzhZEl8gwEb7/OVTOHAXO KXew== X-Gm-Message-State: AOJu0YwIGhekqMz+EC2K9Al3YDSvVgLjUMtDaXPN4JsH0uzN+qNuu3jc f7uEE7UGqaFZLF04cXIW5wprDhNzv2EIkzhjtw== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:2893:b0:5d3:b449:e58e with SMTP id ee19-20020a05690c289300b005d3b449e58emr47985ywb.6.1701996778990; Thu, 07 Dec 2023 16:52:58 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:33 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-3-almasrymina@google.com> Subject: [net-next v1 02/16] net: page_pool: create hooks for custom page providers From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784672993356150981 X-GMAIL-MSGID: 1784672993356150981 From: Jakub Kicinski The page providers which try to reuse the same pages will need to hold onto the ref, even if page gets released from the pool - as in releasing the page from the pp just transfers the "ownership" reference from pp to the provider, and provider will wait for other references to be gone before feeding this page back into the pool. Signed-off-by: Jakub Kicinski Signed-off-by: Mina Almasry --- This is implemented by Jakub in his RFC: https://lore.kernel.org/netdev/f8270765-a27b-6ccf-33ea-cda097168d79@redhat.com/T/ I take no credit for the idea or implementation; I only added minor edits to make this workable with device memory TCP, and removed some hacky test code. This is a critical dependency of device memory TCP and thus I'm pulling it into this series to make it revewable and mergable. RFC v3 -> v1 - Removed unusued mem_provider. (Yunsheng). - Replaced memory_provider & mp_priv with netdev_rx_queue (Jakub). --- include/net/page_pool/types.h | 12 ++++++++++ net/core/page_pool.c | 43 +++++++++++++++++++++++++++++++---- 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index ac286ea8ce2d..0e9fa79a5ef1 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -51,6 +51,7 @@ struct pp_alloc_cache { * @dev: device, for DMA pre-mapping purposes * @netdev: netdev this pool will serve (leave as NULL if none or multiple) * @napi: NAPI which is the sole consumer of pages, otherwise NULL + * @queue: struct netdev_rx_queue this page_pool is being created for. * @dma_dir: DMA mapping direction * @max_len: max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV * @offset: DMA sync address offset for PP_FLAG_DMA_SYNC_DEV @@ -63,6 +64,7 @@ struct page_pool_params { int nid; struct device *dev; struct napi_struct *napi; + struct netdev_rx_queue *queue; enum dma_data_direction dma_dir; unsigned int max_len; unsigned int offset; @@ -125,6 +127,13 @@ struct page_pool_stats { }; #endif +struct memory_provider_ops { + int (*init)(struct page_pool *pool); + void (*destroy)(struct page_pool *pool); + struct page *(*alloc_pages)(struct page_pool *pool, gfp_t gfp); + bool (*release_page)(struct page_pool *pool, struct page *page); +}; + struct page_pool { struct page_pool_params_fast p; @@ -174,6 +183,9 @@ struct page_pool { */ struct ptr_ring ring; + void *mp_priv; + const struct memory_provider_ops *mp_ops; + #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ struct page_pool_recycle_stats __percpu *recycle_stats; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index ca1b3b65c9b5..f5c84d2a4510 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -25,6 +25,8 @@ #include "page_pool_priv.h" +static DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); + #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) @@ -174,6 +176,7 @@ static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params) { unsigned int ring_qsize = 1024; /* Default */ + int err; memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow)); @@ -234,10 +237,25 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); + if (pool->mp_ops) { + err = pool->mp_ops->init(pool); + if (err) { + pr_warn("%s() mem-provider init failed %d\n", + __func__, err); + goto free_ptr_ring; + } + + static_branch_inc(&page_pool_mem_providers); + } + if (pool->p.flags & PP_FLAG_DMA_MAP) get_device(pool->p.dev); return 0; + +free_ptr_ring: + ptr_ring_cleanup(&pool->ring, NULL); + return err; } static void page_pool_uninit(struct page_pool *pool) @@ -519,7 +537,10 @@ struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp) return page; /* Slow-path: cache empty, do real allocation */ - page = __page_pool_alloc_pages_slow(pool, gfp); + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + page = pool->mp_ops->alloc_pages(pool, gfp); + else + page = __page_pool_alloc_pages_slow(pool, gfp); return page; } EXPORT_SYMBOL(page_pool_alloc_pages); @@ -576,10 +597,13 @@ void __page_pool_release_page_dma(struct page_pool *pool, struct page *page) void page_pool_return_page(struct page_pool *pool, struct page *page) { int count; + bool put; - __page_pool_release_page_dma(pool, page); - - page_pool_clear_pp_info(page); + put = true; + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + put = pool->mp_ops->release_page(pool, page); + else + __page_pool_release_page_dma(pool, page); /* This may be the last page returned, releasing the pool, so * it is not safe to reference pool afterwards. @@ -587,7 +611,10 @@ void page_pool_return_page(struct page_pool *pool, struct page *page) count = atomic_inc_return_relaxed(&pool->pages_state_release_cnt); trace_page_pool_state_release(pool, page, count); - put_page(page); + if (put) { + page_pool_clear_pp_info(page); + put_page(page); + } /* An optimization would be to call __free_pages(page, pool->p.order) * knowing page is not part of page-cache (thus avoiding a * __page_cache_release() call). @@ -857,6 +884,12 @@ static void __page_pool_destroy(struct page_pool *pool) page_pool_unlist(pool); page_pool_uninit(pool); + + if (pool->mp_ops) { + pool->mp_ops->destroy(pool); + static_branch_dec(&page_pool_mem_providers); + } + kfree(pool); }