From patchwork Fri Dec 8 00:52:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175493 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163134vqy; Thu, 7 Dec 2023 16:53:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IEZwJNSI8G3xjkljJzCI/pkhKq8c953ZBaBc4Uy+egc7tjHkCyuGvyT6ICSjdviNe+nxGsU X-Received: by 2002:a17:90b:1d09:b0:286:bd9d:9997 with SMTP id on9-20020a17090b1d0900b00286bd9d9997mr3110409pjb.18.1701996788181; Thu, 07 Dec 2023 16:53:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996788; cv=none; d=google.com; s=arc-20160816; b=LjNejPtQsMNyp06n2LsX/jNDWaoj10IjW56yBGB/uHppc6xNi9qa+X06YRGT3HGRNa BBdR0anYAUV1fOE49PvsuQqezD1hYXH7I/Ahdle8zKADoUcE+DFUqSUQP9dDYCR5kvcF 6bVR2zJuf69YpfhW7sjcLMZYYITYpYD9fyyQTPa0MAzNbNXeahzMWLkPkikW97QU9jk3 +C51JgS4l08J4RRyKoicsOG/7gqsDXRvlK1M8gghcE2fhdc0wXY9XipHRD7H6Hk00PMX abGW6ZcZrQ57b53zxbQMh9yn6mbTkDIlf16ppRkhSfr+pgtOxmDYaA8TV4G9T2HcPWWV 1iqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=hfCMsfp1IUoZ64EVVtr7gxYSAUWCUtGUA5R8s2RxKzs=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=Ch7WP7UcDRt8Mma3UA8u1cmoIoLCwAfjyAXwk64za6FjjOLNcpQc9bfvwSSsTeUC7Z GEjcI2UC/9/hPVY/m4rjtDS+T1zejtOGkdKPYP714BuvXpxKVIgWEw3J56oARszsH6+6 L+AQzlp4sRA5PBL8JO+E8V6n/mxTANIgdn2EcDWAHBDi9mrH7aKIiJwFSTfWLCd20dqP wYp5xNBGX4yp9U6VyHUa3mwg34ohDJhaJHpjrwtpF3BAMLmkfBgDnno75w2vM7XrKFcA SgYTZkAeqWAV1IsHIVzoTNyr4oip2pOYna8fb9i7rU3pkjj8oyA9I0e6BMKuCdID03kv ZbAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XWkp78Jt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id a2-20020a17090abe0200b00286a1497c76si693143pjs.21.2023.12.07.16.53.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XWkp78Jt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 25BAE80DEA71; Thu, 7 Dec 2023 16:53:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444195AbjLHAwy (ORCPT + 99 others); Thu, 7 Dec 2023 19:52:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235574AbjLHAwv (ORCPT ); Thu, 7 Dec 2023 19:52:51 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF7F21721 for ; Thu, 7 Dec 2023 16:52:57 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d1ed4b268dso18386367b3.0 for ; Thu, 07 Dec 2023 16:52:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996777; x=1702601577; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hfCMsfp1IUoZ64EVVtr7gxYSAUWCUtGUA5R8s2RxKzs=; b=XWkp78Jt2QfvZg8KL7llBcvri5RjxOftzxLo0jx+7924tKzy++MPlqogZva2qqiyHY +bueHNbRUnR3T9ac90rBm7qI3/wDGbnJN/nP8rsklNS96u4ZWHZSFl+syYtRZ0VX77x3 4H+75nfKvExAlegJE/i8inw6F53Al++9/MEQy42vYTBYdujlyUcroYYZtih02yNAIzge 8n9USaAsYZMIKt5sMYTcgQ2a0AtrWPCAh6X7T6YCdxNsTCfWn+sGys4zJnbJwZRVveoz xBNWojBDl1810EOBaMMJh2y57X8lkYKiRr9DE5U1SXABZldjMU/S2aKk893QNxUF4YmV Pt0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996777; x=1702601577; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hfCMsfp1IUoZ64EVVtr7gxYSAUWCUtGUA5R8s2RxKzs=; b=j+6iJ7ABxSbzGJSTgKp+VVtC6qz4Ad7jGj1gmiMdxRwVy9wO2cV6bdjyCvOIEJZeBj lN+vup2yZ2Qt6h0cNxTgpeeQhjex8cBHvHK3abLi0HGCECzCJnLiFFd5eXAVFSkhCuEE hJelNOYCiI62c6mJmiqGWmmBNb2012/onCacLlm099IdJOZkg4Q3ETScOCCWDrehq6aB bOzW2pqv9GEwnMm946K3pcr/dO6PnxYB85zrsBzRwBAhNl6F23WRXOpKr2UWBmv22akw 37c6lxfX2jCjEV/WLFbNH8DF/0bys/nQ8jQVKaRZR2+XWj517TzJENZrvsVy55Km3ies q1Cg== X-Gm-Message-State: AOJu0YwDPqzeaihc0MV7CMb0kHOBPljrHCUGxtAcjZI1bYFbdd70gDnG v9HOen+w49MaH884n1E+40unWgIivm/jx3/zzA== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a81:441f:0:b0:5d5:5183:ebdb with SMTP id r31-20020a81441f000000b005d55183ebdbmr57205ywa.10.1701996776836; Thu, 07 Dec 2023 16:52:56 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:32 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-2-almasrymina@google.com> Subject: [net-next v1 01/16] net: page_pool: factor out releasing DMA from releasing the page From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784672983916037052 X-GMAIL-MSGID: 1784672983916037052 From: Jakub Kicinski Releasing the DMA mapping will be useful for other types of pages, so factor it out. Make sure compiler inlines it, to avoid any regressions. Signed-off-by: Jakub Kicinski Signed-off-by: Mina Almasry Reviewed-by: Shakeel Butt Reviewed-by: Ilias Apalodimas --- This is implemented by Jakub in his RFC: https://lore.kernel.org/netdev/f8270765-a27b-6ccf-33ea-cda097168d79@redhat.com/T/ I take no credit for the idea or implementation. This is a critical dependency of device memory TCP and thus I'm pulling it into this series to make it revewable and mergable. --- net/core/page_pool.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index c2e7c9a6efbe..ca1b3b65c9b5 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -548,21 +548,16 @@ s32 page_pool_inflight(const struct page_pool *pool, bool strict) return inflight; } -/* Disconnects a page (from a page_pool). API users can have a need - * to disconnect a page (from a page_pool), to allow it to be used as - * a regular page (that will eventually be returned to the normal - * page-allocator via put_page). - */ -static void page_pool_return_page(struct page_pool *pool, struct page *page) +static __always_inline +void __page_pool_release_page_dma(struct page_pool *pool, struct page *page) { dma_addr_t dma; - int count; if (!(pool->p.flags & PP_FLAG_DMA_MAP)) /* Always account for inflight pages, even if we didn't * map them */ - goto skip_dma_unmap; + return; dma = page_pool_get_dma_addr(page); @@ -571,7 +566,19 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page) PAGE_SIZE << pool->p.order, pool->p.dma_dir, DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); page_pool_set_dma_addr(page, 0); -skip_dma_unmap: +} + +/* Disconnects a page (from a page_pool). API users can have a need + * to disconnect a page (from a page_pool), to allow it to be used as + * a regular page (that will eventually be returned to the normal + * page-allocator via put_page). + */ +void page_pool_return_page(struct page_pool *pool, struct page *page) +{ + int count; + + __page_pool_release_page_dma(pool, page); + page_pool_clear_pp_info(page); /* This may be the last page returned, releasing the pool, so From patchwork Fri Dec 8 00:52:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175494 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163179vqy; Thu, 7 Dec 2023 16:53:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IFKWeibv9sCOSxs5gxQ1LrDPORi+jBirGVz4nr9stJRmh8xXEH5zu7Z5pev/PQCoDizEQeU X-Received: by 2002:a05:6e02:1d81:b0:35d:5995:7994 with SMTP id h1-20020a056e021d8100b0035d59957994mr4455781ila.46.1701996797153; Thu, 07 Dec 2023 16:53:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996797; cv=none; d=google.com; s=arc-20160816; b=0zDmHHLMGPVntHs6buyvQ3sOLiL9atGt1U6j2fqpEEitJjEPCbvITsp+NqbiRSdn45 SXTepXfz6M0uFOx3TImY55G4dvzh1OEi+bz/EooRQEjiBNDAWdGVM1LGhXHV6kt2msTh uZdgpyl0+CiKzHk/yE90XIXkq1k92RXpYB6bkHDeD4xuqrEuZ6dBt09B8STekSuPSocl pqxgxEsSm9dZGxynnOZ+/gOq6XHfGAEDCiCKYyrhnp6289Icgp2E0sA6PHeIZV5nl5MU QZMrczN5XAGOKZtRTgputs7cnzIfWv/upjcXgO4l6j7FjQ86S8KJkgWa8K9/jzXcoVdg 1u9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=VK/mE0l6dF+pgEQ/a29RIXejus4h8m61K1NdvDJeNVZH2pgJVbvMul5JQQPgpqUP2F P/dDCAL8LNJ8OHtaTtS6AgJT3VMK1Cbu0IKo/cEKN+/Dg3X3uLWqVdly2QuZ5lD6saii grZMHw1KPzSimLNAkl0hrkeWoAhUSN5OrrjDhBjdY1JUqh9+jXsuJC1Iiyy0s9Aht3l5 7nzi0uQii3ON10Ziv2e25VJuasciXv+auuGPn514hjR5tCXgImi+an85YpMOe2TooMZF R5FUl1LrHHwhYmMee2jjyOs/3hwoe0o+NT5NVa8z4oCrBgGRQZttuVgC1tEgW1INV9h9 GNnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=04ViboTO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id l8-20020a6542c8000000b005b902b736f3si534347pgp.615.2023.12.07.16.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=04ViboTO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 1885080EE7F4; Thu, 7 Dec 2023 16:53:14 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444206AbjLHAw6 (ORCPT + 99 others); Thu, 7 Dec 2023 19:52:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444192AbjLHAwy (ORCPT ); Thu, 7 Dec 2023 19:52:54 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D32941737 for ; Thu, 7 Dec 2023 16:52:59 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d1b2153ba1so15605067b3.2 for ; Thu, 07 Dec 2023 16:52:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996779; x=1702601579; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; b=04ViboTOIUXyfwWbPzlEoZFF/dh03VXifu1JKxfqaSlKwEQiqXKJd4X8DHJrmkY2aL 97OtIgxuK3IVGuc49JAKnJVUG/knMMvGrPdklyJPCHl03IC35fZ9Wp4QvjLKs6Q/ukkK 7mMiLBwxGap9Pn5d/nedz+9YoYMlPvtgW6VmozGDNGBMMl1c/wW5m6DXQKIlVW2F+V4I K762D9kq+Wc0lPmU65Tt9pcEJXRpWJ181whoaXiw5AfB+NDam3uzB1a4nkznRdZ9PN/T OhhJKt1tQS2y8sqNIxWHXVWRUrv4JigxPx6ACvC4ccLeUpe3ndg7DksWqd0TcJDw43ey 53FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996779; x=1702601579; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=igrDQ25kHpMvPF+Js5oOYPKq9BDscOjehO2W0ZO2Cow=; b=VZUJvS5EsoE4Ih1juNKa5bIGTsCoahaEo5eIQ8qOhqdIcfKWtqErGKSvV9bt0USnli FkXmWio7xao+WV2qAvD6eimRS3jXsfz3iCMuUBIXMQG0Y+vhbyXNS31REHfKABHAYiMq PCTo5qjZ62KUKMYzbhSzOHZrj8RGOK+OAeIOHbT+umAjv3UO0R4nEzusqFEmHu0fGfbP yTo54qz0clR72YLeap2TVHoxjPyLFJ72llEkMEhryY80yPfp7Ub4x/PPaucJtESu+Jtl iStXw+ZCgXwoLXPv5ovpOjGU4U+drIO0PF6pjoxgY9b16XJbzhZEl8gwEb7/OVTOHAXO KXew== X-Gm-Message-State: AOJu0YwIGhekqMz+EC2K9Al3YDSvVgLjUMtDaXPN4JsH0uzN+qNuu3jc f7uEE7UGqaFZLF04cXIW5wprDhNzv2EIkzhjtw== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:2893:b0:5d3:b449:e58e with SMTP id ee19-20020a05690c289300b005d3b449e58emr47985ywb.6.1701996778990; Thu, 07 Dec 2023 16:52:58 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:33 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-3-almasrymina@google.com> Subject: [net-next v1 02/16] net: page_pool: create hooks for custom page providers From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784672993356150981 X-GMAIL-MSGID: 1784672993356150981 From: Jakub Kicinski The page providers which try to reuse the same pages will need to hold onto the ref, even if page gets released from the pool - as in releasing the page from the pp just transfers the "ownership" reference from pp to the provider, and provider will wait for other references to be gone before feeding this page back into the pool. Signed-off-by: Jakub Kicinski Signed-off-by: Mina Almasry --- This is implemented by Jakub in his RFC: https://lore.kernel.org/netdev/f8270765-a27b-6ccf-33ea-cda097168d79@redhat.com/T/ I take no credit for the idea or implementation; I only added minor edits to make this workable with device memory TCP, and removed some hacky test code. This is a critical dependency of device memory TCP and thus I'm pulling it into this series to make it revewable and mergable. RFC v3 -> v1 - Removed unusued mem_provider. (Yunsheng). - Replaced memory_provider & mp_priv with netdev_rx_queue (Jakub). --- include/net/page_pool/types.h | 12 ++++++++++ net/core/page_pool.c | 43 +++++++++++++++++++++++++++++++---- 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index ac286ea8ce2d..0e9fa79a5ef1 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -51,6 +51,7 @@ struct pp_alloc_cache { * @dev: device, for DMA pre-mapping purposes * @netdev: netdev this pool will serve (leave as NULL if none or multiple) * @napi: NAPI which is the sole consumer of pages, otherwise NULL + * @queue: struct netdev_rx_queue this page_pool is being created for. * @dma_dir: DMA mapping direction * @max_len: max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV * @offset: DMA sync address offset for PP_FLAG_DMA_SYNC_DEV @@ -63,6 +64,7 @@ struct page_pool_params { int nid; struct device *dev; struct napi_struct *napi; + struct netdev_rx_queue *queue; enum dma_data_direction dma_dir; unsigned int max_len; unsigned int offset; @@ -125,6 +127,13 @@ struct page_pool_stats { }; #endif +struct memory_provider_ops { + int (*init)(struct page_pool *pool); + void (*destroy)(struct page_pool *pool); + struct page *(*alloc_pages)(struct page_pool *pool, gfp_t gfp); + bool (*release_page)(struct page_pool *pool, struct page *page); +}; + struct page_pool { struct page_pool_params_fast p; @@ -174,6 +183,9 @@ struct page_pool { */ struct ptr_ring ring; + void *mp_priv; + const struct memory_provider_ops *mp_ops; + #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ struct page_pool_recycle_stats __percpu *recycle_stats; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index ca1b3b65c9b5..f5c84d2a4510 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -25,6 +25,8 @@ #include "page_pool_priv.h" +static DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); + #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) @@ -174,6 +176,7 @@ static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params) { unsigned int ring_qsize = 1024; /* Default */ + int err; memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow)); @@ -234,10 +237,25 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); + if (pool->mp_ops) { + err = pool->mp_ops->init(pool); + if (err) { + pr_warn("%s() mem-provider init failed %d\n", + __func__, err); + goto free_ptr_ring; + } + + static_branch_inc(&page_pool_mem_providers); + } + if (pool->p.flags & PP_FLAG_DMA_MAP) get_device(pool->p.dev); return 0; + +free_ptr_ring: + ptr_ring_cleanup(&pool->ring, NULL); + return err; } static void page_pool_uninit(struct page_pool *pool) @@ -519,7 +537,10 @@ struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp) return page; /* Slow-path: cache empty, do real allocation */ - page = __page_pool_alloc_pages_slow(pool, gfp); + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + page = pool->mp_ops->alloc_pages(pool, gfp); + else + page = __page_pool_alloc_pages_slow(pool, gfp); return page; } EXPORT_SYMBOL(page_pool_alloc_pages); @@ -576,10 +597,13 @@ void __page_pool_release_page_dma(struct page_pool *pool, struct page *page) void page_pool_return_page(struct page_pool *pool, struct page *page) { int count; + bool put; - __page_pool_release_page_dma(pool, page); - - page_pool_clear_pp_info(page); + put = true; + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + put = pool->mp_ops->release_page(pool, page); + else + __page_pool_release_page_dma(pool, page); /* This may be the last page returned, releasing the pool, so * it is not safe to reference pool afterwards. @@ -587,7 +611,10 @@ void page_pool_return_page(struct page_pool *pool, struct page *page) count = atomic_inc_return_relaxed(&pool->pages_state_release_cnt); trace_page_pool_state_release(pool, page, count); - put_page(page); + if (put) { + page_pool_clear_pp_info(page); + put_page(page); + } /* An optimization would be to call __free_pages(page, pool->p.order) * knowing page is not part of page-cache (thus avoiding a * __page_cache_release() call). @@ -857,6 +884,12 @@ static void __page_pool_destroy(struct page_pool *pool) page_pool_unlist(pool); page_pool_uninit(pool); + + if (pool->mp_ops) { + pool->mp_ops->destroy(pool); + static_branch_dec(&page_pool_mem_providers); + } + kfree(pool); } From patchwork Fri Dec 8 00:52:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175496 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163261vqy; Thu, 7 Dec 2023 16:53:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFe2ufli1YMCTK+F0RNez99dZ/lGqEk806EIf/bEZZ6rcz2CmRgKWguEFzbT99qpJazBkCL X-Received: by 2002:a05:6a20:8f02:b0:190:93e:f224 with SMTP id b2-20020a056a208f0200b00190093ef224mr1984757pzk.76.1701996809477; Thu, 07 Dec 2023 16:53:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996809; cv=none; d=google.com; s=arc-20160816; b=Jwn/Dzf2njBFWc38tPUvson6q+urfYiR5mrj1RvavpXOYcA0lQNy3qdxbfouA8PfNs G10jyaSyDXtX/Y2yKqcwI1YLlhCTgqQSnJGuPpEQtFz1oafG0rWeKRC++l2eyxbpT5WK q9pi+1AqOHpgdFNN/opOhutI2luiLBiWEf2rORZMhNp2GFB+aMrbKwDhL+c6B8G52Ptg XcH6tsoj8L4eO7I61IAKuaL5WHeKKwDRbxh61FLHr1yJpu55jgeTeLnUK4Sc6tZQQjgQ Vh1uT7/fqTyvMWEy9I94MkWgppEI8Iasegbz1XgAps0jmofIsbpRagpB+B+8Bp7X6pFp rcWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=V1r8ZXlyktFk/hR6LvKbeClNyv61T/bWHclepM8Rp88=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=fiArZHkvErDZiNIr61OGRMEoEMg3HlvfHrPgmVAYCDs2IIttBWGmtnVE7l3zrDM7c2 G16xqaORosG5/Vc2iBEHgg6Ur2h/QShl1r196PLjaS4ynre2gs0vMtmYmY21/pCImqhs BtbWGGOONFipS3RpBI1wFaoZuU083INySzjZkKNY6s+0rBc+jf5NH6RY8ysXAfW9dVN4 Wk0tqYOX1G32LvuKe/eCbS0PGzlZUozn3qU9/d8lHdQhkB4GPmKLTgYZzMLiTQiiP4CJ HoAtWkWxd8fkpUdOtQUsWYW9LMzk+DwQ650a6dZdjJbuNpD5fMyvbCvkmSmUMLuyzHcT 4hTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nKnwUW+O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id h18-20020a656392000000b005b11e5a69fdsi541177pgv.508.2023.12.07.16.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nKnwUW+O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 8BABB815CDBD; Thu, 7 Dec 2023 16:53:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235659AbjLHAxH (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444201AbjLHAwz (ORCPT ); Thu, 7 Dec 2023 19:52:55 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1AE3172E for ; Thu, 7 Dec 2023 16:53:01 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-db3ef4c7094so2057011276.1 for ; Thu, 07 Dec 2023 16:53:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996781; x=1702601581; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V1r8ZXlyktFk/hR6LvKbeClNyv61T/bWHclepM8Rp88=; b=nKnwUW+ODKImfMaS+HQrnvNEyZvj4JFMXrS8bZ2Yj2CEkAWfRhNkOcAbSNZryzCV40 IEP3lXnGr0DPXlGGLhlUv9I9HMPS4JPnRE7WCrvUzpjlrtvyTGHZ21wStkG0XrDx2NMW UxP6Ekq10yozd0+HCZ2rraY2EqsiLLQf7nZUnVP6fq27YMRIRrL3OzLJnG1+wUA+pcXO vd6SaMDoFYSUNiRcyaZnRvWOhdHS3Y3Lw/f3oaqITRv1LXh3YNJumLBbszFEM4o6mo34 MaFYf2i0lX6QKtI2a/1XI8KLfwQUm79xAeSyNffz3LXeA1Otv48DStwIYBlBzvrjFvl2 oJGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996781; x=1702601581; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V1r8ZXlyktFk/hR6LvKbeClNyv61T/bWHclepM8Rp88=; b=ltuMLv5RoapCmKZYNZehf0RHABdrv1wGLbPPrL6Pyjmr7oyh1yoRkJ6v1/hKVuL/GF sG/3/cMetMWrcKyw3+7R3DyThzXliIY9W3CQLnC/ODLCMABv6RSI2MU5oR9A7ujSeYRw d/lTgFfZMCgv6dhoW55j6kHXsGbyEIs+4ctHNqN7PTNFhn6mnpi0b6HzARvLCL86JI9r hpEiVv44FcP3uF6GsQbGGJiG0jK8A2uNhH0pw8AR1oTL1xEoqRgAD9CZ8I8Tzp/vW10H 7Z6+gahPOVlhij7QsC+BlcUX3vrKt0B9+sFsvIiHDJZ4SuhX4MFnDjsvZJnbrJrPRyys 58aQ== X-Gm-Message-State: AOJu0YxOIF7BJEf8Lmccl5zMrCmrEOYgSQ4TOZahxOackmsrjyMe3/4a sl0UnMab3Zk+OFN8sLRBeb9NRMr4+IrbKgW21w== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:cc4b:0:b0:db5:3bdf:ff55 with SMTP id l72-20020a25cc4b000000b00db53bdfff55mr39860ybf.6.1701996781080; Thu, 07 Dec 2023 16:53:01 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:34 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-4-almasrymina@google.com> Subject: [net-next v1 03/16] queue_api: define queue api From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673006663388071 X-GMAIL-MSGID: 1784673006663388071 This API enables the net stack to reset the queues used for devmem. Signed-off-by: Mina Almasry --- include/linux/netdevice.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 1b935ee341b4..316f7dee86ce 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1432,6 +1432,20 @@ struct netdev_net_notifier { * struct kernel_hwtstamp_config *kernel_config, * struct netlink_ext_ack *extack); * Change the hardware timestamping parameters for NIC device. + * + * void *(*ndo_queue_mem_alloc)(struct net_device *dev, int idx); + * Allocate memory for an RX queue. The memory returned in the form of + * a void * can be passed to ndo_queue_mem_free() for freeing or to + * ndo_queue_start to create an RX queue with this memory. + * + * void (*ndo_queue_mem_free)(struct net_device *dev, void *); + * Free memory from an RX queue. + * + * int (*ndo_queue_start)(struct net_device *dev, int idx, void *); + * Start an RX queue at the specified index. + * + * int (*ndo_queue_stop)(struct net_device *dev, int idx, void **); + * Stop the RX queue at the specified index. */ struct net_device_ops { int (*ndo_init)(struct net_device *dev); @@ -1673,6 +1687,16 @@ struct net_device_ops { int (*ndo_hwtstamp_set)(struct net_device *dev, struct kernel_hwtstamp_config *kernel_config, struct netlink_ext_ack *extack); + void * (*ndo_queue_mem_alloc)(struct net_device *dev, + int idx); + void (*ndo_queue_mem_free)(struct net_device *dev, + void *queue_mem); + int (*ndo_queue_start)(struct net_device *dev, + int idx, + void *queue_mem); + int (*ndo_queue_stop)(struct net_device *dev, + int idx, + void **out_queue_mem); }; /** From patchwork Fri Dec 8 00:52:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175495 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163254vqy; Thu, 7 Dec 2023 16:53:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IFQhMJW9ZZSfOL8hoRhbuPuC1g1A6Deg/D0ULofSpnvHaKyp0CdBjtx0Hi7LBYDNDVCk0uy X-Received: by 2002:a17:90a:c901:b0:286:6cc0:cadd with SMTP id v1-20020a17090ac90100b002866cc0caddmr3270187pjt.84.1701996807586; Thu, 07 Dec 2023 16:53:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996807; cv=none; d=google.com; s=arc-20160816; b=0BVlJwMqbdCGvYCWxTpe8j3LKM4F0RPAK9XpysgOE6aucIIiIaPnLsCzsP0toQ6J2j 71s0gh/VwuUFKhCKPXLyjggkYI9Rx9iZPDLUA1HNMlfF6Pq8lGo9U5C/JxgA6qooccGB 3F/o61MSYgs6dwdjRBuyvj6E6bNFMXOzWMGfcb3U3SyZ/DSD2s0NZ3/i2y2fOnqEDDe1 2iFba2NDEUryDPDSFioAxp3QKOykrJtyd3MnY8u5NAwa9RJOrnwR5Xve2ge1fyLjscKO ZdbuL31LP7pKTRcsbXUjJCFpX8GBUa5TRxwwtmi2w74ugKxZ5HnyxmfwTHe03p+4FYJd HIVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=nhWZE+QRUt+Z8lRUloVgvBF4+Y7IPBx9Q6LUmHNzq1c=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=BYktvO5k2+CHJM+aY+6N332gWmIS8OIPpET98o2TO+B38fRdmpjameAfviVIUocUDA ms5kSxjOlUEoYk5H6KtkXijcnw1FObG5v0vhvSP6IUuSYknqXE3iX1Um5W9udSv51gtk XvyLVKAfovttwspbCVSBzARA3MzdgiE3CSEObh/IW6uwa+glwcSwKMZMHkj6nMe6fPUY TJ1AcP0tzpSPE7hHBoj2pqOW/TaJSl2B+f8NwA9ERgc4Dvdg8qq289wElsLS6b8e1inQ RGJmiezSJBbrstC5eRWkWTW4Ei+ZAkDkYjeQqVU+2/wLipFeREOsj3aYeVc8X18oxClP o5oA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=rkcfCe2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id h16-20020a17090aa89000b002869b4e7d1bsi657232pjq.96.2023.12.07.16.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=rkcfCe2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A06A4853A0E0; Thu, 7 Dec 2023 16:53:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444215AbjLHAxJ (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229531AbjLHAxF (ORCPT ); Thu, 7 Dec 2023 19:53:05 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06643172A for ; Thu, 7 Dec 2023 16:53:04 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dbc4f389835so332527276.2 for ; Thu, 07 Dec 2023 16:53:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996783; x=1702601583; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nhWZE+QRUt+Z8lRUloVgvBF4+Y7IPBx9Q6LUmHNzq1c=; b=rkcfCe2CbupevsyPtC8BxPibduKqF2oeMzVt2UEQVdP/TefH3ZDW9KbjAvoYgtpFUK tDm4J2vHNiRK2BwBUgNhW3uh+sZ8mG+VUdVxGON87ERVbiD39qwODXJCj+/v2yscUmo5 qYdph509uT4HkUoLQh/lp7e0iCBqASHsy9wLVZBRIQsSvqnN++gEhVNqfp9P+CusVvPc 2BogCbkV+yFTMbTr7Fxrro3WS7/J9HD2dqK9BUOgeBlrondIMi7NAwPbHZ66rRq9djGj v6N5IBewPpAHC5/TwOz5aJPkweA1XuUU9ynjnXlcOddJeydoQASnYakLJmlxNGnyiJNb sZFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996783; x=1702601583; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nhWZE+QRUt+Z8lRUloVgvBF4+Y7IPBx9Q6LUmHNzq1c=; b=wRdexUZJ6aHuiZnwcJ3wRSzFCFZ86QFhp+YXRTGVAU2wKUk4Lm9k1M4E2RLAVwQuy7 9ds/ASgmBZ8MXCrtExtNDWnjQc1dI+DAO2w4MdWG8O4j8iT0eZnJYpN+uC5MG0G/+PZC r5Pw2aTs4L3hZg6X0M2clSOz5md8s5c95J/j3Km2dLAw69Vz7E1FjyL0CWnBlUmcSNAk sfYao4GGTqP2dpGeTsMYH2fVw1lEn+L0TIwkZFOoTHxAkcN363xUzkdEGRHUFCzKi6lJ GNi5LEECpaRnrf1Dc2Xb+Uf7cJfR+qx8kRG0RBJhZ7dI7AClTkUGtb14jC1rVpY/oFqv Q+wg== X-Gm-Message-State: AOJu0YxOCKPB2nHbA1Jn+ylpV3KFTpyjUDe+Y5NlH9JQcJ2V/Au6m7Y3 gWpN9uqmx/IGCty3qi2Nc/l6IQVR83XWO1qlBg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:e097:0:b0:db3:fa34:50b0 with SMTP id x145-20020a25e097000000b00db3fa3450b0mr38904ybg.4.1701996783271; Thu, 07 Dec 2023 16:53:03 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:35 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-5-almasrymina@google.com> Subject: [net-next v1 04/16] gve: implement queue api From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673004260581447 X-GMAIL-MSGID: 1784673004260581447 Define a struct that contains all of the memory needed for an RX queue to function. Implement the queue-api in GVE using this struct. Currently the only memory is allocated at the time of queue start are the RX pages in gve_rx_post_buffers_dqo(). That can be moved up to queue_mem_alloc() time in the future. For simplicity the queue API is only supported by the diorite queue out-of-order (DQO) format without queue-page-lists (QPL). Support for other GVE formats can be added in the future as well. Signed-off-by: Mina Almasry --- drivers/net/ethernet/google/gve/gve_adminq.c | 6 +- drivers/net/ethernet/google/gve/gve_adminq.h | 3 + drivers/net/ethernet/google/gve/gve_dqo.h | 2 + drivers/net/ethernet/google/gve/gve_main.c | 286 +++++++++++++++++++ drivers/net/ethernet/google/gve/gve_rx_dqo.c | 5 +- 5 files changed, 296 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index 12fbd723ecc6..e515b7278295 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -348,7 +348,7 @@ static int gve_adminq_parse_err(struct gve_priv *priv, u32 status) /* Flushes all AQ commands currently queued and waits for them to complete. * If there are failures, it will return the first error. */ -static int gve_adminq_kick_and_wait(struct gve_priv *priv) +int gve_adminq_kick_and_wait(struct gve_priv *priv) { int tail, head; int i; @@ -591,7 +591,7 @@ int gve_adminq_create_tx_queues(struct gve_priv *priv, u32 start_id, u32 num_que return gve_adminq_kick_and_wait(priv); } -static int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index) +int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index) { struct gve_rx_ring *rx = &priv->rx[queue_index]; union gve_adminq_command cmd; @@ -691,7 +691,7 @@ int gve_adminq_destroy_tx_queues(struct gve_priv *priv, u32 start_id, u32 num_qu return gve_adminq_kick_and_wait(priv); } -static int gve_adminq_destroy_rx_queue(struct gve_priv *priv, u32 queue_index) +int gve_adminq_destroy_rx_queue(struct gve_priv *priv, u32 queue_index) { union gve_adminq_command cmd; int err; diff --git a/drivers/net/ethernet/google/gve/gve_adminq.h b/drivers/net/ethernet/google/gve/gve_adminq.h index 5865ccdccbd0..265beed965dc 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.h +++ b/drivers/net/ethernet/google/gve/gve_adminq.h @@ -411,6 +411,7 @@ union gve_adminq_command { static_assert(sizeof(union gve_adminq_command) == 64); +int gve_adminq_kick_and_wait(struct gve_priv *priv); int gve_adminq_alloc(struct device *dev, struct gve_priv *priv); void gve_adminq_free(struct device *dev, struct gve_priv *priv); void gve_adminq_release(struct gve_priv *priv); @@ -424,7 +425,9 @@ int gve_adminq_deconfigure_device_resources(struct gve_priv *priv); int gve_adminq_create_tx_queues(struct gve_priv *priv, u32 start_id, u32 num_queues); int gve_adminq_destroy_tx_queues(struct gve_priv *priv, u32 start_id, u32 num_queues); int gve_adminq_create_rx_queues(struct gve_priv *priv, u32 num_queues); +int gve_adminq_create_rx_queue(struct gve_priv *priv, u32 queue_index); int gve_adminq_destroy_rx_queues(struct gve_priv *priv, u32 queue_id); +int gve_adminq_destroy_rx_queue(struct gve_priv *priv, u32 queue_id); int gve_adminq_register_page_list(struct gve_priv *priv, struct gve_queue_page_list *qpl); int gve_adminq_unregister_page_list(struct gve_priv *priv, u32 page_list_id); diff --git a/drivers/net/ethernet/google/gve/gve_dqo.h b/drivers/net/ethernet/google/gve/gve_dqo.h index c36b93f0de15..3eed26a0ed7d 100644 --- a/drivers/net/ethernet/google/gve/gve_dqo.h +++ b/drivers/net/ethernet/google/gve/gve_dqo.h @@ -46,6 +46,8 @@ int gve_clean_tx_done_dqo(struct gve_priv *priv, struct gve_tx_ring *tx, struct napi_struct *napi); void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx); void gve_rx_write_doorbell_dqo(const struct gve_priv *priv, int queue_idx); +void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs, + bool free_page); static inline void gve_tx_put_doorbell_dqo(const struct gve_priv *priv, diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 619bf63ec935..5b23d811afd3 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -22,6 +22,7 @@ #include "gve_dqo.h" #include "gve_adminq.h" #include "gve_register.h" +#include "gve_utils.h" #define GVE_DEFAULT_RX_COPYBREAK (256) @@ -1702,6 +1703,287 @@ static int gve_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } +struct gve_per_rx_queue_mem_dqo { + struct gve_rx_buf_state_dqo *buf_states; + u32 num_buf_states; + + struct gve_rx_compl_desc_dqo *complq_desc_ring; + dma_addr_t complq_bus; + + struct gve_rx_desc_dqo *bufq_desc_ring; + dma_addr_t bufq_bus; + + struct gve_queue_resources *q_resources; + dma_addr_t q_resources_bus; + + size_t completion_queue_slots; + size_t buffer_queue_slots; +}; + +static int gve_rx_queue_stop(struct net_device *dev, int idx, + void **out_per_q_mem) +{ + struct gve_per_rx_queue_mem_dqo *per_q_mem; + struct gve_priv *priv = netdev_priv(dev); + struct gve_notify_block *block; + struct gve_rx_ring *rx; + int ntfy_idx; + int err; + + rx = &priv->rx[idx]; + ntfy_idx = gve_rx_idx_to_ntfy(priv, idx); + block = &priv->ntfy_blocks[ntfy_idx]; + + if (priv->queue_format != GVE_DQO_RDA_FORMAT) + return -EOPNOTSUPP; + + if (!out_per_q_mem) + return -EINVAL; + + /* Stopping queue 0 while other queues are running is unfortunately + * fails silently for GVE at the moment. Disable the queue-api for + * queue 0 until this is resolved. + */ + if (idx == 0) + return -ERANGE; + + per_q_mem = kvcalloc(1, sizeof(*per_q_mem), GFP_KERNEL); + if (!per_q_mem) + return -ENOMEM; + + napi_disable(&block->napi); + err = gve_adminq_destroy_rx_queue(priv, idx); + if (err) + goto err_napi_enable; + + err = gve_adminq_kick_and_wait(priv); + if (err) + goto err_create_rx_queue; + + gve_remove_napi(priv, ntfy_idx); + + per_q_mem->buf_states = rx->dqo.buf_states; + per_q_mem->num_buf_states = rx->dqo.num_buf_states; + + per_q_mem->complq_desc_ring = rx->dqo.complq.desc_ring; + per_q_mem->complq_bus = rx->dqo.complq.bus; + + per_q_mem->bufq_desc_ring = rx->dqo.bufq.desc_ring; + per_q_mem->bufq_bus = rx->dqo.bufq.bus; + + per_q_mem->q_resources = rx->q_resources; + per_q_mem->q_resources_bus = rx->q_resources_bus; + + per_q_mem->buffer_queue_slots = rx->dqo.bufq.mask + 1; + per_q_mem->completion_queue_slots = rx->dqo.complq.mask + 1; + + *out_per_q_mem = per_q_mem; + + return 0; + +err_create_rx_queue: + /* There is nothing we can do here if these fail. */ + gve_adminq_create_rx_queue(priv, idx); + gve_adminq_kick_and_wait(priv); + +err_napi_enable: + napi_enable(&block->napi); + kvfree(per_q_mem); + + return err; +} + +static void gve_rx_queue_mem_free(struct net_device *dev, void *per_q_mem) +{ + struct gve_per_rx_queue_mem_dqo *gve_q_mem; + struct gve_priv *priv = netdev_priv(dev); + struct gve_rx_buf_state_dqo *bs; + struct device *hdev; + size_t size; + int i; + + priv = netdev_priv(dev); + gve_q_mem = (struct gve_per_rx_queue_mem_dqo *)per_q_mem; + hdev = &priv->pdev->dev; + + if (!gve_q_mem) + return; + + if (priv->queue_format != GVE_DQO_RDA_FORMAT) + return; + + for (i = 0; i < gve_q_mem->num_buf_states; i++) { + bs = &gve_q_mem->buf_states[i]; + if (bs->page_info.page) + gve_free_page_dqo(priv, bs, true); + } + + if (gve_q_mem->q_resources) + dma_free_coherent(hdev, sizeof(*gve_q_mem->q_resources), + gve_q_mem->q_resources, + gve_q_mem->q_resources_bus); + + if (gve_q_mem->bufq_desc_ring) { + size = sizeof(gve_q_mem->bufq_desc_ring[0]) * + gve_q_mem->buffer_queue_slots; + dma_free_coherent(hdev, size, gve_q_mem->bufq_desc_ring, + gve_q_mem->bufq_bus); + } + + if (gve_q_mem->complq_desc_ring) { + size = sizeof(gve_q_mem->complq_desc_ring[0]) * + gve_q_mem->completion_queue_slots; + dma_free_coherent(hdev, size, gve_q_mem->complq_desc_ring, + gve_q_mem->complq_bus); + } + + kvfree(gve_q_mem->buf_states); + + kvfree(per_q_mem); +} + +static void *gve_rx_queue_mem_alloc(struct net_device *dev, int idx) +{ + struct gve_per_rx_queue_mem_dqo *gve_q_mem; + struct gve_priv *priv = netdev_priv(dev); + struct device *hdev = &priv->pdev->dev; + size_t size; + + if (priv->queue_format != GVE_DQO_RDA_FORMAT) + return NULL; + + /* See comment in gve_rx_queue_stop() */ + if (idx == 0) + return NULL; + + gve_q_mem = kvcalloc(1, sizeof(*gve_q_mem), GFP_KERNEL); + if (!gve_q_mem) + goto err; + + gve_q_mem->buffer_queue_slots = + priv->options_dqo_rda.rx_buff_ring_entries; + gve_q_mem->completion_queue_slots = priv->rx_desc_cnt; + + gve_q_mem->num_buf_states = + min_t(s16, S16_MAX, gve_q_mem->buffer_queue_slots * 4); + + gve_q_mem->buf_states = kvcalloc(gve_q_mem->num_buf_states, + sizeof(gve_q_mem->buf_states[0]), + GFP_KERNEL); + if (!gve_q_mem->buf_states) + goto err; + + size = sizeof(struct gve_rx_compl_desc_dqo) * + gve_q_mem->completion_queue_slots; + gve_q_mem->complq_desc_ring = dma_alloc_coherent(hdev, size, + &gve_q_mem->complq_bus, + GFP_KERNEL); + if (!gve_q_mem->complq_desc_ring) + goto err; + + size = sizeof(struct gve_rx_desc_dqo) * gve_q_mem->buffer_queue_slots; + gve_q_mem->bufq_desc_ring = dma_alloc_coherent(hdev, size, + &gve_q_mem->bufq_bus, + GFP_KERNEL); + if (!gve_q_mem->bufq_desc_ring) + goto err; + + gve_q_mem->q_resources = dma_alloc_coherent(hdev, + sizeof(*gve_q_mem->q_resources), + &gve_q_mem->q_resources_bus, + GFP_KERNEL); + if (!gve_q_mem->q_resources) + goto err; + + return gve_q_mem; + +err: + gve_rx_queue_mem_free(dev, gve_q_mem); + return NULL; +} + +static int gve_rx_queue_start(struct net_device *dev, int idx, void *per_q_mem) +{ + struct gve_per_rx_queue_mem_dqo *gve_q_mem; + struct gve_priv *priv = netdev_priv(dev); + struct gve_rx_ring *rx = &priv->rx[idx]; + struct gve_notify_block *block; + int ntfy_idx; + int err; + int i; + + if (priv->queue_format != GVE_DQO_RDA_FORMAT) + return -EOPNOTSUPP; + + /* See comment in gve_rx_queue_stop() */ + if (idx == 0) + return -ERANGE; + + gve_q_mem = (struct gve_per_rx_queue_mem_dqo *)per_q_mem; + ntfy_idx = gve_rx_idx_to_ntfy(priv, idx); + block = &priv->ntfy_blocks[ntfy_idx]; + + netif_dbg(priv, drv, priv->dev, "starting rx ring DQO\n"); + + memset(rx, 0, sizeof(*rx)); + rx->gve = priv; + rx->q_num = idx; + rx->dqo.bufq.mask = gve_q_mem->buffer_queue_slots - 1; + rx->dqo.complq.num_free_slots = gve_q_mem->completion_queue_slots; + rx->dqo.complq.mask = gve_q_mem->completion_queue_slots - 1; + rx->ctx.skb_head = NULL; + rx->ctx.skb_tail = NULL; + + rx->dqo.num_buf_states = gve_q_mem->num_buf_states; + + rx->dqo.buf_states = gve_q_mem->buf_states; + + /* Set up linked list of buffer IDs */ + for (i = 0; i < rx->dqo.num_buf_states - 1; i++) + rx->dqo.buf_states[i].next = i + 1; + + rx->dqo.buf_states[rx->dqo.num_buf_states - 1].next = -1; + rx->dqo.recycled_buf_states.head = -1; + rx->dqo.recycled_buf_states.tail = -1; + rx->dqo.used_buf_states.head = -1; + rx->dqo.used_buf_states.tail = -1; + + rx->dqo.complq.desc_ring = gve_q_mem->complq_desc_ring; + rx->dqo.complq.bus = gve_q_mem->complq_bus; + + rx->dqo.bufq.desc_ring = gve_q_mem->bufq_desc_ring; + rx->dqo.bufq.bus = gve_q_mem->bufq_bus; + + rx->q_resources = gve_q_mem->q_resources; + rx->q_resources_bus = gve_q_mem->q_resources_bus; + + gve_rx_add_to_block(priv, idx); + + err = gve_adminq_create_rx_queue(priv, idx); + if (err) + return err; + + err = gve_adminq_kick_and_wait(priv); + if (err) + goto err_destroy_rx_queue; + + /* TODO, pull the memory allocations in this to gve_rx_queue_mem_alloc() + */ + gve_rx_post_buffers_dqo(&priv->rx[idx]); + + napi_enable(&block->napi); + gve_set_itr_coalesce_usecs_dqo(priv, block, priv->rx_coalesce_usecs); + + return 0; + +err_destroy_rx_queue: + /* There is nothing we can do if these fail. */ + gve_adminq_destroy_rx_queue(priv, idx); + gve_adminq_kick_and_wait(priv); + + return err; +} + int gve_adjust_queues(struct gve_priv *priv, struct gve_queue_config new_rx_config, struct gve_queue_config new_tx_config) @@ -1900,6 +2182,10 @@ static const struct net_device_ops gve_netdev_ops = { .ndo_bpf = gve_xdp, .ndo_xdp_xmit = gve_xdp_xmit, .ndo_xsk_wakeup = gve_xsk_wakeup, + .ndo_queue_mem_alloc = gve_rx_queue_mem_alloc, + .ndo_queue_mem_free = gve_rx_queue_mem_free, + .ndo_queue_start = gve_rx_queue_start, + .ndo_queue_stop = gve_rx_queue_stop, }; static void gve_handle_status(struct gve_priv *priv, u32 status) diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c index f281e42a7ef9..e729f04d3f60 100644 --- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c +++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c @@ -21,9 +21,8 @@ static int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs) return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias; } -static void gve_free_page_dqo(struct gve_priv *priv, - struct gve_rx_buf_state_dqo *bs, - bool free_page) +void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs, + bool free_page) { page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1); if (free_page) From patchwork Fri Dec 8 00:52:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175500 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163424vqy; Thu, 7 Dec 2023 16:53:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IFVPO4yDwrV7pYPjClZNbxoDFhMY7TBYALo8c0XEFdgZfZp9/VIwfE8E8B+K+3raV7x4Fjf X-Received: by 2002:a05:6808:3d4:b0:3b9:e153:a610 with SMTP id o20-20020a05680803d400b003b9e153a610mr1505902oie.59.1701996835977; Thu, 07 Dec 2023 16:53:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996835; cv=none; d=google.com; s=arc-20160816; b=oNvcnzUI/K01vWDjTb8D0Ba8cN+qiYdeCXF4O7+zp9x2PZBQzVoLZwV62w/roGlNc9 OTh2/AyDBxGR8lxtrDkGGvpgzuPv8fdL1djdNtq2ffHgvk610lF9YeaZ2ykGm7dK8Y7R EYkr5+DjggVHnkF19aBG3YStOHoB/mzblkCMyF/VqgX/4YUiXY7GJ/bokUVLtSlcpK6k SysLGV4p6e9PyMHytjw5E0StE8bxTI+5ZkZZREbv6ya71aVKZA5IJemdnW/BNzOy7wce 0J4utvP66LluSUjCWGob2U/Acbj+eKhLtTAhJJbeOmzUfoAMYKhDi5EQbmtLHKEGszWm sH0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=aoV0tDbNZgYvivQmYBKAP9GCcy/xYSu5tcLB8R1UGG0=; fh=dQ0Z1OOFK5ChiAlIZaZLnTEZu41VePMcBmOe3O0ONkE=; b=Ewd0KYeLnArSY9LvxH3L5h/cLgebqq0WXAXfGLf79d5YTPO1MEJWSuJmXLgDimA1rh rcMZpAIyr+r/mm4EHWWRa6SmQEwl9EFJLk2RvSkIXtjstme5LouWKhSO4xUJOS29/v+9 4BKDR3d9GQTddhrF1br0ULAQQ7It0DdrxBJuux7pgC53L5nsNhMeMrUUCz3JBChpsawK DFRCGHSrBkd5JUu0dpIdMXHPj2207xSttAhI8I5nZFCY18althHRf0X2y6XGUv02uxga E5x4HzZ1ZBH8xvoE+ZviBU7STaboh949H6cknwCFtEToXHVE06Pgq59ODVsEC90JloG4 WxWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=VeAzctvT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id x29-20020a056a00189d00b006cb997a5f83si576999pfh.31.2023.12.07.16.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=VeAzctvT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 2D9AB8228815; Thu, 7 Dec 2023 16:53:47 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230225AbjLHAxM (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235639AbjLHAxF (ORCPT ); Thu, 7 Dec 2023 19:53:05 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE266173E for ; Thu, 7 Dec 2023 16:53:05 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d7e7e10231so9953137b3.1 for ; Thu, 07 Dec 2023 16:53:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996785; x=1702601585; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aoV0tDbNZgYvivQmYBKAP9GCcy/xYSu5tcLB8R1UGG0=; b=VeAzctvT1kcS8w5gV/5/iAZvKWCS4Eh1NZboSist2gYXvzP+4bfJmEGNu2ljTmU8Tj lm83lQKNgFs1DAIIdJlTn3ETMug/o8n8k3a+BTug7cE5M7Vz8Tur0uQss6ZH2gaXNyDS 2cmS6jK/ovrzsVqicpl+EPe3FIFFlarY+RSAEMOhriuEy5vjjN4hrBmLzR4OYPOJY4vg 645Iq/RuFhqRFLMSqiTydphJicdLlH4je3gkmsB67fDI7fDhRnNLyzIQYNU2PIPHgp8r yrhHYqg97U1tlPMnk/owoAcSofETreXdj2MXh+rGAMOJ2BefNdiio+cU3RXmowHwDMm+ Pdjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996785; x=1702601585; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aoV0tDbNZgYvivQmYBKAP9GCcy/xYSu5tcLB8R1UGG0=; b=CU6yjFo1cPuv3/BnlR4RfwGgSWsHGhSFZ75KRieIxkb5r1yvIzJR1RbVjDJoUvqUWU Zs4dSERONgE5ASddK7s2xskKkJ028/ihQLAzOfpsPhJJIY0aZrOs+5S0h9FA1y0Hc9xB 5nYlmLNERkBGmFpGcDVXGi8GthU0VBzVxI9FuhGTjXUIqOEROJv29IeyvyIhazD8qMqj CuLfDOutgQBdAHVyKzAGfJ5QNNhnjQygzJTjERdCHE1Im8FQO2CqmlaMQacx3ibzEJbK 0S3mYbfV0f0IRQuAuO8MLsdHvpXLzOBv3vsyWvfPK6U0xHvDMmehyNPFlG3GumFIUfDQ NtPA== X-Gm-Message-State: AOJu0YyDlyFQHNydE6HOCFvi1Of3dfMIa53zhx7aVleCOoby1RXzLIAX 3TW0JIM6x1cRRnY76HPLTo0ORMXORaTiC5Gwmg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:c1c:b0:5d9:452e:d653 with SMTP id cl28-20020a05690c0c1c00b005d9452ed653mr2146ywb.5.1701996785141; Thu, 07 Dec 2023 16:53:05 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:36 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-6-almasrymina@google.com> Subject: [net-next v1 05/16] net: netdev netlink api to bind dma-buf to a net device From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Stanislav Fomichev X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673034353409532 X-GMAIL-MSGID: 1784673034353409532 API takes the dma-buf fd as input, and binds it to the netdevice. The user can specify the rx queues to bind the dma-buf to. Suggested-by: Stanislav Fomichev Signed-off-by: Mina Almasry --- Changes in v1: - Add rx-queue-type to distingish rx from tx (Jakub) - Return dma-buf ID from netlink API (David, Stan) Changes in RFC-v3: - Support binding multiple rx rx-queues --- Documentation/netlink/specs/netdev.yaml | 52 +++++++++++++++++++++++++ include/uapi/linux/netdev.h | 19 +++++++++ net/core/netdev-genl-gen.c | 19 +++++++++ net/core/netdev-genl-gen.h | 2 + net/core/netdev-genl.c | 6 +++ tools/include/uapi/linux/netdev.h | 19 +++++++++ 6 files changed, 117 insertions(+) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml index f2c76d103bd8..df6a11d47006 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -260,6 +260,45 @@ attribute-sets: name: napi-id doc: ID of the NAPI instance which services this queue. type: u32 + - + name: queue-dmabuf + attributes: + - + name: type + doc: rx or tx queue + type: u8 + enum: queue-type + - + name: idx + doc: queue index + type: u32 + + - + name: bind-dmabuf + attributes: + - + name: ifindex + doc: netdev ifindex to bind the dma-buf to. + type: u32 + checks: + min: 1 + - + name: queues + doc: receive queues to bind the dma-buf to. + type: nest + nested-attributes: queue-dmabuf + multi-attr: true + - + name: dmabuf-fd + doc: dmabuf file descriptor to bind. + type: u32 + - + name: dmabuf-id + doc: id of the dmabuf binding + type: u32 + checks: + min: 1 + operations: list: @@ -382,6 +421,19 @@ operations: attributes: - ifindex reply: *queue-get-op + - + name: bind-rx + doc: Bind dmabuf to netdev + attribute-set: bind-dmabuf + do: + request: + attributes: + - ifindex + - dmabuf-fd + - queues + reply: + attributes: + - dmabuf-id - name: napi-get doc: Get information about NAPI instances configured on the system. diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index 424c5e28f495..35d201dc4b05 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -129,6 +129,24 @@ enum { NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX - 1) }; +enum { + NETDEV_A_QUEUE_DMABUF_TYPE = 1, + NETDEV_A_QUEUE_DMABUF_IDX, + + __NETDEV_A_QUEUE_DMABUF_MAX, + NETDEV_A_QUEUE_DMABUF_MAX = (__NETDEV_A_QUEUE_DMABUF_MAX - 1) +}; + +enum { + NETDEV_A_BIND_DMABUF_IFINDEX = 1, + NETDEV_A_BIND_DMABUF_QUEUES, + NETDEV_A_BIND_DMABUF_DMABUF_FD, + NETDEV_A_BIND_DMABUF_DMABUF_ID, + + __NETDEV_A_BIND_DMABUF_MAX, + NETDEV_A_BIND_DMABUF_MAX = (__NETDEV_A_BIND_DMABUF_MAX - 1) +}; + enum { NETDEV_CMD_DEV_GET = 1, NETDEV_CMD_DEV_ADD_NTF, @@ -140,6 +158,7 @@ enum { NETDEV_CMD_PAGE_POOL_CHANGE_NTF, NETDEV_CMD_PAGE_POOL_STATS_GET, NETDEV_CMD_QUEUE_GET, + NETDEV_CMD_BIND_RX, NETDEV_CMD_NAPI_GET, __NETDEV_CMD_MAX, diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index be7f2ebd61b2..3384b1ae3f40 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -27,6 +27,11 @@ const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFIND [NETDEV_A_PAGE_POOL_IFINDEX] = NLA_POLICY_FULL_RANGE(NLA_U32, &netdev_a_page_pool_ifindex_range), }; +const struct nla_policy netdev_queue_dmabuf_nl_policy[NETDEV_A_QUEUE_DMABUF_IDX + 1] = { + [NETDEV_A_QUEUE_DMABUF_TYPE] = NLA_POLICY_MAX(NLA_U8, 1), + [NETDEV_A_QUEUE_DMABUF_IDX] = { .type = NLA_U32, }, +}; + /* NETDEV_CMD_DEV_GET - do */ static const struct nla_policy netdev_dev_get_nl_policy[NETDEV_A_DEV_IFINDEX + 1] = { [NETDEV_A_DEV_IFINDEX] = NLA_POLICY_MIN(NLA_U32, 1), @@ -58,6 +63,13 @@ static const struct nla_policy netdev_queue_get_dump_nl_policy[NETDEV_A_QUEUE_IF [NETDEV_A_QUEUE_IFINDEX] = NLA_POLICY_MIN(NLA_U32, 1), }; +/* NETDEV_CMD_BIND_RX - do */ +static const struct nla_policy netdev_bind_rx_nl_policy[NETDEV_A_BIND_DMABUF_DMABUF_FD + 1] = { + [NETDEV_A_BIND_DMABUF_IFINDEX] = NLA_POLICY_MIN(NLA_U32, 1), + [NETDEV_A_BIND_DMABUF_DMABUF_FD] = { .type = NLA_U32, }, + [NETDEV_A_BIND_DMABUF_QUEUES] = NLA_POLICY_NESTED(netdev_queue_dmabuf_nl_policy), +}; + /* NETDEV_CMD_NAPI_GET - do */ static const struct nla_policy netdev_napi_get_do_nl_policy[NETDEV_A_NAPI_ID + 1] = { [NETDEV_A_NAPI_ID] = { .type = NLA_U32, }, @@ -124,6 +136,13 @@ static const struct genl_split_ops netdev_nl_ops[] = { .maxattr = NETDEV_A_QUEUE_IFINDEX, .flags = GENL_CMD_CAP_DUMP, }, + { + .cmd = NETDEV_CMD_BIND_RX, + .doit = netdev_nl_bind_rx_doit, + .policy = netdev_bind_rx_nl_policy, + .maxattr = NETDEV_A_BIND_DMABUF_DMABUF_FD, + .flags = GENL_CMD_CAP_DO, + }, { .cmd = NETDEV_CMD_NAPI_GET, .doit = netdev_nl_napi_get_doit, diff --git a/net/core/netdev-genl-gen.h b/net/core/netdev-genl-gen.h index a47f2bcbe4fa..a7ede514eccd 100644 --- a/net/core/netdev-genl-gen.h +++ b/net/core/netdev-genl-gen.h @@ -13,6 +13,7 @@ /* Common nested types */ extern const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PAGE_POOL_IFINDEX + 1]; +extern const struct nla_policy netdev_queue_dmabuf_nl_policy[NETDEV_A_QUEUE_DMABUF_IDX + 1]; int netdev_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb); @@ -26,6 +27,7 @@ int netdev_nl_page_pool_stats_get_dumpit(struct sk_buff *skb, int netdev_nl_queue_get_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_queue_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb); +int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_napi_get_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_napi_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb); diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index fd98936da3ae..0ed292d87ae0 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -469,6 +469,12 @@ int netdev_nl_queue_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) return skb->len; } +/* Stub */ +int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info) +{ + return 0; +} + static int netdev_genl_netdevice_event(struct notifier_block *nb, unsigned long event, void *ptr) { diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/netdev.h index 424c5e28f495..35d201dc4b05 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -129,6 +129,24 @@ enum { NETDEV_A_QUEUE_MAX = (__NETDEV_A_QUEUE_MAX - 1) }; +enum { + NETDEV_A_QUEUE_DMABUF_TYPE = 1, + NETDEV_A_QUEUE_DMABUF_IDX, + + __NETDEV_A_QUEUE_DMABUF_MAX, + NETDEV_A_QUEUE_DMABUF_MAX = (__NETDEV_A_QUEUE_DMABUF_MAX - 1) +}; + +enum { + NETDEV_A_BIND_DMABUF_IFINDEX = 1, + NETDEV_A_BIND_DMABUF_QUEUES, + NETDEV_A_BIND_DMABUF_DMABUF_FD, + NETDEV_A_BIND_DMABUF_DMABUF_ID, + + __NETDEV_A_BIND_DMABUF_MAX, + NETDEV_A_BIND_DMABUF_MAX = (__NETDEV_A_BIND_DMABUF_MAX - 1) +}; + enum { NETDEV_CMD_DEV_GET = 1, NETDEV_CMD_DEV_ADD_NTF, @@ -140,6 +158,7 @@ enum { NETDEV_CMD_PAGE_POOL_CHANGE_NTF, NETDEV_CMD_PAGE_POOL_STATS_GET, NETDEV_CMD_QUEUE_GET, + NETDEV_CMD_BIND_RX, NETDEV_CMD_NAPI_GET, __NETDEV_CMD_MAX, From patchwork Fri Dec 8 00:52:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175498 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163362vqy; Thu, 7 Dec 2023 16:53:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IHApTwoyVxI9HYC1DOfxQu/uA91Owl6YQnvaq7ikrFxiej4o7Xk5qWMlLHPQs7FnDZK/+fT X-Received: by 2002:a05:6358:7f01:b0:169:9c45:ca12 with SMTP id p1-20020a0563587f0100b001699c45ca12mr4845120rwn.23.1701996825593; Thu, 07 Dec 2023 16:53:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996825; cv=none; d=google.com; s=arc-20160816; b=btno+VjuAvK0BG/Ayob96NIQGkbLb4tupKdfMfeDZChi7vMyryeLV+ydtJSTlgMAuG H9Tehv4vJGYPNLF5c8hUpxUwE3zX+6TZ3EHhzMPiSc7WzewrI04VTe6y1G7UDdXvVSdR v0nWlxmKf1/cj6R8xANIuTnuzH6KmPI78LRwCWQQuGAluxora3TF7JkQi88Tnf4niQDs jfge5ZJYM2IE9623wq/4DtfwotRktRQor5XzLfPXasxxupafOyEL8apzadBmU3vtCzy2 LIc+gVhA9tKET1/++TcDdX/jwKACJwBhxm1x69UWBuBQCpg+pGu9lscPMMuAcVCYk6ek TPWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=IJSREQV89hlugaW+sm/AZT6Rr4mGIq6Ap66sB5d+pEY=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=gqKInTFu9ieilNZwoZ9PYFBg0lhQWwDUAd8+7ZCTg6mMTPrH15eTxc8+9Xay1cDp+c DQjVVzHy3qjbbPZ5wnIcE4GhsXlLVYXqCBunU4H9YhIezRIGFdkp66ohfBwYKuVN0qy4 CUotufAtcoZ0G9kAyYrf/fZvfZmo4EL6RGT2fkVBrOf+RlJ3QQhP3onxiDAdycTXrvsf tI/2R2TMsIYCosE2ml6l3275PvtURwLfNDIn/5OvOshsEI3tyQowxL/fG0wHMrDj4I7O V2lcL5z25EjKN94TAH+lPM68DW3dSKwQ26Dsxp41C/qrN6nWIH7qfATC0isjPFrSTmWM yI2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=q8qOBwMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id n128-20020a632786000000b005b11ce9c97esi552572pgn.353.2023.12.07.16.53.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=q8qOBwMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id E26B18108BC8; Thu, 7 Dec 2023 16:53:38 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572888AbjLHAxP (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235614AbjLHAxG (ORCPT ); Thu, 7 Dec 2023 19:53:06 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 460331738 for ; Thu, 7 Dec 2023 16:53:08 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-daee86e2d70so2099610276.0 for ; Thu, 07 Dec 2023 16:53:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996787; x=1702601587; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IJSREQV89hlugaW+sm/AZT6Rr4mGIq6Ap66sB5d+pEY=; b=q8qOBwMVDo+EhxalRraVbQIFxhroHw+/hJVeuCzPAxpTRkLGepzEHgOkjTc5r8+X/e dhZcdCs45zYqXAldsPXBTlt7GGMKqdGKVB8tjf7uoGE5jcezHR+goh9l8gM7wveJdeGx Gc3nnfU2CO7Uh8RsBYFuQYhxMVaiO/ObEcVuANEsywJ0Mf9Rb6ZrDaoCM9HmR5o5AnCl YtlYg2JXgA1korBkbGG0W3IVdZFRTmBdKy+NpKXBkiTOyamHGxrbnOcny3eM8qyKLfQ8 R7VzOJmyQX/FjXT1CpQUNwjomAAkW0m7Fboa46LtEHtECVTb6ZlPlDvjA2uYwi6bxVRW WjgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996787; x=1702601587; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IJSREQV89hlugaW+sm/AZT6Rr4mGIq6Ap66sB5d+pEY=; b=qgXaQ2HTS0zv2hjPUsV/r6fnJ6kMBmGA2SAJy1OrEXO7o5qbTa+JAv/BMj17VsEwF1 iD7nw2IDK+6anmg1qgi0mSmhhTlBfC/XIYmo6XO8nGQQNLpu3cmuxlsPnEj0CiyokOXd W/SGfOE9lwAkg3iI2MKFblKrnZg6dobYa01qOIFdKEIrDk8r0SyBEiEAQd7MMXhAmbk8 YKDpOX6RAdw876Hd67uwDS+VS67m1VzB95X0xDoMN7JQUQ1JLRrfbVEWKJw81LY0kSkX 8YPR1CfcMppSkCgidt4pv961QX3c1hUr4jROXO2HHvJK9JWDapDM0EJQD61CLXfWYZyi RmJg== X-Gm-Message-State: AOJu0YzKATSchoxaQpGjnCVittQRtWsBgWGEJ3fMf3q+/k0KRa9FITYL msoZ/sX2TaDdPEAGqijY89D9ckwLDH77mKbmog== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:d84e:0:b0:db3:5b0a:f274 with SMTP id p75-20020a25d84e000000b00db35b0af274mr47808ybg.0.1701996787378; Thu, 07 Dec 2023 16:53:07 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:37 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-7-almasrymina@google.com> Subject: [net-next v1 06/16] netdev: support binding dma-buf to netdevice From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:39 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673023081464898 X-GMAIL-MSGID: 1784673023081464898 Add a netdev_dmabuf_binding struct which represents the dma-buf-to-netdevice binding. The netlink API will bind the dma-buf to rx queues on the netdevice. On the binding, the dma_buf_attach & dma_buf_map_attachment will occur. The entries in the sg_table from mapping will be inserted into a genpool to make it ready for allocation. The chunks in the genpool are owned by a dmabuf_chunk_owner struct which holds the dma-buf offset of the base of the chunk and the dma_addr of the chunk. Both are needed to use allocations that come from this chunk. We create a new type that represents an allocation from the genpool: page_pool_iov. We setup the page_pool_iov allocation size in the genpool to PAGE_SIZE for simplicity: to match the PAGE_SIZE normally allocated by the page pool and given to the drivers. The user can unbind the dmabuf from the netdevice by closing the netlink socket that established the binding. We do this so that the binding is automatically unbound even if the userspace process crashes. The binding and unbinding leaves an indicator in struct netdev_rx_queue that the given queue is bound, but the binding doesn't take effect until the driver actually reconfigures its queues, and re-initializes its page pool. The netdev_dmabuf_binding struct is refcounted, and releases its resources only when all the refs are released. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- v1: - Introduce devmem.h instead of bloating netdevice.h (Jakub) - ENOTSUPP -> EOPNOTSUPP (checkpatch.pl I think) - Remove unneeded rcu protection for binding->list (rtnl protected) - Removed extraneous err_binding_put: label. - Removed dma_addr += len (Paolo). - Don't override err on netdev_bind_dmabuf_to_queue failure. - Rename devmem -> dmabuf (David). - Add id to dmabuf binding (David/Stan). - Fix missing xa_destroy bound_rq_list. - Use queue api to reset bound RX queues (Jakub). - Update netlink API for rx-queue type (tx/re) (Jakub). RFC v3: - Support multi rx-queue binding --- include/net/devmem.h | 96 ++++++++++++ include/net/netdev_rx_queue.h | 1 + include/net/page_pool/types.h | 27 ++++ net/core/dev.c | 276 ++++++++++++++++++++++++++++++++++ net/core/netdev-genl.c | 122 ++++++++++++++- 5 files changed, 520 insertions(+), 2 deletions(-) create mode 100644 include/net/devmem.h diff --git a/include/net/devmem.h b/include/net/devmem.h new file mode 100644 index 000000000000..29ff125f9815 --- /dev/null +++ b/include/net/devmem.h @@ -0,0 +1,96 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Device memory TCP support + * + * Authors: Mina Almasry + * Willem de Bruijn + * Kaiyuan Zhang + * + */ +#ifndef _NET_DEVMEM_H +#define _NET_DEVMEM_H + +struct netdev_dmabuf_binding { + struct dma_buf *dmabuf; + struct dma_buf_attachment *attachment; + struct sg_table *sgt; + struct net_device *dev; + struct gen_pool *chunk_pool; + + /* The user holds a ref (via the netlink API) for as long as they want + * the binding to remain alive. Each page pool using this binding holds + * a ref to keep the binding alive. Each allocated page_pool_iov holds a + * ref. + * + * The binding undos itself and unmaps the underlying dmabuf once all + * those refs are dropped and the binding is no longer desired or in + * use. + */ + refcount_t ref; + + /* The portid of the user that owns this binding. Used for netlink to + * notify us of the user dropping the bind. + */ + u32 owner_nlportid; + + /* The list of bindings currently active. Used for netlink to notify us + * of the user dropping the bind. + */ + struct list_head list; + + /* rxq's this binding is active on. */ + struct xarray bound_rxq_list; + + /* ID of this binding. Globally unique to all bindings currently + * active. + */ + u32 id; +}; + +#ifdef CONFIG_DMA_SHARED_BUFFER +void __netdev_dmabuf_binding_free(struct netdev_dmabuf_binding *binding); +int netdev_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + struct netdev_dmabuf_binding **out); +void netdev_unbind_dmabuf(struct netdev_dmabuf_binding *binding); +int netdev_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct netdev_dmabuf_binding *binding); +#else +static inline void +__netdev_dmabuf_binding_free(struct netdev_dmabuf_binding *binding) +{ +} + +static inline int netdev_bind_dmabuf(struct net_device *dev, + unsigned int dmabuf_fd, + struct netdev_dmabuf_binding **out) +{ + return -EOPNOTSUPP; +} +static inline void netdev_unbind_dmabuf(struct netdev_dmabuf_binding *binding) +{ +} + +static inline int +netdev_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct netdev_dmabuf_binding *binding) +{ + return -EOPNOTSUPP; +} +#endif + +static inline void +netdev_dmabuf_binding_get(struct netdev_dmabuf_binding *binding) +{ + refcount_inc(&binding->ref); +} + +static inline void +netdev_dmabuf_binding_put(struct netdev_dmabuf_binding *binding) +{ + if (!refcount_dec_and_test(&binding->ref)) + return; + + __netdev_dmabuf_binding_free(binding); +} + +#endif /* _NET_DEVMEM_H */ diff --git a/include/net/netdev_rx_queue.h b/include/net/netdev_rx_queue.h index aa1716fb0e53..5dc35628633a 100644 --- a/include/net/netdev_rx_queue.h +++ b/include/net/netdev_rx_queue.h @@ -25,6 +25,7 @@ struct netdev_rx_queue { * Readers and writers must hold RTNL */ struct napi_struct *napi; + struct netdev_dmabuf_binding *binding; } ____cacheline_aligned_in_smp; /* diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 0e9fa79a5ef1..44faee7a7b02 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -134,6 +134,33 @@ struct memory_provider_ops { bool (*release_page)(struct page_pool *pool, struct page *page); }; +/* page_pool_iov support */ + +/* Owner of the dma-buf chunks inserted into the gen pool. Each scatterlist + * entry from the dmabuf is inserted into the genpool as a chunk, and needs + * this owner struct to keep track of some metadata necessary to create + * allocations from this chunk. + */ +struct dmabuf_genpool_chunk_owner { + /* Offset into the dma-buf where this chunk starts. */ + unsigned long base_virtual; + + /* dma_addr of the start of the chunk. */ + dma_addr_t base_dma_addr; + + /* Array of page_pool_iovs for this chunk. */ + struct page_pool_iov *ppiovs; + size_t num_ppiovs; + + struct netdev_dmabuf_binding *binding; +}; + +struct page_pool_iov { + struct dmabuf_genpool_chunk_owner *owner; + + refcount_t refcount; +}; + struct page_pool { struct page_pool_params_fast p; diff --git a/net/core/dev.c b/net/core/dev.c index 0432b04cf9b0..b8c8be5a912e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -153,6 +153,10 @@ #include #include #include +#include +#include +#include +#include #include "dev.h" #include "net-sysfs.h" @@ -2041,6 +2045,278 @@ static int call_netdevice_notifiers_mtu(unsigned long val, return call_netdevice_notifiers_info(val, &info.info); } +/* Device memory support */ + +#ifdef CONFIG_DMA_SHARED_BUFFER +static void netdev_dmabuf_free_chunk_owner(struct gen_pool *genpool, + struct gen_pool_chunk *chunk, + void *not_used) +{ + struct dmabuf_genpool_chunk_owner *owner = chunk->owner; + + kvfree(owner->ppiovs); + kfree(owner); +} + +void __netdev_dmabuf_binding_free(struct netdev_dmabuf_binding *binding) +{ + size_t size, avail; + + gen_pool_for_each_chunk(binding->chunk_pool, + netdev_dmabuf_free_chunk_owner, NULL); + + size = gen_pool_size(binding->chunk_pool); + avail = gen_pool_avail(binding->chunk_pool); + + if (!WARN(size != avail, "can't destroy genpool. size=%lu, avail=%lu", + size, avail)) + gen_pool_destroy(binding->chunk_pool); + + dma_buf_unmap_attachment(binding->attachment, binding->sgt, + DMA_BIDIRECTIONAL); + dma_buf_detach(binding->dmabuf, binding->attachment); + dma_buf_put(binding->dmabuf); + xa_destroy(&binding->bound_rxq_list); + kfree(binding); +} + +static int netdev_restart_rx_queue(struct net_device *dev, int rxq_idx) +{ + void *new_mem; + void *old_mem; + int err; + + if (!dev || !dev->netdev_ops) + return -EINVAL; + + if (!dev->netdev_ops->ndo_queue_stop || + !dev->netdev_ops->ndo_queue_mem_free || + !dev->netdev_ops->ndo_queue_mem_alloc || + !dev->netdev_ops->ndo_queue_start) + return -EOPNOTSUPP; + + new_mem = dev->netdev_ops->ndo_queue_mem_alloc(dev, rxq_idx); + if (!new_mem) + return -ENOMEM; + + err = dev->netdev_ops->ndo_queue_stop(dev, rxq_idx, &old_mem); + if (err) + goto err_free_new_mem; + + err = dev->netdev_ops->ndo_queue_start(dev, rxq_idx, new_mem); + if (err) + goto err_start_queue; + + dev->netdev_ops->ndo_queue_mem_free(dev, old_mem); + + return 0; + +err_start_queue: + dev->netdev_ops->ndo_queue_start(dev, rxq_idx, old_mem); + +err_free_new_mem: + dev->netdev_ops->ndo_queue_mem_free(dev, new_mem); + + return err; +} + +/* Protected by rtnl_lock() */ +static DEFINE_XARRAY_FLAGS(netdev_dmabuf_bindings, XA_FLAGS_ALLOC1); + +void netdev_unbind_dmabuf(struct netdev_dmabuf_binding *binding) +{ + struct netdev_rx_queue *rxq; + unsigned long xa_idx; + unsigned int rxq_idx; + + if (!binding) + return; + + if (binding->list.next) + list_del(&binding->list); + + xa_for_each(&binding->bound_rxq_list, xa_idx, rxq) { + if (rxq->binding == binding) { + /* We hold the rtnl_lock while binding/unbinding + * dma-buf, so we can't race with another thread that + * is also modifying this value. However, the driver + * may read this config while it's creating its + * rx-queues. WRITE_ONCE() here to match the + * READ_ONCE() in the driver. + */ + WRITE_ONCE(rxq->binding, NULL); + + rxq_idx = get_netdev_rx_queue_index(rxq); + + netdev_restart_rx_queue(binding->dev, rxq_idx); + } + } + + xa_erase(&netdev_dmabuf_bindings, binding->id); + + netdev_dmabuf_binding_put(binding); +} + +int netdev_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct netdev_dmabuf_binding *binding) +{ + struct netdev_rx_queue *rxq; + u32 xa_idx; + int err; + + rxq = __netif_get_rx_queue(dev, rxq_idx); + + if (rxq->binding) + return -EEXIST; + + err = xa_alloc(&binding->bound_rxq_list, &xa_idx, rxq, xa_limit_32b, + GFP_KERNEL); + if (err) + return err; + + /* We hold the rtnl_lock while binding/unbinding dma-buf, so we can't + * race with another thread that is also modifying this value. However, + * the driver may read this config while it's creating its * rx-queues. + * WRITE_ONCE() here to match the READ_ONCE() in the driver. + */ + WRITE_ONCE(rxq->binding, binding); + + err = netdev_restart_rx_queue(dev, rxq_idx); + if (err) + goto err_xa_erase; + + return 0; + +err_xa_erase: + xa_erase(&binding->bound_rxq_list, xa_idx); + WRITE_ONCE(rxq->binding, NULL); + + return err; +} + +int netdev_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + struct netdev_dmabuf_binding **out) +{ + struct netdev_dmabuf_binding *binding; + static u32 id_alloc_next; + struct scatterlist *sg; + struct dma_buf *dmabuf; + unsigned int sg_idx, i; + unsigned long virtual; + int err; + + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + dmabuf = dma_buf_get(dmabuf_fd); + if (IS_ERR_OR_NULL(dmabuf)) + return -EBADFD; + + binding = kzalloc_node(sizeof(*binding), GFP_KERNEL, + dev_to_node(&dev->dev)); + if (!binding) { + err = -ENOMEM; + goto err_put_dmabuf; + } + binding->dev = dev; + + err = xa_alloc_cyclic(&netdev_dmabuf_bindings, &binding->id, binding, + xa_limit_32b, &id_alloc_next, GFP_KERNEL); + if (err < 0) + goto err_free_binding; + + xa_init_flags(&binding->bound_rxq_list, XA_FLAGS_ALLOC); + + refcount_set(&binding->ref, 1); + + binding->dmabuf = dmabuf; + + binding->attachment = dma_buf_attach(binding->dmabuf, dev->dev.parent); + if (IS_ERR(binding->attachment)) { + err = PTR_ERR(binding->attachment); + goto err_free_id; + } + + binding->sgt = dma_buf_map_attachment(binding->attachment, + DMA_BIDIRECTIONAL); + if (IS_ERR(binding->sgt)) { + err = PTR_ERR(binding->sgt); + goto err_detach; + } + + /* For simplicity we expect to make PAGE_SIZE allocations, but the + * binding can be much more flexible than that. We may be able to + * allocate MTU sized chunks here. Leave that for future work... + */ + binding->chunk_pool = gen_pool_create(PAGE_SHIFT, + dev_to_node(&dev->dev)); + if (!binding->chunk_pool) { + err = -ENOMEM; + goto err_unmap; + } + + virtual = 0; + for_each_sgtable_dma_sg(binding->sgt, sg, sg_idx) { + dma_addr_t dma_addr = sg_dma_address(sg); + struct dmabuf_genpool_chunk_owner *owner; + size_t len = sg_dma_len(sg); + struct page_pool_iov *ppiov; + + owner = kzalloc_node(sizeof(*owner), GFP_KERNEL, + dev_to_node(&dev->dev)); + owner->base_virtual = virtual; + owner->base_dma_addr = dma_addr; + owner->num_ppiovs = len / PAGE_SIZE; + owner->binding = binding; + + err = gen_pool_add_owner(binding->chunk_pool, dma_addr, + dma_addr, len, dev_to_node(&dev->dev), + owner); + if (err) { + err = -EINVAL; + goto err_free_chunks; + } + + owner->ppiovs = kvmalloc_array(owner->num_ppiovs, + sizeof(*owner->ppiovs), + GFP_KERNEL); + if (!owner->ppiovs) { + err = -ENOMEM; + goto err_free_chunks; + } + + for (i = 0; i < owner->num_ppiovs; i++) { + ppiov = &owner->ppiovs[i]; + ppiov->owner = owner; + refcount_set(&ppiov->refcount, 1); + } + + virtual += len; + } + + *out = binding; + + return 0; + +err_free_chunks: + gen_pool_for_each_chunk(binding->chunk_pool, + netdev_dmabuf_free_chunk_owner, NULL); + gen_pool_destroy(binding->chunk_pool); +err_unmap: + dma_buf_unmap_attachment(binding->attachment, binding->sgt, + DMA_BIDIRECTIONAL); +err_detach: + dma_buf_detach(dmabuf, binding->attachment); +err_free_id: + xa_erase(&netdev_dmabuf_bindings, binding->id); +err_free_binding: + kfree(binding); +err_put_dmabuf: + dma_buf_put(dmabuf); + return err; +} +#endif + #ifdef CONFIG_NET_INGRESS static DEFINE_STATIC_KEY_FALSE(ingress_needed_key); diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 0ed292d87ae0..b3323812d0b0 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "netdev-genl-gen.h" #include "dev.h" @@ -469,10 +470,94 @@ int netdev_nl_queue_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) return skb->len; } -/* Stub */ +static LIST_HEAD(netdev_rbinding_list); + int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info) { - return 0; + struct nlattr *tb[ARRAY_SIZE(netdev_queue_dmabuf_nl_policy)]; + struct netdev_dmabuf_binding *out_binding; + u32 ifindex, dmabuf_fd, rxq_idx; + struct net_device *netdev; + struct sk_buff *rsp; + struct nlattr *attr; + int rem, err = 0; + void *hdr; + + if (GENL_REQ_ATTR_CHECK(info, NETDEV_A_DEV_IFINDEX) || + GENL_REQ_ATTR_CHECK(info, NETDEV_A_BIND_DMABUF_DMABUF_FD) || + GENL_REQ_ATTR_CHECK(info, NETDEV_A_BIND_DMABUF_QUEUES)) + return -EINVAL; + + ifindex = nla_get_u32(info->attrs[NETDEV_A_DEV_IFINDEX]); + dmabuf_fd = nla_get_u32(info->attrs[NETDEV_A_BIND_DMABUF_DMABUF_FD]); + + rtnl_lock(); + + netdev = __dev_get_by_index(genl_info_net(info), ifindex); + if (!netdev) { + err = -ENODEV; + goto err_unlock; + } + + err = netdev_bind_dmabuf(netdev, dmabuf_fd, &out_binding); + if (err) + goto err_unlock; + + nla_for_each_attr(attr, genlmsg_data(info->genlhdr), + genlmsg_len(info->genlhdr), rem) { + if (nla_type(attr) != NETDEV_A_BIND_DMABUF_QUEUES) + continue; + + err = nla_parse_nested(tb, + ARRAY_SIZE(netdev_queue_dmabuf_nl_policy) - 1, + attr, netdev_queue_dmabuf_nl_policy, + info->extack); + + if (err < 0) + goto err_unbind; + + rxq_idx = nla_get_u32(tb[NETDEV_A_QUEUE_DMABUF_IDX]); + + if (rxq_idx >= netdev->num_rx_queues) { + err = -ERANGE; + goto err_unbind; + } + + err = netdev_bind_dmabuf_to_queue(netdev, rxq_idx, out_binding); + if (err) + goto err_unbind; + } + + out_binding->owner_nlportid = info->snd_portid; + list_add(&out_binding->list, &netdev_rbinding_list); + + rsp = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) { + err = -ENOMEM; + goto err_unbind; + } + + hdr = genlmsg_put(rsp, info->snd_portid, info->snd_seq, + &netdev_nl_family, 0, info->genlhdr->cmd); + if (!hdr) { + err = -EMSGSIZE; + goto err_genlmsg_free; + } + + nla_put_u32(rsp, NETDEV_A_BIND_DMABUF_DMABUF_ID, out_binding->id); + genlmsg_end(rsp, hdr); + + rtnl_unlock(); + + return genlmsg_reply(rsp, info); + +err_genlmsg_free: + nlmsg_free(rsp); +err_unbind: + netdev_unbind_dmabuf(out_binding); +err_unlock: + rtnl_unlock(); + return err; } static int netdev_genl_netdevice_event(struct notifier_block *nb, @@ -495,10 +580,37 @@ static int netdev_genl_netdevice_event(struct notifier_block *nb, return NOTIFY_OK; } +static int netdev_netlink_notify(struct notifier_block *nb, unsigned long state, + void *_notify) +{ + struct netlink_notify *notify = _notify; + struct netdev_dmabuf_binding *rbinding; + + if (state != NETLINK_URELEASE || notify->protocol != NETLINK_GENERIC) + return NOTIFY_DONE; + + rtnl_lock(); + + list_for_each_entry(rbinding, &netdev_rbinding_list, list) { + if (rbinding->owner_nlportid == notify->portid) { + netdev_unbind_dmabuf(rbinding); + break; + } + } + + rtnl_unlock(); + + return NOTIFY_OK; +} + static struct notifier_block netdev_genl_nb = { .notifier_call = netdev_genl_netdevice_event, }; +static struct notifier_block netdev_netlink_notifier = { + .notifier_call = netdev_netlink_notify, +}; + static int __init netdev_genl_init(void) { int err; @@ -511,8 +623,14 @@ static int __init netdev_genl_init(void) if (err) goto err_unreg_ntf; + err = netlink_register_notifier(&netdev_netlink_notifier); + if (err) + goto err_unreg_family; + return 0; +err_unreg_family: + genl_unregister_family(&netdev_nl_family); err_unreg_ntf: unregister_netdevice_notifier(&netdev_genl_nb); return err; From patchwork Fri Dec 8 00:52:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175497 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163357vqy; Thu, 7 Dec 2023 16:53:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IELXsjrfu34rAtEnpLn9XlHwJbYfqVq6Ru1SbcGkwhzQV5/YgNSc/IsWKFOXH2m9nDV3LP6 X-Received: by 2002:a17:90b:3686:b0:286:6cc1:28d with SMTP id mj6-20020a17090b368600b002866cc1028dmr3180449pjb.88.1701996825100; Thu, 07 Dec 2023 16:53:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996825; cv=none; d=google.com; s=arc-20160816; b=lbK9l4ebEavCvim9LZbLnBTy7BBfYxR5bolioAJ+tZTuIHXiXZmbvjkqQe9a8O1RnJ Oks1Fr0u03ERfxaatcRfrz89GfyI9R8gJKy67bq0jPZfBbQMhsGdHj6vCfvBIvG7pZZl 1yG17LGf//wwuUtMqfQahLwMzJSmKyrwPSOaubQkXftuUb21SFmg3n6MlqfeOott5Y3B OqIUzx+FP7U0yXlrufNOh6fB4OfSQVq1+fmy+1WfyAtxeTXn8WB69tU2ncpoy558JIae r3/ptKl5SmaYsphz2a2rouPvJ9mIfeMlFFDHI/8dTU2OsefMklSesxcuDY2pqktBhsQF a6Sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=SdsQlA52rmqBL7OXbh7npWQi67Syf8S/UwksNo2UDhs=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=s+k86tVZO4sajKViszaEjhWfNneXV6d/HqlM11800GkjejYqT4NI43d7AmQ9H/jies ridxlri4dYGEkfcfdK8VHr+M4vaR7WfWXlEZmMZxyY8kN96W7xB4bSe2azIForjv0oBx K8ssZFGgTilf9179vciNTsYhL+TE7kHr3Yb7e3kBBCs/HK4LDbe5ormsg3BUBltKHxu8 pNzzFugw6gakyECPU6EzmEv96A2ACrYpIrdhXpibAcjPTupTs0iskRoIhCfRSJ357Wav dijL8Z/F5y5I/TygGVDR2kEoVFUauPrSrHb92DtSWHYEsHpzvmQjO8SjeAfO0B6Jkeu8 bCtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=q95z+wyk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id w10-20020a17090abc0a00b0028524fc80c4si471718pjr.55.2023.12.07.16.53.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=q95z+wyk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 059FC822880D; Thu, 7 Dec 2023 16:53:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444239AbjLHAxZ (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235655AbjLHAxH (ORCPT ); Thu, 7 Dec 2023 19:53:07 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70362198B for ; Thu, 7 Dec 2023 16:53:10 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-db084a0a2e9so1907667276.2 for ; Thu, 07 Dec 2023 16:53:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996789; x=1702601589; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SdsQlA52rmqBL7OXbh7npWQi67Syf8S/UwksNo2UDhs=; b=q95z+wyk4H31xzCDd2pjmnTHHhFDhyJRKmdJ59urvQwlbJlWHRMLKw1kdiGNp8Dplp pdgZF2KZZf1mSoDEH9v+iSK7zkalHdgB7c8SDCI3LvY3RZP6jzvSnmteLY4DXOXGU4h+ uMyyFz1mwB8H05lV/zqEauumoXhvaEWLEkVmsdICX7e2oHFSHySbsvXO3Z3pNjD5Uww3 mYpRE2V/fkzJ/hnfYi2oWBnnPHNIfkGa8HxPhaJGbvFYZO5L8Dfc8O75E6yc+6yqwdH7 iHGapPKuvH2M8YrXT9T/WA9TGGkxNvM7UFnIEwLgeEV1Ir4o1bpsh+nKq9geSemdJ/rz UR3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996789; x=1702601589; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SdsQlA52rmqBL7OXbh7npWQi67Syf8S/UwksNo2UDhs=; b=X+PZlSN2Nw0pyIRtOKuOse8oSyxxCBT51t4BCt8Kx/4isMPBZo75nCbsOIR23rB716 5Op/pbgCUgddhG6y2v9YoPguW3Jg10R1MypYa/eNBHT9hmZPGP46HKSJ7HPqcueVimUG IHuLrQnD3aQaDE1DcTwJcakto9uems0DF2eSwi4QHWZnkP3waGzlcy/NhaE/TDU6fdY8 5mZqQdFoVL8w17O+mWqGAvSLiEFsXXkYMNBxxvmFuJJ59m/VGM1xAPLxbjN4xOs/fXZS LGUOS50IcdtqitmvAk1WFjdf+BduvfHvkFRqPRXTndrSvZaQnnMUVd0VpK5STlt9j58q 9ZNQ== X-Gm-Message-State: AOJu0YxGw8UtIeFxiaDgoj4WUH/0IU8RNH16eaceLHadhNnohriZemTT OdrLcPFvubTbtnF8grOCXQyyTJS6jMddQENOZg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:7690:0:b0:db4:6936:48b7 with SMTP id r138-20020a257690000000b00db4693648b7mr52142ybc.2.1701996789635; Thu, 07 Dec 2023 16:53:09 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:38 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-8-almasrymina@google.com> Subject: [net-next v1 07/16] netdev: netdevice devmem allocator From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673022856577074 X-GMAIL-MSGID: 1784673022856577074 Implement netdev devmem allocator. The allocator takes a given struct netdev_dmabuf_binding as input and allocates page_pool_iov from that binding. The allocation simply delegates to the binding's genpool for the allocation logic and wraps the returned memory region in a page_pool_iov struct. page_pool_iov are refcounted and are freed back to the binding when the refcount drops to 0. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- v1: - Rename devmem -> dmabuf (David). --- include/net/devmem.h | 13 ++++++++++++ include/net/page_pool/helpers.h | 28 +++++++++++++++++++++++++ net/core/dev.c | 37 ++++++++++++++++++++++++++++++++- 3 files changed, 77 insertions(+), 1 deletion(-) diff --git a/include/net/devmem.h b/include/net/devmem.h index 29ff125f9815..29bc337c7743 100644 --- a/include/net/devmem.h +++ b/include/net/devmem.h @@ -48,6 +48,9 @@ struct netdev_dmabuf_binding { }; #ifdef CONFIG_DMA_SHARED_BUFFER +struct page_pool_iov * +netdev_alloc_dmabuf(struct netdev_dmabuf_binding *binding); +void netdev_free_dmabuf(struct page_pool_iov *ppiov); void __netdev_dmabuf_binding_free(struct netdev_dmabuf_binding *binding); int netdev_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, struct netdev_dmabuf_binding **out); @@ -55,6 +58,16 @@ void netdev_unbind_dmabuf(struct netdev_dmabuf_binding *binding); int netdev_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, struct netdev_dmabuf_binding *binding); #else +static inline struct page_pool_iov * +netdev_alloc_dmabuf(struct netdev_dmabuf_binding *binding) +{ + return NULL; +} + +static inline void netdev_free_dmabuf(struct page_pool_iov *ppiov) +{ +} + static inline void __netdev_dmabuf_binding_free(struct netdev_dmabuf_binding *binding) { diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 7dc65774cde5..8bfc2d43efd4 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -79,6 +79,34 @@ static inline u64 *page_pool_ethtool_stats_get(u64 *data, void *stats) } #endif +/* page_pool_iov support */ + +static inline struct dmabuf_genpool_chunk_owner * +page_pool_iov_owner(const struct page_pool_iov *ppiov) +{ + return ppiov->owner; +} + +static inline unsigned int page_pool_iov_idx(const struct page_pool_iov *ppiov) +{ + return ppiov - page_pool_iov_owner(ppiov)->ppiovs; +} + +static inline dma_addr_t +page_pool_iov_dma_addr(const struct page_pool_iov *ppiov) +{ + struct dmabuf_genpool_chunk_owner *owner = page_pool_iov_owner(ppiov); + + return owner->base_dma_addr + + ((dma_addr_t)page_pool_iov_idx(ppiov) << PAGE_SHIFT); +} + +static inline struct netdev_dmabuf_binding * +page_pool_iov_binding(const struct page_pool_iov *ppiov) +{ + return page_pool_iov_owner(ppiov)->binding; +} + /** * page_pool_dev_alloc_pages() - allocate a page. * @pool: pool from which to allocate diff --git a/net/core/dev.c b/net/core/dev.c index b8c8be5a912e..30667e4c3b95 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -155,8 +155,8 @@ #include #include #include -#include #include +#include #include "dev.h" #include "net-sysfs.h" @@ -2120,6 +2120,41 @@ static int netdev_restart_rx_queue(struct net_device *dev, int rxq_idx) return err; } +struct page_pool_iov *netdev_alloc_dmabuf(struct netdev_dmabuf_binding *binding) +{ + struct dmabuf_genpool_chunk_owner *owner; + struct page_pool_iov *ppiov; + unsigned long dma_addr; + ssize_t offset; + ssize_t index; + + dma_addr = gen_pool_alloc_owner(binding->chunk_pool, PAGE_SIZE, + (void **)&owner); + if (!dma_addr) + return NULL; + + offset = dma_addr - owner->base_dma_addr; + index = offset / PAGE_SIZE; + ppiov = &owner->ppiovs[index]; + + netdev_dmabuf_binding_get(binding); + + return ppiov; +} + +void netdev_free_dmabuf(struct page_pool_iov *ppiov) +{ + struct netdev_dmabuf_binding *binding = page_pool_iov_binding(ppiov); + unsigned long dma_addr = page_pool_iov_dma_addr(ppiov); + + refcount_set(&ppiov->refcount, 1); + + if (gen_pool_has_addr(binding->chunk_pool, dma_addr, PAGE_SIZE)) + gen_pool_free(binding->chunk_pool, dma_addr, PAGE_SIZE); + + netdev_dmabuf_binding_put(binding); +} + /* Protected by rtnl_lock() */ static DEFINE_XARRAY_FLAGS(netdev_dmabuf_bindings, XA_FLAGS_ALLOC1); From patchwork Fri Dec 8 00:52:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175503 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163478vqy; Thu, 7 Dec 2023 16:54:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IHdCmWFVOi0nIbd7e6QHrpaBR7UG3p1SGOsMmpOJ3hyhPxthcWSbM58milfX6Nd65Q1bfag X-Received: by 2002:a05:6359:29c9:b0:170:3684:315d with SMTP id qf9-20020a05635929c900b001703684315dmr2799012rwb.45.1701996845393; Thu, 07 Dec 2023 16:54:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996845; cv=none; d=google.com; s=arc-20160816; b=l4ZUtzldKv33YrUTVlpkbuSEp489NlPpGg62zt9n68bByQ421acOOcpFOUuMved3Nl v1ntoAdh/jdRsShtQ+QS6IUIsPen1iVatqW0sjjYdNI9tiPZ6Uo9NiQ31lMURmTgXUaq ZGyADqRqNzgcRnL/1y2X0tpLe5vMXRkIxPdJjkxK4Lrggct4LmmblI6rvEaL7sCDt2rG 7cRM2jy4U20L/rX7eiRkDvdimqvEPgIiSY+jlvCKwu5UETI9nFGp4fMxLOohnQ7HdeAk srETJDIrNkUjf37dG7dqyHaMb1kXzAoWd3TDjaskiAnsE/DxpIwjtcdAVAVXsfEL6f31 B9QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=6PanYp2ouKVS9w/roipcdSrUh/6HC+Cc/BaaByD7oTA=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=GApPhxiQFquNmDL8JQodqJR8u6DmjIPAT/RnXgJyjV8PBIBXSHJk6R9b9OA42oYpYB upIIyx+2vPCJYLbxwKsj1ji15YCt9jHmx3E+YpPXX2VVzcvMcQ5am+9SN4Ykrn+hG3MF MxXjAY0T/5yzFx2Nzn13zOMUp4Y7XgtPgNtJO9cOG7LKpx5+Wxvy4AioVmUpmF5Hsojm CjaIOCOmF+CCW3xvlU+IVm2AmSgNLL4NbFSJwHKHCrBHhnOwnS1nNm7kbWuVrXNLHlyT ZGe5N+yxszoRfVmoE3ZNnNOBpcWc8er/0LBpuqYoOJKUzzpLQ09fkmxVa10tqlHycoUo X1kQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Sx3jBrZI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id f9-20020a056a00228900b006cde5241c15si571711pfe.306.2023.12.07.16.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Sx3jBrZI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 412C2816F0D6; Thu, 7 Dec 2023 16:53:56 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572889AbjLHAx2 (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444202AbjLHAxH (ORCPT ); Thu, 7 Dec 2023 19:53:07 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D6C21996 for ; Thu, 7 Dec 2023 16:53:12 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5d7e7e10231so9953567b3.1 for ; Thu, 07 Dec 2023 16:53:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996792; x=1702601592; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6PanYp2ouKVS9w/roipcdSrUh/6HC+Cc/BaaByD7oTA=; b=Sx3jBrZI4T6YrKjUMUH9z0Sz2AvSxc2M8OH4NsG9eDH8mI9chEIQSDiBDrj/B//ONR KobKTJieRbC9h93UyUSAkwX/hdomluyklANx3EKcOOEHr/gPEgIXuigX3BDDVF7lcg/4 hMxosGqRiEEcqWGvcIrPoVVKx88cq9Gry0tOdicDTyKrzOxO7NV4B5foeHRE6rUIjk5P tZXLbxSLdhn7gUJbhYeXb8thl/h/peSjQVW+f1Z0WRI9RI6V3AeYEs+QsxakxFxBYpqf H2qYoH/bfwaIyvKwVpPut3S6+G6AbR1i0tdlkyOZ47a+lUugZ/V3KG1v0P9Pf56kbU6p k8Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996792; x=1702601592; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6PanYp2ouKVS9w/roipcdSrUh/6HC+Cc/BaaByD7oTA=; b=SBZ+LIfJVq6ZH3JiW/3eNKH0Buu2FeeCEsCF/YV+NbR+usKHq08rz/6NWQmIb0TuqQ WmiOXH7rR83uP3bhOpZgDQ+zIMfzENmC0ffUcp842V9wp4bvh3n+XPo2MpFzr4Uh+2er PuYeBQK6dqUP4RDPu7V/vPDIETz7DXEdugWhjmC6C7cksXw9Mik9kmK7O8ylwVzGCPqD cPhfKzanqLP4sdhbGvDMMA1SJXmOo4kUadH733sDHeThOn40+RpJpWDiqTfhQc36mDbx xUabPKeTmSFT/hRBWTGAj9lvr8q0H+y248INi3K220z89Aja7+eIYpc24Bp06e6B9oSx oP7A== X-Gm-Message-State: AOJu0Yy5I/KypPryeO5n3VOkrtCvmf5qBT+9+0vxIyAQkUdxO+WGlUeB Cv5KMQ06PxaD3lDpuLIfrLCi4bIc1Z+a/C2nMQ== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a0d:e208:0:b0:5d9:340c:292d with SMTP id l8-20020a0de208000000b005d9340c292dmr1876ywe.4.1701996791753; Thu, 07 Dec 2023 16:53:11 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:39 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-9-almasrymina@google.com> Subject: [net-next v1 08/16] memory-provider: dmabuf devmem memory provider From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:56 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673043967574966 X-GMAIL-MSGID: 1784673043967574966 Implement a memory provider that allocates dmabuf devmem page_pool_iovs. The provider receives a reference to the struct netdev_dmabuf_binding via the pool->mp_priv pointer. The driver needs to set this pointer for the provider in the page_pool_params. The provider obtains a reference on the netdev_dmabuf_binding which guarantees the binding and the underlying mapping remains alive until the provider is destroyed. Usage of PP_FLAG_DMA_MAP is required for this memory provide such that the page_pool can provide the driver with the dma-addrs of the devmem. Support for PP_FLAG_DMA_SYNC_DEV is omitted for simplicity. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- v1: - static_branch check in page_is_page_pool_iov() (Willem & Paolo). - PP_DEVMEM -> PP_IOV (David). - Require PP_FLAG_DMA_MAP (Jakub). --- include/net/page_pool/helpers.h | 47 +++++++++++++++++ include/net/page_pool/types.h | 9 ++++ net/core/page_pool.c | 89 ++++++++++++++++++++++++++++++++- 3 files changed, 144 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 8bfc2d43efd4..00197f14aa87 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -53,6 +53,8 @@ #define _NET_PAGE_POOL_HELPERS_H #include +#include +#include #ifdef CONFIG_PAGE_POOL_STATS /* Deprecated driver-facing API, use netlink instead */ @@ -92,6 +94,11 @@ static inline unsigned int page_pool_iov_idx(const struct page_pool_iov *ppiov) return ppiov - page_pool_iov_owner(ppiov)->ppiovs; } +static inline u32 page_pool_iov_binding_id(const struct page_pool_iov *ppiov) +{ + return page_pool_iov_owner(ppiov)->binding->id; +} + static inline dma_addr_t page_pool_iov_dma_addr(const struct page_pool_iov *ppiov) { @@ -107,6 +114,46 @@ page_pool_iov_binding(const struct page_pool_iov *ppiov) return page_pool_iov_owner(ppiov)->binding; } +static inline int page_pool_iov_refcount(const struct page_pool_iov *ppiov) +{ + return refcount_read(&ppiov->refcount); +} + +static inline void page_pool_iov_get_many(struct page_pool_iov *ppiov, + unsigned int count) +{ + refcount_add(count, &ppiov->refcount); +} + +void __page_pool_iov_free(struct page_pool_iov *ppiov); + +static inline void page_pool_iov_put_many(struct page_pool_iov *ppiov, + unsigned int count) +{ + if (!refcount_sub_and_test(count, &ppiov->refcount)) + return; + + __page_pool_iov_free(ppiov); +} + +/* page pool mm helpers */ + +DECLARE_STATIC_KEY_FALSE(page_pool_mem_providers); +static inline bool page_is_page_pool_iov(const struct page *page) +{ + return static_branch_unlikely(&page_pool_mem_providers) && + (unsigned long)page & PP_IOV; +} + +static inline struct page_pool_iov *page_to_page_pool_iov(struct page *page) +{ + if (page_is_page_pool_iov(page)) + return (struct page_pool_iov *)((unsigned long)page & ~PP_IOV); + + DEBUG_NET_WARN_ON_ONCE(true); + return NULL; +} + /** * page_pool_dev_alloc_pages() - allocate a page. * @pool: pool from which to allocate diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 44faee7a7b02..136930a238de 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -134,8 +134,15 @@ struct memory_provider_ops { bool (*release_page)(struct page_pool *pool, struct page *page); }; +extern const struct memory_provider_ops dmabuf_devmem_ops; + /* page_pool_iov support */ +/* We overload the LSB of the struct page pointer to indicate whether it's + * a page or page_pool_iov. + */ +#define PP_IOV 0x01UL + /* Owner of the dma-buf chunks inserted into the gen pool. Each scatterlist * entry from the dmabuf is inserted into the genpool as a chunk, and needs * this owner struct to keep track of some metadata necessary to create @@ -159,6 +166,8 @@ struct page_pool_iov { struct dmabuf_genpool_chunk_owner *owner; refcount_t refcount; + + struct page_pool *pp; }; struct page_pool { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index f5c84d2a4510..423c88564a00 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -12,6 +12,7 @@ #include #include +#include #include #include @@ -20,12 +21,15 @@ #include #include #include +#include +#include #include #include "page_pool_priv.h" -static DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); +DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); +EXPORT_SYMBOL(page_pool_mem_providers); #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) @@ -175,6 +179,7 @@ static void page_pool_producer_unlock(struct page_pool *pool, static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params) { + struct netdev_dmabuf_binding *binding = NULL; unsigned int ring_qsize = 1024; /* Default */ int err; @@ -237,6 +242,14 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); + if (pool->p.queue) + binding = READ_ONCE(pool->p.queue->binding); + + if (binding) { + pool->mp_ops = &dmabuf_devmem_ops; + pool->mp_priv = binding; + } + if (pool->mp_ops) { err = pool->mp_ops->init(pool); if (err) { @@ -1020,3 +1033,77 @@ void page_pool_update_nid(struct page_pool *pool, int new_nid) } } EXPORT_SYMBOL(page_pool_update_nid); + +void __page_pool_iov_free(struct page_pool_iov *ppiov) +{ + if (WARN_ON(ppiov->pp->mp_ops != &dmabuf_devmem_ops)) + return; + + netdev_free_dmabuf(ppiov); +} +EXPORT_SYMBOL_GPL(__page_pool_iov_free); + +/*** "Dmabuf devmem memory provider" ***/ + +static int mp_dmabuf_devmem_init(struct page_pool *pool) +{ + struct netdev_dmabuf_binding *binding = pool->mp_priv; + + if (!binding) + return -EINVAL; + + if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + return -EOPNOTSUPP; + + if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + return -EOPNOTSUPP; + + netdev_dmabuf_binding_get(binding); + return 0; +} + +static struct page *mp_dmabuf_devmem_alloc_pages(struct page_pool *pool, + gfp_t gfp) +{ + struct netdev_dmabuf_binding *binding = pool->mp_priv; + struct page_pool_iov *ppiov; + + ppiov = netdev_alloc_dmabuf(binding); + if (!ppiov) + return NULL; + + ppiov->pp = pool; + pool->pages_state_hold_cnt++; + trace_page_pool_state_hold(pool, (struct page *)ppiov, + pool->pages_state_hold_cnt); + return (struct page *)((unsigned long)ppiov | PP_IOV); +} + +static void mp_dmabuf_devmem_destroy(struct page_pool *pool) +{ + struct netdev_dmabuf_binding *binding = pool->mp_priv; + + netdev_dmabuf_binding_put(binding); +} + +static bool mp_dmabuf_devmem_release_page(struct page_pool *pool, + struct page *page) +{ + struct page_pool_iov *ppiov; + + if (WARN_ON_ONCE(!page_is_page_pool_iov(page))) + return false; + + ppiov = page_to_page_pool_iov(page); + page_pool_iov_put_many(ppiov, 1); + /* We don't want the page pool put_page()ing our page_pool_iovs. */ + return false; +} + +const struct memory_provider_ops dmabuf_devmem_ops = { + .init = mp_dmabuf_devmem_init, + .destroy = mp_dmabuf_devmem_destroy, + .alloc_pages = mp_dmabuf_devmem_alloc_pages, + .release_page = mp_dmabuf_devmem_release_page, +}; +EXPORT_SYMBOL(dmabuf_devmem_ops); From patchwork Fri Dec 8 00:52:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175501 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163427vqy; Thu, 7 Dec 2023 16:53:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IGynIfZBuwbyXuf2ldVmGYX2uUEKKwy2oSgHfqVp5AxP+eG1YnkvofxI0wtj8vdBGNUquSQ X-Received: by 2002:a05:6a20:729b:b0:18f:ef63:467b with SMTP id o27-20020a056a20729b00b0018fef63467bmr3178746pzk.38.1701996836197; Thu, 07 Dec 2023 16:53:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996836; cv=none; d=google.com; s=arc-20160816; b=iDwOwHdxO95dDobAG0N+Ixm+7dmTSt3NaE2Hc7uuteniyiH0WzTprgGRcG9vINNke/ ttMGFStjIMjHqfYf6ccu8Wfbzn/Ks21mdOlboLfctfrtMxIiERCkjwe0BGov5Gm9byEn A4hRLBuqJBzQWvuz37gS/hTakR3aqBdgTGD9/3jknJkz2qr+FqRoaq11N333BU4n1yLj NZTS10qA+t3yYjgGBj9ofjvGbBYV4PsBhXoBvjIeDYLMPFJILwjINS0Jk7FWKvKkEtVL Rd1U4ckXONnrX1DfbxIW3ZG5FWU3HgvY30sEomJt7qTXsU3ehiFxVSxAKsipTtmKYT69 aucw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=/frlnDGiPBjsiHjgXvPuD20HndGYXPjOBYCQWwtD/DY=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=zbyN2BAeFGFnd5AD3UM3iE0jppSFsK08L35eXIajIET11u5CROdLo6JMp2uzgxDcrj DuhzVqRo6nQ+JFst5ADMBba0uQBZMoNmV5OYvncOjiectoTZX0zBifuKK2q12MYK1WUJ M3Vf54JrLJJ7rhZBYx8qbNnobUjO8TlUxjhw5j3z9b9zWlagXsMtkyW4JlXz+czzaJv+ 1hQXWfh71pPSnrEEZDGWpFfF4qKxHQ7g6w57FVb1uyi7+EEnjRyVFq6Vnabb1FAJVd4k XN/P3iYq/EvPeuUtQNzkYc91p9EHcslOsBNHLjo3/nnw+9N+d30kWgYMpjmC/33EZkFp rfVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=vBRsbNqo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id d1-20020a170902aa8100b001d07c56fd79si568342plr.638.2023.12.07.16.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=vBRsbNqo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id CFF158087B7C; Thu, 7 Dec 2023 16:53:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572908AbjLHAxb (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444216AbjLHAxJ (ORCPT ); Thu, 7 Dec 2023 19:53:09 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A21311C for ; Thu, 7 Dec 2023 16:53:14 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5d10f5bf5d9so15541067b3.3 for ; Thu, 07 Dec 2023 16:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996794; x=1702601594; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/frlnDGiPBjsiHjgXvPuD20HndGYXPjOBYCQWwtD/DY=; b=vBRsbNqot8epSwQSXKoU0GSDp5CGWwhdLD5KF0Qt9Mu9kp72t7wzVIhS5MEJqxZ2BV QpX80+/4QvpUwor0wDW2xagZ8shlZhTdIE3lbCZE4fW+zt6/iDKWxhD/GQCewRV5IAu8 mhTjQfoNpeth0A1An06Kybm2KqhDr25iDXtBku2PNArFTqimTVJKv4bi7z5y9wnzEWcp riL6zGtHKWcDIDCjS129E+dN9BiKnipPShGT0lXrtm4jBFMfRczWpCbxS/rAAU0o8tF2 +ciQqTqyb9+LJYefdGuVJuOfWuBQUwa+OwQbjyiYsFY+Bdi7bwPNNMbkUlkWEeH+sIcU NavA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996794; x=1702601594; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/frlnDGiPBjsiHjgXvPuD20HndGYXPjOBYCQWwtD/DY=; b=Mcelc9qO6X3GjA9Zp6IzjUxeFCiikps0/mermQY6m/xFz2VQvKbcbBShhrJ8ld93n6 F6FL7yt0KLV1zBcLyNHBdmwJivEuV5MuR8j5aWYwQyolTGUQXyScaCwzvawLjQ+YOLcu RKwlWcNgTJpTY2HdwZC4maGQvWFe/fHOyILEDJmaTCU/cwikaaaFhSJ/y00kzUVgfe57 nzOuEaFdqs9WAQgxvniirmLEzhDmfXJ883C5jUYNQTcC9BuI4tz9wF7Gn0Us8s6VrsPj qEHEuqsutG3k/ShXoBikuJ8w4tALdZpTayLsrIz5fzXvojrKkCM9gDREzMYqGFhqKT/K Typw== X-Gm-Message-State: AOJu0YxcHXeGfTH1/q5NxiAZMBVyv71WGITlUhwh+jFhTSa7aRvRNX4A AaSy+aN0iIqf0HGmfzp3KhvVvpR8BkIqOxlHzQ== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:2893:b0:5d3:b449:e58e with SMTP id ee19-20020a05690c289300b005d3b449e58emr47990ywb.6.1701996793843; Thu, 07 Dec 2023 16:53:13 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:40 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-10-almasrymina@google.com> Subject: [net-next v1 09/16] page_pool: device memory support From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:49 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673034256835944 X-GMAIL-MSGID: 1784673034256835944 Overload the LSB of struct page* to indicate that it's a page_pool_iov. Refactor mm calls on struct page* into helpers, and add page_pool_iov handling on those helpers. Modify callers of these mm APIs with calls to these helpers instead. In areas where struct page* is dereferenced, add a check for special handling of page_pool_iov. Signed-off-by: Mina Almasry --- v1: - Disable fragmentation support for iov properly. - fix napi_pp_put_page() path (Yunsheng). --- include/net/page_pool/helpers.h | 78 ++++++++++++++++++++++++++++++++- net/core/page_pool.c | 67 ++++++++++++++++++++-------- net/core/skbuff.c | 28 +++++++----- 3 files changed, 141 insertions(+), 32 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 00197f14aa87..2d4e0a2c5620 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -154,6 +154,64 @@ static inline struct page_pool_iov *page_to_page_pool_iov(struct page *page) return NULL; } +static inline int page_pool_page_ref_count(struct page *page) +{ + if (page_is_page_pool_iov(page)) + return page_pool_iov_refcount(page_to_page_pool_iov(page)); + + return page_ref_count(page); +} + +static inline void page_pool_page_get_many(struct page *page, + unsigned int count) +{ + if (page_is_page_pool_iov(page)) + return page_pool_iov_get_many(page_to_page_pool_iov(page), + count); + + return page_ref_add(page, count); +} + +static inline void page_pool_page_put_many(struct page *page, + unsigned int count) +{ + if (page_is_page_pool_iov(page)) + return page_pool_iov_put_many(page_to_page_pool_iov(page), + count); + + if (count > 1) + page_ref_sub(page, count - 1); + + put_page(page); +} + +static inline bool page_pool_page_is_pfmemalloc(struct page *page) +{ + if (page_is_page_pool_iov(page)) + return false; + + return page_is_pfmemalloc(page); +} + +static inline bool page_pool_page_is_pref_nid(struct page *page, int pref_nid) +{ + /* Assume page_pool_iov are on the preferred node without actually + * checking... + * + * This check is only used to check for recycling memory in the page + * pool's fast paths. Currently the only implementation of page_pool_iov + * is dmabuf device memory. It's a deliberate decision by the user to + * bind a certain dmabuf to a certain netdev, and the netdev rx queue + * would not be able to reallocate memory from another dmabuf that + * exists on the preferred node, so, this check doesn't make much sense + * in this case. Assume all page_pool_iovs can be recycled for now. + */ + if (page_is_page_pool_iov(page)) + return true; + + return page_to_nid(page) == pref_nid; +} + /** * page_pool_dev_alloc_pages() - allocate a page. * @pool: pool from which to allocate @@ -304,6 +362,10 @@ static inline long page_pool_defrag_page(struct page *page, long nr) { long ret; + /* fragmentation support hasn't been added to ppiov yet */ + if (WARN_ON_ONCE(page_is_page_pool_iov(page))) + return 0; + /* If nr == pp_frag_count then we have cleared all remaining * references to the page: * 1. 'n == 1': no need to actually overwrite it. @@ -347,7 +409,8 @@ static inline long page_pool_defrag_page(struct page *page, long nr) static inline bool page_pool_is_last_frag(struct page *page) { /* If page_pool_defrag_page() returns 0, we were the last user */ - return page_pool_defrag_page(page, 1) == 0; + return page_is_page_pool_iov(page) || + page_pool_defrag_page(page, 1) == 0; } /** @@ -434,7 +497,12 @@ static inline void page_pool_free_va(struct page_pool *pool, void *va, */ static inline dma_addr_t page_pool_get_dma_addr(struct page *page) { - dma_addr_t ret = page->dma_addr; + dma_addr_t ret; + + if (page_is_page_pool_iov(page)) + return page_pool_iov_dma_addr(page_to_page_pool_iov(page)); + + ret = page->dma_addr; if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) ret <<= PAGE_SHIFT; @@ -444,6 +512,12 @@ static inline dma_addr_t page_pool_get_dma_addr(struct page *page) static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t addr) { + /* page_pool_iovs are mapped and their dma-addr can't be modified. */ + if (page_is_page_pool_iov(page)) { + DEBUG_NET_WARN_ON_ONCE(true); + return false; + } + if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { page->dma_addr = addr >> PAGE_SHIFT; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 423c88564a00..f0148d66371b 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -346,7 +346,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool) if (unlikely(!page)) break; - if (likely(page_to_nid(page) == pref_nid)) { + if (likely(page_pool_page_is_pref_nid(page, pref_nid))) { pool->alloc.cache[pool->alloc.count++] = page; } else { /* NUMA mismatch; @@ -391,7 +391,15 @@ static void page_pool_dma_sync_for_device(struct page_pool *pool, struct page *page, unsigned int dma_sync_size) { - dma_addr_t dma_addr = page_pool_get_dma_addr(page); + dma_addr_t dma_addr; + + /* page_pool_iov memory provider do not support PP_FLAG_DMA_SYNC_DEV */ + if (page_is_page_pool_iov(page)) { + DEBUG_NET_WARN_ON_ONCE(true); + return; + } + + dma_addr = page_pool_get_dma_addr(page); dma_sync_size = min(dma_sync_size, pool->p.max_len); dma_sync_single_range_for_device(pool->p.dev, dma_addr, @@ -403,6 +411,12 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) { dma_addr_t dma; + if (page_is_page_pool_iov(page)) { + /* page_pool_iovs are already mapped */ + DEBUG_NET_WARN_ON_ONCE(true); + return true; + } + /* Setup DMA mapping: use 'struct page' area for storing DMA-addr * since dma_addr_t can be either 32 or 64 bits and does not always fit * into page private data (i.e 32bit cpu with 64bit DMA caps) @@ -434,22 +448,33 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) static void page_pool_set_pp_info(struct page_pool *pool, struct page *page) { - page->pp = pool; - page->pp_magic |= PP_SIGNATURE; - - /* Ensuring all pages have been split into one fragment initially: - * page_pool_set_pp_info() is only called once for every page when it - * is allocated from the page allocator and page_pool_fragment_page() - * is dirtying the same cache line as the page->pp_magic above, so - * the overhead is negligible. - */ - page_pool_fragment_page(page, 1); + if (!page_is_page_pool_iov(page)) { + page->pp = pool; + page->pp_magic |= PP_SIGNATURE; + + /* Ensuring all pages have been split into one fragment + * initially: + * page_pool_set_pp_info() is only called once for every page + * when it is allocated from the page allocator and + * page_pool_fragment_page() is dirtying the same cache line as + * the page->pp_magic above, so * the overhead is negligible. + */ + page_pool_fragment_page(page, 1); + } else { + page_to_page_pool_iov(page)->pp = pool; + } + if (pool->has_init_callback) pool->slow.init_callback(page, pool->slow.init_arg); } static void page_pool_clear_pp_info(struct page *page) { + if (page_is_page_pool_iov(page)) { + page_to_page_pool_iov(page)->pp = NULL; + return; + } + page->pp_magic = 0; page->pp = NULL; } @@ -664,7 +689,7 @@ static bool page_pool_recycle_in_cache(struct page *page, return false; } - /* Caller MUST have verified/know (page_ref_count(page) == 1) */ + /* Caller MUST have verified/know (page_pool_page_ref_count(page) == 1) */ pool->alloc.cache[pool->alloc.count++] = page; recycle_stat_inc(pool, cached); return true; @@ -689,9 +714,10 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, * refcnt == 1 means page_pool owns page, and can recycle it. * * page is NOT reusable when allocated when system is under - * some pressure. (page_is_pfmemalloc) + * some pressure. (page_pool_page_is_pfmemalloc) */ - if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { + if (likely(page_pool_page_ref_count(page) == 1 && + !page_pool_page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) @@ -806,7 +832,8 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, if (likely(page_pool_defrag_page(page, drain_count))) return NULL; - if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { + if (page_pool_page_ref_count(page) == 1 && + !page_pool_page_is_pfmemalloc(page)) { if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) page_pool_dma_sync_for_device(pool, page, -1); @@ -840,6 +867,10 @@ struct page *page_pool_alloc_frag(struct page_pool *pool, if (WARN_ON(size > max_size)) return NULL; + /* page_pool_iov's don't currently support fragmentation */ + if (WARN_ON_ONCE(pool->mp_ops == &dmabuf_devmem_ops)) + return NULL; + size = ALIGN(size, dma_get_cache_alignment()); *offset = pool->frag_offset; @@ -882,9 +913,9 @@ static void page_pool_empty_ring(struct page_pool *pool) /* Empty recycle ring */ while ((page = ptr_ring_consume_bh(&pool->ring))) { /* Verify the refcnt invariant of cached pages */ - if (!(page_ref_count(page) == 1)) + if (!(page_pool_page_ref_count(page) == 1)) pr_crit("%s() page_pool refcnt %d violation\n", - __func__, page_ref_count(page)); + __func__, page_pool_page_ref_count(page)); page_pool_return_page(pool, page); } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..07f802f1adf1 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -896,19 +896,23 @@ bool napi_pp_put_page(struct page *page, bool napi_safe) bool allow_direct = false; struct page_pool *pp; - page = compound_head(page); - - /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation - * in order to preserve any existing bits, such as bit 0 for the - * head page of compound page and bit 1 for pfmemalloc page, so - * mask those bits for freeing side when doing below checking, - * and page_is_pfmemalloc() is checked in __page_pool_put_page() - * to avoid recycling the pfmemalloc page. - */ - if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) - return false; + if (!page_is_page_pool_iov(page)) { + page = compound_head(page); + + /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation + * in order to preserve any existing bits, such as bit 0 for the + * head page of compound page and bit 1 for pfmemalloc page, so + * mask those bits for freeing side when doing below checking, + * and page_is_pfmemalloc() is checked in __page_pool_put_page() + * to avoid recycling the pfmemalloc page. + */ + if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) + return false; - pp = page->pp; + pp = page->pp; + } else { + pp = page_to_page_pool_iov(page)->pp; + } /* Allow direct recycle if we have reasons to believe that we are * in the same context as the consumer would run, so there's From patchwork Fri Dec 8 00:52:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175499 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163388vqy; Thu, 7 Dec 2023 16:53:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IE/ezrv3GsZb5YZEIkTE0B/ssCekHmUzUt+ymXvnAeqluhUd0ECF+K8CXSprX3kQE5ZJXl6 X-Received: by 2002:a05:6a20:2618:b0:18b:31eb:8b66 with SMTP id i24-20020a056a20261800b0018b31eb8b66mr2610686pze.50.1701996830099; Thu, 07 Dec 2023 16:53:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996830; cv=none; d=google.com; s=arc-20160816; b=DrKVHibicjxaGdkBlrJzdBa2VDBZJNwm4iesL36lUKOrp4pc44W/n4meTaIcFGmy7Z 9+afJBVgfygzYceGhJoGKfspOQ4C07DJ/162mD2QxTGaVMTAWj3EvmBGvo/n3D2yfe6I 83HGZ9/XTmrYiztHwTl/QjXIMps6/3RpZcZ75Z9U3/fPcNP3suNpbuLq4KnJnHanQRWo SXd9q0pWjxy2x2B3JOnHEZd/0/W6f43UF249bvlbf2XNlAa8/t3z00BqDXfq/bUiSZ1u x3t7WoMdr+ZDBVJR1ODf/3rrS5FKdu0HhpR9TflW5MvvCXKhMoURiDasPDRtpHmvt771 QQXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=lPnDX1rYMAtCe+hQ8qdPonfWEnFnkE5EPZQcdSbkY2c=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=zXpdNdNeAcQbdMu2kloUIYry/ZUXfYoQ4RsU75A9qdftTNr7GIDNMhEiYDnYbgSEtk m9wZMD7g8Az1U3+uldF+71xq78L07dCLrnFAinDO3p8VNvLBcMa00+WC/wikEsSepZAp MLzzclG6XX75/FCUSML+3RxmW6e2d00Nd3K4ZabBDuqiKR7smqCir6fm3N88lMMssJEO PoDGKguzxmGJijtU8ziNJ0EFzqTUNJRyYQTzKWD/9V65HCjItrQ7bmZK+OLrVZrz7OCw tPK5mg7E8J6n6okEL3NqxxzdSM7Ws/PlRqR03TFnKnF81cOyhGd8KKLYDAFciiq26uSZ uEoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tItI9ksJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id ci2-20020a17090afc8200b0028694fc7a61si669739pjb.13.2023.12.07.16.53.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:53:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tItI9ksJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 1B5A280DECD6; Thu, 7 Dec 2023 16:53:49 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572899AbjLHAxf (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235574AbjLHAxK (ORCPT ); Thu, 7 Dec 2023 19:53:10 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A843172D for ; Thu, 7 Dec 2023 16:53:16 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5c5daf2baccso17492397b3.3 for ; Thu, 07 Dec 2023 16:53:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996795; x=1702601595; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lPnDX1rYMAtCe+hQ8qdPonfWEnFnkE5EPZQcdSbkY2c=; b=tItI9ksJxKZZeSA4cIlz1awMH257FvktV3a5FQasmksOiVe6EDUTdc38DlVmY9AMUv 1yUbt++nhyY11JmQKVKMdqq/CKI60A8ETh2zdhXhofyLY1NvfvxI27vAS+LaBEfcsC0m dMAr4tkoEokr4qCBaTMo4v6nkfqhHxyI7j+wwuw3MRT2niwmzCRi6fGzm0jpDy+wGnvE PZvJaSLBlM5QaFpeBtM0oCVky9+C8Jg5N9L//cguRIlLlMuU524zOQUz7g3+6xyaACVK H4SLrLLPCGbX164C9j9lRBfetzidTop22PysOSNLg7487MDm0dggC/tegdH5rd8KCgry Sp0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996795; x=1702601595; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lPnDX1rYMAtCe+hQ8qdPonfWEnFnkE5EPZQcdSbkY2c=; b=wfrh5rYuJBkAmgEK5GW85mB1dsq0hqHr7SMLlDp01xPRGW8ggiq9shlhOkZfixXZco MbMdZTtPrhdI4MeLGuHb9E7uEEulb/nv5da9oaNJ9bDEH+Lt5dOXhWIonxJbqD6Kl9+T 6LsJLShhCuEII8KUJOrBPjrP0f1Tmp/lf2gn2k/aJKcdCohQ5bygUNYIpo7amD9CO9l9 c3Aa3IvDSroE3okcBEQoF8uSaCKwVBVA/rteYjc1E8r7SWe1ZmCtWh9dEJOBGgNTsA7j 08jsTWuO6lG65pRipLcIklJhMHebudtcVAz8C9X8awxjPYKWPkpFW/yA7em8Q6Nw1TKN lOVA== X-Gm-Message-State: AOJu0Ywx6dcR+VAyXIYojEHvdTyGPywDVEAa6E2lMsXoGXGcxfM8pTUt /hYLtPtUt80GljocxmjTh1GuR0RkPzGOrkr0ag== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:4707:b0:5d4:ce2:e908 with SMTP id gz7-20020a05690c470700b005d40ce2e908mr53364ywb.3.1701996795620; Thu, 07 Dec 2023 16:53:15 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:41 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-11-almasrymina@google.com> Subject: [net-next v1 10/16] page_pool: don't release iov on elevanted refcount From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:53:49 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673027956269146 X-GMAIL-MSGID: 1784673027956269146 Currently the page_pool behavior is that a page is considered for recycling only once, the first time __page_pool_put_page() is called on it. This works because in practice the net stack only holds 1 reference to the skb frags. In that case, the page_pool recycling works as expected, as the skb frags will have 1 reference on the pages from the net stack when __page_pool_put_page() is called (if the driver is not holding extra references for recycling), and so the page will be recycled. However, this is not compatible with devmem TCP. For devmem TCP, the net stack holds 2 references for each frag, 1 reference is part of the SKB, and the second reference is for the user holding the frag until they call SO_DEVMEM_DONTNEED. This causes a bug in the page_pool recycling where, when the skb is freed, the reference count goes from 2->1, the page_pool sees a pending reference, releases the page, and so no devmem iovs get recycled. To fix this, don't release iovs on elevated refcount. Signed-off-by: Mina Almasry --- net/core/page_pool.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index f0148d66371b..dc2a148f5b06 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -731,6 +731,29 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, /* Page found as candidate for recycling */ return page; } + + if (page_is_page_pool_iov(page)) { + /* With devmem TCP and ppiovs, we can't release pages if the + * refcount is > 1. This is because the net stack holds + * 2 references: + * - 1 for the skb, and + * - 1 for the user until they call SO_DEVMEM_DONTNEED. + * Releasing pages for elevated refcounts completely disables + * page_pool recycling. Instead, simply don't release pages and + * the next call to napi_pp_put_page() via SO_DEVMEM_DONTNEED + * will consider the page again for recycling. As a result, + * devmem TCP incompatible with drivers doing refcnt based + * recycling unless those drivers: + * + * - don't mark skb_mark_for_recycle() + * - are sure to release the last reference with + * page_pool_put_full_page() to consider the page for + * page_pool recycling. + */ + page_pool_page_put_many(page, 1); + return NULL; + } + /* Fallback/non-XDP mode: API user have elevated refcnt. * * Many drivers split up the page into fragments, and some From patchwork Fri Dec 8 00:52:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175506 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163591vqy; Thu, 7 Dec 2023 16:54:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IHq2ifbejI5bDDL4KubANejkUOAhMG33mZ/RUfs4IQ+TbJxfIjY1YUzUZ8nw77EAsOgqw7G X-Received: by 2002:a17:90b:30cb:b0:286:c398:841b with SMTP id hi11-20020a17090b30cb00b00286c398841bmr3326386pjb.58.1701996864587; Thu, 07 Dec 2023 16:54:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996864; cv=none; d=google.com; s=arc-20160816; b=RDgrgfYanySoyWmbrXbNTXDwIflKDIzO+2ZY/bIX9OEe2SG7X2rIUQFFm95EjomIuU 2XlIEK2lrgHUDW2aFccRa6PVZm+KpUbtPVUSkGQbcVUm9+8TMr0QLC5EUDacwclOZnoo szyEb9wOcVBjhQSbcq0pei2hikZysxbHfxDXrC1qsol3j/aiTTBqUt4pE2t8zG9JO5Q/ 3dgvM6cHHrbmmYlr64DelFeKgaMRnTtGinC0OyNeCht2l5FpFsyvfLh/9pBS+K8PO4z3 S68ODUUiToN27H15GFGoKh+Go+hQDrZC8J5IVPPsCtCRODpUCEZA3QX8n3Fk9pvy+C3j XJAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=ffR5lRGBwRWz92voabMKfnUxOBX9ei9jc/R/TuPbdfY=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=eaWuvI35VO/E/JdDry32AmWfXBfGhQihZxh//tO9yoPQzEgKXaWPmLwO67q7lYJP2K KxeXTDe2BBXne+HCwH6iwtpkGEP7+I+Bc3JPf8z4CuK7dn36vtFVhtqCgvDygIi8skhy 5/g6DuZJpMt79ssHdZby/g4a9+HaJ1nNsdmwxuxd3KcFu+AdAvPPcoQGTi/c+FGR9z9B m8VAWGd5HUOgpwbTB7luDE/xhCI4EonHlRto1U/SgUHhM99enCcpZonrKjp4Aj742mwR QBpMMaBj4mswq42yEk8Xl2gMvRNjoAqyekmeemJOxJ70wPiP5TTdN/pG8/hviK1ptqDT xdRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CARj6FzP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id b8-20020a170903228800b001d09c539cd5si645897plh.133.2023.12.07.16.54.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CARj6FzP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id B98F3801B483; Thu, 7 Dec 2023 16:54:19 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572907AbjLHAxi (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1572881AbjLHAxN (ORCPT ); Thu, 7 Dec 2023 19:53:13 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67DA51989 for ; Thu, 7 Dec 2023 16:53:18 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d988d69812so18524337b3.0 for ; Thu, 07 Dec 2023 16:53:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996797; x=1702601597; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ffR5lRGBwRWz92voabMKfnUxOBX9ei9jc/R/TuPbdfY=; b=CARj6FzPCQ1sS6YCVmsJB2NuIjXU5lWVvtkUA/k/Ll427ms3YefKFgMH2SMQQf4uBm Y2ixiCLENlCGWmGmNRwE04NsaWWaoc4v0jHlbOrFkzgIyF6Pu+sLahVyJ5wZPPRlSi4L kLQRg9ZKrOYxNvgxMBRA/rmTJwWc4ML4fjSMjZRL50MXqbO50Ww/Kf4PZtBI4Q1sNKsS 7nI8xXGI/uXnpLQUYfGgpCCEL7LIrUJhauFjUzSAypMIPyixSiMFc2VlLXB8UKV9yYkX ZP3x0ifY9/bD9o8CH1cl3tnMc12Q65gmXLdDzgfdTONQe3SLK3C9tEsoS2nOSd2zRAxy g3Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996797; x=1702601597; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ffR5lRGBwRWz92voabMKfnUxOBX9ei9jc/R/TuPbdfY=; b=vApjSlgQ+lbNBSwNcTnytLCAMA7euktLxEJrTPohKPqAdzsUspMKgALUkYXKOblENp seXEAszsjReMCooFVUOtb5JinV/i8adGMieFxbuBs79G+N3M3nAuimyNL0vTbLM90D0S 5WWURJkOOB5wQJYUWaGNRoqhnP7ySs+YnFdVrIhXApFyOUUfn29IA6XwaigIa8Waie7z 0Uh+H3fMHwfcraCQjYTo5WgMjmGaKbRFId1ydJtnoHvLuMfImIIf8Ft+ALewlBqNz59Q /oCHQVfIyOewffQ8PgQO7mFjXk6ZXuTloTsVp8z+ASjOnwft74xkTcWNvGElVfyWuL1b 0xgw== X-Gm-Message-State: AOJu0YzYRGd4pUrVZkipShWb+V9gZGvV50MzVaDbpJKWlJluG9vXznYc jHYTd7syuIAc1ykEQJ/CbSe7waUYtakRccyQbg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:ccd5:0:b0:daf:6333:17c3 with SMTP id l204-20020a25ccd5000000b00daf633317c3mr42079ybf.1.1701996797568; Thu, 07 Dec 2023 16:53:17 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:42 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-12-almasrymina@google.com> Subject: [net-next v1 11/16] net: support non paged skb frags From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:19 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673064519870424 X-GMAIL-MSGID: 1784673064519870424 Make skb_frag_page() fail in the case where the frag is not backed by a page, and fix its relevant callers to handle this case. Correctly handle skb_frag refcounting in the page_pool_iovs case. Signed-off-by: Mina Almasry --- Changes in v1: - Fix illegal_highdma() (Yunsheng). - Rework napi_pp_put_page() slightly to reduce code churn (Willem). --- include/linux/skbuff.h | 42 +++++++++++++++++++++++++++++++++++------- net/core/dev.c | 3 ++- net/core/gro.c | 2 +- net/core/skbuff.c | 3 +++ net/ipv4/tcp.c | 3 +++ 5 files changed, 44 insertions(+), 9 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index b370eb8d70f7..851f448d2181 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -37,6 +37,8 @@ #endif #include #include +#include +#include /** * DOC: skb checksums @@ -3414,15 +3416,38 @@ static inline void skb_frag_off_copy(skb_frag_t *fragto, fragto->bv_offset = fragfrom->bv_offset; } +/* Returns true if the skb_frag contains a page_pool_iov. */ +static inline bool skb_frag_is_page_pool_iov(const skb_frag_t *frag) +{ + return page_is_page_pool_iov(frag->bv_page); +} + /** * skb_frag_page - retrieve the page referred to by a paged fragment * @frag: the paged fragment * - * Returns the &struct page associated with @frag. + * Returns the &struct page associated with @frag. Returns NULL if this frag + * has no associated page. */ static inline struct page *skb_frag_page(const skb_frag_t *frag) { - return frag->bv_page; + if (!page_is_page_pool_iov(frag->bv_page)) + return frag->bv_page; + + return NULL; +} + +/** + * skb_frag_page_pool_iov - retrieve the page_pool_iov referred to by fragment + * @frag: the fragment + * + * Returns the &struct page_pool_iov associated with @frag. Returns NULL if this + * frag has no associated page_pool_iov. + */ +static inline struct page_pool_iov * +skb_frag_page_pool_iov(const skb_frag_t *frag) +{ + return page_to_page_pool_iov(frag->bv_page); } /** @@ -3433,7 +3458,7 @@ static inline struct page *skb_frag_page(const skb_frag_t *frag) */ static inline void __skb_frag_ref(skb_frag_t *frag) { - get_page(skb_frag_page(frag)); + page_pool_page_get_many(frag->bv_page, 1); } /** @@ -3453,13 +3478,13 @@ bool napi_pp_put_page(struct page *page, bool napi_safe); static inline void napi_frag_unref(skb_frag_t *frag, bool recycle, bool napi_safe) { - struct page *page = skb_frag_page(frag); - #ifdef CONFIG_PAGE_POOL - if (recycle && napi_pp_put_page(page, napi_safe)) + if (recycle && napi_pp_put_page(frag->bv_page, napi_safe)) return; + page_pool_page_put_many(frag->bv_page, 1); +#else + put_page(skb_frag_page(frag)); #endif - put_page(page); } /** @@ -3499,6 +3524,9 @@ static inline void skb_frag_unref(struct sk_buff *skb, int f) */ static inline void *skb_frag_address(const skb_frag_t *frag) { + if (!skb_frag_page(frag)) + return NULL; + return page_address(skb_frag_page(frag)) + skb_frag_off(frag); } diff --git a/net/core/dev.c b/net/core/dev.c index 30667e4c3b95..1ae9257df441 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3709,8 +3709,9 @@ static int illegal_highdma(struct net_device *dev, struct sk_buff *skb) if (!(dev->features & NETIF_F_HIGHDMA)) { for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; + struct page *page = skb_frag_page(frag); - if (PageHighMem(skb_frag_page(frag))) + if (page && PageHighMem(page)) return 1; } } diff --git a/net/core/gro.c b/net/core/gro.c index 0759277dc14e..42d7f6755f32 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -376,7 +376,7 @@ static inline void skb_gro_reset_offset(struct sk_buff *skb, u32 nhoff) NAPI_GRO_CB(skb)->frag0 = NULL; NAPI_GRO_CB(skb)->frag0_len = 0; - if (!skb_headlen(skb) && pinfo->nr_frags && + if (!skb_headlen(skb) && pinfo->nr_frags && skb_frag_page(frag0) && !PageHighMem(skb_frag_page(frag0)) && (!NET_IP_ALIGN || !((skb_frag_off(frag0) + nhoff) & 3))) { NAPI_GRO_CB(skb)->frag0 = skb_frag_address(frag0); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 07f802f1adf1..2ce64f57a0f6 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2999,6 +2999,9 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, for (seg = 0; seg < skb_shinfo(skb)->nr_frags; seg++) { const skb_frag_t *f = &skb_shinfo(skb)->frags[seg]; + if (WARN_ON_ONCE(!skb_frag_page(f))) + return false; + if (__splice_segment(skb_frag_page(f), skb_frag_off(f), skb_frag_size(f), offset, len, spd, false, sk, pipe)) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 70a1bafbefba..e22681c4bfac 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2157,6 +2157,9 @@ static int tcp_zerocopy_receive(struct sock *sk, break; } page = skb_frag_page(frags); + if (WARN_ON_ONCE(!page)) + break; + prefetchw(page); pages[pages_to_map++] = page; length += PAGE_SIZE; From patchwork Fri Dec 8 00:52:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175507 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163626vqy; Thu, 7 Dec 2023 16:54:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IGM2+iE02x91j4kPt9sq+ql6rxpkiVZTjs4hgAOfyCDfo+xOrF2YzBft/UjiWbmBnxwHan/ X-Received: by 2002:a05:6358:51ce:b0:170:17eb:1e6 with SMTP id 14-20020a05635851ce00b0017017eb01e6mr3034684rwl.41.1701996870947; Thu, 07 Dec 2023 16:54:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996870; cv=none; d=google.com; s=arc-20160816; b=BJCFxFhvAlS8lvp4C11y5F2MoBCN6J5acPW44XISd7tKD2pHz8Dq48uUxRj+Db6PKo KqzKOowtMDoi56DUMjkty+ztLwje6jSU4S1cSg4XQr4ov9xUyjWH2giC3noa3XcO1YvO z5Sbg5yg/BXfMe+fVBWXBn1p9lgyLdc8cySrwR5icymTcz9hlA3nGfIIgLPR2jTDcG4M 7m9hQhA0Hr+kLWGhzUeIZV97Rc/v1wgMMj3G6ZqgDzgNdNSl35sEPJZ2ykep9xlCWOJr H52Zt8OyKK9GXW1OsOU0Z174sKFpxxiVhV3P2o0NuNGius5AISUL5QjOX/kq9T7HOk0S MBIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=+q3Q0nSqUx5ks54zyV6WkT43wIv1KXasQh2KaO7Camc=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=GP6kOn+iQY1sGho7Wnd0iOYVrAUAAIVwKn0fcXOPlZuskz82HI/ysJnkKYkMe+aZb5 NnAmHKfFPgphdpA27Xbu+UAPUE8WNO17XUpnTT/0wAIQaFak2Mh15vDTpB0wZshlk4eP pbq9bMy2pRhXnSSprHeEGmPZe73BNcD/48nK6kKRWySh+P/9Sb9STM5MBdLCEv5zr04Z JoRJGsrdZv1kuswHlIwSrmvLl5vmLMNDKr8pUgvPEUo7RqigAFTiTFrrE/zQjIqDyDyI lUFE2fsZ4SpTD1fV+NMnTpna8Y5at1GGpwux2USTaQWPXbLPmffzbhvrIZI36SkmWPdX RPDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=gmqAQl+a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id b15-20020a630c0f000000b005bcfc60520asi549015pgl.586.2023.12.07.16.54.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=gmqAQl+a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 0FBDB8088A84; Thu, 7 Dec 2023 16:54:27 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572928AbjLHAxl (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1572895AbjLHAxQ (ORCPT ); Thu, 7 Dec 2023 19:53:16 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CEA01999 for ; Thu, 7 Dec 2023 16:53:20 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-db547d41413so1395982276.0 for ; Thu, 07 Dec 2023 16:53:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996799; x=1702601599; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+q3Q0nSqUx5ks54zyV6WkT43wIv1KXasQh2KaO7Camc=; b=gmqAQl+aXWfBigYktGXkKzQ0WeVGdIx2AxomgDeKeLHHWTxCHEy7acwmd7fLUB9Tvw 6lsc27J39RpEpOQoU99ET3b+di0Oz+ghz9ASOJ3n6NRK5AXNwPTT6GiLUUTKzJb2dNOX sANS4MCFXy5wmjVaGVvDfK4bGFUwBQ0X8M3NiDFsRAANZNE6+hNiLVSpPL7iaVyYagbp SZ7/FPGhGcICK0j2zHEAgoQBxLtXpZRKCFGrEkX3VGHsuPOWRFEqEfVnRo9QaVA9YIwM JIJGVx5fgNVofvOD315vGHrmqJBWLLsB8BkHnO1d1uMp/+4YgBUvw/S8qHP6378KqUfQ AuFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996799; x=1702601599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+q3Q0nSqUx5ks54zyV6WkT43wIv1KXasQh2KaO7Camc=; b=ruiMIepvwWg0hNdX71Z8KKQIIb9n5CCMvAU+dRAn5F+CPAOYTfXhelyT+lBdYLi0Bb 5B/dZ7CeZ+wWibJ0ovLfCHVEzY6BRWxZWrm1D3iLuhYcQirQMpuybM4hr6pTneuTGCfF RREAg4C/4mIspGoi01/hsvjULEjNzhUSdBnVgTxtx7asQiJmrLdX1oNwxQtFcUmenI0O W0NnWYjPflSynyX0WH/J8P4Y8ddKwN3dHWjDhNgvWfl6Aj3fF6jHotYnqWyY8ifo5TaR ntH8YmvWeTnwstDmtrmsJXKbmhSg2IwoSbe5TNQ2FGmq/bP64Maw0p1XSN7BEY4ItzfW hY9A== X-Gm-Message-State: AOJu0Yx2BkSrRnezCjfegWSC0Qqs+ZUKyBxwiIoAcRoGlBWIPfpEKq6S 45igXYH5eHGaxHKzO2G1hwM5u7aMIwetYtqu1g== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:6902:14d:b0:db5:3aaf:5207 with SMTP id p13-20020a056902014d00b00db53aaf5207mr1545ybh.3.1701996799475; Thu, 07 Dec 2023 16:53:19 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:43 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-13-almasrymina@google.com> Subject: [net-next v1 12/16] net: add support for skbs with unreadable frags From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:27 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673071106408973 X-GMAIL-MSGID: 1784673071106408973 For device memory TCP, we expect the skb headers to be available in host memory for access, and we expect the skb frags to be in device memory and unaccessible to the host. We expect there to be no mixing and matching of device memory frags (unaccessible) with host memory frags (accessible) in the same skb. Add a skb->devmem flag which indicates whether the frags in this skb are device memory frags or not. __skb_fill_page_desc() now checks frags added to skbs for page_pool_iovs, and marks the skb as skb->devmem accordingly. Add checks through the network stack to avoid accessing the frags of devmem skbs and avoid coalescing devmem skbs with non devmem skbs. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- Changes in v1: - Rename devmem -> dmabuf (David). - Flip skb_frags_not_readable (Jakub). --- include/linux/skbuff.h | 14 +++++++- include/net/tcp.h | 5 +-- net/core/datagram.c | 6 ++++ net/core/gro.c | 5 ++- net/core/skbuff.c | 77 ++++++++++++++++++++++++++++++++++++------ net/ipv4/tcp.c | 3 ++ net/ipv4/tcp_input.c | 13 +++++-- net/ipv4/tcp_output.c | 5 ++- net/packet/af_packet.c | 4 +-- 9 files changed, 112 insertions(+), 20 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 851f448d2181..61de32ab04ea 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -817,6 +817,8 @@ typedef unsigned char *sk_buff_data_t; * @csum_level: indicates the number of consecutive checksums found in * the packet minus one that have been verified as * CHECKSUM_UNNECESSARY (max 3) + * @dmabuf: indicates that all the fragments in this skb are backed by + * dmabuf. * @dst_pending_confirm: need to confirm neighbour * @decrypted: Decrypted SKB * @slow_gro: state present at GRO time, slower prepare step required @@ -1003,7 +1005,7 @@ struct sk_buff { #if IS_ENABLED(CONFIG_IP_SCTP) __u8 csum_not_inet:1; #endif - + __u8 dmabuf:1; #if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS) __u16 tc_index; /* traffic control index */ #endif @@ -1778,6 +1780,12 @@ static inline void skb_zcopy_downgrade_managed(struct sk_buff *skb) __skb_zcopy_downgrade_managed(skb); } +/* Return true if frags in this skb are readable by the host. */ +static inline bool skb_frags_readable(const struct sk_buff *skb) +{ + return !skb->dmabuf; +} + static inline void skb_mark_not_on_list(struct sk_buff *skb) { skb->next = NULL; @@ -2480,6 +2488,10 @@ static inline void __skb_fill_page_desc(struct sk_buff *skb, int i, struct page *page, int off, int size) { __skb_fill_page_desc_noacc(skb_shinfo(skb), i, page, off, size); + if (page_is_page_pool_iov(page)) { + skb->dmabuf = true; + return; + } /* Propagate page pfmemalloc to the skb if we can. The problem is * that not all callers have unique ownership of the page but rely diff --git a/include/net/tcp.h b/include/net/tcp.h index 973555cb1d3f..0fbf198bdb55 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1017,7 +1017,7 @@ static inline int tcp_skb_mss(const struct sk_buff *skb) static inline bool tcp_skb_can_collapse_to(const struct sk_buff *skb) { - return likely(!TCP_SKB_CB(skb)->eor); + return likely(!TCP_SKB_CB(skb)->eor && skb_frags_readable(skb)); } static inline bool tcp_skb_can_collapse(const struct sk_buff *to, @@ -1025,7 +1025,8 @@ static inline bool tcp_skb_can_collapse(const struct sk_buff *to, { return likely(tcp_skb_can_collapse_to(to) && mptcp_skb_can_collapse(to, from) && - skb_pure_zcopy_same(to, from)); + skb_pure_zcopy_same(to, from) && + skb_frags_readable(to) == skb_frags_readable(from)); } /* Events passed to congestion control interface */ diff --git a/net/core/datagram.c b/net/core/datagram.c index 103d46fa0eeb..f28472ddbaa4 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -426,6 +426,9 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset, return 0; } + if (!skb_frags_readable(skb)) + goto short_copy; + /* Copy paged appendix. Hmm... why does this look so complicated? */ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; @@ -638,6 +641,9 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, if (msg && msg->msg_ubuf && msg->sg_from_iter) return msg->sg_from_iter(sk, skb, from, length); + if (!skb_frags_readable(skb)) + return -EFAULT; + frag = skb_shinfo(skb)->nr_frags; while (length && iov_iter_count(from)) { diff --git a/net/core/gro.c b/net/core/gro.c index 42d7f6755f32..26df48f1b355 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -390,6 +390,9 @@ static void gro_pull_from_frag0(struct sk_buff *skb, int grow) { struct skb_shared_info *pinfo = skb_shinfo(skb); + if (WARN_ON_ONCE(!skb_frags_readable(skb))) + return; + BUG_ON(skb->end - skb->tail < grow); memcpy(skb_tail_pointer(skb), NAPI_GRO_CB(skb)->frag0, grow); @@ -411,7 +414,7 @@ static void gro_try_pull_from_frag0(struct sk_buff *skb) { int grow = skb_gro_offset(skb) - skb_headlen(skb); - if (grow > 0) + if (grow > 0 && skb_frags_readable(skb)) gro_pull_from_frag0(skb, grow); } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 2ce64f57a0f6..50b1b7c2ef7b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1235,6 +1235,14 @@ void skb_dump(const char *level, const struct sk_buff *skb, bool full_pkt) struct page *p; u8 *vaddr; + if (skb_frag_is_page_pool_iov(frag)) { + printk("%sskb frag %d: not readable\n", level, i); + len -= frag->bv_len; + if (!len) + break; + continue; + } + skb_frag_foreach_page(frag, skb_frag_off(frag), skb_frag_size(frag), p, p_off, p_len, copied) { @@ -1812,6 +1820,9 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) if (skb_shared(skb) || skb_unclone(skb, gfp_mask)) return -EINVAL; + if (!skb_frags_readable(skb)) + return -EFAULT; + if (!num_frags) goto release; @@ -1982,8 +1993,12 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask) { int headerlen = skb_headroom(skb); unsigned int size = skb_end_offset(skb) + skb->data_len; - struct sk_buff *n = __alloc_skb(size, gfp_mask, - skb_alloc_rx_flag(skb), NUMA_NO_NODE); + struct sk_buff *n; + + if (!skb_frags_readable(skb)) + return NULL; + + n = __alloc_skb(size, gfp_mask, skb_alloc_rx_flag(skb), NUMA_NO_NODE); if (!n) return NULL; @@ -2309,14 +2324,16 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb, int newheadroom, int newtailroom, gfp_t gfp_mask) { - /* - * Allocate the copy buffer - */ - struct sk_buff *n = __alloc_skb(newheadroom + skb->len + newtailroom, - gfp_mask, skb_alloc_rx_flag(skb), - NUMA_NO_NODE); int oldheadroom = skb_headroom(skb); int head_copy_len, head_copy_off; + struct sk_buff *n; + + if (!skb_frags_readable(skb)) + return NULL; + + /* Allocate the copy buffer */ + n = __alloc_skb(newheadroom + skb->len + newtailroom, gfp_mask, + skb_alloc_rx_flag(skb), NUMA_NO_NODE); if (!n) return NULL; @@ -2655,6 +2672,9 @@ void *__pskb_pull_tail(struct sk_buff *skb, int delta) */ int i, k, eat = (skb->tail + delta) - skb->end; + if (!skb_frags_readable(skb)) + return NULL; + if (eat > 0 || skb_cloned(skb)) { if (pskb_expand_head(skb, 0, eat > 0 ? eat + 128 : 0, GFP_ATOMIC)) @@ -2808,6 +2828,9 @@ int skb_copy_bits(const struct sk_buff *skb, int offset, void *to, int len) to += copy; } + if (!skb_frags_readable(skb)) + goto fault; + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; skb_frag_t *f = &skb_shinfo(skb)->frags[i]; @@ -2996,6 +3019,9 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, /* * then map the fragments */ + if (!skb_frags_readable(skb)) + return false; + for (seg = 0; seg < skb_shinfo(skb)->nr_frags; seg++) { const skb_frag_t *f = &skb_shinfo(skb)->frags[seg]; @@ -3219,6 +3245,9 @@ int skb_store_bits(struct sk_buff *skb, int offset, const void *from, int len) from += copy; } + if (!skb_frags_readable(skb)) + goto fault; + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; int end; @@ -3298,6 +3327,9 @@ __wsum __skb_checksum(const struct sk_buff *skb, int offset, int len, pos = copy; } + if (!skb_frags_readable(skb)) + return 0; + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; @@ -3398,6 +3430,9 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, pos = copy; } + if (!skb_frags_readable(skb)) + return 0; + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; @@ -3888,7 +3923,9 @@ static inline void skb_split_inside_header(struct sk_buff *skb, skb_shinfo(skb1)->frags[i] = skb_shinfo(skb)->frags[i]; skb_shinfo(skb1)->nr_frags = skb_shinfo(skb)->nr_frags; + skb1->dmabuf = skb->dmabuf; skb_shinfo(skb)->nr_frags = 0; + skb->dmabuf = 0; skb1->data_len = skb->data_len; skb1->len += skb1->data_len; skb->data_len = 0; @@ -3902,6 +3939,7 @@ static inline void skb_split_no_header(struct sk_buff *skb, { int i, k = 0; const int nfrags = skb_shinfo(skb)->nr_frags; + const int dmabuf = skb->dmabuf; skb_shinfo(skb)->nr_frags = 0; skb1->len = skb1->data_len = skb->len - len; @@ -3935,6 +3973,16 @@ static inline void skb_split_no_header(struct sk_buff *skb, pos += size; } skb_shinfo(skb1)->nr_frags = k; + + if (skb_shinfo(skb)->nr_frags) + skb->dmabuf = dmabuf; + else + skb->dmabuf = 0; + + if (skb_shinfo(skb1)->nr_frags) + skb1->dmabuf = dmabuf; + else + skb1->dmabuf = 0; } /** @@ -4170,6 +4218,9 @@ unsigned int skb_seq_read(unsigned int consumed, const u8 **data, return block_limit - abs_offset; } + if (!skb_frags_readable(st->cur_skb)) + return 0; + if (st->frag_idx == 0 && !st->frag_data) st->stepped_offset += skb_headlen(st->cur_skb); @@ -5784,7 +5835,10 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, (from->pp_recycle && skb_cloned(from))) return false; - if (len <= skb_tailroom(to)) { + if (skb_frags_readable(from) != skb_frags_readable(to)) + return false; + + if (len <= skb_tailroom(to) && skb_frags_readable(from)) { if (len) BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len)); *delta_truesize = 0; @@ -5959,6 +6013,9 @@ int skb_ensure_writable(struct sk_buff *skb, unsigned int write_len) if (!pskb_may_pull(skb, write_len)) return -ENOMEM; + if (!skb_frags_readable(skb)) + return -EFAULT; + if (!skb_cloned(skb) || skb_clone_writable(skb, write_len)) return 0; @@ -6613,7 +6670,7 @@ void skb_condense(struct sk_buff *skb) { if (skb->data_len) { if (skb->data_len > skb->end - skb->tail || - skb_cloned(skb)) + skb_cloned(skb) || !skb_frags_readable(skb)) return; /* Nice, we can free page frag(s) right now */ diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index e22681c4bfac..5a3135e93d3d 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2140,6 +2140,9 @@ static int tcp_zerocopy_receive(struct sock *sk, skb = tcp_recv_skb(sk, seq, &offset); } + if (!skb_frags_readable(skb)) + break; + if (TCP_SKB_CB(skb)->has_rxtstamp) { tcp_update_recv_tstamps(skb, tss); zc->msg_flags |= TCP_CMSG_TS; diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0548f0c12155..a47f98187656 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5309,6 +5309,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, for (end_of_skbs = true; skb != NULL && skb != tail; skb = n) { n = tcp_skb_next(skb, list); + if (!skb_frags_readable(skb)) + goto skip_this; + /* No new bits? It is possible on ofo queue. */ if (!before(start, TCP_SKB_CB(skb)->end_seq)) { skb = tcp_collapse_one(sk, skb, list, root); @@ -5329,17 +5332,20 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, break; } - if (n && n != tail && mptcp_skb_can_collapse(skb, n) && + if (n && n != tail && skb_frags_readable(n) && + mptcp_skb_can_collapse(skb, n) && TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(n)->seq) { end_of_skbs = false; break; } +skip_this: /* Decided to skip this, advance start seq. */ start = TCP_SKB_CB(skb)->end_seq; } if (end_of_skbs || - (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN))) + (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) || + !skb_frags_readable(skb)) return; __skb_queue_head_init(&tmp); @@ -5383,7 +5389,8 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, if (!skb || skb == tail || !mptcp_skb_can_collapse(nskb, skb) || - (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN))) + (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) || + !skb_frags_readable(skb)) goto end; #ifdef CONFIG_TLS_DEVICE if (skb->decrypted != nskb->decrypted) diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index eb13a55d660c..c8c0a1cbaca5 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -2343,7 +2343,8 @@ static bool tcp_can_coalesce_send_queue_head(struct sock *sk, int len) if (unlikely(TCP_SKB_CB(skb)->eor) || tcp_has_tx_tstamp(skb) || - !skb_pure_zcopy_same(skb, next)) + !skb_pure_zcopy_same(skb, next) || + skb_frags_readable(skb) != skb_frags_readable(next)) return false; len -= skb->len; @@ -3227,6 +3228,8 @@ static bool tcp_can_collapse(const struct sock *sk, const struct sk_buff *skb) return false; if (skb_cloned(skb)) return false; + if (!skb_frags_readable(skb)) + return false; /* Some heuristics for collapsing over SACK'd could be invented */ if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) return false; diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index f92edba4c40f..33988106f237 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -2156,7 +2156,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev, } } - snaplen = skb->len; + snaplen = skb_frags_readable(skb) ? skb->len : skb_headlen(skb); res = run_filter(skb, sk, snaplen); if (!res) @@ -2276,7 +2276,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, } } - snaplen = skb->len; + snaplen = skb_frags_readable(skb) ? skb->len : skb_headlen(skb); res = run_filter(skb, sk, snaplen); if (!res) From patchwork Fri Dec 8 00:52:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175504 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163481vqy; Thu, 7 Dec 2023 16:54:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IFhP5sw829QlEKfPHs6SiIup85UqKaYs/ZuBPW56rCKdhL18EVlp90roXeqBEpPBJH57pf0 X-Received: by 2002:a17:90a:e612:b0:286:7471:41e4 with SMTP id j18-20020a17090ae61200b00286747141e4mr148112pjy.0.1701996845508; Thu, 07 Dec 2023 16:54:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996845; cv=none; d=google.com; s=arc-20160816; b=V7eA9pS3t3aQpHwgFxhTIpEjlIpv9brtBxLWQD4NzfqLUDTXncfsmor1IMST7Cyoju 0yxWj6MOEr7jW5uEzA1lm/5NUit1syUD2I6Dje3RUR5xnjLGvwZV0jqJp8r4jRX9dmfe wjCjK+ku9/T31Pi76MsKPY8iyJCqk+w7ZHZV6fJaikf5TUzXcVYRoOojESvh0kvBk129 fuQvFNakGz7Q2fV5XRz+OPky4v58L4IL4IAJmxl2f+Cz5K5u+ZULrwrPbWpJ6IXDqwZB C8NRDyI0zsSTtNN999Xv61EbC+VaVAvtwhPrrCESq01wzeoZJt50RoH3yEPrjIjXnmSs ngrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=o2HtSBl/MmwRbvRf64o4B77I6D1fDAMKeKnhqiyczlI=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=uZKTUZL2VRlF0n9cGonzwrvygxb8gDDrAXfpIBDf941wjcPanJTUZYpPl0SAGy82Ue zw/HTY1nRW55qq6jEIa5Y1D/e8lusA0+Z2bSkYpB+feXRQaC7s22rjQjPQ1HgZT74ggD OMFizqRMZavSZWFd0I/V0yHeo6C2np0Y3tFTBDzmcSigDWf5PexpsUTt23Kvi5/RYqsO sfDhCyhuVY64xmV8DAen6ltDdINKJ2avY2n2KIJuEEvykMT1SnGl8x5YbrcNWR1GUcEw IBvM5bTzfKGRRHzhK/hY9GCygCzLzr9uUqZGXw4SzogfHubc2t57EMi7viLI4hYM5zqt MdEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=t9QOWBmc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id n4-20020a17090ade8400b00286e36b62efsi668558pjv.74.2023.12.07.16.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=t9QOWBmc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7B6DB80DECDD; Thu, 7 Dec 2023 16:54:04 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572960AbjLHAxx (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444231AbjLHAxY (ORCPT ); Thu, 7 Dec 2023 19:53:24 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B77B4199A for ; Thu, 7 Dec 2023 16:53:22 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5d8d3271ff5so18156187b3.2 for ; Thu, 07 Dec 2023 16:53:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996802; x=1702601602; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o2HtSBl/MmwRbvRf64o4B77I6D1fDAMKeKnhqiyczlI=; b=t9QOWBmcCVW9kxtBFM+wAKQVV93wS7g3VfFemYMdEeaPZtIDwMZzdTNYx8yXB2dELJ uUoHM6LgTg3AjLWp8QC1IXwDw85OxdjFidRI/lt2tlr/HJk/QvS39vK1obXn7O+1ZJp1 H/n2V05TVoCPnRYlMWUYCuMERPL5KGj9rT8kjx2TuAkqtN0ydRnXoNu+h5TAVpJzt7EO 7f6Ky8BVEYCIsU/vAbhNMCBO/yLf5jnbDivg0lx0f+1Oq1wj4QyWsILjH7kcYIwRoKqq JWa/wzNfV550kLoID3xLDOBFn/MYKtRwTBVHKYYeuKwnT+ULGRrqltG8CrpuibvotqHe mYtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996802; x=1702601602; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o2HtSBl/MmwRbvRf64o4B77I6D1fDAMKeKnhqiyczlI=; b=M7ExZBfkwuO1fckc+gG0w6ZncJmR3LWxrBQCfyEFK67J+Ww14+MFjMN/eUgrvjYGDW Borj4sDBLLzVUJD70FdnKt/DlrJMyhkbWmC6TDy5rcBcJH41n70KrhuFtbIQTmmO6qZQ IZJYoKZKYLS5xaxw4znx7gtKCgSiSMcdKXAIRI1+JqK+PLEH5bW4N1c2P59VTTAz1sxa rJvGyN2rqZ9I/h6B/9yrMtcvuhzdlUp87XOV34RPuThW23LAqoxbbDfs+ANcZP4r1WYC Gwwzst6gzSVp1iPcXGz/haki7Dqjabx/i9mp9OyPfUDHan6zWRIsMl0JzgNnkzo6WyzB Jhpg== X-Gm-Message-State: AOJu0Yy0ktbRdKbRjA1famIm3ZUUTlD6UxS3XMuX/e+25ZQRyzsZ86Rx 8q8HZ5lDR7ZgGigBfNV4q/CZuDK4Vk74ZS+geQ== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a81:af41:0:b0:5d4:1846:3121 with SMTP id x1-20020a81af41000000b005d418463121mr43133ywj.8.1701996801728; Thu, 07 Dec 2023 16:53:21 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:44 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-14-almasrymina@google.com> Subject: [net-next v1 13/16] tcp: RX path for devmem TCP From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:04 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673043832741005 X-GMAIL-MSGID: 1784673043832741005 In tcp_recvmsg_locked(), detect if the skb being received by the user is a devmem skb. In this case - if the user provided the MSG_SOCK_DEVMEM flag - pass it to tcp_recvmsg_devmem() for custom handling. tcp_recvmsg_devmem() copies any data in the skb header to the linear buffer, and returns a cmsg to the user indicating the number of bytes returned in the linear buffer. tcp_recvmsg_devmem() then loops over the unaccessible devmem skb frags, and returns to the user a cmsg_devmem indicating the location of the data in the dmabuf device memory. cmsg_devmem contains this information: 1. the offset into the dmabuf where the payload starts. 'frag_offset'. 2. the size of the frag. 'frag_size'. 3. an opaque token 'frag_token' to return to the kernel when the buffer is to be released. The pages awaiting freeing are stored in the newly added sk->sk_user_pages, and each page passed to userspace is get_page()'d. This reference is dropped once the userspace indicates that it is done reading this page. All pages are released when the socket is destroyed. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- Changes in v1: - Added dmabuf_id to dmabuf_cmsg (David/Stan). - Devmem -> dmabuf (David). - Change tcp_recvmsg_dmabuf() check to skb->dmabuf (Paolo). - Use __skb_frag_ref() & napi_pp_put_page() for refcounting (Yunsheng). RFC v3: - Fixed issue with put_cmsg() failing silently. --- include/linux/socket.h | 1 + include/net/page_pool/helpers.h | 9 ++ include/net/sock.h | 2 + include/uapi/asm-generic/socket.h | 5 + include/uapi/linux/uio.h | 10 ++ net/ipv4/tcp.c | 190 +++++++++++++++++++++++++++++- net/ipv4/tcp_ipv4.c | 8 ++ 7 files changed, 220 insertions(+), 5 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index cfcb7e2c3813..fe2b9e2081bb 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -326,6 +326,7 @@ struct ucred { * plain text and require encryption */ +#define MSG_SOCK_DEVMEM 0x2000000 /* Receive devmem skbs as cmsg */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 2d4e0a2c5620..e7e2e89d3663 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -108,6 +108,15 @@ page_pool_iov_dma_addr(const struct page_pool_iov *ppiov) ((dma_addr_t)page_pool_iov_idx(ppiov) << PAGE_SHIFT); } +static inline unsigned long +page_pool_iov_virtual_addr(const struct page_pool_iov *ppiov) +{ + struct dmabuf_genpool_chunk_owner *owner = page_pool_iov_owner(ppiov); + + return owner->base_virtual + + ((unsigned long)page_pool_iov_idx(ppiov) << PAGE_SHIFT); +} + static inline struct netdev_dmabuf_binding * page_pool_iov_binding(const struct page_pool_iov *ppiov) { diff --git a/include/net/sock.h b/include/net/sock.h index 1d6931caf0c3..01029c855c1b 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -353,6 +353,7 @@ struct sk_filter; * @sk_txtime_unused: unused txtime flags * @ns_tracker: tracker for netns reference * @sk_bind2_node: bind node in the bhash2 table + * @sk_user_pages: xarray of pages the user is holding a reference on. */ struct sock { /* @@ -545,6 +546,7 @@ struct sock { struct rcu_head sk_rcu; netns_tracker ns_tracker; struct hlist_node sk_bind2_node; + struct xarray sk_user_pages; }; enum sk_pacing { diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..25a2f5255f52 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,11 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SO_DEVMEM_LINEAR 98 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 99 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/uio.h b/include/uapi/linux/uio.h index 059b1a9147f4..ad92e37699da 100644 --- a/include/uapi/linux/uio.h +++ b/include/uapi/linux/uio.h @@ -20,6 +20,16 @@ struct iovec __kernel_size_t iov_len; /* Must be size_t (1003.1g) */ }; +struct dmabuf_cmsg { + __u64 frag_offset; /* offset into the dmabuf where the frag starts. + */ + __u32 frag_size; /* size of the frag. */ + __u32 frag_token; /* token representing this frag for + * DEVMEM_DONTNEED. + */ + __u32 dmabuf_id; /* dmabuf id this frag belongs to. */ +}; + /* * UIO_MAXIOV shall be at least 16 1003.1g (5.4.1.1) */ diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 5a3135e93d3d..088b2b48bee0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -461,6 +461,7 @@ void tcp_init_sock(struct sock *sk) set_bit(SOCK_SUPPORT_ZC, &sk->sk_socket->flags); sk_sockets_allocated_inc(sk); + xa_init_flags(&sk->sk_user_pages, XA_FLAGS_ALLOC1); } EXPORT_SYMBOL(tcp_init_sock); @@ -2303,6 +2304,155 @@ static int tcp_inq_hint(struct sock *sk) return inq; } +/* On error, returns the -errno. On success, returns number of bytes sent to the + * user. May not consume all of @remaining_len. + */ +static int tcp_recvmsg_dmabuf(const struct sock *sk, const struct sk_buff *skb, + unsigned int offset, struct msghdr *msg, + int remaining_len) +{ + struct dmabuf_cmsg dmabuf_cmsg = { 0 }; + unsigned int start; + int i, copy, n; + int sent = 0; + int err = 0; + + do { + start = skb_headlen(skb); + + if (!skb->dmabuf) { + err = -ENODEV; + goto out; + } + + /* Copy header. */ + copy = start - offset; + if (copy > 0) { + copy = min(copy, remaining_len); + + n = copy_to_iter(skb->data + offset, copy, + &msg->msg_iter); + if (n != copy) { + err = -EFAULT; + goto out; + } + + offset += copy; + remaining_len -= copy; + + /* First a dmabuf_cmsg for # bytes copied to user + * buffer. + */ + memset(&dmabuf_cmsg, 0, sizeof(dmabuf_cmsg)); + dmabuf_cmsg.frag_size = copy; + err = put_cmsg(msg, SOL_SOCKET, SO_DEVMEM_LINEAR, + sizeof(dmabuf_cmsg), &dmabuf_cmsg); + if (err || msg->msg_flags & MSG_CTRUNC) { + msg->msg_flags &= ~MSG_CTRUNC; + if (!err) + err = -ETOOSMALL; + goto out; + } + + sent += copy; + + if (remaining_len == 0) + goto out; + } + + /* after that, send information of dmabuf pages through a + * sequence of cmsg + */ + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { + skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; + struct page_pool_iov *ppiov; + u64 frag_offset; + u32 user_token; + int end; + + /* skb->dmabuf should indicate that ALL the frags in + * this skb are dmabuf page_pool_iovs. We're checking + * for that flag above, but also check individual frags + * here. If the tcp stack is not setting skb->dmabuf + * correctly, we still don't want to crash here when + * accessing pgmap or priv below. + */ + if (!skb_frag_page_pool_iov(frag)) { + net_err_ratelimited("Found non-dmabuf skb with page_pool_iov"); + err = -ENODEV; + goto out; + } + + ppiov = skb_frag_page_pool_iov(frag); + end = start + skb_frag_size(frag); + copy = end - offset; + + if (copy > 0) { + copy = min(copy, remaining_len); + + frag_offset = page_pool_iov_virtual_addr(ppiov) + + skb_frag_off(frag) + offset - + start; + dmabuf_cmsg.frag_offset = frag_offset; + dmabuf_cmsg.frag_size = copy; + err = xa_alloc((struct xarray *)&sk->sk_user_pages, + &user_token, frag->bv_page, + xa_limit_31b, GFP_KERNEL); + if (err) + goto out; + + dmabuf_cmsg.frag_token = user_token; + dmabuf_cmsg.dmabuf_id = page_pool_iov_binding_id(ppiov); + + offset += copy; + remaining_len -= copy; + + err = put_cmsg(msg, SOL_SOCKET, + SO_DEVMEM_DMABUF, + sizeof(dmabuf_cmsg), + &dmabuf_cmsg); + if (err || msg->msg_flags & MSG_CTRUNC) { + msg->msg_flags &= ~MSG_CTRUNC; + xa_erase((struct xarray *)&sk->sk_user_pages, + user_token); + if (!err) + err = -ETOOSMALL; + goto out; + } + + __skb_frag_ref(frag); + + sent += copy; + + if (remaining_len == 0) + goto out; + } + start = end; + } + + if (!remaining_len) + goto out; + + /* if remaining_len is not satisfied yet, we need to go to the + * next frag in the frag_list to satisfy remaining_len. + */ + skb = skb_shinfo(skb)->frag_list ?: skb->next; + + offset = offset - start; + } while (skb); + + if (remaining_len) { + err = -EFAULT; + goto out; + } + +out: + if (!sent) + sent = err; + + return sent; +} + /* * This routine copies from a sock struct into the user buffer. * @@ -2316,6 +2466,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, int *cmsg_flags) { struct tcp_sock *tp = tcp_sk(sk); + int last_copied_dmabuf = -1; /* uninitialized */ int copied = 0; u32 peek_seq; u32 *seq; @@ -2493,15 +2644,44 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, } if (!(flags & MSG_TRUNC)) { - err = skb_copy_datagram_msg(skb, offset, msg, used); - if (err) { - /* Exception. Bailout! */ - if (!copied) - copied = -EFAULT; + if (last_copied_dmabuf != -1 && + last_copied_dmabuf != skb->dmabuf) break; + + if (!skb->dmabuf) { + err = skb_copy_datagram_msg(skb, offset, msg, + used); + if (err) { + /* Exception. Bailout! */ + if (!copied) + copied = -EFAULT; + break; + } + } else { + if (!(flags & MSG_SOCK_DEVMEM)) { + /* skb->dmabuf skbs can only be received + * with the MSG_SOCK_DEVMEM flag. + */ + if (!copied) + copied = -EFAULT; + + break; + } + + err = tcp_recvmsg_dmabuf(sk, skb, offset, msg, + used); + if (err <= 0) { + if (!copied) + copied = -EFAULT; + + break; + } + used = err; } } + last_copied_dmabuf = skb->dmabuf; + WRITE_ONCE(*seq, *seq + used); copied += used; len -= used; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 86cc6d36f818..986398cc2f65 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -2501,6 +2501,14 @@ static void tcp_md5sig_info_free_rcu(struct rcu_head *head) void tcp_v4_destroy_sock(struct sock *sk) { struct tcp_sock *tp = tcp_sk(sk); + struct page *page; + unsigned long index; + + xa_for_each(&sk->sk_user_pages, index, page) + if (WARN_ON_ONCE(!napi_pp_put_page(page, false))) + page_pool_page_put_many(page, 1); + + xa_destroy(&sk->sk_user_pages); trace_tcp_destroy_sock(sk); From patchwork Fri Dec 8 00:52:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175502 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163472vqy; Thu, 7 Dec 2023 16:54:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHDwC5aXIsAjRkQGvhyyqjfCj7w/N79U9/cGIS/AlLcW8KdDb/liIy+3srwDKdPXYTzSZ11 X-Received: by 2002:a17:902:7808:b0:1d0:6ffd:8375 with SMTP id p8-20020a170902780800b001d06ffd8375mr2399589pll.128.1701996844518; Thu, 07 Dec 2023 16:54:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996844; cv=none; d=google.com; s=arc-20160816; b=i9bOBtQWqPMnX6IFIzqsxHJymV/Y6rshb9NEPQEMq4Bdgp3/h8LhDdrc2wDGiesz0e DtLo4MGBEHQuRkU54Js0DQt8KgpEsPqZpXqneiPFHiUDSWxugEDKf8PDNHnJLRv3tBG9 HvYcYPs5rJnIk6G+x3XDhzSEMrHzyG2aU9JCKwZ5zbaW6+diLEGMXsVATMh6bUynYfsm HJyGW16+qYy8MK4UVQGuPa0vdF4MJBT/BepzNxjRtd7VSSuP35a7h7LZBRBVx3OmZF0F 0hv3dCnJf3RdI0Yblxoz482jK/RC1d7QuXSYSHstWv3ijMqK909cV2/jhxf2GhTq4iNZ 5Apg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=wUh6+R8PyJI5UjQ2PsAiU6Bt8ES298Z5QR8+6xQcmYE=; fh=jiJDDAUhxxqnLDOaLfA63yYuVUdQlpct3IK6rtRzo8Q=; b=NRxmaIaJg/8ToxnIo2WRHgE+0iBdYDZtk/wbiFjbMOYNilW1lQyUDs6SEdcAtX+sgJ 6xfazByl2WSJ42ibihpqVW9lRJTkHzJbhYVD7F1gDKHlr3e0xbMiW2afZmWRfg7BO4ye IM5o0advLTyHZD0XERGbRKZfjC+2twUfI25FkVbuD00DxJD/fQazv1RY960rDVd1KW/l 0X9nFI3esTrvWioGac4/4GA7NNKGZT+RQZsc3ZUd5j8VLLzEnqqY1FXfVd89WNn3WMpq Nsk1Nip+GsrycuNXjKjM6laBr9dk7zLWDnWFkpDq7tpGeJpuPjVb4Kw93kRhOingW57A 9wOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=2TIiaxC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id g20-20020a170902c39400b001cfb65f4c9asi607135plg.570.2023.12.07.16.54.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=2TIiaxC0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 77FCF80DECE1; Thu, 7 Dec 2023 16:54:03 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572937AbjLHAxp (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444228AbjLHAxY (ORCPT ); Thu, 7 Dec 2023 19:53:24 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D495A128 for ; Thu, 7 Dec 2023 16:53:24 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5d42c43d8daso10065457b3.0 for ; Thu, 07 Dec 2023 16:53:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996804; x=1702601604; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wUh6+R8PyJI5UjQ2PsAiU6Bt8ES298Z5QR8+6xQcmYE=; b=2TIiaxC0IIxvIXQ99aZ23/ECXzt89xzHAz8vDzuidfu7txru05e/Q95jZRIPfbTIRU hKjqdzpA4u+6RZOKq1RxaEedOUSHBV4IFACdVO/GOECX4sBca8VsOFp719jGF7hwka+5 D1DFwVkJFRieD79PWHpIq/q1ywYDWoPRKhhzPLqlWMyTj5IaUR12wdT3MC+XwbcUg/us 8SviaMHAuKerm8BZEksmxb1prLWrynhmjxZlYSMM2d7P6g1tiWrSIMPgbKHAb8TSC5xA 4IwY5GVqli0f0OOEvtMFo5C3NXialwES/oiAPvLfxFdl3IkWXpgA60r4JaCVij2Lzv2C Wuyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996804; x=1702601604; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wUh6+R8PyJI5UjQ2PsAiU6Bt8ES298Z5QR8+6xQcmYE=; b=VjUM29HcFIBRVIqq7820kWuAMj3Ac1ON8AoKysZZg/2VS8e3fQMptpt721lWQVnfSe Z9FcKvOtnKMUCfGk5k8+HIaAqu4HflgHr5HSNu5L/G88Qi7CoTvp8HKtdT+nGb/6bqnT 5URqLLsr8CUyXaIttMJ1+rX7dxT+QEUjkD9GTc6yerbheACfES9vyot3ihl6mwjySs0q sBuO+SoJDPdXu87zvNI+hnqC283oqdMe8pu0WQySDTOL9SmLq6dCTGum92LCvfd7MuuV vXnzAAlnEGJv5ryxDp9oXk2VHE5uai9g4K5Pnpm3rIKitzvt+ZD7NsxapzTRI79fZ/Cf kt3w== X-Gm-Message-State: AOJu0YwlMRnxK5TkcVSEoTk9/Ym4Sf5eDekW3RwJJVzZNCwucRqYGPHm aAU0/MgqDqHEAWboRjYUzfAeGUza/ErYy1/W9w== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a25:9d10:0:b0:ca3:3341:6315 with SMTP id i16-20020a259d10000000b00ca333416315mr2159ybp.0.1701996804012; Thu, 07 Dec 2023 16:53:24 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:45 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-15-almasrymina@google.com> Subject: [net-next v1 14/16] net: add SO_DEVMEM_DONTNEED setsockopt to release RX frags From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Willem de Bruijn , Kaiyuan Zhang X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:03 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673043178892670 X-GMAIL-MSGID: 1784673043178892670 Add an interface for the user to notify the kernel that it is done reading the devmem dmabuf frags returned as cmsg. The kernel will drop the reference on the frags to make them available for re-use. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- Changes in v1: - devmemtoken -> dmabuf_token (David). - Use napi_pp_put_page() for refcounting (Yunsheng). --- include/uapi/asm-generic/socket.h | 1 + include/uapi/linux/uio.h | 4 ++++ net/core/sock.c | 38 +++++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+) diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 25a2f5255f52..1acb77780f10 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,7 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SO_DEVMEM_DONTNEED 97 #define SO_DEVMEM_LINEAR 98 #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 99 diff --git a/include/uapi/linux/uio.h b/include/uapi/linux/uio.h index ad92e37699da..65f33178a601 100644 --- a/include/uapi/linux/uio.h +++ b/include/uapi/linux/uio.h @@ -30,6 +30,10 @@ struct dmabuf_cmsg { __u32 dmabuf_id; /* dmabuf id this frag belongs to. */ }; +struct dmabuf_token { + __u32 token_start; + __u32 token_count; +}; /* * UIO_MAXIOV shall be at least 16 1003.1g (5.4.1.1) */ diff --git a/net/core/sock.c b/net/core/sock.c index fef349dd72fa..521bdc4ff260 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1051,6 +1051,41 @@ static int sock_reserve_memory(struct sock *sk, int bytes) return 0; } +static noinline_for_stack int +sock_devmem_dontneed(struct sock *sk, sockptr_t optval, unsigned int optlen) +{ + struct dmabuf_token tokens[128]; + unsigned int num_tokens, i, j; + int ret; + + if (sk->sk_type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP) + return -EBADF; + + if (optlen % sizeof(struct dmabuf_token) || optlen > sizeof(tokens)) + return -EINVAL; + + num_tokens = optlen / sizeof(struct dmabuf_token); + if (copy_from_sockptr(tokens, optval, optlen)) + return -EFAULT; + + ret = 0; + for (i = 0; i < num_tokens; i++) { + for (j = 0; j < tokens[i].token_count; j++) { + struct page *page = xa_erase(&sk->sk_user_pages, + tokens[i].token_start + j); + + if (page) { + if (WARN_ON_ONCE(!napi_pp_put_page(page, + false))) + page_pool_page_put_many(page, 1); + ret++; + } + } + } + + return ret; +} + void sockopt_lock_sock(struct sock *sk) { /* When current->bpf_ctx is set, the setsockopt is called from @@ -1538,6 +1573,9 @@ int sk_setsockopt(struct sock *sk, int level, int optname, break; } + case SO_DEVMEM_DONTNEED: + ret = sock_devmem_dontneed(sk, optval, optlen); + break; default: ret = -ENOPROTOOPT; break; From patchwork Fri Dec 8 00:52:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163505vqy; Thu, 7 Dec 2023 16:54:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmuw1Dss3lmBvXN9FW3sD+s/kKPJEbkkzMJf3CxTFCGrpw+KNwKusM9f5CqLq0Vu/68cOx X-Received: by 2002:a17:902:c943:b0:1d0:7535:8b94 with SMTP id i3-20020a170902c94300b001d075358b94mr4065874pla.97.1701996849342; Thu, 07 Dec 2023 16:54:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996849; cv=none; d=google.com; s=arc-20160816; b=kN0wIvNGqz8kjzDJQt9ODnVjOdU8NwiK0JFjY0wJrHnwoHxIZLGQ6+TB273OBRkslA StskB7OlQDj3zqgEYG2U9hu9IP9E4EkPDww42QKJw6C2jTlgrV0KLKgdE9JJ/5S38PgL 8hDeJKUP9u/Zl6KMGGpFdPrKh9vC1P18pib5zwj5hutCBDaeV315inM7HafQMkavFtiv cZqzrnPvWsiOrRl7ivYm97Vv6kq1I6/IrH2UO26JFgblGqoTs/Fjk/wUC5msdYgNYyVY tv59ehQEyFG6RlsihztO4b6kzBW5Ysjoy1yLtMscwzVN28dVbmWH+aD4weugtjHFXeaT z8Ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=DG/lnECqv2VNc8nFmC6Bul9upujMtEgUOT8jD86bXxEFOmvHVOfl8TIdPpXhRZvU3t dcqs42fvSDCK/3Deug61iRusm0oRI8heonIumTMz1Gki8lt1nlMtFJrfkZrUYoUD6QCs f5qOhSsAbtOatepgqPmp8+8hPCX2Snw0EnL2XFNYfkjLgXR8F9f3y4u0te0/wuW7H+LP HC3IHLA8v1npzlLU3Pj2dQvPJcyrhuIxEYykX8HU7EN44Cp+KDyPdbA2E/BJZsfULydT EKHuP/Vb3RA/lUSd+BfaqJYJYR2gkLso3DLi4MFAkeHWb8Ywz44MdzbIkAKxQQrhTXt8 ZJpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mKG3LIYS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id u8-20020a170902e5c800b001d2e6c3f136si643214plf.96.2023.12.07.16.54.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mKG3LIYS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B391280DECEA; Thu, 7 Dec 2023 16:54:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444267AbjLHAx6 (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444192AbjLHAxY (ORCPT ); Thu, 7 Dec 2023 19:53:24 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C08DF19A5 for ; Thu, 7 Dec 2023 16:53:26 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d10f5bf5d9so15543227b3.3 for ; Thu, 07 Dec 2023 16:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996806; x=1702601606; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; b=mKG3LIYSvgUaZXWqG3b0c+K1M73nhULqOdoFoCOgyTKk66VBWzbX96bgjj6q2coEaB +alO1hqSD/lN3OsaBmALW4aN/tPw1MVkUDbOCh4T2mdmCu1uNA90oIUd9rQSly0q8ySC 8F3adQtYkoDGvOFufA+3Ig4bABNF5tDVnew6MHT3sdge26xEWlNaMlGfYhzXcLSBKgLe ur51g6f3iPQT8ses4LqLU3GkMuxWErC6QeP/+h+Nqk+DxPq5JR0KvlUq2R0WuIwOggHy GfqR22DwgU5QoDj18nEnB80rCnzSG39OeKXJby62ESV6fqAgVpSL8isnufEvZn5hEtay nmUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996806; x=1702601606; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; b=fYIcgWk32xucZIDloOTnxgSWCG7DJdGWjCu9nI0gLvWVjcQu8QJ8xdzY6I1tVKLdso LdTVjNzQ2RF9BwuJzBxrga5J53N2YUa5OH2zAN/M40LOsY4J2/O8ctu9DGUMxbfiAfAn fDMF3Xc2AM+NxRm3UKWS/Oakk8XiL1U/lBDA/irGt7AieT8Yh9SmMO6zaP7JMzVRzA95 ncTZtf3eonw7nnysgx3EVuxDD/6ypWn+yxu3qndAy7fvI56loVU/QBEXnwMBGs9ZQAQt nuPwZEcz8nOLN7orkxtuguhYaFw09Lns3i7cx47+bmCxaJ3TJAC4wkVUhVeUnViyBUN5 /k1A== X-Gm-Message-State: AOJu0YzXZX2vVq3AMahzaT5+ZiYtcZSJcGfOCQ2vr+EsEoLsIJTlDw9a JRSBiw7m055al+NckYhVAmQro7tFMPKtLOgfBg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:844:b0:5d6:cb62:4793 with SMTP id bz4-20020a05690c084400b005d6cb624793mr51347ywb.0.1701996806041; Thu, 07 Dec 2023 16:53:26 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:46 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-16-almasrymina@google.com> Subject: [net-next v1 15/16] net: add devmem TCP documentation From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:08 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673048401873348 X-GMAIL-MSGID: 1784673048401873348 Signed-off-by: Mina Almasry --- Documentation/networking/devmem.rst | 270 ++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 Documentation/networking/devmem.rst diff --git a/Documentation/networking/devmem.rst b/Documentation/networking/devmem.rst new file mode 100644 index 000000000000..ed0d9c88b708 --- /dev/null +++ b/Documentation/networking/devmem.rst @@ -0,0 +1,270 @@ + +================= +Device Memory TCP +================= + + +Intro +===== + +Device memory TCP (devmem TCP) enables receiving data directly into device +memory (dmabuf). The feature is currently implemented for TCP sockets. + + +Opportunity +----------- + +A large amount of data transfers have device memory as the source and/or +destination. Accelerators drastically increased the volume of such transfers. +Some examples include: + +- Distributed training, where ML accelerators, such as GPUs on different hosts, + exchange data among them. + +- Distributed raw block storage applications transfer large amounts of data with + remote SSDs, much of this data does not require host processing. + +Today, the majority of the Device-to-Device data transfers the network are +implemented as the following low level operations: Device-to-Host copy, +Host-to-Host network transfer, and Host-to-Device copy. + +The implementation is suboptimal, especially for bulk data transfers, and can +put significant strains on system resources such as host memory bandwidth and +PCIe bandwidth. + +Devmem TCP optimizes this use case by implementing socket APIs that enable +the user to receive incoming network packets directly into device memory. + +Packet payloads go directly from the NIC to device memory. + +Packet headers go to host memory and are processed by the TCP/IP stack +normally. The NIC must support header split to achieve this. + +Advantages: + +- Alleviate host memory bandwidth pressure, compared to existing + network-transfer + device-copy semantics. + +- Alleviate PCIe bandwidth pressure, by limiting data transfer to the lowest + level of the PCIe tree, compared to traditional path which sends data through + the root complex. + + +More Info +--------- + + slides, video + https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html + + patchset + [RFC PATCH v3 00/12] Device Memory TCP + https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@google.com/T/ + + +Interface +========= + +Example +------- + +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up +the RX path of this API. + +NIC Setup +--------- + +Header split, flow steering, & RSS are required features for devmem TCP. + +Header split is used to split incoming packets into a header buffer in host +memory, and a payload buffer in device memory. + +Flow steering & RSS are used to ensure that only flows targeting devmem land on +RX queue bound to devmem. + +Enable header split & flow steering: + +:: + + # enable header split (assuming priv-flag) + ethtool --set-priv-flags eth1 enable-header-split on + + # enable flow steering + ethtool -K eth1 ntuple on + +Configure RSS to steer all traffic away from the target RX queue (queue 15 in +this example): + +:: + + ethtool --set-rxfh-indir eth1 equal 15 + + +The user must bind a dmabuf to any number of RX queues on a given NIC using +netlink API: + +:: + + /* Bind dmabuf to NIC RX queue 15 */ + struct netdev_queue *queues; + queues = malloc(sizeof(*queues) * 1); + + queues[0]._present.type = 1; + queues[0]._present.idx = 1; + queues[0].type = NETDEV_RX_QUEUE_TYPE_RX; + queues[0].idx = 15; + + *ys = ynl_sock_create(&ynl_netdev_family, &yerr); + + req = netdev_bind_rx_req_alloc(); + netdev_bind_rx_req_set_ifindex(req, 1 /* ifindex */); + netdev_bind_rx_req_set_dmabuf_fd(req, dmabuf_fd); + __netdev_bind_rx_req_set_queues(req, queues, n_queue_index); + + rsp = netdev_bind_rx(*ys, req); + + dmabuf_id = rsp->dmabuf_id; + + +The netlink API returns a dmabuf_id: a unique ID that refers to this dmabuf +that has been bound. + +Socket Setup +------------ + +The socket must be flow steering to the dmabuf bound RX queue: + +:: + + ethtool -N eth1 flow-type tcp4 ... queue 15, + + +Receiving data +-------------- + +The user application must signal to the kernel that it is capable of receiving +devmem data by passing the MSG_SOCK_DEVMEM flag to recvmsg: + +:: + + ret = recvmsg(fd, &msg, MSG_SOCK_DEVMEM); + +Applications that do not specify the MSG_SOCK_DEVMEM flag will receive an EFAULT +on devmem data. + +Devmem data is received directly into the dmabuf bound to the NIC in 'NIC +Setup', and the kernel signals such to the user via the SCM_DEVMEM_* cmsgs: + +:: + + for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level != SOL_SOCKET || + (cm->cmsg_type != SCM_DEVMEM_DMABUF && + cm->cmsg_type != SCM_DEVMEM_LINEAR)) + continue; + + dmabuf_cmsg = (struct dmabuf_cmsg *)CMSG_DATA(cm); + + if (cm->cmsg_type == SCM_DEVMEM_DMABUF) { + /* Frag landed in dmabuf. + * + * dmabuf_cmsg->dmabuf_id is the dmabuf the + * frag landed on. + * + * dmabuf_cmsg->frag_offset is the offset into + * the dmabuf where the frag starts. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + * + * dmabuf_cmsg->frag_token is a token used to + * refer to this frag for later freeing. + */ + + struct dmabuf_token token; + token.token_start = dmabuf_cmsg->frag_token; + token.token_count = 1; + continue; + } + + if (cm->cmsg_type == SCM_DEVMEM_LINEAR) + /* Frag landed in linear buffer. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + */ + continue; + + } + +Applications may receive 2 cmsgs: + +- SCM_DEVMEM_DMABUF: this indicates the fragment landed in the dmabuf indicated + by dmabuf_id. + +- SCM_DEVMEM_LINEAR: this indicates the fragment landed in the linear buffer. + This typically happens when the NIC is unable to split the packet at the + header boundary, such that part (or all) of the payload landed in host + memory. + +Applications may receive no SO_DEVMEM_* cmsgs. That indicates non-devmem, +regular TCP data that landed on an RX queue not bound to a dmabuf. + + +Freeing frags +------------- + +Frags received via SCM_DEVMEM_DMABUF are pinned by the kernel while the user +processes the frag. The user must return the frag to the kernel via +SO_DEVMEM_DONTNEED: + +:: + + ret = setsockopt(client_fd, SOL_SOCKET, SO_DEVMEM_DONTNEED, &token, + sizeof(token)); + +The user must ensure the tokens are returned to the kernel in a timely manner. +Failure to do so will exhaust the limited dmabuf that is bound to the RX queue +and will lead to packet drops. + + +Implementation & Caveats +======================== + +Unreadable skbs +--------------- + +Devmem payloads are inaccessible to the kernel processing the packets. This +results in a few quirks for payloads of devmem skbs: + +- Loopback is not functional. Loopback relies on copying the payload, which is + not possible with devmem skbs. + +- Software checksum calculation fails. + +- TCP Dump and bpf can't access devmem packet payloads. + + +Testing +======= + +More realistic example code can be found in the kernel source under +tools/testing/selftests/net/ncdevmem.c + +ncdevmem is a devmem TCP netcat. It works very similarly to netcat, but +receives data directly into a udmabuf. + +To run ncdevmem, you need to run it a server on the machine under test, and you +need to run netcat on a peer to provide the TX data. + +ncdevmem has a validation mode as well that expects a repeating pattern of +incoming data and validates it as such: + +:: + + # On server: + ncdevmem -s -c -f eth1 -d 3 -n 0000:06:00.0 -l \ + -p 5201 -v 7 + + # On client: + yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ + tr \\n \\0 | head -c 5G | nc 5201 -p 5201 From patchwork Fri Dec 8 00:52:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175508 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163766vqy; Thu, 7 Dec 2023 16:54:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IFV0Br7qeCdaIIE3wOqklkC2cvvpr/x6Fo0hVSZGBL63KK5Xjgf8rQdGLABig/kRoBTCwB/ X-Received: by 2002:a17:903:120e:b0:1d0:6ffd:e2cf with SMTP id l14-20020a170903120e00b001d06ffde2cfmr4121197plh.105.1701996896756; Thu, 07 Dec 2023 16:54:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996896; cv=none; d=google.com; s=arc-20160816; b=Px1sefeZNK5Xlk0+zUzZmQT0VCsG85362n9Uhfdp46bWN83RrCWVXZxEzcnAISaT0F Z4cuX9uimBBXXYfe+lsBbApDNFKCwg19XUZJTf9LtMEtemT6/EG1rorcoFLZSenRvwb4 BEuieHbBvN1FPasqgVUoyhzkuKl1/z8Fcv3hcV21U2Yee1kGo6W0v+zkY4WIb5PmDSep s9MuVc+7HOU25zrLRf8QTc0eUp3QISBoqnv/RepmF9wC/n6XQ+0hrPmxDQZCubEA16fF 48lUnx386qW3fmVsjTZt9+VHzNp7IScSYi70uzjik5zmRg7KEERKsN1kt1PrfG/dt1PW Eerw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=8LAFahE4f6lW6U4R0EfMD5JqvdtDASlh+SiOPl8ncLM=; fh=dQ0Z1OOFK5ChiAlIZaZLnTEZu41VePMcBmOe3O0ONkE=; b=R6qGm/HoOUxVMy44VoKQyVXNGpAA0hcXVjtwAF4rMfwTXxCxIOkyqSYXEwgQVEnK9U h8Am5ox2JX/RsMoSbyJpvLETmXx9JVlAtx0w+Jiod0PtXX5TSldh85iFjnYMhxq8DXET fBdRi8EYNrdx/Ybwu9qU+maXj6cfkmbgTUArdKDgpjnUB/yPQLIdhjA1d/nUvP96nXey 3lu6pqlNptV6Fy9LR1hnEkaJTB82egD657q8xcAlYxNesyGS0fiaKVmecKp7DJ7FzewH tAO8JKF21X/MyjVwJowK4J8ooXy1/VgrXMe2YBw5XLu7mXWktIDa2LZKY6PuHIrWuNju m54A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Y7oNXzs9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id p8-20020a170903248800b001d0c906f613si584773plw.492.2023.12.07.16.54.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Y7oNXzs9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 922D88228820; Thu, 7 Dec 2023 16:54:52 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1572897AbjLHAyD (ORCPT + 99 others); Thu, 7 Dec 2023 19:54:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444242AbjLHAxZ (ORCPT ); Thu, 7 Dec 2023 19:53:25 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD61319AC for ; Thu, 7 Dec 2023 16:53:28 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5c941936f7fso9790307b3.0 for ; Thu, 07 Dec 2023 16:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996808; x=1702601608; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8LAFahE4f6lW6U4R0EfMD5JqvdtDASlh+SiOPl8ncLM=; b=Y7oNXzs9yj0JOSbCMwO0P7k/0Gy/PW8g8Rdbsr9VG8Es5LP04F0RuLDkAFnvq/0hF6 zVuuJ1lsgTLvDykmNGZBKy8vDuYxwV7VtIu/2qOS3LQ0r7mOoEuc4jJySCMCOeaCw5ia qKhmoEraG3DdceIscMlsEI1rWDGpl5H9fj0eSfk6umEsmFVZMdphBnkCnBgFE2uorHHj Oz/Ogw7pO1Cv/m8u5MccDhdGyOR6my9NqA+VWeCUYeoWl9VFkFPwgfBVeXh5y6JP29W7 x3stEq1KceVAfMSEg3787QXnS4lET/iZt2t82Z0RP0YdBZGHMqO+k5ecVidDaNkQkAY4 PmNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996808; x=1702601608; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8LAFahE4f6lW6U4R0EfMD5JqvdtDASlh+SiOPl8ncLM=; b=mvKFw9IJV63K52qtG5vNhBbYtTuFD17sYeLaVwQJ/ZXu7/KoBOrBKsoBB4EKQeqLFK YIrB8sfOKOb1Z6pSFqVuGWdRlbB0ro1W0TDQRVOTqRI/GMcL/3hzcXci5HmAd0KfNYxY NttbtuWnHwTa8WEHA9yE06sLvMNI49v0IIR6ZmcE+KmcYf9oR8EvcVMJquTsBAoLaFGz iEkQNU6vwYbH44zJL0aUAO1kpoCXAP3DcD6k/lWKbKc/iAv2OvfA6IiL9zaXDUBWUmgT xp5alUYQ2HwFv1MzeWXLMK8xcq0Oqk4119V4rSZJqP4mWCxkbX9W6TkdYH9Io+GmXScG 0zqQ== X-Gm-Message-State: AOJu0Yxoj/RJG5FMl54JMmfAJzxXELUlVdQRiO1onxdx2fBYBBeI+SXE uDU+fmltDjply4rOCRpLEo+n3M7NrBNJb5pslQ== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:6902:14d:b0:db5:3aaf:5207 with SMTP id p13-20020a056902014d00b00db53aaf5207mr1548ybh.3.1701996807958; Thu, 07 Dec 2023 16:53:27 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:47 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-17-almasrymina@google.com> Subject: [net-next v1 16/16] selftests: add ncdevmem, netcat for devmem TCP From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt , Stanislav Fomichev X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:52 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673097600445676 X-GMAIL-MSGID: 1784673097600445676 ncdevmem is a devmem TCP netcat. It works similarly to netcat, but it sends and receives data using the devmem TCP APIs. It uses udmabuf as the dmabuf provider. It is compatible with a regular netcat running on a peer, or a ncdevmem running on a peer. In addition to normal netcat support, ncdevmem has a validation mode, where it sends a specific pattern and validates this pattern on the receiver side to ensure data integrity. Suggested-by: Stanislav Fomichev Signed-off-by: Mina Almasry --- Changes in v1: - Many more general cleanups (Willem). - Removed driver reset (Jakub). - Removed hardcoded if index (Paolo). RFC v2: - General cleanups (Willem). --- tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 5 + tools/testing/selftests/net/ncdevmem.c | 489 +++++++++++++++++++++++++ 3 files changed, 495 insertions(+) create mode 100644 tools/testing/selftests/net/ncdevmem.c diff --git a/tools/testing/selftests/net/.gitignore b/tools/testing/selftests/net/.gitignore index 2f9d378edec3..b644dbae58b7 100644 --- a/tools/testing/selftests/net/.gitignore +++ b/tools/testing/selftests/net/.gitignore @@ -17,6 +17,7 @@ ipv6_flowlabel ipv6_flowlabel_mgr log.txt msg_zerocopy +ncdevmem nettest psock_fanout psock_snd diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile index 14bd68da7466..d7a66563ffe7 100644 --- a/tools/testing/selftests/net/Makefile +++ b/tools/testing/selftests/net/Makefile @@ -5,6 +5,10 @@ CFLAGS = -Wall -Wl,--no-as-needed -O2 -g CFLAGS += -I../../../../usr/include/ $(KHDR_INCLUDES) # Additional include paths needed by kselftest.h CFLAGS += -I../ +CFLAGS += -I../../../net/ynl/generated/ +CFLAGS += -I../../../net/ynl/lib/ + +LDLIBS += ../../../net/ynl/lib/ynl.a ../../../net/ynl/generated/protos.a TEST_PROGS := run_netsocktests run_afpackettests test_bpf.sh netdevice.sh \ rtnetlink.sh xfrm_policy.sh test_blackhole_dev.sh @@ -92,6 +96,7 @@ TEST_PROGS += test_vxlan_nolocalbypass.sh TEST_PROGS += test_bridge_backup_port.sh TEST_PROGS += fdb_flush.sh TEST_PROGS += fq_band_pktlimit.sh +TEST_GEN_FILES += ncdevmem TEST_FILES := settings diff --git a/tools/testing/selftests/net/ncdevmem.c b/tools/testing/selftests/net/ncdevmem.c new file mode 100644 index 000000000000..7fbeee02b9a2 --- /dev/null +++ b/tools/testing/selftests/net/ncdevmem.c @@ -0,0 +1,489 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#define __EXPORTED_HEADERS__ + +#include +#include +#include +#include +#include +#include +#include +#define __iovec_defined +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "netdev-user.h" +#include + +#define PAGE_SHIFT 12 +#define TEST_PREFIX "ncdevmem" +#define NUM_PAGES 16000 + +#ifndef MSG_SOCK_DEVMEM +#define MSG_SOCK_DEVMEM 0x2000000 +#endif + +/* + * tcpdevmem netcat. Works similarly to netcat but does device memory TCP + * instead of regular TCP. Uses udmabuf to mock a dmabuf provider. + * + * Usage: + * + * On server: + * ncdevmem -s -c -f eth1 -d 3 -n 0000:06:00.0 -l \ + * -p 5201 -v 7 + * + * On client: + * yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ + * tr \\n \\0 | \ + * head -c 5G | \ + * nc 5201 -p 5201 + * + * Note this is compatible with regular netcat. i.e. the sender or receiver can + * be replaced with regular netcat to test the RX or TX path in isolation. + */ + +static char *server_ip = "192.168.1.4"; +static char *client_ip = "192.168.1.2"; +static char *port = "5201"; +static size_t do_validation; +static int queue_num = 15; +static char *ifname = "eth1"; +static unsigned int ifindex = 3; +static char *nic_pci_addr = "0000:06:00.0"; +static unsigned int iterations; +static unsigned int dmabuf_id; + +void print_bytes(void *ptr, size_t size) +{ + unsigned char *p = ptr; + int i; + + for (i = 0; i < size; i++) + printf("%02hhX ", p[i]); + printf("\n"); +} + +void print_nonzero_bytes(void *ptr, size_t size) +{ + unsigned char *p = ptr; + unsigned int i; + + for (i = 0; i < size; i++) + putchar(p[i]); + printf("\n"); +} + +void validate_buffer(void *line, size_t size) +{ + static unsigned char seed = 1; + unsigned char *ptr = line; + int errors = 0; + size_t i; + + for (i = 0; i < size; i++) { + if (ptr[i] != seed) { + fprintf(stderr, + "Failed validation: expected=%u, actual=%u, index=%lu\n", + seed, ptr[i], i); + errors++; + if (errors > 20) + error(1, 0, "validation failed."); + } + seed++; + if (seed == do_validation) + seed = 0; + } + + fprintf(stdout, "Validated buffer\n"); +} + +static void reset_flow_steering(void) +{ + char command[256]; + + memset(command, 0, sizeof(command)); + snprintf(command, sizeof(command), "sudo ethtool -K %s ntuple off", + "eth1"); + system(command); + + memset(command, 0, sizeof(command)); + snprintf(command, sizeof(command), "sudo ethtool -K %s ntuple on", + "eth1"); + system(command); +} + +static void configure_flow_steering(void) +{ + char command[256]; + + memset(command, 0, sizeof(command)); + snprintf(command, sizeof(command), + "sudo ethtool -N %s flow-type tcp4 src-ip %s dst-ip %s src-port %s dst-port %s queue %d", + ifname, client_ip, server_ip, port, port, queue_num); + system(command); +} + +static int bind_rx_queue(unsigned int ifindex, unsigned int dmabuf_fd, + struct netdev_queue_dmabuf *queues, + unsigned int n_queue_index, struct ynl_sock **ys) +{ + struct netdev_bind_rx_req *req = NULL; + struct netdev_bind_rx_rsp *rsp = NULL; + struct ynl_error yerr; + + *ys = ynl_sock_create(&ynl_netdev_family, &yerr); + if (!*ys) { + fprintf(stderr, "YNL: %s\n", yerr.msg); + return -1; + } + + req = netdev_bind_rx_req_alloc(); + netdev_bind_rx_req_set_ifindex(req, ifindex); + netdev_bind_rx_req_set_dmabuf_fd(req, dmabuf_fd); + __netdev_bind_rx_req_set_queues(req, queues, n_queue_index); + + rsp = netdev_bind_rx(*ys, req); + if (!rsp) { + perror("netdev_bind_rx"); + goto err_close; + } + + if (!rsp->_present.dmabuf_id) { + perror("dmabuf_id not present"); + goto err_close; + } + + printf("got dmabuf id=%d\n", rsp->dmabuf_id); + dmabuf_id = rsp->dmabuf_id; + + netdev_bind_rx_req_free(req); + netdev_bind_rx_rsp_free(rsp); + + return 0; + +err_close: + fprintf(stderr, "YNL failed: %s\n", (*ys)->err.msg); + netdev_bind_rx_req_free(req); + ynl_sock_destroy(*ys); + return -1; +} + +static void create_udmabuf(int *devfd, int *memfd, int *buf, size_t dmabuf_size) +{ + struct udmabuf_create create; + int ret; + + *devfd = open("/dev/udmabuf", O_RDWR); + if (*devfd < 0) { + error(70, 0, + "%s: [skip,no-udmabuf: Unable to access DMA buffer device file]\n", + TEST_PREFIX); + } + + *memfd = memfd_create("udmabuf-test", MFD_ALLOW_SEALING); + if (*memfd < 0) + error(70, 0, "%s: [skip,no-memfd]\n", TEST_PREFIX); + + /* Required for udmabuf */ + ret = fcntl(*memfd, F_ADD_SEALS, F_SEAL_SHRINK); + if (ret < 0) + error(73, 0, "%s: [skip,fcntl-add-seals]\n", TEST_PREFIX); + + ret = ftruncate(*memfd, dmabuf_size); + if (ret == -1) + error(74, 0, "%s: [FAIL,memfd-truncate]\n", TEST_PREFIX); + + memset(&create, 0, sizeof(create)); + + create.memfd = *memfd; + create.offset = 0; + create.size = dmabuf_size; + *buf = ioctl(*devfd, UDMABUF_CREATE, &create); + if (*buf < 0) + error(75, 0, "%s: [FAIL, create udmabuf]\n", TEST_PREFIX); +} + +int do_server(void) +{ + char ctrl_data[sizeof(int) * 20000]; + struct netdev_queue_dmabuf *queues; + size_t non_page_aligned_frags = 0; + struct sockaddr_in client_addr; + struct sockaddr_in server_sin; + size_t page_aligned_frags = 0; + int devfd, memfd, buf, ret; + size_t total_received = 0; + socklen_t client_addr_len; + bool is_devmem = false; + char *buf_mem = NULL; + struct ynl_sock *ys; + size_t dmabuf_size; + char iobuf[819200]; + char buffer[256]; + int socket_fd; + int client_fd; + size_t i = 0; + int opt = 1; + + dmabuf_size = getpagesize() * NUM_PAGES; + + create_udmabuf(&devfd, &memfd, &buf, dmabuf_size); + + reset_flow_steering(); + configure_flow_steering(); + + sleep(1); + + queues = malloc(sizeof(*queues) * 1); + + queues[0]._present.type = 1; + queues[0]._present.idx = 1; + queues[0].type = NETDEV_QUEUE_TYPE_RX; + queues[0].idx = queue_num; + if (bind_rx_queue(ifindex, buf, queues, 1, &ys)) + error(1, 0, "Failed to bind\n"); + + buf_mem = mmap(NULL, dmabuf_size, PROT_READ | PROT_WRITE, MAP_SHARED, + buf, 0); + if (buf_mem == MAP_FAILED) + error(1, 0, "mmap()"); + + server_sin.sin_family = AF_INET; + server_sin.sin_port = htons(atoi(port)); + + ret = inet_pton(server_sin.sin_family, server_ip, &server_sin.sin_addr); + if (socket < 0) + error(79, 0, "%s: [FAIL, create socket]\n", TEST_PREFIX); + + socket_fd = socket(server_sin.sin_family, SOCK_STREAM, 0); + if (socket < 0) + error(errno, errno, "%s: [FAIL, create socket]\n", TEST_PREFIX); + + ret = setsockopt(socket_fd, SOL_SOCKET, SO_REUSEPORT, &opt, + sizeof(opt)); + if (ret) + error(errno, errno, "%s: [FAIL, set sock opt]\n", TEST_PREFIX); + + ret = setsockopt(socket_fd, SOL_SOCKET, SO_REUSEADDR, &opt, + sizeof(opt)); + if (ret) + error(errno, errno, "%s: [FAIL, set sock opt]\n", TEST_PREFIX); + + printf("binding to address %s:%d\n", server_ip, + ntohs(server_sin.sin_port)); + + ret = bind(socket_fd, &server_sin, sizeof(server_sin)); + if (ret) + error(errno, errno, "%s: [FAIL, bind]\n", TEST_PREFIX); + + ret = listen(socket_fd, 1); + if (ret) + error(errno, errno, "%s: [FAIL, listen]\n", TEST_PREFIX); + + client_addr_len = sizeof(client_addr); + + inet_ntop(server_sin.sin_family, &server_sin.sin_addr, buffer, + sizeof(buffer)); + printf("Waiting or connection on %s:%d\n", buffer, + ntohs(server_sin.sin_port)); + client_fd = accept(socket_fd, &client_addr, &client_addr_len); + + inet_ntop(client_addr.sin_family, &client_addr.sin_addr, buffer, + sizeof(buffer)); + printf("Got connection from %s:%d\n", buffer, + ntohs(client_addr.sin_port)); + + while (1) { + struct iovec iov = { .iov_base = iobuf, + .iov_len = sizeof(iobuf) }; + struct dmabuf_cmsg *dmabuf_cmsg = NULL; + struct dma_buf_sync sync = { 0 }; + struct cmsghdr *cm = NULL; + struct msghdr msg = { 0 }; + struct dmabuf_token token; + ssize_t ret; + + is_devmem = false; + printf("\n\n"); + + msg.msg_iov = &iov; + msg.msg_iovlen = 1; + msg.msg_control = ctrl_data; + msg.msg_controllen = sizeof(ctrl_data); + ret = recvmsg(client_fd, &msg, MSG_SOCK_DEVMEM); + printf("recvmsg ret=%ld\n", ret); + if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) + continue; + if (ret < 0) { + perror("recvmsg"); + continue; + } + if (ret == 0) { + printf("client exited\n"); + goto cleanup; + } + + i++; + for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level != SOL_SOCKET || + (cm->cmsg_type != SCM_DEVMEM_DMABUF && + cm->cmsg_type != SCM_DEVMEM_LINEAR)) { + fprintf(stdout, "skipping non-devmem cmsg\n"); + continue; + } + + dmabuf_cmsg = (struct dmabuf_cmsg *)CMSG_DATA(cm); + is_devmem = true; + + if (cm->cmsg_type == SCM_DEVMEM_LINEAR) { + /* TODO: process data copied from skb's linear + * buffer. + */ + fprintf(stdout, + "SCM_DEVMEM_LINEAR. dmabuf_cmsg->frag_size=%u\n", + dmabuf_cmsg->frag_size); + + continue; + } + + token.token_start = dmabuf_cmsg->frag_token; + token.token_count = 1; + + total_received += dmabuf_cmsg->frag_size; + printf("received frag_page=%llu, in_page_offset=%llu, frag_offset=%llu, frag_size=%u, token=%u, total_received=%lu, dmabuf_id=%u\n", + dmabuf_cmsg->frag_offset >> PAGE_SHIFT, + dmabuf_cmsg->frag_offset % getpagesize(), + dmabuf_cmsg->frag_offset, dmabuf_cmsg->frag_size, + dmabuf_cmsg->frag_token, total_received, + dmabuf_cmsg->dmabuf_id); + + if (dmabuf_cmsg->dmabuf_id != dmabuf_id) + error(1, 0, + "received on wrong dmabuf_id: flow steering error\n"); + + if (dmabuf_cmsg->frag_size % getpagesize()) + non_page_aligned_frags++; + else + page_aligned_frags++; + + sync.flags = DMA_BUF_SYNC_READ | DMA_BUF_SYNC_START; + ioctl(buf, DMA_BUF_IOCTL_SYNC, &sync); + + if (do_validation) + validate_buffer( + ((unsigned char *)buf_mem) + + dmabuf_cmsg->frag_offset, + dmabuf_cmsg->frag_size); + else + print_nonzero_bytes( + ((unsigned char *)buf_mem) + + dmabuf_cmsg->frag_offset, + dmabuf_cmsg->frag_size); + + sync.flags = DMA_BUF_SYNC_READ | DMA_BUF_SYNC_END; + ioctl(buf, DMA_BUF_IOCTL_SYNC, &sync); + + ret = setsockopt(client_fd, SOL_SOCKET, + SO_DEVMEM_DONTNEED, &token, + sizeof(token)); + if (ret != 1) + error(1, 0, + "SO_DEVMEM_DONTNEED not enough tokens"); + } + if (!is_devmem) + error(1, 0, "flow steering error\n"); + + printf("total_received=%lu\n", total_received); + } + + fprintf(stdout, "%s: ok\n", TEST_PREFIX); + + fprintf(stdout, "page_aligned_frags=%lu, non_page_aligned_frags=%lu\n", + page_aligned_frags, non_page_aligned_frags); + + fprintf(stdout, "page_aligned_frags=%lu, non_page_aligned_frags=%lu\n", + page_aligned_frags, non_page_aligned_frags); + +cleanup: + + munmap(buf_mem, dmabuf_size); + close(client_fd); + close(socket_fd); + close(buf); + close(memfd); + close(devfd); + ynl_sock_destroy(ys); + + return 0; +} + +int main(int argc, char *argv[]) +{ + int is_server = 0, opt; + + while ((opt = getopt(argc, argv, "ls:c:p:v:q:f:n:i:d:")) != -1) { + switch (opt) { + case 'l': + is_server = 1; + break; + case 's': + server_ip = optarg; + break; + case 'c': + client_ip = optarg; + break; + case 'p': + port = optarg; + break; + case 'v': + do_validation = atoll(optarg); + break; + case 'q': + queue_num = atoi(optarg); + break; + case 'f': + ifname = optarg; + break; + case 'd': + ifindex = atoi(optarg); + break; + case 'n': + nic_pci_addr = optarg; + break; + case 'i': + iterations = atoll(optarg); + break; + case '?': + printf("unknown option: %c\n", optopt); + break; + } + } + + for (; optind < argc; optind++) + printf("extra arguments: %s\n", argv[optind]); + + if (is_server) + return do_server(); + + return 0; +}