From patchwork Thu Jul 27 14:43:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1175419vqo; Thu, 27 Jul 2023 08:21:03 -0700 (PDT) X-Google-Smtp-Source: APBJJlHAD5yaRAqem++CiS6Zp2evjC6aNJps2WxTuj3xIkRNNQABxAQTRozZDXuZam3A+hBrPpU4 X-Received: by 2002:a05:6a00:1798:b0:686:a10b:e8b with SMTP id s24-20020a056a00179800b00686a10b0e8bmr6664185pfg.8.1690471263297; Thu, 27 Jul 2023 08:21:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471263; cv=none; d=google.com; s=arc-20160816; b=nne8TAfFDzxyWuEtPwypK8Sum0RKj3Dt4gpr1PH1EKZ80XxYZJ+xfthGqmNI8MUT2X geUwvuHIIHGWguTMON0BOyjyqAwI6fwIB0H2D4gc5aKR/AxpKIDnuryRL5U+sEQypHND 6WzH9SSgT9zh0qx964BysX1uzZVdTe7aG5s4BACml6bYjPYz7p+FEGxqzRVCXJ96akU/ L63/UTdmLPFxKbMlwG7zyNJrgkcaqlnjE4N0F5Di4xu/D9DSlXX//tX6FGE48bKTwsJM RH0Xjl24mmmcERs/aTnN2ymZbQ7AHJhboNO48n2EyCKXFyR9lovc6Cg+W5tloxZnuWCC M8ZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FH0Uln6g7FOd1R95BQB7JD7hvOEwNADm3maRtCllv8k=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=YDoMuyvAXq/nM+pSBpNG/vQ10JPrglb5bXk0IgLrdcSFQx6s9g3xuKt8QvplZV545m 8yYZ5Ll2tFV0EciAi6gRvW70UQOJ+5k2X5i1FJHoTD8FbKvneDo0nR4JUCrpWaTDIHE3 cJ5mf0i74xjH868h3SlSpYILxCFTEA8X58uoMLh3FBV5QKkCdd915EDgv8UZtAUB4t8k PRuzh85NeJnonDD8TVJHJNbwtGYZfV+hg/29a/gwDRLouL2MltpZY55Hotrf83aTld4J Umuqk3m+CIq+8/vPxXYpylZVH1TD4ZXMFb2c50tyloJJ4+wE5BaNrJ8qaUKGxdEa7Wg1 Je1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KBGpY2BY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o20-20020a056a0015d400b00653c0edfd34si1573288pfu.139.2023.07.27.08.20.46; Thu, 27 Jul 2023 08:21:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=KBGpY2BY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233920AbjG0Opo (ORCPT + 99 others); Thu, 27 Jul 2023 10:45:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233902AbjG0Opj (ORCPT ); Thu, 27 Jul 2023 10:45:39 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D634F2; Thu, 27 Jul 2023 07:45:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469136; x=1722005136; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AKUBmlFpIsP8FOTuhCgHejCXQyGo5SgEdM8TgCFAfgs=; b=KBGpY2BYKXyOq1vdylfFmyGEsLG+0SEGAYUJYWM9Ron3rYRwKxapdkrS ejwdSLRQIf5FKXSU3ZJjZUnivCitqmRLotD1XY5deN3qiT/+4w0YVxvK+ CSSiItKN782HyQ2dnoVkLJFF6LzDbZhVFpzOv1lxEfPrvUE+U41z4Pger nZYUsxJVM5whTPxjz76boMFkyxjrDHKeJMH0oRqSIIZkcnUrv5k18Zz+3 hPvpy1/awQhBAZVBctu7jvnZgkOSn2UneRxtwET4LFp+taXNZtdkzd2Fv U+8mTaJJhJiPZb+cM0tpAmdyA89Q90y+TYXZjHfLiovBjy6+jPweKpZrH g==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139673" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139673" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119873" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119873" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:32 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 1/9] page_pool: split types and declarations from page_pool.h Date: Thu, 27 Jul 2023 16:43:28 +0200 Message-ID: <20230727144336.1646454-2-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772587595795913197 X-GMAIL-MSGID: 1772587595795913197 From: Yunsheng Lin Split types and pure function declarations from page_pool.h and add them in page_page/types.h, so that C sources can include page_pool.h and headers should generally only include page_pool/types.h as suggested by jakub. Rename page_pool.h to page_pool/helpers.h to have both in one place. Signed-off-by: Yunsheng Lin Suggested-by: Jakub Kicinski Signed-off-by: Alexander Lobakin --- MAINTAINERS | 3 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 2 +- drivers/net/ethernet/engleder/tsnep_main.c | 1 + drivers/net/ethernet/freescale/fec_main.c | 1 + .../net/ethernet/hisilicon/hns3/hns3_enet.c | 1 + .../net/ethernet/hisilicon/hns3/hns3_enet.h | 2 +- drivers/net/ethernet/marvell/mvneta.c | 2 +- drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 2 +- .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 1 + .../marvell/octeontx2/nic/otx2_common.c | 1 + .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 1 + drivers/net/ethernet/mediatek/mtk_eth_soc.c | 1 + drivers/net/ethernet/mediatek/mtk_eth_soc.h | 2 +- .../ethernet/mellanox/mlx5/core/en/params.c | 1 + .../net/ethernet/mellanox/mlx5/core/en/trap.c | 1 - .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 2 +- .../ethernet/mellanox/mlx5/core/en_stats.c | 2 +- .../ethernet/microchip/lan966x/lan966x_fdma.c | 1 + .../ethernet/microchip/lan966x/lan966x_main.h | 2 +- drivers/net/ethernet/socionext/netsec.c | 2 +- drivers/net/ethernet/stmicro/stmmac/stmmac.h | 2 +- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 1 + drivers/net/ethernet/ti/cpsw.c | 2 +- drivers/net/ethernet/ti/cpsw_new.c | 2 +- drivers/net/ethernet/ti/cpsw_priv.c | 2 +- drivers/net/ethernet/wangxun/libwx/wx_lib.c | 2 +- drivers/net/veth.c | 2 +- drivers/net/wireless/mediatek/mt76/mac80211.c | 1 - drivers/net/wireless/mediatek/mt76/mt76.h | 1 + drivers/net/xen-netfront.c | 2 +- include/linux/skbuff.h | 2 +- include/net/page_pool/helpers.h | 193 +++++++++++++++++ .../net/{page_pool.h => page_pool/types.h} | 197 +----------------- include/trace/events/page_pool.h | 2 +- net/bpf/test_run.c | 2 +- net/core/page_pool.c | 2 +- net/core/skbuff.c | 2 +- net/core/xdp.c | 2 +- 41 files changed, 235 insertions(+), 220 deletions(-) create mode 100644 include/net/page_pool/helpers.h rename include/net/{page_pool.h => page_pool/types.h} (51%) diff --git a/MAINTAINERS b/MAINTAINERS index d0553ad37865..30037d39b82d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16015,8 +16015,7 @@ M: Ilias Apalodimas L: netdev@vger.kernel.org S: Supported F: Documentation/networking/page_pool.rst -F: include/net/page_pool.h -F: include/trace/events/page_pool.h +F: include/net/page_pool/*.h F: net/core/page_pool.c PAGE TABLE CHECK diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index a3bbd13c070f..2e8a1b79bf3f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -54,7 +54,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c index 5b6fbdc4dc40..0fc2c7514aa6 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include "bnxt_hsi.h" #include "bnxt.h" #include "bnxt_xdp.h" diff --git a/drivers/net/ethernet/engleder/tsnep_main.c b/drivers/net/ethernet/engleder/tsnep_main.c index 079f9f6ae21a..f61bd89734c5 100644 --- a/drivers/net/ethernet/engleder/tsnep_main.c +++ b/drivers/net/ethernet/engleder/tsnep_main.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #define TSNEP_RX_OFFSET (max(NET_SKB_PAD, XDP_PACKET_HEADROOM) + NET_IP_ALIGN) diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c index 03ac7690b5c4..27420f3447fc 100644 --- a/drivers/net/ethernet/freescale/fec_main.c +++ b/drivers/net/ethernet/freescale/fec_main.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index 9f6890059666..e5e37a33fd81 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h index 88af34bbee34..acd756b0c7c9 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h @@ -6,7 +6,7 @@ #include #include -#include +#include #include #include "hnae3.h" diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index acf4f6ba73a6..d483b8c00ec0 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -37,7 +37,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h index 11e603686a27..e809f91c08fb 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c index 1fec84b4c068..68fef2027221 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c index 8cdd92dd9762..8336cea16aff 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c @@ -7,6 +7,7 @@ #include #include +#include #include #include diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c index 9551b422622a..bfcc13ad9514 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "otx2_reg.h" #include "otx2_common.h" diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 7490d48000c2..b72e127d7d3a 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -25,6 +25,7 @@ #include #include #include +#include #include "mtk_eth_soc.h" #include "mtk_wed.h" diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h index 28adda0c90c0..5ebe151f0763 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h @@ -18,7 +18,7 @@ #include #include #include -#include +#include #include #include "mtk_ppe.h" diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c index 5ce28ff7685f..e097f336e1c4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c @@ -6,6 +6,7 @@ #include "en/port.h" #include "en_accel/en_accel.h" #include "en_accel/ipsec.h" +#include #include static u8 mlx5e_mpwrq_min_page_shift(struct mlx5_core_dev *mdev) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/trap.c b/drivers/net/ethernet/mellanox/mlx5/core/en/trap.c index 201ac7dd338f..698647cc8c0f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/trap.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/trap.c @@ -1,7 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB /* Copyright (c) 2020 Mellanox Technologies */ -#include #include "en/txrx.h" #include "en/params.h" #include "en/trap.h" diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index 40589cebb773..12f56d0db0af 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -35,6 +35,7 @@ #include "en/xdp.h" #include "en/params.h" #include +#include int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index defb1efccb78..d9b17ab75fa4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -38,7 +38,7 @@ #include #include #include -#include +#include #include #include #include "eswitch.h" diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 41d37159e027..d01b7f501d9a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -36,7 +36,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index 4d77055abd4b..07b84d668fcc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -38,7 +38,7 @@ #include "en/port.h" #ifdef CONFIG_PAGE_POOL_STATS -#include +#include #endif static unsigned int stats_grps_num(struct mlx5e_priv *priv) diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c index bd72fbc2220f..3960534ac2ad 100644 --- a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c @@ -2,6 +2,7 @@ #include #include +#include #include "lan966x_main.h" diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h index 27f272831ea5..b0be99fa8174 100644 --- a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/net/ethernet/socionext/netsec.c b/drivers/net/ethernet/socionext/netsec.c index 2d7347b71c41..0cde22a01e03 100644 --- a/drivers/net/ethernet/socionext/netsec.c +++ b/drivers/net/ethernet/socionext/netsec.c @@ -15,7 +15,7 @@ #include #include -#include +#include #include #define NETSEC_REG_SOFT_RST 0x104 diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h index 4ce5eaaae513..a57f5f362ff5 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h @@ -21,7 +21,7 @@ #include #include #include -#include +#include #include struct stmmac_resources { diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 19e46e8f626a..be676ffefeca 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -39,6 +39,7 @@ #include #include #include +#include #include #include #include "stmmac_ptp.h" diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c index f9cd566d1c9b..ca4d4548f85e 100644 --- a/drivers/net/ethernet/ti/cpsw.c +++ b/drivers/net/ethernet/ti/cpsw.c @@ -31,7 +31,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c index c61e4e44a78f..0e4f526b1753 100644 --- a/drivers/net/ethernet/ti/cpsw_new.c +++ b/drivers/net/ethernet/ti/cpsw_new.c @@ -30,7 +30,7 @@ #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/ti/cpsw_priv.c b/drivers/net/ethernet/ti/cpsw_priv.c index e966dd47e2db..1e81e95bcc4b 100644 --- a/drivers/net/ethernet/ti/cpsw_priv.c +++ b/drivers/net/ethernet/ti/cpsw_priv.c @@ -18,7 +18,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/net/ethernet/wangxun/libwx/wx_lib.c b/drivers/net/ethernet/wangxun/libwx/wx_lib.c index 2c3f08be8c37..e04d4a5eed7b 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_lib.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_lib.c @@ -3,7 +3,7 @@ #include #include -#include +#include #include #include #include diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 614f3e3efab0..953f6d8f8db0 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -26,7 +26,7 @@ #include #include #include -#include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c index 467afef98ba2..c8f7f80746e8 100644 --- a/drivers/net/wireless/mediatek/mt76/mac80211.c +++ b/drivers/net/wireless/mediatek/mt76/mac80211.c @@ -4,7 +4,6 @@ */ #include #include -#include #include "mt76.h" #define CHAN2G(_idx, _freq) { \ diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h index 6b07b8fafec2..81192a07f17f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -15,6 +15,7 @@ #include #include #include +#include #include "util.h" #include "testmode.h" diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index 47d54d8ea59d..ad29f370034e 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -45,7 +45,7 @@ #include #include #include -#include +#include #include #include diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 16a49ba534e4..888e3d7e74c1 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -32,7 +32,7 @@ #include #include #include -#include +#include #if IS_ENABLED(CONFIG_NF_CONNTRACK) #include #endif diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h new file mode 100644 index 000000000000..e2d8d3a8810c --- /dev/null +++ b/include/net/page_pool/helpers.h @@ -0,0 +1,193 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * page_pool/helpers.h + * Author: Jesper Dangaard Brouer + * Copyright (C) 2016 Red Hat, Inc. + */ + +/** + * DOC: page_pool allocator + * + * This page_pool allocator is optimized for the XDP mode that + * uses one-frame-per-page, but have fallbacks that act like the + * regular page allocator APIs. + * + * Basic use involve replacing alloc_pages() calls with the + * page_pool_alloc_pages() call. Drivers should likely use + * page_pool_dev_alloc_pages() replacing dev_alloc_pages(). + * + * API keeps track of in-flight pages, in-order to let API user know + * when it is safe to dealloactor page_pool object. Thus, API users + * must call page_pool_put_page() where appropriate and only attach + * the page to a page_pool-aware objects, like skbs marked for recycling. + * + * API user must only call page_pool_put_page() once on a page, as it + * will either recycle the page, or in case of elevated refcnt, it + * will release the DMA mapping and in-flight state accounting. We + * hope to lift this requirement in the future. + */ +#ifndef _NET_PAGE_POOL_HELPERS_H +#define _NET_PAGE_POOL_HELPERS_H + +#include + +#ifdef CONFIG_PAGE_POOL_STATS +int page_pool_ethtool_stats_get_count(void); +u8 *page_pool_ethtool_stats_get_strings(u8 *data); +u64 *page_pool_ethtool_stats_get(u64 *data, void *stats); + +/* + * Drivers that wish to harvest page pool stats and report them to users + * (perhaps via ethtool, debugfs, or another mechanism) can allocate a + * struct page_pool_stats call page_pool_get_stats to get stats for the specified pool. + */ +bool page_pool_get_stats(struct page_pool *pool, + struct page_pool_stats *stats); +#else +static inline int page_pool_ethtool_stats_get_count(void) +{ + return 0; +} + +static inline u8 *page_pool_ethtool_stats_get_strings(u8 *data) +{ + return data; +} + +static inline u64 *page_pool_ethtool_stats_get(u64 *data, void *stats) +{ + return data; +} +#endif + +static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool) +{ + gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN); + + return page_pool_alloc_pages(pool, gfp); +} + +static inline struct page *page_pool_dev_alloc_frag(struct page_pool *pool, + unsigned int *offset, + unsigned int size) +{ + gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN); + + return page_pool_alloc_frag(pool, offset, size, gfp); +} + +/* get the stored dma direction. A driver might decide to treat this locally and + * avoid the extra cache line from page_pool to determine the direction + */ +static +inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) +{ + return pool->p.dma_dir; +} + +/* pp_frag_count represents the number of writers who can update the page + * either by updating skb->data or via DMA mappings for the device. + * We can't rely on the page refcnt for that as we don't know who might be + * holding page references and we can't reliably destroy or sync DMA mappings + * of the fragments. + * + * When pp_frag_count reaches 0 we can either recycle the page if the page + * refcnt is 1 or return it back to the memory allocator and destroy any + * mappings we have. + */ +static inline void page_pool_fragment_page(struct page *page, long nr) +{ + atomic_long_set(&page->pp_frag_count, nr); +} + +static inline long page_pool_defrag_page(struct page *page, long nr) +{ + long ret; + + /* If nr == pp_frag_count then we have cleared all remaining + * references to the page. No need to actually overwrite it, instead + * we can leave this to be overwritten by the calling function. + * + * The main advantage to doing this is that an atomic_read is + * generally a much cheaper operation than an atomic update, + * especially when dealing with a page that may be partitioned + * into only 2 or 3 pieces. + */ + if (atomic_long_read(&page->pp_frag_count) == nr) + return 0; + + ret = atomic_long_sub_return(nr, &page->pp_frag_count); + WARN_ON(ret < 0); + return ret; +} + +static inline bool page_pool_is_last_frag(struct page_pool *pool, + struct page *page) +{ + /* If fragments aren't enabled or count is 0 we were the last user */ + return !(pool->p.flags & PP_FLAG_PAGE_FRAG) || + (page_pool_defrag_page(page, 1) == 0); +} + +static inline void page_pool_put_page(struct page_pool *pool, + struct page *page, + unsigned int dma_sync_size, + bool allow_direct) +{ + /* When page_pool isn't compiled-in, net/core/xdp.c doesn't + * allow registering MEM_TYPE_PAGE_POOL, but shield linker. + */ +#ifdef CONFIG_PAGE_POOL + if (!page_pool_is_last_frag(pool, page)) + return; + + page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); +#endif +} + +/* Same as above but will try to sync the entire area pool->max_len */ +static inline void page_pool_put_full_page(struct page_pool *pool, + struct page *page, bool allow_direct) +{ + page_pool_put_page(pool, page, -1, allow_direct); +} + +/* Same as above but the caller must guarantee safe context. e.g NAPI */ +static inline void page_pool_recycle_direct(struct page_pool *pool, + struct page *page) +{ + page_pool_put_full_page(pool, page, true); +} + +#define PAGE_POOL_DMA_USE_PP_FRAG_COUNT \ + (sizeof(dma_addr_t) > sizeof(unsigned long)) + +static inline dma_addr_t page_pool_get_dma_addr(struct page *page) +{ + dma_addr_t ret = page->dma_addr; + + if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) + ret |= (dma_addr_t)page->dma_addr_upper << 16 << 16; + + return ret; +} + +static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr) +{ + page->dma_addr = addr; + if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) + page->dma_addr_upper = upper_32_bits(addr); +} + +static inline bool page_pool_put(struct page_pool *pool) +{ + return refcount_dec_and_test(&pool->user_cnt); +} + +static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) +{ + if (unlikely(pool->p.nid != new_nid)) + page_pool_update_nid(pool, new_nid); +} + +#endif /* _NET_PAGE_POOL_HELPERS_H */ diff --git a/include/net/page_pool.h b/include/net/page_pool/types.h similarity index 51% rename from include/net/page_pool.h rename to include/net/page_pool/types.h index f1d5cc1fa13b..4a0270291deb 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool/types.h @@ -1,37 +1,10 @@ -/* SPDX-License-Identifier: GPL-2.0 - * - * page_pool.h - * Author: Jesper Dangaard Brouer - * Copyright (C) 2016 Red Hat, Inc. - */ +/* SPDX-License-Identifier: GPL-2.0 */ -/** - * DOC: page_pool allocator - * - * This page_pool allocator is optimized for the XDP mode that - * uses one-frame-per-page, but have fallbacks that act like the - * regular page allocator APIs. - * - * Basic use involve replacing alloc_pages() calls with the - * page_pool_alloc_pages() call. Drivers should likely use - * page_pool_dev_alloc_pages() replacing dev_alloc_pages(). - * - * API keeps track of in-flight pages, in-order to let API user know - * when it is safe to dealloactor page_pool object. Thus, API users - * must call page_pool_put_page() where appropriate and only attach - * the page to a page_pool-aware objects, like skbs marked for recycling. - * - * API user must only call page_pool_put_page() once on a page, as it - * will either recycle the page, or in case of elevated refcnt, it - * will release the DMA mapping and in-flight state accounting. We - * hope to lift this requirement in the future. - */ -#ifndef _NET_PAGE_POOL_H -#define _NET_PAGE_POOL_H +#ifndef _NET_PAGE_POOL_TYPES_H +#define _NET_PAGE_POOL_TYPES_H -#include /* Needed by ptr_ring */ -#include #include +#include #define PP_FLAG_DMA_MAP BIT(0) /* Should page_pool do the DMA * map/unmap @@ -116,35 +89,6 @@ struct page_pool_stats { struct page_pool_alloc_stats alloc_stats; struct page_pool_recycle_stats recycle_stats; }; - -int page_pool_ethtool_stats_get_count(void); -u8 *page_pool_ethtool_stats_get_strings(u8 *data); -u64 *page_pool_ethtool_stats_get(u64 *data, void *stats); - -/* - * Drivers that wish to harvest page pool stats and report them to users - * (perhaps via ethtool, debugfs, or another mechanism) can allocate a - * struct page_pool_stats call page_pool_get_stats to get stats for the specified pool. - */ -bool page_pool_get_stats(struct page_pool *pool, - struct page_pool_stats *stats); -#else - -static inline int page_pool_ethtool_stats_get_count(void) -{ - return 0; -} - -static inline u8 *page_pool_ethtool_stats_get_strings(u8 *data) -{ - return data; -} - -static inline u64 *page_pool_ethtool_stats_get(u64 *data, void *stats) -{ - return data; -} - #endif struct page_pool { @@ -188,7 +132,7 @@ struct page_pool { * association with allocation resource. * * Use ptr_ring, as it separates consumer and producer - * effeciently, it a way that doesn't bounce cache-lines. + * efficiently, it a way that doesn't bounce cache-lines. * * TODO: Implement bulk return pages into this structure. */ @@ -210,35 +154,8 @@ struct page_pool { }; struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); - -static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool) -{ - gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN); - - return page_pool_alloc_pages(pool, gfp); -} - struct page *page_pool_alloc_frag(struct page_pool *pool, unsigned int *offset, unsigned int size, gfp_t gfp); - -static inline struct page *page_pool_dev_alloc_frag(struct page_pool *pool, - unsigned int *offset, - unsigned int size) -{ - gfp_t gfp = (GFP_ATOMIC | __GFP_NOWARN); - - return page_pool_alloc_frag(pool, offset, size, gfp); -} - -/* get the stored dma direction. A driver might decide to treat this locally and - * avoid the extra cache line from page_pool to determine the direction - */ -static -inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) -{ - return pool->p.dma_dir; -} - bool page_pool_return_skb_page(struct page *page, bool napi_safe); struct page_pool *page_pool_create(const struct page_pool_params *params); @@ -277,100 +194,6 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, unsigned int dma_sync_size, bool allow_direct); -/* pp_frag_count represents the number of writers who can update the page - * either by updating skb->data or via DMA mappings for the device. - * We can't rely on the page refcnt for that as we don't know who might be - * holding page references and we can't reliably destroy or sync DMA mappings - * of the fragments. - * - * When pp_frag_count reaches 0 we can either recycle the page if the page - * refcnt is 1 or return it back to the memory allocator and destroy any - * mappings we have. - */ -static inline void page_pool_fragment_page(struct page *page, long nr) -{ - atomic_long_set(&page->pp_frag_count, nr); -} - -static inline long page_pool_defrag_page(struct page *page, long nr) -{ - long ret; - - /* If nr == pp_frag_count then we have cleared all remaining - * references to the page. No need to actually overwrite it, instead - * we can leave this to be overwritten by the calling function. - * - * The main advantage to doing this is that an atomic_read is - * generally a much cheaper operation than an atomic update, - * especially when dealing with a page that may be partitioned - * into only 2 or 3 pieces. - */ - if (atomic_long_read(&page->pp_frag_count) == nr) - return 0; - - ret = atomic_long_sub_return(nr, &page->pp_frag_count); - WARN_ON(ret < 0); - return ret; -} - -static inline bool page_pool_is_last_frag(struct page_pool *pool, - struct page *page) -{ - /* If fragments aren't enabled or count is 0 we were the last user */ - return !(pool->p.flags & PP_FLAG_PAGE_FRAG) || - (page_pool_defrag_page(page, 1) == 0); -} - -static inline void page_pool_put_page(struct page_pool *pool, - struct page *page, - unsigned int dma_sync_size, - bool allow_direct) -{ - /* When page_pool isn't compiled-in, net/core/xdp.c doesn't - * allow registering MEM_TYPE_PAGE_POOL, but shield linker. - */ -#ifdef CONFIG_PAGE_POOL - if (!page_pool_is_last_frag(pool, page)) - return; - - page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); -#endif -} - -/* Same as above but will try to sync the entire area pool->max_len */ -static inline void page_pool_put_full_page(struct page_pool *pool, - struct page *page, bool allow_direct) -{ - page_pool_put_page(pool, page, -1, allow_direct); -} - -/* Same as above but the caller must guarantee safe context. e.g NAPI */ -static inline void page_pool_recycle_direct(struct page_pool *pool, - struct page *page) -{ - page_pool_put_full_page(pool, page, true); -} - -#define PAGE_POOL_DMA_USE_PP_FRAG_COUNT \ - (sizeof(dma_addr_t) > sizeof(unsigned long)) - -static inline dma_addr_t page_pool_get_dma_addr(struct page *page) -{ - dma_addr_t ret = page->dma_addr; - - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) - ret |= (dma_addr_t)page->dma_addr_upper << 16 << 16; - - return ret; -} - -static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr) -{ - page->dma_addr = addr; - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT) - page->dma_addr_upper = upper_32_bits(addr); -} - static inline bool is_page_pool_compiled_in(void) { #ifdef CONFIG_PAGE_POOL @@ -380,17 +203,7 @@ static inline bool is_page_pool_compiled_in(void) #endif } -static inline bool page_pool_put(struct page_pool *pool) -{ - return refcount_dec_and_test(&pool->user_cnt); -} - /* Caller must provide appropriate safe context, e.g. NAPI. */ void page_pool_update_nid(struct page_pool *pool, int new_nid); -static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) -{ - if (unlikely(pool->p.nid != new_nid)) - page_pool_update_nid(pool, new_nid); -} #endif /* _NET_PAGE_POOL_H */ diff --git a/include/trace/events/page_pool.h b/include/trace/events/page_pool.h index ca534501158b..6834356b2d2a 100644 --- a/include/trace/events/page_pool.h +++ b/include/trace/events/page_pool.h @@ -9,7 +9,7 @@ #include #include -#include +#include TRACE_EVENT(page_pool_release, diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 7d47f53f20c1..f892698c8829 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 7ca456bfab71..2a75f61264c5 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -10,7 +10,7 @@ #include #include -#include +#include #include #include diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a298992060e6..ac8f421f8ab3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -73,7 +73,7 @@ #include #include #include -#include +#include #include #include diff --git a/net/core/xdp.c b/net/core/xdp.c index 8362130bf085..a70670fe9a2d 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include /* struct xdp_mem_allocator */ From patchwork Thu Jul 27 14:43:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1176267vqo; Thu, 27 Jul 2023 08:22:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlHNxpjFZ2llL5mtDlhQjoAmySFY3K7JvnD5IV2mMRbRmrDImbyKCFgCZdtpOQP4V027Xhig X-Received: by 2002:a17:902:da87:b0:1b8:ae12:5628 with SMTP id j7-20020a170902da8700b001b8ae125628mr4152168plx.2.1690471354163; Thu, 27 Jul 2023 08:22:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471354; cv=none; d=google.com; s=arc-20160816; b=WXwV+guVt9DpdVg7PhWxtGhY1PfZfPAmlXSOkTqw8xMF+Fsz/RCQejisRAv4T6daC/ 313DHV/Jj+OwNOzBux9Asa/1g+NJvYDQZewtJu2GCGoVCAoZFvvTzkhAMpQlj4dy4CF8 jJuMvFEOHnJILvbk92nFdUO0I3yO6tHjnEmcdIJ1ls9z/YfErfxPvFMd+s8WNome7UBR dK3xxsRyaghj4PKGxPON6VULcVGg1CGir4O2tYPO4CvYh99FyONSsXsOQcwRl3zNNWqB zKUYUfaW/BsAT+foZctwzAppzgKRsI2/5IjKSjqiuGtXiUAomnlBPynn3wsWjpi3MbtS wwTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9GpKE204rmLuYRqmsRAHbKWYaB8LYa5OAq+4KRpIrMo=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=uVGZPgBWqpvZiz/4NrtsP8cGbEouOIu77N8piLd1PP3cfb8dV1jfk3tyOLFplXJylz vJWPkec9GCDu5X7k369KxKqtSVXD5N87nOWFQdTWg49ILh0PKGnUP64YPcOXC8h8XPWh AZ9+2cF+VIWwVJhoHQRJhQge0S2pOXqp7tQ0exGMkItpoTrymXTWA0gP4bhp6rCP1XbM AXyKMOkhI/BUAQuLPgtiB7EMGfDb8BKkJsFxNokYbYztBhPUgH/BA58lMPa4DJxxV6DK T7JnPedi6mCbV74KCnWUEhf/SmxelqU+rqxrL8Im1eNwoSkkiUj6exGKYiD+iyGy0LUa js1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=B2PYvzOu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e4-20020a170902e0c400b001a51368b098si1390136pla.209.2023.07.27.08.22.20; Thu, 27 Jul 2023 08:22:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=B2PYvzOu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233945AbjG0Opq (ORCPT + 99 others); Thu, 27 Jul 2023 10:45:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233906AbjG0Opk (ORCPT ); Thu, 27 Jul 2023 10:45:40 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C765930D8; Thu, 27 Jul 2023 07:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469139; x=1722005139; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bfJssYGznC4Y168BbqD0WxjJ4ba14FCskmkH0A8esDA=; b=B2PYvzOuA3qphg0obtMDa9dKkepBqSdweqHo+OsoJRKostS+c8LXA0NQ 43aY0rFDaFdIB1sDkVKkViS2cT7wUO8lHLjlq1Ws8XOP4w9Ep3Daec9VB ho5zWKGGCg4zxDp81VTB34JfEVuZOdJnE9SViLEKbncDq2TOHJf6Rxe+q w7/T8cC0ink5MWa210F7XOfMZOhuml8L9VoAlfH81r8m9M8tS0NVBSGKD TiTQx4dUHD+pYblkuQW6gkF/pnrWt0xrYdbYLWG6V68q/LGEisOmUqGGt R8g5wmLDvp868rwEw2rFZ/Xvbk388o5l92sN9v2miRJvs3u86Ht83okW3 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139693" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139693" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119884" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119884" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:35 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 2/9] net: skbuff: don't include to Date: Thu, 27 Jul 2023 16:43:29 +0200 Message-ID: <20230727144336.1646454-3-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772587690359286366 X-GMAIL-MSGID: 1772587690359286366 Currently, touching triggers a rebuild of more than half of the kernel. That's because it's included in . And each new include to page_pool/types.h adds more [useless] data for the toolchain to process per each source file from that pile. In commit 6a5bcd84e886 ("page_pool: Allow drivers to hint on SKB recycling"), Matteo included it to be able to call a couple of functions defined there. Then, in commit 57f05bc2ab24 ("page_pool: keep pp info as long as page pool owns the page") one of the calls was removed, so only one was left. It's the call to page_pool_return_skb_page() in napi_frag_unref(). The function is external and doesn't have any dependencies. Having very niche page_pool_types.h included only for that looks like an overkill. As %PP_SIGNATURE is not local to page_pool.c (was only in the early submissions), nothing holds this function there. Teleport page_pool_return_skb_page() to skbuff.c, just next to the main consumer, skb_pp_recycle(). It's used also in napi_frag_unref() -> {__,}skb_frag_unref(), so no `static` unfortunately. Maybe next time. Now, touching page_pool_types.h only triggers rebuilding of the drivers using it and a couple of core networking files. Suggested-by: Jakub Kicinski # make skbuff.h less heavy Suggested-by: Alexander Duyck # move to skbuff.c Signed-off-by: Alexander Lobakin --- include/linux/skbuff.h | 3 ++- include/net/page_pool/types.h | 2 -- net/core/page_pool.c | 39 --------------------------------- net/core/skbuff.c | 41 ++++++++++++++++++++++++++++++++++- 4 files changed, 42 insertions(+), 43 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 888e3d7e74c1..7effd94efd6c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -32,7 +32,6 @@ #include #include #include -#include #if IS_ENABLED(CONFIG_NF_CONNTRACK) #include #endif @@ -3421,6 +3420,8 @@ static inline void skb_frag_ref(struct sk_buff *skb, int f) __skb_frag_ref(&skb_shinfo(skb)->frags[f]); } +bool page_pool_return_skb_page(struct page *page, bool napi_safe); + static inline void napi_frag_unref(skb_frag_t *frag, bool recycle, bool napi_safe) { diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 4a0270291deb..c7aef6c75935 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -156,8 +156,6 @@ struct page_pool { struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); struct page *page_pool_alloc_frag(struct page_pool *pool, unsigned int *offset, unsigned int size, gfp_t gfp); -bool page_pool_return_skb_page(struct page *page, bool napi_safe); - struct page_pool *page_pool_create(const struct page_pool_params *params); struct xdp_mem_info; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 2a75f61264c5..7a23ca6b1124 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -906,42 +906,3 @@ void page_pool_update_nid(struct page_pool *pool, int new_nid) } } EXPORT_SYMBOL(page_pool_update_nid); - -bool page_pool_return_skb_page(struct page *page, bool napi_safe) -{ - struct napi_struct *napi; - struct page_pool *pp; - bool allow_direct; - - page = compound_head(page); - - /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation - * in order to preserve any existing bits, such as bit 0 for the - * head page of compound page and bit 1 for pfmemalloc page, so - * mask those bits for freeing side when doing below checking, - * and page_is_pfmemalloc() is checked in __page_pool_put_page() - * to avoid recycling the pfmemalloc page. - */ - if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) - return false; - - pp = page->pp; - - /* Allow direct recycle if we have reasons to believe that we are - * in the same context as the consumer would run, so there's - * no possible race. - */ - napi = READ_ONCE(pp->p.napi); - allow_direct = napi_safe && napi && - READ_ONCE(napi->list_owner) == smp_processor_id(); - - /* Driver set this to memory recycling info. Reset it on recycle. - * This will *not* work for NIC using a split-page memory model. - * The page will be returned to the pool here regardless of the - * 'flipped' fragment being in use or not. - */ - page_pool_put_full_page(pp, page, allow_direct); - - return true; -} -EXPORT_SYMBOL(page_pool_return_skb_page); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index ac8f421f8ab3..3084ef59400b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -73,7 +73,7 @@ #include #include #include -#include +#include #include #include @@ -879,6 +879,45 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } +bool page_pool_return_skb_page(struct page *page, bool napi_safe) +{ + struct napi_struct *napi; + struct page_pool *pp; + bool allow_direct; + + page = compound_head(page); + + /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation + * in order to preserve any existing bits, such as bit 0 for the + * head page of compound page and bit 1 for pfmemalloc page, so + * mask those bits for freeing side when doing below checking, + * and page_is_pfmemalloc() is checked in __page_pool_put_page() + * to avoid recycling the pfmemalloc page. + */ + if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) + return false; + + pp = page->pp; + + /* Allow direct recycle if we have reasons to believe that we are + * in the same context as the consumer would run, so there's + * no possible race. + */ + napi = READ_ONCE(pp->p.napi); + allow_direct = napi_safe && napi && + READ_ONCE(napi->list_owner) == smp_processor_id(); + + /* Driver set this to memory recycling info. Reset it on recycle. + * This will *not* work for NIC using a split-page memory model. + * The page will be returned to the pool here regardless of the + * 'flipped' fragment being in use or not. + */ + page_pool_put_full_page(pp, page, allow_direct); + + return true; +} +EXPORT_SYMBOL(page_pool_return_skb_page); + static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) { if (!IS_ENABLED(CONFIG_PAGE_POOL) || !skb->pp_recycle) From patchwork Thu Jul 27 14:43:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127044 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1207775vqo; Thu, 27 Jul 2023 09:11:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlH/nscdi9xzx0zIjZEnh2CTPtFQFxd9IseYheZ2is1gLcTfgAKeJbrQ4X53E8J2d7d0kqC3 X-Received: by 2002:a17:90a:fab:b0:268:3299:ef22 with SMTP id 40-20020a17090a0fab00b002683299ef22mr4160670pjz.24.1690474293666; Thu, 27 Jul 2023 09:11:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690474293; cv=none; d=google.com; s=arc-20160816; b=U/VvwTsfOCg2yEZ/NhNC/+causzRM9hh1stbZq7iNAb2Uc51yWFNzVA4Fv1Jthh24+ m6Zz7qF3nyptqnPOMkoYmUiSy0hi06JWvxuZTw/Mv1mu/PIeSTNTTYen/cD5Vxd5s0a7 6DQzfC6EvwT/qzAQre96T+3BouItyQ2/aFUMOTaQxmSuAB7X4a/hMC9cVIJF6Z9wXIyv qN3uDI36QjlUwb2xhidu6uZYrtH4KPj4cGH5j0syEe8os3UEU3xHMlmmmyo070KEpMS8 lABsx5iZk1kfK1LkQrhAebmItElFJaBflKhlOh8jvoxwyCSLTKj0nLkdw8DS3DZwhU+I jX5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=a6uz0/98z0hvHTrzfEb2t1bDNi0CoSOtc/ifICcaiJw=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=rUOBgoOjXyNnt+J3pzPc/6DvATWPn0Iep4htGstxH6FrNAcfV01KK+tNihVqixM1Fa KWo+BbHYzv4cFZ1HmLsyKLIl4LTYpmtU7sWAcpVrYL/bNkhhJaaKZALPfXinDCPRnCwg BYW6T9mmIYogkt4rkk6aSjm64pkOLtvU77gTRiihGKTvEM3ByhHRCeS4Hed9NM358tQS r487gOQZNfDyKSMpdDTR3q8dVCCXE1ZmJCqoC6Z9tCMxCbpSXNoM6VTuPvouao+WxuoN lZocSL8TlBmAIAhztlzn38GFVrUBLmynKyKR10oUVaE84L0z+2n5T1juP66HpulU4wbT 6qCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WlpcAYbY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fz4-20020a17090b024400b00263638bb354si3070233pjb.93.2023.07.27.09.11.19; Thu, 27 Jul 2023 09:11:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WlpcAYbY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233682AbjG0Opw (ORCPT + 99 others); Thu, 27 Jul 2023 10:45:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233918AbjG0Opn (ORCPT ); Thu, 27 Jul 2023 10:45:43 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A40530D2; Thu, 27 Jul 2023 07:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469142; x=1722005142; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=31u+MUzH6+dKE9DPXFl5YBb0OxGcml0B6f7mnB7yaio=; b=WlpcAYbYb2itepcebcCzJdE+7Xr1o+J7p5mmk+RsDk2OoGdVqIdgWKP7 Gu0EhIDNa3AVXk3VlG8Ibl+0ArAiR25cMz52Oc+bsDBxrtjVTAf1DExYp YginGj2IVwqLAKKldByUxSy2FFLDFoEN9dKzrdrAZJcZv/Q4rGiw8XYYq 8zlzextyfBj97FKeTeCQu6epq0M85d0Q0jycG2kCvCCc3ewRTAauw3ZP9 G0P7PUHe/wI5G0RhX5xHFIuGCk+xenkmeECSEZgBpILEScsOm3K2e0vUO aWQJMzOyUbjnFRqaE6VMdmKvauyGxuQI7quEyvQ8fYxDAWFSQoE5D79Ne A==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139720" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139720" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119899" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119899" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:38 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 3/9] page_pool: place frag_* fields in one cacheline Date: Thu, 27 Jul 2023 16:43:30 +0200 Message-ID: <20230727144336.1646454-4-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772590772928038081 X-GMAIL-MSGID: 1772590772928038081 On x86_64, frag_* fields of struct page_pool are scattered across two cachelines despite the summary size of 24 bytes. All three fields are used in pretty much the same places, but the last field, ::frag_users, is pushed out to the next CL, provoking unwanted false-sharing on hotpath (frags allocation code). There are some holes and cold members to move around. Move frag_* one block up, placing them right after &page_pool_params perfectly at the beginning of CL2. This doesn't do any meaningful to the second block, as those are some destroy-path cold structures, and doesn't do anything to ::alloc_stats, which still starts at 200-byte offset, 8 bytes after CL3 (still fitting into 1 cacheline). On my setup, this yields 1-2% of Mpps when using PP frags actively. When it comes to 32-bit architectures with 32-byte CL: &page_pool_params plus ::pad is 44 bytes, the block taken care of is 16 bytes within one CL, so there should be at least no regressions from the actual change. ::pages_state_hold_cnt is not related directly to that triple, but is paired currently with ::frags_offset and decoupling them would mean either two 4-byte holes or more invasive layout changes. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index c7aef6c75935..664a787948e1 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -94,16 +94,16 @@ struct page_pool_stats { struct page_pool { struct page_pool_params p; + long frag_users; + struct page *frag_page; + unsigned int frag_offset; + u32 pages_state_hold_cnt; + struct delayed_work release_dw; void (*disconnect)(void *); unsigned long defer_start; unsigned long defer_warn; - u32 pages_state_hold_cnt; - unsigned int frag_offset; - struct page *frag_page; - long frag_users; - #ifdef CONFIG_PAGE_POOL_STATS /* these stats are incremented while in softirq context */ struct page_pool_alloc_stats alloc_stats; From patchwork Thu Jul 27 14:43:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1175864vqo; Thu, 27 Jul 2023 08:21:50 -0700 (PDT) X-Google-Smtp-Source: APBJJlGJUYU4AxHsrADFUcUFOw0nb+RCGWVoKLXQMNziGw3BBYDAuoY5zEPP51DcZGBzupZZcOZc X-Received: by 2002:a05:6a00:24d2:b0:686:254c:9d47 with SMTP id d18-20020a056a0024d200b00686254c9d47mr5987444pfv.30.1690471309950; Thu, 27 Jul 2023 08:21:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471309; cv=none; d=google.com; s=arc-20160816; b=Tq8Vuuh2zNRKXCTBdy/AbvpGOhITNxoAQksGPxODDXP+9uaGLyqCTUy4XQgnvs2r61 8DB/qoZN9bnpurzxwp3swprBp3xapP6QChoqyHs8njwfKeW1gqGET1XL7zyPIxhZ8Awe //jZpYiJsCYv9Qdc2UdHZ5n2aSY2GUizyIrpfuc5YzgtDOTuQsaUSIalq9eQklyFMPHw YoH2PrpA5w+zldvNXs4FYTjcJ7yRoN8VS2msGrQ+4nNrl9sDoCi2H6I1u57FhLQbniAf UQbZo0xJ87U0hHQRJjmj+mdwH0N4fLL5MUBKD5JBKFlA8MiybylrEfJXh7nKcdszghOg 48Sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RVogmhotSCFWbMFUrm+0WjikbESirTSIAwFxJKaC7do=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=AaVi6Q4Nhhj/BJj0eqJ0bqcVTJZX5SCev/ermD1e05gFvVKYPmstGB2dsBhBbrwEQi mc0vAqTMBXEP91W1hbzYkDRXJvVBZcaVNFOzKsevdUP/Frmng3AAuCiDZmp2ZyWut/Ul IVflWX60Y3BaWdv5zCMw+QYY/bbYiteVacRq/6IuNrUEs5Ds09GGZykMjWroNdoBzLyC Zn6HBi+vePU/UIJ3roI+JYMMVRbp5G93CazsB3BXzMS0CjVmZjGGOMS0bXdb804H4EdU WzmGb8zKTf0Cgw1qqbnAscoeHvzrGba5pk3v90zoYPLLm7n2R7AP2lLagP0XIG5LRkeK qPsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WGsoPrkC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a126-20020a636684000000b00563a0bacbb4si1366074pgc.694.2023.07.27.08.21.35; Thu, 27 Jul 2023 08:21:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=WGsoPrkC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234004AbjG0OqJ (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233995AbjG0Op5 (ORCPT ); Thu, 27 Jul 2023 10:45:57 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ACB130E3; Thu, 27 Jul 2023 07:45:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469146; x=1722005146; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2UbKf7v2UR/BH4b/sLY6ZcXfEMndoDL0gkvxE3Ft3Vg=; b=WGsoPrkCUaCL2T8DvRzS42o1JA+UajI5CE95h0D8H09MOuhuLF1tb6c+ 1tnMNJSD3JXe9eTIN3pmjVFq+DJzisp+ZuzKvSSkrqWFMrxrefNA7TAD5 YLmLeQGEkMGBtwqn3IIpyW4R7sB2F6eWL30mm1JjoJ76iMigxgkokSUOJ xx7RT4ZlDKW4IwFsJsgjyRUtyiLwUrcpag7Vj9J3QMj6jDfeXMXTTyAB2 yW96b1Wpv/A1ffysmOcp6xcK4mLVh5ypqeSo5rQ/dlmW5REc1gmLktyid K3nxbP8ZiOscdVQ5rclPtGiRUQmVpvSPEuREYhbCZojgtgvc+NFt4vxFI g==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139738" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139738" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119912" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119912" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:42 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 4/9] page_pool: shrink &page_pool_params a tiny bit Date: Thu, 27 Jul 2023 16:43:31 +0200 Message-ID: <20230727144336.1646454-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772587644364856839 X-GMAIL-MSGID: 1772587644364856839 For now, this structure takes a whole 64-byte cacheline on x86_64. But in fact, it has a 4-byte hole before ::init_callback() (yet not sufficient to change its sizeof()). ::dma_dir is whole 4 bytes, although its values can only be 0 and 2. Merge it with ::flags and, so that its slot gets freed and reduces the structure's size to 56 bytes. This adds instruction when reading that field, but the upcoming change will make those reads happen way less often. Pad the freed slot explicitly in &page_pool to not alter cacheline layout while it's not used. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 664a787948e1..c86f65e57614 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -44,13 +44,13 @@ struct pp_alloc_cache { }; struct page_pool_params { - unsigned int flags; + unsigned int flags:30; + enum dma_data_direction dma_dir:2; /* DMA mapping direction */ unsigned int order; unsigned int pool_size; int nid; /* Numa node id to allocate from pages from */ struct device *dev; /* device, for DMA pre-mapping purposes */ struct napi_struct *napi; /* Sole consumer of pages, otherwise NULL */ - enum dma_data_direction dma_dir; /* DMA mapping direction */ unsigned int max_len; /* max DMA sync memory size */ unsigned int offset; /* DMA addr offset */ void (*init_callback)(struct page *page, void *arg); @@ -93,6 +93,7 @@ struct page_pool_stats { struct page_pool { struct page_pool_params p; + long pad; long frag_users; struct page *frag_page; From patchwork Thu Jul 27 14:43:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127006 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1173849vqo; Thu, 27 Jul 2023 08:18:29 -0700 (PDT) X-Google-Smtp-Source: APBJJlF3QS6UlU83Fxt8G7OqT1HpDCP1krbt11ZmfVjj7D/kpNTcUegTX/z7/zpZs9BuryEB+zM7 X-Received: by 2002:a17:90a:f30a:b0:268:326:1935 with SMTP id ca10-20020a17090af30a00b0026803261935mr4184038pjb.0.1690471109295; Thu, 27 Jul 2023 08:18:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471109; cv=none; d=google.com; s=arc-20160816; b=T9xboK5rozxXWL3SDcYpzu/6FISCECIBT3C7876mOuIlOaSS/JeutKrEVmFyHN9kTj FOIJHdRfQ/12PFodNQ2r6qBWTULez4JcoNU4BBa61qEHytWUCr39uWuYUsKVvbOayIi4 9PIdwj6VBjyjnPJJWF29OUSZT1awmPx6VqoTWV9ipSLFvIApxI9MeYCGuDWjhEg+IzGz ncn5CrZXK9+yfcWcoA0prpx8xoil/b1IUTPZ3L1pZA1nYMPgDP5FS/1zElZatf7t3KYt tL6URkfy/wzAIS5o7BIsGcAk13EWZfgKaKrb6s+bhAIRntEtLgo7tz4v+w4Gw4Z2jujx AXcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9N5PewloIGMiMX3BBme4t/y1yhLPIhM2TLc4lr7IYsQ=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=Vz/+/zF1qE1xh3UzSXX3Je8KfUoMIXQxaafu8Sru2n7bN5BmgkBUq9CgQ5n18HtePa 3qbFTdbnBRdfcGpdDnDrnCXj5+wRC9ZuV/NkYxIXGTZcUcZGxyQBjGg5BtpWjDmccpRz +aiTIuirhtDg0+C7zxZcNihnHqLA6LSRitoHE0D/ulNksRo9tlMA0UcmyvEyAlk9pogz FFNjvOGJbdGAklc5fcPmRlEaMGRK6ZEhFmOYLzKQo7MwUNfFRptANSA/y2JYsb7w75tk 2xo5DGJO6KtfhyM4r0dEr1sXyc5/RscrT2xop5FSfvr0XL6c/SNpJ/E7RTug++GZ9Uqh XGig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h5MuQr4W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q7-20020a17090aa00700b00263035570e1si1481321pjp.89.2023.07.27.08.18.14; Thu, 27 Jul 2023 08:18:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h5MuQr4W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234100AbjG0Oq1 (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234029AbjG0OqF (ORCPT ); Thu, 27 Jul 2023 10:46:05 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD3E830FF; Thu, 27 Jul 2023 07:45:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469149; x=1722005149; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jgDLGoKjKbbAnNkiXtnAT0R4SkigoAbWXqx73iWEsDQ=; b=h5MuQr4WD/ejuQyxPzoxM1PlNZVTOzwtzbwRqcraVSiY77ivHuGCvS6V NBJN4H6JmZw7rvT9rMtESZ8qVK+jh/xX56QvOWqm5eCelt4NcuDwGU4y8 BADyHVGJjlLmya/CPKCJN0RAPI5ltLM5ZfDYPVCiuZ8Ir4YWwh+WqLUbO HeMq0RJOwyOfYM/keShyTQbuLRN3hdAPUIUQytfLyxpxpt4cos6XnNkYL lY4zj833WAcWd6sfgEuD1nHLMjQkZH4VtY591CqdWIG4a6nt7nT/OGzhZ iFnRHejT8CIsAPu4UobT+rfCuQvyIlR7aM7JFhrW151ow68TruGu8y6du Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139769" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139769" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119920" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119920" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:45 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 5/9] page_pool: don't use driver-set flags field directly Date: Thu, 27 Jul 2023 16:43:32 +0200 Message-ID: <20230727144336.1646454-6-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772587434114400974 X-GMAIL-MSGID: 1772587434114400974 page_pool::p is driver-defined params, copied directly from the structure passed via page_pool_create(). The structure isn't meant to be modified by the Page Pool core code and this even might look confusing[0][1]. In order to be able to alter some flags, let's define our own, internal fields. Use the slot freed earlier to stay within the same cacheline as before (or almost if it's shorter than 64 bytes). The flags indicating whether to perform DMA mapping and use frags can be bool; as for DMA sync, define it as an enum to be able to extend it later on. They are defined as bits in the driver-set params, leave them so here as well, to not waste byte-per-bit or so. Now there are 29 free bits left in those 4 bytes + 4 free bytes more before the cacheline boundary. We could've defined only new flags here or only the ones we may need to alter, but checking some flags in one place while others in another doesn't sound convenient or intuitive. Suggested-by: Jakub Kicinski Link[0]: https://lore.kernel.org/netdev/20230703133207.4f0c54ce@kernel.org Suggested-by: Alexander Duyck Link[1]: https://lore.kernel.org/netdev/CAKgT0UfZCGnWgOH96E4GV3ZP6LLbROHM7SHE8NKwq+exX+Gk_Q@mail.gmail.com Signed-off-by: Alexander Lobakin Signed-off-by: Jakub Kicinski --- include/net/page_pool/helpers.h | 2 +- include/net/page_pool/types.h | 8 +++++++- net/core/page_pool.c | 33 +++++++++++++++++---------------- 3 files changed, 25 insertions(+), 18 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index e2d8d3a8810c..a09ba80b889e 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -125,7 +125,7 @@ static inline bool page_pool_is_last_frag(struct page_pool *pool, struct page *page) { /* If fragments aren't enabled or count is 0 we were the last user */ - return !(pool->p.flags & PP_FLAG_PAGE_FRAG) || + return !pool->page_frag || (page_pool_defrag_page(page, 1) == 0); } diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index c86f65e57614..dd26f4b2b66c 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -93,7 +93,13 @@ struct page_pool_stats { struct page_pool { struct page_pool_params p; - long pad; + + bool dma_map:1; /* Perform DMA mapping */ + enum { + PP_DMA_SYNC_ACT_DISABLED = 0, /* Driver didn't ask to sync */ + PP_DMA_SYNC_ACT_DO, /* Perform DMA sync ops */ + } dma_sync_act:1; + bool page_frag:1; /* Allow page fragments */ long frag_users; struct page *frag_page; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 7a23ca6b1124..6a8f105e2df5 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -183,6 +183,8 @@ static int page_pool_init(struct page_pool *pool, if ((pool->p.dma_dir != DMA_FROM_DEVICE) && (pool->p.dma_dir != DMA_BIDIRECTIONAL)) return -EINVAL; + + pool->dma_map = true; } if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) { @@ -195,13 +197,15 @@ static int page_pool_init(struct page_pool *pool, if (!pool->p.max_len) return -EINVAL; + pool->dma_sync_act = PP_DMA_SYNC_ACT_DO; + /* pool->p.offset has to be set according to the address * offset used by the DMA engine to start copying rx data */ } - if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT && - pool->p.flags & PP_FLAG_PAGE_FRAG) + pool->page_frag = !!(pool->p.flags & PP_FLAG_PAGE_FRAG); + if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT && pool->page_frag) return -EINVAL; #ifdef CONFIG_PAGE_POOL_STATS @@ -218,7 +222,7 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) get_device(pool->p.dev); return 0; @@ -346,7 +350,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) page_pool_set_dma_addr(page, dma); - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync_act == PP_DMA_SYNC_ACT_DO) page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -377,8 +381,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, if (unlikely(!page)) return NULL; - if ((pool->p.flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); return NULL; } @@ -398,8 +401,8 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, gfp_t gfp) { const int bulk = PP_ALLOC_CACHE_REFILL; - unsigned int pp_flags = pool->p.flags; unsigned int pp_order = pool->p.order; + bool dma_map = pool->dma_map; struct page *page; int i, nr_pages; @@ -424,8 +427,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, */ for (i = 0; i < nr_pages; i++) { page = pool->alloc.cache[i]; - if ((pp_flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); continue; } @@ -497,7 +499,7 @@ static void page_pool_return_page(struct page_pool *pool, struct page *page) dma_addr_t dma; int count; - if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + if (!pool->dma_map) /* Always account for inflight pages, even if we didn't * map them */ @@ -563,7 +565,7 @@ static bool page_pool_recycle_in_cache(struct page *page, } /* If the page refcnt == 1, this will try to recycle the page. - * if PP_FLAG_DMA_SYNC_DEV is set, we'll try to sync the DMA area for + * if pool->dma_sync_act is set, we'll try to sync the DMA area for * the configured size min(dma_sync_size, pool->max_len). * If the page refcnt != 1, then the page will be returned to memory * subsystem. @@ -584,7 +586,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync_act == PP_DMA_SYNC_ACT_DO) page_pool_dma_sync_for_device(pool, page, dma_sync_size); @@ -683,7 +685,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync_act == PP_DMA_SYNC_ACT_DO) page_pool_dma_sync_for_device(pool, page, -1); return page; @@ -713,8 +715,7 @@ struct page *page_pool_alloc_frag(struct page_pool *pool, unsigned int max_size = PAGE_SIZE << pool->p.order; struct page *page = pool->frag_page; - if (WARN_ON(!(pool->p.flags & PP_FLAG_PAGE_FRAG) || - size > max_size)) + if (WARN_ON(!pool->page_frag || size > max_size)) return NULL; size = ALIGN(size, dma_get_cache_alignment()); @@ -774,7 +775,7 @@ static void page_pool_free(struct page_pool *pool) ptr_ring_cleanup(&pool->ring, NULL); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) put_device(pool->p.dev); #ifdef CONFIG_PAGE_POOL_STATS From patchwork Thu Jul 27 14:43:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127047 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1210956vqo; Thu, 27 Jul 2023 09:16:21 -0700 (PDT) X-Google-Smtp-Source: APBJJlHUS4Z1B7GYWDQCgsrmK1ATF3KDnovE0LZivWAvCNYJ7jLQNj7OXYP7YwtFvqz4HcJ3RETT X-Received: by 2002:a05:6a00:c95:b0:668:82fe:16f1 with SMTP id a21-20020a056a000c9500b0066882fe16f1mr6931457pfv.1.1690474581109; Thu, 27 Jul 2023 09:16:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690474581; cv=none; d=google.com; s=arc-20160816; b=U1oYufdLZ/bQO5Mh3dntTMa4xD0z42VToD0lhJNzA5YVNsBNIcwH4+EwQEdHVhIfhP la8ydumwIP/1jS+N2dwBNKfMfEuTgiil3nUkZc+n/hoPoQ8k/4zjsLOubSjbCf5eqQFK eTfHH9rGvRMiiabUaTBd1r0URG2XZuzikJxdl3a6xBgG00PLhqKxN5cckY26H7jUTu0M WEcz8gM4IqeQGQeNQUAz7CukxM8DYKO9ko2t5Iu5AePUSV27gJ5POCONAWrJF5M+y4Rl 3gpbti0BNpgnjSRldGxomG74GMi73hyxzWsbG321qJfXIDyn8Cy9Fe16ceYlAaORQ74s O2Pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bBZqnXUbAUipWdT4Vd89hWOc1PTqacu4fQCklI1Mmko=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=fRA5ludD8Q2i/d10Eo9j676LYHgLhh1ZxwuFK6tQ4/mHY2n+SLDxsQ6eoaQUBMX2gj r9X4njS0DBSqETFVLg1PKvePuIFr4S0IRlna3Ixx12hkm+lAbytKHiMdzzpgilcELW26 b75X5DxFyrQeeaEHyrwRS6S99SUY9KNORGrUoNY+vb4RD+HvKs2SdpWZ11JTJmp9hjw+ S3yjWN+JJ3z7soi4g5agTxkzxhumzcaLz2cBEKbvLMIkP3B3CcrTPHW+oCMDK5An1ble Xvw1HEkMMnEIE2keipCmb05+RdM7VXDE/ITE+saYSj+a4/Z30Gv0JtQuneZFBdNpF6k8 xg8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=niqM5UzI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z14-20020a63190e000000b00563f871ca69si1396379pgl.859.2023.07.27.09.16.07; Thu, 27 Jul 2023 09:16:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=niqM5UzI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233175AbjG0Oq2 (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234036AbjG0OqF (ORCPT ); Thu, 27 Jul 2023 10:46:05 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2F513592; Thu, 27 Jul 2023 07:45:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469151; x=1722005151; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rXg0fjSPcukCgI8rvn6/BURKoRJDLNE56JJQvNgZt28=; b=niqM5UzIfKyOlPuLyVIxnK7ECzCCpVK5ALYUdrapFK5VtqNQ9guLZRbZ tYAQblKHYRT6kO9diepxWbiFHRKnZBJ10Qa43LiipUK1MchVb44Ht6ATW 5nLRZZQQfkFYmBe2BckijfUi8GWP43AgV3cGu+zBpDBFMDuvOQ3WUXhMV lDxlkxnq9frAa/hdhO+xrSm8uhb7s7dwHoyA5YBZiBUNVHaYYWDDdQoGP 8mu48CstiyIxu2plBP9CL+KBASYY46/R9q5t+GTzzmQFRronDvCUNQfyc d1IiOyG1nXkL2bJin6Nso/m56c+qTlBs0dSfCSi9IH5wMqATWjduP4+72 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139796" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139796" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119940" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119940" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:48 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 6/9] page_pool: avoid calling no-op externals when possible Date: Thu, 27 Jul 2023 16:43:33 +0200 Message-ID: <20230727144336.1646454-7-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772591074299536008 X-GMAIL-MSGID: 1772591074299536008 Turned out page_pool_put{,_full}_page() can burn quite a bunch of cycles even when on DMA-coherent platforms (like x86) with no active IOMMU or swiotlb, just for the call ladder. Indeed, it's page_pool_put_page() page_pool_put_defragged_page() <- external __page_pool_put_page() page_pool_dma_sync_for_device() <- non-inline dma_sync_single_range_for_device() dma_sync_single_for_device() <- external dma_direct_sync_single_for_device() dev_is_dma_coherent() <- exit For the inline functions, no guarantees the compiler won't uninline them (they're clearly not one-liners and sometimes compilers uninline even 2 + 2). The first external call is necessary, but the rest 2+ are done for nothing each time, plus a bunch of checks here and there. Since Page Pool mappings are long-term and for one "device + addr" pair dma_need_sync() will always return the same value (basically, whether it belongs to an swiotlb pool), addresses can be tested once right after they're obtained and the result can be reused until the page is unmapped. Define the new PP DMA sync operation type, which will mean "do DMA syncs for the device, but only when needed" and turn it on by default when the driver asks to sync pages. When a page is mapped, check whether it needs syncs and if so, replace that "sync when needed" back to "always do syncs" globally for the whole pool (better safe than sorry). As long as the pool has no pages requiring DMA syncs, this cuts off a good piece of calls and checks. When at least one page required it, the pool conservatively falls back to "always call sync functions", no per-page verdicts. It's a fairly rare case anyway that only a few pages would require syncing. On my x86_64, this gives from 2% to 5% performance benefit with no negative impact for cases when IOMMU is on and the shortcut can't be used. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 3 ++- net/core/page_pool.c | 7 ++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index dd26f4b2b66c..9134fb458cb2 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -97,8 +97,9 @@ struct page_pool { bool dma_map:1; /* Perform DMA mapping */ enum { PP_DMA_SYNC_ACT_DISABLED = 0, /* Driver didn't ask to sync */ + PP_DMA_SYNC_ACT_SKIP, /* Syncs can be skipped */ PP_DMA_SYNC_ACT_DO, /* Perform DMA sync ops */ - } dma_sync_act:1; + } dma_sync_act:2; bool page_frag:1; /* Allow page fragments */ long frag_users; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 6a8f105e2df5..529e4b41e9eb 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -197,7 +197,8 @@ static int page_pool_init(struct page_pool *pool, if (!pool->p.max_len) return -EINVAL; - pool->dma_sync_act = PP_DMA_SYNC_ACT_DO; + /* Try to avoid calling no-op syncs */ + pool->dma_sync_act = PP_DMA_SYNC_ACT_SKIP; /* pool->p.offset has to be set according to the address * offset used by the DMA engine to start copying rx data @@ -350,6 +351,10 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) page_pool_set_dma_addr(page, dma); + if (pool->dma_sync_act == PP_DMA_SYNC_ACT_SKIP && + dma_need_sync(pool->p.dev, dma)) + pool->dma_sync_act = PP_DMA_SYNC_ACT_DO; + if (pool->dma_sync_act == PP_DMA_SYNC_ACT_DO) page_pool_dma_sync_for_device(pool, page, pool->p.max_len); From patchwork Thu Jul 27 14:43:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127084 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1256335vqo; Thu, 27 Jul 2023 10:34:09 -0700 (PDT) X-Google-Smtp-Source: APBJJlEbgAbRTq0U3NYELjK5CHwvpG6ORwytI0h22VcGhyoPUVqIb+iSnlCE88IW5Y4ahHpIYqf0 X-Received: by 2002:a05:6a00:1882:b0:66a:2ff1:dee2 with SMTP id x2-20020a056a00188200b0066a2ff1dee2mr6427601pfh.10.1690479248871; Thu, 27 Jul 2023 10:34:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690479248; cv=none; d=google.com; s=arc-20160816; b=e/mEyMb0wl8Q/H8C5AC8byV8GWmxTp8/Pd4MQwGMT//LGUt2jpvdmOTyfVLd7yc5or r94VekxcnUfPZBg7v/6gL2m3EKaAEoZykSmLxGPXUELQSKbG2lXaCrcuRP0PazWJ+SXm 2a5n1QE49YOSR2mAkEmJbUc4CCHgTzizAM4DNSEW0pcskSC8JDTApg10TaTCTtNvhsHB rTOFqaWy9OBBu9THs6EBwe3CLFD2TZsy9B8f9iqsInkl2gfu71dAtmTkAkrMKpP8QL81 uhvqxV9Pg7Rt4Y9vaX3oDf5coLL+PpP8tf4DObdku2SlBv/JLnyPiCIqku11pA4B4Kdl 4xWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7z0gLwcNsgULFHJy+w4ndgOM1rkJ0SjVu1N70b7LYUs=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=IsFKyt11BHXlcbM/cr5TsGMzADOnBwLkDZjpdtLbqUObCyt7e3yXfvfly6Z4yCEccD setTmVjVoHZbMjfsQ43gJOsMakLbqc88NtSVdm9jbQWOSESKoRtf7t40mdN6g5RQQJBc JiNMSlUj01hC4k9IImKKalEvu/5MYLcVZSlNF1+1KEaS9ZvWxvOLe+mfBbyP6uiuaK09 l0gXiNbF04IoMmcLqNXsuEoWE9XVEO+Zc3d8JPtjJMcMkEMxYfBMU8AG8mR39EaodJVO gZtDzDiSCe3WRjEwqlcr4aQsujWekzRt42TWPcfqOIHMDFIIo17t1vO9MOkJXZurEWuE Kcww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=c74bP8y3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o6-20020a056a001bc600b006815bf78628si1555820pfw.356.2023.07.27.10.33.55; Thu, 27 Jul 2023 10:34:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=c74bP8y3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234201AbjG0Oqk (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234071AbjG0Oq0 (ORCPT ); Thu, 27 Jul 2023 10:46:26 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 396BE35B6; Thu, 27 Jul 2023 07:45:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469155; x=1722005155; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5UQmg0hLv8TkdVz/X6+MyXQQIYyvi5xDXayA3nnMuec=; b=c74bP8y341PVLLh37xGgpNzZG+c1qCCKvaZ+BaipaxeL4e8wohqMWPQR o0GrclNJvoQGQSiaBEmVjXFyKRTwpAeY+pBUnRXpGfPAz2kItOBiX5Ns+ ayR9dxyAahTw9iYHUm5kYm3v/oGiHp5XcqhuMI0dRSLlY8VtRrQPTFKiS BzetrKHdI+YRCW8aNTsbR1GZMJmW9Yt/iLnl1Ok4KQdGumgYzauXz45fL ojTl7S44EGxnzAVWJsg8S6emAaPW9LpBvldZpsPPytJ7ZAjk/dEiD7gFD P7hCFE7/oe/Km54nTqJHsE5UAVcAJAEA4ZQUDLxysT94uWXzNd4mwPEwM w==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139821" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139821" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119944" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119944" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:51 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 7/9] net: skbuff: avoid accessing page_pool if !napi_safe when returning page Date: Thu, 27 Jul 2023 16:43:34 +0200 Message-ID: <20230727144336.1646454-8-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772595968456993738 X-GMAIL-MSGID: 1772595968456993738 Currently, pp->p.napi is always read, but the actual variable it gets assigned to is read-only when @napi_safe is true. For the !napi_safe cases, which yet is still a pack, it's an unneeded operation. Moreover, it can lead to premature or even redundant page_pool cacheline access. For example, when page_pool_is_last_frag() returns false (with the recent frag improvements). Thus, read it only when @napi_safe is true. This also allows moving @napi inside the condition block itself. Constify it while we are here, because why not. Signed-off-by: Alexander Lobakin --- net/core/skbuff.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3084ef59400b..e701401092d7 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -881,9 +881,8 @@ static void skb_clone_fraglist(struct sk_buff *skb) bool page_pool_return_skb_page(struct page *page, bool napi_safe) { - struct napi_struct *napi; + bool allow_direct = false; struct page_pool *pp; - bool allow_direct; page = compound_head(page); @@ -903,9 +902,12 @@ bool page_pool_return_skb_page(struct page *page, bool napi_safe) * in the same context as the consumer would run, so there's * no possible race. */ - napi = READ_ONCE(pp->p.napi); - allow_direct = napi_safe && napi && - READ_ONCE(napi->list_owner) == smp_processor_id(); + if (napi_safe) { + const struct napi_struct *napi = READ_ONCE(pp->p.napi); + + allow_direct = napi && + READ_ONCE(napi->list_owner) == smp_processor_id(); + } /* Driver set this to memory recycling info. Reset it on recycle. * This will *not* work for NIC using a split-page memory model. From patchwork Thu Jul 27 14:43:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127026 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1180243vqo; Thu, 27 Jul 2023 08:29:35 -0700 (PDT) X-Google-Smtp-Source: APBJJlFFlIUyd5/Qmn9UYo4NeVhf7InfTeWhkAdtdGwQPGvX2p3l6aPbLt5gst+Jo/U5O4E/m+cZ X-Received: by 2002:a19:6452:0:b0:4fb:91c5:fd38 with SMTP id b18-20020a196452000000b004fb91c5fd38mr1963660lfj.0.1690471775053; Thu, 27 Jul 2023 08:29:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471775; cv=none; d=google.com; s=arc-20160816; b=Ws2um6KyZq2L8o1e9vSZuJK//fgthTOMMFIa2GrK8zce3TDMOCq90hW9gajHjQr3Jo +m3hUSAAihrylw4uzP6esfGoC+KjBgJUcB5jvAYiEiCuEBSRRM9LqbZGWFsnREljA4pc uaOzpcwUniP0vmZ1ZmeIZ6QI+zKiwWN1sxi3Uwc9vY4UKYlR8tFaFz25fmD9++2ctLPz +8uSduR+lXUkklJYqIryjlYKlAmF6/4zdcOqfnI+mfmNP746t39wetVAIhFENtuw2g73 r/H/yPV3zOL+ewpuuN4ZuoWYpGIIPOrWpzlpqCa0ErGe8t3uBRO9ayFZlznrd/+QETJC TMzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vhAons5/pd1uGUwYpWdWJPRFHMq70h55jOgp8PZoeWA=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=mfjrQhXbIKwpftv9FEa8byQaJsvOIQbG2rxkFRAxSdoEsAUtqNFBrzg1QnL10OPWCM UrCbJN3RRybX2zVbsGmoUdYhI+HkueVcA+rdJCGchZf/XL+JYhTS3N+qWIc9B7iyxmYB wbE7a7+C6cpRUvGgtJgEV9GFEsiTjrO6EnavAsdXDHeSM6pCg/MoTnT8SA8ac1zsyyw7 plm8+D5iboioQzuT1prufExGbuNVxJBOX07ezrYsrPLzNiWhd14PGONMsCbgW8Vf2h75 /uExfKKPMILhrp9V1Hw1DaNh5TOwq7LkQjCSyNgFeASduq9sCgsdOUahQE09HhynqUkF dUSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="E1rw8pp/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e2-20020a50fb82000000b0052228fad0f4si1064569edq.24.2023.07.27.08.29.09; Thu, 27 Jul 2023 08:29:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="E1rw8pp/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234032AbjG0Oqn (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234028AbjG0Oq1 (ORCPT ); Thu, 27 Jul 2023 10:46:27 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EBB230DA; Thu, 27 Jul 2023 07:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469158; x=1722005158; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KlYBSALGjCnGn6k2NRIvrpGdhSLAEVVLVU2ig0CAvyw=; b=E1rw8pp/7blEWzZ30GBezU/a32XZ1bZtX4JhsYd9TC6z0+LkW5HwkN+N A0r1ZIDYW7v7OYEriIAzfvItykKq51XJ5dGIdeDx8IjGz6Z9dfUtnYd7p bnRwUF0MfiZZ54rcadt5OVV26Wn9NsuqRpEBa07V+ibWpcHWwkWBKhAws 9rCk502nY6mm5j98ZIbKxwLyaQSS7OWdb6UrAqc9cb4J4jaAMoB4+e8Eb n82LFweB6wFPBOMntjWukbjOA4XCTKYA1wH5RWiNgEhIY7YmURFWfGhh2 gMHutjSJbBaOESLaQMQU6bXLL83AEfCu6Nvm6sx/LNVU1UKp9JaxRAxqT Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139848" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139848" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:45:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119955" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119955" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:54 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 8/9] page_pool: add a lockdep check for recycling in hardirq Date: Thu, 27 Jul 2023 16:43:35 +0200 Message-ID: <20230727144336.1646454-9-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772588131906636236 X-GMAIL-MSGID: 1772588131906636236 From: Jakub Kicinski Page pool use in hardirq is prohibited, add debug checks to catch misuses. IIRC we previously discussed using DEBUG_NET_WARN_ON_ONCE() for this, but there were concerns that people will have DEBUG_NET enabled in perf testing. I don't think anyone enables lockdep in perf testing, so use lockdep to avoid pushback and arguing :) Signed-off-by: Jakub Kicinski Signed-off-by: Alexander Lobakin --- include/linux/lockdep.h | 7 +++++++ net/core/page_pool.c | 2 ++ 2 files changed, 9 insertions(+) diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 310f85903c91..dc2844b071c2 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -625,6 +625,12 @@ do { \ WARN_ON_ONCE(__lockdep_enabled && !this_cpu_read(hardirq_context)); \ } while (0) +#define lockdep_assert_no_hardirq() \ +do { \ + WARN_ON_ONCE(__lockdep_enabled && (this_cpu_read(hardirq_context) || \ + !this_cpu_read(hardirqs_enabled))); \ +} while (0) + #define lockdep_assert_preemption_enabled() \ do { \ WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_COUNT) && \ @@ -659,6 +665,7 @@ do { \ # define lockdep_assert_irqs_enabled() do { } while (0) # define lockdep_assert_irqs_disabled() do { } while (0) # define lockdep_assert_in_irq() do { } while (0) +# define lockdep_assert_no_hardirq() do { } while (0) # define lockdep_assert_preemption_enabled() do { } while (0) # define lockdep_assert_preemption_disabled() do { } while (0) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 529e4b41e9eb..be9371cd9ac7 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -579,6 +579,8 @@ static __always_inline struct page * __page_pool_put_page(struct page_pool *pool, struct page *page, unsigned int dma_sync_size, bool allow_direct) { + lockdep_assert_no_hardirq(); + /* This allocator is optimized for the XDP mode that uses * one-frame-per-page, but have fallbacks that act like the * regular page allocator APIs. From patchwork Thu Jul 27 14:43:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 127032 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1181420vqo; Thu, 27 Jul 2023 08:31:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlFVU1Ao8QqBA2HpswY+iZAUqOCmz881co36FR7ofaNSyTkFwtv5MvlqPYtKa0/+FDYSSfaI X-Received: by 2002:a05:6a20:3cb:b0:134:a478:5e4a with SMTP id 11-20020a056a2003cb00b00134a4785e4amr3283618pzu.17.1690471876723; Thu, 27 Jul 2023 08:31:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690471876; cv=none; d=google.com; s=arc-20160816; b=eb87WZL76NxmieFgDq8jSNuXJTGp5pYwTEIUYcI9kJrhklJX+7ufNBlduQP9HdE90t meA1XSgl0ZvtJbq7aps6hGWPYraXQtsRQF+EHGOI4Fwn8tIMp1x7Z4+zSecw49JytMHj 3v85ahY2jL7o+yrrLhZZkDILhH2Xx4k5zWrIoXaRSYvPhf5nbjmqpsCnLgaZmvtsFrcg PK/iDcKk2GTJSHpZYB87+Wzui/2vhWpgsCKIg74oWT8sdgb3DbHSAy9kQMgs5M1NtylI Qu45Xs/BE71Is+0AhRKzeqMUPnW1xaHRpArXXlZ5xKV89+B28AxjO7MzVtn9Vk8OoPTT ZOFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FFj0ROMS6VWyNkPnCiyySUDzg/2jHUVOn3I8XdOvcXw=; fh=3HE1xXs4s0z3NpKSg+MkyPw1+1QpKlxno2HY1HR9bAc=; b=C2f5tMGZVHYiyIDgeTD5W5CIw190ty52WthnseBv0C7xXO7dp6q656/RnowkOigxgX 1wgcZrkUWcQ4BADpLuHSbDIDPaNoUc35Yqlgi0s7HNILMZzMtwmn/LHkjb05XRTxvp5F 9O9EbFMIconi1vO8b3ZzU72dzmkNVodJrVgnGw9/whpCLOiiKBLHw1k1TGPug2roJmQG vIAvt06IGIKVCN+m0wNwJhAmP8DkHIOYEVeFC3OAHv/NUf3DWcaHW5QsNr0L5Q+rLAKn dtxS+jVdjm6C2cPSstNZwZoLuSQQexfJDILKn8BlPLMy1xCcZP/Xsv5KZEnkg+wj+Ewa 8EJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IkgpUQGN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l20-20020a635714000000b0055337ca9cefsi1373242pgb.857.2023.07.27.08.31.02; Thu, 27 Jul 2023 08:31:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IkgpUQGN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234248AbjG0Oqy (ORCPT + 99 others); Thu, 27 Jul 2023 10:46:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234143AbjG0Oq3 (ORCPT ); Thu, 27 Jul 2023 10:46:29 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A92473AAA; Thu, 27 Jul 2023 07:46:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690469161; x=1722005161; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eYl99+L8eoZb9TdbyaCeQo6lRhwlNSjOXE6qobDW3K0=; b=IkgpUQGNIbKtfpljDJGk10+TiBLptw0DunKPCOtyK55fN2OgBAW8gYxh eAejiz0Qn+pA/f/3udRKxN9koAJPdaQbkAhqmPLdbMJt3s/TmXZSbgtIu GOHKwafQ11X6w1VQcvfY/7h2Iza0yEkwY/OyqxzAYtUjAHtjGE3kR1itl nBs0rbe+RYJIKim6+gGQg8LP5VHFJddzobUnjCF58eoRQR0MjE1TctsnU AlrcR8gyKvFuD7zLnuxzjq2pZVDI2WY0aa5wy9ocflw5Hr52fnSUMGdLF wxASAuNvr3TR5FqIM4P2wuiY3gfWRAGbSvb0902opVK/rHCVxVyP5mfsJ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="432139872" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="432139872" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 07:46:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="817119964" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="817119964" Received: from newjersey.igk.intel.com ([10.102.20.203]) by FMSMGA003.fm.intel.com with ESMTP; 27 Jul 2023 07:45:57 -0700 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Maciej Fijalkowski , Larysa Zaremba , Yunsheng Lin , Alexander Duyck , Jesper Dangaard Brouer , Ilias Apalodimas , Simon Horman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next 9/9] net: skbuff: always try to recycle PP pages directly when in softirq Date: Thu, 27 Jul 2023 16:43:36 +0200 Message-ID: <20230727144336.1646454-10-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727144336.1646454-1-aleksander.lobakin@intel.com> References: <20230727144336.1646454-1-aleksander.lobakin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772588238640203798 X-GMAIL-MSGID: 1772588238640203798 Commit 8c48eea3adf3 ("page_pool: allow caching from safely localized NAPI") allowed direct recycling of skb pages to their PP for some cases, but unfortunately missed a couple of other majors. For example, %XDP_DROP in skb mode. The netstack just calls kfree_skb(), which unconditionally passes `false` as @napi_safe. Thus, all pages go through ptr_ring and locks, although most of time we're actually inside the NAPI polling this PP is linked with, so that it would be perfectly safe to recycle pages directly. Let's address such. If @napi_safe is true, we're fine, don't change anything for this path. But if it's false, check whether we are in the softirq context. It will most likely be so and then if ->list_owner is our current CPU, we're good to use direct recycling, even though @napi_safe is false -- concurrent access is excluded. in_softirq() protection is needed mostly due to we can hit this place in the process context (not the hardirq though). For the mentioned xdp-drop-skb-mode case, the improvement I got is 3-4% in Mpps. As for page_pool stats, recycle_ring is now 0 and alloc_slow counter doesn't change most of time, which means the MM layer is not even called to allocate any new pages. Suggested-by: Jakub Kicinski # in_softirq() Signed-off-by: Alexander Lobakin --- net/core/skbuff.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index e701401092d7..5ba3948cceed 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -901,8 +901,10 @@ bool page_pool_return_skb_page(struct page *page, bool napi_safe) /* Allow direct recycle if we have reasons to believe that we are * in the same context as the consumer would run, so there's * no possible race. + * __page_pool_put_page() makes sure we're not in hardirq context + * and interrupts are enabled prior to accessing the cache. */ - if (napi_safe) { + if (napi_safe || in_softirq()) { const struct napi_struct *napi = READ_ONCE(pp->p.napi); allow_direct = napi &&