From patchwork Mon Feb 5 11:04:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196751 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp813106dyb; Mon, 5 Feb 2024 03:35:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IE398+yB5vNQhL88+iHOdywkXa85AEppQkTZb+GejbRG0A34U46PbmHwpNsmL0URqTgtw/L X-Received: by 2002:a05:6a20:ce48:b0:19e:4e80:27f0 with SMTP id id8-20020a056a20ce4800b0019e4e8027f0mr11390428pzb.37.1707132922996; Mon, 05 Feb 2024 03:35:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707132922; cv=pass; d=google.com; s=arc-20160816; b=hk6zoo+8BDe7s+6p9Pjy0f8xNG38FpGNFVUsn6PMJmSiag+3ZJhoKjkxmxrld8bmnd bWu7hdC3qiSx+5ASmVie2B5B0UtjxT09FOfPfQItgCw7S6Tp2FbJTTMTdFDND2bQUlMl 9DPXU77PDoj/bCjDKF2n1H7qQ8PGKRByYIS0XfclaYhbzWBY53eseQ8S8MRfgOw9cD1H SCuPbm5m0FpoARj4XTzFv3GBSl4u5V/LmtUh4I0Qw2tBz1l11apR1djccdxRPP/7yMm7 GRecC06wYPnI0xZu2EZWVyXG7/+mUcHwwCO6+XYVUBDZbuNCpAFCSSTX1RkSnaki9g/T 0+AA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=HbIH3TV3juYGUsMClwv1Ryzw25I+G/uciu/ZrnxNIak=; fh=q2O/nm10gU5huKr2cWbpJE+fJILFeBU/FbYJADgFEYA=; b=wvdhXa+Dd8u9EOuYQM6FvMhB4TjtyIxePg4oxoio1UVqucejCfVKNhZvxhclCOG1HL eV95EWsWs2KTN3gloapcZrdwL+Ax2nfTroEwTpghA/OYIPH6kV5EbmpTDUMvI6nGtMMn KfLpY/LoTFxJCeL9PNpRNjZoSv3ZTmf8EPf1Jw0X5Tsu3kuNRuSc3l8goB5l4grxL9vB aVvIUwRjFJ1WT+pVeDxmyzEIBdJvIGwHbWVr7snD326xG/BipVSvQ6yZWiVOXsx2EyGI lwVzsc+wbEsA18csHC4e/+gpK3dyd0JFG6571qVAzmSGEWWUgnxSH+DM3XGaqSqQ5wvh f2kg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lOUsi0Jo; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52449-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCViH5GFwwMeA+tijgJnipLVBOjyetH0OUCi5t+lM31PpBQcU8w0kDnXJCLH1M5/tgfttOYsCP/fBXCjOjEfb7INzFcvyA== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id 13-20020a63154d000000b005d8bb44b21asi6029722pgv.542.2024.02.05.03.35.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:35:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lOUsi0Jo; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52449-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E2CE0B279FA for ; Mon, 5 Feb 2024 11:06:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E3507199A2; Mon, 5 Feb 2024 11:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="lOUsi0Jo" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83ED618027; Mon, 5 Feb 2024 11:05:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131114; cv=none; b=l+CVwjYNSuXhF3dQAC9fY+brr9HlRYvCKWmv8uVX6kRj68ArKW5NXx5pca7bZAhSAd3L/nEDqHZOpISZVeBOaJ90yM5XUhYR+h+iYZOB0ule92bqjasDKmBa9Tvuz1h7N/9TDgKx+nIo41UvodbVOgcWGyp+i7t1c1DliMvXKOo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131114; c=relaxed/simple; bh=qZtBthYGqwmEme9KNh+V0p9VPVK4TqaRNwdPkzk3SD4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CwIyqn2RwAKYseMCvu4gqravDCfZRmuAaKylAsSiqkuyrSsNSVemRzVGfnEbBh6foWonKyumlxobrCEvR3ReBfCIlFboPmXEbnJ1MX/Gdu0cWNBxmjbBUaaubYnVWMhHZ0BZyggVZbFzylq5sp1fihg45TXj17lAtpllPNg8vYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=lOUsi0Jo; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131112; x=1738667112; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qZtBthYGqwmEme9KNh+V0p9VPVK4TqaRNwdPkzk3SD4=; b=lOUsi0Jo+leegoCPtovMvT2o4LjbbN9LEAXw0HVyIZKJK/EEDcv35nXP OQOnm4pB1xo7Yszp3RB1pvts6iwHRodIi6fbpNoLop2f/5QQymOHIo0i5 QqHxTawVwZUD/m+FxrmU/D2MylF2Hc8DM8YFNuQN9Rn+95zcFj6l1go0T z9VoCHW6iDRtJfhKLl0P2AxQ28JSqztoRlXiuW2+Nvfs+YS/x3EWNGY+D NJXmMA5S3y2mTt36plg6vsJWFKvbN9GXXxDg8xEUUVed2Kt0nQoyTwsT3 aSvJxar5qzSYBOgyQFi3wfpjbE++RP6uJ/EFYNnTCpATNxSTmLPzcEEMk w==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945232" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945232" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5327677" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:07 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 1/7] dma: compile-out DMA sync op calls when not used Date: Mon, 5 Feb 2024 12:04:20 +0100 Message-ID: <20240205110426.764393-2-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790058612052510099 X-GMAIL-MSGID: 1790058612052510099 Some platforms do have DMA, but DMA there is always direct and coherent. Currently, even on such platforms DMA sync operations are compiled and called. Add a new hidden Kconfig symbol, DMA_NEED_SYNC, and set it only when either sync operations are needed or there is DMA ops or swiotlb enabled. Set dma_need_sync() and dma_skip_sync() depending on this symbol state and don't call sync ops when dma_skip_sync() is true. The change allows for future optimizations of DMA sync calls depending on compile-time or runtime conditions. Signed-off-by: Alexander Lobakin --- kernel/dma/Kconfig | 4 ++ include/linux/dma-mapping.h | 118 ++++++++++++++++++++++++++++-------- kernel/dma/mapping.c | 28 ++++----- 3 files changed, 110 insertions(+), 40 deletions(-) diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index d62f5957f36b..1c9ff05b1ecb 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -107,6 +107,10 @@ config DMA_BOUNCE_UNALIGNED_KMALLOC bool depends on SWIOTLB +config DMA_NEED_SYNC + def_bool ARCH_HAS_SYNC_DMA_FOR_DEVICE || ARCH_HAS_SYNC_DMA_FOR_CPU || \ + ARCH_HAS_SYNC_DMA_FOR_CPU_ALL || DMA_OPS || SWIOTLB + config DMA_RESTRICTED_POOL bool "DMA Restricted Pool" depends on OF && OF_RESERVED_MEM && SWIOTLB diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 4a658de44ee9..569a4da68f56 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -117,14 +117,14 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); -void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, - enum dma_data_direction dir); -void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir); -void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir); -void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir); +void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir); +void __dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir); +void __dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); +void __dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs); void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, @@ -147,7 +147,7 @@ u64 dma_get_required_mask(struct device *dev); bool dma_addressing_limited(struct device *dev); size_t dma_max_mapping_size(struct device *dev); size_t dma_opt_mapping_size(struct device *dev); -bool dma_need_sync(struct device *dev, dma_addr_t dma_addr); +bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr); unsigned long dma_get_merge_boundary(struct device *dev); struct sg_table *dma_alloc_noncontiguous(struct device *dev, size_t size, enum dma_data_direction dir, gfp_t gfp, unsigned long attrs); @@ -195,20 +195,24 @@ static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) { } -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) +static inline void __dma_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, + enum dma_data_direction dir) { } -static inline void dma_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) +static inline void __dma_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, + enum dma_data_direction dir) { } -static inline void dma_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sg, int nelems, enum dma_data_direction dir) +static inline void __dma_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sg, int nelems, + enum dma_data_direction dir) { } -static inline void dma_sync_sg_for_device(struct device *dev, - struct scatterlist *sg, int nelems, enum dma_data_direction dir) +static inline void __dma_sync_sg_for_device(struct device *dev, + struct scatterlist *sg, int nelems, + enum dma_data_direction dir) { } static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) @@ -277,7 +281,7 @@ static inline size_t dma_opt_mapping_size(struct device *dev) { return 0; } -static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +static inline bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) { return false; } @@ -348,18 +352,80 @@ static inline void dma_unmap_single_attrs(struct device *dev, dma_addr_t addr, return dma_unmap_page_attrs(dev, addr, size, dir, attrs); } -static inline void dma_sync_single_range_for_cpu(struct device *dev, - dma_addr_t addr, unsigned long offset, size_t size, - enum dma_data_direction dir) +static inline void +__dma_sync_single_range_for_cpu(struct device *dev, dma_addr_t addr, + unsigned long offset, size_t size, + enum dma_data_direction dir) +{ + __dma_sync_single_for_cpu(dev, addr + offset, size, dir); +} + +static inline void +__dma_sync_single_range_for_device(struct device *dev, dma_addr_t addr, + unsigned long offset, size_t size, + enum dma_data_direction dir) +{ + __dma_sync_single_for_device(dev, addr + offset, size, dir); +} + +static inline bool dma_skip_sync(const struct device *dev) +{ + return !IS_ENABLED(CONFIG_DMA_NEED_SYNC); +} + +static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +{ + return !dma_skip_sync(dev) ? __dma_need_sync(dev, dma_addr) : false; +} + +static inline void dma_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, + enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_single_for_cpu(dev, addr, size, dir); +} + +static inline void dma_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, + enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_single_for_device(dev, addr, size, dir); +} + +static inline void dma_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sg, int nelems, + enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_sg_for_cpu(dev, sg, nelems, dir); +} + +static inline void dma_sync_sg_for_device(struct device *dev, + struct scatterlist *sg, int nelems, + enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_sg_for_device(dev, sg, nelems, dir); +} + +static inline void +dma_sync_single_range_for_cpu(struct device *dev, dma_addr_t addr, + unsigned long offset, size_t size, + enum dma_data_direction dir) { - return dma_sync_single_for_cpu(dev, addr + offset, size, dir); + if (!dma_skip_sync(dev)) + __dma_sync_single_for_cpu(dev, addr + offset, size, dir); } -static inline void dma_sync_single_range_for_device(struct device *dev, - dma_addr_t addr, unsigned long offset, size_t size, - enum dma_data_direction dir) +static inline void +dma_sync_single_range_for_device(struct device *dev, dma_addr_t addr, + unsigned long offset, size_t size, + enum dma_data_direction dir) { - return dma_sync_single_for_device(dev, addr + offset, size, dir); + if (!dma_skip_sync(dev)) + __dma_sync_single_for_device(dev, addr + offset, size, dir); } /** diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..8716e5e8281c 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -329,8 +329,8 @@ void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, } EXPORT_SYMBOL(dma_unmap_resource); -void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, - enum dma_data_direction dir) +void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -341,10 +341,10 @@ void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, ops->sync_single_for_cpu(dev, addr, size, dir); debug_dma_sync_single_for_cpu(dev, addr, size, dir); } -EXPORT_SYMBOL(dma_sync_single_for_cpu); +EXPORT_SYMBOL(__dma_sync_single_for_cpu); -void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) +void __dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -355,10 +355,10 @@ void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, ops->sync_single_for_device(dev, addr, size, dir); debug_dma_sync_single_for_device(dev, addr, size, dir); } -EXPORT_SYMBOL(dma_sync_single_for_device); +EXPORT_SYMBOL(__dma_sync_single_for_device); -void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) +void __dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -369,10 +369,10 @@ void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, ops->sync_sg_for_cpu(dev, sg, nelems, dir); debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); } -EXPORT_SYMBOL(dma_sync_sg_for_cpu); +EXPORT_SYMBOL(__dma_sync_sg_for_cpu); -void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) +void __dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -383,7 +383,7 @@ void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, ops->sync_sg_for_device(dev, sg, nelems, dir); debug_dma_sync_sg_for_device(dev, sg, nelems, dir); } -EXPORT_SYMBOL(dma_sync_sg_for_device); +EXPORT_SYMBOL(__dma_sync_sg_for_device); /* * The whole dma_get_sgtable() idea is fundamentally unsafe - it seems @@ -841,7 +841,7 @@ size_t dma_opt_mapping_size(struct device *dev) } EXPORT_SYMBOL_GPL(dma_opt_mapping_size); -bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -849,7 +849,7 @@ bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) return dma_direct_need_sync(dev, dma_addr); return ops->sync_single_for_cpu || ops->sync_single_for_device; } -EXPORT_SYMBOL_GPL(dma_need_sync); +EXPORT_SYMBOL_GPL(__dma_need_sync); unsigned long dma_get_merge_boundary(struct device *dev) { From patchwork Mon Feb 5 11:04:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196737 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp800340dyb; Mon, 5 Feb 2024 03:06:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IEs9WSGYyrUjogDrOxqQmfFLg7gcPCpmKB2sUyx88wrfxt9QSfLsoFid3MC3RvnePLVJoRD X-Received: by 2002:a6b:7219:0:b0:7c0:2daf:74aa with SMTP id n25-20020a6b7219000000b007c02daf74aamr13522543ioc.19.1707131182640; Mon, 05 Feb 2024 03:06:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707131182; cv=pass; d=google.com; s=arc-20160816; b=OpzjHunW9QrbpDiddQPsi7BJBTHqgTEUdHMDoP9qsiaDbkuLI1rCp+8aPY96ZpBokB EmYVd+Dr7yC/C0URltYGXTpxcKQNmMGEIh1NuBrOy+69IcKmtbTiq6+oEh9SR2ftrqOb uVif8jFKlezNFFBVXxUoSUm6uOUZLgsviZy7bKKy680AOry2kTp1Kc9qSbZBMUAiJYuj mY4ZwIeFoSXbVXAsg+Df+Lkt3twHUvaIXCz8s4Z1+MZ8JuZms+egYzaP+rOIyGZZzkAq omKeXSkasCNmdOCioFJJtvBBxBkKfVCzNxoNm8NcwRw4LkD0CSkue2F/kyXiN2pBYeYK KtKg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=ijQEPwCzqukoseHBSMsuYt+SddyslIcA2JQRXcDDWBo=; fh=5AiE9j8oMBwMMFVRrpeWj5EI9hfkL63OaBfs0DSqsnA=; b=hxNFZIx9w6HYEByWAO1beAhp/uukzuGhhY+ZcHf88vvAJk8eZCi/NUAyRV6Hwy1mpH Z0yw9Ih+SByfnb50EIrRA7XVmZX6Cby4eT2wRT09/CLN2zT4MvI55kVba3G0ZqZi6jAC 9VI04wac/3eW9RRYleGlD/+EE+LcZaotxIBifTL+StycToCkjpH+xOGcjO46z2LmqMjn eBU0r+Z4jEN9tgkgmOYLUNppChKpv0/piDZfhH3+H9Q1Qfok6xY4o0T57X7MAsf9bH5l hIoik6P/5VDuGfHXHLq3ZQvQIOA8Vc6f97GqNYaribUvkJ1+P456ga2Oemy1UbOrhXE0 bUbQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=isyPU9kM; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52450-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52450-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCX/LtJW4ar05L1Q0Z0yS18lxu9wi4LGhzcM1pKuDN5E/EfbC0COf+tt6gEh0XrsEA2eX8lP4F2sHwukhriOQhVJAdc8RQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id q2-20020a63bc02000000b005d6a0068c1csi5980156pge.95.2024.02.05.03.06.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:06:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52450-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=isyPU9kM; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52450-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52450-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 5C3EA28126C for ; Mon, 5 Feb 2024 11:06:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0B1E318027; Mon, 5 Feb 2024 11:05:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="isyPU9kM" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AE5718E3F; Mon, 5 Feb 2024 11:05:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131118; cv=none; b=RqplZx86FbQxssFhm/f4PDfFjFNzHsFdWwJ0PPX1cuceG4qryMuntZi/Kd2XILT39H6CbbP5TN/QUyqKX2qbVodXcrDBYyRLp5Ltg11wlcMtWbnZhuhgpmyS7yyWIa7tlo9VWyJWUcb+79vmbzqdQN3tSFOYdgbi+qgGwXqJDv4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131118; c=relaxed/simple; bh=ZBi16G6nTGV1woqfWZjvwuEuRwW/kOjMRUhMHdPBUnI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ek2TqCSSNXh/PdsvT7O7A4pAQAbdrGmOI8HOJBkZumHxkofURpJ9ZcX8UV+ApCb6rzZh/WhNscye7roSJBjjnvwqwEaeMFKnVtVGJooGIJkkj3DdT0YTZ8b7MzQQ+ZGI60e35LSihs9Rkc7JGvSOj9OoAqkTPVOoB7oWCbkKbFg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=isyPU9kM; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131116; x=1738667116; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZBi16G6nTGV1woqfWZjvwuEuRwW/kOjMRUhMHdPBUnI=; b=isyPU9kMIlsqXC5nVw40wkKpZrkkr7LE7a30PbuzS6IMQ/GHX9bKMSLz /E9s1RvQ+46YBHW0q3gKUsKUcpVVDQVitEHOOc0ROjbShyVSX7rDcxSjM gMberNiDxGDL3Spw1+QyEKMwfJnG1Hxkoz7c8AN+4Ew3lyZivO4oSSyHl kLMHOvc8Dzlg+EjkiQ4ClreOzNVpldTNF6PzMvkZyWjor2zKNiUc+FAT0 YiG2G0Wvaf4ZvKz20QZM63jfBmMdAL6sLqreHJPMzEFg9VydDpVuatRFw OR1ONmn29zj/nHcUWx1ovcUwpUv3eMpGv734shYLItiiKCijpsiSf7Dqb g==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945285" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945285" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5327858" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:11 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 2/7] dma: avoid redundant calls for sync operations Date: Mon, 5 Feb 2024 12:04:21 +0100 Message-ID: <20240205110426.764393-3-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790056787271789933 X-GMAIL-MSGID: 1790056787271789933 Quite often, NIC devices do not need dma_sync operations on x86_64 at least. Indeed, when dev_is_dma_coherent(dev) is true and dev_use_swiotlb(dev) is false, iommu_dma_sync_single_for_cpu() and friends do nothing. However, indirectly calling them when CONFIG_RETPOLINE=y consumes about 10% of cycles on a cpu receiving packets from softirq at ~100Gbit rate. Even if/when CONFIG_RETPOLINE is not set, there is a cost of about 3%. Add dev->skip_dma_sync boolean which is set during the device initialization depending on the setup: dev_is_dma_coherent() for the direct DMA, !(sync_single_for_device || sync_single_for_cpu) or the new dma_map_ops flag, %DMA_F_CAN_SKIP_SYNC, advertised for non-NULL DMA ops. Then later, if/when swiotlb is used for the first time, the flag is turned off, from swiotlb_tbl_map_single(). On iavf, the UDP trafficgen with XDP_DROP in skb mode test shows +3-5% increase for direct DMA. In case some device doesn't work with the shortcut: * include to the driver source; * call dma_set_skip_sync(dev, false) at the beginning of the probe callback. This will disable the shortcut and force DMA syncs. Suggested-by: Christoph Hellwig # direct DMA shortcut Co-developed-by: Eric Dumazet Signed-off-by: Eric Dumazet Signed-off-by: Alexander Lobakin --- include/linux/device.h | 5 +++++ include/linux/dma-map-ops.h | 20 ++++++++++++++++++ include/linux/dma-mapping.h | 6 +++++- drivers/base/dd.c | 2 ++ kernel/dma/mapping.c | 42 ++++++++++++++++++++++++++++++++++++- kernel/dma/swiotlb.c | 14 +++++++++++++ 6 files changed, 87 insertions(+), 2 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index 97c4b046c09d..f23e6a32bea0 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -686,6 +686,8 @@ struct device_physical_location { * other devices probe successfully. * @dma_coherent: this particular device is dma coherent, even if the * architecture supports non-coherent devices. + * @dma_skip_sync: DMA sync operations can be skipped for coherent non-SWIOTLB + * buffers. * @dma_ops_bypass: If set to %true then the dma_ops are bypassed for the * streaming DMA operations (->map_* / ->unmap_* / ->sync_*), * and optionall (if the coherent mask is large enough) also @@ -800,6 +802,9 @@ struct device { defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) bool dma_coherent:1; #endif +#ifdef CONFIG_DMA_NEED_SYNC + bool dma_skip_sync:1; +#endif #ifdef CONFIG_DMA_OPS_BYPASS bool dma_ops_bypass : 1; #endif diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..3406fb950980 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -18,8 +18,11 @@ struct iommu_ops; * * DMA_F_PCI_P2PDMA_SUPPORTED: Indicates the dma_map_ops implementation can * handle PCI P2PDMA pages in the map_sg/unmap_sg operation. + * DMA_F_CAN_SKIP_SYNC: DMA sync operations can be skipped if the device is + * coherent and it's not an SWIOTLB buffer. */ #define DMA_F_PCI_P2PDMA_SUPPORTED (1 << 0) +#define DMA_F_CAN_SKIP_SYNC BIT(1) struct dma_map_ops { unsigned int flags; @@ -111,6 +114,23 @@ static inline void set_dma_ops(struct device *dev, } #endif /* CONFIG_DMA_OPS */ +#ifdef CONFIG_DMA_NEED_SYNC +void dma_setup_skip_sync(struct device *dev); + +static inline void dma_set_skip_sync(struct device *dev, bool skip) +{ + dev->dma_skip_sync = skip; +} +#else /* !CONFIG_DMA_NEED_SYNC */ +static inline void dma_setup_skip_sync(struct device *dev) +{ +} + +static inline void dma_set_skip_sync(struct device *dev, bool skip) +{ +} +#endif /* !CONFIG_DMA_NEED_SYNC */ + #ifdef CONFIG_DMA_CMA extern struct cma *dma_contiguous_default_area; diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 569a4da68f56..03711ae6c4db 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -370,7 +370,11 @@ __dma_sync_single_range_for_device(struct device *dev, dma_addr_t addr, static inline bool dma_skip_sync(const struct device *dev) { - return !IS_ENABLED(CONFIG_DMA_NEED_SYNC); +#ifdef CONFIG_DMA_NEED_SYNC + return dev->dma_skip_sync; +#else + return true; +#endif } static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) diff --git a/drivers/base/dd.c b/drivers/base/dd.c index 85152537dbf1..67ad3e1d51f6 100644 --- a/drivers/base/dd.c +++ b/drivers/base/dd.c @@ -642,6 +642,8 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto pinctrl_bind_failed; } + dma_setup_skip_sync(dev); + ret = driver_sysfs_add(dev); if (ret) { pr_err("%s: driver_sysfs_add(%s) failed\n", diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 8716e5e8281c..b815e1bbc2d0 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -846,8 +846,14 @@ bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) const struct dma_map_ops *ops = get_dma_ops(dev); if (dma_map_direct(dev, ops)) + /* + * dma_skip_sync could've been set to false on first SWIOTLB + * buffer mapping, but @dma_addr is not necessary an SWIOTLB + * buffer. In this case, fall back to more granular check. + */ return dma_direct_need_sync(dev, dma_addr); - return ops->sync_single_for_cpu || ops->sync_single_for_device; + + return true; } EXPORT_SYMBOL_GPL(__dma_need_sync); @@ -861,3 +867,37 @@ unsigned long dma_get_merge_boundary(struct device *dev) return ops->get_merge_boundary(dev); } EXPORT_SYMBOL_GPL(dma_get_merge_boundary); + +#ifdef CONFIG_DMA_NEED_SYNC +void dma_setup_skip_sync(struct device *dev) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + bool skip; + + if (dma_map_direct(dev, ops)) + /* + * dma_skip_sync will be set to false on first SWIOTLB buffer + * mapping, if any. During the device initialization, it's + * enough to check only for DMA coherence. + */ + skip = dev_is_dma_coherent(dev); + else if (!ops->sync_single_for_device && !ops->sync_single_for_cpu) + /* + * Synchronization is not possible when none of DMA sync ops + * is set. This check precedes the below one as it disables + * the synchronization unconditionally. + */ + skip = true; + else if (ops->flags & DMA_F_CAN_SKIP_SYNC) + /* + * Assume that when ``DMA_F_CAN_SKIP_SYNC`` is advertised, + * the conditions for synchronizing are the same as with + * the direct DMA. + */ + skip = dev_is_dma_coherent(dev); + else + skip = false; + + dma_set_skip_sync(dev, skip); +} +#endif /* CONFIG_DMA_NEED_SYNC */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index b079a9a8e087..b62ea0a4f106 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1286,6 +1286,16 @@ static unsigned long mem_used(struct io_tlb_mem *mem) #endif /* CONFIG_DEBUG_FS */ +static inline void swiotlb_disable_dma_skip_sync(struct device *dev) +{ + /* + * If dma_skip_sync was set, reset it to false on first SWIOTLB buffer + * mapping/allocation to always sync SWIOTLB buffers. + */ + if (unlikely(dma_skip_sync(dev))) + dma_set_skip_sync(dev, false); +} + phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, unsigned int alloc_align_mask, enum dma_data_direction dir, @@ -1323,6 +1333,8 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } + swiotlb_disable_dma_skip_sync(dev); + /* * Save away the mapping from the original address to the DMA address. * This is needed when we sync the memory. Then we sync the buffer if @@ -1640,6 +1652,8 @@ struct page *swiotlb_alloc(struct device *dev, size_t size) if (index == -1) return NULL; + swiotlb_disable_dma_skip_sync(dev); + tlb_addr = slot_addr(pool->start, index); return pfn_to_page(PFN_DOWN(tlb_addr)); From patchwork Mon Feb 5 11:04:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196738 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp800644dyb; Mon, 5 Feb 2024 03:06:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IE37uOlJqMFSHYpBiSdEIBOBpDn2wu6S2a+AEh9Z7FfL0u4blAu339sfrhAyXMYgNw9DHOk X-Received: by 2002:a17:907:784a:b0:a37:249c:fc73 with SMTP id lb10-20020a170907784a00b00a37249cfc73mr5536376ejc.15.1707131216177; Mon, 05 Feb 2024 03:06:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707131216; cv=pass; d=google.com; s=arc-20160816; b=jmpKVMy9DgPcxUZSVZZKc17A48ZDaa1fm5lo+9J03a5MpZckzu+YXJxPTPxHWMU+eh rHVgS1yTTGQ+XFzC4iMZkEWxy0MHrIexuSY37i9pr31hkrMSJBuluS8HZF1DuLiDLh/3 YiH+Zrp1Q4NWqo5/RLtCyArO526Hmd+ri+n+gVd0zXXwqCcRCqxJXqcfXwgrnEKAwbCK nIgxQA6oSJCZDbQX/rtdqnJKK/xyzgpf8b4TnGwU/9pInZcW9zwsZrgLxrykTFXhxpYT P567Eq3T5YBkLlj1PzaCth2OIU29ZsoWBFqRVWa670qU4D0sTMajZh5NORBIo2niYW+q 40UQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=3RbYCi97CBUYpb3Qk7i6iB9pJIZcKC+4jz+RMzWKVIE=; fh=q9VfXgg+a8UYJzmEIxPxV3XF3aNZpKZdfjHgQurc4dE=; b=Dpv8SAtENW0fvKS3wGVgi6aMWTB3CYy3ZmS4qI8n2zcqK2Y/oEx+UZ76ZTo41vEq68 6Vc8C+JyKZ9FPqifwQ9tHSYGwfOzVkZqqcachGZ7Y4zBsxa337aI9eV08no290y+SH60 mMHGyYiWtH/m8MvIm3BxLOIyNJjbCOlze+rJK5FYrmiS6s8UyxVmUJH/+MiT49plZ2+C BXyxqVEf9QQym9DLlBzVCn3RcaKTZ7s0vyht44b8zS80/jbZqS6pwOGw+wt5TaYtzX9H k+2cJw6VndDy16WM+Usute3JhVjBOydLYrc1GivwEVrKGiT7l8oc6dvrGSIExxG3QSkn FHQg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OFn7ejKn; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52451-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52451-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCWaEcZAQ3NXG/6ku4SJmU83cRrc8XdRgxiW8tPKauiaZGUQLPKhgAA0Ssdg5mwCqKVB6MMm7G8t2xgjfQXAJclVky+O9g== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id gq8-20020a170906e24800b00a37f5b4225asi28853ejb.707.2024.02.05.03.06.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:06:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52451-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OFn7ejKn; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52451-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52451-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 853E01F21099 for ; Mon, 5 Feb 2024 11:06:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E44531B7FC; Mon, 5 Feb 2024 11:05:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OFn7ejKn" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33A241A5BA; Mon, 5 Feb 2024 11:05:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131122; cv=none; b=D0ShCv9zCDsh8jIdjUAnEnbuC/p63X/N2nhsA1I6Op6vm5rgCFbbb+yJbXz9/Xm4BB/99sXvlxKPRKPqHho0E1CShcJ7nj6EPcpmAF309f7kdvHE6unP9umtARKNYbHIXeh/XuBZTnHk+TEa4VVjSrkJ0MdRYidqrKwtaTbdqPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131122; c=relaxed/simple; bh=LVf5BgWT9OSmbGH+Qwko5u5SJBVWf1jKCf7S7n+8WTs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tC3pSOWbc108x9GxKS5KzXIs6lxA75TqBhDd2cgPfUURonUcO7SrzMGGBggEY0PVLJAheqbn7oRm6pB6A4NwQzYdGa/y8y2Rup1jVwzAdTPhHUagwk+Zu1308HjZokAa4nIAifk7FWp+I/T9asvVFQ9gVP5vtpIAj7ZQNu+DiuM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OFn7ejKn; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131120; x=1738667120; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LVf5BgWT9OSmbGH+Qwko5u5SJBVWf1jKCf7S7n+8WTs=; b=OFn7ejKnfVsTz6m1MRO70tcOjMaFMF+XZPm/Zk6SoyjQV7gEyCq6xDo1 Isut9fXYTBYalKqIy6erni7Wjb8FHh7eKFdMaFhpa0zfYKuQ6TF+Y3qZu duG7l0vYo7YdjOmcle33LruJrkr1/mNISGmhNs7KA2U6xRtwavma3bOzu 1JsbTNMqlbyQU27obpMk8iymkKGhkEvnULjS8ZqjnYgD964kQT4pR8AZ6 T6iWi2mSZ37B+VOOXLV0hjy+maAHtOoow7QuywTLvTYItt9wI1BiqQX6h Ug9NA7tyuut7O6xGL+2WAb2nhDzh7iEMTxswhA+4LN9bptKnt6r+NNzMj w==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945326" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945326" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328048" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:15 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 3/7] iommu/dma: avoid expensive indirect calls for sync operations Date: Mon, 5 Feb 2024 12:04:22 +0100 Message-ID: <20240205110426.764393-4-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790056822061821877 X-GMAIL-MSGID: 1790056822061821877 When IOMMU is on, the actual synchronization happens in the same cases as with the direct DMA. Advertise %DMA_F_CAN_SKIP_SYNC in IOMMU DMA to skip sync ops calls (indirect) for non-SWIOTLB buffers. perf profile before the patch: 18.53% [kernel] [k] gq_rx_skb 14.77% [kernel] [k] napi_reuse_skb 8.95% [kernel] [k] skb_release_data 5.42% [kernel] [k] dev_gro_receive 5.37% [kernel] [k] memcpy <*> 5.26% [kernel] [k] iommu_dma_sync_sg_for_cpu 4.78% [kernel] [k] tcp_gro_receive <*> 4.42% [kernel] [k] iommu_dma_sync_sg_for_device 4.12% [kernel] [k] ipv6_gro_receive 3.65% [kernel] [k] gq_pool_get 3.25% [kernel] [k] skb_gro_receive 2.07% [kernel] [k] napi_gro_frags 1.98% [kernel] [k] tcp6_gro_receive 1.27% [kernel] [k] gq_rx_prep_buffers 1.18% [kernel] [k] gq_rx_napi_handler 0.99% [kernel] [k] csum_partial 0.74% [kernel] [k] csum_ipv6_magic 0.72% [kernel] [k] free_pcp_prepare 0.60% [kernel] [k] __napi_poll 0.58% [kernel] [k] net_rx_action 0.56% [kernel] [k] read_tsc <*> 0.50% [kernel] [k] __x86_indirect_thunk_r11 0.45% [kernel] [k] memset After patch, lines with <*> no longer show up, and overall cpu usage looks much better (~60% instead of ~72%): 25.56% [kernel] [k] gq_rx_skb 9.90% [kernel] [k] napi_reuse_skb 7.39% [kernel] [k] dev_gro_receive 6.78% [kernel] [k] memcpy 6.53% [kernel] [k] skb_release_data 6.39% [kernel] [k] tcp_gro_receive 5.71% [kernel] [k] ipv6_gro_receive 4.35% [kernel] [k] napi_gro_frags 4.34% [kernel] [k] skb_gro_receive 3.50% [kernel] [k] gq_pool_get 3.08% [kernel] [k] gq_rx_napi_handler 2.35% [kernel] [k] tcp6_gro_receive 2.06% [kernel] [k] gq_rx_prep_buffers 1.32% [kernel] [k] csum_partial 0.93% [kernel] [k] csum_ipv6_magic 0.65% [kernel] [k] net_rx_action iavf yields +10% of Mpps on Rx. This also unblocks batched allocations of XSk buffers when IOMMU is active. Co-developed-by: Eric Dumazet Signed-off-by: Eric Dumazet Signed-off-by: Alexander Lobakin --- drivers/iommu/dma-iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 50ccc4f1ef81..4ab9ac13d362 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1707,7 +1707,8 @@ static size_t iommu_dma_opt_mapping_size(void) } static const struct dma_map_ops iommu_dma_ops = { - .flags = DMA_F_PCI_P2PDMA_SUPPORTED, + .flags = DMA_F_PCI_P2PDMA_SUPPORTED | + DMA_F_CAN_SKIP_SYNC, .alloc = iommu_dma_alloc, .free = iommu_dma_free, .alloc_pages = dma_common_alloc_pages, From patchwork Mon Feb 5 11:04:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196739 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp800723dyb; Mon, 5 Feb 2024 03:07:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IEuwMxGLMHsqlU1QtJbxr/Dqq1zDMfblgdDbBZ0wk6XC2hmbk4l5tPi23OYjoOcRcdExhZi X-Received: by 2002:a05:620a:1a0c:b0:785:3adb:f145 with SMTP id bk12-20020a05620a1a0c00b007853adbf145mr17515946qkb.57.1707131227296; Mon, 05 Feb 2024 03:07:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707131227; cv=pass; d=google.com; s=arc-20160816; b=JEYRZH20v9TnDwYDs/pAwlnc6Fxk9WfBnBYJtzobO3RHnNn6i9IuIc8jz2ZNEpbxPR Ouulr4m2n2QvBcgeNAQkx18sYeJDv1Zp+2SiuYd2s9hRqqO8EgPkLZohuxWwDsdcfFTM 7qqNSbGUCHjRcaL7VsKyryBiKBVHfBnSuE7SAYYZXMOzYR/7txkiGpr+MxLP0kU3H6i+ KlqmIz6Mj3DOJKWY5T+8rHQEwMKj1xoZLE3d21ejQ9uX9F2ptkaQeFYesGxYSlQCvD4t lAto8GSWJWCzfcX0jCOtycMKmLNv6qYLYvEvxNx+EQ55klqYULMk/j+WQNgakZKzm4/C KNVw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=L9I6zKyaecDW+yXld8UoYqEVvpbYMSKipidpIcy5GJs=; fh=hhyLtz0vZ9QD5d/CKxQgwFCSe7tamXfYwxAudrFaUPQ=; b=Z35SqllXVtg+M8eSHSFXaym+W2KHnEcZ+rxPW79PDOj4OjYMoiRM7lCHHYjfntg1hV X5RXOq9KPiARjPPWwgRjrMkMlVBh9tSvgbAne0LLwzSBLzj/qREiaUoXSOZUOmVNRBaV WjR6QC59vTd8sAiSzD+cQCUSeqEbJQjmzw8/g44Hmf2F1E3zTZEdeLvQY4g7fMrZWQrP aW+r2rfjUztR3P8Fq9D70OQr+OrJX7WO726ym3whi2sPadDqCHOOUK827+g+ZJy2GsyH gfYRTAAm1ueZwFBVP4fzpbQrUfQxwvzYZkmktliaLttX8P5U1euE0va3N9bPR4iVWvMU kF1Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=maTViyZH; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52452-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52452-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCW8ZVnwmvL08is5lqeEqSqAuVbo5qVSLdqquZe8qlbMjkZS4UIQT1f7svAtCzvBwaRGfCknGBxqrvysP7c5W8Gpf3A4eA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id u14-20020a05620a120e00b00783f8dfa491si8062015qkj.201.2024.02.05.03.07.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:07:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52452-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=maTViyZH; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52452-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52452-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 0D4C91C215B5 for ; Mon, 5 Feb 2024 11:07:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 878351B94B; Mon, 5 Feb 2024 11:05:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="maTViyZH" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8221E1B7F6; Mon, 5 Feb 2024 11:05:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131126; cv=none; b=khG+ZKnaBhdLoUiW1hXYDuEexZCO9qFAMv2NMtAYvOlNyAfhp704Gg67zuq6ZQ0Iv9WXNt6UnGiO2vxUj4098PaKfS8/I6qa+dhAvHsBGzU/Nq6hzET0q9VOs3a+lQXuEQUJb04K8j6HHv7TVNcEiQzFlfwAyxItKT9sQo1abaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131126; c=relaxed/simple; bh=vQ9pXrWvroiooYs41SwiRJil8dTyqvM8/gTLYDeQYJU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HSUYgWjVyhvvbxJA9CJiFDhP2dAR04PsM1QrPaBHCdpflZ54Bg/HGNDLU/xcthBwU8W7gSZkBABKetc9B9qs04wHo0sRxuYrRwVIbxzXVsXvlc+FWaVfp1JRL9wrAq2g9Waq2njQzSXhElqiJzpB9Sw8kqHAjqa8AMIKgPC9ZHs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=maTViyZH; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131124; x=1738667124; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vQ9pXrWvroiooYs41SwiRJil8dTyqvM8/gTLYDeQYJU=; b=maTViyZHIn0zQ3rqhi6Ur7JpQFkWl+HvKBmLCKtmOg/IfPqzAHC5oV3Z Tm/6T2L0evXKH9iXVtjxo5vxdKAWM/BhJfSZsaNHdflcRQYBO7ov1uzPe 6JdoxBaC0ib4rZGq4tLF4ws6ZJYM/1EEtrVMAQh/DEpC5SJxUTZlZZZi8 c9uW1GXdNq0hh9/r/AZd9AKd9XZGiJbEspNM2A3H34Pu+175MpDtxcydT j4pLS1jmAq6clp2aU5rJuGljKZrYtL/MhXc1l3MmRD+1HbGmpo7dQGnoH vEu8Odgd53Ht0w87wsalv+1RzPhUwFs3B2FiDAfmxN4pEbpZvAqxh+ZUm A==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945400" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945400" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328252" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:20 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 4/7] page_pool: make sure frag API fields don't span between cachelines Date: Mon, 5 Feb 2024 12:04:23 +0100 Message-ID: <20240205110426.764393-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790056834144274790 X-GMAIL-MSGID: 1790056834144274790 After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again. To not bother with this each time fast params get expanded or shrunk, let's just align them to `4 * sizeof(long)`, the closest upper pow-2 to their actual size (2 longs + 1 int). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing. ::page_state_hold_cnt is used quite intensively on hotpath no matter if frag API is used, so move it to the newly created hole in the first cacheline. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 12 +++++++++++- net/core/page_pool.c | 9 +++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 76481c465375..217e73b7e4fc 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -128,12 +128,22 @@ struct page_pool_stats { struct page_pool { struct page_pool_params_fast p; + u32 pages_state_hold_cnt; bool has_init_callback; + /* The following block must stay within one cacheline. On 32-bit + * systems, sizeof(long) == sizeof(int), so that the block size is + * ``3 * sizeof(long)``. On 64-bit systems, the actual size is + * ``2 * sizeof(long) + sizeof(int)``. The closest pow-2 to both of + * them is ``4 * sizeof(long)``, so just use that one for simplicity. + * Having it aligned to a cacheline boundary may be excessive and + * doesn't bring any good. + */ + __cacheline_group_begin(frag) __aligned(4 * sizeof(long)); long frag_users; struct page *frag_page; unsigned int frag_offset; - u32 pages_state_hold_cnt; + __cacheline_group_end(frag); struct delayed_work release_dw; void (*disconnect)(void *pool); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 4933762e5a6b..be1219816990 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -170,11 +170,20 @@ static void page_pool_producer_unlock(struct page_pool *pool, spin_unlock_bh(&pool->ring.producer_lock); } +static void page_pool_struct_check(void) +{ + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_users); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_page); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_offset); +} + static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params) { unsigned int ring_qsize = 1024; /* Default */ + page_pool_struct_check(); + memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow)); From patchwork Mon Feb 5 11:04:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196740 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp800990dyb; Mon, 5 Feb 2024 03:07:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IG6RRpacwVN5OJkAGzsdYWMq53Q2REeqWZZBg/EJxT9pvy23iZRLxj8npsD72smGVsll9Tu X-Received: by 2002:a0c:e34f:0:b0:68c:88fe:a40c with SMTP id a15-20020a0ce34f000000b0068c88fea40cmr5279018qvm.36.1707131256650; Mon, 05 Feb 2024 03:07:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707131256; cv=pass; d=google.com; s=arc-20160816; b=E42OWcac4BSeCRoJNyFoHTKhliJdPygRcD+1vB7yk5ofIG+lM0LkP4BKUQPvGy+OFi 65hYDDn3HJSFcl11er1K+Yd74cqdM0DFrZgC04L0Gl4yxu3L++BwS+1dbMxf96ogL1jc UG5ta7QcW0izBjr20Vv3MWqqa2J+Udf8Cm/5AVWfg90LL15jAHg2RitzZBMU5lb148oO 759Ei5G4mE777JeF1atOZRzPqER/Th2ObcvbbxihKKo2lhMOB2ooTMq1hbhLOIvr8yyj cSge9r+R5TR2zclhUuToRwNDsdvmaOBLcKTms027D+F5+HRLR1vSrcwda8UkUXUHgwfc XMoQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=HCgKfU4pneQXUQIp/h87D2YbNfsaDuqTvWBWySRCeEE=; fh=nIusjPXX8w/oYU05iwG5VjHAy+14xY3qle4ZaNCVcYw=; b=xlM6GW0noAD7VTbKPQj30v/gymqAUfCPR9DjAKA8OVUfnhRISihGUY3ckNGyd+5yfL 8PJat356CS4O9JZamesVMk+tbrEA0U550lmVdYg5ZgAaYSYChBCLZr8IbkG/kOzxuV8c sbXKetxVJNKz9jLWea6oq4/s2G4emdraeSbHyY3PUV8FmS5W2jmsU0pw6Ekhljb//cuZ ecU8TBOV+r4p/tc+je2/FEepUCO/m2/xmnpiRdwvuqbpby9oc2HHFKJ/TmhgDJwt7UAb eLOPMbHoROn2p8cp/wkOp8vUvfef1P5OnUpyK3HYHKA7HOAUyDdxnbGuVPsibm5rzq6l 68fw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DM6bXYFB; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52453-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52453-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCXPHY1MCVr3ql8DfzTh5fS2234UpmJvSkyhGhPiIjJu2RvCE6i5/30GuGY5AKMth3T9FL+LNq75mapt7Sz+MBTR35kEpg== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id a4-20020a0ce904000000b0068c78eccd4bsi7970281qvo.269.2024.02.05.03.07.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:07:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52453-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DM6bXYFB; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52453-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52453-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3EEB81C21A42 for ; Mon, 5 Feb 2024 11:07:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 54B591BC20; Mon, 5 Feb 2024 11:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="DM6bXYFB" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 975B01B940; Mon, 5 Feb 2024 11:05:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131130; cv=none; b=UeRPTa0OYcZUnHC5o6bggXNU8G19I/B8A2BS0fHGzn2klKKWYNMCFEXZRZYoqyAhMIEfCAz/H2wWoSG9brkotc+Zpm4yULoIcrr/Z4abYaq3y/heKPP2j17WZN/OSJ/ZHgDSnMDHEdrLYwfd7bwf5ijSAclLLA+NlbveSalKoSQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131130; c=relaxed/simple; bh=9vXQ5bF/Xi6PCf2WeVv071d/6ZrsHREf5IxWopzq0LY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RsiMPVwZmPcLqzwfGentRMtUZnne6NFijE+H9AUxXF3mUX1dURFpkiZp5pmr8TV4XxgfGKHlYvZDpGHv//85I4CYToQy7s7xM88NhxaoCCwyBPDv/RwUEXjLiTI5yuNAckJo/WKdQzFjOz0pDfttOtCDJgB/B5iTE4CyCAUYCYk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=DM6bXYFB; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131128; x=1738667128; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9vXQ5bF/Xi6PCf2WeVv071d/6ZrsHREf5IxWopzq0LY=; b=DM6bXYFBd02QtM7k/0sy0cuvfDEan3CLafRV+Yfqdb6WO5sBCJKSTzXO ywYDAFqtWC3fVjAoFAMHHTumQLO2rVvLtF6H6XpyhAavpGnslXS4B03hS nybMZOcR8phTX9bOgtZdwtBYp7gLlgKgp7pAAW+GUnI+OIUL9FyBVADgk YiBwE7dd6R2BTPjUR77Xh0tRAX0w875bwD0ZuGDWz3pDx485Sc1o278CC X4ytablxFdWTgF2WgpNyLG3DOzx4tNFQ02GET/4jVQ42qSkqZgdOKuBvi JuZTFnd1gBGjkEpqc48WxzlwCeKLNGurQWOcmCjp0SNuLBebK4XGSRMuI Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945465" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945465" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328255" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:24 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 5/7] page_pool: don't use driver-set flags field directly Date: Mon, 5 Feb 2024 12:04:24 +0100 Message-ID: <20240205110426.764393-6-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790056864460631158 X-GMAIL-MSGID: 1790056864460631158 page_pool::p is driver-defined params, copied directly from the structure passed to page_pool_create(). The structure isn't meant to be modified by the Page Pool core code and this even might look confusing[0][1]. In order to be able to alter some flags, let's define our own, internal fields the same way as the already existing one (::has_init_callback). They are defined as bits in the driver-set params, leave them so here as well, to not waste byte-per-bit or so. Almost 30 bits are still free for future extensions. We could've defined only new flags here or only the ones we may need to alter, but checking some flags in one place while others in another doesn't sound convenient or intuitive. ::flags passed by the driver can now go to the "slow" PP params. Suggested-by: Jakub Kicinski Link[0]: https://lore.kernel.org/netdev/20230703133207.4f0c54ce@kernel.org Suggested-by: Alexander Duyck Link[1]: https://lore.kernel.org/netdev/CAKgT0UfZCGnWgOH96E4GV3ZP6LLbROHM7SHE8NKwq+exX+Gk_Q@mail.gmail.com Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 9 ++++++--- net/core/page_pool.c | 34 ++++++++++++++++++---------------- 2 files changed, 24 insertions(+), 19 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 217e73b7e4fc..6a767ad1c572 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -44,7 +44,6 @@ struct pp_alloc_cache { /** * struct page_pool_params - page pool parameters - * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV * @order: 2^order pages on allocation * @pool_size: size of the ptr_ring * @nid: NUMA node id to allocate from pages from @@ -54,10 +53,10 @@ struct pp_alloc_cache { * @dma_dir: DMA mapping direction * @max_len: max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV * @offset: DMA sync address offset for PP_FLAG_DMA_SYNC_DEV + * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV */ struct page_pool_params { struct_group_tagged(page_pool_params_fast, fast, - unsigned int flags; unsigned int order; unsigned int pool_size; int nid; @@ -68,6 +67,7 @@ struct page_pool_params { unsigned int offset; ); struct_group_tagged(page_pool_params_slow, slow, + unsigned int flags; struct net_device *netdev; /* private: used by test code only */ void (*init_callback)(struct page *page, void *arg); @@ -129,7 +129,10 @@ struct page_pool { struct page_pool_params_fast p; u32 pages_state_hold_cnt; - bool has_init_callback; + + bool dma_map:1; /* Perform DMA mapping */ + bool dma_sync:1; /* Perform DMA sync */ + bool has_init_callback:1; /* slow.init_callback is set */ /* The following block must stay within one cacheline. On 32-bit * systems, sizeof(long) == sizeof(int), so that the block size is diff --git a/net/core/page_pool.c b/net/core/page_pool.c index be1219816990..2c353906407c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -188,7 +188,7 @@ static int page_pool_init(struct page_pool *pool, memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow)); /* Validate only known flags were used */ - if (pool->p.flags & ~(PP_FLAG_ALL)) + if (pool->slow.flags & ~(PP_FLAG_ALL)) return -EINVAL; if (pool->p.pool_size) @@ -202,22 +202,26 @@ static int page_pool_init(struct page_pool *pool, * DMA_BIDIRECTIONAL is for allowing page used for DMA sending, * which is the XDP_TX use-case. */ - if (pool->p.flags & PP_FLAG_DMA_MAP) { + if (pool->slow.flags & PP_FLAG_DMA_MAP) { if ((pool->p.dma_dir != DMA_FROM_DEVICE) && (pool->p.dma_dir != DMA_BIDIRECTIONAL)) return -EINVAL; + + pool->dma_map = true; } - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) { + if (pool->slow.flags & PP_FLAG_DMA_SYNC_DEV) { /* In order to request DMA-sync-for-device the page * needs to be mapped */ - if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + if (!(pool->slow.flags & PP_FLAG_DMA_MAP)) return -EINVAL; if (!pool->p.max_len) return -EINVAL; + pool->dma_sync = true; + /* pool->p.offset has to be set according to the address * offset used by the DMA engine to start copying rx data */ @@ -243,7 +247,7 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) get_device(pool->p.dev); return 0; @@ -253,7 +257,7 @@ static void page_pool_uninit(struct page_pool *pool) { ptr_ring_cleanup(&pool->ring, NULL); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) put_device(pool->p.dev); #ifdef CONFIG_PAGE_POOL_STATS @@ -396,7 +400,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) if (page_pool_set_dma_addr(page, dma)) goto unmap_failed; - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -442,8 +446,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, if (unlikely(!page)) return NULL; - if ((pool->p.flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); return NULL; } @@ -463,8 +466,8 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, gfp_t gfp) { const int bulk = PP_ALLOC_CACHE_REFILL; - unsigned int pp_flags = pool->p.flags; unsigned int pp_order = pool->p.order; + bool dma_map = pool->dma_map; struct page *page; int i, nr_pages; @@ -489,8 +492,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, */ for (i = 0; i < nr_pages; i++) { page = pool->alloc.cache[i]; - if ((pp_flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); continue; } @@ -562,7 +564,7 @@ void __page_pool_release_page_dma(struct page_pool *pool, struct page *page) { dma_addr_t dma; - if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + if (!pool->dma_map) /* Always account for inflight pages, even if we didn't * map them */ @@ -640,7 +642,7 @@ static bool page_pool_recycle_in_cache(struct page *page, } /* If the page refcnt == 1, this will try to recycle the page. - * if PP_FLAG_DMA_SYNC_DEV is set, we'll try to sync the DMA area for + * If pool->dma_sync is set, we'll try to sync the DMA area for * the configured size min(dma_sync_size, pool->max_len). * If the page refcnt != 1, then the page will be returned to memory * subsystem. @@ -663,7 +665,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, dma_sync_size); @@ -776,7 +778,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, -1); return page; From patchwork Mon Feb 5 11:04:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196753 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp813795dyb; Mon, 5 Feb 2024 03:36:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IGpKzy4leMauSoQtfibFIOKKDSHyc/qPZ1tiXrvzN2J9oww/TuVmFRSh8NX8/CEMfICa2B1 X-Received: by 2002:a05:6358:6e95:b0:176:9bfd:d08d with SMTP id q21-20020a0563586e9500b001769bfdd08dmr16265898rwm.11.1707133016319; Mon, 05 Feb 2024 03:36:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707133016; cv=pass; d=google.com; s=arc-20160816; b=rb0hP6h4DTg77vF4kQSuQLn6efXyioFgv1ZUOOXiauxGuKg7vhP8al6YYiW+IP+Iby n+5HmO22tzYbUXiwFRlhrRLHmE7UZ/dvy/Pm6S5j8yrQHO4QxRsZranH9wj1FULWCZhi PZKGgdvmk0rP1wcSGza3jVuGZmGu5agkk52TEVcuyagS0n9B9jtwk6wdpHALF4D87LXx ZtQVu1aTB6JvpXjdGSmSHqBnMAilOHAfTbdKuZYoj2cMgAZSsNh7IPo9axeOku3C2xQm Qi5lUK7K8gg1FNavWCFCzgr/owl8tDsbj52QnUK0hpHBk7SmJX33m3+Y/D2I37lwpDWx Bd6g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=AV/c1cWWIWUkdT8pnihfHEAqVlIV0cM5qidI+eui4fg=; fh=fg+zuvzw4pM0GbH5VG/X8QpDscjf+hwjV4TDfqhKdKA=; b=t2JFK899+6iFOeT8P4nbLitXhTatCeAS7VyYhUwiQ0Xd99VEwwOmS8hgaSlcUaB+LD mODJoa/3GasSZbgh0Kj2yb0L9R0e2uRxJ2KQIK6gxpLEivevkLCEWtU/fF2MMrdESCkb C7itXipWK63PQw4+UBStQeOf51nftJXvrE+QlMDMMPk4obeEcBjyxxg2ZM3MX3HouEAq tQ7JXkY4DHmh2JxZNQM5jhbroPYqk4o5krIT4SCRk9f5/OdDKPnzZ7KqPvUH7W31L/+g wH+xUT/n4T4iR6E8REGPs6Cxk6lnuIh2xRmKJCEcWJAvkoPukQymgfWCowPQ7i/Nx0rQ /+PA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkWhEaVb; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCW4vvEsfdn9tbv3jNaw7w5UPXfT+7OBDhTbvqQOV0SSSX53jQE9m+2BH5qMd8OJIxfID+lgigu5FuIZGIcQSn61AzmV7g== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id s71-20020a63774a000000b005dbf1fcc429si3387005pgc.162.2024.02.05.03.36.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:36:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkWhEaVb; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 2E896B28726 for ; Mon, 5 Feb 2024 11:07:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C9F971BDCD; Mon, 5 Feb 2024 11:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SkWhEaVb" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D3931B964; Mon, 5 Feb 2024 11:05:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131134; cv=none; b=pmkpeQq7Juq41QBOsGWI3lysJAB7eGhWmAbdVYWrtaH97PC/seyEJ1wB2i4THVN+QMDSHiiMhSXWnBq38lOxR1dhF0Jp8wtYr+SxRIposH23la+OKzF6rKjKNh7uveFFDO0J38OebDZJseBzcTDeSUz0u+TwUKXaX6LzCITwiBA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131134; c=relaxed/simple; bh=j5bCxYGboU2nsbzGInCNZJ3ihY7kPTLpx8cHouNYyPI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SRyCQQwbQmnMgTFW+xxh4FZVKTht6/AO26MlNqK1JkD9Me/0x8B731uuHhObE+T0H07D/cm4AgNPRi+5WOkwfO6rU1b1ldujSbh4kO0cQ6YPMrEGkswOb3kbUBjTQPHMtBgOhQ1w1SviHnh66fKelX+3lXE/HgOfQHO+HZ2dcT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SkWhEaVb; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131132; x=1738667132; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j5bCxYGboU2nsbzGInCNZJ3ihY7kPTLpx8cHouNYyPI=; b=SkWhEaVbVUJyrNvLim5AORJbEsuJnWCH3c1X+xCOwSHdIKQRc0v4kykV ZxIAOES6AnOJKygByB26fiSNJecg4NQN9NXeGhb2ZttJ7xgTMmAjAYpIs kOGxXjYnOYTh0J3k8j0v7qL5JnvPfHPYOBqgT2EwvU9ZnO+giCN1lRroM GcMV5dFBXwxnCo8jGiq6C9oYB6HnMhKz6ZFfoBgfcL4fxT2Cz/cvGq/Th A5KC65CWGhuuuHku1+4QONXGS5jkNaauyGaKErNCq8pwn6Y9aZWsirey/ FRXEFsXJHvieYhtF99qGLd2LrMGTP86sTiv5rVVJOo/7JjcW5Mtaaq1bh A==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945508" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945508" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328262" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:28 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 6/7] page_pool: check for DMA sync shortcut earlier Date: Mon, 5 Feb 2024 12:04:25 +0100 Message-ID: <20240205110426.764393-7-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790058709631276367 X-GMAIL-MSGID: 1790058709631276367 We can save a couple more function calls in the Page Pool code if we check for dma_skip_sync() earlier, just when we test pp->p.dma_sync. Move both these checks into an inline wrapper and call the PP wrapper over the generic DMA sync function only when both are true. You can't cache the result of dma_skip_sync() in &page_pool, as it may change anytime if an SWIOTLB buffer is allocated or mapped. Signed-off-by: Alexander Lobakin --- net/core/page_pool.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 2c353906407c..d2b411095131 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -369,16 +369,24 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) return page; } -static void page_pool_dma_sync_for_device(struct page_pool *pool, - struct page *page, - unsigned int dma_sync_size) +static void __page_pool_dma_sync_for_device(struct page_pool *pool, + struct page *page, + u32 dma_sync_size) { dma_addr_t dma_addr = page_pool_get_dma_addr(page); dma_sync_size = min(dma_sync_size, pool->p.max_len); - dma_sync_single_range_for_device(pool->p.dev, dma_addr, - pool->p.offset, dma_sync_size, - pool->p.dma_dir); + __dma_sync_single_range_for_device(pool->p.dev, dma_addr, + pool->p.offset, dma_sync_size, + pool->p.dma_dir); +} + +static __always_inline void +page_pool_dma_sync_for_device(struct page_pool *pool, struct page *page, + u32 dma_sync_size) +{ + if (pool->dma_sync && !dma_skip_sync(pool->p.dev)) + __page_pool_dma_sync_for_device(pool, page, dma_sync_size); } static bool page_pool_dma_map(struct page_pool *pool, struct page *page) @@ -400,8 +408,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) if (page_pool_set_dma_addr(page, dma)) goto unmap_failed; - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, pool->p.max_len); + page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -665,9 +672,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, - dma_sync_size); + page_pool_dma_sync_for_device(pool, page, dma_sync_size); if (allow_direct && in_softirq() && page_pool_recycle_in_cache(page, pool)) @@ -778,8 +783,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, -1); + page_pool_dma_sync_for_device(pool, page, -1); return page; } From patchwork Mon Feb 5 11:04:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196741 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp801392dyb; Mon, 5 Feb 2024 03:08:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IFF6WX14mqs88JrsCI/ZxW7RiXuZF5/F7a8uQW4vzmGd3j2aTtlEU9z+F37+eEfEMBrrVGw X-Received: by 2002:aa7:d854:0:b0:560:d8f:7def with SMTP id f20-20020aa7d854000000b005600d8f7defmr5077155eds.17.1707131301642; Mon, 05 Feb 2024 03:08:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707131301; cv=pass; d=google.com; s=arc-20160816; b=lSXaSjOv9HcGq5H+IU+r9fIbM/vUL+IQxbI0PyyXhBm3suppOstOeiZ1uLA/Gh1mV/ Wdun+a7OoemepHJBCQo3SnGiAsm7aDQD3/JK+MBlzUIXpkJuJulN6T3giVG16c6MmuPY nQLwoZ7/aSwM4rg+6q9FqYtfQ2ICOWQG85z0OeEmJzFoTrNgRRhaZ/HN244sLZpEFsXw QifAC7Lpr01izE37h7DcNdMhyVyd9ppnHzgRAd1HwbJoqYXnX41ofES7RlZlIVTnxq4C w/zp75SLLqgwJGas7U+4J5aEKiPGtnFKzVqmcWoiCWj5GtQxnvpbK265J/yqWSXGq0cr 29Kw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=IFzDX2HQAJruh+mHNSYv9kEiAHSP9xPgg+Pn5oMrkzQ=; fh=xdErQTDCNJlVbppbgxxXR2dKN/hn1oVGuEeKHB1RVtg=; b=RW5FxzPjRx7ED14aks5FtCDE6hbMe58+7elqGteUnX+81wCUmo7tBXFe2tjWYxljzw 2V/+zxTZKzZl38XWt52FoFI4B0608w7TZ62WgO4iH9SXvNsyZa1c/Q/Bmmnkztdebrk7 edqanWYGAlT5ckSOwLHpbN3kq1EDrXAPtMuFD1qjDPawlYr13NGrq/cB6pczMMV+wH6j SlLT0VSYQ4QSba4yQLo8E5iHXN3/UJh2zzl3DGc0mPe4uRPzk6fzTjDf9ny23dQuWRox eCyG01Zp/hIOHrUXTPYHmVQZY7DbW518k8NMZilaah1iks9AqWWFvl+NE5PKBYx8xJAq oRUg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=cIXI3KOs; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52455-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52455-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCV4ZhIfCDdKQlCh3jox3CdJVbqGcaGaAFmZvPon1USaz9gnKEkNz0mofJm8C/zb94B56RSx5Xv9eRkqefriYhHOkVfZxw== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id f3-20020aa7d843000000b0056012ce9cfasi2912556eds.555.2024.02.05.03.08.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:08:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52455-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=cIXI3KOs; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52455-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52455-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 19C1F1F225A4 for ; Mon, 5 Feb 2024 11:08:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1F4C81BF32; Mon, 5 Feb 2024 11:05:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="cIXI3KOs" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A471D1BDC9; Mon, 5 Feb 2024 11:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131138; cv=none; b=mqjHlPFNFtEwfLuwp06467KUIl+e84oBlLKyzE7FPDzzzgerW9cW8hzP7z9ODOzCieGv6SBkvZpejlpJ7CsdW6El5+xHwgdfg7fe4nby7BW48dpPdPhhTSwUrUrjKifZbNTo58vjAThHo7NdLgi4lGLBhd3u5NShhwu/JaN6ieM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131138; c=relaxed/simple; bh=+fP5W4+oVndeU76RM6P85LN5H2O/z5jAuq/WWrmDgwA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Yhqnd101igoa/fxv2QPJVxCGRzP2tW2RYQJT5wUDkEvOhFqjUWBKKJ7htR616J/MWLzLG5zu4jPD5KP4+p++XNz2nIJ4isKORWrO911GH0aBXcc1Wn5rDWSrae1A6KkKT3mqREoBp/AswvS8G2TPOvIYX+MRVK7Gx32CNqOTNCY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=cIXI3KOs; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131136; x=1738667136; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+fP5W4+oVndeU76RM6P85LN5H2O/z5jAuq/WWrmDgwA=; b=cIXI3KOsefVJHoxvjGc0omBc4d2fCHm/Evlcjs5hl3LF1NEgUWyYGOgu 8ZpDd0G99bf84KJ3JGR4vrcQ5wtjP/an8CQ7eL/I7EjnKk7R+l2zaHOew aq+Vb+DQb5UpqttcK2h28LKfaVwX0zWhce8nvj+zrBH3oyACEsZ1uOMSV cNDv1Bq/FIeyBau4oVXcOZDOr30Wk44pF+JgxweWpKpnR3hVMWqFrZcXn m/uNwALlut6wFuk+FR0Pax66antvWophS4hOKnWSUp2D3KauGAXA+dmE4 CYHYJQaBsiPGHba/Cz2NwS8NL75H5hhl7BS/2Tk/8EMGGdo/Ekke00UY3 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945535" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945535" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328271" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:32 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 7/7] xsk: use generic DMA sync shortcut instead of a custom one Date: Mon, 5 Feb 2024 12:04:26 +0100 Message-ID: <20240205110426.764393-8-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790056911767449946 X-GMAIL-MSGID: 1790056911767449946 XSk infra's been using its own DMA sync shortcut to try avoiding redundant function calls. Now that there is a generic one, remove the custom implementation and rely on the generic helpers. xsk_buff_dma_sync_for_cpu() doesn't need the second argument anymore, remove it. Signed-off-by: Alexander Lobakin --- include/net/xdp_sock_drv.h | 7 ++--- include/net/xsk_buff_pool.h | 13 ++------- drivers/net/ethernet/engleder/tsnep_main.c | 2 +- .../net/ethernet/freescale/dpaa2/dpaa2-xsk.c | 2 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +- drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +- drivers/net/ethernet/intel/igc/igc_main.c | 2 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 4 +-- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 2 +- drivers/net/ethernet/netronome/nfp/nfd3/xsk.c | 2 +- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 2 +- net/xdp/xsk_buff_pool.c | 29 +++---------------- 13 files changed, 20 insertions(+), 51 deletions(-) diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h index c9aec9ab6191..0a5dca2b2b3f 100644 --- a/include/net/xdp_sock_drv.h +++ b/include/net/xdp_sock_drv.h @@ -219,13 +219,10 @@ static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool return meta; } -static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool) +static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp) { struct xdp_buff_xsk *xskb = container_of(xdp, struct xdp_buff_xsk, xdp); - if (!pool->dma_need_sync) - return; - xp_dma_sync_for_cpu(xskb); } @@ -402,7 +399,7 @@ static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool return NULL; } -static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool) +static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp) { } diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h index 99dd7376df6a..b61e787a0ee5 100644 --- a/include/net/xsk_buff_pool.h +++ b/include/net/xsk_buff_pool.h @@ -43,7 +43,6 @@ struct xsk_dma_map { refcount_t users; struct list_head list; /* Protected by the RTNL_LOCK */ u32 dma_pages_cnt; - bool dma_need_sync; }; struct xsk_buff_pool { @@ -82,7 +81,6 @@ struct xsk_buff_pool { u8 tx_metadata_len; /* inherited from umem */ u8 cached_need_wakeup; bool uses_need_wakeup; - bool dma_need_sync; bool unaligned; bool tx_sw_csum; void *addrs; @@ -155,21 +153,16 @@ static inline dma_addr_t xp_get_frame_dma(struct xdp_buff_xsk *xskb) return xskb->frame_dma; } -void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb); static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb) { - xp_dma_sync_for_cpu_slow(xskb); + dma_sync_single_for_cpu(xskb->pool->dev, xskb->dma, + xskb->pool->frame_len, DMA_BIDIRECTIONAL); } -void xp_dma_sync_for_device_slow(struct xsk_buff_pool *pool, dma_addr_t dma, - size_t size); static inline void xp_dma_sync_for_device(struct xsk_buff_pool *pool, dma_addr_t dma, size_t size) { - if (!pool->dma_need_sync) - return; - - xp_dma_sync_for_device_slow(pool, dma, size); + dma_sync_single_for_device(pool->dev, dma, size, DMA_BIDIRECTIONAL); } /* Masks for xdp_umem_page flags. diff --git a/drivers/net/ethernet/engleder/tsnep_main.c b/drivers/net/ethernet/engleder/tsnep_main.c index eb64118f5b18..2e5ccacb78b9 100644 --- a/drivers/net/ethernet/engleder/tsnep_main.c +++ b/drivers/net/ethernet/engleder/tsnep_main.c @@ -1571,7 +1571,7 @@ static int tsnep_rx_poll_zc(struct tsnep_rx *rx, struct napi_struct *napi, length = __le32_to_cpu(entry->desc_wb->properties) & TSNEP_DESC_LENGTH_MASK; xsk_buff_set_size(entry->xdp, length - ETH_FCS_LEN); - xsk_buff_dma_sync_for_cpu(entry->xdp, rx->xsk_pool); + xsk_buff_dma_sync_for_cpu(entry->xdp); /* RX metadata with timestamps is in front of actual data, * subtract metadata size to get length of actual data and diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c index 051748b997f3..a466c2379146 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c @@ -55,7 +55,7 @@ static u32 dpaa2_xsk_run_xdp(struct dpaa2_eth_priv *priv, xdp_set_data_meta_invalid(xdp_buff); xdp_buff->rxq = &ch->xdp_rxq; - xsk_buff_dma_sync_for_cpu(xdp_buff, ch->xsk_pool); + xsk_buff_dma_sync_for_cpu(xdp_buff); xdp_act = bpf_prog_run_xdp(xdp_prog, xdp_buff); /* xdp.data pointer may have changed */ diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 11500003af0d..d20ce517426e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -483,7 +483,7 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) bi = *i40e_rx_bi(rx_ring, next_to_process); xsk_buff_set_size(bi, size); - xsk_buff_dma_sync_for_cpu(bi, rx_ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi); if (!first) first = bi; diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c index 8b81a1677045..5d4aabf7e1b1 100644 --- a/drivers/net/ethernet/intel/ice/ice_xsk.c +++ b/drivers/net/ethernet/intel/ice/ice_xsk.c @@ -892,7 +892,7 @@ int ice_clean_rx_irq_zc(struct ice_rx_ring *rx_ring, int budget) ICE_RX_FLX_DESC_PKT_LEN_M; xsk_buff_set_size(xdp, size); - xsk_buff_dma_sync_for_cpu(xdp, xsk_pool); + xsk_buff_dma_sync_for_cpu(xdp); if (!first) { first = xdp; diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index ba8d3fe186ae..ad9ebbd9d61d 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -2817,7 +2817,7 @@ static int igc_clean_rx_irq_zc(struct igc_q_vector *q_vector, const int budget) } bi->xdp->data_end = bi->xdp->data + size; - xsk_buff_dma_sync_for_cpu(bi->xdp, ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi->xdp); res = __igc_xdp_run_prog(adapter, prog, bi->xdp); switch (res) { diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c index 59798bc33298..ebda0cebe910 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c @@ -304,7 +304,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, } bi->xdp->data_end = bi->xdp->data + size; - xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi->xdp); xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp); if (likely(xdp_res & (IXGBE_XDP_TX | IXGBE_XDP_REDIR))) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c index b8dd74453655..1b7132fa70de 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c @@ -270,7 +270,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, /* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ mxbuf->cqe = cqe; xsk_buff_set_size(&mxbuf->xdp, cqe_bcnt); - xsk_buff_dma_sync_for_cpu(&mxbuf->xdp, rq->xsk_pool); + xsk_buff_dma_sync_for_cpu(&mxbuf->xdp); net_prefetch(mxbuf->xdp.data); /* Possible flows: @@ -319,7 +319,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq, /* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ mxbuf->cqe = cqe; xsk_buff_set_size(&mxbuf->xdp, cqe_bcnt); - xsk_buff_dma_sync_for_cpu(&mxbuf->xdp, rq->xsk_pool); + xsk_buff_dma_sync_for_cpu(&mxbuf->xdp); net_prefetch(mxbuf->xdp.data); prog = rcu_dereference(rq->xdp_prog); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index d601b5faaed5..5e5d9fd0bfd5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -917,7 +917,7 @@ INDIRECT_CALLABLE_SCOPE bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq) if (!rq->xsk_pool) { count = mlx5e_refill_rx_wqes(rq, head, wqe_bulk); - } else if (likely(!rq->xsk_pool->dma_need_sync)) { + } else if (likely(dma_skip_sync(rq->pdev))) { mlx5e_xsk_free_rx_wqes(rq, head, wqe_bulk); count = mlx5e_xsk_alloc_rx_wqes_batched(rq, head, wqe_bulk); } else { diff --git a/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c b/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c index 45be6954d5aa..01cfa9cc1b5e 100644 --- a/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c +++ b/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c @@ -184,7 +184,7 @@ nfp_nfd3_xsk_rx(struct nfp_net_rx_ring *rx_ring, int budget, xrxbuf->xdp->data += meta_len; xrxbuf->xdp->data_end = xrxbuf->xdp->data + pkt_len; xdp_set_data_meta_invalid(xrxbuf->xdp); - xsk_buff_dma_sync_for_cpu(xrxbuf->xdp, r_vec->xsk_pool); + xsk_buff_dma_sync_for_cpu(xrxbuf->xdp); net_prefetch(xrxbuf->xdp->data); if (meta_len) { diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 04d817dc5899..a280dfa8420d 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -5335,7 +5335,7 @@ static int stmmac_rx_zc(struct stmmac_priv *priv, int limit, u32 queue) /* RX buffer is good and fit into a XSK pool buffer */ buf->xdp->data_end = buf->xdp->data + buf1_len; - xsk_buff_dma_sync_for_cpu(buf->xdp, rx_q->xsk_pool); + xsk_buff_dma_sync_for_cpu(buf->xdp); prog = READ_ONCE(priv->xdp_prog); res = __stmmac_xdp_run_prog(priv, prog, buf->xdp); diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index ce60ecd48a4d..ecea2a329b1d 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -338,7 +338,6 @@ static struct xsk_dma_map *xp_create_dma_map(struct device *dev, struct net_devi dma_map->netdev = netdev; dma_map->dev = dev; - dma_map->dma_need_sync = false; dma_map->dma_pages_cnt = nr_pages; refcount_set(&dma_map->users, 1); list_add(&dma_map->list, &umem->xsk_dma_list); @@ -424,7 +423,6 @@ static int xp_init_dma_info(struct xsk_buff_pool *pool, struct xsk_dma_map *dma_ pool->dev = dma_map->dev; pool->dma_pages_cnt = dma_map->dma_pages_cnt; - pool->dma_need_sync = dma_map->dma_need_sync; memcpy(pool->dma_pages, dma_map->dma_pages, pool->dma_pages_cnt * sizeof(*pool->dma_pages)); @@ -460,8 +458,6 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev, __xp_dma_unmap(dma_map, attrs); return -ENOMEM; } - if (dma_need_sync(dev, dma)) - dma_map->dma_need_sync = true; dma_map->dma_pages[i] = dma; } @@ -557,11 +553,9 @@ struct xdp_buff *xp_alloc(struct xsk_buff_pool *pool) xskb->xdp.data_meta = xskb->xdp.data; xskb->xdp.flags = 0; - if (pool->dma_need_sync) { - dma_sync_single_range_for_device(pool->dev, xskb->dma, 0, - pool->frame_len, - DMA_BIDIRECTIONAL); - } + dma_sync_single_for_device(pool->dev, xskb->dma, pool->frame_len, + DMA_BIDIRECTIONAL); + return &xskb->xdp; } EXPORT_SYMBOL(xp_alloc); @@ -633,7 +627,7 @@ u32 xp_alloc_batch(struct xsk_buff_pool *pool, struct xdp_buff **xdp, u32 max) { u32 nb_entries1 = 0, nb_entries2; - if (unlikely(pool->dma_need_sync)) { + if (unlikely(!dma_skip_sync(pool->dev))) { struct xdp_buff *buff; /* Slow path */ @@ -693,18 +687,3 @@ dma_addr_t xp_raw_get_dma(struct xsk_buff_pool *pool, u64 addr) (addr & ~PAGE_MASK); } EXPORT_SYMBOL(xp_raw_get_dma); - -void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb) -{ - dma_sync_single_range_for_cpu(xskb->pool->dev, xskb->dma, 0, - xskb->pool->frame_len, DMA_BIDIRECTIONAL); -} -EXPORT_SYMBOL(xp_dma_sync_for_cpu_slow); - -void xp_dma_sync_for_device_slow(struct xsk_buff_pool *pool, dma_addr_t dma, - size_t size) -{ - dma_sync_single_range_for_device(pool->dev, dma, 0, - size, DMA_BIDIRECTIONAL); -} -EXPORT_SYMBOL(xp_dma_sync_for_device_slow);