From patchwork Wed Feb 14 16:21:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201041 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1346724dyb; Wed, 14 Feb 2024 08:48:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVCtDNTxSEuw/aq+/2w0pTDA/7NLLq+sGYVleF3zq6D76g0ZTq86PfsJ/W/UUycO1OOMZKEsjBwlmC+WLdg5o2wvKHPNQ== X-Google-Smtp-Source: AGHT+IGCz9F9OpvH8oOFlsYLtLdKrW72gjAj38irhyXgf3eT9OUQuA8k4K/h/WRV7rgsncjqZieT X-Received: by 2002:a05:6a21:1789:b0:19e:a1ae:7f14 with SMTP id nx9-20020a056a21178900b0019ea1ae7f14mr4073301pzb.4.1707929320301; Wed, 14 Feb 2024 08:48:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707929320; cv=pass; d=google.com; s=arc-20160816; b=1Kj0H9MC3tjeLCFsmSo+VBtJNNNuF3pli6O0ZMZOMMd+VkDolie5HqEEUvSN+Vm9iY IuWCi+W6xTv26VRQuXLt9Cg9MmhhchLVVLJcJIR28eMVJG6hAgucRxs7zNtqDvz44Pza /7YKdnEbuL0qGECIU42CsJ2dDpQm/qR+DuzoL1kZqfYpcC9ky+u184O5wG0T3DSFwxix uZOMhGIs5aJPoUOmRYWDEWFqPxnPNZcbK4t1jmin//o5FXXF6gZBxCV/EIbiLMivXqLV bS4pHMa92nfW32ZpFktZdjGPtYwXFqvZlFZYMSGV0fjOXLMXvjOA8r72seNQz7eeMVei X9MQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=CvQ0P29OuW2rz1RNNSIiQxzhYpNErH9WsYtLDnbf62Q=; fh=pEj5nAGAu9D5qykFvsCQ5+o4kolXYmnnTHqzD0F4jDk=; b=wvh9Q/5ApF+h2IvreURZrTrJxZic2Gno+zSDkjxN6RBlpN6qBy9R1OzurQjsGXptfu aIxusudEYJhGcaSOFljJ7CZ2Mnn98k0fvUDqjdRMOAn1TMoqO9JI076LEeylT8x3LtIO tFGnn1midgn24q0E4PFLoGr9Qrgwjrzz2t8b0ySCeMTLMJ4/KdtyAMhRaLbbc9jrw3zz SbFHQ407mmkEoTLAX7W7qagedN8qeyqnyZwR9f+5CYkof5jzVB4DUQ9Vrw7f7AvFabne AsRv4eb/OUSsSSEDhKecz6u2jv96Dpnq9LpN+dlfcBV8RPxfFeCYCzCCZggE/XTWtNhC 5KZg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=afDpTSMW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65512-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCUY3tomueKRzmQJglMKZw1hKJdqY87zLUzMZRED2SBMf5BPewA+Nehh7PysKSN6t2PkIQxOKO72C8V08R0rB1ryExphLw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id l5-20020a656805000000b005d8b49c9b43si4092459pgt.693.2024.02.14.08.48.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:48:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=afDpTSMW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65512-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65512-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 7B3DFB28C86 for ; Wed, 14 Feb 2024 16:23:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 81D9E60BA5; Wed, 14 Feb 2024 16:22:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="afDpTSMW" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 941D06024D; Wed, 14 Feb 2024 16:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927768; cv=none; b=dk+jIYamffrEOE8a9Hbc4tFe/0eZw4OpdSWnbM0GM6L2k05289+ayuY2TTgD/e/KhO6NaoSVm45N3YKsxncN9D/e80Tl+MA12m+98/V+XWm+44UFUXpTHyFwjz947UCC3JiT/6qPyA8eadH5XL/lwKtAsWXQTdl+n3y+aVVkR44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927768; c=relaxed/simple; bh=XjQnDnhe7CydUIm6so/zShzYAWo2nbePKGaBcANZkSc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N8oEuemFelGj2ZInsLXyoQTerWjl/MwqQ3smTVoVGCe1mzM/QW5kbg3+a7pGHTM3lrlEGUTS//YCR7D1R2Irmp6xbB+YnVGe/GbIbBHbsf9l8Ylt+VNsWo93QEQP0SlqOHphuIgaXBCjXzLsn8jD/fasRIaPdPcNkNASRA/IxNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=afDpTSMW; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927766; x=1739463766; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XjQnDnhe7CydUIm6so/zShzYAWo2nbePKGaBcANZkSc=; b=afDpTSMWf8rsPpJ2L7tE/eRGlSEBqg0t5rBne79yfPbR8z6nC2/XtcxG 0173RMUiejIJ8S/E7t/d77X9WMIySJiBTQKLMED2ZRGxPCZ8Nypm6tVO7 prypJ37G+79vFNT27R0m453BRmepjnY5Jmko6F4AwvIwct9pLR3NYGeoM 0AnHIvaRnZLTMn8Dn8VtiEBeRForzW+HnSGwK2Zg451Xanq6x/WXpyTmz Z520PxiL8wCnsOn+aefgU3dFGlRfIqCSqzTgk9/jelpVKB1QM1Bldl8s9 tkstuhAOx1HkCZnORNj+WMGfXpMaXmbrzTtpvs+YcPFN6iiAikpcMdOko Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755532" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755532" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:22:46 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26399959" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:42 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 1/7] dma: compile-out DMA sync op calls when not used Date: Wed, 14 Feb 2024 17:21:55 +0100 Message-ID: <20240214162201.4168778-2-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790893694998907749 X-GMAIL-MSGID: 1790893694998907749 Some platforms do have DMA, but DMA there is always direct and coherent. Currently, even on such platforms DMA sync operations are compiled and called. Add a new hidden Kconfig symbol, DMA_NEED_SYNC, and set it only when either sync operations are needed or there is DMA ops or swiotlb enabled. Set dma_need_sync() and dma_skip_sync() depending on this symbol state and don't call sync ops when dma_skip_sync() is true. The change allows for future optimizations of DMA sync calls depending on compile-time or runtime conditions. Signed-off-by: Alexander Lobakin --- kernel/dma/Kconfig | 4 ++ include/linux/dma-mapping.h | 80 +++++++++++++++++++++++++++++++------ kernel/dma/mapping.c | 20 +++++----- 3 files changed, 81 insertions(+), 23 deletions(-) diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index d62f5957f36b..1c9ff05b1ecb 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -107,6 +107,10 @@ config DMA_BOUNCE_UNALIGNED_KMALLOC bool depends on SWIOTLB +config DMA_NEED_SYNC + def_bool ARCH_HAS_SYNC_DMA_FOR_DEVICE || ARCH_HAS_SYNC_DMA_FOR_CPU || \ + ARCH_HAS_SYNC_DMA_FOR_CPU_ALL || DMA_OPS || SWIOTLB + config DMA_RESTRICTED_POOL bool "DMA Restricted Pool" depends on OF && OF_RESERVED_MEM && SWIOTLB diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 4a658de44ee9..6c7640441214 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -117,13 +117,13 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); -void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, +void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, +void __dma_sync_single_for_device(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, +void __dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir); -void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, +void __dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir); void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs); @@ -147,7 +147,7 @@ u64 dma_get_required_mask(struct device *dev); bool dma_addressing_limited(struct device *dev); size_t dma_max_mapping_size(struct device *dev); size_t dma_opt_mapping_size(struct device *dev); -bool dma_need_sync(struct device *dev, dma_addr_t dma_addr); +bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr); unsigned long dma_get_merge_boundary(struct device *dev); struct sg_table *dma_alloc_noncontiguous(struct device *dev, size_t size, enum dma_data_direction dir, gfp_t gfp, unsigned long attrs); @@ -195,19 +195,19 @@ static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) { } -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) +static inline void __dma_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) { } -static inline void dma_sync_single_for_device(struct device *dev, +static inline void __dma_sync_single_for_device(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { } -static inline void dma_sync_sg_for_cpu(struct device *dev, +static inline void __dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir) { } -static inline void dma_sync_sg_for_device(struct device *dev, +static inline void __dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir) { } @@ -277,7 +277,7 @@ static inline size_t dma_opt_mapping_size(struct device *dev) { return 0; } -static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +static inline bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) { return false; } @@ -348,18 +348,72 @@ static inline void dma_unmap_single_attrs(struct device *dev, dma_addr_t addr, return dma_unmap_page_attrs(dev, addr, size, dir, attrs); } +static inline void __dma_sync_single_range_for_cpu(struct device *dev, + dma_addr_t addr, unsigned long offset, size_t size, + enum dma_data_direction dir) +{ + __dma_sync_single_for_cpu(dev, addr + offset, size, dir); +} + +static inline void __dma_sync_single_range_for_device(struct device *dev, + dma_addr_t addr, unsigned long offset, size_t size, + enum dma_data_direction dir) +{ + __dma_sync_single_for_device(dev, addr + offset, size, dir); +} + +static inline bool dma_skip_sync(const struct device *dev) +{ + return !IS_ENABLED(CONFIG_DMA_NEED_SYNC); +} + +static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +{ + return !dma_skip_sync(dev) ? __dma_need_sync(dev, dma_addr) : false; +} + +static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_single_for_cpu(dev, addr, size, dir); +} + +static inline void dma_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_single_for_device(dev, addr, size, dir); +} + +static inline void dma_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sg, int nelems, enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_sg_for_cpu(dev, sg, nelems, dir); +} + +static inline void dma_sync_sg_for_device(struct device *dev, + struct scatterlist *sg, int nelems, enum dma_data_direction dir) +{ + if (!dma_skip_sync(dev)) + __dma_sync_sg_for_device(dev, sg, nelems, dir); +} + static inline void dma_sync_single_range_for_cpu(struct device *dev, dma_addr_t addr, unsigned long offset, size_t size, enum dma_data_direction dir) { - return dma_sync_single_for_cpu(dev, addr + offset, size, dir); + if (!dma_skip_sync(dev)) + __dma_sync_single_for_cpu(dev, addr + offset, size, dir); } static inline void dma_sync_single_range_for_device(struct device *dev, dma_addr_t addr, unsigned long offset, size_t size, enum dma_data_direction dir) { - return dma_sync_single_for_device(dev, addr + offset, size, dir); + if (!dma_skip_sync(dev)) + __dma_sync_single_for_device(dev, addr + offset, size, dir); } /** diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..85feaa0e008c 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -329,7 +329,7 @@ void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, } EXPORT_SYMBOL(dma_unmap_resource); -void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, +void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -341,9 +341,9 @@ void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, ops->sync_single_for_cpu(dev, addr, size, dir); debug_dma_sync_single_for_cpu(dev, addr, size, dir); } -EXPORT_SYMBOL(dma_sync_single_for_cpu); +EXPORT_SYMBOL(__dma_sync_single_for_cpu); -void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, +void __dma_sync_single_for_device(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -355,9 +355,9 @@ void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, ops->sync_single_for_device(dev, addr, size, dir); debug_dma_sync_single_for_device(dev, addr, size, dir); } -EXPORT_SYMBOL(dma_sync_single_for_device); +EXPORT_SYMBOL(__dma_sync_single_for_device); -void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, +void __dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -369,9 +369,9 @@ void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, ops->sync_sg_for_cpu(dev, sg, nelems, dir); debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); } -EXPORT_SYMBOL(dma_sync_sg_for_cpu); +EXPORT_SYMBOL(__dma_sync_sg_for_cpu); -void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, +void __dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, int nelems, enum dma_data_direction dir) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -383,7 +383,7 @@ void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, ops->sync_sg_for_device(dev, sg, nelems, dir); debug_dma_sync_sg_for_device(dev, sg, nelems, dir); } -EXPORT_SYMBOL(dma_sync_sg_for_device); +EXPORT_SYMBOL(__dma_sync_sg_for_device); /* * The whole dma_get_sgtable() idea is fundamentally unsafe - it seems @@ -841,7 +841,7 @@ size_t dma_opt_mapping_size(struct device *dev) } EXPORT_SYMBOL_GPL(dma_opt_mapping_size); -bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) +bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) { const struct dma_map_ops *ops = get_dma_ops(dev); @@ -849,7 +849,7 @@ bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) return dma_direct_need_sync(dev, dma_addr); return ops->sync_single_for_cpu || ops->sync_single_for_device; } -EXPORT_SYMBOL_GPL(dma_need_sync); +EXPORT_SYMBOL_GPL(__dma_need_sync); unsigned long dma_get_merge_boundary(struct device *dev) { From patchwork Wed Feb 14 16:21:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201005 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1331260dyb; Wed, 14 Feb 2024 08:24:20 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVis8Q8y7bmDypZaTr54vPjLzgfugh3XdkkdxhC6oXQY/rGjryOnHT1EOczntI4+MfpQ0PtmbD+KFpSBmNCHo837Kxc0A== X-Google-Smtp-Source: AGHT+IHsKeCFMIkKNuJmut7iKjus3n5o9igJA8cIvoj4M8nq491AKHWZFjRvROWD4v3xVMnp7XkH X-Received: by 2002:a05:6a20:438a:b0:19e:b6da:da1b with SMTP id i10-20020a056a20438a00b0019eb6dada1bmr3798880pzl.4.1707927859691; Wed, 14 Feb 2024 08:24:19 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707927859; cv=pass; d=google.com; s=arc-20160816; b=dEZX7tgQ0HkkyTX5eLJ1tvkKRP35Zb92aUpkLoZd8c6zPsZJXIaUM4trifNR3c7Xok hzjjFXlPK00O6YHuHvejYiG8JWzfouzBeX825m6b2z3nuKWvmYbISRghyY/e07nlTcW1 lyWge1+kWlY46eHn66qKngiJQg6NENzdqOAZFln8Wb+cDkSgV0pucNCDhtamzyhLooKf CWY64a814On5rmdbyRGy9bipzfm2f9Wbjn89OnWjGIttGUEJjjOSgrv/xJUAIoeiKMZl ZrayVgrRcSZFJdH5eI2xnb4ll+E/erqnX4U9CLgOQP8SsEOoJUxdowbfeoeFhyVg6Dm/ JFeg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=TdZirWx2ISe9phMWwn8x43ihntRYRHHm3AqyBUF0j0g=; fh=CIQbUrYKvoEk0QA3bUZrkMxxijWXkJ7SSqLm8nOlxTI=; b=RTAi7Ly0u8Kz/Cqem91WPYpfXqvfXdRyFijzeValeJViW4EYtd0ulkm6klIVHZ7XX2 avoVUWBR8wGtKBzRSiXVlLu8Nal20+USEuPOZcs3DC00+AS/mRTVSK05S5MF7QN7FM8k BmYBFMbRmB1+Jkjm9wloSKLjVI4CoLvdnZHsEJa1jy0zUVqGJMBxh3MATl0bIebzEWy6 ZPhKjGV3I2LNm2f6gIBSTqM9C7F6wfFjkjF+6h3Yp8fOCs262HYDJpKazpIMhXQ46rgY MeE3JGqqipvJrT7TFVoVlG4COR0QAN0bPTRiDI9DLLzOHXhiWZPDE8tL4l+0Q9DMLKS1 1kGA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gr21UQ3Q; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65513-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65513-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCUrP4PZSp/xfdzun2TVOh2id2gQKv4dPGszXwCm0/FN6asQEHlxnBxk7HFBrF4J+vBCGs2qXRltGlPetcVhLUHiSXCNsg== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id bo12-20020a056a02038c00b005cf94b8d9ddsi4219743pgb.869.2024.02.14.08.24.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:24:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65513-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gr21UQ3Q; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65513-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65513-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 73DFA28223D for ; Wed, 14 Feb 2024 16:24:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2FF8160DF8; Wed, 14 Feb 2024 16:22:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gr21UQ3Q" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 882B060884; Wed, 14 Feb 2024 16:22:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927772; cv=none; b=Lpnrv5vtMqWP1wlBAjYC0BQ5HF1GVoaHjdvnqCSmJYUivPpcOvUs0osld7dKTfvHNse2jM8UdsXZmvDFkNkTLa86j21c3pfdC0ybTE2D/I9De6+yzR27aQ070l+d7fONzyVGE7Ss0cW+ROaEoWMbsCEbBjoQxMkvNvjGayKMQww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927772; c=relaxed/simple; bh=hcetF/PnslkcN3jX68+J8IOB6E8XYwzSEUS+T5IkqzM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=djwIgagdBoSEy4SahK/eNZ9d+P3H2Aby2umHrYLnCh9RmaS+O7jyEdjNo7GuJnqsAT6WSecapjrEEEZTEKKsmkmPvgYJEvzFkMcvxnBziBMKCCnDOqm6CDIMPP89pzYlBbINzTClkYAU+85WK6LzHK4y2ORftxoTXWSXYcCdfXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gr21UQ3Q; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927770; x=1739463770; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hcetF/PnslkcN3jX68+J8IOB6E8XYwzSEUS+T5IkqzM=; b=gr21UQ3QREOJ6BZSuoENygQp3FGqEXUPwpNipsLaKXUsDTUN2MYrAHA7 q2a2DuEqabi0yxdQ2H2uMuR3GZ+AP2pmopRcRiRlrB3lSzY0zY+I+eaMh tSIYwXJ4gBbKKbrscC1PC8l3kg0Ux3G7MHXxbeX1sca5WlqlJwm9TOMsy 4fDtowc8YqnUjCsId3pegQAivsvaLTj598/kuAh9SDXW7pk7XsEKfpR/L EP0HkFyHhC5N1EX5FYChjQHqXILaIALMjiPH48p4gBSfJ/0fQ2/PRaKwa zMesdr0DEWX9xjT87mKCRSwJAGbTkXfnKgQOjPbZCYuoHXYrUN6He4Tah Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755550" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755550" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:22:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26399979" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:46 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 2/7] dma: avoid redundant calls for sync operations Date: Wed, 14 Feb 2024 17:21:56 +0100 Message-ID: <20240214162201.4168778-3-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790892163099714824 X-GMAIL-MSGID: 1790892163099714824 Quite often, devices do not need dma_sync operations on x86_64 at least. Indeed, when dev_is_dma_coherent(dev) is true and dev_use_swiotlb(dev) is false, iommu_dma_sync_single_for_cpu() and friends do nothing. However, indirectly calling them when CONFIG_RETPOLINE=y consumes about 10% of cycles on a cpu receiving packets from softirq at ~100Gbit rate. Even if/when CONFIG_RETPOLINE is not set, there is a cost of about 3%. Add dev->skip_dma_sync boolean which is set during the device initialization depending on the setup: dev_is_dma_coherent() for the direct DMA, !(sync_single_for_device || sync_single_for_cpu) or the new dma_map_ops flag, %DMA_F_CAN_SKIP_SYNC, advertised for non-NULL DMA ops. Then later, if/when swiotlb is used for the first time, the flag is turned off, from swiotlb_tbl_map_single(). On iavf, the UDP trafficgen with XDP_DROP in skb mode test shows +3-5% increase for direct DMA. Suggested-by: Christoph Hellwig # direct DMA shortcut Co-developed-by: Eric Dumazet Signed-off-by: Eric Dumazet Signed-off-by: Alexander Lobakin --- include/linux/device.h | 5 +++++ include/linux/dma-map-ops.h | 21 ++++++++++++++++++++ include/linux/dma-mapping.h | 6 +++++- drivers/base/dd.c | 2 ++ kernel/dma/mapping.c | 39 ++++++++++++++++++++++++++++++++++++- kernel/dma/swiotlb.c | 8 ++++++++ 6 files changed, 79 insertions(+), 2 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index 97c4b046c09d..f23e6a32bea0 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -686,6 +686,8 @@ struct device_physical_location { * other devices probe successfully. * @dma_coherent: this particular device is dma coherent, even if the * architecture supports non-coherent devices. + * @dma_skip_sync: DMA sync operations can be skipped for coherent non-SWIOTLB + * buffers. * @dma_ops_bypass: If set to %true then the dma_ops are bypassed for the * streaming DMA operations (->map_* / ->unmap_* / ->sync_*), * and optionall (if the coherent mask is large enough) also @@ -800,6 +802,9 @@ struct device { defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) bool dma_coherent:1; #endif +#ifdef CONFIG_DMA_NEED_SYNC + bool dma_skip_sync:1; +#endif #ifdef CONFIG_DMA_OPS_BYPASS bool dma_ops_bypass : 1; #endif diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..327b73f653ad 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -18,8 +18,11 @@ struct iommu_ops; * * DMA_F_PCI_P2PDMA_SUPPORTED: Indicates the dma_map_ops implementation can * handle PCI P2PDMA pages in the map_sg/unmap_sg operation. + * DMA_F_CAN_SKIP_SYNC: DMA sync operations can be skipped if the device is + * coherent and it's not an SWIOTLB buffer. */ #define DMA_F_PCI_P2PDMA_SUPPORTED (1 << 0) +#define DMA_F_CAN_SKIP_SYNC BIT(1) struct dma_map_ops { unsigned int flags; @@ -111,6 +114,24 @@ static inline void set_dma_ops(struct device *dev, } #endif /* CONFIG_DMA_OPS */ +#ifdef CONFIG_DMA_NEED_SYNC +void dma_setup_skip_sync(struct device *dev); + +static inline void dma_clear_skip_sync(struct device *dev) +{ + /* Clear it only once so that the function can be called on hotpath */ + if (unlikely(dev->dma_skip_sync)) + dev->dma_skip_sync = false; +} +#else /* !CONFIG_DMA_NEED_SYNC */ +static inline void dma_setup_skip_sync(struct device *dev) +{ +} +static inline void dma_clear_skip_sync(struct device *dev) +{ +} +#endif /* !CONFIG_DMA_NEED_SYNC */ + #ifdef CONFIG_DMA_CMA extern struct cma *dma_contiguous_default_area; diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 6c7640441214..d85ae541c267 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -364,7 +364,11 @@ static inline void __dma_sync_single_range_for_device(struct device *dev, static inline bool dma_skip_sync(const struct device *dev) { - return !IS_ENABLED(CONFIG_DMA_NEED_SYNC); +#ifdef CONFIG_DMA_NEED_SYNC + return dev->dma_skip_sync; +#else + return true; +#endif } static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) diff --git a/drivers/base/dd.c b/drivers/base/dd.c index 85152537dbf1..67ad3e1d51f6 100644 --- a/drivers/base/dd.c +++ b/drivers/base/dd.c @@ -642,6 +642,8 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto pinctrl_bind_failed; } + dma_setup_skip_sync(dev); + ret = driver_sysfs_add(dev); if (ret) { pr_err("%s: driver_sysfs_add(%s) failed\n", diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 85feaa0e008c..5f588e31ea89 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -846,8 +846,14 @@ bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) const struct dma_map_ops *ops = get_dma_ops(dev); if (dma_map_direct(dev, ops)) + /* + * dma_skip_sync could've been set to false on first SWIOTLB + * buffer mapping, but @dma_addr is not necessary an SWIOTLB + * buffer. In this case, fall back to more granular check. + */ return dma_direct_need_sync(dev, dma_addr); - return ops->sync_single_for_cpu || ops->sync_single_for_device; + + return true; } EXPORT_SYMBOL_GPL(__dma_need_sync); @@ -861,3 +867,34 @@ unsigned long dma_get_merge_boundary(struct device *dev) return ops->get_merge_boundary(dev); } EXPORT_SYMBOL_GPL(dma_get_merge_boundary); + +#ifdef CONFIG_DMA_NEED_SYNC +void dma_setup_skip_sync(struct device *dev) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + if (dma_map_direct(dev, ops)) + /* + * dma_skip_sync will be set to false on first SWIOTLB buffer + * mapping, if any. During the device initialization, it's + * enough to check only for DMA coherence. + */ + dev->dma_skip_sync = dev_is_dma_coherent(dev); + else if (!ops->sync_single_for_device && !ops->sync_single_for_cpu) + /* + * Synchronization is not possible when none of DMA sync ops + * is set. This check precedes the below one as it disables + * the synchronization unconditionally. + */ + dev->dma_skip_sync = true; + else if (ops->flags & DMA_F_CAN_SKIP_SYNC) + /* + * Assume that when ``DMA_F_CAN_SKIP_SYNC`` is advertised, + * the conditions for synchronizing are the same as with + * the direct DMA. + */ + dev->dma_skip_sync = dev_is_dma_coherent(dev); + else + dev->dma_skip_sync = false; +} +#endif /* CONFIG_DMA_NEED_SYNC */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index b079a9a8e087..0b737eab4d48 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1323,6 +1323,12 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } + /* + * If dma_skip_sync was set, reset it to false on first SWIOTLB buffer + * mapping to always sync SWIOTLB buffers. + */ + dma_clear_skip_sync(dev); + /* * Save away the mapping from the original address to the DMA address. * This is needed when we sync the memory. Then we sync the buffer if @@ -1640,6 +1646,8 @@ struct page *swiotlb_alloc(struct device *dev, size_t size) if (index == -1) return NULL; + dma_clear_skip_sync(dev); + tlb_addr = slot_addr(pool->start, index); return pfn_to_page(PFN_DOWN(tlb_addr)); From patchwork Wed Feb 14 16:21:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201006 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1331589dyb; Wed, 14 Feb 2024 08:24:48 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVxJJ3Q4MlIk/6uavLzr8hgiO01VXK42FqIJ2wNegKj5iDvDsjCr9TuV0Axj0g/CkVVS3O1tuEzZfLpsUKr6+4bG5XNDA== X-Google-Smtp-Source: AGHT+IF6qndabFFNG7rdhXqrckR9M0jKObzQg5sWke0DtsYziYkhPXmLu+y7FQJhMh9+WVlqhYaR X-Received: by 2002:a67:eb90:0:b0:46e:dca6:ef9f with SMTP id e16-20020a67eb90000000b0046edca6ef9fmr2878616vso.18.1707927888016; Wed, 14 Feb 2024 08:24:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707927888; cv=pass; d=google.com; s=arc-20160816; b=i1CI4eo5Vn+if7g7bEr6aaFc9Djb31z+PIzqpWVqsSWJYT9eTMgogzbV1N8dXtUQpd LFUceFRf6W86uAEwQwnJ3bL9w/KpcwKJq1h58mLVBzPILs1m8+qtYqurxpAHAlTf/WbK oyw4y1HYhWcfLjUpI5a9rJk9QxnQgatzRfdieuywMMQ2gKbiPOYCHya1XL7laV7sLYZt 26VgK88wv79m1av3C5ViyIdK4FsNcOyTeJMtC12wnzhtOFTiDyPIw+KCsAkvV2qGrPza F7h3Eb2QjXsQ46g2Ha9LRmvJZknom1eQwiqvaBAlOQnyG1VyYCqMRJwlyGg1p8OvtlZK xLwg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=3RbYCi97CBUYpb3Qk7i6iB9pJIZcKC+4jz+RMzWKVIE=; fh=GAN7d7D4LRBZIhr23XOwqsCMoYimUBAymNTp8Bu82UU=; b=bTQ4dL9m5WbqFhpEeASCsHvuBKCZq/6818QZr+V5KYqXwDoyPHl7MgtZaQDrt8x0HE t+hTVyaw8QbYmjr+QXqxpJMGNEoEPDRjyhISJvnUjvISCTtGx6f5BTBy2rio5ZYka4YM hCmWTk2PYBInr/BeErjVwuBIIXApsOyNFTCvEX0ZamS8TkkdIwOvUmpt32iyGPla8Myn LGcBLYJN6Yv2vCeXi9XaCZ3jj/PPgul9yoiMDi5uJ2q+AN+gxJfuOYKciRTKGD8NHcl9 ZyH6Gkd4C3Jo4gm3FWTip601elYx8qyHVy12jA579vjB8k7x0OI9kJJxx2ZtsfbAEM6Y /U6A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XZCI7xAs; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65515-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65515-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCXERy65IPAN2FoHGd4cRG2TzkcK04utD6upi8BLaUZSA6isKadjltaxi/9i/H/gLdsJFmSKjQsbahDPr1cNZj49VdzhRA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id r16-20020a67ef90000000b0046d37166398si1414740vsp.65.2024.02.14.08.24.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:24:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65515-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XZCI7xAs; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65515-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65515-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C5C821C213E3 for ; Wed, 14 Feb 2024 16:24:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 05689612C6; Wed, 14 Feb 2024 16:22:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XZCI7xAs" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B02060DE2; Wed, 14 Feb 2024 16:22:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927776; cv=none; b=GUF7SGDak/LU/bzMD+eNKQbrfaJYwNATPoGM++kbZnMtw3ASLP3ivd+rCKk4oHyiPFo0rlAGaL+l7Uh64blyOvwEINb2BuCdUUXM5al+Zw4HYrZZROcLqP+Anf5vE4062F/KBoLGpqRSys0W0KR30F9ZT1I+uZ7CAs81Kj3x+gw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927776; c=relaxed/simple; bh=LVf5BgWT9OSmbGH+Qwko5u5SJBVWf1jKCf7S7n+8WTs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cr8JNv9ggdVKV2zUaXJAn2ZOTqdpbQtJpKPIdmTuMq87EM8A2RCzNwxq4DVtKReDZzN+MCWSnTYrWIuNTD3iFhpsQTM/7BQ7M9oBsE9SbbOfK/ch4Qa3bIG5tmiFuN/hBk37iDk3lsIijl7BIXWBPnEEuzOMzHXjIJNiSdNlk3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XZCI7xAs; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927774; x=1739463774; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LVf5BgWT9OSmbGH+Qwko5u5SJBVWf1jKCf7S7n+8WTs=; b=XZCI7xAseoMQmku7hs3afH1TFpoM5LdApzAmTc9FxyvFOvpVfcVFlySy fA7YruRI1xUBsp0VjL8TKuIQM0Ugv2jElBxsc3ZcpsOcGvPpml76qLiaF ResO5SLLX2Lwp9ml0w4oddszgZEqI72oy6VGdFHgnfjti/3hitP4C2Pqj WIwLVTJrrUvrIno3L9+AGhdJWYy08N3DUgzE7LJ5zdCohBxjqoOn3/XV2 pLzLrNguAv/ND9DnPPoX0D9Do9I7d9BEppJpZ8RBWHm+m/LHyBEl8c5jF JQ/EKpsRL/hayVc+P/foNiGzaR3nJTWxYSoe69r0mYgzVF1m1EPVg5Sof A==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755570" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755570" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:22:54 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400001" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:50 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 3/7] iommu/dma: avoid expensive indirect calls for sync operations Date: Wed, 14 Feb 2024 17:21:57 +0100 Message-ID: <20240214162201.4168778-4-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790892193406425925 X-GMAIL-MSGID: 1790892193406425925 When IOMMU is on, the actual synchronization happens in the same cases as with the direct DMA. Advertise %DMA_F_CAN_SKIP_SYNC in IOMMU DMA to skip sync ops calls (indirect) for non-SWIOTLB buffers. perf profile before the patch: 18.53% [kernel] [k] gq_rx_skb 14.77% [kernel] [k] napi_reuse_skb 8.95% [kernel] [k] skb_release_data 5.42% [kernel] [k] dev_gro_receive 5.37% [kernel] [k] memcpy <*> 5.26% [kernel] [k] iommu_dma_sync_sg_for_cpu 4.78% [kernel] [k] tcp_gro_receive <*> 4.42% [kernel] [k] iommu_dma_sync_sg_for_device 4.12% [kernel] [k] ipv6_gro_receive 3.65% [kernel] [k] gq_pool_get 3.25% [kernel] [k] skb_gro_receive 2.07% [kernel] [k] napi_gro_frags 1.98% [kernel] [k] tcp6_gro_receive 1.27% [kernel] [k] gq_rx_prep_buffers 1.18% [kernel] [k] gq_rx_napi_handler 0.99% [kernel] [k] csum_partial 0.74% [kernel] [k] csum_ipv6_magic 0.72% [kernel] [k] free_pcp_prepare 0.60% [kernel] [k] __napi_poll 0.58% [kernel] [k] net_rx_action 0.56% [kernel] [k] read_tsc <*> 0.50% [kernel] [k] __x86_indirect_thunk_r11 0.45% [kernel] [k] memset After patch, lines with <*> no longer show up, and overall cpu usage looks much better (~60% instead of ~72%): 25.56% [kernel] [k] gq_rx_skb 9.90% [kernel] [k] napi_reuse_skb 7.39% [kernel] [k] dev_gro_receive 6.78% [kernel] [k] memcpy 6.53% [kernel] [k] skb_release_data 6.39% [kernel] [k] tcp_gro_receive 5.71% [kernel] [k] ipv6_gro_receive 4.35% [kernel] [k] napi_gro_frags 4.34% [kernel] [k] skb_gro_receive 3.50% [kernel] [k] gq_pool_get 3.08% [kernel] [k] gq_rx_napi_handler 2.35% [kernel] [k] tcp6_gro_receive 2.06% [kernel] [k] gq_rx_prep_buffers 1.32% [kernel] [k] csum_partial 0.93% [kernel] [k] csum_ipv6_magic 0.65% [kernel] [k] net_rx_action iavf yields +10% of Mpps on Rx. This also unblocks batched allocations of XSk buffers when IOMMU is active. Co-developed-by: Eric Dumazet Signed-off-by: Eric Dumazet Signed-off-by: Alexander Lobakin Acked-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 50ccc4f1ef81..4ab9ac13d362 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1707,7 +1707,8 @@ static size_t iommu_dma_opt_mapping_size(void) } static const struct dma_map_ops iommu_dma_ops = { - .flags = DMA_F_PCI_P2PDMA_SUPPORTED, + .flags = DMA_F_PCI_P2PDMA_SUPPORTED | + DMA_F_CAN_SKIP_SYNC, .alloc = iommu_dma_alloc, .free = iommu_dma_free, .alloc_pages = dma_common_alloc_pages, From patchwork Wed Feb 14 16:21:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201042 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1348122dyb; Wed, 14 Feb 2024 08:51:11 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVXMJUe0atV/HORHfrK6zWmqQUEK/bf9oMi98Zxwb+A7AuQd1r1YwqEZHQt4aB4HQ8dfsPlHjlejnQbVmrvWldiKLp3Hw== X-Google-Smtp-Source: AGHT+IH97jwFXFbuXBFdOA8vkorWAxyFk9s48FVfdH2Z00d+k3s9viMeE9THvYJFk3FY20jRLh2D X-Received: by 2002:a17:90a:e506:b0:296:1ee:a54d with SMTP id t6-20020a17090ae50600b0029601eea54dmr3099414pjy.48.1707929471463; Wed, 14 Feb 2024 08:51:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707929471; cv=pass; d=google.com; s=arc-20160816; b=RXjKer4SkfKs+JNm2Apn8L34pS1pShOSKMDimJ7+I9rQ9Hoigq3pt6K5vZr8t6DhVv B4l+IlFmsUjU5Vt7gAdypge4n3q377kwIxONHxS4HSU+cjlmarTe5umgUgAFlkmK7YPo zRHIV/o7wxx2V8OCe4JKvSgyh4FoT4oUq4VTLqOD2k7gRkl3djoSHROhxupzL3GWVNJH 6fWQZgqE2w87+kK3fEYxGvBHXoLk6gd1oy7f5ml9tVDxK2izNeODrvx0/vuv3CfizY+T Xd95e+/g7rQLUyQO+J8UXyhpJULNW55YGrk8qUdZuPyy2Cd4VIIDhn2V9JITNlLv3Qdo MnUw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=OemP9CZ81RJmmITlED5kKKv7ot0ld8HBt6qj4vXDcs4=; fh=p9UVfHZrpOpaw7yYkEMPSqtecQpykP9kauv+FmIg5io=; b=MSVijkXdgjNf6HuVKLtY70Xzp2SK0tYWr2ARQVBYnj1nfWndWZRb5nG606RMzgS5o6 7aTCj9+ENYOKOQxX0jlKS5OWJpfJ1Jtxd4ZhPW24N/QRTof/AFJcXRswKIORdGUIHjk6 Sv2Rxs/sw+ppSeSfOoOFu/Yvp2o3iLIW/ozxibuSRI2JxaGszK20ymcaR+oeC/Dy30bS KWaY0pj4mT++/QGYkLMldRMPhSnbkFfXT/V3auZO8Xbubs9SgomtoXiCAJO0DSazoREa nEhY8OoJTzaXsSNDpBEDml5G1BkkEpdbC+I2wlz17BMEjA0gk+KA6bQehXV9/vQmNz7y BdFQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ctzg4LqY; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCWHx5hdLiLSKv54Jozs3jGJ9gEWBpVCAe3N0g1me/pJ13JG3PosjnNKVa4DiodQ90kSZGPRmMXdxlnX3uJrdoT45K1B/Q== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id f15-20020a17090aec8f00b0029619430cebsi1406384pjy.155.2024.02.14.08.51.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:51:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ctzg4LqY; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E497CB22561 for ; Wed, 14 Feb 2024 16:25:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CF3C361672; Wed, 14 Feb 2024 16:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Ctzg4LqY" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9960D612C1; Wed, 14 Feb 2024 16:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927780; cv=none; b=fWZ5opUfZEFL8stLMRcvUBC11qqP31X1DqgvkC5YEobcwhfALaLyaQg3rlGPUpuxidCnKbFhWRlDqTjMLUQnKxkYYhRVU9Lykg6KoJHvbpk9CkHiZQc1MFazznyhrN8jaecBQeE7fMrW31bzav+OD4elVP5YbtQnlBO13v2NqWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927780; c=relaxed/simple; bh=2jYWs6cVfcBDrhRNnxCoaJhB0s1tUX2UFowyoAY2brs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nlzF1/Y2NNmX+07zv8oHCieFRfLHCFEm3xUZDF/7lO2UFNGDhwXkca5etvuwUtt6QQy6TXUH5Z7wYF1o+HaCXzPOSmEdsVu2hhQ73jqXL6YN4By/TINvRitPT7CGdIheQXxu7GXa+heEVfQxHTjKAYJ7A8FbU/bpRUj3Mvc8QMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Ctzg4LqY; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927778; x=1739463778; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2jYWs6cVfcBDrhRNnxCoaJhB0s1tUX2UFowyoAY2brs=; b=Ctzg4LqYepotQaI30P7e9QCC/QcFQ9Wv0O17xfPz+2ELGyOeAF33Oa7U e1f80RrFykaFwkDWfoc8Kjy21JnHC+JkKZJzmlTsiGwOg+GcWaVO/MgmK gRlhLa+YGsEVChke0L3ADtIT9Eq/tD1yoTO/DF+NGl5fYwIdVnc+ZxAxV Zd0Jk3oo57yhAzbbvM9IrrG6NZyN6EL8lfgl+Al1+XhCGEzSCGBOGPNCS k1CNM1J/h8I2UIxK9vZlsIjbTAsRaYPJHqYBkyk5feEwC1knw7dtcRcLx CsUKsQ8Qy8dT91nS672v54uVqgQ2W+gQFinA5saE2NqEvMd7hSh/scm5o A==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755588" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755588" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:22:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400020" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:54 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 4/7] page_pool: make sure frag API fields don't span between cachelines Date: Wed, 14 Feb 2024 17:21:58 +0100 Message-ID: <20240214162201.4168778-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790893853073900823 X-GMAIL-MSGID: 1790893853073900823 After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again. To not bother with this each time fast params get expanded or shrunk, let's just align them to `4 * sizeof(long)`, the closest upper pow-2 to their actual size (2 longs + 1 int). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing. ::page_state_hold_cnt is used quite intensively on hotpath no matter if frag API is used, so move it to the newly created hole in the first cacheline. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 12 +++++++++++- net/core/page_pool.c | 9 +++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 3828396ae60c..b2b93bec7bce 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -129,12 +129,22 @@ struct page_pool { struct page_pool_params_fast p; int cpuid; + u32 pages_state_hold_cnt; bool has_init_callback; + /* The following block must stay within one cacheline. On 32-bit + * systems, sizeof(long) == sizeof(int), so that the block size is + * ``3 * sizeof(long)``. On 64-bit systems, the actual size is + * ``2 * sizeof(long) + sizeof(int)``. The closest pow-2 to both of + * them is ``4 * sizeof(long)``, so just use that one for simplicity. + * Having it aligned to a cacheline boundary may be excessive and + * doesn't bring any good. + */ + __cacheline_group_begin(frag) __aligned(4 * sizeof(long)); long frag_users; struct page *frag_page; unsigned int frag_offset; - u32 pages_state_hold_cnt; + __cacheline_group_end(frag); struct delayed_work release_dw; void (*disconnect)(void *pool); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 89c835fcf094..3c464852e228 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -170,12 +170,21 @@ static void page_pool_producer_unlock(struct page_pool *pool, spin_unlock_bh(&pool->ring.producer_lock); } +static void page_pool_struct_check(void) +{ + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_users); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_page); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_offset); +} + static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params, int cpuid) { unsigned int ring_qsize = 1024; /* Default */ + page_pool_struct_check(); + memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow)); From patchwork Wed Feb 14 16:21:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201007 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1332337dyb; Wed, 14 Feb 2024 08:25:54 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVZR9Gtb453Ipj2VJDh3g3IA/wag9JqsnWMhlhpBiOK/je8Bqn6J9yYxMgfyJ5UvZsa+xa5G8Dr0kFL2oj4tVEz/urSpQ== X-Google-Smtp-Source: AGHT+IHEMKSXrSNTPc4uRzu5VgYRqgNYHUAme6FY4R7SJbHjo3y80w7y0MhELv6fQSBBnklsyJ7F X-Received: by 2002:a05:6871:5b1c:b0:219:6c49:69e9 with SMTP id op28-20020a0568715b1c00b002196c4969e9mr3417485oac.21.1707927953854; Wed, 14 Feb 2024 08:25:53 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707927953; cv=pass; d=google.com; s=arc-20160816; b=EnbzUHPsS+UBiVz235j7WnBj9WjBuV8WnlubwnoImcufkpzDOlBgugvGOm9XLLf2C+ kOD6H6w7GPDgDj1EoWLUy2kLTcMbVrAl2VIL7thWgaYgBIacdls8uK5D3DoS/6sYasyJ jMANN5XvDLX2aUEMtZBrnxtCVA3iGjTrs13M5Qc7mAv9jy6zB3McXhybyfhK/vTnj0dc 0BkzSZxzWM5902OlXXDMJa9TUYoffRF1webB9DKrNK9o64XjRjK9jYBhlo0+mu1BB1+e dQt+B+aIuWGRmU+ufxmzRGUpK4B6Zc9Zqn9b8nymYM/IBa4Jt9qsYArmwsPIi/mlGeXN bD2A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=T57NM6QRTSbZ4gakr0qavkveDWlsRNW9FGO1GtIcHcI=; fh=4SVO90owGYmo+vPdoFK6Ex9J6EIb/I6gUNrk4pXmBZw=; b=TW9QKGROfZHxvlJcviQbG9qktcWjq0Ke98Yo9cWWC/5T1Iy5j6tj5Hzko22ZYXmAg5 y5H2VvXoj29pMSAwqO1uFoQ4/m65L4SDtD8CVlPZESzcZ91skR15jYkKRQPbnkm9k7K/ qRpRyG796IDpRVyAfkbvZo/CMKeJEV7W5w678f7rUVHKcW7ZntbAr6NnywKLwMR9u0HV U9lIT+VcVnvC/1oh1y9WT2m888lG0ecY1NxFHvATTtTQ/QK1sDkPleL/+52O1cF729aY 3mwUzKYaOucH6c0LcxDsN1O5MCqAtMak0K/b+1GQ/rRjS6mPch39J2eKbUVW+yAAZCL3 ZB1A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FjLV+lTE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65517-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65517-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCWgLiyD5uEY0GooRwqphcf3Gc7Mr8SyKdQwXGk2djATzczT8wmfKi/mpM49kNwBMlmBQvrGUysfo+cg+jKywIcN4qdctw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id po30-20020a05620a385e00b00785b189cc93si10965800qkn.353.2024.02.14.08.25.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:25:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65517-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FjLV+lTE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65517-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65517-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id F06771C28205 for ; Wed, 14 Feb 2024 16:25:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E4448626DF; Wed, 14 Feb 2024 16:23:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="FjLV+lTE" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A50A4612CD; Wed, 14 Feb 2024 16:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927784; cv=none; b=N81W+ba8zrFZ9jxL3JbbQUZBZ8CTvbUdYlxgk197Xnywp48tKhfpVZfmAXBFa/Wq+4Q8H1S2dW+a5CYwU9/Lbq0estmkMx910M5kvWNniTChebiQftNEl80uYDZJRViWZyrsHLUaGtOugxy9PbYm2O10HrCPYCXdO37wZxe3aHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927784; c=relaxed/simple; bh=ivEhI9z6xPg9OHonv02GVsSHqZLDifgZ7XtI9c4EjDc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P6DMNJ2kWL+0JYaJJIUGFGt/msRrvgosPrcXn7qBxtfoP6T1SUcCIw5dMMQpGomoXfJTD71Y5oviJGovYclOd1tedVT1XhB0j4wfWxzxVlFQVDtJSX2gHhu57759m5pGhZW37Dw+9uH2PSRuFZ7/91eXmZL7JuWjyvug3SmJBdo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=FjLV+lTE; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927782; x=1739463782; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ivEhI9z6xPg9OHonv02GVsSHqZLDifgZ7XtI9c4EjDc=; b=FjLV+lTEndBU2SgrVV5SKf2Yy1+Gd8ZBektE6PWugsE8jxgwO9uJ1gjU /a/z9elbJPGmvwtrAp2QqbOJ8IZAuZwhBOBTEG6UU5+sF+wIQKNjJd/JI O7jRFJCt8pe5NnAuqMe51wpD4hg5TCuc5VpnLiXHKOwjiBpQNsStqGNPM jbbT0i6CQmjGitm8aTwT+6BjMR4pTMB4J+aDl7R4KE/PMR90bPF0c1vMF g5wAtWg2e/nXBCqPtLJ1iMlffl1ihgi840CYVDErp22G8rT5mQ1nivdWe nVWyMAAaom2YNgBMfEKI7sX5AvNEVPp62BHwitCkt0jgOI/amcNr0QWa3 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755610" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755610" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:23:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400037" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:58 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 5/7] page_pool: don't use driver-set flags field directly Date: Wed, 14 Feb 2024 17:21:59 +0100 Message-ID: <20240214162201.4168778-6-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790892262182540657 X-GMAIL-MSGID: 1790892262182540657 page_pool::p is driver-defined params, copied directly from the structure passed to page_pool_create(). The structure isn't meant to be modified by the Page Pool core code and this even might look confusing[0][1]. In order to be able to alter some flags, let's define our own, internal fields the same way as the already existing one (::has_init_callback). They are defined as bits in the driver-set params, leave them so here as well, to not waste byte-per-bit or so. Almost 30 bits are still free for future extensions. We could've defined only new flags here or only the ones we may need to alter, but checking some flags in one place while others in another doesn't sound convenient or intuitive. ::flags passed by the driver can now go to the "slow" PP params. Suggested-by: Jakub Kicinski Link[0]: https://lore.kernel.org/netdev/20230703133207.4f0c54ce@kernel.org Suggested-by: Alexander Duyck Link[1]: https://lore.kernel.org/netdev/CAKgT0UfZCGnWgOH96E4GV3ZP6LLbROHM7SHE8NKwq+exX+Gk_Q@mail.gmail.com Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 9 ++++++--- net/core/page_pool.c | 34 ++++++++++++++++++---------------- 2 files changed, 24 insertions(+), 19 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index b2b93bec7bce..938980dafa7d 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -44,7 +44,6 @@ struct pp_alloc_cache { /** * struct page_pool_params - page pool parameters - * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV * @order: 2^order pages on allocation * @pool_size: size of the ptr_ring * @nid: NUMA node id to allocate from pages from @@ -54,10 +53,10 @@ struct pp_alloc_cache { * @dma_dir: DMA mapping direction * @max_len: max DMA sync memory size for PP_FLAG_DMA_SYNC_DEV * @offset: DMA sync address offset for PP_FLAG_DMA_SYNC_DEV + * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV */ struct page_pool_params { struct_group_tagged(page_pool_params_fast, fast, - unsigned int flags; unsigned int order; unsigned int pool_size; int nid; @@ -68,6 +67,7 @@ struct page_pool_params { unsigned int offset; ); struct_group_tagged(page_pool_params_slow, slow, + unsigned int flags; struct net_device *netdev; /* private: used by test code only */ void (*init_callback)(struct page *page, void *arg); @@ -130,7 +130,10 @@ struct page_pool { int cpuid; u32 pages_state_hold_cnt; - bool has_init_callback; + + bool dma_map:1; /* Perform DMA mapping */ + bool dma_sync:1; /* Perform DMA sync */ + bool has_init_callback:1; /* slow.init_callback is set */ /* The following block must stay within one cacheline. On 32-bit * systems, sizeof(long) == sizeof(int), so that the block size is diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 3c464852e228..a9f3c032c45f 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -191,7 +191,7 @@ static int page_pool_init(struct page_pool *pool, pool->cpuid = cpuid; /* Validate only known flags were used */ - if (pool->p.flags & ~(PP_FLAG_ALL)) + if (pool->slow.flags & ~(PP_FLAG_ALL)) return -EINVAL; if (pool->p.pool_size) @@ -205,22 +205,26 @@ static int page_pool_init(struct page_pool *pool, * DMA_BIDIRECTIONAL is for allowing page used for DMA sending, * which is the XDP_TX use-case. */ - if (pool->p.flags & PP_FLAG_DMA_MAP) { + if (pool->slow.flags & PP_FLAG_DMA_MAP) { if ((pool->p.dma_dir != DMA_FROM_DEVICE) && (pool->p.dma_dir != DMA_BIDIRECTIONAL)) return -EINVAL; + + pool->dma_map = true; } - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) { + if (pool->slow.flags & PP_FLAG_DMA_SYNC_DEV) { /* In order to request DMA-sync-for-device the page * needs to be mapped */ - if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + if (!(pool->slow.flags & PP_FLAG_DMA_MAP)) return -EINVAL; if (!pool->p.max_len) return -EINVAL; + pool->dma_sync = true; + /* pool->p.offset has to be set according to the address * offset used by the DMA engine to start copying rx data */ @@ -246,7 +250,7 @@ static int page_pool_init(struct page_pool *pool, /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) get_device(pool->p.dev); return 0; @@ -256,7 +260,7 @@ static void page_pool_uninit(struct page_pool *pool) { ptr_ring_cleanup(&pool->ring, NULL); - if (pool->p.flags & PP_FLAG_DMA_MAP) + if (pool->dma_map) put_device(pool->p.dev); #ifdef CONFIG_PAGE_POOL_STATS @@ -411,7 +415,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) if (page_pool_set_dma_addr(page, dma)) goto unmap_failed; - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -457,8 +461,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, if (unlikely(!page)) return NULL; - if ((pool->p.flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); return NULL; } @@ -478,8 +481,8 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, gfp_t gfp) { const int bulk = PP_ALLOC_CACHE_REFILL; - unsigned int pp_flags = pool->p.flags; unsigned int pp_order = pool->p.order; + bool dma_map = pool->dma_map; struct page *page; int i, nr_pages; @@ -504,8 +507,7 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, */ for (i = 0; i < nr_pages; i++) { page = pool->alloc.cache[i]; - if ((pp_flags & PP_FLAG_DMA_MAP) && - unlikely(!page_pool_dma_map(pool, page))) { + if (dma_map && unlikely(!page_pool_dma_map(pool, page))) { put_page(page); continue; } @@ -577,7 +579,7 @@ void __page_pool_release_page_dma(struct page_pool *pool, struct page *page) { dma_addr_t dma; - if (!(pool->p.flags & PP_FLAG_DMA_MAP)) + if (!pool->dma_map) /* Always account for inflight pages, even if we didn't * map them */ @@ -655,7 +657,7 @@ static bool page_pool_recycle_in_cache(struct page *page, } /* If the page refcnt == 1, this will try to recycle the page. - * if PP_FLAG_DMA_SYNC_DEV is set, we'll try to sync the DMA area for + * If pool->dma_sync is set, we'll try to sync the DMA area for * the configured size min(dma_sync_size, pool->max_len). * If the page refcnt != 1, then the page will be returned to memory * subsystem. @@ -678,7 +680,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, dma_sync_size); @@ -791,7 +793,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV) + if (pool->dma_sync) page_pool_dma_sync_for_device(pool, page, -1); return page; From patchwork Wed Feb 14 16:22:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201043 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1348644dyb; Wed, 14 Feb 2024 08:52:07 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXXBJT3DsgIed6BiscRRnakhtke8eKQJyxuJYx6y+1QbA2yiXqI+No8NjbAbhgCLUKmC2xYl2ZnKbnHBLFAuO/kwjSzZA== X-Google-Smtp-Source: AGHT+IGgYqjpkM7GO/Qek+6r0FwcxcgLTA8IUDfyfy5XhCxxgXrHYMkUMvdGWu+RKmhubzhEB7AU X-Received: by 2002:a05:6a00:1990:b0:6e0:3d8c:72bb with SMTP id d16-20020a056a00199000b006e03d8c72bbmr3600567pfl.33.1707929527235; Wed, 14 Feb 2024 08:52:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707929527; cv=pass; d=google.com; s=arc-20160816; b=rN6sNnwOBMwQBJQ8IZ7Ee54gSss5sZBZaIOZVi6WstNyqAGB/YbbckK+6R7D/NyO88 Ui67VlY4nM7pA69o/wFtu9neboNm14oc5MGQoecfodvglq9Ldlw6e+4mluI0rBuWx8/T gzZ0NG2yyq/EXDzsMlHm01SWn6M2xRR/SWqJFZxHsObqo2jSVyMh5GnB1+xH6vSE8D+V KMBFxDZtzVjFWghUnCob2l/CODyfXNtGfzGXq8SHEKHev7mWrBq4p4MJLcYOUeR1y42n ETipw+pY/zTl3BrXbKO8dG/IJqJjWMP20jSt1UrRVuCVdmTyVNmE5Khv1EhqmjYL+4dZ uUdw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=dakEawkeyRq5Sq42nEsPJO6K0hSqeAe2ZiZPRS7onbs=; fh=jOYc1jZUlxL2+vhCQGAsFV5ISIAMvcnzkyzKJrVm5qk=; b=n2B3dEYvqz44OwnX8ICvDu1g39uUM0m0XEGBuzq9gRch9+VgCDyklI76qSi+Du++xN sM+8uokIXCuRWxioTJqjec9QDriCJR5ZPlYg/XLrwrftLyIrTS7dzWp/exZaxtgPWm5I EFbZdsXCgUNjoRSch0x0sdyXURQFLUQEaKqniPiR+d2aLgt59g4oLNiPnSzjxXAOYRxQ VRen1KyxcCosJ269R6Po2pMb51f0j23Tgn2lN2cCF26edXYm9uVwWQrHnbrgnXYhTS1H vOOKj28sKu+NAEHl7ilxqs7yUq8bEQiC2Dm5rhEtEy43zaG8xsHT+XmmT8ye/SBSemvq IIHQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=a0yjvQGq; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65518-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65518-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCXIh+MPvUbJDOzDgfSwZfv3AFzAKqTfmW3AFSChTnnGIg6uhDOMeSa7tPx0ov3bQCCe9TLW1SqfJwSbtE/N3Zk7xaOuXw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id jc3-20020a056a006c8300b006e0dd734307si5537630pfb.157.2024.02.14.08.52.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:52:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65518-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=a0yjvQGq; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65518-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65518-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 0BA8CB23765 for ; Wed, 14 Feb 2024 16:25:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id ADFC76312F; Wed, 14 Feb 2024 16:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="a0yjvQGq" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 897DF626A3; Wed, 14 Feb 2024 16:23:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927788; cv=none; b=NC4tlOr/z6EttneRhU8CzoKOIycr5pOP+BnLFxkoJVgsWr9IkM9DwdahoR7P160CocNYOLeiiNVJyKTBWWbe8Kv41AwOsORAiR0RMPh0USF65CCtutxN/3K94wGi3QJykWpAtWbG+KB867+Kj99a2mCCVD4+KyRKAgrBBvj6B48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927788; c=relaxed/simple; bh=8/xfsN2JJ5K0biHSMOPQftsQpWMqQOkJvAI5f9o0N8U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mPKXK4/vVQmUm7QRTwAm8wo0a7/kVg8Y2pMEdYHT8XZbcRjDK0itITmFdW1jWRizoByXgowUA83Nt8cmzbevYF7qlAfpI3kIp8qtd4VCZ+QIdHEzzXtF0thrlEK7SPf/cOnLhW3d5dXKEiz0FV08n32P4yiaaAkm14MKrmDjRFo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=a0yjvQGq; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927786; x=1739463786; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8/xfsN2JJ5K0biHSMOPQftsQpWMqQOkJvAI5f9o0N8U=; b=a0yjvQGqwXDQLecMnmxWH2ZVhfquECBYNfjMEI2sIhd0WCz3YaLR5Op2 MvGtgcO8nbXmjRqZhwNVPzRIuapU9QfOZGTD9ZUfX9wmk5EoaOJTWH1t7 UI3iwBkBkT9x8RciFVNx7FfYxkmDouXKGoN+47zACHjt0Va0Q4k58f+zB OnXQI7GBdKbubbnqFLFlOd0/C5CXBkkcY6ZNlUxp6vkrY/1Lpx7LUQy49 fzyA/KBP5P8hyZL7fgr2/rltZb6KFI/hfosZuMbdhgNxQf/2nvA8m2aqx s8szjHOGygNZNvvgg5JLFZQXJxRh67afva4c7QjF+cqAIfxAmkBDKYiHX Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755633" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755633" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:23:06 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400053" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:23:02 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 6/7] page_pool: check for DMA sync shortcut earlier Date: Wed, 14 Feb 2024 17:22:00 +0100 Message-ID: <20240214162201.4168778-7-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790893911997241861 X-GMAIL-MSGID: 1790893911997241861 We can save a couple more function calls in the Page Pool code if we check for dma_skip_sync() earlier, just when we test pp->p.dma_sync. Move both these checks into an inline wrapper and call the PP wrapper over the generic DMA sync function only when both are true. You can't cache the result of dma_skip_sync() in &page_pool, as it may change anytime if an SWIOTLB buffer is allocated or mapped. Signed-off-by: Alexander Lobakin --- net/core/page_pool.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index a9f3c032c45f..a48895d824e2 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -384,16 +384,24 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) return page; } -static void page_pool_dma_sync_for_device(struct page_pool *pool, - struct page *page, - unsigned int dma_sync_size) +static void __page_pool_dma_sync_for_device(struct page_pool *pool, + struct page *page, + u32 dma_sync_size) { dma_addr_t dma_addr = page_pool_get_dma_addr(page); dma_sync_size = min(dma_sync_size, pool->p.max_len); - dma_sync_single_range_for_device(pool->p.dev, dma_addr, - pool->p.offset, dma_sync_size, - pool->p.dma_dir); + __dma_sync_single_range_for_device(pool->p.dev, dma_addr, + pool->p.offset, dma_sync_size, + pool->p.dma_dir); +} + +static __always_inline void +page_pool_dma_sync_for_device(struct page_pool *pool, struct page *page, + u32 dma_sync_size) +{ + if (pool->dma_sync && !dma_skip_sync(pool->p.dev)) + __page_pool_dma_sync_for_device(pool, page, dma_sync_size); } static bool page_pool_dma_map(struct page_pool *pool, struct page *page) @@ -415,8 +423,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) if (page_pool_set_dma_addr(page, dma)) goto unmap_failed; - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, pool->p.max_len); + page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -680,9 +687,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, - dma_sync_size); + page_pool_dma_sync_for_device(pool, page, dma_sync_size); if (allow_direct && in_softirq() && page_pool_recycle_in_cache(page, pool)) @@ -793,8 +798,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, -1); + page_pool_dma_sync_for_device(pool, page, -1); return page; } From patchwork Wed Feb 14 16:22:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201008 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1332935dyb; Wed, 14 Feb 2024 08:27:00 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCX9UGgLRlfZT2FdKZzJXljzLJ/fhl8HeCgsN3SlILR/oJAqSybnFu+1/A2p4fy+AW6barjBqDQ7p6xf0bH2z5QbGoa05A== X-Google-Smtp-Source: AGHT+IGPNvL70XH2TNxTobbvRywwIaO02IqSj2cFuMOnStLwL5i44dLswy2R4xitjrJdQQlVXCMS X-Received: by 2002:a05:6358:1283:b0:17a:f931:60ca with SMTP id e3-20020a056358128300b0017af93160camr3653570rwi.9.1707928020466; Wed, 14 Feb 2024 08:27:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707928020; cv=pass; d=google.com; s=arc-20160816; b=r6DpMFF63cZNz0JChzvD8VZLJhcGZA2nMMOJWz6lfqthIAM0IE8dEd1UDcsCCpbwDw GAPFq/rzV227ykThF9XZzno7HNuo/6lD0iQWr6wOCeaoQyfH0FYQfFafxPPwQJ9mhCI9 EagrPqDQbfHF94AeQ20MP683DqatnC4JFbNmqOdny9Vwn+06xQxj2fqPdONZpr0SNOCS 9Q4kqHGYjpfrLWM+l8qzZwchieB6pmmt6aWiuo3NuVoh+fT7jC0TZl41PQQ2IoNsDRvq /yVoF5JP6ZgPHgBzkG/T5oiTvyc/yv5899nH9rPzDT4vJn69bF6WJKNNzc8uYdI6OLan n36Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=98rkN4/QtXqC6BceMScYqx13OCyEKpAazSFVXY29Xqk=; fh=ZtcVJ6aw4G6nFCKnhFXD9ulyLaAfoY2svseU7DdiiZ8=; b=rGHvRw7gQ36uGBJ9xjxuxrZLgZEJsI2dhHUSs/+7SmzruTM57P83/QWu0wNiOA7aQI ecSTPSZ8ibmjPfwxPuX62TDr2HQDT7z2eboq5Ekokg3UW3VjLi8dLT6mCV4Vz4+2Fh27 XKFamKxtuCeI0HrHDA3nZXVllP4VHQxLCXrXADh0TmUYXq7eaDa4lvsdwU/NPC8Mf3b3 xkWUKA5sXdxX1lxBMpwWt2UshUxqyiQ0LKOACqaI+BXQge9CEHKvl9ukShqCtF0y+Blv GKCDp0qapj9WZYZXEx7W/lxOBKi8Ly+hv4B9md0LvQeNcJvcchspEYL+xpeHHZ6cLzv6 dYqw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="k/JEmdav"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65519-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65519-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCWDx94xcYZltXai08pHTDtH4mQuF59HACKoPNYOzKNKoq3YZLRDch2V6F/xrXpVl2+oogcRrqffwTAqpSShl8u6ku5Qag== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id y28-20020a637d1c000000b005dc7fe1218dsi3501783pgc.284.2024.02.14.08.27.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:27:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65519-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="k/JEmdav"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65519-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65519-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 79675281569 for ; Wed, 14 Feb 2024 16:26:23 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CB1F6634FB; Wed, 14 Feb 2024 16:23:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="k/JEmdav" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D19DA63134; Wed, 14 Feb 2024 16:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927793; cv=none; b=mZLA7Y7JI1BHJGQdoKAb7RqcR4qDYMvJmKYrVLXehOahj9SEBPoyOm8ZtWHiw9DY/p6Kh8YYH+G1hvmiPmXYi1kdQgcwnfi4bPUfcZE36kdUtkozhvSQEfad4rWkcrkbgP/8zxSdmCKtK6prQmrGabtISJ4VAcBt06YEjM7wTto= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927793; c=relaxed/simple; bh=8O515Gvu1dafk1rw+MhXdSKFdsnsTiqAswRK09tlEWU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FJDzgI3KJip6uwggPxTT6Zrq/HQA6qrKeDOqatIpLQ7y1TxsUP5jyQeqPQnUyoREIpjoHLDQRa2bWudkxcyiu3ipQpv4XhvGYXKDCB2HKTXEB/qG6IBdXOqjx5OH3QIn/UQAYWF3Y8K+h0Uj9/d941qspONvPfEFXdZKci9At78= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=k/JEmdav; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927791; x=1739463791; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8O515Gvu1dafk1rw+MhXdSKFdsnsTiqAswRK09tlEWU=; b=k/JEmdavi2VsteHQXLmoOA0ZSqnDnyu2VzNo95yKdaU56oR8fgHco0Bo XQh3WtlW+T0h0rf1tl8nnzvh15v+KEyWxr5qRSN1AJHi1aNDr01VI5yjD QZdFB3I5lAmw/fyvIXVo0EA+N+UaY4ER7EPBUwiRSrV5XkZu9AK2QDods 6i35DB3OZKbg+1bP+lpFOxfyej0oZv8iopkSwmML1ZgllTOaDmYfuuODV MQJRFeXT0yEaIyLPAaKQGnGGo0GZUQ0CvTgjGBs7HaUQ0nXhULDN+7CYJ W7hYhosmR9rwslKPNiwjieri+tdrnuCpDzUIHS1sBnHdHzsd5P8UT6GdX w==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755650" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755650" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:23:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400068" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:23:06 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 7/7] xsk: use generic DMA sync shortcut instead of a custom one Date: Wed, 14 Feb 2024 17:22:01 +0100 Message-ID: <20240214162201.4168778-8-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790892331700247832 X-GMAIL-MSGID: 1790892331700247832 XSk infra's been using its own DMA sync shortcut to try avoiding redundant function calls. Now that there is a generic one, remove the custom implementation and rely on the generic helpers. xsk_buff_dma_sync_for_cpu() doesn't need the second argument anymore, remove it. Signed-off-by: Alexander Lobakin --- include/net/xdp_sock_drv.h | 7 ++--- include/net/xsk_buff_pool.h | 13 ++------- drivers/net/ethernet/engleder/tsnep_main.c | 2 +- .../net/ethernet/freescale/dpaa2/dpaa2-xsk.c | 2 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +- drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +- drivers/net/ethernet/intel/igc/igc_main.c | 2 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 4 +-- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 2 +- drivers/net/ethernet/netronome/nfp/nfd3/xsk.c | 2 +- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 2 +- net/xdp/xsk_buff_pool.c | 29 +++---------------- 13 files changed, 20 insertions(+), 51 deletions(-) diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h index c9aec9ab6191..0a5dca2b2b3f 100644 --- a/include/net/xdp_sock_drv.h +++ b/include/net/xdp_sock_drv.h @@ -219,13 +219,10 @@ static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool return meta; } -static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool) +static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp) { struct xdp_buff_xsk *xskb = container_of(xdp, struct xdp_buff_xsk, xdp); - if (!pool->dma_need_sync) - return; - xp_dma_sync_for_cpu(xskb); } @@ -402,7 +399,7 @@ static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool return NULL; } -static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp, struct xsk_buff_pool *pool) +static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp) { } diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h index 99dd7376df6a..b61e787a0ee5 100644 --- a/include/net/xsk_buff_pool.h +++ b/include/net/xsk_buff_pool.h @@ -43,7 +43,6 @@ struct xsk_dma_map { refcount_t users; struct list_head list; /* Protected by the RTNL_LOCK */ u32 dma_pages_cnt; - bool dma_need_sync; }; struct xsk_buff_pool { @@ -82,7 +81,6 @@ struct xsk_buff_pool { u8 tx_metadata_len; /* inherited from umem */ u8 cached_need_wakeup; bool uses_need_wakeup; - bool dma_need_sync; bool unaligned; bool tx_sw_csum; void *addrs; @@ -155,21 +153,16 @@ static inline dma_addr_t xp_get_frame_dma(struct xdp_buff_xsk *xskb) return xskb->frame_dma; } -void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb); static inline void xp_dma_sync_for_cpu(struct xdp_buff_xsk *xskb) { - xp_dma_sync_for_cpu_slow(xskb); + dma_sync_single_for_cpu(xskb->pool->dev, xskb->dma, + xskb->pool->frame_len, DMA_BIDIRECTIONAL); } -void xp_dma_sync_for_device_slow(struct xsk_buff_pool *pool, dma_addr_t dma, - size_t size); static inline void xp_dma_sync_for_device(struct xsk_buff_pool *pool, dma_addr_t dma, size_t size) { - if (!pool->dma_need_sync) - return; - - xp_dma_sync_for_device_slow(pool, dma, size); + dma_sync_single_for_device(pool->dev, dma, size, DMA_BIDIRECTIONAL); } /* Masks for xdp_umem_page flags. diff --git a/drivers/net/ethernet/engleder/tsnep_main.c b/drivers/net/ethernet/engleder/tsnep_main.c index 4b15af6b7122..44da335d66bd 100644 --- a/drivers/net/ethernet/engleder/tsnep_main.c +++ b/drivers/net/ethernet/engleder/tsnep_main.c @@ -1587,7 +1587,7 @@ static int tsnep_rx_poll_zc(struct tsnep_rx *rx, struct napi_struct *napi, length = __le32_to_cpu(entry->desc_wb->properties) & TSNEP_DESC_LENGTH_MASK; xsk_buff_set_size(entry->xdp, length - ETH_FCS_LEN); - xsk_buff_dma_sync_for_cpu(entry->xdp, rx->xsk_pool); + xsk_buff_dma_sync_for_cpu(entry->xdp); /* RX metadata with timestamps is in front of actual data, * subtract metadata size to get length of actual data and diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c index 051748b997f3..a466c2379146 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-xsk.c @@ -55,7 +55,7 @@ static u32 dpaa2_xsk_run_xdp(struct dpaa2_eth_priv *priv, xdp_set_data_meta_invalid(xdp_buff); xdp_buff->rxq = &ch->xdp_rxq; - xsk_buff_dma_sync_for_cpu(xdp_buff, ch->xsk_pool); + xsk_buff_dma_sync_for_cpu(xdp_buff); xdp_act = bpf_prog_run_xdp(xdp_prog, xdp_buff); /* xdp.data pointer may have changed */ diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 11500003af0d..d20ce517426e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -483,7 +483,7 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) bi = *i40e_rx_bi(rx_ring, next_to_process); xsk_buff_set_size(bi, size); - xsk_buff_dma_sync_for_cpu(bi, rx_ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi); if (!first) first = bi; diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c index 8a051420fa19..0d3d56aca424 100644 --- a/drivers/net/ethernet/intel/ice/ice_xsk.c +++ b/drivers/net/ethernet/intel/ice/ice_xsk.c @@ -878,7 +878,7 @@ int ice_clean_rx_irq_zc(struct ice_rx_ring *rx_ring, int budget) ICE_RX_FLX_DESC_PKT_LEN_M; xsk_buff_set_size(xdp, size); - xsk_buff_dma_sync_for_cpu(xdp, xsk_pool); + xsk_buff_dma_sync_for_cpu(xdp); if (!first) { first = xdp; diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index c3fe62813f43..6e3ce15053b1 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -2817,7 +2817,7 @@ static int igc_clean_rx_irq_zc(struct igc_q_vector *q_vector, const int budget) } bi->xdp->data_end = bi->xdp->data + size; - xsk_buff_dma_sync_for_cpu(bi->xdp, ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi->xdp); res = __igc_xdp_run_prog(adapter, prog, bi->xdp); switch (res) { diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c index 59798bc33298..ebda0cebe910 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c @@ -304,7 +304,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, } bi->xdp->data_end = bi->xdp->data + size; - xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool); + xsk_buff_dma_sync_for_cpu(bi->xdp); xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp); if (likely(xdp_res & (IXGBE_XDP_TX | IXGBE_XDP_REDIR))) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c index b8dd74453655..1b7132fa70de 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c @@ -270,7 +270,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, /* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ mxbuf->cqe = cqe; xsk_buff_set_size(&mxbuf->xdp, cqe_bcnt); - xsk_buff_dma_sync_for_cpu(&mxbuf->xdp, rq->xsk_pool); + xsk_buff_dma_sync_for_cpu(&mxbuf->xdp); net_prefetch(mxbuf->xdp.data); /* Possible flows: @@ -319,7 +319,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq, /* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ mxbuf->cqe = cqe; xsk_buff_set_size(&mxbuf->xdp, cqe_bcnt); - xsk_buff_dma_sync_for_cpu(&mxbuf->xdp, rq->xsk_pool); + xsk_buff_dma_sync_for_cpu(&mxbuf->xdp); net_prefetch(mxbuf->xdp.data); prog = rcu_dereference(rq->xdp_prog); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index d601b5faaed5..5e5d9fd0bfd5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -917,7 +917,7 @@ INDIRECT_CALLABLE_SCOPE bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq) if (!rq->xsk_pool) { count = mlx5e_refill_rx_wqes(rq, head, wqe_bulk); - } else if (likely(!rq->xsk_pool->dma_need_sync)) { + } else if (likely(dma_skip_sync(rq->pdev))) { mlx5e_xsk_free_rx_wqes(rq, head, wqe_bulk); count = mlx5e_xsk_alloc_rx_wqes_batched(rq, head, wqe_bulk); } else { diff --git a/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c b/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c index 45be6954d5aa..01cfa9cc1b5e 100644 --- a/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c +++ b/drivers/net/ethernet/netronome/nfp/nfd3/xsk.c @@ -184,7 +184,7 @@ nfp_nfd3_xsk_rx(struct nfp_net_rx_ring *rx_ring, int budget, xrxbuf->xdp->data += meta_len; xrxbuf->xdp->data_end = xrxbuf->xdp->data + pkt_len; xdp_set_data_meta_invalid(xrxbuf->xdp); - xsk_buff_dma_sync_for_cpu(xrxbuf->xdp, r_vec->xsk_pool); + xsk_buff_dma_sync_for_cpu(xrxbuf->xdp); net_prefetch(xrxbuf->xdp->data); if (meta_len) { diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index ae2ffa9595d6..d988f5ac8166 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -5364,7 +5364,7 @@ static int stmmac_rx_zc(struct stmmac_priv *priv, int limit, u32 queue) /* RX buffer is good and fit into a XSK pool buffer */ buf->xdp->data_end = buf->xdp->data + buf1_len; - xsk_buff_dma_sync_for_cpu(buf->xdp, rx_q->xsk_pool); + xsk_buff_dma_sync_for_cpu(buf->xdp); prog = READ_ONCE(priv->xdp_prog); res = __stmmac_xdp_run_prog(priv, prog, buf->xdp); diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index ce60ecd48a4d..ecea2a329b1d 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -338,7 +338,6 @@ static struct xsk_dma_map *xp_create_dma_map(struct device *dev, struct net_devi dma_map->netdev = netdev; dma_map->dev = dev; - dma_map->dma_need_sync = false; dma_map->dma_pages_cnt = nr_pages; refcount_set(&dma_map->users, 1); list_add(&dma_map->list, &umem->xsk_dma_list); @@ -424,7 +423,6 @@ static int xp_init_dma_info(struct xsk_buff_pool *pool, struct xsk_dma_map *dma_ pool->dev = dma_map->dev; pool->dma_pages_cnt = dma_map->dma_pages_cnt; - pool->dma_need_sync = dma_map->dma_need_sync; memcpy(pool->dma_pages, dma_map->dma_pages, pool->dma_pages_cnt * sizeof(*pool->dma_pages)); @@ -460,8 +458,6 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev, __xp_dma_unmap(dma_map, attrs); return -ENOMEM; } - if (dma_need_sync(dev, dma)) - dma_map->dma_need_sync = true; dma_map->dma_pages[i] = dma; } @@ -557,11 +553,9 @@ struct xdp_buff *xp_alloc(struct xsk_buff_pool *pool) xskb->xdp.data_meta = xskb->xdp.data; xskb->xdp.flags = 0; - if (pool->dma_need_sync) { - dma_sync_single_range_for_device(pool->dev, xskb->dma, 0, - pool->frame_len, - DMA_BIDIRECTIONAL); - } + dma_sync_single_for_device(pool->dev, xskb->dma, pool->frame_len, + DMA_BIDIRECTIONAL); + return &xskb->xdp; } EXPORT_SYMBOL(xp_alloc); @@ -633,7 +627,7 @@ u32 xp_alloc_batch(struct xsk_buff_pool *pool, struct xdp_buff **xdp, u32 max) { u32 nb_entries1 = 0, nb_entries2; - if (unlikely(pool->dma_need_sync)) { + if (unlikely(!dma_skip_sync(pool->dev))) { struct xdp_buff *buff; /* Slow path */ @@ -693,18 +687,3 @@ dma_addr_t xp_raw_get_dma(struct xsk_buff_pool *pool, u64 addr) (addr & ~PAGE_MASK); } EXPORT_SYMBOL(xp_raw_get_dma); - -void xp_dma_sync_for_cpu_slow(struct xdp_buff_xsk *xskb) -{ - dma_sync_single_range_for_cpu(xskb->pool->dev, xskb->dma, 0, - xskb->pool->frame_len, DMA_BIDIRECTIONAL); -} -EXPORT_SYMBOL(xp_dma_sync_for_cpu_slow); - -void xp_dma_sync_for_device_slow(struct xsk_buff_pool *pool, dma_addr_t dma, - size_t size) -{ - dma_sync_single_range_for_device(pool->dev, dma, 0, - size, DMA_BIDIRECTIONAL); -} -EXPORT_SYMBOL(xp_dma_sync_for_device_slow);