From patchwork Mon Feb 5 11:04:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 196753 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp813795dyb; Mon, 5 Feb 2024 03:36:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IGpKzy4leMauSoQtfibFIOKKDSHyc/qPZ1tiXrvzN2J9oww/TuVmFRSh8NX8/CEMfICa2B1 X-Received: by 2002:a05:6358:6e95:b0:176:9bfd:d08d with SMTP id q21-20020a0563586e9500b001769bfdd08dmr16265898rwm.11.1707133016319; Mon, 05 Feb 2024 03:36:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707133016; cv=pass; d=google.com; s=arc-20160816; b=rb0hP6h4DTg77vF4kQSuQLn6efXyioFgv1ZUOOXiauxGuKg7vhP8al6YYiW+IP+Iby n+5HmO22tzYbUXiwFRlhrRLHmE7UZ/dvy/Pm6S5j8yrQHO4QxRsZranH9wj1FULWCZhi PZKGgdvmk0rP1wcSGza3jVuGZmGu5agkk52TEVcuyagS0n9B9jtwk6wdpHALF4D87LXx ZtQVu1aTB6JvpXjdGSmSHqBnMAilOHAfTbdKuZYoj2cMgAZSsNh7IPo9axeOku3C2xQm Qi5lUK7K8gg1FNavWCFCzgr/owl8tDsbj52QnUK0hpHBk7SmJX33m3+Y/D2I37lwpDWx Bd6g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=AV/c1cWWIWUkdT8pnihfHEAqVlIV0cM5qidI+eui4fg=; fh=fg+zuvzw4pM0GbH5VG/X8QpDscjf+hwjV4TDfqhKdKA=; b=t2JFK899+6iFOeT8P4nbLitXhTatCeAS7VyYhUwiQ0Xd99VEwwOmS8hgaSlcUaB+LD mODJoa/3GasSZbgh0Kj2yb0L9R0e2uRxJ2KQIK6gxpLEivevkLCEWtU/fF2MMrdESCkb C7itXipWK63PQw4+UBStQeOf51nftJXvrE+QlMDMMPk4obeEcBjyxxg2ZM3MX3HouEAq tQ7JXkY4DHmh2JxZNQM5jhbroPYqk4o5krIT4SCRk9f5/OdDKPnzZ7KqPvUH7W31L/+g wH+xUT/n4T4iR6E8REGPs6Cxk6lnuIh2xRmKJCEcWJAvkoPukQymgfWCowPQ7i/Nx0rQ /+PA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkWhEaVb; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=1; AJvYcCW4vvEsfdn9tbv3jNaw7w5UPXfT+7OBDhTbvqQOV0SSSX53jQE9m+2BH5qMd8OJIxfID+lgigu5FuIZGIcQSn61AzmV7g== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id s71-20020a63774a000000b005dbf1fcc429si3387005pgc.162.2024.02.05.03.36.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 03:36:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SkWhEaVb; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-52454-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 2E896B28726 for ; Mon, 5 Feb 2024 11:07:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C9F971BDCD; Mon, 5 Feb 2024 11:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SkWhEaVb" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D3931B964; Mon, 5 Feb 2024 11:05:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131134; cv=none; b=pmkpeQq7Juq41QBOsGWI3lysJAB7eGhWmAbdVYWrtaH97PC/seyEJ1wB2i4THVN+QMDSHiiMhSXWnBq38lOxR1dhF0Jp8wtYr+SxRIposH23la+OKzF6rKjKNh7uveFFDO0J38OebDZJseBzcTDeSUz0u+TwUKXaX6LzCITwiBA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707131134; c=relaxed/simple; bh=j5bCxYGboU2nsbzGInCNZJ3ihY7kPTLpx8cHouNYyPI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SRyCQQwbQmnMgTFW+xxh4FZVKTht6/AO26MlNqK1JkD9Me/0x8B731uuHhObE+T0H07D/cm4AgNPRi+5WOkwfO6rU1b1ldujSbh4kO0cQ6YPMrEGkswOb3kbUBjTQPHMtBgOhQ1w1SviHnh66fKelX+3lXE/HgOfQHO+HZ2dcT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SkWhEaVb; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707131132; x=1738667132; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j5bCxYGboU2nsbzGInCNZJ3ihY7kPTLpx8cHouNYyPI=; b=SkWhEaVbVUJyrNvLim5AORJbEsuJnWCH3c1X+xCOwSHdIKQRc0v4kykV ZxIAOES6AnOJKygByB26fiSNJecg4NQN9NXeGhb2ZttJ7xgTMmAjAYpIs kOGxXjYnOYTh0J3k8j0v7qL5JnvPfHPYOBqgT2EwvU9ZnO+giCN1lRroM GcMV5dFBXwxnCo8jGiq6C9oYB6HnMhKz6ZFfoBgfcL4fxT2Cz/cvGq/Th A5KC65CWGhuuuHku1+4QONXGS5jkNaauyGaKErNCq8pwn6Y9aZWsirey/ FRXEFsXJHvieYhtF99qGLd2LrMGTP86sTiv5rVVJOo/7JjcW5Mtaaq1bh A==; X-IronPort-AV: E=McAfee;i="6600,9927,10974"; a="25945508" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="25945508" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 03:05:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5328262" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa004.fm.intel.com with ESMTP; 05 Feb 2024 03:05:28 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 6/7] page_pool: check for DMA sync shortcut earlier Date: Mon, 5 Feb 2024 12:04:25 +0100 Message-ID: <20240205110426.764393-7-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240205110426.764393-1-aleksander.lobakin@intel.com> References: <20240205110426.764393-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790058709631276367 X-GMAIL-MSGID: 1790058709631276367 We can save a couple more function calls in the Page Pool code if we check for dma_skip_sync() earlier, just when we test pp->p.dma_sync. Move both these checks into an inline wrapper and call the PP wrapper over the generic DMA sync function only when both are true. You can't cache the result of dma_skip_sync() in &page_pool, as it may change anytime if an SWIOTLB buffer is allocated or mapped. Signed-off-by: Alexander Lobakin --- net/core/page_pool.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 2c353906407c..d2b411095131 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -369,16 +369,24 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) return page; } -static void page_pool_dma_sync_for_device(struct page_pool *pool, - struct page *page, - unsigned int dma_sync_size) +static void __page_pool_dma_sync_for_device(struct page_pool *pool, + struct page *page, + u32 dma_sync_size) { dma_addr_t dma_addr = page_pool_get_dma_addr(page); dma_sync_size = min(dma_sync_size, pool->p.max_len); - dma_sync_single_range_for_device(pool->p.dev, dma_addr, - pool->p.offset, dma_sync_size, - pool->p.dma_dir); + __dma_sync_single_range_for_device(pool->p.dev, dma_addr, + pool->p.offset, dma_sync_size, + pool->p.dma_dir); +} + +static __always_inline void +page_pool_dma_sync_for_device(struct page_pool *pool, struct page *page, + u32 dma_sync_size) +{ + if (pool->dma_sync && !dma_skip_sync(pool->p.dev)) + __page_pool_dma_sync_for_device(pool, page, dma_sync_size); } static bool page_pool_dma_map(struct page_pool *pool, struct page *page) @@ -400,8 +408,7 @@ static bool page_pool_dma_map(struct page_pool *pool, struct page *page) if (page_pool_set_dma_addr(page, dma)) goto unmap_failed; - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, pool->p.max_len); + page_pool_dma_sync_for_device(pool, page, pool->p.max_len); return true; @@ -665,9 +672,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, if (likely(page_ref_count(page) == 1 && !page_is_pfmemalloc(page))) { /* Read barrier done in page_ref_count / READ_ONCE */ - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, - dma_sync_size); + page_pool_dma_sync_for_device(pool, page, dma_sync_size); if (allow_direct && in_softirq() && page_pool_recycle_in_cache(page, pool)) @@ -778,8 +783,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { - if (pool->dma_sync) - page_pool_dma_sync_for_device(pool, page, -1); + page_pool_dma_sync_for_device(pool, page, -1); return page; }