From patchwork Wed Feb 14 16:21:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 201042 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1348122dyb; Wed, 14 Feb 2024 08:51:11 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVXMJUe0atV/HORHfrK6zWmqQUEK/bf9oMi98Zxwb+A7AuQd1r1YwqEZHQt4aB4HQ8dfsPlHjlejnQbVmrvWldiKLp3Hw== X-Google-Smtp-Source: AGHT+IH97jwFXFbuXBFdOA8vkorWAxyFk9s48FVfdH2Z00d+k3s9viMeE9THvYJFk3FY20jRLh2D X-Received: by 2002:a17:90a:e506:b0:296:1ee:a54d with SMTP id t6-20020a17090ae50600b0029601eea54dmr3099414pjy.48.1707929471463; Wed, 14 Feb 2024 08:51:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707929471; cv=pass; d=google.com; s=arc-20160816; b=RXjKer4SkfKs+JNm2Apn8L34pS1pShOSKMDimJ7+I9rQ9Hoigq3pt6K5vZr8t6DhVv B4l+IlFmsUjU5Vt7gAdypge4n3q377kwIxONHxS4HSU+cjlmarTe5umgUgAFlkmK7YPo zRHIV/o7wxx2V8OCe4JKvSgyh4FoT4oUq4VTLqOD2k7gRkl3djoSHROhxupzL3GWVNJH 6fWQZgqE2w87+kK3fEYxGvBHXoLk6gd1oy7f5ml9tVDxK2izNeODrvx0/vuv3CfizY+T Xd95e+/g7rQLUyQO+J8UXyhpJULNW55YGrk8qUdZuPyy2Cd4VIIDhn2V9JITNlLv3Qdo MnUw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=OemP9CZ81RJmmITlED5kKKv7ot0ld8HBt6qj4vXDcs4=; fh=p9UVfHZrpOpaw7yYkEMPSqtecQpykP9kauv+FmIg5io=; b=MSVijkXdgjNf6HuVKLtY70Xzp2SK0tYWr2ARQVBYnj1nfWndWZRb5nG606RMzgS5o6 7aTCj9+ENYOKOQxX0jlKS5OWJpfJ1Jtxd4ZhPW24N/QRTof/AFJcXRswKIORdGUIHjk6 Sv2Rxs/sw+ppSeSfOoOFu/Yvp2o3iLIW/ozxibuSRI2JxaGszK20ymcaR+oeC/Dy30bS KWaY0pj4mT++/QGYkLMldRMPhSnbkFfXT/V3auZO8Xbubs9SgomtoXiCAJO0DSazoREa nEhY8OoJTzaXsSNDpBEDml5G1BkkEpdbC+I2wlz17BMEjA0gk+KA6bQehXV9/vQmNz7y BdFQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ctzg4LqY; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCWHx5hdLiLSKv54Jozs3jGJ9gEWBpVCAe3N0g1me/pJ13JG3PosjnNKVa4DiodQ90kSZGPRmMXdxlnX3uJrdoT45K1B/Q== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id f15-20020a17090aec8f00b0029619430cebsi1406384pjy.155.2024.02.14.08.51.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 08:51:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ctzg4LqY; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65516-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E497CB22561 for ; Wed, 14 Feb 2024 16:25:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CF3C361672; Wed, 14 Feb 2024 16:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Ctzg4LqY" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9960D612C1; Wed, 14 Feb 2024 16:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927780; cv=none; b=fWZ5opUfZEFL8stLMRcvUBC11qqP31X1DqgvkC5YEobcwhfALaLyaQg3rlGPUpuxidCnKbFhWRlDqTjMLUQnKxkYYhRVU9Lykg6KoJHvbpk9CkHiZQc1MFazznyhrN8jaecBQeE7fMrW31bzav+OD4elVP5YbtQnlBO13v2NqWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707927780; c=relaxed/simple; bh=2jYWs6cVfcBDrhRNnxCoaJhB0s1tUX2UFowyoAY2brs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nlzF1/Y2NNmX+07zv8oHCieFRfLHCFEm3xUZDF/7lO2UFNGDhwXkca5etvuwUtt6QQy6TXUH5Z7wYF1o+HaCXzPOSmEdsVu2hhQ73jqXL6YN4By/TINvRitPT7CGdIheQXxu7GXa+heEVfQxHTjKAYJ7A8FbU/bpRUj3Mvc8QMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Ctzg4LqY; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707927778; x=1739463778; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2jYWs6cVfcBDrhRNnxCoaJhB0s1tUX2UFowyoAY2brs=; b=Ctzg4LqYepotQaI30P7e9QCC/QcFQ9Wv0O17xfPz+2ELGyOeAF33Oa7U e1f80RrFykaFwkDWfoc8Kjy21JnHC+JkKZJzmlTsiGwOg+GcWaVO/MgmK gRlhLa+YGsEVChke0L3ADtIT9Eq/tD1yoTO/DF+NGl5fYwIdVnc+ZxAxV Zd0Jk3oo57yhAzbbvM9IrrG6NZyN6EL8lfgl+Al1+XhCGEzSCGBOGPNCS k1CNM1J/h8I2UIxK9vZlsIjbTAsRaYPJHqYBkyk5feEwC1knw7dtcRcLx CsUKsQ8Qy8dT91nS672v54uVqgQ2W+gQFinA5saE2NqEvMd7hSh/scm5o A==; X-IronPort-AV: E=McAfee;i="6600,9927,10984"; a="5755588" X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="5755588" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Feb 2024 08:22:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,159,1705392000"; d="scan'208";a="26400020" Received: from newjersey.igk.intel.com ([10.102.20.203]) by fmviesa002.fm.intel.com with ESMTP; 14 Feb 2024 08:22:54 -0800 From: Alexander Lobakin To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Alexander Lobakin , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , Will Deacon , Greg Kroah-Hartman , "Rafael J. Wysocki" , Magnus Karlsson , Maciej Fijalkowski , Alexander Duyck , bpf@vger.kernel.org, netdev@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH net-next v3 4/7] page_pool: make sure frag API fields don't span between cachelines Date: Wed, 14 Feb 2024 17:21:58 +0100 Message-ID: <20240214162201.4168778-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240214162201.4168778-1-aleksander.lobakin@intel.com> References: <20240214162201.4168778-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790893853073900823 X-GMAIL-MSGID: 1790893853073900823 After commit 5027ec19f104 ("net: page_pool: split the page_pool_params into fast and slow") that made &page_pool contain only "hot" params at the start, cacheline boundary chops frag API fields group in the middle again. To not bother with this each time fast params get expanded or shrunk, let's just align them to `4 * sizeof(long)`, the closest upper pow-2 to their actual size (2 longs + 1 int). This ensures 16-byte alignment for the 32-bit architectures and 32-byte alignment for the 64-bit ones, excluding unnecessary false-sharing. ::page_state_hold_cnt is used quite intensively on hotpath no matter if frag API is used, so move it to the newly created hole in the first cacheline. Signed-off-by: Alexander Lobakin --- include/net/page_pool/types.h | 12 +++++++++++- net/core/page_pool.c | 9 +++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 3828396ae60c..b2b93bec7bce 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -129,12 +129,22 @@ struct page_pool { struct page_pool_params_fast p; int cpuid; + u32 pages_state_hold_cnt; bool has_init_callback; + /* The following block must stay within one cacheline. On 32-bit + * systems, sizeof(long) == sizeof(int), so that the block size is + * ``3 * sizeof(long)``. On 64-bit systems, the actual size is + * ``2 * sizeof(long) + sizeof(int)``. The closest pow-2 to both of + * them is ``4 * sizeof(long)``, so just use that one for simplicity. + * Having it aligned to a cacheline boundary may be excessive and + * doesn't bring any good. + */ + __cacheline_group_begin(frag) __aligned(4 * sizeof(long)); long frag_users; struct page *frag_page; unsigned int frag_offset; - u32 pages_state_hold_cnt; + __cacheline_group_end(frag); struct delayed_work release_dw; void (*disconnect)(void *pool); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 89c835fcf094..3c464852e228 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -170,12 +170,21 @@ static void page_pool_producer_unlock(struct page_pool *pool, spin_unlock_bh(&pool->ring.producer_lock); } +static void page_pool_struct_check(void) +{ + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_users); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_page); + CACHELINE_ASSERT_GROUP_MEMBER(struct page_pool, frag, frag_offset); +} + static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params, int cpuid) { unsigned int ring_qsize = 1024; /* Default */ + page_pool_struct_check(); + memcpy(&pool->p, ¶ms->fast, sizeof(pool->p)); memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow));