From patchwork Thu Feb 22 02:06:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 204516 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp1418326dyc; Wed, 21 Feb 2024 18:06:57 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWSObnNT9NV5Monj04snzMdd40g3eSgMv8oR9dPaGoeZD7Bg/Tw43fRil4Bt1suygSPCylcFJesck1pYDSnt8c69kh9AA== X-Google-Smtp-Source: AGHT+IEPA8Hb9hjE8FsWUNXdbH64ucQLgLjdi8vR5war/0y0b/b37wMrsMxVEog6FlofQ8MyuLj/ X-Received: by 2002:a19:7407:0:b0:512:9e9f:2f1d with SMTP id v7-20020a197407000000b005129e9f2f1dmr9792690lfe.58.1708567617282; Wed, 21 Feb 2024 18:06:57 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708567617; cv=pass; d=google.com; s=arc-20160816; b=eJWNhMMKXJzU09AxpVUXWhaaWcFJ0wXDLuBaCkjRB8/swoHHgJIx8roiBUhlUNwHkb qU1L8b2lFzR+ms/XBQ36kgKHkdznvKK8HX6Ba0XYOqd0drHtyfRfMafG2A227ph7f4gf 7licrWnafByb8sIekI3SxuFO5rHMZFJrP3ppVNe1FiQf0FmSCDALNedTn7F8Y0o2d5sx eVDoqmBosjRR34Eo505wvxexXdVwZcB5watflvXTu0PGJfl7C04J0BypdoDa/qg1Np2x A3tiEhSujn96/iIx2Ednxq+F19nW7yJax6y7OB05kHvPzrNuICq1sP+7EvqKWVdM0Xu6 q8mg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=w64efItMlr6vgLfvpO0/b9Hh1Adb3EB97Cl6D2DGNWI=; fh=aidW9zUGPLK+UX+MFiPjjMTi3iA/3dw4si2m6JNhz/c=; b=flzmLw6NFdTY7P8PIqUC9UZ77kZUm/uxvVMplM2gtwM34MqU/CuCuXYyS74ki4QAIM mgIpjUKCIPDyMCy78tu21ykPGJ7gQsad6ZUXUwASgg2r2jC5fdqS3nGUKmt66fZgxmcn B7wnBUu5to671qlwsy9lzdp3tXN4hAIw3RSunePh6I07Q6wcyj3IxmKhU3X5+DEUQGS2 jdT+Abudl/U5iwBRAKtQ5WAwulIcnijcbFzdvRlUzvUWTIu8DD8iQix6eUbIPueLgRsJ 7QCChHhvSvIcAVmNJOcKQmTLAeVC0qrDDBVAOJZ2QHDAHmIBDCEpYEbyP8IGF5HZKP/r GgcA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="T/EWDyal"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-75784-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-75784-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id l25-20020a170906415900b00a3f69d73c7dsi443852ejk.361.2024.02.21.18.06.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 18:06:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-75784-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="T/EWDyal"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-75784-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-75784-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A32091F217AC for ; Thu, 22 Feb 2024 02:06:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 97FF3168D8; Thu, 22 Feb 2024 02:06:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T/EWDyal" Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38CAD17548; Thu, 22 Feb 2024 02:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708567598; cv=none; b=RTT4Vzg4j8mX2aR8C2ymLg1/AG9Shuaz2TRvzb8yTe+A/XECwR5G0smq9GiCsyEc2t5DlIgMY6CrdrgzvQztAnMi9CJajV7R6PxlG4mrqFAu0IP2GR8klhHOyB38g6iU924x5RzoMUrXI7soPzCHggZ6S9XK8SvaJmUdnYWuf58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708567598; c=relaxed/simple; bh=Hgwue3VsYVuGAHidF8xeWMBthAFTNIHrbSPwF1lYYYo=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=D1dzXI3DAb4efQUl2lsTnXG5a0GrbuD0Nl0Hbu21MhXQsxpqD0CnyP/rarpjbz7P7AcQnmtLqz29wbOrFTIaRgRvIAgmLeCCiyLGRdPh2z+60pXfQJTQ65jTqTYrdLfOK6Na078sDsp3e06aOK5mYguXieesQc583qMezgjh+uQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=T/EWDyal; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-299540b4d18so3182486a91.1; Wed, 21 Feb 2024 18:06:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708567596; x=1709172396; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=w64efItMlr6vgLfvpO0/b9Hh1Adb3EB97Cl6D2DGNWI=; b=T/EWDyalSUo5nA2o13oq7qtrKgyZR7TSPb+LqN0FjeujB9sQkfcKJ3kwL3kRUAJt9R TY4U4e8O6uD9gnG3h1wMQLE+wwBgDt03pk+RqNbZ0Fi6XfhxaevbEZXS9hN7/D0qoOeb 3fNzjrEOqvBHLDY8HdzOq3phMfjq95yEEvvi0bL4dOgx68WTElaiIObEdUK1RXj+jYfL pAIuwmzFln9dz06/CsU7in8jXinFmKuKztcCbU6RdRBqp6oHesqYiwWd1jV9hURzCMzM aSQkvrBjzccvm6TPyt4uD1sCMPEc4eVZQtZoW3xC3B3Q2dnLdxdiJ3PWEZ7Yy6n0Z52G Txbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708567596; x=1709172396; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=w64efItMlr6vgLfvpO0/b9Hh1Adb3EB97Cl6D2DGNWI=; b=imPxQOFQCWsD5PEtz2lGzu8uX+i7BAxRTSIHo+aAcDjcT1S2a4z81GG4rK79V7S8mv QQmijE/g8NSb5x5yRBarqkTpKgNVNz1hDZFmTH6wHiT3vnlQ30habAbxSUeaPUn8t6br l1iAi0xIR13b2kXTkKBG06OfqQ0DgmUI3mxKAlE/5W3fU2kiZbuY5GpRILUVT18txuSC QQ8T2zAQedGfTDwSuU1x+9Kz/mb3Yf8Pq3brPdat8Xd3ZqCPjrRvgXlSDdKPSpR8X4h5 OQ8m4FNcPuuR6GxZyE62I0oOZsV1TdrKqD4XzvwBQVXz6QTkQrDd2frXA2CCHQNbrEV8 PoyQ== X-Forwarded-Encrypted: i=1; AJvYcCUQsm/CG2YjLpuLVn/JkyourYbmKiZIilq8ArmD9w4JOk4/UL18/vqcKTpe+FL8uiiUr0BNa6ZjpbMiKe6QnUOKJlnbhGrSTXl2C4PUBzZbOnPvzmJ9g9Rt+lL5n4mGmbtzRvp5sppEfAPA X-Gm-Message-State: AOJu0Yy0b8mgBfuqoXaemJ+50Il5DxvBjkaxDF9mkt3ZpUTVyJescVp+ h9MwyF/4EiE7HMH6bLQNXWswjSzvz2vlS1j7XuwiGwuXcK/4fdYm X-Received: by 2002:a17:90b:344f:b0:299:2038:11e5 with SMTP id lj15-20020a17090b344f00b00299203811e5mr15017066pjb.27.1708567596274; Wed, 21 Feb 2024 18:06:36 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:3b18:a2e2:f00b:c965]) by smtp.gmail.com with ESMTPSA id sl6-20020a17090b2e0600b00298ca3a93f1sm2650070pjb.4.2024.02.21.18.06.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 18:06:35 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: herbert@gondor.apana.org.au, davem@davemloft.net, linux-crypto@vger.kernel.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, chriscli@google.com, chrisl@kernel.org, sjenning@redhat.com, vitaly.wool@konsulko.com, Barry Song , Nhat Pham , Yosry Ahmed , Chengming Zhou Subject: [PATCH v6] crypto: scompress: remove memcpy if sg_nents is 1 and pages are lowmem Date: Thu, 22 Feb 2024 15:06:17 +1300 Message-Id: <20240222020617.31041-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791562997678126957 X-GMAIL-MSGID: 1791562997678126957 From: Barry Song while sg_nents is 1, which is always true for the current kernel as the only user - zswap is this case, we might have a chance to remove memcpy, thus improve the performance. Though sg_nents is 1, its buffer might cross two pages. If those pages are highmem, we have no cheap way to map them to contiguous virtual address because kmap doesn't support more than one page (kmap single higmem page could be still expensive for tlb) and vmap is expensive. So we also test and enure page is not highmem in order to safely use page_to_virt before removing the memcpy. The good news is that in the most majority of cases, we are lowmem, and we are always lowmem in those modern and popular hardware. Cc: Johannes Weiner Cc: Nhat Pham Cc: Yosry Ahmed Signed-off-by: Barry Song Tested-by: Chengming Zhou --- -v6: * separate it from zswap & crypto series[1] according to Herbert; * make sure the code is still safe when buffers cross two pages to address Herbert's comment [1] https://lore.kernel.org/linux-mm/20240220064414.262582-1-21cnbao@gmail.com/ crypto/scompress.c | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/crypto/scompress.c b/crypto/scompress.c index b108a30a7600..185d2359f28b 100644 --- a/crypto/scompress.c +++ b/crypto/scompress.c @@ -117,6 +117,7 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) struct crypto_scomp *scomp = *tfm_ctx; void **ctx = acomp_request_ctx(req); struct scomp_scratch *scratch; + void *src, *dst; unsigned int dlen; int ret; @@ -134,13 +135,25 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) scratch = raw_cpu_ptr(&scomp_scratch); spin_lock(&scratch->lock); - scatterwalk_map_and_copy(scratch->src, req->src, 0, req->slen, 0); + if (sg_nents(req->src) == 1 && !PageHighMem(sg_page(req->src))) { + src = page_to_virt(sg_page(req->src)) + req->src->offset; + } else { + scatterwalk_map_and_copy(scratch->src, req->src, 0, + req->slen, 0); + src = scratch->src; + } + + if (req->dst && sg_nents(req->dst) == 1 && !PageHighMem(sg_page(req->dst))) + dst = page_to_virt(sg_page(req->dst)) + req->dst->offset; + else + dst = scratch->dst; + if (dir) - ret = crypto_scomp_compress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_compress(scomp, src, req->slen, + dst, &req->dlen, *ctx); else - ret = crypto_scomp_decompress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_decompress(scomp, src, req->slen, + dst, &req->dlen, *ctx); if (!ret) { if (!req->dst) { req->dst = sgl_alloc(req->dlen, GFP_ATOMIC, NULL); @@ -152,8 +165,12 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) ret = -ENOSPC; goto out; } - scatterwalk_map_and_copy(scratch->dst, req->dst, 0, req->dlen, - 1); + if (dst == scratch->dst) { + scatterwalk_map_and_copy(scratch->dst, req->dst, 0, + req->dlen, 1); + } else { + flush_dcache_page(sg_page(req->dst)); + } } out: spin_unlock(&scratch->lock);