From patchwork Thu Oct 27 20:43:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Safonov X-Patchwork-Id: 11949 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp447840wru; Thu, 27 Oct 2022 13:45:50 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7XSuiFK4IRQmu1Q0gOsgmadjzXsiiLpdNjGNVEQqXAe3lfJHQSEHYmGAJSD3bHoS3ChL6f X-Received: by 2002:a05:6402:35d1:b0:45d:3661:567e with SMTP id z17-20020a05640235d100b0045d3661567emr48393738edc.343.1666903550808; Thu, 27 Oct 2022 13:45:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666903550; cv=none; d=google.com; s=arc-20160816; b=bVRT/k0i2fCyo6vKA6XlP0uo34imx7nktEwe1LITd3y1PZrw/F0JB2/r8IRRWpQZEA b/fH/jkHjbtzwSPH2NBZ8dkp+KWcmObu5m4oW2KpVHpl9m8o+ODVoH3xein9SvjAXdWa 7S2u7Ql6FsxM9vszlTtz0xotdbmlIpX+2xC+OofDo7nuBlH1V0zcpaNgRvQof+HVh612 +ibSNNqH8h0744hIfNZUJphVFNh2cYxg33+5mYeAs4K7jCnLe8nS9y8DsEOTGiyI7Pb7 QFlIg4fPdK5CFrJ0vMQexfez4mQyptscGHEgYDWDpp7Q3MNo+/4eSy+sZJCghHHT/kvF jJig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RxfT3udHjY/1MFp4/+Y9TLXrCrikupuEdzT2r1qucRk=; b=Htw8E6ZXS2SK2hPOzH7ZOT/5MK5fpxmiMSSLFPBLFB+YxFsPXTn3Di791bVsJPsq7s E7frqXEw+UXnGpi5IX6IDH9v/cJAawEgMHy6T/gjXFMEKJULy9pSplfoQKd5ru9/Nlk7 o8Qygq7JvVUL6U//H3d76BUjKOxVG8BNOb2lG+2XRBLp7LH5ttwL+vYEDX+QSZkNGRpU RSZVI1QYfyWxL0uqHusrwSWHijiHVib6guOteo15Ga3ebAQgw8Cx8FFry+/XtRrsGZiM otIJtxZZTU1dzTd3Nz1L4nvxVLfBI1hvj2xv1gAavrnCJl8lZ8SaGnYKsy7jj+sEe5Wx 2Nfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=WObTEuSp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d7-20020a056402516700b00461f5bb2b79si2332920ede.458.2022.10.27.13.45.24; Thu, 27 Oct 2022 13:45:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=WObTEuSp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236785AbiJ0UoJ (ORCPT + 99 others); Thu, 27 Oct 2022 16:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236638AbiJ0UoB (ORCPT ); Thu, 27 Oct 2022 16:44:01 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF3678B2C9 for ; Thu, 27 Oct 2022 13:43:59 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id o4so4161438wrq.6 for ; Thu, 27 Oct 2022 13:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RxfT3udHjY/1MFp4/+Y9TLXrCrikupuEdzT2r1qucRk=; b=WObTEuSpCZ6kMlzBm/Y1HjE9FvJSeykn2DUTcsVTC2eDycf2ZDGPUcrb6r7L/423El ZuPxAEamLFG8SeagtCXwBy5pSPP+SPHLPCAZ69QkfrsuhUAbe8J3usllgZj0Wb0pak7D NvVxE45Ve5j4+4at2dVO+/HKBYzr1h1+M4dDQtm3VDf0fEnOOhBaQRI4rBw80CwPGew8 5GnuzdG1g15c5nYNOEiTVTrMlgrL4s9YKXC0J6XyJKcYiXhBSzPr/nY7HlQVFjw+fvvx aMFfDbEp+su/jwS7NkHAVa08sXoo//f0GGNa0Zbsa6mccpRHxksq0jQcP4biMrmUBdgj uUuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RxfT3udHjY/1MFp4/+Y9TLXrCrikupuEdzT2r1qucRk=; b=bsu8+oBOrf0ByfjigaEMHxVGAR+p4zoHRi2+edzEGSlkQzoiUeX9v5Fs9K4jHGR+Rp mtETPSI5vwNnERegQfnq3FUAugcnT6OZc7MQVuQcdKnJPaKi1O+YWbIQCNQ70h80bmYY CIoDrIvl1cKzjiLBUj25Nv7yiQfz78SQ3i//0rruu2o+wdTw+acMl+1NwgHu2WRiykeb ETIll1M/Ab7hR5hJwgHT3TrwAERyKkl8LdFLze9QFOejeFkLoom2tm246NfKYJxO/c8P zVvZK13d0AyiV2LryfXdZWgrWWIUcXRZkeVIa1OX+JrzMO089Y/Mj5acoDrTNMrXRlg7 lrAw== X-Gm-Message-State: ACrzQf2pPrpvWbZPYLyGMWtuX6YjoUQL13OrnbFP5XGrDNXCCK3HUhRg rJQHyCKryX4w74R3XavFUfRD0OjCm4Z7iOKt X-Received: by 2002:a05:6000:811:b0:236:622d:a7d6 with SMTP id bt17-20020a056000081100b00236622da7d6mr19178175wrb.258.1666903438049; Thu, 27 Oct 2022 13:43:58 -0700 (PDT) Received: from Mindolluin.ire.aristanetworks.com ([217.173.96.166]) by smtp.gmail.com with ESMTPSA id n3-20020a5d6b83000000b00236644228besm1968739wrx.40.2022.10.27.13.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Oct 2022 13:43:57 -0700 (PDT) From: Dmitry Safonov To: linux-kernel@vger.kernel.org, David Ahern , Eric Dumazet Cc: Dmitry Safonov , Andy Lutomirski , Ard Biesheuvel , Bob Gilligan , Dan Carpenter , "David S. Miller" , Dmitry Safonov <0x7f454c46@gmail.com>, Eric Biggers , "Eric W. Biederman" , Francesco Ruggeri , Herbert Xu , Hideaki YOSHIFUJI , Ivan Delalande , Jakub Kicinski , Leonard Crestez , Paolo Abeni , Salam Noureddine , Shuah Khan , netdev@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 02/36] crypto_pool: Add crypto_pool_reserve_scratch() Date: Thu, 27 Oct 2022 21:43:13 +0100 Message-Id: <20221027204347.529913-3-dima@arista.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221027204347.529913-1-dima@arista.com> References: <20221027204347.529913-1-dima@arista.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747875057516105363?= X-GMAIL-MSGID: =?utf-8?q?1747875057516105363?= Instead of having build-time hardcoded constant, reallocate scratch area, if needed by user. Different algos, different users may need different size of temp per-CPU buffer. Only up-sizing supported for simplicity. Signed-off-by: Dmitry Safonov --- crypto/Kconfig | 6 ++++ crypto/crypto_pool.c | 77 ++++++++++++++++++++++++++++++++++--------- include/crypto/pool.h | 3 +- 3 files changed, 69 insertions(+), 17 deletions(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 768d331e626b..e002cd321e79 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1397,6 +1397,12 @@ config CRYPTO_POOL help Per-CPU pool of crypto requests ready for usage in atomic contexts. +config CRYPTO_POOL_DEFAULT_SCRATCH_SIZE + hex "Per-CPU default scratch area size" + depends on CRYPTO_POOL + default 0x100 + range 0x100 0x10000 + if !KMSAN # avoid false positives from assembly if ARM source "arch/arm/crypto/Kconfig" diff --git a/crypto/crypto_pool.c b/crypto/crypto_pool.c index 37131952c5a7..2f1deb3f5218 100644 --- a/crypto/crypto_pool.c +++ b/crypto/crypto_pool.c @@ -1,13 +1,14 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include +#include #include #include #include #include #include -static unsigned long scratch_size = DEFAULT_CRYPTO_POOL_SCRATCH_SZ; +static unsigned long scratch_size = CONFIG_CRYPTO_POOL_DEFAULT_SCRATCH_SIZE; static DEFINE_PER_CPU(void *, crypto_pool_scratch); struct crypto_pool_entry { @@ -22,26 +23,69 @@ static struct crypto_pool_entry cpool[CPOOL_SIZE]; static unsigned int cpool_populated; static DEFINE_MUTEX(cpool_mutex); -static int crypto_pool_scratch_alloc(void) +/* Slow-path */ +/** + * crypto_pool_reserve_scratch - re-allocates scratch buffer, slow-path + * @size: request size for the scratch/temp buffer + */ +#define FREE_BATCH_SIZE 64 +int crypto_pool_reserve_scratch(unsigned long size) { - int cpu; - - lockdep_assert_held(&cpool_mutex); + void *free_batch[FREE_BATCH_SIZE]; + int cpu, err = 0; + unsigned int i = 0; + mutex_lock(&cpool_mutex); + if (size == scratch_size) { + for_each_possible_cpu(cpu) { + if (per_cpu(crypto_pool_scratch, cpu)) + continue; + goto allocate_scratch; + } + mutex_unlock(&cpool_mutex); + return 0; + } +allocate_scratch: + size = max(size, scratch_size); + cpus_read_lock(); for_each_possible_cpu(cpu) { - void *scratch = per_cpu(crypto_pool_scratch, cpu); + void *scratch, *old_scratch; - if (scratch) + scratch = kmalloc_node(size, GFP_KERNEL, cpu_to_node(cpu)); + if (!scratch) { + err = -ENOMEM; + break; + } + + old_scratch = per_cpu(crypto_pool_scratch, cpu); + /* Pairs with crypto_pool_get() */ + WRITE_ONCE(*per_cpu_ptr(&crypto_pool_scratch, cpu), scratch); + if (!cpu_online(cpu)) { + kfree(old_scratch); continue; + } + free_batch[i++] = old_scratch; + if (i == FREE_BATCH_SIZE) { + cpus_read_unlock(); + synchronize_rcu(); + while (i > 0) + kfree(free_batch[--i]); + cpus_read_lock(); + } + } + cpus_read_unlock(); + if (!err) + scratch_size = size; + mutex_unlock(&cpool_mutex); - scratch = kmalloc_node(scratch_size, GFP_KERNEL, - cpu_to_node(cpu)); - if (!scratch) - return -ENOMEM; - per_cpu(crypto_pool_scratch, cpu) = scratch; + if (i > 0) { + synchronize_rcu(); + while (i > 0) + kfree(free_batch[--i]); } - return 0; + return err; } +EXPORT_SYMBOL_GPL(crypto_pool_reserve_scratch); static void crypto_pool_scratch_free(void) { @@ -138,7 +182,6 @@ int crypto_pool_alloc_ahash(const char *alg) /* slow-path */ mutex_lock(&cpool_mutex); - for (i = 0; i < cpool_populated; i++) { if (cpool[i].alg && !strcmp(cpool[i].alg, alg)) { if (kref_read(&cpool[i].kref) > 0) { @@ -263,7 +306,11 @@ int crypto_pool_get(unsigned int id, struct crypto_pool *c) return -EINVAL; } ret->req = *this_cpu_ptr(cpool[id].req); - ret->base.scratch = this_cpu_read(crypto_pool_scratch); + /* + * Pairs with crypto_pool_reserve_scratch(), scartch area is + * valid (allocated) until crypto_pool_put(). + */ + ret->base.scratch = READ_ONCE(*this_cpu_ptr(&crypto_pool_scratch)); return 0; } EXPORT_SYMBOL_GPL(crypto_pool_get); diff --git a/include/crypto/pool.h b/include/crypto/pool.h index 2c61aa45faff..c7d817860cc3 100644 --- a/include/crypto/pool.h +++ b/include/crypto/pool.h @@ -4,8 +4,6 @@ #include -#define DEFAULT_CRYPTO_POOL_SCRATCH_SZ 128 - struct crypto_pool { void *scratch; }; @@ -20,6 +18,7 @@ struct crypto_pool_ahash { struct ahash_request *req; }; +int crypto_pool_reserve_scratch(unsigned long size); int crypto_pool_alloc_ahash(const char *alg); void crypto_pool_add(unsigned int id); void crypto_pool_release(unsigned int id);