From patchwork Wed Jan 18 21:41:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Safonov X-Patchwork-Id: 45517 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2577712wrn; Wed, 18 Jan 2023 13:44:44 -0800 (PST) X-Google-Smtp-Source: AMrXdXsOu64wzwGTXhrfxiETxBNgScXL5lrX7Y7IfQH8OI3EEbwpvrzD7uGFW+vFv+0y7LqVeSN5 X-Received: by 2002:a17:907:a585:b0:872:ec40:65e9 with SMTP id vs5-20020a170907a58500b00872ec4065e9mr8063853ejc.18.1674078284327; Wed, 18 Jan 2023 13:44:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674078284; cv=none; d=google.com; s=arc-20160816; b=MPX3FEKKFLs314ATmHji6cp2V1pDXHHp+R3XHJ4kmpyRoR152fSa2V9eHFm87aclFw xaYmi/YAfym0C7bV7PV8OEcLValvw2pNFXbz/ZN5xT8zXTWGKd1lrUNHhz2y1nF7FBC7 Fq9XQeHsgoMuG71QwnPJCJwSALijSUZvbnenUr4XmMhI6f/faMicxOOGr5yVH1wJlRJx MVxMz27xYm/8duco3uy2PtZQ4ZPLKmM3ELauMXseJd4lQM+7F5bN04sG+uELWxupE0+F WkNKa/u3vQft7m62gfdYj6/rcND3NwNLmRylI5rrziq3/NkLrZ1EJ3wuQG6A9MJkuUKr PgxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0pMAF7eLOpf48HDHSGi29DRHvDztNug+IGFby4/8CII=; b=rtjoIzxlFREZ5GTnKj/nTMr3tlNdYmVgcswvluNZOC410qiilDU/zSAWpc8cs0xDUq h61/JG6jK1j+1Cj/4oVS5jalcLTfeQjTP+fuXfkcfjgH1xOIMFPebjorjeE506vUWS0K C1Prqcec8+JQHTJly2cqiVXGtj4M71dEiYNXwPHoqCCIy+Mp0Cx9UVtCSe+pnP4IDbvx 1J+So8CEiZrbTubkR7zDEPUb9WMorLT9DvWdk58jRMBdCtH4ka16/n80bpB+9uFNUXgB ItMh3uRTvhyoYsyyoYLLfV8D49JnUQTHavhelDL5Yk1fVuwG/0Mm7Qe301OF0J+qe9KO At0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=eldce92x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f14-20020a056402194e00b00499e52b07d0si14976512edz.65.2023.01.18.13.44.20; Wed, 18 Jan 2023 13:44:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=eldce92x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231199AbjARVl3 (ORCPT + 99 others); Wed, 18 Jan 2023 16:41:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231159AbjARVlY (ORCPT ); Wed, 18 Jan 2023 16:41:24 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 145C623108 for ; Wed, 18 Jan 2023 13:41:22 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id n7so6399wrx.5 for ; Wed, 18 Jan 2023 13:41:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0pMAF7eLOpf48HDHSGi29DRHvDztNug+IGFby4/8CII=; b=eldce92xmDUkAwn5ojEy4APbipBNx8mgDPZYTycUGyalea/bbOA+8CybcRIua6S8Pa t49YwyhOXDoAqfOS2YfYKCydYYPMujgrEOBgUgBbJb9eL30yv0BlVfvqieOpblNd1AmX gr6pheUhMa1xLfrDSC8P62jHxts2H0GwPg2ZVFjF7hlEQ2cKRK6CGxNorQNONNnLXD/q d1bF1kVpN0f/8m/71xI+YW9gI1JvUiFsfpoIQiU6jv+HGQdOup0R1GvHwcRtgLoF91MQ SR/HX6VjxQ6MoPRajStynJxwVfTJO4lUyqSq6gYFoSuMd1hdRDOtcDOnHQYSxBVgtDem s0gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0pMAF7eLOpf48HDHSGi29DRHvDztNug+IGFby4/8CII=; b=AKcgGCWSqJn/eXqR40rO5T3Jr5e1y7+luEmEA+p1jDKVzFtvacj2m0daEbkQnAiD41 1+HKBgbb2sQdpP7FluWhy8PlnIncKO2v+wnSrX5aWQAJdWuws9CZuRmn2RWlQkr0cWok dkwxGmDhIfA47+jsBHLEqbDdpLzfxDNMsny4xDq/nbyPV+deLDGEprkc8ySZUPyffef7 HeOWOiq4kdSe79bO5mcoA7V0ZUKxOacfONwwYaFtpjL2Q3m6CEzuYYmjOWUu+3Omd/Fq dtelbMvR93ufWGBUfwGqoE8RPmBdqubDDUj08dFzgKH6T2joebXV4ZDnTQUCoWOCIv+Y G7WQ== X-Gm-Message-State: AFqh2kp13YRccXkhEtRKf76tM8jrzw1RZ2RAk+PSqkK44wv1ircd394S EY/7ySiWqwnunkf0yGPx34jJZxPa+O3D/EIi X-Received: by 2002:a05:6000:603:b0:2bd:f651:3a67 with SMTP id bn3-20020a056000060300b002bdf6513a67mr8614999wrb.64.1674078080361; Wed, 18 Jan 2023 13:41:20 -0800 (PST) Received: from Mindolluin.ire.aristanetworks.com ([217.173.96.166]) by smtp.gmail.com with ESMTPSA id m5-20020a056000024500b00267bcb1bbe5sm33186349wrz.56.2023.01.18.13.41.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 13:41:19 -0800 (PST) From: Dmitry Safonov To: linux-kernel@vger.kernel.org, David Ahern , Eric Dumazet , Herbert Xu , Jakub Kicinski , "David S. Miller" Cc: Dmitry Safonov , Andy Lutomirski , Bob Gilligan , Dmitry Safonov <0x7f454c46@gmail.com>, Hideaki YOSHIFUJI , Leonard Crestez , Paolo Abeni , Salam Noureddine , netdev@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v4 1/4] crypto: Introduce crypto_pool Date: Wed, 18 Jan 2023 21:41:08 +0000 Message-Id: <20230118214111.394416-2-dima@arista.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118214111.394416-1-dima@arista.com> References: <20230118214111.394416-1-dima@arista.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755398311190433293?= X-GMAIL-MSGID: =?utf-8?q?1755398311190433293?= Introduce a per-CPU pool of async crypto requests that can be used in bh-disabled contexts (designed with net RX/TX softirqs as users in mind). Allocation can sleep and is a slow-path. Initial implementation has only ahash as a backend and a fix-sized array of possible algorithms used in parallel. Signed-off-by: Dmitry Safonov --- crypto/Kconfig | 3 + crypto/Makefile | 1 + crypto/crypto_pool.c | 333 ++++++++++++++++++++++++++++++++++++++++++ include/crypto/pool.h | 46 ++++++ 4 files changed, 383 insertions(+) create mode 100644 crypto/crypto_pool.c create mode 100644 include/crypto/pool.h diff --git a/crypto/Kconfig b/crypto/Kconfig index 9c86f7045157..7096654419cb 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1388,6 +1388,9 @@ endmenu config CRYPTO_HASH_INFO bool +config CRYPTO_POOL + tristate + if !KMSAN # avoid false positives from assembly if ARM source "arch/arm/crypto/Kconfig" diff --git a/crypto/Makefile b/crypto/Makefile index d0126c915834..eed8f61bc93b 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_CRYPTO_ACOMP2) += crypto_acompress.o cryptomgr-y := algboss.o testmgr.o obj-$(CONFIG_CRYPTO_MANAGER2) += cryptomgr.o +obj-$(CONFIG_CRYPTO_POOL) += crypto_pool.o obj-$(CONFIG_CRYPTO_USER) += crypto_user.o crypto_user-y := crypto_user_base.o crypto_user-$(CONFIG_CRYPTO_STATS) += crypto_user_stat.o diff --git a/crypto/crypto_pool.c b/crypto/crypto_pool.c new file mode 100644 index 000000000000..0237dad937b8 --- /dev/null +++ b/crypto/crypto_pool.c @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include +#include +#include +#include +#include +#include +#include + +static size_t __scratch_size; +static DEFINE_PER_CPU(void __rcu *, crypto_pool_scratch); + +struct crypto_pool_entry { + struct ahash_request * __percpu *req; + const char *alg; + struct kref kref; + bool needs_key; +}; + +#define CPOOL_SIZE (PAGE_SIZE/sizeof(struct crypto_pool_entry)) +static struct crypto_pool_entry cpool[CPOOL_SIZE]; +static unsigned int cpool_populated; +static DEFINE_MUTEX(cpool_mutex); + +/* Slow-path */ +struct scratches_to_free { + struct rcu_head rcu; + unsigned int cnt; + void *scratches[]; +}; +static void free_old_scratches(struct rcu_head *head) +{ + struct scratches_to_free *stf; + + stf = container_of(head, struct scratches_to_free, rcu); + while (stf->cnt--) + kfree(stf->scratches[stf->cnt]); + kfree(stf); +} +/* + * crypto_pool_reserve_scratch - re-allocates scratch buffer, slow-path + * @size: request size for the scratch/temp buffer + */ +static int crypto_pool_reserve_scratch(size_t size) +{ + struct scratches_to_free *stf; + size_t stf_sz = struct_size(stf, scratches, num_possible_cpus()); + int cpu, err = 0; + + lockdep_assert_held(&cpool_mutex); + if (__scratch_size >= size) + return 0; + + stf = kmalloc(stf_sz, GFP_KERNEL); + if (!stf) + return -ENOMEM; + stf->cnt = 0; + + cpus_read_lock(); + for_each_possible_cpu(cpu) { + void *scratch, *old_scratch; + + scratch = kmalloc_node(size, GFP_KERNEL, cpu_to_node(cpu)); + if (!scratch) { + err = -ENOMEM; + break; + } + + old_scratch = rcu_replace_pointer(per_cpu(crypto_pool_scratch, cpu), scratch, lockdep_is_held(&cpool_mutex)); + if (!cpu_online(cpu) || !old_scratch) { + kfree(old_scratch); + continue; + } + stf->scratches[stf->cnt++] = old_scratch; + } + cpus_read_unlock(); + if (!err) + __scratch_size = size; + + call_rcu(&stf->rcu, free_old_scratches); + return err; +} + +static void crypto_pool_scratch_free(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + kfree(rcu_replace_pointer(per_cpu(crypto_pool_scratch, cpu), + NULL, lockdep_is_held(&cpool_mutex))); + __scratch_size = 0; +} + +static int __cpool_alloc_ahash(struct crypto_pool_entry *e, const char *alg) +{ + struct crypto_ahash *hash, *cpu0_hash; + int cpu, ret = -ENOMEM; + + e->alg = kstrdup(alg, GFP_KERNEL); + if (!e->alg) + return -ENOMEM; + + e->req = alloc_percpu(struct ahash_request *); + if (!e->req) + goto out_free_alg; + + cpu0_hash = crypto_alloc_ahash(alg, 0, CRYPTO_ALG_ASYNC); + if (IS_ERR(cpu0_hash)) { + ret = PTR_ERR(cpu0_hash); + goto out_free_req; + } + + /* If hash has .setkey(), allocate ahash per-CPU, not only request */ + e->needs_key = crypto_ahash_get_flags(cpu0_hash) & CRYPTO_TFM_NEED_KEY; + + hash = cpu0_hash; + for_each_possible_cpu(cpu) { + struct ahash_request *req; + + /* + * If ahash has a key - it has to be allocated per-CPU. + * In such case re-use for CPU0 hash that just have been + * allocated above. + */ + if (!hash) + hash = crypto_alloc_ahash(alg, 0, CRYPTO_ALG_ASYNC); + if (IS_ERR(hash)) + goto out_free_per_cpu; + + req = ahash_request_alloc(hash, GFP_KERNEL); + if (!req) + goto out_free_hash; + + ahash_request_set_callback(req, 0, NULL, NULL); + + *per_cpu_ptr(e->req, cpu) = req; + + if (e->needs_key) + hash = NULL; + } + kref_init(&e->kref); + return 0; + +out_free_hash: + if (hash != cpu0_hash) + crypto_free_ahash(hash); + +out_free_per_cpu: + for_each_possible_cpu(cpu) { + struct ahash_request *req = *per_cpu_ptr(e->req, cpu); + struct crypto_ahash *pcpu_hash; + + if (req == NULL) + break; + pcpu_hash = crypto_ahash_reqtfm(req); + ahash_request_free(req); + /* hash per-CPU, e->needs_key == true */ + if (pcpu_hash != cpu0_hash) + crypto_free_ahash(pcpu_hash); + } + + crypto_free_ahash(cpu0_hash); +out_free_req: + free_percpu(e->req); +out_free_alg: + kfree(e->alg); + e->alg = NULL; + return ret; +} + +/** + * crypto_pool_alloc_ahash - allocates pool for ahash requests + * @alg: name of async hash algorithm + * @scratch_size: reserve a crypto_pool::scratch buffer of this size + */ +int crypto_pool_alloc_ahash(const char *alg, size_t scratch_size) +{ + int i, ret; + + /* slow-path */ + mutex_lock(&cpool_mutex); + ret = crypto_pool_reserve_scratch(scratch_size); + if (ret) + goto out; + for (i = 0; i < cpool_populated; i++) { + if (cpool[i].alg && !strcmp(cpool[i].alg, alg)) { + if (kref_read(&cpool[i].kref) > 0) + kref_get(&cpool[i].kref); + else + kref_init(&cpool[i].kref); + ret = i; + goto out; + } + } + + for (i = 0; i < cpool_populated; i++) { + if (!cpool[i].alg) + break; + } + if (i >= CPOOL_SIZE) { + ret = -ENOSPC; + goto out; + } + + ret = __cpool_alloc_ahash(&cpool[i], alg); + if (!ret) { + ret = i; + if (i == cpool_populated) + cpool_populated++; + } +out: + mutex_unlock(&cpool_mutex); + return ret; +} +EXPORT_SYMBOL_GPL(crypto_pool_alloc_ahash); + +static void __cpool_free_entry(struct crypto_pool_entry *e) +{ + struct crypto_ahash *hash = NULL; + int cpu; + + for_each_possible_cpu(cpu) { + if (*per_cpu_ptr(e->req, cpu) == NULL) + continue; + + hash = crypto_ahash_reqtfm(*per_cpu_ptr(e->req, cpu)); + ahash_request_free(*per_cpu_ptr(e->req, cpu)); + if (e->needs_key) { + crypto_free_ahash(hash); + hash = NULL; + } + } + if (hash) + crypto_free_ahash(hash); + free_percpu(e->req); + kfree(e->alg); + memset(e, 0, sizeof(*e)); +} + +static void cpool_cleanup_work_cb(struct work_struct *work) +{ + unsigned int i; + bool free_scratch = true; + + mutex_lock(&cpool_mutex); + for (i = 0; i < cpool_populated; i++) { + if (kref_read(&cpool[i].kref) > 0) { + free_scratch = false; + continue; + } + if (!cpool[i].alg) + continue; + __cpool_free_entry(&cpool[i]); + } + if (free_scratch) + crypto_pool_scratch_free(); + mutex_unlock(&cpool_mutex); +} + +static DECLARE_WORK(cpool_cleanup_work, cpool_cleanup_work_cb); +static void cpool_schedule_cleanup(struct kref *kref) +{ + schedule_work(&cpool_cleanup_work); +} + +/** + * crypto_pool_release - decreases number of users for a pool. If it was + * the last user of the pool, releases any memory that was consumed. + * @id: crypto_pool that was previously allocated by crypto_pool_alloc_ahash() + */ +void crypto_pool_release(unsigned int id) +{ + if (WARN_ON_ONCE(id > cpool_populated || !cpool[id].alg)) + return; + + /* slow-path */ + kref_put(&cpool[id].kref, cpool_schedule_cleanup); +} +EXPORT_SYMBOL_GPL(crypto_pool_release); + +/** + * crypto_pool_get - increases number of users (refcounter) for a pool + * @id: crypto_pool that was previously allocated by crypto_pool_alloc_ahash() + */ +void crypto_pool_get(unsigned int id) +{ + if (WARN_ON_ONCE(id > cpool_populated || !cpool[id].alg)) + return; + kref_get(&cpool[id].kref); +} +EXPORT_SYMBOL_GPL(crypto_pool_get); + +int crypto_pool_start(unsigned int id, struct crypto_pool *c) +{ + struct crypto_pool_ahash *ret = (struct crypto_pool_ahash *)c; + + rcu_read_lock_bh(); + if (WARN_ON_ONCE(id > cpool_populated || !cpool[id].alg)) { + rcu_read_unlock_bh(); + return -EINVAL; + } + ret->req = *this_cpu_ptr(cpool[id].req); + /* + * Pairs with crypto_pool_reserve_scratch(), scratch area is + * valid (allocated) until crypto_pool_end(). + */ + ret->base.scratch = rcu_dereference_bh(*this_cpu_ptr(&crypto_pool_scratch)); + return 0; +} +EXPORT_SYMBOL_GPL(crypto_pool_start); + +/** + * crypto_pool_algo - return algorithm of crypto_pool + * @id: crypto_pool that was previously allocated by crypto_pool_alloc_ahash() + * @buf: buffer to return name of algorithm + * @buf_len: size of @buf + */ +size_t crypto_pool_algo(unsigned int id, char *buf, size_t buf_len) +{ + size_t ret = 0; + + /* slow-path */ + mutex_lock(&cpool_mutex); + if (cpool[id].alg) + ret = strscpy(buf, cpool[id].alg, buf_len); + mutex_unlock(&cpool_mutex); + return ret; +} +EXPORT_SYMBOL_GPL(crypto_pool_algo); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Per-CPU pool of crypto requests"); diff --git a/include/crypto/pool.h b/include/crypto/pool.h new file mode 100644 index 000000000000..e266c1cba7de --- /dev/null +++ b/include/crypto/pool.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _CRYPTO_POOL_H +#define _CRYPTO_POOL_H + +#include + +/** + * struct crypto_pool - generic type for different crypto requests + * @scratch: per-CPU temporary area, that can be used between + * crypto_pool_start() and crypto_pool_end() to perform + * crypto requests + */ +struct crypto_pool { + void *scratch; +}; + +/** + * struct crypto_pool_ahash - per-CPU pool of ahash_requests + * @base: common members that can be used by any async crypto ops + * @req: pre-allocated ahash request + */ +struct crypto_pool_ahash { + struct crypto_pool base; + struct ahash_request *req; +}; + +int crypto_pool_alloc_ahash(const char *alg, size_t scratch_size); +void crypto_pool_get(unsigned int id); +void crypto_pool_release(unsigned int id); + +/** + * crypto_pool_start - disable bh and start using crypto_pool + * @id: crypto_pool that was previously allocated by crypto_pool_alloc_ahash() + * @c: returned crypto_pool for usage (uninitialized on failure) + */ +int crypto_pool_start(unsigned int id, struct crypto_pool *c); +/** + * crypto_pool_end - enable bh and stop using crypto_pool + */ +static inline void crypto_pool_end(void) +{ + rcu_read_unlock_bh(); +} +size_t crypto_pool_algo(unsigned int id, char *buf, size_t buf_len); + +#endif /* _CRYPTO_POOL_H */ From patchwork Wed Jan 18 21:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Safonov X-Patchwork-Id: 45518 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2577906wrn; Wed, 18 Jan 2023 13:45:22 -0800 (PST) X-Google-Smtp-Source: AMrXdXuzVMrZw8QIMJ54vHW7zflgEIzWsy195krmCMt8QHoHfFiSTKwdXkgKa4B4l8Of0RZA67sK X-Received: by 2002:a05:6402:3898:b0:45c:835b:ac64 with SMTP id fd24-20020a056402389800b0045c835bac64mr9987641edb.31.1674078322454; Wed, 18 Jan 2023 13:45:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674078322; cv=none; d=google.com; s=arc-20160816; b=Txs5jqPSMFEwZXt3/6Z3wk3bdxplYhz8rVOQvvhNJsXlsr292/Nche7yq8kqLKXWH+ r/jm9nE4efXFguL7oqUeeuWu2R+FV7nHdeZl7sP5bkxzP/NENRaPjeKXBw9Ir7h91uMM K3yl1bMCRdz4PBb6L2coIvjvHnzVLztPraiTefpSNgNCh0BqYuZNRL+UlKTT7L7S6Bfv nX/Qt3P3amlsCbNcXH9foGKcxAmJprmPRo7wW12d52FwfxpoOm1YHVBK3lFtSxI07QyE Af+vgHKzb4MI3BStmJcx+cMIV65zAVP5u/KlPJH3WNf5nAObYPVAWPDRBkHOxOIawCAi v32A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W9IuZSSGRVZfqc0uwsTQaNNx3sjy+h0W9pWP413z0cw=; b=ctQJqGD1n7Tay8b8sMnQI/TmLUWjRkedNUpG39ssDE4Q2dYzyi/a3iPOZI6HttpF3i FdTj8B6cmyJPV914fsq6r/ixHmiaQJQluiFHl9dxbzVGyMbxY8tbTVIIQQ2cbI0s7TWI +kTUMcGOQ5h+t/bqZyfRx86DLzKNkFOG5spocqIsTxQ8iU19jV7zydbfucwGWkGWDRtl EixzvKJs675BxPgSttCp1M40dzl57bQ7kkyzWGfW52xx8tp6LmZlBANcyvmtzZEHla4v wXXEwSNe5bzD+iDVPrHwZPBXrw3cKuFiOsDfgZvvULBzP6AOZ4fcte4XNP2FD2f9HQUA Xpag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=dU25ql4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w14-20020a05640234ce00b0049e439020c1si5014680edc.11.2023.01.18.13.44.28; Wed, 18 Jan 2023 13:45:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=dU25ql4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231223AbjARVmD (ORCPT + 99 others); Wed, 18 Jan 2023 16:42:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231194AbjARVlZ (ORCPT ); Wed, 18 Jan 2023 16:41:25 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85DE0BBB9 for ; Wed, 18 Jan 2023 13:41:23 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id r9so10711wrw.4 for ; Wed, 18 Jan 2023 13:41:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W9IuZSSGRVZfqc0uwsTQaNNx3sjy+h0W9pWP413z0cw=; b=dU25ql4SICWAuGG+5RC/8lb6ZexXBfFXjnYwj2uKb0rgX1p9S2mgSjNJs6CiceN1Ub xZSSVuocIKqSYsoX1i7L9DzWhIuV+jg+DlT5kmJMsI6tdrymbzxpPO6pCVvtX14gWtIV yxkIlxiv9Pv4jdYAfZEmKExmK7mcAKYqvyZvsIm/HsFRayLhSUGo0OClA1ZEDJiVcxI4 awbHzfvPIJIuOrb7FmutK7Nhi/+oDeJAzampliwCVjdM+60nCcuY5V1a1/l/2qQeX1Uj 9+T2OF3itpFnrxbVOfDx99BPJmLRcehVhXeeaaiDK12toLVMyYeGBsv5Q1iAgeanFK3+ xSPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W9IuZSSGRVZfqc0uwsTQaNNx3sjy+h0W9pWP413z0cw=; b=3Q10u4I2WOa0fDXNFTcy9TJxSWwrE6rgYV8ACIgm/CmlQCSdVf67VYyE9phS4x8JVb BnzfWcAW6Is6RvA1Y/Dq0wiqH3uH9/cjBdYKDl+pIixlKHfW38l/ohNV0kbmqOyHWtpR N1x+WF8E7+3MufocB/2q/o6O+gBNM3tDzNsGZDKZHTSVmMbnnnyvkKmfm0ihhERnCq7x R63bPbHex37T1JWJiml6WxWcmYySV7LTw5Dz3GAY8AQtqYH6j8j8URQWq+MlJZQV9Gvt n9bO+DkBDgnVM5l0GoZSruWWwVpcbBUrtPm9eGmrD9NrQA4r800gaOzVIdIsgY/ukh5I 3qwA== X-Gm-Message-State: AFqh2krgG6AxNhh11PBsSDb+cQlrE2TZY4S/QyC76ivaMKo91bpTj1il LBP1mXB/ItKpZrBEPSuXFfDh9ztASkGLtVNh X-Received: by 2002:a05:6000:1f14:b0:2bd:c484:1b01 with SMTP id bv20-20020a0560001f1400b002bdc4841b01mr7686472wrb.53.1674078081638; Wed, 18 Jan 2023 13:41:21 -0800 (PST) Received: from Mindolluin.ire.aristanetworks.com ([217.173.96.166]) by smtp.gmail.com with ESMTPSA id m5-20020a056000024500b00267bcb1bbe5sm33186349wrz.56.2023.01.18.13.41.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 13:41:21 -0800 (PST) From: Dmitry Safonov To: linux-kernel@vger.kernel.org, David Ahern , Eric Dumazet , Herbert Xu , Jakub Kicinski , "David S. Miller" Cc: Dmitry Safonov , Andy Lutomirski , Bob Gilligan , Dmitry Safonov <0x7f454c46@gmail.com>, Hideaki YOSHIFUJI , Leonard Crestez , Paolo Abeni , Salam Noureddine , netdev@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v4 2/4] crypto/net/tcp: Use crypto_pool for TCP-MD5 Date: Wed, 18 Jan 2023 21:41:09 +0000 Message-Id: <20230118214111.394416-3-dima@arista.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118214111.394416-1-dima@arista.com> References: <20230118214111.394416-1-dima@arista.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755398350986614908?= X-GMAIL-MSGID: =?utf-8?q?1755398350986614908?= Use crypto_pool API that was designed with tcp_md5sig_pool in mind. The conversion to use crypto_pool will allow: - to reuse ahash_request(s) for different users - to allocate only one per-CPU scratch buffer rather than a new one for each user - to have a common API for net/ users that need ahash on RX/TX fast path Signed-off-by: Dmitry Safonov --- include/net/tcp.h | 24 +++------ net/ipv4/Kconfig | 1 + net/ipv4/tcp.c | 104 ++++++++++----------------------------- net/ipv4/tcp_ipv4.c | 100 +++++++++++++++++++++---------------- net/ipv4/tcp_minisocks.c | 21 +++++--- net/ipv6/tcp_ipv6.c | 61 +++++++++++------------ 6 files changed, 135 insertions(+), 176 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index db9f828e9d1e..048057cb4c2e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1664,12 +1664,6 @@ union tcp_md5sum_block { #endif }; -/* - pool: digest algorithm, hash description and scratch buffer */ -struct tcp_md5sig_pool { - struct ahash_request *md5_req; - void *scratch; -}; - /* - functions */ int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key, const struct sock *sk, const struct sk_buff *skb); @@ -1725,17 +1719,15 @@ tcp_inbound_md5_hash(const struct sock *sk, const struct sk_buff *skb, #define tcp_twsk_md5_key(twsk) NULL #endif -bool tcp_alloc_md5sig_pool(void); - -struct tcp_md5sig_pool *tcp_get_md5sig_pool(void); -static inline void tcp_put_md5sig_pool(void) -{ - local_bh_enable(); -} +struct crypto_pool_ahash; +int tcp_md5_alloc_crypto_pool(void); +void tcp_md5_release_crypto_pool(void); +void tcp_md5_add_crypto_pool(void); +extern int tcp_md5_crypto_pool_id; -int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *, const struct sk_buff *, - unsigned int header_len); -int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, +int tcp_md5_hash_skb_data(struct crypto_pool_ahash *hp, + const struct sk_buff *skb, unsigned int header_len); +int tcp_md5_hash_key(struct crypto_pool_ahash *hp, const struct tcp_md5sig_key *key); /* From tcp_fastopen.c */ diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig index 2dfb12230f08..7e851ec0fc0e 100644 --- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -744,6 +744,7 @@ config DEFAULT_TCP_CONG config TCP_MD5SIG bool "TCP: MD5 Signature Option support (RFC2385)" select CRYPTO + select CRYPTO_POOL select CRYPTO_MD5 help RFC2385 specifies a method of giving MD5 protection to TCP sessions. diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index c567d5e8053e..e226771f5985 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -244,6 +244,7 @@ #define pr_fmt(fmt) "TCP: " fmt #include +#include #include #include #include @@ -4411,98 +4412,42 @@ int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, EXPORT_SYMBOL(tcp_getsockopt); #ifdef CONFIG_TCP_MD5SIG -static DEFINE_PER_CPU(struct tcp_md5sig_pool, tcp_md5sig_pool); -static DEFINE_MUTEX(tcp_md5sig_mutex); -static bool tcp_md5sig_pool_populated = false; +int tcp_md5_crypto_pool_id = -1; +EXPORT_SYMBOL(tcp_md5_crypto_pool_id); -static void __tcp_alloc_md5sig_pool(void) +int tcp_md5_alloc_crypto_pool(void) { - struct crypto_ahash *hash; - int cpu; - - hash = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC); - if (IS_ERR(hash)) - return; - - for_each_possible_cpu(cpu) { - void *scratch = per_cpu(tcp_md5sig_pool, cpu).scratch; - struct ahash_request *req; - - if (!scratch) { - scratch = kmalloc_node(sizeof(union tcp_md5sum_block) + - sizeof(struct tcphdr), - GFP_KERNEL, - cpu_to_node(cpu)); - if (!scratch) - return; - per_cpu(tcp_md5sig_pool, cpu).scratch = scratch; - } - if (per_cpu(tcp_md5sig_pool, cpu).md5_req) - continue; - - req = ahash_request_alloc(hash, GFP_KERNEL); - if (!req) - return; - - ahash_request_set_callback(req, 0, NULL, NULL); + size_t scratch_size; + int ret; - per_cpu(tcp_md5sig_pool, cpu).md5_req = req; + scratch_size = sizeof(union tcp_md5sum_block) + sizeof(struct tcphdr); + ret = crypto_pool_alloc_ahash("md5", scratch_size); + if (ret >= 0) { + tcp_md5_crypto_pool_id = ret; + return 0; } - /* before setting tcp_md5sig_pool_populated, we must commit all writes - * to memory. See smp_rmb() in tcp_get_md5sig_pool() - */ - smp_wmb(); - /* Paired with READ_ONCE() from tcp_alloc_md5sig_pool() - * and tcp_get_md5sig_pool(). - */ - WRITE_ONCE(tcp_md5sig_pool_populated, true); + return ret; } +EXPORT_SYMBOL(tcp_md5_alloc_crypto_pool); -bool tcp_alloc_md5sig_pool(void) +void tcp_md5_release_crypto_pool(void) { - /* Paired with WRITE_ONCE() from __tcp_alloc_md5sig_pool() */ - if (unlikely(!READ_ONCE(tcp_md5sig_pool_populated))) { - mutex_lock(&tcp_md5sig_mutex); - - if (!tcp_md5sig_pool_populated) - __tcp_alloc_md5sig_pool(); - - mutex_unlock(&tcp_md5sig_mutex); - } - /* Paired with WRITE_ONCE() from __tcp_alloc_md5sig_pool() */ - return READ_ONCE(tcp_md5sig_pool_populated); + crypto_pool_release(tcp_md5_crypto_pool_id); } -EXPORT_SYMBOL(tcp_alloc_md5sig_pool); +EXPORT_SYMBOL(tcp_md5_release_crypto_pool); - -/** - * tcp_get_md5sig_pool - get md5sig_pool for this user - * - * We use percpu structure, so if we succeed, we exit with preemption - * and BH disabled, to make sure another thread or softirq handling - * wont try to get same context. - */ -struct tcp_md5sig_pool *tcp_get_md5sig_pool(void) +void tcp_md5_add_crypto_pool(void) { - local_bh_disable(); - - /* Paired with WRITE_ONCE() from __tcp_alloc_md5sig_pool() */ - if (READ_ONCE(tcp_md5sig_pool_populated)) { - /* coupled with smp_wmb() in __tcp_alloc_md5sig_pool() */ - smp_rmb(); - return this_cpu_ptr(&tcp_md5sig_pool); - } - local_bh_enable(); - return NULL; + crypto_pool_get(tcp_md5_crypto_pool_id); } -EXPORT_SYMBOL(tcp_get_md5sig_pool); +EXPORT_SYMBOL(tcp_md5_add_crypto_pool); -int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, +int tcp_md5_hash_skb_data(struct crypto_pool_ahash *hp, const struct sk_buff *skb, unsigned int header_len) { struct scatterlist sg; const struct tcphdr *tp = tcp_hdr(skb); - struct ahash_request *req = hp->md5_req; + struct ahash_request *req = hp->req; unsigned int i; const unsigned int head_data_len = skb_headlen(skb) > header_len ? skb_headlen(skb) - header_len : 0; @@ -4536,16 +4481,17 @@ int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, } EXPORT_SYMBOL(tcp_md5_hash_skb_data); -int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key) +int tcp_md5_hash_key(struct crypto_pool_ahash *hp, + const struct tcp_md5sig_key *key) { u8 keylen = READ_ONCE(key->keylen); /* paired with WRITE_ONCE() in tcp_md5_do_add */ struct scatterlist sg; sg_init_one(&sg, key->key, keylen); - ahash_request_set_crypt(hp->md5_req, &sg, NULL, keylen); + ahash_request_set_crypt(hp->req, &sg, NULL, keylen); /* We use data_race() because tcp_md5_do_add() might change key->key under us */ - return data_race(crypto_ahash_update(hp->md5_req)); + return data_race(crypto_ahash_update(hp->req)); } EXPORT_SYMBOL(tcp_md5_hash_key); diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 8320d0ecb13a..53938e080c5f 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -79,6 +79,7 @@ #include #include +#include #include #include @@ -1212,10 +1213,6 @@ static int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr, key = sock_kmalloc(sk, sizeof(*key), gfp | __GFP_ZERO); if (!key) return -ENOMEM; - if (!tcp_alloc_md5sig_pool()) { - sock_kfree_s(sk, key, sizeof(*key)); - return -ENOMEM; - } memcpy(key->key, newkey, newkeylen); key->keylen = newkeylen; @@ -1237,8 +1234,13 @@ int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr, struct tcp_sock *tp = tcp_sk(sk); if (!rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk))) { - if (tcp_md5sig_info_add(sk, GFP_KERNEL)) + if (tcp_md5_alloc_crypto_pool()) + return -ENOMEM; + + if (tcp_md5sig_info_add(sk, GFP_KERNEL)) { + tcp_md5_release_crypto_pool(); return -ENOMEM; + } if (!static_branch_inc(&tcp_md5_needed.key)) { struct tcp_md5sig_info *md5sig; @@ -1246,6 +1248,7 @@ int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr, md5sig = rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk)); rcu_assign_pointer(tp->md5sig_info, NULL); kfree_rcu(md5sig, rcu); + tcp_md5_release_crypto_pool(); return -EUSERS; } } @@ -1262,8 +1265,12 @@ int tcp_md5_key_copy(struct sock *sk, const union tcp_md5_addr *addr, struct tcp_sock *tp = tcp_sk(sk); if (!rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk))) { - if (tcp_md5sig_info_add(sk, sk_gfp_mask(sk, GFP_ATOMIC))) + tcp_md5_add_crypto_pool(); + + if (tcp_md5sig_info_add(sk, sk_gfp_mask(sk, GFP_ATOMIC))) { + tcp_md5_release_crypto_pool(); return -ENOMEM; + } if (!static_key_fast_inc_not_disabled(&tcp_md5_needed.key.key)) { struct tcp_md5sig_info *md5sig; @@ -1272,6 +1279,7 @@ int tcp_md5_key_copy(struct sock *sk, const union tcp_md5_addr *addr, net_warn_ratelimited("Too many TCP-MD5 keys in the system\n"); rcu_assign_pointer(tp->md5sig_info, NULL); kfree_rcu(md5sig, rcu); + tcp_md5_release_crypto_pool(); return -EUSERS; } } @@ -1371,7 +1379,7 @@ static int tcp_v4_parse_md5_keys(struct sock *sk, int optname, cmd.tcpm_key, cmd.tcpm_keylen); } -static int tcp_v4_md5_hash_headers(struct tcp_md5sig_pool *hp, +static int tcp_v4_md5_hash_headers(struct crypto_pool_ahash *hp, __be32 daddr, __be32 saddr, const struct tcphdr *th, int nbytes) { @@ -1379,7 +1387,7 @@ static int tcp_v4_md5_hash_headers(struct tcp_md5sig_pool *hp, struct scatterlist sg; struct tcphdr *_th; - bp = hp->scratch; + bp = hp->base.scratch; bp->saddr = saddr; bp->daddr = daddr; bp->pad = 0; @@ -1391,38 +1399,35 @@ static int tcp_v4_md5_hash_headers(struct tcp_md5sig_pool *hp, _th->check = 0; sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th)); - ahash_request_set_crypt(hp->md5_req, &sg, NULL, + ahash_request_set_crypt(hp->req, &sg, NULL, sizeof(*bp) + sizeof(*th)); - return crypto_ahash_update(hp->md5_req); + return crypto_ahash_update(hp->req); } static int tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key, __be32 daddr, __be32 saddr, const struct tcphdr *th) { - struct tcp_md5sig_pool *hp; - struct ahash_request *req; + struct crypto_pool_ahash hp; - hp = tcp_get_md5sig_pool(); - if (!hp) - goto clear_hash_noput; - req = hp->md5_req; + if (crypto_pool_start(tcp_md5_crypto_pool_id, &hp.base)) + goto clear_hash_nostart; - if (crypto_ahash_init(req)) + if (crypto_ahash_init(hp.req)) goto clear_hash; - if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2)) + if (tcp_v4_md5_hash_headers(&hp, daddr, saddr, th, th->doff << 2)) goto clear_hash; - if (tcp_md5_hash_key(hp, key)) + if (tcp_md5_hash_key(&hp, key)) goto clear_hash; - ahash_request_set_crypt(req, NULL, md5_hash, 0); - if (crypto_ahash_final(req)) + ahash_request_set_crypt(hp.req, NULL, md5_hash, 0); + if (crypto_ahash_final(hp.req)) goto clear_hash; - tcp_put_md5sig_pool(); + crypto_pool_end(); return 0; clear_hash: - tcp_put_md5sig_pool(); -clear_hash_noput: + crypto_pool_end(); +clear_hash_nostart: memset(md5_hash, 0, 16); return 1; } @@ -1431,8 +1436,7 @@ int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key, const struct sock *sk, const struct sk_buff *skb) { - struct tcp_md5sig_pool *hp; - struct ahash_request *req; + struct crypto_pool_ahash hp; const struct tcphdr *th = tcp_hdr(skb); __be32 saddr, daddr; @@ -1445,30 +1449,28 @@ int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key, daddr = iph->daddr; } - hp = tcp_get_md5sig_pool(); - if (!hp) - goto clear_hash_noput; - req = hp->md5_req; + if (crypto_pool_start(tcp_md5_crypto_pool_id, &hp.base)) + goto clear_hash_nostart; - if (crypto_ahash_init(req)) + if (crypto_ahash_init(hp.req)) goto clear_hash; - if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, skb->len)) + if (tcp_v4_md5_hash_headers(&hp, daddr, saddr, th, skb->len)) goto clear_hash; - if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2)) + if (tcp_md5_hash_skb_data(&hp, skb, th->doff << 2)) goto clear_hash; - if (tcp_md5_hash_key(hp, key)) + if (tcp_md5_hash_key(&hp, key)) goto clear_hash; - ahash_request_set_crypt(req, NULL, md5_hash, 0); - if (crypto_ahash_final(req)) + ahash_request_set_crypt(hp.req, NULL, md5_hash, 0); + if (crypto_ahash_final(hp.req)) goto clear_hash; - tcp_put_md5sig_pool(); + crypto_pool_end(); return 0; clear_hash: - tcp_put_md5sig_pool(); -clear_hash_noput: + crypto_pool_end(); +clear_hash_nostart: memset(md5_hash, 0, 16); return 1; } @@ -2285,6 +2287,18 @@ static int tcp_v4_init_sock(struct sock *sk) return 0; } +#ifdef CONFIG_TCP_MD5SIG +static void tcp_md5sig_info_free_rcu(struct rcu_head *head) +{ + struct tcp_md5sig_info *md5sig; + + md5sig = container_of(head, struct tcp_md5sig_info, rcu); + kfree(md5sig); + static_branch_slow_dec_deferred(&tcp_md5_needed); + tcp_md5_release_crypto_pool(); +} +#endif + void tcp_v4_destroy_sock(struct sock *sk) { struct tcp_sock *tp = tcp_sk(sk); @@ -2309,10 +2323,12 @@ void tcp_v4_destroy_sock(struct sock *sk) #ifdef CONFIG_TCP_MD5SIG /* Clean up the MD5 key list, if any */ if (tp->md5sig_info) { + struct tcp_md5sig_info *md5sig; + + md5sig = rcu_dereference_protected(tp->md5sig_info, 1); tcp_clear_md5_list(sk); - kfree_rcu(rcu_dereference_protected(tp->md5sig_info, 1), rcu); - tp->md5sig_info = NULL; - static_branch_slow_dec_deferred(&tcp_md5_needed); + call_rcu(&md5sig->rcu, tcp_md5sig_info_free_rcu); + rcu_assign_pointer(tp->md5sig_info, NULL); } #endif diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index e002f2e1d4f2..6fbf2d4a4a97 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -261,10 +261,9 @@ static void tcp_time_wait_init(struct sock *sk, struct tcp_timewait_sock *tcptw) tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC); if (!tcptw->tw_md5_key) return; - if (!tcp_alloc_md5sig_pool()) - goto out_free; if (!static_key_fast_inc_not_disabled(&tcp_md5_needed.key.key)) goto out_free; + tcp_md5_add_crypto_pool(); } return; out_free: @@ -349,16 +348,26 @@ void tcp_time_wait(struct sock *sk, int state, int timeo) } EXPORT_SYMBOL(tcp_time_wait); +#ifdef CONFIG_TCP_MD5SIG +static void tcp_md5_twsk_free_rcu(struct rcu_head *head) +{ + struct tcp_md5sig_key *key; + + key = container_of(head, struct tcp_md5sig_key, rcu); + kfree(key); + static_branch_slow_dec_deferred(&tcp_md5_needed); + tcp_md5_release_crypto_pool(); +} +#endif + void tcp_twsk_destructor(struct sock *sk) { #ifdef CONFIG_TCP_MD5SIG if (static_branch_unlikely(&tcp_md5_needed.key)) { struct tcp_timewait_sock *twsk = tcp_twsk(sk); - if (twsk->tw_md5_key) { - kfree_rcu(twsk->tw_md5_key, rcu); - static_branch_slow_dec_deferred(&tcp_md5_needed); - } + if (twsk->tw_md5_key) + call_rcu(&twsk->tw_md5_key->rcu, tcp_md5_twsk_free_rcu); } #endif } diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 11b736a76bd7..eb02224c7725 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -64,6 +64,7 @@ #include #include +#include #include #include @@ -672,7 +673,7 @@ static int tcp_v6_parse_md5_keys(struct sock *sk, int optname, cmd.tcpm_key, cmd.tcpm_keylen); } -static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp, +static int tcp_v6_md5_hash_headers(struct crypto_pool_ahash *hp, const struct in6_addr *daddr, const struct in6_addr *saddr, const struct tcphdr *th, int nbytes) @@ -681,7 +682,7 @@ static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp, struct scatterlist sg; struct tcphdr *_th; - bp = hp->scratch; + bp = hp->base.scratch; /* 1. TCP pseudo-header (RFC2460) */ bp->saddr = *saddr; bp->daddr = *daddr; @@ -693,39 +694,36 @@ static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp, _th->check = 0; sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th)); - ahash_request_set_crypt(hp->md5_req, &sg, NULL, + ahash_request_set_crypt(hp->req, &sg, NULL, sizeof(*bp) + sizeof(*th)); - return crypto_ahash_update(hp->md5_req); + return crypto_ahash_update(hp->req); } static int tcp_v6_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key, const struct in6_addr *daddr, struct in6_addr *saddr, const struct tcphdr *th) { - struct tcp_md5sig_pool *hp; - struct ahash_request *req; + struct crypto_pool_ahash hp; - hp = tcp_get_md5sig_pool(); - if (!hp) - goto clear_hash_noput; - req = hp->md5_req; + if (crypto_pool_start(tcp_md5_crypto_pool_id, &hp.base)) + goto clear_hash_nostart; - if (crypto_ahash_init(req)) + if (crypto_ahash_init(hp.req)) goto clear_hash; - if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2)) + if (tcp_v6_md5_hash_headers(&hp, daddr, saddr, th, th->doff << 2)) goto clear_hash; - if (tcp_md5_hash_key(hp, key)) + if (tcp_md5_hash_key(&hp, key)) goto clear_hash; - ahash_request_set_crypt(req, NULL, md5_hash, 0); - if (crypto_ahash_final(req)) + ahash_request_set_crypt(hp.req, NULL, md5_hash, 0); + if (crypto_ahash_final(hp.req)) goto clear_hash; - tcp_put_md5sig_pool(); + crypto_pool_end(); return 0; clear_hash: - tcp_put_md5sig_pool(); -clear_hash_noput: + crypto_pool_end(); +clear_hash_nostart: memset(md5_hash, 0, 16); return 1; } @@ -736,8 +734,7 @@ static int tcp_v6_md5_hash_skb(char *md5_hash, const struct sk_buff *skb) { const struct in6_addr *saddr, *daddr; - struct tcp_md5sig_pool *hp; - struct ahash_request *req; + struct crypto_pool_ahash hp; const struct tcphdr *th = tcp_hdr(skb); if (sk) { /* valid for establish/request sockets */ @@ -749,30 +746,28 @@ static int tcp_v6_md5_hash_skb(char *md5_hash, daddr = &ip6h->daddr; } - hp = tcp_get_md5sig_pool(); - if (!hp) - goto clear_hash_noput; - req = hp->md5_req; + if (crypto_pool_start(tcp_md5_crypto_pool_id, &hp.base)) + goto clear_hash_nostart; - if (crypto_ahash_init(req)) + if (crypto_ahash_init(hp.req)) goto clear_hash; - if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, skb->len)) + if (tcp_v6_md5_hash_headers(&hp, daddr, saddr, th, skb->len)) goto clear_hash; - if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2)) + if (tcp_md5_hash_skb_data(&hp, skb, th->doff << 2)) goto clear_hash; - if (tcp_md5_hash_key(hp, key)) + if (tcp_md5_hash_key(&hp, key)) goto clear_hash; - ahash_request_set_crypt(req, NULL, md5_hash, 0); - if (crypto_ahash_final(req)) + ahash_request_set_crypt(hp.req, NULL, md5_hash, 0); + if (crypto_ahash_final(hp.req)) goto clear_hash; - tcp_put_md5sig_pool(); + crypto_pool_end(); return 0; clear_hash: - tcp_put_md5sig_pool(); -clear_hash_noput: + crypto_pool_end(); +clear_hash_nostart: memset(md5_hash, 0, 16); return 1; } From patchwork Wed Jan 18 21:41:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Safonov X-Patchwork-Id: 45515 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2577132wrn; Wed, 18 Jan 2023 13:43:12 -0800 (PST) X-Google-Smtp-Source: AMrXdXvimJRcM/54KhVUWlfD5LBHFMb2oHjYAOTcWopc5nQ36cdudmq4VKrs3aCP7j4HBQwqzqaC X-Received: by 2002:a17:906:2dd6:b0:86b:119c:7ea8 with SMTP id h22-20020a1709062dd600b0086b119c7ea8mr8749844eji.49.1674078192030; Wed, 18 Jan 2023 13:43:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674078192; cv=none; d=google.com; s=arc-20160816; b=JQ5yGdJu8kT3YPYyNzHCD3Gv4gMgguFm2WK3D5hmbA0Aa147pp+8kcqozYEDm6YOF/ MKOS5GxOu8EudKnPT2n/NpNIpoopBYoZwjzx3fCDQAH3vvDunTuQRKIhpWFEvTea1a2o xDGkiwq50BHx2E56xPxlCbJdPCdNH5P4Hb+SA40nwhBhCpouVZOcn2MH4SGNnGeM3NzH TvLOr47331kZlt3qHgIlQ9zQnS75S8BfQCZAhnn6NAzsAsLe+heLPMvXCb05xtezS0es cltsKuprLv8l73T7ltCcugFpL5SZX9P781dZXmImAELaYTSbd5i+u7Bf/po10RF4fiGe G9vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3HZPKYb2PIVdmeKOK5m1pvrqCNkMbBHNc0IpoK2AHjU=; b=J+KdMVR3pFn5NfTAD13wkqagRZaLAsHQGe1cfwT1IUhF1UI+XkFLHFZakJMqG5LUBz NXslvmEGgSc5GLL53otugdZcSn4n4526bEspSj7smz2/4Qd5seselxT4VSyqsRYMWzCS vaHLPEv4WqYZs7i/c7wWHwUMoVMSKCOuV2XTLnUEwPL09Ud1jUiplftYJCpvmX7sG4KY DG1tcBAk3oCCcWB3EoBTQ9b/196gHFdrpSA8ahQmWgo0NAZv05VwPnzC0ylt6RZ32CQM xgH4YlvsIiZMpFIlLohwGf9AEeok+jezuZ8PgGwZBJedvFe++v5FQl6lsp0JNS0hrCBO MorQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=VKHWS9hx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u3-20020a170906654300b007c19a5b7697si19361565ejn.233.2023.01.18.13.42.48; Wed, 18 Jan 2023 13:43:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=VKHWS9hx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231210AbjARVlx (ORCPT + 99 others); Wed, 18 Jan 2023 16:41:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231193AbjARVlZ (ORCPT ); Wed, 18 Jan 2023 16:41:25 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA313EC65 for ; Wed, 18 Jan 2023 13:41:23 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id e3so26111296wru.13 for ; Wed, 18 Jan 2023 13:41:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3HZPKYb2PIVdmeKOK5m1pvrqCNkMbBHNc0IpoK2AHjU=; b=VKHWS9hxSMvklXJwfn/LnCRZqfrdx5Eo5olrqvSKObW/WSn3uMMG04tdASYTVH5k+C NBWdW0KR9AjLXGWFrVq0h7/KV5H4aLXcx6AqAZqBSnzlqmUeJiUS60zM0VSgJNmZaeg7 GJhndnVcMFGhJA5RD6ewKcE5XPSE7Y/LV69iv2yHm3vvcbyBVymRfSNtumnaGonO2qWn 1lYIVQZYzanD6f1xNbx86Lzxsu2MHh6VD5TIZ7zmyiyGOkUjNNSjdSe0XXekYWKXaCrP aXC928rjrNgb+FGnlQDFrLbQ0XRT11BRGv3nvaA+zBai2v7yxFAvJSgp4UvXFSGcABu4 /SNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3HZPKYb2PIVdmeKOK5m1pvrqCNkMbBHNc0IpoK2AHjU=; b=eoNfnv/SZar10aiBpTydsVDvrwggyFKAnz59T/YT7RbcehM2DL95p7CK3zLASX8sAd pAaFb9WesO44dt8JzC+mRFJtwa+PHBRfa+gpigpk6FiaSfRm4bSW8s5gDoS3DaxPbbzQ O0BN+xSIJdWyKHwJgXfLtbqY0MrOA08TXj2Zb/WPbNv33byKK8drMFEAwDA4mYUPf5vk l5DHeG4I496Y0eOK754MU3KEdFTa58WxGNA2W/vblxJktCsueakM9g1bRveuTJI8vCxy TudxunTLLqbj2fJKLcoZHSCkssCBmaudtzh3Xq8VPfMxhysfCiLFZ8dDYW9FLMGMz5Ca kVwA== X-Gm-Message-State: AFqh2koqiklkL/a9cXV9BXHfsfkSCEXSokNFAf47+HiGnKR8sRs7bQn8 xzg3sEd1UbB/7Y8w5ShswXHusZOH5XYShCK4 X-Received: by 2002:a5d:5227:0:b0:275:618c:83ea with SMTP id i7-20020a5d5227000000b00275618c83eamr7288209wra.29.1674078083019; Wed, 18 Jan 2023 13:41:23 -0800 (PST) Received: from Mindolluin.ire.aristanetworks.com ([217.173.96.166]) by smtp.gmail.com with ESMTPSA id m5-20020a056000024500b00267bcb1bbe5sm33186349wrz.56.2023.01.18.13.41.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 13:41:22 -0800 (PST) From: Dmitry Safonov To: linux-kernel@vger.kernel.org, David Ahern , Eric Dumazet , Herbert Xu , Jakub Kicinski , "David S. Miller" Cc: Dmitry Safonov , Andy Lutomirski , Bob Gilligan , Dmitry Safonov <0x7f454c46@gmail.com>, Hideaki YOSHIFUJI , Leonard Crestez , Paolo Abeni , Salam Noureddine , netdev@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v4 3/4] crypto/net/ipv6: sr: Switch to using crypto_pool Date: Wed, 18 Jan 2023 21:41:10 +0000 Message-Id: <20230118214111.394416-4-dima@arista.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118214111.394416-1-dima@arista.com> References: <20230118214111.394416-1-dima@arista.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755398214094472735?= X-GMAIL-MSGID: =?utf-8?q?1755398214094472735?= The conversion to use crypto_pool has the following upsides: - now SR uses asynchronous API which may potentially free CPU cycles and improve performance for of CPU crypto algorithm providers; - hash descriptors now don't have to be allocated on boot, but only at the moment SR starts using HMAC and until the last HMAC secret is deleted; - potentially reuse ahash_request(s) for different users - allocate only one per-CPU scratch buffer rather than a new one for each user - have a common API for net/ users that need ahash on RX/TX fast path Signed-off-by: Dmitry Safonov --- include/net/seg6_hmac.h | 9 -- net/ipv6/Kconfig | 1 + net/ipv6/seg6.c | 14 +-- net/ipv6/seg6_hmac.c | 207 +++++++++++++++------------------------- 4 files changed, 81 insertions(+), 150 deletions(-) diff --git a/include/net/seg6_hmac.h b/include/net/seg6_hmac.h index 2b5d2ee5613e..8aba24036143 100644 --- a/include/net/seg6_hmac.h +++ b/include/net/seg6_hmac.h @@ -32,13 +32,6 @@ struct seg6_hmac_info { u8 alg_id; }; -struct seg6_hmac_algo { - u8 alg_id; - char name[64]; - struct crypto_shash * __percpu *tfms; - struct shash_desc * __percpu *shashs; -}; - extern int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr, struct in6_addr *saddr, u8 *output); @@ -49,8 +42,6 @@ extern int seg6_hmac_info_del(struct net *net, u32 key); extern int seg6_push_hmac(struct net *net, struct in6_addr *saddr, struct ipv6_sr_hdr *srh); extern bool seg6_hmac_validate_skb(struct sk_buff *skb); -extern int seg6_hmac_init(void); -extern void seg6_hmac_exit(void); extern int seg6_hmac_net_init(struct net *net); extern void seg6_hmac_net_exit(struct net *net); diff --git a/net/ipv6/Kconfig b/net/ipv6/Kconfig index 658bfed1df8b..e9aa99180f85 100644 --- a/net/ipv6/Kconfig +++ b/net/ipv6/Kconfig @@ -305,6 +305,7 @@ config IPV6_SEG6_HMAC bool "IPv6: Segment Routing HMAC support" depends on IPV6 select CRYPTO + select CRYPTO_POOL select CRYPTO_HMAC select CRYPTO_SHA1 select CRYPTO_SHA256 diff --git a/net/ipv6/seg6.c b/net/ipv6/seg6.c index 29346a6eec9f..a1e4f3079c49 100644 --- a/net/ipv6/seg6.c +++ b/net/ipv6/seg6.c @@ -526,12 +526,6 @@ int __init seg6_init(void) goto out_unregister_pernet; err = seg6_local_init(); - if (err) - goto out_unregister_pernet; -#endif - -#ifdef CONFIG_IPV6_SEG6_HMAC - err = seg6_hmac_init(); if (err) goto out_unregister_iptun; #endif @@ -540,13 +534,12 @@ int __init seg6_init(void) out: return err; -#ifdef CONFIG_IPV6_SEG6_HMAC -out_unregister_iptun: + #ifdef CONFIG_IPV6_SEG6_LWTUNNEL seg6_local_exit(); +out_unregister_iptun: seg6_iptunnel_exit(); #endif -#endif #ifdef CONFIG_IPV6_SEG6_LWTUNNEL out_unregister_pernet: unregister_pernet_subsys(&ip6_segments_ops); @@ -558,9 +551,6 @@ int __init seg6_init(void) void seg6_exit(void) { -#ifdef CONFIG_IPV6_SEG6_HMAC - seg6_hmac_exit(); -#endif #ifdef CONFIG_IPV6_SEG6_LWTUNNEL seg6_iptunnel_exit(); #endif diff --git a/net/ipv6/seg6_hmac.c b/net/ipv6/seg6_hmac.c index d43c50a7310d..2395d227018c 100644 --- a/net/ipv6/seg6_hmac.c +++ b/net/ipv6/seg6_hmac.c @@ -35,6 +35,7 @@ #include #include +#include #include #include #include @@ -70,6 +71,12 @@ static const struct rhashtable_params rht_params = { .obj_cmpfn = seg6_hmac_cmpfn, }; +struct seg6_hmac_algo { + u8 alg_id; + char name[64]; + int crypto_pool_id; +}; + static struct seg6_hmac_algo hmac_algos[] = { { .alg_id = SEG6_HMAC_ALGO_SHA1, @@ -115,55 +122,17 @@ static struct seg6_hmac_algo *__hmac_get_algo(u8 alg_id) return NULL; } -static int __do_hmac(struct seg6_hmac_info *hinfo, const char *text, u8 psize, - u8 *output, int outlen) -{ - struct seg6_hmac_algo *algo; - struct crypto_shash *tfm; - struct shash_desc *shash; - int ret, dgsize; - - algo = __hmac_get_algo(hinfo->alg_id); - if (!algo) - return -ENOENT; - - tfm = *this_cpu_ptr(algo->tfms); - - dgsize = crypto_shash_digestsize(tfm); - if (dgsize > outlen) { - pr_debug("sr-ipv6: __do_hmac: digest size too big (%d / %d)\n", - dgsize, outlen); - return -ENOMEM; - } - - ret = crypto_shash_setkey(tfm, hinfo->secret, hinfo->slen); - if (ret < 0) { - pr_debug("sr-ipv6: crypto_shash_setkey failed: err %d\n", ret); - goto failed; - } - - shash = *this_cpu_ptr(algo->shashs); - shash->tfm = tfm; - - ret = crypto_shash_digest(shash, text, psize, output); - if (ret < 0) { - pr_debug("sr-ipv6: crypto_shash_digest failed: err %d\n", ret); - goto failed; - } - - return dgsize; - -failed: - return ret; -} - int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr, struct in6_addr *saddr, u8 *output) { __be32 hmackeyid = cpu_to_be32(hinfo->hmackeyid); - u8 tmp_out[SEG6_HMAC_MAX_DIGESTSIZE]; + struct crypto_pool_ahash hp; + struct seg6_hmac_algo *algo; int plen, i, dgsize, wrsize; + struct crypto_ahash *tfm; + struct scatterlist sg; char *ring, *off; + int err; /* a 160-byte buffer for digest output allows to store highest known * hash function (RadioGatun) with up to 1216 bits @@ -176,6 +145,10 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr, if (plen >= SEG6_HMAC_RING_SIZE) return -EMSGSIZE; + algo = __hmac_get_algo(hinfo->alg_id); + if (!algo) + return -ENOENT; + /* Let's build the HMAC text on the ring buffer. The text is composed * as follows, in order: * @@ -186,8 +159,36 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr, * 5. All segments in the segments list (n * 128 bits) */ - local_bh_disable(); + err = crypto_pool_start(algo->crypto_pool_id, &hp.base); + if (err) + return err; + ring = this_cpu_ptr(hmac_ring); + + sg_init_one(&sg, ring, plen); + + tfm = crypto_ahash_reqtfm(hp.req); + dgsize = crypto_ahash_digestsize(tfm); + if (dgsize > SEG6_HMAC_MAX_DIGESTSIZE) { + pr_debug("digest size too big (%d / %d)\n", + dgsize, SEG6_HMAC_MAX_DIGESTSIZE); + err = -ENOMEM; + goto err_end_pool; + } + + err = crypto_ahash_setkey(tfm, hinfo->secret, hinfo->slen); + if (err) { + pr_debug("crypto_ahash_setkey failed: err %d\n", err); + goto err_end_pool; + } + + err = crypto_ahash_init(hp.req); + if (err) + goto err_end_pool; + + ahash_request_set_crypt(hp.req, &sg, + hp.base.scratch, SEG6_HMAC_MAX_DIGESTSIZE); + off = ring; /* source address */ @@ -210,21 +211,25 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr, off += 16; } - dgsize = __do_hmac(hinfo, ring, plen, tmp_out, - SEG6_HMAC_MAX_DIGESTSIZE); - local_bh_enable(); + err = crypto_ahash_update(hp.req); + if (err) + goto err_end_pool; - if (dgsize < 0) - return dgsize; + err = crypto_ahash_final(hp.req); + if (err) + goto err_end_pool; wrsize = SEG6_HMAC_FIELD_LEN; if (wrsize > dgsize) wrsize = dgsize; memset(output, 0, SEG6_HMAC_FIELD_LEN); - memcpy(output, tmp_out, wrsize); + memcpy(output, hp.base.scratch, wrsize); + +err_end_pool: + crypto_pool_end(); - return 0; + return err; } EXPORT_SYMBOL(seg6_hmac_compute); @@ -291,12 +296,24 @@ EXPORT_SYMBOL(seg6_hmac_info_lookup); int seg6_hmac_info_add(struct net *net, u32 key, struct seg6_hmac_info *hinfo) { struct seg6_pernet_data *sdata = seg6_pernet(net); - int err; + struct seg6_hmac_algo *algo; + int ret; + + algo = __hmac_get_algo(hinfo->alg_id); + if (!algo) + return -ENOENT; + + ret = crypto_pool_alloc_ahash(algo->name, SEG6_HMAC_MAX_DIGESTSIZE); + if (ret < 0) + return ret; + algo->crypto_pool_id = ret; - err = rhashtable_lookup_insert_fast(&sdata->hmac_infos, &hinfo->node, + ret = rhashtable_lookup_insert_fast(&sdata->hmac_infos, &hinfo->node, rht_params); + if (ret) + crypto_pool_release(algo->crypto_pool_id); - return err; + return ret; } EXPORT_SYMBOL(seg6_hmac_info_add); @@ -304,6 +321,7 @@ int seg6_hmac_info_del(struct net *net, u32 key) { struct seg6_pernet_data *sdata = seg6_pernet(net); struct seg6_hmac_info *hinfo; + struct seg6_hmac_algo *algo; int err = -ENOENT; hinfo = rhashtable_lookup_fast(&sdata->hmac_infos, &key, rht_params); @@ -315,6 +333,12 @@ int seg6_hmac_info_del(struct net *net, u32 key) if (err) goto out; + algo = __hmac_get_algo(hinfo->alg_id); + if (algo) + crypto_pool_release(algo->crypto_pool_id); + else + WARN_ON_ONCE(1); + seg6_hinfo_release(hinfo); out: @@ -348,58 +372,6 @@ int seg6_push_hmac(struct net *net, struct in6_addr *saddr, } EXPORT_SYMBOL(seg6_push_hmac); -static int seg6_hmac_init_algo(void) -{ - struct seg6_hmac_algo *algo; - struct crypto_shash *tfm; - struct shash_desc *shash; - int i, alg_count, cpu; - - alg_count = ARRAY_SIZE(hmac_algos); - - for (i = 0; i < alg_count; i++) { - struct crypto_shash **p_tfm; - int shsize; - - algo = &hmac_algos[i]; - algo->tfms = alloc_percpu(struct crypto_shash *); - if (!algo->tfms) - return -ENOMEM; - - for_each_possible_cpu(cpu) { - tfm = crypto_alloc_shash(algo->name, 0, 0); - if (IS_ERR(tfm)) - return PTR_ERR(tfm); - p_tfm = per_cpu_ptr(algo->tfms, cpu); - *p_tfm = tfm; - } - - p_tfm = raw_cpu_ptr(algo->tfms); - tfm = *p_tfm; - - shsize = sizeof(*shash) + crypto_shash_descsize(tfm); - - algo->shashs = alloc_percpu(struct shash_desc *); - if (!algo->shashs) - return -ENOMEM; - - for_each_possible_cpu(cpu) { - shash = kzalloc_node(shsize, GFP_KERNEL, - cpu_to_node(cpu)); - if (!shash) - return -ENOMEM; - *per_cpu_ptr(algo->shashs, cpu) = shash; - } - } - - return 0; -} - -int __init seg6_hmac_init(void) -{ - return seg6_hmac_init_algo(); -} - int __net_init seg6_hmac_net_init(struct net *net) { struct seg6_pernet_data *sdata = seg6_pernet(net); @@ -407,29 +379,6 @@ int __net_init seg6_hmac_net_init(struct net *net) return rhashtable_init(&sdata->hmac_infos, &rht_params); } -void seg6_hmac_exit(void) -{ - struct seg6_hmac_algo *algo = NULL; - int i, alg_count, cpu; - - alg_count = ARRAY_SIZE(hmac_algos); - for (i = 0; i < alg_count; i++) { - algo = &hmac_algos[i]; - for_each_possible_cpu(cpu) { - struct crypto_shash *tfm; - struct shash_desc *shash; - - shash = *per_cpu_ptr(algo->shashs, cpu); - kfree(shash); - tfm = *per_cpu_ptr(algo->tfms, cpu); - crypto_free_shash(tfm); - } - free_percpu(algo->tfms); - free_percpu(algo->shashs); - } -} -EXPORT_SYMBOL(seg6_hmac_exit); - void __net_exit seg6_hmac_net_exit(struct net *net) { struct seg6_pernet_data *sdata = seg6_pernet(net); From patchwork Wed Jan 18 21:41:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Safonov X-Patchwork-Id: 45516 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2577284wrn; Wed, 18 Jan 2023 13:43:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXsOmpa3KNveyhAmGT2y0nYAVfgUFXsjc4puiEltfLGgKPf36SK+BaeovKEr05ww8VDQ0nCo X-Received: by 2002:a17:906:3515:b0:7c1:1b89:1fe0 with SMTP id r21-20020a170906351500b007c11b891fe0mr8512976eja.65.1674078217757; Wed, 18 Jan 2023 13:43:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674078217; cv=none; d=google.com; s=arc-20160816; b=zpbf2KZLLPecAdkT+kQYIvcc6zD3nhUCjWLqcAUKqaGUCOiz24YnJFZ5LDVQPN+m/c 5QkgU9L5xMqHm6NgWWjdMVOvo6tk6YhPXx0HkB37JFX75lGJ1iP+mSZFaJmj0b5Ai7es WtAqpfjtYUV1vztNrCTdEQqw3JohFceviJQ4enP7zVtBrOjfDacVGXTWX0ujFxHpFeGw +v0yOx3k6P4CicvyxyyGOp7NeWGqw3cztmoZzzGfj/OlpbQd7T9wl6TGyWrTYj1vFSWB rckH5NCNcHFavjUMB4FjLx0PEkuych8LSlHyjPKrprsA/h8uavb3oajaw/Ol52Em5tVo JIHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FMLRObJikupEwzTrRnh2PGmgjZ9Cqtc3WwiwTSH8KzM=; b=pEgR71TVOUUxLaZYPTrPEWn1k6z78RxU+VaFY+4ySbCGOjrAjSx0wfQjgVF3NwK8ND 2t8meuOzEKUByI7e7O3rRpOuMEYkRXT2Nr61iXC521/3e0Jyo3sdm4swSX0wWU0w/FTE wEzE7M3dEM5+Sp3CSurQa2+Ewyis1sDjLowjos5wrAiRMvCyp/0I5UdqRoJeVxSoXo3n mOE+udQVd0DGF4f+egaAJF5AKQbHHOe1rja1woB3PMg9Ugetjp7pG+dr+8fe8HeCNZ+G JT2MGDdWu2I9ZByojBrCnypPrAYZ1Zx1zCcho9v3X+E/rzGDQNtd/c15qfr0tcJJ9VHP jJsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=eCtXeYsP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hp4-20020a1709073e0400b007c106ff40ddsi43779463ejc.631.2023.01.18.13.43.14; Wed, 18 Jan 2023 13:43:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@arista.com header.s=google header.b=eCtXeYsP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=arista.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231137AbjARVmK (ORCPT + 99 others); Wed, 18 Jan 2023 16:42:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231205AbjARVlt (ORCPT ); Wed, 18 Jan 2023 16:41:49 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F315F63E0C for ; Wed, 18 Jan 2023 13:41:25 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id k8so20846843wrc.9 for ; Wed, 18 Jan 2023 13:41:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arista.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FMLRObJikupEwzTrRnh2PGmgjZ9Cqtc3WwiwTSH8KzM=; b=eCtXeYsP+ADhvJqyR5obWxJN2h47MVmxSPM3WN/yFWVLL7Bofz6ALMqkQ5FmPFSW8x UWaka0KuTIy7rIQGm6zFjHry+P1eL6Sk8lHn9fP+GlFKwmKfj/Q6EGvopu1kINYq/tef zm8zNYOALlS2vhNm+EkXcXVPzdDc/TXwTPpHmjgDI6AlVHBF2UVGGn5IwZJjA1mkQrTd W7/8pHqfbBiiPSMO3n8KjIvPtsQCwLJhGfGrg9lAgqPV+bv5lqOuwfh93c0xePh+Wkpp 2YLAsoMIKNpdfYZLV7cFr9MhZ4e3JlmJFQlcJhphihQuV+4yzmbAlgwjxUanzoyThCaw x8Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FMLRObJikupEwzTrRnh2PGmgjZ9Cqtc3WwiwTSH8KzM=; b=aI6PZEt+N2DPN1o6U6SAcc3u5tBOzb31+4HYpxE8VhTR3hThQeeFuGwzJwQbXX4FN9 lI7JRWJ+3Yq0d/UgMuQ2ShgRalKbzsNMIhV/wa/OjkAAAYqrBl55WnJLBkGyzJZbxBbt 6HNJMrt0E6ohXAWOWpx4Yl9/FQoqG1D74JyYpUZl24MWu6zE8EO4mnsS7UR7rAhXzAD1 OZAcCM3D/Y3upvv/ekXji/M1VGJL0KOCbFzmk6bDtjDJ57GLEJh4r2oIBeHbzFCrZ0cc PCZDWTKejMM5U3BJH0MlOxPH+l2/2njWwFJAsnM6vd3rgkmGlnCcV7vK8x0jDyPfFm5l txLg== X-Gm-Message-State: AFqh2krsEIFiGwfUKK8UB0Jub19Zb/wWFgbDk98E6e/XNqKTyXRNznb/ 6P7AKiV2GjyBzuOltwhqYUKMQKXhlYsvXvRM X-Received: by 2002:a5d:5b0e:0:b0:250:22e4:b89e with SMTP id bx14-20020a5d5b0e000000b0025022e4b89emr8426197wrb.65.1674078084270; Wed, 18 Jan 2023 13:41:24 -0800 (PST) Received: from Mindolluin.ire.aristanetworks.com ([217.173.96.166]) by smtp.gmail.com with ESMTPSA id m5-20020a056000024500b00267bcb1bbe5sm33186349wrz.56.2023.01.18.13.41.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 13:41:23 -0800 (PST) From: Dmitry Safonov To: linux-kernel@vger.kernel.org, David Ahern , Eric Dumazet , Herbert Xu , Jakub Kicinski , "David S. Miller" Cc: Dmitry Safonov , Andy Lutomirski , Bob Gilligan , Dmitry Safonov <0x7f454c46@gmail.com>, Hideaki YOSHIFUJI , Leonard Crestez , Paolo Abeni , Salam Noureddine , netdev@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v4 4/4] crypto/Documentation: Add crypto_pool kernel API Date: Wed, 18 Jan 2023 21:41:11 +0000 Message-Id: <20230118214111.394416-5-dima@arista.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118214111.394416-1-dima@arista.com> References: <20230118214111.394416-1-dima@arista.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755398241338317806?= X-GMAIL-MSGID: =?utf-8?q?1755398241338317806?= Signed-off-by: Dmitry Safonov --- Documentation/crypto/crypto_pool.rst | 36 ++++++++++++++++++++++++++++ Documentation/crypto/index.rst | 1 + 2 files changed, 37 insertions(+) create mode 100644 Documentation/crypto/crypto_pool.rst diff --git a/Documentation/crypto/crypto_pool.rst b/Documentation/crypto/crypto_pool.rst new file mode 100644 index 000000000000..84abd1f2ee80 --- /dev/null +++ b/Documentation/crypto/crypto_pool.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Per-CPU pool of crypto requests +=============================== + +Overview +-------- +The crypto pool API manages pre-allocated per-CPU pool of crypto requests, +providing ability to use crypto requests on fast paths, potentially in atomic +contexts. The allocation and initialization of the requests should be done +before their usage as it's slow-path and may sleep. + +Order of operations +------------------- +You are required to allocate a new pool prior using it and manage its lifetime. +You can allocate a per-CPU pool of ahash requests by crypto_pool_alloc_ahash(). +It will give you a pool id that you can use further on fast-path for hashing. +You can increase the reference counter for an allocated pool via +crypto_pool_get(). Decrease the reference counter by crypto_pool_release(). +When the refcounter hits zero, the pool is scheduled for destruction and you +can't use the corresponding crypto pool id anymore. +Note that crypto_pool_get() and crypto_pool_release() must be called +only for an already existing pool and can be called in atomic contexts. + +crypto_pool_start() disables bh and returns you back ``struct crypto_pool *``, +which is a generic type for different crypto requests and has ``scratch`` area +that can be used as a temporary buffer for your operation. + +crypto_pool_end() enables bh back once you've done with your crypto +operation. + +.. kernel-doc:: include/crypto/pool.h + :identifiers: + +.. kernel-doc:: crypto/crypto_pool.c + :identifiers: diff --git a/Documentation/crypto/index.rst b/Documentation/crypto/index.rst index 21338fa92642..3eaf4e964e5b 100644 --- a/Documentation/crypto/index.rst +++ b/Documentation/crypto/index.rst @@ -25,6 +25,7 @@ for cryptographic use cases, as well as programming examples. devel-algos userspace-if crypto_engine + crypto_pool api api-samples descore-readme