From patchwork Wed Nov 16 04:13:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 20704 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp3084340wru; Tue, 15 Nov 2022 20:17:13 -0800 (PST) X-Google-Smtp-Source: AA0mqf6vJ9FE7fCIMk5cJf6gpo7SKu93O7bDo307cn2S5giKQLglE7Hx5HbwXJ5iVZ0gZ/4RKIdH X-Received: by 2002:a17:90a:4d47:b0:200:2069:7702 with SMTP id l7-20020a17090a4d4700b0020020697702mr1722332pjh.239.1668572223649; Tue, 15 Nov 2022 20:17:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668572223; cv=none; d=google.com; s=arc-20160816; b=XP9tlyA+pIFJi2poyXWUD1GStvoZgx1GQWzAMHLqsO/NCjDDLvCXNqAWz44/yiZJNX PHfyTVgjTO1sM5lSui/wqoufQkWt9tQk/e41C6j+LN2/8NMJ1bTOQxdRC5vw/sMbW/G0 7dI5FRKvwvflX5yRsnUVFTa4F0Sx+5NwpeUcNYCrmbvHdJz5CfODwh0Xr+A/D3k6EhjB E/5AxJXpXl1OKblOslZFjMLAsssxqYRHVkjPs2dtwgRDHQMTPZd7cxL2KiOBRg4bCDeg Hzm6Vc2PcnsgXrMyrqO6aLth5H2q43U0osMjhnqKMzGP5RP6dC6e9HJR/vPA3Xf9hkHq E3CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cTzqPQFdhTvUVOcsBjcnPT2NWlxFqM19tqppsEYjuYM=; b=Wru2gqG5rcTAshbr1odRn8CBAgGJZFtRBjQWlOSEs806NYPktIKfwGVxtgh452gIh6 Opz6KFl0aWyGhnJFZGiQnsYCXyivW6t31TXltU3irdUehzqzwe1WoFoaJJM4UQQN/CxT igJnYJ6rFtsY2NAd4C/sRo4A+RqOfspVb9Tyji60QRAHvYiXqvSR9m0rDMVYmiqAAtEX ASi+008wprMGrT7Uvycc785bzyjYNVPy3xGC+CplzH/VuoY+B+/DhyL8sNItyKPvbz1X lcshqG/TCvsV6Y7W6lKRvw5a2Ip6CWzMFQu7Mb2F60l7hVTVmmzMWOpLh2mHTg0m3bLn KrXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=gG3oz65q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u9-20020a170902e80900b001868d5fc27csi16850528plg.254.2022.11.15.20.16.49; Tue, 15 Nov 2022 20:17:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=gG3oz65q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232413AbiKPEP7 (ORCPT + 99 others); Tue, 15 Nov 2022 23:15:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229495AbiKPEOc (ORCPT ); Tue, 15 Nov 2022 23:14:32 -0500 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 561163205E; Tue, 15 Nov 2022 20:14:25 -0800 (PST) Received: from pps.filterd (m0150241.ppops.net [127.0.0.1]) by mx0a-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AG3Zd6V015695; Wed, 16 Nov 2022 04:14:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=cTzqPQFdhTvUVOcsBjcnPT2NWlxFqM19tqppsEYjuYM=; b=gG3oz65q2sA8UpQnUKLlfZu9X3sSK8E6WnisyHzRLbbdryTKn++4uqJg2Q21hi80yHYl 6T5GDkhbneJerWV5h27PFn0c176350ZaY8kI886qTKzqU7/aKdIaVHrtEwarGm+TwF9u zWLXpyvfIUf9cMS70MU6xv49SkIYoL/lFD2JNuc91i+UHeEdTZljrp3rul/cS3SvLLtY ISM1d5ZcxPrGMLXlQgPRocsZPer0gftY1flBACb6mrxoS6m4/U8COWf/XJ05n4pcl8OK 7CN+EmjMQxj6h9MUJHOtl5rY4+1fzxdtKS+C5UJuJlaqxlmB4R+dBRHNmadpVeGtXAVP XQ== Received: from p1lg14879.it.hpe.com (p1lg14879.it.hpe.com [16.230.97.200]) by mx0a-002e3701.pphosted.com (PPS) with ESMTPS id 3kvqkbreg5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 16 Nov 2022 04:14:14 +0000 Received: from p1lg14885.dc01.its.hpecorp.net (unknown [10.119.18.236]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14879.it.hpe.com (Postfix) with ESMTPS id 0A42A4B5C9; Wed, 16 Nov 2022 04:14:14 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14885.dc01.its.hpecorp.net (Postfix) with ESMTP id 8E539808BA4; Wed, 16 Nov 2022 04:14:13 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net, tim.c.chen@linux.intel.com, ap420073@gmail.com, ardb@kernel.org, Jason@zx2c4.com, David.Laight@ACULAB.COM, ebiggers@kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Robert Elliott Subject: [PATCH v4 17/24] crypto: x86/poly - load based on CPU features Date: Tue, 15 Nov 2022 22:13:35 -0600 Message-Id: <20221116041342.3841-18-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221116041342.3841-1-elliott@hpe.com> References: <20221103042740.6556-1-elliott@hpe.com> <20221116041342.3841-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-GUID: SD-tb7NqfoFHqzT8wvROtWvqBdLg81oS X-Proofpoint-ORIG-GUID: SD-tb7NqfoFHqzT8wvROtWvqBdLg81oS X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-15_08,2022-11-15_03,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 impostorscore=0 spamscore=0 clxscore=1015 mlxscore=0 suspectscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211160029 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749624788122797376?= X-GMAIL-MSGID: =?utf-8?q?1749624788122797376?= Like commit aa031b8f702e ("crypto: x86/sha512 - load based on CPU features"), these x86-optimized crypto modules already have module aliases based on CPU feature bits: nhpoly1305 poly1305 polyval Rename the unique device table data structure to a generic name so the code has the same pattern in all the modules. Remove the __maybe_unused attribute from polyval since it is always used. Signed-off-by: Robert Elliott --- v4 Removed CPU feature checks that are unreachable because the x86_match_cpu call already handles them. Made poly1305 match on all features since it does provide an x86_64 asm function if avx, avx2, and avx512f are not available. Move polyval into this patch rather than pair with ghash. Remove __maybe_unused from polyval. --- arch/x86/crypto/nhpoly1305-avx2-glue.c | 13 +++++++++++-- arch/x86/crypto/nhpoly1305-sse2-glue.c | 9 ++++++++- arch/x86/crypto/poly1305_glue.c | 10 ++++++++++ arch/x86/crypto/polyval-clmulni_glue.c | 6 +++--- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/arch/x86/crypto/nhpoly1305-avx2-glue.c b/arch/x86/crypto/nhpoly1305-avx2-glue.c index f7dc9c563bb5..fa415fec5793 100644 --- a/arch/x86/crypto/nhpoly1305-avx2-glue.c +++ b/arch/x86/crypto/nhpoly1305-avx2-glue.c @@ -11,6 +11,7 @@ #include #include #include +#include #include /* avoid kernel_fpu_begin/end scheduler/rcu stalls */ @@ -60,10 +61,18 @@ static struct shash_alg nhpoly1305_alg = { .descsize = sizeof(struct nhpoly1305_state), }; +static const struct x86_cpu_id module_cpu_ids[] = { + X86_MATCH_FEATURE(X86_FEATURE_AVX2, NULL), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, module_cpu_ids); + static int __init nhpoly1305_mod_init(void) { - if (!boot_cpu_has(X86_FEATURE_AVX2) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) + if (!x86_match_cpu(module_cpu_ids)) + return -ENODEV; + + if (!boot_cpu_has(X86_FEATURE_OSXSAVE)) return -ENODEV; return crypto_register_shash(&nhpoly1305_alg); diff --git a/arch/x86/crypto/nhpoly1305-sse2-glue.c b/arch/x86/crypto/nhpoly1305-sse2-glue.c index daffcc7019ad..c47765e46236 100644 --- a/arch/x86/crypto/nhpoly1305-sse2-glue.c +++ b/arch/x86/crypto/nhpoly1305-sse2-glue.c @@ -11,6 +11,7 @@ #include #include #include +#include #include /* avoid kernel_fpu_begin/end scheduler/rcu stalls */ @@ -60,9 +61,15 @@ static struct shash_alg nhpoly1305_alg = { .descsize = sizeof(struct nhpoly1305_state), }; +static const struct x86_cpu_id module_cpu_ids[] = { + X86_MATCH_FEATURE(X86_FEATURE_XMM2, NULL), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, module_cpu_ids); + static int __init nhpoly1305_mod_init(void) { - if (!boot_cpu_has(X86_FEATURE_XMM2)) + if (!x86_match_cpu(module_cpu_ids)) return -ENODEV; return crypto_register_shash(&nhpoly1305_alg); diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c index 16831c036d71..f1e39e23b2a3 100644 --- a/arch/x86/crypto/poly1305_glue.c +++ b/arch/x86/crypto/poly1305_glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -268,8 +269,17 @@ static struct shash_alg alg = { }, }; +static const struct x86_cpu_id module_cpu_ids[] = { + X86_MATCH_FEATURE(X86_FEATURE_ANY, NULL), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, module_cpu_ids); + static int __init poly1305_simd_mod_init(void) { + if (!x86_match_cpu(module_cpu_ids)) + return -ENODEV; + if (boot_cpu_has(X86_FEATURE_AVX) && cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) static_branch_enable(&poly1305_use_avx); diff --git a/arch/x86/crypto/polyval-clmulni_glue.c b/arch/x86/crypto/polyval-clmulni_glue.c index de1c908f7412..b98e32f8e2a4 100644 --- a/arch/x86/crypto/polyval-clmulni_glue.c +++ b/arch/x86/crypto/polyval-clmulni_glue.c @@ -176,15 +176,15 @@ static struct shash_alg polyval_alg = { }, }; -__maybe_unused static const struct x86_cpu_id pcmul_cpu_id[] = { +static const struct x86_cpu_id module_cpu_ids[] = { X86_MATCH_FEATURE(X86_FEATURE_PCLMULQDQ, NULL), {} }; -MODULE_DEVICE_TABLE(x86cpu, pcmul_cpu_id); +MODULE_DEVICE_TABLE(x86cpu, module_cpu_ids); static int __init polyval_clmulni_mod_init(void) { - if (!x86_match_cpu(pcmul_cpu_id)) + if (!x86_match_cpu(module_cpu_ids)) return -ENODEV; if (!boot_cpu_has(X86_FEATURE_AVX))