From patchwork Mon Dec 19 20:37:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2604390wrn; Mon, 19 Dec 2022 12:39:35 -0800 (PST) X-Google-Smtp-Source: AA0mqf7lNG9CEZ02/7ra/JmOe82XE17T4PQ0MiIHqt0lDnP5xDTN285Vh1lgKWs7hJWXiZOeuNsG X-Received: by 2002:a17:90a:17e8:b0:21a:2306:a9bc with SMTP id q95-20020a17090a17e800b0021a2306a9bcmr44135032pja.7.1671482375483; Mon, 19 Dec 2022 12:39:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671482375; cv=none; d=google.com; s=arc-20160816; b=ebVWy3fjGC3WWqy4JhZbYS99vErxN64FZifi3/7mhxCgcNO3O2tVojYFPB5w8psDwS jhrrhY3hzTqImp6PgSJft637DlanL8qfnh5MgZtqv8yCqH6Qc5tOzrhHi7cBpYqhkqbH 24Y7O3UccjjySK1ZvAdUDYu3mRYvbTN6uPDlsnwDzvjzoJxa/ao9dhZ25YcstVM7XMGT S8pIJHr9af9kcvP0TojDx33Wnn26bpGGa4NhpeLUFKfv/Q+37s0hnIW6BvIppt78hYSk dpnOvFf1mQjF2UC8kTiU7BdTYjPrq9WNI078zsgoBDZFML0nxdPEwvuOP251dIaFmdIE kvPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hxDUwTsXgf/F0pCYDtIB+NfpqhGa9UxYzYrva7hoWO4=; b=zl5B9C409VpZVVvYuW79pYDAhvPxRmS6uac8cU1hmzcqaqJpmeeweyPsS9mcdOG2pe 1EBRkVY77jiK8aWJZ5bQaxVDsARDoVFRwb6qDAhcE6GAdeZHpXW1n9rigBJVytRlajGt 6IsmwLhRCvW3pFGpkdk8Hj9j120MuQAGdeSi8/E9Ub52r8EGhsbfd4EpHm3BJDv/zL3q 6rYKtYKZdBeuvl6/EviBqFE9bqyjOUUCti9vafytjJ2dXjVxIZzqGbnDN3varDHt4t4T uhkxZQWWtiJW8GwRlWvlGRPhU4ELIUUgY6Cd4V/xoESKTq3m0VCTMzSHr50SwOI/aK7I ToFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=bdwGlHDm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g71-20020a636b4a000000b00477632e6011si10784407pgc.308.2022.12.19.12.39.22; Mon, 19 Dec 2022 12:39:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=bdwGlHDm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232541AbiLSUht (ORCPT + 99 others); Mon, 19 Dec 2022 15:37:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232394AbiLSUhq (ORCPT ); Mon, 19 Dec 2022 15:37:46 -0500 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AFDAD2FC; Mon, 19 Dec 2022 12:37:44 -0800 (PST) Received: from pps.filterd (m0148663.ppops.net [127.0.0.1]) by mx0a-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJKVxbp001379; Mon, 19 Dec 2022 20:37:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=hxDUwTsXgf/F0pCYDtIB+NfpqhGa9UxYzYrva7hoWO4=; b=bdwGlHDmJVGNT9TxaBzhIRCsCpWGGZL9J3HKrwKqG0UgBVE4QKsX5bKKsTGLb805qo/r SU48jqI/CfQ3icUgybiUw96ANFbgN80lonJ3WlOvJ7tjzlE+Yt8XcI+S7ccI5h9aHee5 WwlMMq2/BtpG4FOCbbxs+37klzwgV4oUBVgi5qQnqmTfRWcL4/E8JNvLKrjY/2EiVC+h glYj/o8abVUUVYloNY34Hyv7lZdkV09+GZs0RnG+8ljgv3cfTcYKOPUJd2mP5lpkq0Ls 3iUzsEozcTMHjES7krXLZOI7M4OhGVQzZ+eeUjbl8IUM9W/+wUVPBhlpaqglTgJ4KubG dA== Received: from p1lg14881.it.hpe.com (p1lg14881.it.hpe.com [16.230.97.202]) by mx0a-002e3701.pphosted.com (PPS) with ESMTPS id 3mjx3mrg6r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:37:41 +0000 Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14881.it.hpe.com (Postfix) with ESMTPS id 52F46801723; Mon, 19 Dec 2022 20:37:41 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTP id 0A8F0807B0B; Mon, 19 Dec 2022 20:37:41 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 1/3] crypto: skcipher - always yield at end of walk Date: Mon, 19 Dec 2022 14:37:31 -0600 Message-Id: <20221219203733.3063192-2-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219203733.3063192-1-elliott@hpe.com> References: <20221219203733.3063192-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-GUID: 8tTJD82xn1hsm32dpy-VUM8DmsXSWayU X-Proofpoint-ORIG-GUID: 8tTJD82xn1hsm32dpy-VUM8DmsXSWayU X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 priorityscore=1501 mlxlogscore=999 mlxscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190181 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752676303365396667?= X-GMAIL-MSGID: =?utf-8?q?1752676303365396667?= Always yield to the scheduler at the end of skcipher_walk_done(), not just if additional bytes are left to be processed. This avoids soft lockups if drivers are invoked back-to-back to process data that is an integer multiple of their block size. Example: while processing 1 MiB buffers, multiple skciphers run from 192 s to 218 s without ever yielding to the scheduler, causing three soft lockup complaints. The kernel is configured for CONFIG_PREEMPT_NONE=y (or preempt=none on the kernel command line), so only explicit cond_resched() calls trigger scheduling - might_resched() and preempt_enable() do not (see kernel/sched/core.c). [ 190.865601] tcrypt: PERL my %speeds_skcipher = ( [ 192.041157] tcrypt: PERL "cbc-aes-aesni" => 2396490, [ 192.373934] tcrypt: PERL "ctr-aes-aesni" => 574888, [ 193.548967] tcrypt: PERL "cts-cbc-aes-aesni" => 2396018, [ 193.909077] tcrypt: PERL "ecb-aes-aesni" => 631824, [ 194.223801] tcrypt: PERL "xctr-aes-aesni" => 535778, [ 194.608548] tcrypt: PERL "xts-aes-aesni" => 676518, [ 196.440247] tcrypt: PERL "ctr-aria-avx" => 3804224, [ 196.675788] tcrypt: PERL "xchacha12-simd" => 368668, [ 196.988868] tcrypt: PERL "xchacha20-simd" => 535314, [ 197.301510] tcrypt: PERL "chacha20-simd" => 535142, [ 198.590113] tcrypt: PERL "ctr-sm4-aesni-avx2" => 2642930, [ 208.975253] tcrypt: PERL "cfb-sm4-aesni-avx2" => 22499840, [ 218.187217] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [modprobe:3433] [ 246.181238] Modules linked in: tcrypt(+) hctr2 essiv adiantum ... [ 246.185048] RIP: 0010:measure_skcipher_bigbuf.constprop.0.isra.0+0x282/0x393 [tcrypt] [ 246.185304] Code: 00 0f 31 ... ... [ 218.197313] Call Trace: [ 218.197567] [ 218.197822] ? 0xffffffffc052a000 [ 218.198079] do_test.cold+0x989/0xc7a [tcrypt] [ 218.198340] ? 0xffffffffc052a000 [ 218.198595] tcrypt_mod_init+0x50/0x1000 [tcrypt] [ 218.198857] ? 0xffffffffc052a000 [ 218.199112] do_one_initcall+0x41/0x200 ... [ 219.391776] tcrypt: PERL "cbc-sm4-aesni-avx2" => 22528138, [ 221.560847] tcrypt: PERL "ctr-sm4-aesni-avx" => 4560732, [ 231.960414] tcrypt: PERL "cfb-sm4-aesni-avx" => 22498380, [ 242.350070] tcrypt: PERL "cbc-sm4-aesni-avx" => 22527668, [ 244.471181] tcrypt: PERL "ecb-sm4-aesni-avx" => 4469626, ... [ 246.181064] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [modprobe:3433] ... [ 250.168239] tcrypt: PERL "cbc-camellia-aesni-avx2" => 12202738, [ 255.840094] tcrypt: PERL "cbc-camellia-aesni" => 12203096, [ 264.047440] tcrypt: PERL "cbc-cast5-avx" => 17744280, [ 273.091258] tcrypt: PERL "cbc-cast6-avx" => 19375400, [ 274.183249] watchdog: BUG: soft lockup - CPU#1 stuck for 78s! [modprobe:3433] ... [ 283.066260] tcrypt: PERL "cbc-serpent-avx2" => 21454930, [ 292.983848] tcrypt: PERL "cbc-serpent-avx" => 21452996, ... By adding a unilateral call to crypto_yield(), which calls cond_resched() and lets the scheduler use the CPU for another thread, that no longer happens. Starting at 2218 s, there is no soft lockup reported at 2244 s: [ 2217.202692] tcrypt: PERL my %speeds_skcipher = ( [ 2218.450215] tcrypt: PERL "cbc-aes-aesni" => 2179138, [ 2218.950960] tcrypt: PERL "cbc-aes-aesni" => 538738, [ 2219.460618] tcrypt: PERL "ctr-aes-aesni" => 575212, [ 2219.983006] tcrypt: PERL "ctr-aes-aesni" => 574402, [ 2221.329550] tcrypt: PERL "cts-cbc-aes-aesni" => 2182864, [ 2221.840599] tcrypt: PERL "cts-cbc-aes-aesni" => 539064, [ 2222.344290] tcrypt: PERL "ecb-aes-aesni" => 537402, [ 2222.869201] tcrypt: PERL "ecb-aes-aesni" => 537730, [ 2223.400315] tcrypt: PERL "xctr-aes-aesni" => 534824, [ 2223.897915] tcrypt: PERL "xctr-aes-aesni" => 534782, [ 2224.414956] tcrypt: PERL "xts-aes-aesni" => 539592, [ 2224.923715] tcrypt: PERL "xts-aes-aesni" => 539356, [ 2226.740211] tcrypt: PERL "ctr-aria-avx" => 3392444, [ 2228.545624] tcrypt: PERL "ctr-aria-avx" => 3392068, [ 2228.869883] tcrypt: PERL "xchacha12-simd" => 368932, [ 2229.204980] tcrypt: PERL "xchacha12-simd" => 374122, [ 2229.609975] tcrypt: PERL "xchacha20-simd" => 535596, [ 2230.022425] tcrypt: PERL "xchacha20-simd" => 537500, [ 2230.429674] tcrypt: PERL "chacha20-simd" => 535474, [ 2230.831041] tcrypt: PERL "chacha20-simd" => 534264, [ 2232.278150] tcrypt: PERL "ctr-sm4-aesni-avx2" => 2640770, [ 2233.744781] tcrypt: PERL "ctr-sm4-aesni-avx2" => 2642520, [ 2244.290542] tcrypt: PERL "cfb-sm4-aesni-avx2" => 22497308, [ 2245.725044] tcrypt: PERL "cfb-sm4-aesni-avx2" => 2604468, [ 2256.279228] tcrypt: PERL "cbc-sm4-aesni-avx2" => 22526084, [ 2257.729868] tcrypt: PERL "cbc-sm4-aesni-avx2" => 2600460, [ 2260.068782] tcrypt: PERL "ctr-sm4-aesni-avx" => 4560650, [ 2262.414663] tcrypt: PERL "ctr-sm4-aesni-avx" => 4561468, [ 2272.943000] tcrypt: PERL "cfb-sm4-aesni-avx" => 22496026, [ 2275.233755] tcrypt: PERL "cfb-sm4-aesni-avx" => 4456984, [ 2285.779516] tcrypt: PERL "cbc-sm4-aesni-avx" => 22525908, [ 2288.081160] tcrypt: PERL "cbc-sm4-aesni-avx" => 4457036, [ 2290.374086] tcrypt: PERL "ecb-sm4-aesni-avx" => 4465790, [ 2292.677381] tcrypt: PERL "ecb-sm4-aesni-avx" => 4466014, [ 2298.544718] tcrypt: PERL "cbc-camellia-aesni-avx2" => 12246268, [ 2299.869611] tcrypt: PERL "cbc-camellia-aesni-avx2" => 2349440, [ 2305.734078] tcrypt: PERL "cbc-camellia-aesni" => 12246930, [ 2307.746065] tcrypt: PERL "cbc-camellia-aesni" => 3832992, [ 2316.127414] tcrypt: PERL "cbc-cast5-avx" => 17737348, [ 2318.703437] tcrypt: PERL "cbc-cast5-avx" => 5061014, [ 2327.694881] tcrypt: PERL "cbc-cast6-avx" => 19065488, [ 2331.672188] tcrypt: PERL "cbc-cast6-avx" => 8145590, [ 2341.750274] tcrypt: PERL "cbc-serpent-avx2" => 21453172, [ 2343.209420] tcrypt: PERL "cbc-serpent-avx2" => 2611702, Fixes: b286d8b1a690 ("crypto: skcipher - Add skcipher walk interface") Signed-off-by: Robert Elliott --- crypto/skcipher.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 0ecab31cfe79..cdead632117a 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -153,13 +153,20 @@ int skcipher_walk_done(struct skcipher_walk *walk, int err) scatterwalk_done(&walk->in, 0, nbytes); scatterwalk_done(&walk->out, 1, nbytes); - if (nbytes) { - crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? - CRYPTO_TFM_REQ_MAY_SLEEP : 0); + /* + * Allow scheduler to use the CPU since it has been busy, + * regardless of whether another loop pass is due + */ + crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? + CRYPTO_TFM_REQ_MAY_SLEEP : 0); + + if (nbytes) return skcipher_walk_next(walk); - } finish: + crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? + CRYPTO_TFM_REQ_MAY_SLEEP : 0); + /* Short-circuit for the common/fast path. */ if (!((unsigned long)walk->buffer | (unsigned long)walk->page)) goto out; From patchwork Mon Dec 19 20:37:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34759 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2604621wrn; Mon, 19 Dec 2022 12:40:06 -0800 (PST) X-Google-Smtp-Source: AA0mqf6rzy5WYJqQuD3dY/5j2IU47hdeCRGAgKXUQRkWHbFSvTT9K9dBVcO93/BnV5/fLVs3sIj7 X-Received: by 2002:a17:90b:2685:b0:21b:c5d9:3478 with SMTP id pl5-20020a17090b268500b0021bc5d93478mr44571246pjb.0.1671482406627; Mon, 19 Dec 2022 12:40:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671482406; cv=none; d=google.com; s=arc-20160816; b=bwRdfnlz8laGM3BuafSsy/VaYyzp+yHZdnXJnSYSXtd1ur4zNe7KsLHpnm5m2e6heF EvIqxCBV+MNy5aUk1pAmIOntazjPbwtsYFgLLRwXErCdZsB1YK5yFgJ1LSpM30trxrwc kblU0XOes7m7zwa0lXBGi+DwsKSvtljV+f03xBiHLtgu1cPVC4e94kHhLq6/pV1O8fr8 8/u/9tM6t1gnN0rDErwSsPZV0vzYpqc0emPcem9XMLgEKhQ0etVcjgoFZ7MrAuwt8FG6 dTMmMMs8mWq209AGWOvwk0rv8PDRCebTMY2AVGjqATxnKCshRnVVZ/xK1byb5cbpu4n8 kW8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0VSNaOL1jHOqTtf/WTfp7k2gMnlP7/VAaH2DyvhO7vE=; b=jitJuVUibKC5l4SSRLaKIPy7kqD7RK9CtSuOeMtFihOVDuRen26inRyY+rwoSE31Mr Z1V8iFMZ6FAWAYE2qUBvtWz4FZSbYoXIbRc5X6AjSrbunffp3q+5s1P6lyuo4Lb/pncL gQ6R9Ig5R6F8XAgBX98lWZ+fYMs9f30FivVf5xIybvdXzFveLlw20w+APldu9kRMnYT4 iX2pABNuYOhUcoawaW1kBpnoIG/2Zdas4pXXPE/GNyU4K8nFxnNUb5QBuzBo2FiRJAJG m7SHAxsx+piTr8gcCguri0zfnXcy/HXjIGwGmNK6J4BCdQfSqYUSRsJx0R8A28UzmwxQ SdSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=XaRQpNta; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oc5-20020a17090b1c0500b0020b15fcac76si19860514pjb.4.2022.12.19.12.39.54; Mon, 19 Dec 2022 12:40:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=XaRQpNta; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbiLSUh5 (ORCPT + 99 others); Mon, 19 Dec 2022 15:37:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232512AbiLSUhq (ORCPT ); Mon, 19 Dec 2022 15:37:46 -0500 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49579D13F; Mon, 19 Dec 2022 12:37:46 -0800 (PST) Received: from pps.filterd (m0134424.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJHcBaJ025331; Mon, 19 Dec 2022 20:37:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=0VSNaOL1jHOqTtf/WTfp7k2gMnlP7/VAaH2DyvhO7vE=; b=XaRQpNtawbs6fYmBkeRKeHiWgPzALDRuFchXjxuS5BcsVf6cife0cwZlC+WCBZ1fjpr6 UtKxgbpr/Q8Cnd/Ct0NlyE35dsO4ZsuMVg2WIyox5uHVXf5Jh13dxe0u77Uu2B9H7Jzr uD2tcSOST4LRJAACHZ3WeI5Am6mLU61WUehsCSCUyKlGvvedJNSxOZZtwSxuc3pXLgjT FERrQemYQlemdyJFto1RBNp6fzR9zlHRBYE1DsBoER1nrPKWCm2BUGKfCUUxNC3Km8sa BdOTnjORxAWrAptZG1N3I8oY2+6lLvpvimo4OJuiAwfJeVf8vvIIL2UyhhtyJ1NqGSyA tQ== Received: from p1lg14878.it.hpe.com (p1lg14878.it.hpe.com [16.230.97.204]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 3mjvh693mq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:37:43 +0000 Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14878.it.hpe.com (Postfix) with ESMTPS id 40CDF2F1EC; Mon, 19 Dec 2022 20:37:42 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTP id F1B5A807B14; Mon, 19 Dec 2022 20:37:41 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 2/3] crypto: aead/shash - yield at end of operations Date: Mon, 19 Dec 2022 14:37:32 -0600 Message-Id: <20221219203733.3063192-3-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219203733.3063192-1-elliott@hpe.com> References: <20221219203733.3063192-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-GUID: EzcKw8IYXUKEw2q5tOsCcHTz0u6rdnfy X-Proofpoint-ORIG-GUID: EzcKw8IYXUKEw2q5tOsCcHTz0u6rdnfy X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=0 malwarescore=0 bulkscore=0 phishscore=0 adultscore=0 clxscore=1015 impostorscore=0 mlxlogscore=943 mlxscore=0 spamscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190181 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752676335927397208?= X-GMAIL-MSGID: =?utf-8?q?1752676335927397208?= Add crypto_yield() calls at the end of all the encrypt and decrypt functions to let the scheduler use the CPU after possibly a long tenure by the crypto driver. This reduces RCU stalls and soft lockups when running crypto functions back-to-back that don't have their own yield calls (e.g., aligned generic functions). Signed-off-by: Robert Elliott --- crypto/aead.c | 4 ++++ crypto/shash.c | 32 ++++++++++++++++++++++++-------- 2 files changed, 28 insertions(+), 8 deletions(-) diff --git a/crypto/aead.c b/crypto/aead.c index 16991095270d..f88378f4d4f5 100644 --- a/crypto/aead.c +++ b/crypto/aead.c @@ -93,6 +93,8 @@ int crypto_aead_encrypt(struct aead_request *req) else ret = crypto_aead_alg(aead)->encrypt(req); crypto_stats_aead_encrypt(cryptlen, alg, ret); + + crypto_yield(crypto_aead_get_flags(aead)); return ret; } EXPORT_SYMBOL_GPL(crypto_aead_encrypt); @@ -112,6 +114,8 @@ int crypto_aead_decrypt(struct aead_request *req) else ret = crypto_aead_alg(aead)->decrypt(req); crypto_stats_aead_decrypt(cryptlen, alg, ret); + + crypto_yield(crypto_aead_get_flags(aead)); return ret; } EXPORT_SYMBOL_GPL(crypto_aead_decrypt); diff --git a/crypto/shash.c b/crypto/shash.c index 868b6ba2b3b7..6fea17a50048 100644 --- a/crypto/shash.c +++ b/crypto/shash.c @@ -114,11 +114,15 @@ int crypto_shash_update(struct shash_desc *desc, const u8 *data, struct crypto_shash *tfm = desc->tfm; struct shash_alg *shash = crypto_shash_alg(tfm); unsigned long alignmask = crypto_shash_alignmask(tfm); + int ret; if ((unsigned long)data & alignmask) - return shash_update_unaligned(desc, data, len); + ret = shash_update_unaligned(desc, data, len); + else + ret = shash->update(desc, data, len); - return shash->update(desc, data, len); + crypto_yield(crypto_shash_get_flags(tfm)); + return ret; } EXPORT_SYMBOL_GPL(crypto_shash_update); @@ -155,11 +159,15 @@ int crypto_shash_final(struct shash_desc *desc, u8 *out) struct crypto_shash *tfm = desc->tfm; struct shash_alg *shash = crypto_shash_alg(tfm); unsigned long alignmask = crypto_shash_alignmask(tfm); + int ret; if ((unsigned long)out & alignmask) - return shash_final_unaligned(desc, out); + ret = shash_final_unaligned(desc, out); + else + ret = shash->final(desc, out); - return shash->final(desc, out); + crypto_yield(crypto_shash_get_flags(tfm)); + return ret; } EXPORT_SYMBOL_GPL(crypto_shash_final); @@ -176,11 +184,15 @@ int crypto_shash_finup(struct shash_desc *desc, const u8 *data, struct crypto_shash *tfm = desc->tfm; struct shash_alg *shash = crypto_shash_alg(tfm); unsigned long alignmask = crypto_shash_alignmask(tfm); + int ret; if (((unsigned long)data | (unsigned long)out) & alignmask) - return shash_finup_unaligned(desc, data, len, out); + ret = shash_finup_unaligned(desc, data, len, out); + else + ret = shash->finup(desc, data, len, out); - return shash->finup(desc, data, len, out); + crypto_yield(crypto_shash_get_flags(tfm)); + return ret; } EXPORT_SYMBOL_GPL(crypto_shash_finup); @@ -197,14 +209,18 @@ int crypto_shash_digest(struct shash_desc *desc, const u8 *data, struct crypto_shash *tfm = desc->tfm; struct shash_alg *shash = crypto_shash_alg(tfm); unsigned long alignmask = crypto_shash_alignmask(tfm); + int ret; if (crypto_shash_get_flags(tfm) & CRYPTO_TFM_NEED_KEY) return -ENOKEY; if (((unsigned long)data | (unsigned long)out) & alignmask) - return shash_digest_unaligned(desc, data, len, out); + ret = shash_digest_unaligned(desc, data, len, out); + else + ret = shash->digest(desc, data, len, out); - return shash->digest(desc, data, len, out); + crypto_yield(crypto_shash_get_flags(tfm)); + return ret; } EXPORT_SYMBOL_GPL(crypto_shash_digest); From patchwork Mon Dec 19 20:37:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34760 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2604962wrn; Mon, 19 Dec 2022 12:41:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXsjtTBGDMRqyV0LlAmge49Vp6dJNby5MQSLeS7bfYM+f8kRaR/Aq7ZEDcWAwIgXnlxu36dH X-Received: by 2002:aa7:8753:0:b0:57d:3aa9:3278 with SMTP id g19-20020aa78753000000b0057d3aa93278mr21529150pfo.17.1671482462344; Mon, 19 Dec 2022 12:41:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671482462; cv=none; d=google.com; s=arc-20160816; b=S3ogf4pP+1gghXz8+LhUXTimmw79F8cvaSX0iNwI4BlTRRzERxDcFfLHdDSzGpFzBj bm+x6FMxKWhB2Zn0Gj7Dvjv1uowVWDITTZvKcCPG9xZQEU6ErZ7RdLQIlhHQkak9OJkE gbikzHPuiTAzW07VnDsUC3rMUx6v9imlC9vjFxVp/CZc19j8ABd9kFELVEP7Zrfc1tvN GMuU16P72DY3tpnO0czUbaD0HqbekfyEh51U0BgDVzTjlCl1QcK9s1i5a3JhhFDQnZrN Ty318l98A+faSpUKkdilqdkYoqMCYwCb913Y5TQY6U3R2fSszqsDe8yOPlNTTavb+JMH +b8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2dYGUGoiYlGjuRM5PMmfeHA1r1xQ9XcKfQBofK1Sca0=; b=ss6FP7/0mBFx6BJm5mtmtAHSoPRlyj2PuGUQIwQ+ObhKyzxhkeYcJxW3iH887/62eX b8EFJTBMX9pJEZAb9tlRzF8lr5x7kyYmbkkdPTePtq8gAYD4A9hzylThxi9wSXDdNlPh 7TQN1ekl0yPS4TRkX96LkLg44XaF5IB9tZFB9e6xb2rfGUgyN8Bloj346kLHzXwDWOZs vpcp2DuCrz6AkcvY02sOlAXSTWYLcBS8atXZZMMm7p7lGp42OgexdXmUqdkZOTALgMvN FAp2mqnFvP9/y7p+z26AsgxxYGluUk9Fa8bAzB9XSReMHdDq8O9RXb15A/PQw9aECT8j x3bg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=BAOJknJI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o23-20020a63fb17000000b0047009247403si13179910pgh.409.2022.12.19.12.40.49; Mon, 19 Dec 2022 12:41:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=BAOJknJI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232320AbiLSUiD (ORCPT + 99 others); Mon, 19 Dec 2022 15:38:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232524AbiLSUhs (ORCPT ); Mon, 19 Dec 2022 15:37:48 -0500 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 790CBD2E6; Mon, 19 Dec 2022 12:37:47 -0800 (PST) Received: from pps.filterd (m0134425.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJIMMT8023118; Mon, 19 Dec 2022 20:37:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=2dYGUGoiYlGjuRM5PMmfeHA1r1xQ9XcKfQBofK1Sca0=; b=BAOJknJIXqnDhbu1T2hlzXUdF5Z27k1OfU8oKQ0IFbkS9kcnCOLxDWk1DxcsbMOkRRe2 CJ+yYbgIrPuQnm2VHekNfEVCdRfrXJpOZs21I91WO9Qz14b8U+kqSOrlvHEKs60OE/6o PIOwOPsju6KnrB9bPd5uMhvpA6EV1iuoE3ur4sGJSQNZBsyCiRsWeqhqi8MhxJCu6xKj 2hLYu2oW0BETFDkivD0DS4Nxnl7uEsyZW3Cm+CGotUK+ujgcKdCYJ2g+Ewb/J5YBLOFl sm4Pd299WAkjgfZ9IUdl81Xefje8eAhHNivdMbSzrIOvnq47/B+qPOpRkdG1BB8oYMPi Iw== Received: from p1lg14880.it.hpe.com (p1lg14880.it.hpe.com [16.230.97.201]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 3mjw610qpd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:37:43 +0000 Received: from p1lg14886.dc01.its.hpecorp.net (unknown [10.119.18.237]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14880.it.hpe.com (Postfix) with ESMTPS id 3065D807131; Mon, 19 Dec 2022 20:37:43 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14886.dc01.its.hpecorp.net (Postfix) with ESMTP id D3E3D809224; Mon, 19 Dec 2022 20:37:42 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 3/3] crypto: tcrypt - yield at end of test Date: Mon, 19 Dec 2022 14:37:33 -0600 Message-Id: <20221219203733.3063192-4-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219203733.3063192-1-elliott@hpe.com> References: <20221219203733.3063192-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: SXAnjrNOQ-cAN-_GH4MGD5ESQQ-mfTh3 X-Proofpoint-GUID: SXAnjrNOQ-cAN-_GH4MGD5ESQQ-mfTh3 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 spamscore=0 malwarescore=0 phishscore=0 clxscore=1015 mlxscore=0 bulkscore=0 mlxlogscore=999 adultscore=0 lowpriorityscore=0 suspectscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190181 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752676394297930430?= X-GMAIL-MSGID: =?utf-8?q?1752676394297930430?= Call cond_resched() to let the scheduler reschedule the CPU at the end of each test pass. If the kernel is configured with CONFIG_PREEMPT_NONE=y (or preempt=none is used on the kernel command line), the only time the scheduler will intervene is when cond_resched() is called. So, repeated calls to modprobe tcrypt mode= hold the CPU for a long time. Signed-off-by: Robert Elliott --- crypto/tcrypt.c | 1 + 1 file changed, 1 insertion(+) diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c index 3e9e4adeef02..916bddbf4e75 100644 --- a/crypto/tcrypt.c +++ b/crypto/tcrypt.c @@ -3027,6 +3027,7 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) } + cond_resched(); return ret; }