From patchwork Mon Dec 19 20:29:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34755 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2601030wrn; Mon, 19 Dec 2022 12:31:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf71DiE04BM8DistPAMkkRRciK5UFD+o+++3rBiyK73kAmQSJo7qhIfRTC+eionWbGs8p9DA X-Received: by 2002:a05:6a20:bca9:b0:ac:ab16:442 with SMTP id fx41-20020a056a20bca900b000acab160442mr48464412pzb.4.1671481879400; Mon, 19 Dec 2022 12:31:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671481879; cv=none; d=google.com; s=arc-20160816; b=f0m+hADNOmagJ79P+WfeZTPfv6qzoDc5SEBqsZel4MOLx2Lxwae3LBTbHLuI4z2whZ 5aM0cXig0sDe5rffcZ9lui+UuPPsTcAZYhDcjWGRvHxF+j0fAJSN0f99zxbDyjryjvBr Oe66VBeBoruzMqsK+WbOlVg4E4PYMvV2sF7jdEG/jLBQQTY3HbCblb8NHD7nzmzuXGFK /wF02kALUt+B62ZLWPa2VijLjsob0HdaVfCEaJvRruNJ4JDDphpq9dilvRE6Xq50UbZd tyJRAfsPx5yLkfa9qkQW6hncGJPhhC7G1bBMIuAPD7SSRuwMtoFiPsu6D0f2JfCtTUj9 Ty1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sM4Tumpims+OtpD2j4ymbG2RWDwxkOit/SUN90kBxjg=; b=Ly+J81nUYUlnn1NekyKl1W9akkg6PTCYRFjgH+Q9VdOy7kQxMDebkrNtRKGyu1kqPc yg9a6YajZ+UOKW3yui/YH9hyJ7BYyLC49vX2ao724KUOwEsqTpJbArqqE01K67pW0NPZ KLqDv8IJFSKJqFlB3XDSuyJmFzZcVB3gGgfKq+lXTYqjN279y7f2gKL/lPGFRb7kaCHe Z6Q7HKazdPt5kN+I00ih6+JlYi38uqQcFgUqKmsukioRPqqXlF79qhNwAiaAcyFwI+Yf 9f1jNSTivfi6Ge5EmTeP76Xoe1/ZSI8QGD8QtOKHIDZ6FA9R+Np6jkOr3OsrFhsFGdTn ih7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=TBtBjm9J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s69-20020a637748000000b0047959422029si11207446pgc.395.2022.12.19.12.31.06; Mon, 19 Dec 2022 12:31:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=TBtBjm9J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232606AbiLSUa1 (ORCPT + 99 others); Mon, 19 Dec 2022 15:30:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232634AbiLSUaD (ORCPT ); Mon, 19 Dec 2022 15:30:03 -0500 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C989767F; Mon, 19 Dec 2022 12:29:58 -0800 (PST) Received: from pps.filterd (m0148663.ppops.net [127.0.0.1]) by mx0a-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJJoue0011960; Mon, 19 Dec 2022 20:29:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=sM4Tumpims+OtpD2j4ymbG2RWDwxkOit/SUN90kBxjg=; b=TBtBjm9J4r239rCE2d4UwtxqNvr0bo5JjAMWCkE1jOJDPAp2NCP45L7TJDhpBav/Z+0f 5f1xDFM9O6wY3HIDpgal3YZ1O8VFsW/r3OOw02Xg77qp9B7iwOSXpPlxj8EulsGC6xOO CzFM6WfIgG/tS3ujUCqxnNAZHo/mRAyaTvQ5BnbXUQXB3PZKVmkhqolWBVRVnkldjURw N1+NYQaxDixTMQMvodCizrU7+MZU2HHxZKn9tJ/9zKP70JYfzhn4x/ULUCiwNY7qlVlN atzmQWYyIxZQWZZHBASuWKbGvxB4f9eYa5zvzM2H5H8SHZiiDMKfj2gEZDHLwyvMwHlj dA== Received: from p1lg14881.it.hpe.com (p1lg14881.it.hpe.com [16.230.97.202]) by mx0a-002e3701.pphosted.com (PPS) with ESMTPS id 3mjx3mrdjp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:29:43 +0000 Received: from p1lg14885.dc01.its.hpecorp.net (unknown [10.119.18.236]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14881.it.hpe.com (Postfix) with ESMTPS id 5C2E2800EC3; Mon, 19 Dec 2022 20:29:42 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14885.dc01.its.hpecorp.net (Postfix) with ESMTP id E05168064A8; Mon, 19 Dec 2022 20:29:41 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net, paulmck@kernel.org, frederic@kernel.org, quic_neeraju@quicinc.com, josh@joshtriplett.org Cc: linux-crypto@vger.kernel.org, rcu@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 1/3] rcu: genericize RCU stall suppression functions Date: Mon, 19 Dec 2022 14:29:08 -0600 Message-Id: <20221219202910.3063036-2-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219202910.3063036-1-elliott@hpe.com> References: <20221219202910.3063036-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-GUID: _DWClsvXR85ZUkNJB2RiHeq_UM1-A9eC X-Proofpoint-ORIG-GUID: _DWClsvXR85ZUkNJB2RiHeq_UM1-A9eC X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 priorityscore=1501 mlxlogscore=999 mlxscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190180 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752675783074579172?= X-GMAIL-MSGID: =?utf-8?q?1752675783074579172?= Convert the functions that temporarily suppress RCU CPU stall reporting: rcu_sysrq_start() rcu_sysrq_end() to more generic functions that may be called by functions other than the SysRq handler: rcu_suppress_start() rcu_suppress_end() Covert the underlying variable: rcu_cpu_stall_suppress to an atomic variable so multiple threads can manipulate it at the same time, incrementing it in start() and decrementing in end(). Expose that in sysfs in a read-only /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress_dyn Keep /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress as writeable by userspace, but OR that into the equations rather than directly manipulate the atomic value. Signed-off-by: Robert Elliott --- .../RCU/Design/Requirements/Requirements.rst | 6 +++--- Documentation/RCU/stallwarn.rst | 19 +++++++++++++++---- arch/parisc/kernel/process.c | 2 +- drivers/tty/sysrq.c | 4 ++-- include/linux/rcupdate.h | 8 ++++---- kernel/rcu/rcu.h | 11 ++++++----- kernel/rcu/tree_stall.h | 18 ++++++++++-------- kernel/rcu/update.c | 11 ++++++++++- 8 files changed, 51 insertions(+), 28 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst index 49387d823619..5083490bb6fc 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -1639,9 +1639,9 @@ against mishaps and misuse: ``rcupdate.rcu_cpu_stall_suppress`` to suppress the splats. This kernel parameter may also be set via ``sysfs``. Furthermore, RCU CPU stall warnings are counter-productive during sysrq dumps and during - panics. RCU therefore supplies the rcu_sysrq_start() and - rcu_sysrq_end() API members to be called before and after long - sysrq dumps. RCU also supplies the rcu_panic() notifier that is + panics. RCU therefore supplies the rcu_suppress_start() and + rcu_suppress_end() API members to be called before and after long + CPU usage. RCU also supplies the rcu_panic() notifier that is automatically invoked at the beginning of a panic to suppress further RCU CPU stall warnings. diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.rst index e38c587067fc..9e6fba72f56d 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -135,13 +135,24 @@ see include/trace/events/rcu.h. Fine-Tuning the RCU CPU Stall Detector ====================================== -The rcuupdate.rcu_cpu_stall_suppress module parameter disables RCU's -CPU stall detector, which detects conditions that unduly delay RCU grace -periods. This module parameter enables CPU stall detection by default, -but may be overridden via boot-time parameter or at runtime via sysfs. +RCU's CPU stall detector detects conditions that unduly delay RCU grace +periods. CPU stall detection is enabled by default, but may be overridden +via boot-time parameter or at runtime via sysfs (accessible via +/sys/module/rcupdate/parameters). + +The rcupdate.rcu_cpu_stall_suppress module parameter disables RCU's +CPU stall detector. + +/sys/module/rcu_cpu_stall_suppress_dyn reports an internal semaphore +used by the RCU's CPU stall detector. Functions holding a CPU for a +long time (e.g., sysrq printouts) increment this value before use +and decrement it when done, so the value reports the number of +functions currently disabling stalls. + The stall detector's idea of what constitutes "unduly delayed" is controlled by a set of kernel configuration variables and cpp macros: + CONFIG_RCU_CPU_STALL_TIMEOUT ---------------------------- diff --git a/arch/parisc/kernel/process.c b/arch/parisc/kernel/process.c index c4f8374c7018..038378fe7bfa 100644 --- a/arch/parisc/kernel/process.c +++ b/arch/parisc/kernel/process.c @@ -126,7 +126,7 @@ void machine_power_off(void) "Please power this system off now."); /* prevent soft lockup/stalled CPU messages for endless loop. */ - rcu_sysrq_start(); + rcu_suppress_start(); lockup_detector_soft_poweroff(); for (;;); } diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c index d2b2720db6ca..81ab63a473a7 100644 --- a/drivers/tty/sysrq.c +++ b/drivers/tty/sysrq.c @@ -579,7 +579,7 @@ void __handle_sysrq(int key, bool check_mask) orig_suppress_printk = suppress_printk; suppress_printk = 0; - rcu_sysrq_start(); + rcu_suppress_start(); rcu_read_lock(); /* * Raise the apparent loglevel to maximum so that the sysrq header @@ -623,7 +623,7 @@ void __handle_sysrq(int key, bool check_mask) console_loglevel = orig_log_level; } rcu_read_unlock(); - rcu_sysrq_end(); + rcu_suppress_end(); suppress_printk = orig_suppress_printk; } diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 03abf883a281..c0d8a4aae7ff 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -131,11 +131,11 @@ static inline void rcu_init_tasks_generic(void) { } #endif #ifdef CONFIG_RCU_STALL_COMMON -void rcu_sysrq_start(void); -void rcu_sysrq_end(void); +void rcu_suppress_start(void); +void rcu_suppress_end(void); #else /* #ifdef CONFIG_RCU_STALL_COMMON */ -static inline void rcu_sysrq_start(void) { } -static inline void rcu_sysrq_end(void) { } +static inline void rcu_suppress_start(void) { } +static inline void rcu_suppress_end(void) { } #endif /* #else #ifdef CONFIG_RCU_STALL_COMMON */ #if defined(CONFIG_NO_HZ_FULL) && (!defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK)) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index c5aa934de59b..a658955a1174 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -224,24 +224,25 @@ extern int rcu_cpu_stall_ftrace_dump; extern int rcu_cpu_stall_suppress; extern int rcu_cpu_stall_timeout; extern int rcu_exp_cpu_stall_timeout; +extern atomic_t rcu_cpu_stall_suppress_dyn; int rcu_jiffies_till_stall_check(void); int rcu_exp_jiffies_till_stall_check(void); static inline bool rcu_stall_is_suppressed(void) { - return rcu_stall_is_suppressed_at_boot() || rcu_cpu_stall_suppress; + return rcu_stall_is_suppressed_at_boot() || + rcu_cpu_stall_suppress || + atomic_read(&rcu_cpu_stall_suppress_dyn); } #define rcu_ftrace_dump_stall_suppress() \ do { \ - if (!rcu_cpu_stall_suppress) \ - rcu_cpu_stall_suppress = 3; \ + atomic_inc(&rcu_cpu_stall_suppress_dyn); \ } while (0) #define rcu_ftrace_dump_stall_unsuppress() \ do { \ - if (rcu_cpu_stall_suppress == 3) \ - rcu_cpu_stall_suppress = 0; \ + atomic_dec(&rcu_cpu_stall_suppress_dyn); \ } while (0) #else /* #endif #ifdef CONFIG_RCU_STALL_COMMON */ diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 5653560573e2..260748bc5bc8 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -103,23 +103,25 @@ bool rcu_gp_might_be_stalled(void) return !time_before(j, READ_ONCE(rcu_state.gp_start) + d); } -/* Don't do RCU CPU stall warnings during long sysrq printouts. */ -void rcu_sysrq_start(void) +/* Don't do RCU CPU stall warnings during functions holding the CPU + * for a long period of time such as long sysrq printouts. + */ +void rcu_suppress_start(void) { - if (!rcu_cpu_stall_suppress) - rcu_cpu_stall_suppress = 2; + atomic_inc(&rcu_cpu_stall_suppress_dyn); } +EXPORT_SYMBOL_GPL(rcu_suppress_start); -void rcu_sysrq_end(void) +void rcu_suppress_end(void) { - if (rcu_cpu_stall_suppress == 2) - rcu_cpu_stall_suppress = 0; + atomic_dec(&rcu_cpu_stall_suppress_dyn); } +EXPORT_SYMBOL_GPL(rcu_suppress_end); /* Don't print RCU CPU stall warnings during a kernel panic. */ static int rcu_panic(struct notifier_block *this, unsigned long ev, void *ptr) { - rcu_cpu_stall_suppress = 1; + atomic_inc(&rcu_cpu_stall_suppress_dyn); return NOTIFY_DONE; } diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index f5e6a2f95a2a..ceee9d777384 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -501,11 +501,18 @@ EXPORT_SYMBOL_GPL(rcutorture_sched_setaffinity); #ifdef CONFIG_RCU_STALL_COMMON int rcu_cpu_stall_ftrace_dump __read_mostly; module_param(rcu_cpu_stall_ftrace_dump, int, 0644); -int rcu_cpu_stall_suppress __read_mostly; // !0 = suppress stall warnings. + +int rcu_cpu_stall_suppress __read_mostly; // !0 = suppress stall warnings from sysfs EXPORT_SYMBOL_GPL(rcu_cpu_stall_suppress); module_param(rcu_cpu_stall_suppress, int, 0644); + +atomic_t rcu_cpu_stall_suppress_dyn __read_mostly; // !0 = suppress stall warnings from functions +EXPORT_SYMBOL_GPL(rcu_cpu_stall_suppress_dyn); +module_param_named(rcu_cpu_stall_suppress_dyn, rcu_cpu_stall_suppress_dyn.counter, int, 0444); + int rcu_cpu_stall_timeout __read_mostly = CONFIG_RCU_CPU_STALL_TIMEOUT; module_param(rcu_cpu_stall_timeout, int, 0644); + int rcu_exp_cpu_stall_timeout __read_mostly = CONFIG_RCU_EXP_CPU_STALL_TIMEOUT; module_param(rcu_exp_cpu_stall_timeout, int, 0644); #endif /* #ifdef CONFIG_RCU_STALL_COMMON */ @@ -616,6 +623,8 @@ void __init rcupdate_announce_bootup_oddness(void) pr_info("\tAll grace periods are expedited (rcu_expedited).\n"); if (rcu_cpu_stall_suppress) pr_info("\tRCU CPU stall warnings suppressed (rcu_cpu_stall_suppress).\n"); + if (atomic_read(&rcu_cpu_stall_suppress_dyn)) + pr_info("\tDynamic RCU CPU stall warnings suppressed (rcu_cpu_stall_suppress_dyn).\n"); if (rcu_cpu_stall_timeout != CONFIG_RCU_CPU_STALL_TIMEOUT) pr_info("\tRCU CPU stall warnings timeout set to %d (rcu_cpu_stall_timeout).\n", rcu_cpu_stall_timeout); rcu_tasks_bootup_oddness(); From patchwork Mon Dec 19 20:29:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34756 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2601437wrn; Mon, 19 Dec 2022 12:32:18 -0800 (PST) X-Google-Smtp-Source: AA0mqf5hv8KhQBtlF6PC3GsbwuqEXRWG1uBqpAFS5IriW2zoW900ukyG6EgoaYFocVfzyOLSnkbT X-Received: by 2002:a62:e219:0:b0:577:d10d:6eab with SMTP id a25-20020a62e219000000b00577d10d6eabmr46007387pfi.21.1671481937936; Mon, 19 Dec 2022 12:32:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671481937; cv=none; d=google.com; s=arc-20160816; b=zGsyLxcSNfud2Ds5DTea5XnTVWL2hdW2Am8g9RFtWggC8iVwZHwcb/oCZTfaFhVbGb X7gGJcSZZDIZDyaHO24Ttq8dcW3NZ5YBedbTdL6ZbneE+6JwqCIhw2lQ7LnEwBQx5ejI BJyqRDJgminthMxioA5Qg3CO4TvtD1iN5Sy2w/5UWOqPgXSTlrpn+/17PS59c/Cq5O4u hhL1XVXllj+HnkoA77CsM9LETX0PCqMzbBh4YmmkX9eOZrJrNs2QAZiKElbieLPkt6uk sdnVuzIqyd5jfroxp2xi9YT+X51/Uea5uLpdz3kiloeK3dcxXurd5Rm/1lrz6U06t4z3 2Z3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jD3LLgSm37nQRHMfKV4CUST8QKYllFqCcPy8zK6SIaM=; b=nP06hmFMVUIsscJhPDt/DPNsIWsGX5W3nskFj9I7VGg/xvPGZiPlHa+GwRzaGui3MN XjI8Z16Zi/qnMqsowAU2pKuMEesEsyzO0Iv6tDv+zvvjIvq7k2ryRJVuVY1r3umzkVVN ecM4E5SXi0wY1XHHKbY2VF7ij+OjiiiwE0XO8439pv5J9fM8aZkRcs/NKprkQ1bXenst +KSRSH6eQ8yUS3BchIjw50qC4IhDJnJQDYbhvMIjOpMG1a3K1WS5S1zabxPYQhSoFyXJ YDeJPV1hsmTUujH19RrDXdQ4kJmzScJl45TWvxwTSQRuS0GKrsKQPuoWHY1aLfVYNJNz EyYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=RyEzalHh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c13-20020a056a000acd00b005787fd98b6bsi12564826pfl.323.2022.12.19.12.32.04; Mon, 19 Dec 2022 12:32:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=RyEzalHh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232657AbiLSUad (ORCPT + 99 others); Mon, 19 Dec 2022 15:30:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232641AbiLSUaD (ORCPT ); Mon, 19 Dec 2022 15:30:03 -0500 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18B488FCC; Mon, 19 Dec 2022 12:29:58 -0800 (PST) Received: from pps.filterd (m0134422.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJKRChb029372; Mon, 19 Dec 2022 20:29:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=jD3LLgSm37nQRHMfKV4CUST8QKYllFqCcPy8zK6SIaM=; b=RyEzalHhrx/aHW3orjqb8LBtPsIHKMSwHC0xpg9eaj/HNB2G6wFZy4pO5S/S8fd+oerW wmCJvFkeF7AxHxMq//9NyHmZM+gepIK8w5ZpT6mPKBLhtknXQLmZruX1Kq+Qo21mbkgo a+zmJ35F2pbb1rSjMZK3zyZtN9egQnXbcluUmf3axRXe/CU1Urc9OltNUTn2EFk9cD96 WyL73c+6rrltYL9kqMV9A2g1Lk/6u51nTSi91Lpg1BWWzjmX4g7QwOL6EJL6VQGiF7tn gFAT7GoqnNyS9arIrU+H2L6n0Y7MKQuLl921TyI0Rl2Cbhr52HHp3B5kNJeDWxVT8YNv 9g== Received: from p1lg14880.it.hpe.com (p1lg14880.it.hpe.com [16.230.97.201]) by mx0b-002e3701.pphosted.com (PPS) with ESMTPS id 3mjy0fr0hu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:29:44 +0000 Received: from p1lg14885.dc01.its.hpecorp.net (unknown [10.119.18.236]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14880.it.hpe.com (Postfix) with ESMTPS id 877EC807EE5; Mon, 19 Dec 2022 20:29:43 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14885.dc01.its.hpecorp.net (Postfix) with ESMTP id 1DC488065EB; Mon, 19 Dec 2022 20:29:43 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net, paulmck@kernel.org, frederic@kernel.org, quic_neeraju@quicinc.com, josh@joshtriplett.org Cc: linux-crypto@vger.kernel.org, rcu@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 2/3] rcu: print first CPU on expedited stall line Date: Mon, 19 Dec 2022 14:29:09 -0600 Message-Id: <20221219202910.3063036-3-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219202910.3063036-1-elliott@hpe.com> References: <20221219202910.3063036-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: wSTApf6gnSbNoJkuVG0vozCGTGg8BbvZ X-Proofpoint-GUID: wSTApf6gnSbNoJkuVG0vozCGTGg8BbvZ X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 malwarescore=0 clxscore=1015 impostorscore=0 phishscore=0 lowpriorityscore=0 mlxlogscore=812 mlxscore=0 bulkscore=0 priorityscore=1501 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190180 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752675844070460032?= X-GMAIL-MSGID: =?utf-8?q?1752675844070460032?= Include the first CPU number in the first pr_err() call reporting an expedited stall warning. Printing the CPU numbers with subsequent pr_cont() calls can result in the prints being many lines away or being dropped entirely in a busy system. This change ensures there is indication of at least one of the CPUs with the problem with the original message. Before (if prints are interspersed with other prints): rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 13-.... } 32 jiffies s: 6685 root: 0x1/. After: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: {13-.... } 32 jiffies s: 6685 root: 0x1/. Signed-off-by: Robert Elliott --- kernel/rcu/tree_exp.h | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index ed6c3cce28f2..ade6a18e6c07 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -624,8 +624,6 @@ static void synchronize_rcu_expedited_wait(void) if (rcu_stall_is_suppressed()) continue; trace_rcu_stall_warning(rcu_state.name, TPS("ExpeditedStall")); - pr_err("INFO: %s detected expedited stalls on CPUs/tasks: {", - rcu_state.name); ndetected = 0; rcu_for_each_leaf_node(rnp) { ndetected += rcu_print_task_exp_stall(rnp); @@ -637,11 +635,21 @@ static void synchronize_rcu_expedited_wait(void) continue; ndetected++; rdp = per_cpu_ptr(&rcu_data, cpu); - pr_cont(" %d-%c%c%c%c", cpu, - "O."[!!cpu_online(cpu)], - "o."[!!(rdp->grpmask & rnp->expmaskinit)], - "N."[!!(rdp->grpmask & rnp->expmaskinitnext)], - "D."[!!(rdp->cpu_no_qs.b.exp)]); + // print the prefix and the first CPU number together + // under heavy load, the pr_cont prints can be far away or dropped + if (ndetected == 1) + pr_err("INFO: %s detected expedited stalls on CPUs/tasks: {%d-%c%c%c%c", + rcu_state.name, cpu, + "O."[!!cpu_online(cpu)], + "o."[!!(rdp->grpmask & rnp->expmaskinit)], + "N."[!!(rdp->grpmask & rnp->expmaskinitnext)], + "D."[!!(rdp->cpu_no_qs.b.exp)]); + else + pr_cont(" %d-%c%c%c%c", cpu, + "O."[!!cpu_online(cpu)], + "o."[!!(rdp->grpmask & rnp->expmaskinit)], + "N."[!!(rdp->grpmask & rnp->expmaskinitnext)], + "D."[!!(rdp->cpu_no_qs.b.exp)]); } } pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n", From patchwork Mon Dec 19 20:29:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elliott, Robert (Servers)" X-Patchwork-Id: 34757 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2604373wrn; Mon, 19 Dec 2022 12:39:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXtfz5qagDOOnVvE7Fjc/uvys+5V4+Az9JwDt+zVhqcjmidVLEMCrwc9K4GwLMtW1Iuu1UhY X-Received: by 2002:a17:90a:be03:b0:219:10b3:c7b5 with SMTP id a3-20020a17090abe0300b0021910b3c7b5mr11739864pjs.49.1671482371832; Mon, 19 Dec 2022 12:39:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671482371; cv=none; d=google.com; s=arc-20160816; b=kIdNU2qaIng+s07v+Zd1kCy+pPNLHoUHGiMUUbCZovEUIjsu23QMUEwgiyw8m9Y6h7 1rqC6ZsgnTH/fsYPsIQRBs76c1WCdMljcSGaqAZuv6hZBtqfLjbRBZ9SC1ngjTczYFks HRIBST5Lq521mv8LpGYQjYmI8blPvssqK7zH/KY7L8fjQatoEfucg3RWQ4t4nHdWk4Tm +FdcdOc6nUzD2l/bEVujZIzG7i/D9XBvvKFzTq44yceuU+/7MYrqiEo8ZzaBJ/fXV73h iYzTSRduclQE7PlKMs50/j2tdh39WZsX63bHfYp+nFtjcLwQpVcwegVA/eD9qgkOtM6O aj7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0ROCRnzk94uQKB9hFXedgn0T1ir63wjm+lzksBC+x48=; b=oVhgVKMSuyvUWutBPQWGuUATY+UwP+iHzKqO5R8R55VbNdVMUwO6qRZVu3bd6V84aJ WgAZHGqZzkokHoJnt4W9qBwTstRlrcnWR6w4CPmm766dQWbH83y3Y3ezOD94W7IENPrg RBXb7D4g7beZ+5/TpYJPrbKXlp8lqGg9YrW8Sb2kUuvLQCDFCeQ1aPL8bNL0VI3V+JcK tNcFesdB3UbO3ry1eTiDEe8v6Bu0ep1n76/QXHWgyfuZ1K8WNQ2giRGvAUpHSeF7B+y5 7JvPf2K8iMbMSU3YfwWX+f35bndC0myugDWPoVfO5RCM6IYpGrS0HfHf4RFIqeeaSTUE iUpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=YEH+2sfq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nu9-20020a17090b1b0900b00205da45d8a5si18074837pjb.124.2022.12.19.12.39.19; Mon, 19 Dec 2022 12:39:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=YEH+2sfq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232689AbiLSUah (ORCPT + 99 others); Mon, 19 Dec 2022 15:30:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232635AbiLSUaD (ORCPT ); Mon, 19 Dec 2022 15:30:03 -0500 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC1B39582; Mon, 19 Dec 2022 12:29:59 -0800 (PST) Received: from pps.filterd (m0150241.ppops.net [127.0.0.1]) by mx0a-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2BJJOigi017410; Mon, 19 Dec 2022 20:29:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pps0720; bh=0ROCRnzk94uQKB9hFXedgn0T1ir63wjm+lzksBC+x48=; b=YEH+2sfquvZx5fwJcv47RHK/ljj/7a87XAzyJW/gr65VyWvBBA4TcvfdV4a6Wdq6fibe zq0drfo0z6RT835kNfy4qAlZ+IYBRJRFCDgHZsowTth0UBNKeU/TfWMUnvsepSeOR+s5 AFM3fDuacsMCcSok09+E94hGLJaqZrQ6eXhZl9TaUYxVHKRa8Ya/FOolBYUoGs3Xa6Iw hgV2iyud08nD6sXrnf84uVOUnLxpsywNYyjVBvAT/fJoQurCQwIjHO8YYYPAw5gJHVkg YggKt4AXboVPkDKoPlp4izRkTob1jpmX2S/w9JGSaqVVS4stggbkVz2cJ2PtWcMKeffh AQ== Received: from p1lg14879.it.hpe.com (p1lg14879.it.hpe.com [16.230.97.200]) by mx0a-002e3701.pphosted.com (PPS) with ESMTPS id 3mjx3b0d4m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 19 Dec 2022 20:29:45 +0000 Received: from p1lg14885.dc01.its.hpecorp.net (unknown [10.119.18.236]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14879.it.hpe.com (Postfix) with ESMTPS id BBBD64AC60; Mon, 19 Dec 2022 20:29:44 +0000 (UTC) Received: from adevxp033-sys.us.rdlabs.hpecorp.net (unknown [16.231.227.36]) by p1lg14885.dc01.its.hpecorp.net (Postfix) with ESMTP id 49DAE8064BE; Mon, 19 Dec 2022 20:29:44 +0000 (UTC) From: Robert Elliott To: herbert@gondor.apana.org.au, davem@davemloft.net, paulmck@kernel.org, frederic@kernel.org, quic_neeraju@quicinc.com, josh@joshtriplett.org Cc: linux-crypto@vger.kernel.org, rcu@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Elliott Subject: [PATCH 3/3] crypto: tcrypt - suppress RCU stall warnings during speed tests Date: Mon, 19 Dec 2022 14:29:10 -0600 Message-Id: <20221219202910.3063036-4-elliott@hpe.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221219202910.3063036-1-elliott@hpe.com> References: <20221219202910.3063036-1-elliott@hpe.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: b8j5S8cYNvBcF4NRwQ6rkCjGZ592I-ZJ X-Proofpoint-GUID: b8j5S8cYNvBcF4NRwQ6rkCjGZ592I-ZJ X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-19_01,2022-12-15_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 spamscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 bulkscore=0 adultscore=0 lowpriorityscore=0 phishscore=0 suspectscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2212190180 X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752676299765304093?= X-GMAIL-MSGID: =?utf-8?q?1752676299765304093?= Suppress RCU CPU stall warnings while running speed tests. The tcrypt module is intended only for developer usage, so RCU stalls induced by those tests are not necessarily representative of real problems. Speed tests need to disable interrupts or preemption to get results that are not distorted by such interruptions. This triggers more RCU stalls than normal invocations of the crypto functions. If an RCU stall is triggered and reported, the time to print to the console distorts the speed results. Signed-off-by: Robert Elliott --- crypto/tcrypt.c | 113 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 94 insertions(+), 19 deletions(-) diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c index 8645e72a7099..3e9e4adeef02 100644 --- a/crypto/tcrypt.c +++ b/crypto/tcrypt.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -191,12 +192,16 @@ static int test_mb_aead_jiffies(struct test_mb_aead_data *data, int enc, if (!rc) return -ENOMEM; + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { ret = do_mult_aead_op(data, enc, num_mb, rc, prefix); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); pr_info("%s %8d operations in %d seconds (%12llu bytes)\n", prefix, bcount * num_mb, secs, (u64)bcount * blen * num_mb); @@ -218,19 +223,25 @@ static int test_mb_aead_cycles(struct test_mb_aead_data *data, int enc, return -ENOMEM; /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { ret = do_mult_aead_op(data, enc, num_mb, rc, prefix); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); ret = do_mult_aead_op(data, enc, num_mb, rc, prefix); end = get_cycles(); + rcu_suppress_end(); if (ret) goto out; @@ -470,6 +481,7 @@ static int test_aead_jiffies(struct aead_request *req, int enc, int bcount; int ret; + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { if (enc) @@ -477,9 +489,12 @@ static int test_aead_jiffies(struct aead_request *req, int enc, else ret = do_one_aead_op(req, crypto_aead_decrypt(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } } + rcu_suppress_end(); pr_info("%s %8d operations in %d seconds (%12llu bytes)\n", prefix, bcount, secs, (u64)bcount * blen); @@ -494,26 +509,32 @@ static int test_aead_cycles(struct aead_request *req, int enc, int i; /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { if (enc) ret = do_one_aead_op(req, crypto_aead_encrypt(req)); else ret = do_one_aead_op(req, crypto_aead_decrypt(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); if (enc) ret = do_one_aead_op(req, crypto_aead_encrypt(req)); else ret = do_one_aead_op(req, crypto_aead_decrypt(req)); end = get_cycles(); + rcu_suppress_end(); if (ret) goto out; @@ -746,12 +767,16 @@ static int test_ahash_jiffies_digest(struct ahash_request *req, int blen, int bcount; int ret; + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { ret = do_one_ahash_op(req, crypto_ahash_digest(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } } + rcu_suppress_end(); pr_info("%s %6u opers/sec, %9lu bytes/sec\n", prefix, bcount / secs, ((long)bcount * blen) / secs); @@ -769,21 +794,29 @@ static int test_ahash_jiffies(struct ahash_request *req, int blen, if (plen == blen) return test_ahash_jiffies_digest(req, blen, secs, prefix); + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { ret = do_one_ahash_op(req, crypto_ahash_init(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } for (pcount = 0; pcount < blen; pcount += plen) { ret = do_one_ahash_op(req, crypto_ahash_update(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } } ret = do_one_ahash_op(req, crypto_ahash_final(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } } + rcu_suppress_end(); pr_info("%s %6u opers/sec, %9lu bytes/sec\n", prefix, bcount / secs, ((long)bcount * blen) / secs); @@ -798,23 +831,31 @@ static int test_ahash_cycles_digest(struct ahash_request *req, int blen, int ret, i; /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { ret = do_one_ahash_op(req, crypto_ahash_digest(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); ret = do_one_ahash_op(req, crypto_ahash_digest(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } end = get_cycles(); + rcu_suppress_end(); cycles += end - start; } @@ -839,24 +880,33 @@ static int test_ahash_cycles(struct ahash_request *req, int blen, return test_ahash_cycles_digest(req, blen, prefix); /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { ret = do_one_ahash_op(req, crypto_ahash_init(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } for (pcount = 0; pcount < blen; pcount += plen) { ret = do_one_ahash_op(req, crypto_ahash_update(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } ret = do_one_ahash_op(req, crypto_ahash_final(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); ret = do_one_ahash_op(req, crypto_ahash_init(req)); @@ -864,14 +914,19 @@ static int test_ahash_cycles(struct ahash_request *req, int blen, goto out; for (pcount = 0; pcount < blen; pcount += plen) { ret = do_one_ahash_op(req, crypto_ahash_update(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } ret = do_one_ahash_op(req, crypto_ahash_final(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } end = get_cycles(); + rcu_suppress_end(); cycles += end - start; } @@ -1039,12 +1094,16 @@ static int test_mb_acipher_jiffies(struct test_mb_skcipher_data *data, int enc, if (!rc) return -ENOMEM; + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { ret = do_mult_acipher_op(data, enc, num_mb, rc, prefix); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); pr_info("%s %8d operations in %d seconds (%12llu bytes)\n", prefix, bcount * num_mb, secs, (u64)bcount * blen * num_mb); @@ -1066,19 +1125,25 @@ static int test_mb_acipher_cycles(struct test_mb_skcipher_data *data, int enc, return -ENOMEM; /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { ret = do_mult_acipher_op(data, enc, num_mb, rc, prefix); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); ret = do_mult_acipher_op(data, enc, num_mb, rc, prefix); end = get_cycles(); + rcu_suppress_end(); if (ret) goto out; @@ -1270,6 +1335,7 @@ static int test_acipher_jiffies(struct skcipher_request *req, int enc, int bcount; int ret; + rcu_suppress_start(); for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { if (enc) @@ -1279,9 +1345,12 @@ static int test_acipher_jiffies(struct skcipher_request *req, int enc, ret = do_one_acipher_op(req, crypto_skcipher_decrypt(req)); - if (ret) + if (ret) { + rcu_suppress_end(); return ret; + } } + rcu_suppress_end(); pr_info("%s %8d operations in %d seconds (%12llu bytes)\n", prefix, bcount, secs, (u64)bcount * blen); @@ -1296,6 +1365,7 @@ static int test_acipher_cycles(struct skcipher_request *req, int enc, int i; /* Warm-up run. */ + rcu_suppress_start(); for (i = 0; i < 4; i++) { if (enc) ret = do_one_acipher_op(req, @@ -1304,14 +1374,18 @@ static int test_acipher_cycles(struct skcipher_request *req, int enc, ret = do_one_acipher_op(req, crypto_skcipher_decrypt(req)); - if (ret) + if (ret) { + rcu_suppress_end(); goto out; + } } + rcu_suppress_end(); /* The real thing. */ for (i = 0; i < 8; i++) { cycles_t start, end; + rcu_suppress_start(); start = get_cycles(); if (enc) ret = do_one_acipher_op(req, @@ -1320,6 +1394,7 @@ static int test_acipher_cycles(struct skcipher_request *req, int enc, ret = do_one_acipher_op(req, crypto_skcipher_decrypt(req)); end = get_cycles(); + rcu_suppress_end(); if (ret) goto out;