From patchwork Tue Nov 29 16:08:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 27324 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp431819wrr; Tue, 29 Nov 2022 08:12:02 -0800 (PST) X-Google-Smtp-Source: AA0mqf5HrwsncP/wW+iLpjDCk0TcfruUnSRp1b0l4ulbjXrhK1kLxO2kkrVFetzkqnWtHjhHSIdD X-Received: by 2002:a05:6402:3892:b0:454:cbef:c161 with SMTP id fd18-20020a056402389200b00454cbefc161mr35039699edb.365.1669738322299; Tue, 29 Nov 2022 08:12:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669738322; cv=none; d=google.com; s=arc-20160816; b=esNvR5/qYK6BSezopWaLF8FMD1z7bxr9e0xhTQ66xt6S7rjtM6gfbE2B7EooeSKjwL k+3uJGC1c+AFlkLcVtXT3f9EduUtIxgby3LOf1/dguXjgbQFRaTTk2aMhIandpcm7RdG Anfib9AYIiQ4IVdk1oWOlEdZuRvd+VpUfnbKdM7YRIX3gNue9zNSlvRokjpBZs1CTRoW MAPXVWvNU1tUiQ7O3tHG+HKcRjPcH7OtEsDaqziIiX/cyi7pkzu9HXe5LDylNZtj8lPj jTEfAtk5Lann/uf1GnbgyXW/rR7AzXLud5lqfGjduxyEX2d70+odoz+mDoZdfA7MSUJ4 EQmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7LUjTHm6kEMXqhLx7VxQfNEptYV9vMbIIcB198xn198=; b=YlIuC+K5iKb/k4BJeVA+WEwXBbANn9J0nEnFJ6isNxrwDwAFK2r0KYRtPTvWDyzEM2 e7ogngQ/VJbSyReQaRmzt94/kIn3mL8e/MGC4BKSwPDBATGs0d6IVoUBI2ynklDNE72z rlJD9lQNmX/N+/AWN3mFSuFq4WQuWcRh78q4luwi9vUGfy/EMQo/OmcLPH5TObcslTFX lTayAGHPoRTtBDw+M8Akb6t/c/k1smbFKzQW5Snli6JlFKPv8HqYXtS/xyQQfRf8bfLQ 3BEAZ/n1IhHzfjVDL42ZtaRqr49MdW6T8FJJtiYv6h2/yKX8m0O4yuzCeCnE7N+J3RzY qp8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=jSW4sRjQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o9-20020a170906974900b0078d0f57b0e2si13425929ejy.412.2022.11.29.08.11.36; Tue, 29 Nov 2022 08:12:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=jSW4sRjQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232468AbiK2QIh (ORCPT + 99 others); Tue, 29 Nov 2022 11:08:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232415AbiK2QIb (ORCPT ); Tue, 29 Nov 2022 11:08:31 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DCEABC9A; Tue, 29 Nov 2022 08:08:28 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 96185B8169F; Tue, 29 Nov 2022 16:08:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 790EBC433D6; Tue, 29 Nov 2022 16:08:25 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="jSW4sRjQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1669738103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7LUjTHm6kEMXqhLx7VxQfNEptYV9vMbIIcB198xn198=; b=jSW4sRjQKLRgBYxtBmNTe2eH1ItnygDF0l+PYQ8E6KhFhRpD7R6L+QpZ7M+mx4iNF3V/q3 8wF6y/3IV8ZFU9hmadJROI0OJj/6J8CRT3HmcQkMid6EYNPdgM6AZIKblEOcswHm92+PXx IRTiGPROC791fB84DA/G+hX+B28pRSI= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id c9d54766 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 29 Nov 2022 16:08:22 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Cc: "Jason A. Donenfeld" , Sultan Alsawaf , Dominik Brodowski , Sebastian Andrzej Siewior , Thomas Gleixner Subject: [PATCH v2] random: spread out jitter callback to different CPUs Date: Tue, 29 Nov 2022 17:08:11 +0100 Message-Id: <20221129160811.563011-1-Jason@zx2c4.com> In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750847530835140965?= X-GMAIL-MSGID: =?utf-8?q?1750847530835140965?= Rather than merely hoping that the callback gets called on another CPU, arrange for that to actually happen, by round robining which CPU the timer fires on. This way, on multiprocessor machines, we exacerbate jitter by touching the same memory from multiple different cores. There's a little bit of tricky bookkeeping involved here, because using timer_setup_on_stack() + add_timer_on() + del_timer_sync() is a recipe for disaster. See this sample code: . Mitigating this by calling del_timer_sync() (or try_to_del_timer_sync()) before each add_timer_on() also isn't satisfactory, because that halts the main entropy collecting loop while it waits, trying to acquire the timer base spinlock. So instead, poll a boolean inside of the main loop to indicate when the timer callback has finished executing. This serializes the callbacks while still ensuring the main loop continues. Cc: Sultan Alsawaf Cc: Dominik Brodowski Cc: Sebastian Andrzej Siewior Cc: Thomas Gleixner Signed-off-by: Jason A. Donenfeld --- drivers/char/random.c | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index 7b71cea6a6ab..40d0395ea566 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1232,7 +1232,9 @@ void __cold rand_initialize_disk(struct gendisk *disk) struct entropy_timer_state { unsigned long entropy; struct timer_list timer; - unsigned int samples, samples_per_bit; + atomic_t samples; + unsigned int samples_per_bit; + bool is_running; }; /* @@ -1250,10 +1252,9 @@ static void __cold entropy_timer(struct timer_list *timer) { struct entropy_timer_state *state = container_of(timer, struct entropy_timer_state, timer); - if (++state->samples == state->samples_per_bit) { + if (atomic_inc_return(&state->samples) % state->samples_per_bit == 0) credit_init_bits(1); - state->samples = 0; - } + smp_store_release(&state->is_running, false); } /* @@ -1263,9 +1264,10 @@ static void __cold entropy_timer(struct timer_list *timer) static void __cold try_to_generate_entropy(void) { enum { NUM_TRIAL_SAMPLES = 8192, MAX_SAMPLES_PER_BIT = HZ / 15 }; - struct entropy_timer_state stack; + struct entropy_timer_state stack = { 0 }; unsigned int i, num_different = 0; unsigned long last = random_get_entropy(); + int cpu = -1; for (i = 0; i < NUM_TRIAL_SAMPLES - 1; ++i) { stack.entropy = random_get_entropy(); @@ -1277,19 +1279,38 @@ static void __cold try_to_generate_entropy(void) if (stack.samples_per_bit > MAX_SAMPLES_PER_BIT) return; - stack.samples = 0; timer_setup_on_stack(&stack.timer, entropy_timer, 0); while (!crng_ready() && !signal_pending(current)) { - if (!timer_pending(&stack.timer)) - mod_timer(&stack.timer, jiffies); + /* + * Check both !timer_pending() and state->is_running to ensure that any previous + * callback has finished executing, before queueing the next one. + */ + if (!timer_pending(&stack.timer) && !smp_load_acquire(&stack.is_running)) { + preempt_disable(); + + /* Basic CPU round-robin, which avoids the current CPU. */ + do { + cpu = cpumask_next(cpu, cpu_online_mask); + if (cpu == nr_cpumask_bits) + cpu = cpumask_first(cpu_online_mask); + } while (cpu == smp_processor_id() && cpumask_weight(cpu_online_mask) > 1); + + /* Expiring the timer at `jiffies` means it's the next tick. */ + stack.timer.expires = jiffies; + + smp_store_release(&stack.is_running, true); + add_timer_on(&stack.timer, cpu); + + preempt_enable(); + } mix_pool_bytes(&stack.entropy, sizeof(stack.entropy)); schedule(); stack.entropy = random_get_entropy(); } + mix_pool_bytes(&stack.entropy, sizeof(stack.entropy)); del_timer_sync(&stack.timer); destroy_timer_on_stack(&stack.timer); - mix_pool_bytes(&stack.entropy, sizeof(stack.entropy)); }