From patchwork Tue Jul 25 23:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 125994 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp82561vqo; Tue, 25 Jul 2023 17:05:32 -0700 (PDT) X-Google-Smtp-Source: APBJJlHr/eSqLOti8OwPKg5dJFmb5cKsXy7odjxcF3QlmRP2qcJKPtVzhf/0sZeJljLuTSccX4a/ X-Received: by 2002:a17:906:cc14:b0:98d:76f8:217b with SMTP id ml20-20020a170906cc1400b0098d76f8217bmr257600ejb.73.1690329932018; Tue, 25 Jul 2023 17:05:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690329932; cv=none; d=google.com; s=arc-20160816; b=jwdo9bQJBqURaInIAG4znp0OVxIZPGoCZLiBQNieh2aGoY04cIfLuN2Ey1mdZ25w9z J6zu+h1+lZRrP6Ea2Ea1+O9hI9NbaZgExDZInOEpKq2TASyx5WHdL7gMTOHN7juVR/mm SiGLr3kGgX2I0Hcvj5H5+RP3kdp8CMB7pYS7QNFbq3tIII19hqe7nqWrK+W6zumUzWN9 ZUID+7u63eBgp0np5MTpHyRBb/Fm7vfe5EoH8P5EfjYTe9vnX7A0rBd/Wt0RMusbLqXL R1CSn6q+rLHaIbaIOOc4xpxtEbV+OgnJwAcx6ogMwNMZFiDTq8ji9cUO7rx3podD8lRc lHeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SjqXIuVtq0u8YshojVwovW+pCVt0mgB/LilQjFvvm4k=; fh=d2L8aFekZ77L5iP9ormEBef4A5rzZzEifl0K7lOf2zs=; b=DANho+8Rpm3afw8SLgZjbGWvXLie+pJh0TeyOWffEPOKdnrrCKLXC0derITUuBEPpW Vxw457geEbeBL6jGrTia71hd/vMs8f8iwngMt5NQ3zerX0/0o7itlzUSohidsY9kp5Nu OOODI9uLYmuG0BE6j8bSWrgIFYTIOAosaSlvwHQYHZJ4ABjZHUaF8VHtDNwbLX7pSBEB dIV3GGxeoa0Qz0RXo5SjY1M1S//k5ZSeYNDyUm6yrbk8Ex94hgEnrhJobEUzOJzpnRUx IOCihBffTAklMS5v5p8WDPDx4bewfUDMCSfSfZLbAiHbWWVLV+ZxNl4106zq+uG/NiWd qq6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=YpU7e+Pj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kd10-20020a17090798ca00b00992f3097ed4si8540613ejc.871.2023.07.25.17.05.06; Tue, 25 Jul 2023 17:05:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=YpU7e+Pj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231805AbjGYX3c (ORCPT + 99 others); Tue, 25 Jul 2023 19:29:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231705AbjGYX3a (ORCPT ); Tue, 25 Jul 2023 19:29:30 -0400 Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E6D819BA for ; Tue, 25 Jul 2023 16:29:29 -0700 (PDT) Received: by mail-io1-xd32.google.com with SMTP id ca18e2360f4ac-7878e573827so224020839f.1 for ; Tue, 25 Jul 2023 16:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690327767; x=1690932567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SjqXIuVtq0u8YshojVwovW+pCVt0mgB/LilQjFvvm4k=; b=YpU7e+Pj0/atr4EbsOJuU3upQRDKhMb6J0aC/Kwb5sK5X1H4MDa7KtxhGNk+JprQa8 xnDQkG8c0nNaGaiUAxLBwoHST4BNpZ2nFNZPTPeF7D0/5SyEMgWEAfwg/HQLLaNJ5GPS CVrYBTezNf2/CmiE7wAzXkmnNs+/HVvnLeI5w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690327767; x=1690932567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SjqXIuVtq0u8YshojVwovW+pCVt0mgB/LilQjFvvm4k=; b=edg07nKGog7ElI3oqPFwG5ySav9qCI2L6R4kWtOf6hmOa1MVaH5exUq5D3++PgpNKL OD3UEF/kGCIC3i0EtE88+ug8zTCaF0vlMqITHcbwFN6cpG0w0KoqzxVzXSDCRD/kT1j6 VG1o4aqIDTbr71Z+WQXSjekwzaTFFDsKMP2U903IyAKC/cZ12N9HuwVj9IXdJTQTQFbD qpZMpaF9ZomnwMxUJ5vwo3pD52GdAy1Re8ad6FvQxHdzx97jdALkl4mQVpYeUF96Q9Hu mZ8uY6RGoteMOqJojW0NghSy3YqCxTFYAeFU4DB9clo1+jBCLkMGnnw4cBbZASepbzvD gzVg== X-Gm-Message-State: ABy/qLbs5NwSFpStFVETaLYVUNDX4FgdrwEV1vEjYAyRgywESrv7VN1V 4Fd1dIFRFM7yd/E4WhCPWdzEa21BIajgUWDk3a0= X-Received: by 2002:a6b:e818:0:b0:783:6ff9:a6c7 with SMTP id f24-20020a6be818000000b007836ff9a6c7mr334374ioh.6.1690327767704; Tue, 25 Jul 2023 16:29:27 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (156.190.123.34.bc.googleusercontent.com. [34.123.190.156]) by smtp.gmail.com with ESMTPSA id m18-20020a02c892000000b0041d859c5721sm3932053jao.64.2023.07.25.16.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:29:26 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett Cc: rcu@vger.kernel.org, "Joel Fernandes (Google)" Subject: [PATCH 1/5] rcutorture: Fix stuttering races and other issues Date: Tue, 25 Jul 2023 23:29:06 +0000 Message-ID: <20230725232913.2981357-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230725232913.2981357-1-joel@joelfernandes.org> References: <20230725232913.2981357-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772439398764976385 X-GMAIL-MSGID: 1772439398764976385 The stuttering code isn't functioning as expected. Ideally, it should pause the torture threads for a designated period before resuming. Yet, it fails to halt the test for the correct duration. Additionally, a race condition exists, potentially causing the stuttering code to pause for an extended period if the 'spt' variable is non-zero due to the stutter orchestration thread's inadequate CPU time. Moreover, over-stuttering can hinder RCU's progress on TREE07 kernels. This happens as the stuttering code may run within a softirq due to RCU callbacks. Consequently, ksoftirqd keeps a CPU busy for several seconds, thus obstructing RCU's progress. This situation triggers a warning message in the logs: [ 2169.481783] rcu_torture_writer: rtort_pipe_count: 9 This warning suggests that an RCU torture object, although invisible to RCU readers, couldn't make it past the pipe array and be freed -- a strong indication that there weren't enough grace periods during the stutter interval. To address these issues, this patch sets the "stutter end" time to an absolute point in the future set by the main stutter thread. This is then used for waiting in stutter_wait(). While the stutter thread still defines this absolute time, the waiters' waiting logic doesn't rely on the stutter thread receiving sufficient CPU time to halt the stuttering as the halting is now self-controlled. Signed-off-by: Joel Fernandes (Google) --- kernel/torture.c | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/kernel/torture.c b/kernel/torture.c index 68dba4ecab5c..63f8f2a7d960 100644 --- a/kernel/torture.c +++ b/kernel/torture.c @@ -719,7 +719,7 @@ static void torture_shutdown_cleanup(void) * suddenly applied to or removed from the system. */ static struct task_struct *stutter_task; -static int stutter_pause_test; +static ktime_t stutter_till_abs_time; static int stutter; static int stutter_gap; @@ -729,30 +729,17 @@ static int stutter_gap; */ bool stutter_wait(const char *title) { - unsigned int i = 0; bool ret = false; - int spt; + ktime_t now_ns, till_ns; cond_resched_tasks_rcu_qs(); - spt = READ_ONCE(stutter_pause_test); - for (; spt; spt = READ_ONCE(stutter_pause_test)) { - if (!ret && !rt_task(current)) { - sched_set_normal(current, MAX_NICE); - ret = true; - } - if (spt == 1) { - torture_hrtimeout_jiffies(1, NULL); - } else if (spt == 2) { - while (READ_ONCE(stutter_pause_test)) { - if (!(i++ & 0xffff)) - torture_hrtimeout_us(10, 0, NULL); - cond_resched(); - } - } else { - torture_hrtimeout_jiffies(round_jiffies_relative(HZ), NULL); - } - torture_shutdown_absorb(title); + now_ns = ktime_get(); + till_ns = READ_ONCE(stutter_till_abs_time); + if (till_ns && ktime_before(now_ns, till_ns)) { + torture_hrtimeout_ns(ktime_sub(till_ns, now_ns), 0, NULL); + ret = true; } + torture_shutdown_absorb(title); return ret; } EXPORT_SYMBOL_GPL(stutter_wait); @@ -763,23 +750,16 @@ EXPORT_SYMBOL_GPL(stutter_wait); */ static int torture_stutter(void *arg) { - DEFINE_TORTURE_RANDOM(rand); - int wtime; + ktime_t till_ns; VERBOSE_TOROUT_STRING("torture_stutter task started"); do { if (!torture_must_stop() && stutter > 1) { - wtime = stutter; - if (stutter > 2) { - WRITE_ONCE(stutter_pause_test, 1); - wtime = stutter - 3; - torture_hrtimeout_jiffies(wtime, &rand); - wtime = 2; - } - WRITE_ONCE(stutter_pause_test, 2); - torture_hrtimeout_jiffies(wtime, NULL); + till_ns = ktime_add_ns(ktime_get(), + jiffies_to_nsecs(stutter)); + WRITE_ONCE(stutter_till_abs_time, till_ns); + torture_hrtimeout_jiffies(stutter - 1, NULL); } - WRITE_ONCE(stutter_pause_test, 0); if (!torture_must_stop()) torture_hrtimeout_jiffies(stutter_gap, NULL); torture_shutdown_absorb("torture_stutter");