Message ID | 20230528084652.5f3b48f0@rorschach.local.home |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp954923vqr; Sun, 28 May 2023 06:27:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5Kx/5vACYdtFjdbPWubI70Pzl7Kah3c/sau99qgwk+rNu+l/l9333EgOWQjsmZNKOZpO+D X-Received: by 2002:a17:90a:1909:b0:256:4189:2b0d with SMTP id 9-20020a17090a190900b0025641892b0dmr5918787pjg.12.1685280472508; Sun, 28 May 2023 06:27:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685280472; cv=none; d=google.com; s=arc-20160816; b=yoRpbYXbgx50MneK6Zv1DzNGU55wXu3UNm1ELCyB/nsLW+7hOtC3yXAEGX8E3ImCE1 DR1fpFmtagy86adXmD5zrknrpKiVAm/xfJ1qbaKWlMM38jqMYP65a/jj4zWyheTf0utq efh8LKD0R2OU0M5cOWSphgCcofFuQEs4g7Fos86ohe4KlmuZ7mjh1eAkHuSfD1cOFYBr VRRGX3sfU+Nq5MgBkUWWuywbotHmlwrvYhmpmDIC4CEdoZctXFCaiQqYCKxtGdJ1tRhD Mg672Jbtx564zMI8f2DoAPBCbgDUjhiv9lND8pPhtneNA0I3d95be+5rFLFio2HG+e1W uDkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:subject:cc:to:from:date; bh=UwMC7IXYUtX8bH/VQ3Zdp7EhrvRFVAE/pFB0Y1XOOEA=; b=uGqvMCJi2tTCv/HdVNcDm1MkWvhhbvAJ2JvjBId89y1CQdv9Jzc0T48Rd/c6vZYsmO WLfQbvj2TTmLlrpXeo5cCEYLhtyCilszvsrFTzENuReYltql5YH2jZWAZy9hP1MUUGeE AjXewItqAk76po5YcewwOXxnqU07LUehhyfqqd7NcHkng6iTB6MxYGzOrhjONP3NR6MY dNfvayJSxZsDM899lcAzNbORnc0QsOKYuGWFtvZuseOpdcgGj5ioY0KMXJtAJPynzpyq QVZRrpuue1yfiCOeZSbo9JNK/vTYRj+IzeSaRvGverFmEDjoJfIOXjG8+9iuAl1zoEzD PvWg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bj19-20020a17090b089300b0025345363ef3si8202109pjb.115.2023.05.28.06.27.40; Sun, 28 May 2023 06:27:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229525AbjE1MrA (ORCPT <rfc822;syz17693488234@gmail.com> + 99 others); Sun, 28 May 2023 08:47:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229447AbjE1Mq7 (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Sun, 28 May 2023 08:46:59 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCC42B9 for <linux-kernel@vger.kernel.org>; Sun, 28 May 2023 05:46:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 71E9961460 for <linux-kernel@vger.kernel.org>; Sun, 28 May 2023 12:46:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1AA83C433D2; Sun, 28 May 2023 12:46:55 +0000 (UTC) Date: Sun, 28 May 2023 08:46:52 -0400 From: Steven Rostedt <rostedt@goodmis.org> To: LKML <linux-kernel@vger.kernel.org>, x86@kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@kernel.org>, Borislav Petkov <bp@alien8.de>, Peter Zijlstra <peterz@infradead.org> Subject: [PATCH] x86/alternatives: Add cond_resched() to text_poke_bp_batch() Message-ID: <20230528084652.5f3b48f0@rorschach.local.home> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.0 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767144656570846093?= X-GMAIL-MSGID: =?utf-8?q?1767144656570846093?= |
Series |
x86/alternatives: Add cond_resched() to text_poke_bp_batch()
|
|
Commit Message
Steven Rostedt
May 28, 2023, 12:46 p.m. UTC
From: "Steven Rostedt (Google)" <rostedt@goodmis.org> Debugging in the kernel has started slowing down the kernel by a noticeable amount. The ftrace start up tests are triggering the softlockup watchdog on some boxes. This is caused by the start up tests that enable function and function graph tracing several times. Sprinkling cond_resched() just in the start up test code was not enough to stop the softlockup from triggering. It would sometimes trigger in the text_poke_bp_batch() code. The text_poke_bp_batch() is run in schedulable context. Add cond_resched() between each phase (adding the int3, updating the code, and removing the int3). This keeps the softlockup from triggering in the start up tests. Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> --- arch/x86/kernel/alternative.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Comments
On Sun, 28 May 2023 08:46:52 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > From: "Steven Rostedt (Google)" <rostedt@goodmis.org> > > Debugging in the kernel has started slowing down the kernel by a > noticeable amount. The ftrace start up tests are triggering the softlockup > watchdog on some boxes. This is caused by the start up tests that enable > function and function graph tracing several times. Sprinkling > cond_resched() just in the start up test code was not enough to stop the > softlockup from triggering. It would sometimes trigger in the > text_poke_bp_batch() code. > > The text_poke_bp_batch() is run in schedulable context. Add > cond_resched() between each phase (adding the int3, updating the code, and > removing the int3). This keeps the softlockup from triggering in the start > up tests. > > Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> > --- > arch/x86/kernel/alternative.c | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c > index f615e0cb6d93..e024eddd457f 100644 > --- a/arch/x86/kernel/alternative.c > +++ b/arch/x86/kernel/alternative.c > @@ -1953,6 +1953,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > */ > atomic_set_release(&bp_desc.refs, 1); > > + /* > + * Function tracing can enable thousands of places that need to be > + * updated. This can take quite some time, and with full kernel debugging > + * enabled, this could cause the softlockup watchdog to trigger. > + * Add cond_resched() calls to each phase. > + */ > + cond_resched(); Hmm, why don't you put this between the first step (put int3) and the second step (put other bytes)? I guess those would takes more time. Thank you, > + > /* > * Corresponding read barrier in int3 notifier for making sure the > * nr_entries and handler are correctly ordered wrt. patching. > @@ -2030,6 +2038,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > * better safe than sorry (plus there's not only Intel). > */ > text_poke_sync(); > + cond_resched(); > } > > /* > @@ -2049,8 +2058,10 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > do_sync++; > } > > - if (do_sync) > + if (do_sync) { > text_poke_sync(); > + cond_resched(); > + } > > /* > * Remove and wait for refs to be zero. > -- > 2.39.2 >
On Mon, 29 May 2023 11:52:46 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > + /* > > + * Function tracing can enable thousands of places that need to be > > + * updated. This can take quite some time, and with full kernel debugging > > + * enabled, this could cause the softlockup watchdog to trigger. > > + * Add cond_resched() calls to each phase. > > + */ > > + cond_resched(); > > Hmm, why don't you put this between the first step (put int3) and the > second step (put other bytes)? I guess those would takes more time. Ah you're right. I still want this here to clear the 'need resched' flag before we start the process, but I did miss one after the first loop. Thanks for the review! -- Steve
On Sun, May 28, 2023 at 08:46:52AM -0400, Steven Rostedt wrote: > From: "Steven Rostedt (Google)" <rostedt@goodmis.org> > > Debugging in the kernel has started slowing down the kernel by a > noticeable amount. The ftrace start up tests are triggering the softlockup > watchdog on some boxes. This is caused by the start up tests that enable > function and function graph tracing several times. Sprinkling > cond_resched() just in the start up test code was not enough to stop the > softlockup from triggering. It would sometimes trigger in the > text_poke_bp_batch() code. > > The text_poke_bp_batch() is run in schedulable context. Add > cond_resched() between each phase (adding the int3, updating the code, and > removing the int3). This keeps the softlockup from triggering in the start > up tests. > > Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> > --- > arch/x86/kernel/alternative.c | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c > index f615e0cb6d93..e024eddd457f 100644 > --- a/arch/x86/kernel/alternative.c > +++ b/arch/x86/kernel/alternative.c > @@ -1953,6 +1953,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > */ > atomic_set_release(&bp_desc.refs, 1); > > + /* > + * Function tracing can enable thousands of places that need to be > + * updated. This can take quite some time, and with full kernel debugging > + * enabled, this could cause the softlockup watchdog to trigger. > + * Add cond_resched() calls to each phase. > + */ > + cond_resched(); But but but... you can only have TP_VEC_MAX pokes queued, which is 256 on normal setups. Please explain how this leads to problems and why you need _3_ reschedule points here.
On Tue, 30 May 2023 14:01:48 +0200 Peter Zijlstra <peterz@infradead.org> wrote: > On Sun, May 28, 2023 at 08:46:52AM -0400, Steven Rostedt wrote: > > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c > > index f615e0cb6d93..e024eddd457f 100644 > > --- a/arch/x86/kernel/alternative.c > > +++ b/arch/x86/kernel/alternative.c > > @@ -1953,6 +1953,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > > */ > > atomic_set_release(&bp_desc.refs, 1); > > > > + /* > > + * Function tracing can enable thousands of places that need to be > > + * updated. This can take quite some time, and with full kernel debugging > > + * enabled, this could cause the softlockup watchdog to trigger. > > + * Add cond_resched() calls to each phase. > > + */ > > + cond_resched(); > > But but but... you can only have TP_VEC_MAX pokes queued, which is 256 > on normal setups. > > Please explain how this leads to problems and why you need _3_ > reschedule points here. Maybe this was me being overly paranoid (and thinking, it doesn't hurt). I could try it with just adding one (to make sure it happens every flush), and see if it doesn't trigger the softlock up. -- Steve
On Tue, 30 May 2023 14:01:48 +0200 Peter Zijlstra <peterz@infradead.org> wrote: > > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c > > index f615e0cb6d93..e024eddd457f 100644 > > --- a/arch/x86/kernel/alternative.c > > +++ b/arch/x86/kernel/alternative.c > > @@ -1953,6 +1953,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries > > */ > > atomic_set_release(&bp_desc.refs, 1); > > > > + /* > > + * Function tracing can enable thousands of places that need to be > > + * updated. This can take quite some time, and with full kernel debugging > > + * enabled, this could cause the softlockup watchdog to trigger. > > + * Add cond_resched() calls to each phase. > > + */ > > + cond_resched(); > > But but but... you can only have TP_VEC_MAX pokes queued, which is 256 > on normal setups. > > Please explain how this leads to problems and why you need _3_ > reschedule points here. I removed all but this first one and it works fine. I'll resend with the update. -- Steve
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index f615e0cb6d93..e024eddd457f 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1953,6 +1953,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries */ atomic_set_release(&bp_desc.refs, 1); + /* + * Function tracing can enable thousands of places that need to be + * updated. This can take quite some time, and with full kernel debugging + * enabled, this could cause the softlockup watchdog to trigger. + * Add cond_resched() calls to each phase. + */ + cond_resched(); + /* * Corresponding read barrier in int3 notifier for making sure the * nr_entries and handler are correctly ordered wrt. patching. @@ -2030,6 +2038,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries * better safe than sorry (plus there's not only Intel). */ text_poke_sync(); + cond_resched(); } /* @@ -2049,8 +2058,10 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries do_sync++; } - if (do_sync) + if (do_sync) { text_poke_sync(); + cond_resched(); + } /* * Remove and wait for refs to be zero.