From patchwork Mon Jan 23 20:50:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47401 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1811964wrn; Mon, 23 Jan 2023 12:59:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXs6qsDYnF5p3Z9HX27J1FJoboXs/2AhrLjqkjBLQCSjymjV1h2VmbIBRG1hgy0cGmRLl1mp X-Received: by 2002:a17:902:930c:b0:194:d54b:e191 with SMTP id bc12-20020a170902930c00b00194d54be191mr16023397plb.52.1674507551472; Mon, 23 Jan 2023 12:59:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507551; cv=none; d=google.com; s=arc-20160816; b=xIms1YaRX2GhnFOfjRGEZF6fdgHIOg2bH8WRYu7J0k+hq7VnOjb+cPDGNzykaiY5CH BH9W40kspac0TrBqXdGJwUHTZLWEL7h3txeTxCc4nFc0KiOckLzKiErWHGoPEVsDkvwa bRZHW6EyrQ/7zla8312pRoBT+rYoh8ABPis1K1DFjLlRjgBtPDWmOFY3DiY7g2otncN7 OdH1uoQhGp86oERSjIuqWmZhyYjEbFQIJbesRXbSDa8yktDJmrgpPElFbBsk82GS+rBG kZfLsdBxmnZeB9IA9Mld+FfcXSiIcQXHhUrxVcMA0kDSCZag5jX/0dj7BEPT6ar4YfxM 7GrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=8Hefq63R1A5T8+z/XoUnS+Yx/XIKnxRs3D+68vKhmj0=; b=Iw53thbE4YbBe79iBDlEC9nYaag6PPa9dJ+tfe4fhmPbKuTaIi7rARqIuVGjYopcJF agaYCYFSrl4mSU1zAd9661cXsbyy3+msBUxNhnha7Cm3q5ut5fluc40QzuMaSIJIrarq LgI7Hq+iN9RMGvyL/fEbImYDWAMl65+nU3fn8c8eYD3omAHP8iT7VHou+cpzBiFLC9UP aYpgZV183clDDgm2WxUZjkp7h61eQToiQ/n1igFqk3xSdeIUtsjlb5rI1oSaIga2cWVA uGJdkXsuwhDp5ROev9dw6T+sps7JGh/yj3Twjkps6e61OFKenOEYua/H6mdQON+eKQNO /U0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=NIXtWepA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e11-20020a17090301cb00b00192d6fb649dsi411836plh.242.2023.01.23.12.58.59; Mon, 23 Jan 2023 12:59:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=NIXtWepA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232709AbjAWU6o (ORCPT + 99 others); Mon, 23 Jan 2023 15:58:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232577AbjAWU6i (ORCPT ); Mon, 23 Jan 2023 15:58:38 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B6EE61BB; Mon, 23 Jan 2023 12:58:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=8Hefq63R1A5T8+z/XoUnS+Yx/XIKnxRs3D+68vKhmj0=; b=NIXtWepAGesH8iiidjMkReQ/Z7 h8yMsVMS+oh33vrOdD3UNCMhlSwRACdr0gR0Ip1CBPek+Ecrceo3df9FmzfBHde2OJYKs+E0+03uk VFu7GwkoPnB+Hkiu9FfZ/mpX6GRxynOT9WsnY/eYCA1KBQZKTI8Fji1sWJLDGkPwh1Ld+VyLpyyH4 No7/ULD+P03PaRIdlcDJDkdKdRoA40L7Hy2D3k1CGJJ9lZdLCBVpmVqPvHH+BwU2wk9lSXKhOh2M3 M+fqQBblf6raJvLZ7EuXUUBqH63riEv2tbLKVx4nMNx3p65CwWYCgwNgJMGfjcXQ4mO35+VTkt3Ui QDQ7VoOg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pK3se-004WTx-5Q; Mon, 23 Jan 2023 20:57:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 66F7D3008ED; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C262A2075A6D7; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205514.944053334@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:10 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org Subject: [PATCH 1/6] x86: Always inline arch_atomic64 References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848430354693162?= X-GMAIL-MSGID: =?utf-8?q?1755848430354693162?= As already done for regular arch_atomic, always inline arch_atomic64. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/atomic64_32.h | 44 ++++++++++++++++++------------------- arch/x86/include/asm/atomic64_64.h | 36 +++++++++++++++--------------- 2 files changed, 40 insertions(+), 40 deletions(-) --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -71,7 +71,7 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 o, s64 n) +static __always_inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 o, s64 n) { return arch_cmpxchg64(&v->counter, o, n); } @@ -85,7 +85,7 @@ static inline s64 arch_atomic64_cmpxchg( * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline s64 arch_atomic64_xchg(atomic64_t *v, s64 n) +static __always_inline s64 arch_atomic64_xchg(atomic64_t *v, s64 n) { s64 o; unsigned high = (unsigned)(n >> 32); @@ -104,7 +104,7 @@ static inline s64 arch_atomic64_xchg(ato * * Atomically sets the value of @v to @n. */ -static inline void arch_atomic64_set(atomic64_t *v, s64 i) +static __always_inline void arch_atomic64_set(atomic64_t *v, s64 i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -119,7 +119,7 @@ static inline void arch_atomic64_set(ato * * Atomically reads the value of @v and returns it. */ -static inline s64 arch_atomic64_read(const atomic64_t *v) +static __always_inline s64 arch_atomic64_read(const atomic64_t *v) { s64 r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -133,7 +133,7 @@ static inline s64 arch_atomic64_read(con * * Atomically adds @i to @v and returns @i + *@v */ -static inline s64 arch_atomic64_add_return(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_add_return(s64 i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -145,7 +145,7 @@ static inline s64 arch_atomic64_add_retu /* * Other variants with different arithmetic operators: */ -static inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -154,7 +154,7 @@ static inline s64 arch_atomic64_sub_retu } #define arch_atomic64_sub_return arch_atomic64_sub_return -static inline s64 arch_atomic64_inc_return(atomic64_t *v) +static __always_inline s64 arch_atomic64_inc_return(atomic64_t *v) { s64 a; alternative_atomic64(inc_return, "=&A" (a), @@ -163,7 +163,7 @@ static inline s64 arch_atomic64_inc_retu } #define arch_atomic64_inc_return arch_atomic64_inc_return -static inline s64 arch_atomic64_dec_return(atomic64_t *v) +static __always_inline s64 arch_atomic64_dec_return(atomic64_t *v) { s64 a; alternative_atomic64(dec_return, "=&A" (a), @@ -179,7 +179,7 @@ static inline s64 arch_atomic64_dec_retu * * Atomically adds @i to @v. */ -static inline s64 arch_atomic64_add(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_add(s64 i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -194,7 +194,7 @@ static inline s64 arch_atomic64_add(s64 * * Atomically subtracts @i from @v. */ -static inline s64 arch_atomic64_sub(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_sub(s64 i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -208,7 +208,7 @@ static inline s64 arch_atomic64_sub(s64 * * Atomically increments @v by 1. */ -static inline void arch_atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); @@ -221,7 +221,7 @@ static inline void arch_atomic64_inc(ato * * Atomically decrements @v by 1. */ -static inline void arch_atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); @@ -237,7 +237,7 @@ static inline void arch_atomic64_dec(ato * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u) +static __always_inline int arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -248,7 +248,7 @@ static inline int arch_atomic64_add_unle } #define arch_atomic64_add_unless arch_atomic64_add_unless -static inline int arch_atomic64_inc_not_zero(atomic64_t *v) +static __always_inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -257,7 +257,7 @@ static inline int arch_atomic64_inc_not_ } #define arch_atomic64_inc_not_zero arch_atomic64_inc_not_zero -static inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) +static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) { s64 r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -269,7 +269,7 @@ static inline s64 arch_atomic64_dec_if_p #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void arch_atomic64_and(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_and(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -277,7 +277,7 @@ static inline void arch_atomic64_and(s64 c = old; } -static inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -288,7 +288,7 @@ static inline s64 arch_atomic64_fetch_an } #define arch_atomic64_fetch_and arch_atomic64_fetch_and -static inline void arch_atomic64_or(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_or(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -296,7 +296,7 @@ static inline void arch_atomic64_or(s64 c = old; } -static inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -307,7 +307,7 @@ static inline s64 arch_atomic64_fetch_or } #define arch_atomic64_fetch_or arch_atomic64_fetch_or -static inline void arch_atomic64_xor(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_xor(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -315,7 +315,7 @@ static inline void arch_atomic64_xor(s64 c = old; } -static inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v) { s64 old, c = 0; @@ -326,7 +326,7 @@ static inline s64 arch_atomic64_fetch_xo } #define arch_atomic64_fetch_xor arch_atomic64_fetch_xor -static inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v) { s64 old, c = 0; --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -17,7 +17,7 @@ * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline s64 arch_atomic64_read(const atomic64_t *v) +static __always_inline s64 arch_atomic64_read(const atomic64_t *v) { return __READ_ONCE((v)->counter); } @@ -29,7 +29,7 @@ static inline s64 arch_atomic64_read(con * * Atomically sets the value of @v to @i. */ -static inline void arch_atomic64_set(atomic64_t *v, s64 i) +static __always_inline void arch_atomic64_set(atomic64_t *v, s64 i) { __WRITE_ONCE(v->counter, i); } @@ -55,7 +55,7 @@ static __always_inline void arch_atomic6 * * Atomically subtracts @i from @v. */ -static inline void arch_atomic64_sub(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_sub(s64 i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -71,7 +71,7 @@ static inline void arch_atomic64_sub(s64 * true if the result is zero, or false for all * other cases. */ -static inline bool arch_atomic64_sub_and_test(s64 i, atomic64_t *v) +static __always_inline bool arch_atomic64_sub_and_test(s64 i, atomic64_t *v) { return GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, e, "er", i); } @@ -113,7 +113,7 @@ static __always_inline void arch_atomic6 * returns true if the result is 0, or false for all other * cases. */ -static inline bool arch_atomic64_dec_and_test(atomic64_t *v) +static __always_inline bool arch_atomic64_dec_and_test(atomic64_t *v) { return GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, e); } @@ -127,7 +127,7 @@ static inline bool arch_atomic64_dec_and * and returns true if the result is zero, or false for all * other cases. */ -static inline bool arch_atomic64_inc_and_test(atomic64_t *v) +static __always_inline bool arch_atomic64_inc_and_test(atomic64_t *v) { return GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, e); } @@ -142,7 +142,7 @@ static inline bool arch_atomic64_inc_and * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool arch_atomic64_add_negative(s64 i, atomic64_t *v) +static __always_inline bool arch_atomic64_add_negative(s64 i, atomic64_t *v) { return GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, s, "er", i); } @@ -161,25 +161,25 @@ static __always_inline s64 arch_atomic64 } #define arch_atomic64_add_return arch_atomic64_add_return -static inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_sub_return(s64 i, atomic64_t *v) { return arch_atomic64_add_return(-i, v); } #define arch_atomic64_sub_return arch_atomic64_sub_return -static inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_add(s64 i, atomic64_t *v) { return xadd(&v->counter, i); } #define arch_atomic64_fetch_add arch_atomic64_fetch_add -static inline s64 arch_atomic64_fetch_sub(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_sub(s64 i, atomic64_t *v) { return xadd(&v->counter, -i); } #define arch_atomic64_fetch_sub arch_atomic64_fetch_sub -static inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) +static __always_inline s64 arch_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) { return arch_cmpxchg(&v->counter, old, new); } @@ -191,13 +191,13 @@ static __always_inline bool arch_atomic6 } #define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg -static inline s64 arch_atomic64_xchg(atomic64_t *v, s64 new) +static __always_inline s64 arch_atomic64_xchg(atomic64_t *v, s64 new) { return arch_xchg(&v->counter, new); } #define arch_atomic64_xchg arch_atomic64_xchg -static inline void arch_atomic64_and(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_and(s64 i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -205,7 +205,7 @@ static inline void arch_atomic64_and(s64 : "memory"); } -static inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_and(s64 i, atomic64_t *v) { s64 val = arch_atomic64_read(v); @@ -215,7 +215,7 @@ static inline s64 arch_atomic64_fetch_an } #define arch_atomic64_fetch_and arch_atomic64_fetch_and -static inline void arch_atomic64_or(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_or(s64 i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -223,7 +223,7 @@ static inline void arch_atomic64_or(s64 : "memory"); } -static inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_or(s64 i, atomic64_t *v) { s64 val = arch_atomic64_read(v); @@ -233,7 +233,7 @@ static inline s64 arch_atomic64_fetch_or } #define arch_atomic64_fetch_or arch_atomic64_fetch_or -static inline void arch_atomic64_xor(s64 i, atomic64_t *v) +static __always_inline void arch_atomic64_xor(s64 i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -241,7 +241,7 @@ static inline void arch_atomic64_xor(s64 : "memory"); } -static inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v) +static __always_inline s64 arch_atomic64_fetch_xor(s64 i, atomic64_t *v) { s64 val = arch_atomic64_read(v); From patchwork Mon Jan 23 20:50:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47405 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1815078wrn; Mon, 23 Jan 2023 13:05:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXvg3EBSccvmcVkaqrG5s9TyTEz4gyaJaBWFNzGLnbq5yrCQxGfoxoz67FBr3+WH4mNprYjJ X-Received: by 2002:a05:6a20:ba87:b0:af:8e92:3eeb with SMTP id fb7-20020a056a20ba8700b000af8e923eebmr25759648pzb.9.1674507928617; Mon, 23 Jan 2023 13:05:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507928; cv=none; d=google.com; s=arc-20160816; b=lXvSPChEy2gnNI5KFWq9dNfQSvPQHh911JhfxM39eUN5ufu++NcHQy1R5KXMiU0/Zd bDUY4OEN+h1vSnui0SvsbpTx0kj5cLtiojATgwJRH+kwa+jh/SvVYWLKgFoqBERlaK+s /2W1Qgsur5tGActvpIG80Oi3JBGonswEYDWHupgRBtg61AcX3SIML66t77iSJHI21iIt jO6stX2nqGcgU8u75zxrlylasdmeq77V9DBeL7KCVBwVpOHn/DQKTDm4awGmTzead+Jv I+Rb3lgC/N/q6R5HpaJeauNNQFHKbK7zKF99vk3ZuZlqV9mov6QdnOvRqkslkBtUE9el BIPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=LKrP1wtwDwtGfmqJdEblQavVdQZHYUX7Rw9EhYSY0po=; b=rNUQIOSL/OjWd9hjectylss4NmdLvel8m/S3lpaNTX/srYGs8pL/0sF+uzrNs1Aiom QU04xjCd3I6uqPeHMDuTQoHLC7zdgrUI6dUUdvF0qda/FBXEatN45MiHXVefncvGzJfv IgEutHOBFbiYPY89iDuS9BIvDbBRY66YNsT7rmIq+XtL8ea1Z9C2PslvpqfCgsUHWFDc UDdpDzkYSDFXjhf1QNXzmbyBOdw1avbfm9sGnNe31eCePG4eh0v5bUBlv+Y2/OGsveSA FNNfb6uleAQ4VHXbislUmsZswHDUPgv5HL5hZF16iWAHCz2hc7xJuC/90pGjNgp6o10I ALOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=LthrIcTz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y24-20020a634958000000b004790510bfe5si49982606pgk.692.2023.01.23.13.05.13; Mon, 23 Jan 2023 13:05:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=LthrIcTz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232870AbjAWU7C (ORCPT + 99 others); Mon, 23 Jan 2023 15:59:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232710AbjAWU6r (ORCPT ); Mon, 23 Jan 2023 15:58:47 -0500 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB2D616315; Mon, 23 Jan 2023 12:58:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LKrP1wtwDwtGfmqJdEblQavVdQZHYUX7Rw9EhYSY0po=; b=LthrIcTz/vqS1NeYOD0uryIINK L5KA0bgSmcB9KMp+I7aQ5EwxmG5qOTNqwwDMaLS8PmkuBC4HSTBPEsoS2bdKiHPtGPN4OFBPidnr2 5zbVMyZP0Y8odEb60DwuQfxT2Eaq/NUfnc1krSs7fj+svrKOAiOd/c5YRTY+/4PYC1xoidVDbYSli ORaftVjMZwg4jYvWLzMGWTGKhY1mBn3TA0jGQksHIcZZqt2zQz/rCGybV3JcBvrHQ6IxXmwVkth8l d1NqgwluXNaHMVCrlyKf/lsJuQ5S//P74orZosbW8IQc/Q1n2PXtJaHc/YqKiZHWCmUT8RmH7O4/Q FhSIaXXw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pK3sA-001dtm-02; Mon, 23 Jan 2023 20:57:07 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 6025C3002BF; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C4CC620754993; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205515.001913664@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:11 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Uros Bizjak Subject: [PATCH 2/6] x86/pvclock: improve atomic update of last_value in pvclock_clocksource_read References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848825886277691?= X-GMAIL-MSGID: =?utf-8?q?1755848825886277691?= From: Uros Bizjak Improve atomic update of last_value in pvclock_clocksource_read: - Atomic update can be skipped if the "last_value" is already equal to "ret". - The detection of atomic update failure is not correct. The value, returned by atomic64_cmpxchg should be compared to the old value from the location to be updated. If these two are the same, then atomic update succeeded and "last_value" location is updated to "ret" in an atomic way. Otherwise, the atomic update failed and it should be retried with the value from "last_value" - exactly what atomic64_try_cmpxchg does in a correct and more optimal way. Signed-off-by: Uros Bizjak Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20230118202330.3740-1-ubizjak@gmail.com --- arch/x86/kernel/pvclock.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/pvclock.c b/arch/x86/kernel/pvclock.c index eda37df016f0..5a2a517dd61b 100644 --- a/arch/x86/kernel/pvclock.c +++ b/arch/x86/kernel/pvclock.c @@ -102,10 +102,9 @@ u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src) */ last = atomic64_read(&last_value); do { - if (ret < last) + if (ret <= last) return last; - last = atomic64_cmpxchg(&last_value, last, ret); - } while (unlikely(last != ret)); + } while (!atomic64_try_cmpxchg(&last_value, &last, ret)); return ret; } From patchwork Mon Jan 23 20:50:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47400 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1811961wrn; Mon, 23 Jan 2023 12:59:10 -0800 (PST) X-Google-Smtp-Source: AMrXdXv6MLofsrH/hhp94Zxf98z5DgTsS/lHLvHgydQJLa+ZabiX1wcgWfJyNJCi/kzKDqv5B3yA X-Received: by 2002:a17:902:7614:b0:194:bcca:5e71 with SMTP id k20-20020a170902761400b00194bcca5e71mr20696575pll.68.1674507550337; Mon, 23 Jan 2023 12:59:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507550; cv=none; d=google.com; s=arc-20160816; b=g0uGLsn5UFuvDOE7DIO0tE34kORAx2eJsMExDQv15QTZktkHPCvpRd0xk2g6JSopLa S+D1/2HgcLE1cx7Y7wicl4TxXmI8X+SnzhELj/73Izycbkg1lj1+lRoFG4HP2BUEViDe KV08PpLA0a0Ks/EYWuIEkfbQ2/dqX/D+kXuNSNcDarpk89vHlFB0OUKP3IFOuNz4gUNj qulvtCOawcKgoXNt/6Sz/QF5xHWMoVRvBMLqJAVpmyAeVlfA2P6mX0scWzuA8DG6FgOs vz/zkdoO9zn/oA1YhbQryQ9eQoPMbYBr9/PfYttd35iQHTEVCR0jfsZG3v8J1/k1hM/H pSAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=bIfFAXl3Gv4EnKDtIDltBqYZ3txjVL6G+NhMkZyfGI4=; b=vE2rpi0bxCp7bPrnG78BK5+02uOZBjU/aRCJ8r3VkMEjan8WVK34KErmH3ZrxlF+Qx /aW2n4hwNWcwTAJoQRsFFZ1Wz1v9Ny3FroCIFtOorXzfvC6xsb4oAc2FLRR3WXnGYYZp SXYi01K1KehqK7A8h/sz37Y9XzcKLy5iMgCEAq2z4YPIJF8wC/BB+oyh93miTAVPA0pp RNoCd+GS4aome88pmfY+GQw/yl0ZcwsvLQ7YUbScXcUwd53yXucgN7RMUuLM9uz3KgIb KU64VwWFWVOjTmvfRsqQ7QxzV9fRuGL1Ndqar4w1wGcZx5cJMBYMfyh+MTZW4DiJmtBZ wj/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=gnYqMu4P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v15-20020a170902e8cf00b001927e835ffasi370529plg.119.2023.01.23.12.58.57; Mon, 23 Jan 2023 12:59:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=gnYqMu4P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232634AbjAWU6l (ORCPT + 99 others); Mon, 23 Jan 2023 15:58:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232553AbjAWU6i (ORCPT ); Mon, 23 Jan 2023 15:58:38 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FA403802B; Mon, 23 Jan 2023 12:58:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=bIfFAXl3Gv4EnKDtIDltBqYZ3txjVL6G+NhMkZyfGI4=; b=gnYqMu4PNJySVrCjnkQZIgqD33 wQXB/rRboPQU3LRLINoxnUhxHyuyRecqTgiKPD7HJmGt1ujJzhYLpFgeknPGh83xjZgsPFTRxusSP 6FrdKCC3GQlR3CMbnpwqtGVsjcmD/Ej0w7wbfIM3vLWi6cNVejzdRqOExLrFK1rAnGCpRPlxr5+Oc wDq1C+RIa8zZ3MBxAWZBdlLHGxcQGyE/awpvRp1BLNR+dLrA+45FqNKyNNuglm+LzlsIk4VYNAJ/q P97sHNR5ecVeHuLhJ2L/MEZt49xD/FKILOzWA0ABELctEl+d01ONaULWBfayuntvr+R+X1nyG1EMb 5ooM8zaA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pK3se-004WTw-59; Mon, 23 Jan 2023 20:57:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 64BDE300747; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C96002075A6F7; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205515.059999893@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:12 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org Subject: [PATCH 3/6] ftrace/x86: Warn and ignore graph tracing when RCU is disabled References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848429284502923?= X-GMAIL-MSGID: =?utf-8?q?1755848429284502923?= All RCU disabled code should be noinstr and hence we should never get here -- when we do, WARN about it and make sure to not actually do tracing. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Peter Zijlstra (Intel) Acked-by: Paul E. McKenney --- arch/x86/kernel/ftrace.c | 3 +++ 1 file changed, 3 insertions(+) --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -646,6 +646,9 @@ void prepare_ftrace_return(unsigned long if (unlikely(atomic_read(¤t->tracing_graph_pause))) return; + if (WARN_ONCE(!rcu_is_watching(), "RCU not on for: %pS\n", (void *)ip)) + return; + bit = ftrace_test_recursion_trylock(ip, *parent); if (bit < 0) return; From patchwork Mon Jan 23 20:50:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47403 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1812081wrn; Mon, 23 Jan 2023 12:59:29 -0800 (PST) X-Google-Smtp-Source: AMrXdXt+RhMoT8NMpEmZ3fv+vmZdiBTCHEye7FzIrVg6ka/2deB25VA8a/TFy5yaK4gRDFoyb2K+ X-Received: by 2002:a17:90b:3d0c:b0:229:4fc8:54eb with SMTP id pt12-20020a17090b3d0c00b002294fc854ebmr27667041pjb.19.1674507569344; Mon, 23 Jan 2023 12:59:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507569; cv=none; d=google.com; s=arc-20160816; b=c5/BNciyS09XoSpM/H/qCDBPcy+WdRvbAarWxmDi6UozrS3kicwppi5j01sPDtX5+D au+Vj6zyLqxoZqzQD9kDoq4XeGSWZ8LzHae8od7oyf5NJ9SUezNxa4mmiS6leX11MRLM 3vF/tiOAU3WDpqqhOwPhDqk8BYky79gHT+Xcw/9FJ20K67yy4usoTgIs7MFxGPtRymbZ jXa6avoB0qqNOZERS7PLxLH+e+xNgo0oCGhPNZqKjhAM7PsEa5epSWB43AKX5vY9Id+v fIa4l3dRfGueVKnt6zfNMJ2zjj4u3qerkI8GHreHGJG1xWBiSs7qpiJBH4M73t4cipTn YK1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=e32s12OZeZPmUAIBV6e3MlO1ukdEIkvaneutSmlGJOw=; b=wC686cBaz3RsvKazwrsYAdG+TYhwFdRF395wu3zYGM6l8HSZlR+lkBzFGnBiK0uvp2 32SO+Cr4tRQylpOzI0xJrlZR/Di44R/U1YE6cf9hLOlT0/DQH/VrX76WjLVLflfqyFYU FloLsVEsrIRzq6+W0ExXKE7yDZR3boWMRmzpw1tRi/sySeRgH0P7z5PNdDYTEsHkSuL2 UbC39vkz9B2vHJZ+rj3E98X6bWhCOP6kKopPwaLaCBDbZWgoU4F2tTq4ZIcKrQyDBx2w 579e8hURDDtcGj8YCJv0jiS0BPCIK4NRsCI0UBN4Ai3ebR2Z1zPJZ/79yArTpdSAc/Bj McaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=RFvWmzO5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t189-20020a6381c6000000b004d08be9515dsi18370542pgd.124.2023.01.23.12.59.16; Mon, 23 Jan 2023 12:59:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=RFvWmzO5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232800AbjAWU67 (ORCPT + 99 others); Mon, 23 Jan 2023 15:58:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232653AbjAWU6q (ORCPT ); Mon, 23 Jan 2023 15:58:46 -0500 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7F4DEC4D; Mon, 23 Jan 2023 12:58:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=e32s12OZeZPmUAIBV6e3MlO1ukdEIkvaneutSmlGJOw=; b=RFvWmzO5XAXR3iWX6Ms10lg1CS 0NAWAMH9UZ2o+rnHkFkj4Z9it6hcaRo3UJcQDEuQ8cEMtZHAx4SphwAzl/CiQPeZPpkAHYAYM5Qn5 f2ykUTICYqjZenTHscGfzcs1sLXklOSBLL13fXfqm3aZn1PS0RnMRU/CleDqzZthaGd7rJuqwG5P/ oap6vCh2BBTw6kSMN8cAiLaw0oqwvFJIhL2vcflmdZHCDOxK9PmgwHtZxKSVcJGRQNYajNlZi0kjV wdzdT4i2XAL9Q2jjPyEowt4jDubGZfAY2GMTdW1Owp3wDRCzvKRWvCZg6VcWwIFEIlDCobi0afqWR IPJsciBw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pK3s9-001dtl-3C; Mon, 23 Jan 2023 20:57:07 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 5DAEB30008D; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D1D702075A6DE; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205515.118011340@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:13 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org Subject: [PATCH 4/6] x86: Mark sched_clock() noinstr References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848449334731599?= X-GMAIL-MSGID: =?utf-8?q?1755848449334731599?= In order to use sched_clock() from noinstr code, mark it and all it's implenentations noinstr. The whole pvclock thing (used by KVM/Xen) is a bit of a pain, since it calls out to watchdogs, create a pvclock_clocksource_read_nowd() variant doesn't do that and can be noinstr. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/kvmclock.h | 2 +- arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/pvclock.h | 3 ++- arch/x86/kernel/cpu/vmware.c | 2 +- arch/x86/kernel/kvmclock.c | 6 +++--- arch/x86/kernel/pvclock.c | 19 +++++++++++++++---- arch/x86/kernel/tsc.c | 7 +++---- arch/x86/xen/time.c | 12 ++++++++++-- include/linux/math64.h | 4 ++-- 9 files changed, 38 insertions(+), 19 deletions(-) --- a/arch/x86/include/asm/kvmclock.h +++ b/arch/x86/include/asm/kvmclock.h @@ -8,7 +8,7 @@ extern struct clocksource kvm_clock; DECLARE_PER_CPU(struct pvclock_vsyscall_time_info *, hv_clock_per_cpu); -static inline struct pvclock_vcpu_time_info *this_cpu_pvti(void) +static __always_inline struct pvclock_vcpu_time_info *this_cpu_pvti(void) { return &this_cpu_read(hv_clock_per_cpu)->pvti; } --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -26,7 +26,7 @@ DECLARE_STATIC_CALL(pv_sched_clock, dumm void paravirt_set_sched_clock(u64 (*func)(void)); -static inline u64 paravirt_sched_clock(void) +static __always_inline u64 paravirt_sched_clock(void) { return static_call(pv_sched_clock)(); } --- a/arch/x86/include/asm/pvclock.h +++ b/arch/x86/include/asm/pvclock.h @@ -7,6 +7,7 @@ /* some helper functions for xen and kvm pv clock sources */ u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src); +u64 pvclock_clocksource_read_nowd(struct pvclock_vcpu_time_info *src); u8 pvclock_read_flags(struct pvclock_vcpu_time_info *src); void pvclock_set_flags(u8 flags); unsigned long pvclock_tsc_khz(struct pvclock_vcpu_time_info *src); @@ -39,7 +40,7 @@ bool pvclock_read_retry(const struct pvc * Scale a 64-bit delta by scaling and multiplying by a 32-bit fraction, * yielding a 64-bit result. */ -static inline u64 pvclock_scale_delta(u64 delta, u32 mul_frac, int shift) +static __always_inline u64 pvclock_scale_delta(u64 delta, u32 mul_frac, int shift) { u64 product; #ifdef __i386__ --- a/arch/x86/kernel/cpu/vmware.c +++ b/arch/x86/kernel/cpu/vmware.c @@ -143,7 +143,7 @@ static __init int parse_no_stealacc(char } early_param("no-steal-acc", parse_no_stealacc); -static unsigned long long notrace vmware_sched_clock(void) +static noinstr u64 vmware_sched_clock(void) { unsigned long long ns; --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -71,12 +71,12 @@ static int kvm_set_wallclock(const struc return -ENODEV; } -static u64 kvm_clock_read(void) +static noinstr u64 kvm_clock_read(void) { u64 ret; preempt_disable_notrace(); - ret = pvclock_clocksource_read(this_cpu_pvti()); + ret = pvclock_clocksource_read_nowd(this_cpu_pvti()); preempt_enable_notrace(); return ret; } @@ -86,7 +86,7 @@ static u64 kvm_clock_get_cycles(struct c return kvm_clock_read(); } -static u64 kvm_sched_clock_read(void) +static noinstr u64 kvm_sched_clock_read(void) { return kvm_clock_read() - kvm_sched_clock_offset; } --- a/arch/x86/kernel/pvclock.c +++ b/arch/x86/kernel/pvclock.c @@ -64,7 +64,8 @@ u8 pvclock_read_flags(struct pvclock_vcp return flags & valid_flags; } -u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src) +static __always_inline +u64 __pvclock_clocksource_read(struct pvclock_vcpu_time_info *src, bool dowd) { unsigned version; u64 ret; @@ -77,7 +78,7 @@ u64 pvclock_clocksource_read(struct pvcl flags = src->flags; } while (pvclock_read_retry(src, version)); - if (unlikely((flags & PVCLOCK_GUEST_STOPPED) != 0)) { + if (dowd && unlikely((flags & PVCLOCK_GUEST_STOPPED) != 0)) { src->flags &= ~PVCLOCK_GUEST_STOPPED; pvclock_touch_watchdogs(); } @@ -100,15 +101,25 @@ u64 pvclock_clocksource_read(struct pvcl * updating at the same time, and one of them could be slightly behind, * making the assumption that last_value always go forward fail to hold. */ - last = atomic64_read(&last_value); + last = arch_atomic64_read(&last_value); do { if (ret <= last) return last; - } while (!atomic64_try_cmpxchg(&last_value, &last, ret)); + } while (!arch_atomic64_try_cmpxchg(&last_value, &last, ret)); return ret; } +u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src) +{ + return __pvclock_clocksource_read(src, true); +} + +noinstr u64 pvclock_clocksource_read_nowd(struct pvclock_vcpu_time_info *src) +{ + return __pvclock_clocksource_read(src, false); +} + void pvclock_read_wallclock(struct pvclock_wall_clock *wall_clock, struct pvclock_vcpu_time_info *vcpu_time, struct timespec64 *ts) --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -215,7 +215,7 @@ static void __init cyc2ns_init_secondary /* * Scheduler clock - returns current time in nanosec units. */ -u64 native_sched_clock(void) +noinstr u64 native_sched_clock(void) { if (static_branch_likely(&__use_tsc)) { u64 tsc_now = rdtsc(); @@ -248,7 +248,7 @@ u64 native_sched_clock_from_tsc(u64 tsc) /* We need to define a real function for sched_clock, to override the weak default version */ #ifdef CONFIG_PARAVIRT -unsigned long long sched_clock(void) +noinstr u64 sched_clock(void) { return paravirt_sched_clock(); } @@ -258,8 +258,7 @@ bool using_native_sched_clock(void) return static_call_query(pv_sched_clock) == native_sched_clock; } #else -unsigned long long -sched_clock(void) __attribute__((alias("native_sched_clock"))); +u64 sched_clock(void) __attribute__((alias("native_sched_clock"))); bool using_native_sched_clock(void) { return true; } #endif --- a/arch/x86/xen/time.c +++ b/arch/x86/xen/time.c @@ -60,9 +60,17 @@ static u64 xen_clocksource_get_cycles(st return xen_clocksource_read(); } -static u64 xen_sched_clock(void) +static noinstr u64 xen_sched_clock(void) { - return xen_clocksource_read() - xen_sched_clock_offset; + struct pvclock_vcpu_time_info *src; + u64 ret; + + preempt_disable_notrace(); + src = &__this_cpu_read(xen_vcpu)->time; + ret = pvclock_clocksource_read_nowd(src); + ret -= xen_sched_clock_offset; + preempt_enable_notrace(); + return ret; } static void xen_read_wallclock(struct timespec64 *ts) --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -161,7 +161,7 @@ static inline u64 mul_u32_u32(u32 a, u32 #if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__) #ifndef mul_u64_u32_shr -static inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift) +static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift) { return (u64)(((unsigned __int128)a * mul) >> shift); } @@ -177,7 +177,7 @@ static inline u64 mul_u64_u64_shr(u64 a, #else #ifndef mul_u64_u32_shr -static inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift) +static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int shift) { u32 ah, al; u64 ret; From patchwork Mon Jan 23 20:50:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47399 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1811947wrn; Mon, 23 Jan 2023 12:59:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXtMBbJKpuQ0kmhP9odKOVrepxDle9FmwT1w076aaMklNtZrNohTbPeLnvGvJN6quqeZ5ysu X-Received: by 2002:a05:6a21:6d90:b0:b6:9f9f:4ba7 with SMTP id wl16-20020a056a216d9000b000b69f9f4ba7mr32543181pzb.47.1674507547453; Mon, 23 Jan 2023 12:59:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507547; cv=none; d=google.com; s=arc-20160816; b=eCBCJTREBXiw68faBF81YZmt2TR87R2cRgLTYtzzyTW5xsS/3XILNRdejtYG+0r2dI 1sC63X7YdvphTXYTFiJJwWJoSTz1WDIiP4hahImPxp42AbXVVmNTVD1kCaJeIK6JOKAK lASBs0eN13VlPfHK/iu7vVtL8yKzxjxB2IuXTYpmWPKfvt8x+YPzDjr+U4gmtD46S/hl 0BuNdX53TH8iVdW8JXN7uo5GhO/S8BOOcpvj+2c4Zy+FBKpkWBI8MYMghPcMI1jLVeKN 81I78tzEpVlkQ3KABkeRgng8TBkGlrEe6RzZHpemVyzWV2ccV03pMubk0puy4N3V4Q2c RjpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=YEKkyvTZ05I2hq93YUH2assl5KDOT02g9Cn+Y86W8Rg=; b=X/AzBHwFJSin/raQcsQYyel2t6GqKnfsX/OeJ6QlSA+MrktmK7gr+VB6F2fQo/k2n8 +CjrGOJvOu4R32SwRAauCVWz3zLFpmWn0M51t7i27o2NN558X+UhOreP5GQKPprEMCDL otzgESlb7wiQlid8K5FVPSDVHePpMfr0Li0hudgTp1uKFaixMKA10Zy1K6HySlpwqKJK O+svlTsiWtAfXL/n+o0VAqnPXcnLdl1d4L2gg6liC5b5PakeMIYIjqJh8GGtuBHb3tQT /hXnm139sx+SkViFlIVZkuhRnoOkvq5YTT4lNT1yLU57sfhs+6nvDt1ZQ+4I4NZiKK/n tyEw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=mnTOOg0X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 31-20020a63125f000000b0045f7fc4b812si36917147pgs.295.2023.01.23.12.58.53; Mon, 23 Jan 2023 12:59:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=mnTOOg0X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232517AbjAWU6g (ORCPT + 99 others); Mon, 23 Jan 2023 15:58:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232315AbjAWU6e (ORCPT ); Mon, 23 Jan 2023 15:58:34 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0CD98A41; Mon, 23 Jan 2023 12:58:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=YEKkyvTZ05I2hq93YUH2assl5KDOT02g9Cn+Y86W8Rg=; b=mnTOOg0XrV4WQaWs8SDkyTd870 AOXy0F8U69ylAPKd0Bj22aigz2fTag/FIRvErdlguRoxjrnM0OypFDE6S1vWQrjBZGeWY4Ll4eifA 2TYoxsVgfCELCo2LAmT6StQd77sf54f/X3g9dtv6SlV7B+6SfiH8vo6Afz0D4LyN041bW5oVZQEmL wJTKOfFCTUiDtudmB5IBHXNLseVMvpkZfx2j3aqj6nukdEvfME1cMgkAvHeKpa2Gl4sVEk2GzwFUI SsquzAD1nbcZnlRrjSBi8t8ssbNVb3NCiWW+m9k8hxMwJDW9OgardpBYukVrHGMgSneTYsH144L+e a2YqqZTQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pK3se-004WU2-SZ; Mon, 23 Jan 2023 20:57:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 6AB95300C22; Mon, 23 Jan 2023 21:57:27 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D40E02075A6FB; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205515.175774650@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:14 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org Subject: [PATCH 5/6] sched/clock: Make local_clock() noinstr References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848426002059804?= X-GMAIL-MSGID: =?utf-8?q?1755848426002059804?= With sched_clock() noinstr, provide a noinstr implementation of local_clock(). Signed-off-by: Peter Zijlstra (Intel) --- include/linux/sched/clock.h | 8 +++----- kernel/sched/clock.c | 27 +++++++++++++++++++++------ 2 files changed, 24 insertions(+), 11 deletions(-) --- a/include/linux/sched/clock.h +++ b/include/linux/sched/clock.h @@ -45,7 +45,7 @@ static inline u64 cpu_clock(int cpu) return sched_clock(); } -static inline u64 local_clock(void) +static __always_inline u64 local_clock(void) { return sched_clock(); } @@ -79,10 +79,8 @@ static inline u64 cpu_clock(int cpu) return sched_clock_cpu(cpu); } -static inline u64 local_clock(void) -{ - return sched_clock_cpu(raw_smp_processor_id()); -} +extern u64 local_clock(void); + #endif #ifdef CONFIG_IRQ_TIME_ACCOUNTING --- a/kernel/sched/clock.c +++ b/kernel/sched/clock.c @@ -93,7 +93,7 @@ struct sched_clock_data { static DEFINE_PER_CPU_SHARED_ALIGNED(struct sched_clock_data, sched_clock_data); -notrace static inline struct sched_clock_data *this_scd(void) +static __always_inline struct sched_clock_data *this_scd(void) { return this_cpu_ptr(&sched_clock_data); } @@ -244,12 +244,12 @@ late_initcall(sched_clock_init_late); * min, max except they take wrapping into account */ -notrace static inline u64 wrap_min(u64 x, u64 y) +static __always_inline u64 wrap_min(u64 x, u64 y) { return (s64)(x - y) < 0 ? x : y; } -notrace static inline u64 wrap_max(u64 x, u64 y) +static __always_inline u64 wrap_max(u64 x, u64 y) { return (s64)(x - y) > 0 ? x : y; } @@ -260,7 +260,7 @@ notrace static inline u64 wrap_max(u64 x * - filter out backward motion * - use the GTOD tick value to create a window to filter crazy TSC values */ -notrace static u64 sched_clock_local(struct sched_clock_data *scd) +static __always_inline u64 sched_clock_local(struct sched_clock_data *scd) { u64 now, clock, old_clock, min_clock, max_clock, gtod; s64 delta; @@ -287,13 +287,28 @@ notrace static u64 sched_clock_local(str clock = wrap_max(clock, min_clock); clock = wrap_min(clock, max_clock); - if (!try_cmpxchg64(&scd->clock, &old_clock, clock)) + if (!arch_try_cmpxchg64(&scd->clock, &old_clock, clock)) goto again; return clock; } -notrace static u64 sched_clock_remote(struct sched_clock_data *scd) +noinstr u64 local_clock(void) +{ + u64 clock; + + if (static_branch_likely(&__sched_clock_stable)) + return sched_clock() + __sched_clock_offset; + + preempt_disable_notrace(); + clock = sched_clock_local(this_scd()); + preempt_enable_notrace(); + + return clock; +} +EXPORT_SYMBOL_GPL(local_clock); + +static notrace u64 sched_clock_remote(struct sched_clock_data *scd) { struct sched_clock_data *my_scd = this_scd(); u64 this_clock, remote_clock; From patchwork Mon Jan 23 20:50:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 47404 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1813646wrn; Mon, 23 Jan 2023 13:02:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXt/73OgC7a/KUz24OlT3+5ksvApmWvCbtzd+hA8jjdpw3icWGuprXfOFwytLCkouECc2KiT X-Received: by 2002:a17:902:8492:b0:18f:438a:cfe1 with SMTP id c18-20020a170902849200b0018f438acfe1mr24347210plo.59.1674507748287; Mon, 23 Jan 2023 13:02:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674507748; cv=none; d=google.com; s=arc-20160816; b=KzYf0SwsNUd7VEgP1xkE2eutcVUXOa/pc1FyxiDSREwYtfUSX73ID/bN6lyEXoBZTj R8qNrPdzN1uxL5uJHUDRo9GiEFQquHOK9fBu4tP0pniuaeJCdA317O3yJOO9nlE0uyiO PwyjZqKb7sL3iYnm3irKpzH/rtw/WJZ6TTgmDccfDbriR8OlwJKSHdjKCBoxE7sYvDGN MslJe/NBO4uo2hIIYX6wgW2+qI3tO7NIUs57O+EKrrAxKdea2HkzJ6yrE0DpKCWmXaQC i2TSQdQLOaa04ifyTvS8PgPaW0NDBiA2PHU82cbytzHREJ7Xe1KFs2Nuqym5Tg0QliEn Jm3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=DwNtSkSaGTU0TFqzb2/WMPSb8JYwGKCIYZsmeR/P6pc=; b=C4BFGvKUaJX16ewhc9iGV7kl+W7U32uPy14+ifH7cWwNqgZ4z5trlEPGDTGcE0nsU+ kb6STWzrolgwE2T25ty3MFKqQmp9aL0j12xoyOTAVRQi/C/pbi+mKXm4Ev5eNxLAcQpu utiuKrjza+HrSkuwNbV8zJB1HJPJdCXJNWn0sd0f3Qbh9KK9C2M/FfysspfbHJdzWkuF pHk6PwqgC///WNfmLpDxtZrO1Op4W5Gv6xvnBi748t/Q4i/tb8fRuJjaxOB394MNVYh8 gzmKkI+CuR74I/TyAbt7Rb5s2CdIHSOyFlheJ7jP6LU0lnPlgUOjFcy/IUOIdZXwYSun yZOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=FueaaZb7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 17-20020a621911000000b00586cc94faafsi71894pfz.227.2023.01.23.13.01.44; Mon, 23 Jan 2023 13:02:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=FueaaZb7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232694AbjAWU7F (ORCPT + 99 others); Mon, 23 Jan 2023 15:59:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232660AbjAWU6r (ORCPT ); Mon, 23 Jan 2023 15:58:47 -0500 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43A2834C29; Mon, 23 Jan 2023 12:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DwNtSkSaGTU0TFqzb2/WMPSb8JYwGKCIYZsmeR/P6pc=; b=FueaaZb7VWO4atG18KkGUgcSzQ FI5cJ88LeuzKk95B4T/ujA5RntI3Q5IOg661T+Nx6Jg1ofr+WaQ3jBAlstIFsm10tCM3ugdjreIdw jekc2aQYG/Zbaa+emGpeAz4Lv6QTZxYV0c5nx4gqJLGh7FKawVx+Ru+OgUlQzQ74ginyYEH3WP7dw 5txleE63/CC3gyBWdpXXfg+uTfS6QliMubretWIhRomQhaeRWZ4/+Y6MJMCwfLlvq3DfaoQCRPIFH +KxmdhhfRE4vj/Ol3tLiIZOlGF2SQI84vVc0Rt8I4gS0JE1bOVgjV0eJCIaInsyf0I1LAQCcZ89zG l5gK1P0g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pK3sB-001dto-1f; Mon, 23 Jan 2023 20:57:02 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 6AB39300BCB; Mon, 23 Jan 2023 21:57:27 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D952B2075A6F8; Mon, 23 Jan 2023 21:57:24 +0100 (CET) Message-ID: <20230123205515.233366796@infradead.org> User-Agent: quilt/0.66 Date: Mon, 23 Jan 2023 21:50:15 +0100 From: Peter Zijlstra To: mingo@kernel.org Cc: will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, rostedt@goodmis.org, mhiramat@kernel.org, wanpengli@tencent.com, vkuznets@redhat.com, boris.ostrovsky@oracle.com, rafael@kernel.org, daniel.lezcano@linaro.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-trace-kernel@vger.kernel.org, linux-pm@vger.kernel.org, kernel test robot Subject: [PATCH 6/6] cpuidle: Fix poll_idle() noinstr annotation References: <20230123205009.790550642@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755848636551849598?= X-GMAIL-MSGID: =?utf-8?q?1755848636551849598?= The instrumentation_begin()/end() annotations in poll_idle() were complete nonsense. Specifically they caused tracing to happen in the middle of noinstr code, resulting in RCU splats. Now that local_clock() is noinstr, mark up the rest and let it rip. Fixes: 00717eb8c955 ("cpuidle: Annotate poll_idle()") Signed-off-by: Peter Zijlstra (Intel) Reported-by: kernel test robot Link: https://lore.kernel.org/oe-lkp/202301192148.58ece903-oliver.sang@intel.com Acked-by: Rafael J. Wysocki --- drivers/cpuidle/cpuidle.c | 2 +- drivers/cpuidle/poll_state.c | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -426,7 +426,7 @@ void cpuidle_reflect(struct cpuidle_devi * @dev: the cpuidle device * */ -u64 cpuidle_poll_time(struct cpuidle_driver *drv, +__cpuidle u64 cpuidle_poll_time(struct cpuidle_driver *drv, struct cpuidle_device *dev) { int i; --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -15,7 +15,6 @@ static int __cpuidle poll_idle(struct cp { u64 time_start; - instrumentation_begin(); time_start = local_clock(); dev->poll_time_limit = false; @@ -42,7 +41,6 @@ static int __cpuidle poll_idle(struct cp raw_local_irq_disable(); current_clr_polling(); - instrumentation_end(); return index; }