From patchwork Mon May 22 10:27:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 97245 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1345668vqo; Mon, 22 May 2023 03:35:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5b7pwfS/M002lzBJ6OvET/1OBN1EnACwU1i/g4qrk9o0WEFYZGImIU8PuJjdLBLIR4durx X-Received: by 2002:a17:902:c944:b0:19f:3d59:e0ac with SMTP id i4-20020a170902c94400b0019f3d59e0acmr14175821pla.44.1684751724427; Mon, 22 May 2023 03:35:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684751724; cv=none; d=google.com; s=arc-20160816; b=Pb7G/Xxhw0uD1Guz6H5Yl7XruzbDlIR6HlaQcTvVCJb/XQ+t1bOMR2KHpBbEwGBOaB 6xAYMkAvthrcxAdcdULpFfbyRqzhJLfsMcQ2xwlAOHSC/Ov/e+AuwLaEbaBcoNn6SeST cb1oi+Y0UbivxeXmUJris5A3FzFomsLiZGm7hJJBPN3XQi89/MM7cUdXm/pIkjOJT6lx t2UkfHRv8EyskEa50m2A6dwBCWkAIpYb82evzUqfCmdUHrNn9ACWyXBkuuPK8PT6wFxD W8lgwD6IY4RFTSbJtSS5CwUptHiI6Zjiw82hKnSLcJowdpYm0Kd05OUZlplRgycnZysw HeFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=2jg+PEFLPaxod2aGd5ToSr/xf/b+owxH3fTbsNQbh18=; b=geXLgz2UlRVZv3f+V9kLZEpnGuPs/5XWrdsIyyMRcPjcEVS/AHGX5aKC3LVVERwIMS FxNl/R+IffM6zLSAr2tzZAQlpIJ5a7CNUq3XZoMKCUAHnfJ4GizvnuGVaW5QVWdEU6pH MrogjYB88adglz0IzVki1VkI303z8/Zqp6K3jXDek89cASd6OkD/1EsJRv6NWIOl0uXM x+EzVyt2YjsR4KBj9hjo9Uw63zWuu4XYPdwgUKOUzFO3uVBjl+uYHIsLuSUWDz1k4vIY RU8orZmpjoh16avmgcD0kk8+U2kH9Sc6QJAoDaBXVgkXopjtmL1lryXWl4TdusW+cCsU T25Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qK0wMMQ0; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 21-20020a170902c11500b001aaed82c2afsi2889647pli.171.2023.05.22.03.35.11; Mon, 22 May 2023 03:35:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qK0wMMQ0; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232924AbjEVK2P (ORCPT + 99 others); Mon, 22 May 2023 06:28:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232018AbjEVK1c (ORCPT ); Mon, 22 May 2023 06:27:32 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3FBF1BE; Mon, 22 May 2023 03:27:29 -0700 (PDT) Date: Mon, 22 May 2023 10:27:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684751248; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2jg+PEFLPaxod2aGd5ToSr/xf/b+owxH3fTbsNQbh18=; b=qK0wMMQ0j1Pnt7fJeMfr7OQoe6mN+2BKXjmhzczLoIZuF8Kx3gvpquLWyKsIwmxm2jTtSN zSpzuO+F5n37iRvoTTvClUpXcFxKkPa8y4nnVkqtTE9zMbd34uM8Gl/DfZP/tQU8stcSUp LalozWSX46gXfG6a/fxKNyYeGu0cOeHxPhSLqdIqMq5xkPcdTJqbIM90ae4sK9M7oafQOt hHBjZY7w2c7XwSfYp+Zlq93swuyVwsoVNZkT2ow5ElJSDjYJAzVD2EaikIVY6eq9abxnuj LQ6c+hMIDBanlB1brBjJLMXL3nhcbSSLDAvQXD+Svv1ddTa64Xbhm5iFrslE4A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684751248; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2jg+PEFLPaxod2aGd5ToSr/xf/b+owxH3fTbsNQbh18=; b=O8ZrWpXD4EUcUE7Ezo1zpT4ol6uhWJuM7nRoASqX7c16HKDjwXkprSDuWY6In8QipMVQrO 8PeAmP4gaCbxhzDQ== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] percpu: Wire up cmpxchg128 Cc: "Peter Zijlstra (Intel)" , Arnd Bergmann , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230515080554.248739380@infradead.org> References: <20230515080554.248739380@infradead.org> MIME-Version: 1.0 Message-ID: <168475124752.404.11683694870847476596.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765946787739398566?= X-GMAIL-MSGID: =?utf-8?q?1766590224056734455?= The following commit has been merged into the locking/core branch of tip: Commit-ID: fed3b1fae1030f87828ff000c69077d660f3bdf3 Gitweb: https://git.kernel.org/tip/fed3b1fae1030f87828ff000c69077d660f3bdf3 Author: Peter Zijlstra AuthorDate: Mon, 15 May 2023 09:57:04 +02:00 Committer: Peter Zijlstra CommitterDate: Mon, 22 May 2023 10:49:49 +02:00 percpu: Wire up cmpxchg128 In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Arnd Bergmann Link: https://lore.kernel.org/r/20230515080554.248739380@infradead.org --- arch/arm64/include/asm/percpu.h | 20 +++++++++++- arch/s390/include/asm/percpu.h | 16 +++++++++- arch/x86/include/asm/percpu.h | 59 ++++++++++++++++++++++++++++++++- include/asm-generic/percpu.h | 16 +++++++++- 4 files changed, 111 insertions(+) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index b9ba19d..2549829 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,22 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = o; \ + new__ = n; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h index 081837b..5603402 100644 --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,22 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, oval, nval) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = oval; \ + new__ = nval; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 13c0d63..4d58b97 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,65 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u64 var; \ + struct { \ + u32 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval); +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval); + +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u128 var; \ + struct { \ + u64 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 6432a7f..53d933e 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,14 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef raw_cpu_cmpxchg128 +#define raw_cpu_cmpxchg128(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +431,14 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg64 +#define this_cpu_cmpxchg64(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg128 +#define this_cpu_cmpxchg128(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \