From patchwork Tue Dec 27 12:13:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 36863 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1361211wrt; Tue, 27 Dec 2022 04:16:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXuY/XpgxZXl6lIpGYAlGcrYU5OqglgLfZuOpJDhHWMGHkdDK4zoFxyPCeBt5F7D/cX7AKSv X-Received: by 2002:a62:6546:0:b0:578:119c:1c52 with SMTP id z67-20020a626546000000b00578119c1c52mr19019116pfb.14.1672143399700; Tue, 27 Dec 2022 04:16:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672143399; cv=none; d=google.com; s=arc-20160816; b=SPxzKeUHkgVEGhohpdUemaAbDo/GWgu1mcprDLUlwkzYetdYSbgb3cidkpoBOnf5qm Rrnr7N0bCB+FrlrBNng8y049OgKDsjAMmi1lTI/VVOx6JAMmH7hekKZuc6WJ3sgDvJ+d sQOxLlwgibpsK4D9J63vtXm/IKATPd15MBfAPr7oTQ7JWklT5lyjjL9zL9b+VcwTYxl/ oT2bh8fxWY5HcCTO0algBgvnYNQslq1g7xLTdIVpf4PYImfy2MWVr7U1pIjHX0YXRuT0 pFi0pyYNPnM8Yz+IHT15Dy/Hx+mbZ89LFIH6PqxEe5AJfNcnQKfwrZJBbWlQGON5NMY+ Xadw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=v1Kv5oWs+xk10ctfCLzH8cy3ViBPTCIu4jALWfPiA5A=; b=VjjjNLpYftzPdkUOK7WVE6bMTTCSxew3Rfr3zLBoyg9scXWyGYqGR7qsmRBgaMRVJ7 iElsQQmEHE/if3i5CRgVmAZk1Hws8qnRTNl4SBGgJHJcsiT2TveddUfOlD3JzcqhLRyN mu6FYbL+RG29eTFhRHp1wuTaOF7afSrMlP3ll8MgY9kNHqIpiCzU7N8eRE4tbvSAJMkk SE+pWg0n/RhilEDKkoynx607tHq/hRoGcacGl4Lxb9CpcmLV+M/0pm7KnAIEh0EhFn8F TIdtTh5ybYTQpEcqYwezfAlaAw8Str7tRw2lk5dGRTm8kki7LEc2IU6TOoa17J2QQLcO uxIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KaWTS3zs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=wdgX61qZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h3-20020a056a001a4300b005631aaf81f2si13897107pfv.183.2022.12.27.04.16.27; Tue, 27 Dec 2022 04:16:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KaWTS3zs; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=wdgX61qZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232355AbiL0MPd (ORCPT + 99 others); Tue, 27 Dec 2022 07:15:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232116AbiL0MOl (ORCPT ); Tue, 27 Dec 2022 07:14:41 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ADC88E6E; Tue, 27 Dec 2022 04:13:52 -0800 (PST) Date: Tue, 27 Dec 2022 12:13:49 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1672143230; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v1Kv5oWs+xk10ctfCLzH8cy3ViBPTCIu4jALWfPiA5A=; b=KaWTS3zsEaVpVQrc2bQaP943Iu4g3OEnZhPxsUPwcAK2Fxy4HTy/2ZDSYAZ3QFATjQuog4 K2GE5voXC1GsJbDc6WLt1Ctgkc960gL41BfD0fQJuY4j4EZjPW0mIMuC8sImk7DW+4HVfj Xg1HmRzpR42wG8xhVv9R+YGP0BHAOKrEStbpNHwYxRwPtzhECQCj1VTXgzN3cPRqHMoq2C KBDu8llqLlQynveFN+UKtafzewP+xZm3AEl/fkTuczk6yCUeTA19VgxX+PaGwdsrOHGFNV nwv2ohaz+P2P9vlMeBg+EnMKB6LK3NuAYY13q/fPzJ5LOmG7+YQ8R7pTpVdMGg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1672143230; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v1Kv5oWs+xk10ctfCLzH8cy3ViBPTCIu4jALWfPiA5A=; b=wdgX61qZn6DbrbATqp2urTj5ufmKxdQd16seXZFWwUPKQEvrp9yRmIOg/MWKD/35D0juN9 b8Uuh7lunB/T4JDA== From: "tip-bot2 for Mathieu Desnoyers" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] selftests/rseq: ppc: Template memory ordering and percpu access mode Cc: Mathieu Desnoyers , "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20221122203932.231377-16-mathieu.desnoyers@efficios.com> References: <20221122203932.231377-16-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 Message-ID: <167214322996.4906.6628139614168626051.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750230411446437492?= X-GMAIL-MSGID: =?utf-8?q?1753369437445559194?= The following commit has been merged into the sched/core branch of tip: Commit-ID: e61bd94c2bbd6b258fb408092172e37d39b4bf5c Gitweb: https://git.kernel.org/tip/e61bd94c2bbd6b258fb408092172e37d39b4bf5c Author: Mathieu Desnoyers AuthorDate: Tue, 22 Nov 2022 15:39:17 -05:00 Committer: Peter Zijlstra CommitterDate: Tue, 27 Dec 2022 12:52:14 +01:00 selftests/rseq: ppc: Template memory ordering and percpu access mode Introduce a rseq-ppc-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-mm-cid per-cpu data access. Signed-off-by: Mathieu Desnoyers Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20221122203932.231377-16-mathieu.desnoyers@efficios.com --- tools/testing/selftests/rseq/rseq-ppc-bits.h | 454 +++++++++++++- tools/testing/selftests/rseq/rseq-ppc.h | 611 +------------------ 2 files changed, 486 insertions(+), 579 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-ppc-bits.h diff --git a/tools/testing/selftests/rseq/rseq-ppc-bits.h b/tools/testing/selftests/rseq/rseq-ppc-bits.h new file mode 100644 index 0000000..98e69ea --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-ppc-bits.h @@ -0,0 +1,454 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-ppc-bits.h + * + * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2018 - Boqun Feng + */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intptr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v not equal to @expectnot */ + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v not equal to @expectnot */ + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) +#endif + /* load the value of @v */ + RSEQ_ASM_OP_R_LOAD(v) + /* store it in @load */ + RSEQ_ASM_OP_R_STORE(load) + /* dereference voffp(v) */ + RSEQ_ASM_OP_R_LOADX(voffp) + /* final store the value at voffp(v) */ + RSEQ_ASM_OP_R_FINAL_STORE(v, 2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "b" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + /* load the value of @v */ + RSEQ_ASM_OP_R_LOAD(v) + /* add @count to it */ + RSEQ_ASM_OP_R_ADD(count) + /* final store */ + RSEQ_ASM_OP_R_FINAL_STORE(v, 2) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) + /* cmp @v2 equal to @expct2 */ + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) + /* cmp @v2 equal to @expct2 */ + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* try store */ + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + /* for 'release' */ + "lwsync\n\t" +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* setup for mempcy */ + "mr %%r19, %[len]\n\t" + "mr %%r20, %[src]\n\t" + "mr %%r21, %[dst]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* try memcpy */ + RSEQ_ASM_OP_R_MEMCPY() + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + /* for 'release' */ + "lwsync\n\t" +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + /* teardown */ + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->RSEQ_TEMPLATE_CPU_ID_FIELD), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17", "r18", "r19", "r20", "r21" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_MM_CID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-ppc.h b/tools/testing/selftests/rseq/rseq-ppc.h index f82d95c..dc9190f 100644 --- a/tools/testing/selftests/rseq/rseq-ppc.h +++ b/tools/testing/selftests/rseq/rseq-ppc.h @@ -2,7 +2,7 @@ /* * rseq-ppc.h * - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers * (C) Copyright 2016-2018 - Boqun Feng */ @@ -205,581 +205,34 @@ do { \ RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ __rseq_str(post_commit_label) ":\n\t" -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v not equal to @expectnot */ - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v not equal to @expectnot */ - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) -#endif - /* load the value of @v */ - RSEQ_ASM_OP_R_LOAD(v) - /* store it in @load */ - RSEQ_ASM_OP_R_STORE(load) - /* dereference voffp(v) */ - RSEQ_ASM_OP_R_LOADX(voffp) - /* final store the value at voffp(v) */ - RSEQ_ASM_OP_R_FINAL_STORE(v, 2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "b" (voffp), - [load] "m" (*load) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - /* load the value of @v */ - RSEQ_ASM_OP_R_LOAD(v) - /* add @count to it */ - RSEQ_ASM_OP_R_ADD(count) - /* final store */ - RSEQ_ASM_OP_R_FINAL_STORE(v, 2) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [count] "r" (count) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try store */ - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try store */ - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - /* for 'release' */ - "lwsync\n\t" - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) - /* cmp @v2 equal to @expct2 */ - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) - /* cmp @v2 equal to @expct2 */ - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) -#endif - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* setup for mempcy */ - "mr %%r19, %[len]\n\t" - "mr %%r20, %[src]\n\t" - "mr %%r21, %[dst]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try memcpy */ - RSEQ_ASM_OP_R_MEMCPY() - RSEQ_INJECT_ASM(5) - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - /* teardown */ - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17", "r18", "r19", "r20", "r21" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* setup for mempcy */ - "mr %%r19, %[len]\n\t" - "mr %%r20, %[src]\n\t" - "mr %%r21, %[dst]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try memcpy */ - RSEQ_ASM_OP_R_MEMCPY() - RSEQ_INJECT_ASM(5) - /* for 'release' */ - "lwsync\n\t" - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - /* teardown */ - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17", "r18", "r19", "r20", "r21" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-cpu-id indexing. */ + +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID + +/* Per-mm-cid indexing. */ + +#define RSEQ_TEMPLATE_MM_CID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_MM_CID + +/* APIs which are not based on cpu ids. */ + +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE