From patchwork Thu Feb 23 15:11:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 61013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp364992wrd; Thu, 23 Feb 2023 07:12:42 -0800 (PST) X-Google-Smtp-Source: AK7set/wjyn9xKnEN5jwEIMYmeOa7BUOkpVcy/2V78PmyV8cdllfJwnuUOy/rADwI6vSuUxFHqpk X-Received: by 2002:a05:6402:34d5:b0:4ae:eae1:2be0 with SMTP id w21-20020a05640234d500b004aeeae12be0mr12242254edc.14.1677165162733; Thu, 23 Feb 2023 07:12:42 -0800 (PST) Received: from sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id j9-20020aa7ca49000000b004ad033944a6si2068128edt.606.2023.02.23.07.12.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 07:12:42 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=RFKLIK+V; arc=fail (signature failed); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B03EA385802F for ; Thu, 23 Feb 2023 15:12:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B03EA385802F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1677165161; bh=37cNnNgjUJmZ4mIwaE/BTu779JCAMfqDI8giFTnEKwo=; h=To:CC:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=RFKLIK+VbMLHkVBSzILCay87fqVn4ffYjLJX9WT2UwtKQEEYKBK+bZlkPk4Fa/1JX +BnVa/OWKAHioHpKOEu3Tg/KL73+3quh8Wc6yk/zdo+4bkU+d5IeHVgLJDiZ0l3dQX fTKsQ+kqXW3IopgVHjPGUPA6CdRie/1bo6TRovxc= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on2085.outbound.protection.outlook.com [40.107.7.85]) by sourceware.org (Postfix) with ESMTPS id 0867B3858C5E for ; Thu, 23 Feb 2023 15:11:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0867B3858C5E Received: from AM6P195CA0040.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:87::17) by GV2PR08MB9206.eurprd08.prod.outlook.com (2603:10a6:150:d5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.19; Thu, 23 Feb 2023 15:11:49 +0000 Received: from AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:87:cafe::8) by AM6P195CA0040.outlook.office365.com (2603:10a6:209:87::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21 via Frontend Transport; Thu, 23 Feb 2023 15:11:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT042.mail.protection.outlook.com (100.127.140.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21 via Frontend Transport; Thu, 23 Feb 2023 15:11:49 +0000 Received: ("Tessian outbound 333ca28169fa:v132"); Thu, 23 Feb 2023 15:11:49 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 961ca1fe8b68b062 X-CR-MTA-TID: 64aa7808 Received: from 3db0d2c4c236.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id FA4547F2-0196-4FCA-92AE-CFE3E9747D94.1; Thu, 23 Feb 2023 15:11:42 +0000 Received: from EUR02-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 3db0d2c4c236.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 23 Feb 2023 15:11:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AN0m0TLHPVJ5fV6d8nFlxLFRLrNsBkPlHbIwWYKr5mBgkzJKAhpr64MUbP/HYQO7euGRU6NgE06fmktHcji7T+f43i9L6ZS9O8uHKQnzfN0/0M4y4G2zREAEGJWNZXG6tYq0QWRhB3Gw9SRPBO2t/bGDWtojkop1ppS8mqje2qXZXGbFyD9q/ByFJ5fjxllvOpBPkOreaAsOj5J2RVi6DY3rkUKsQsc+12Ahu4LxeTe48V2WF9eX7rM9omfvXzKEzVRVkHelUXydTvQYcMF/hZ1NNCxaGqHrHiDQIaZYwIFBFjrLFDeJg61DrLRUtugBSlCLHVYoQ+3UAA6QboqUkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=37cNnNgjUJmZ4mIwaE/BTu779JCAMfqDI8giFTnEKwo=; b=IN/H2Q9OdMHMON55JEIpFpt4G/SGjn7nykBmf9v+hA5YjpdLt0G2jikt7URbJVAht3WvE+60dFCeXOp/EwvbWXr1dnaqbLsLcRPR3P0aLQrW1qXViKEYDVx7EqWV5m44M7lG90NC+38819fQdL1DB15B3FKylkMBfAuUdX63cKl8I89GN2Af0k6tFdOUYNozFUEeHFipIMWEuPP7V4Z7qeAo4yuJTq5DvwkBGLG/bqk8KB6MWjhVLEhoXmZmEGiYHhJnAo8AjrRp6YDOB3sURcKJXXPpe4NnzH4lNSQpYRcL2555Ye/Km3hRsvBVwxB02lSowfSlsRhPEPuL9dQ48w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Received: from PAWPR08MB8982.eurprd08.prod.outlook.com (2603:10a6:102:33f::20) by PAWPR08MB9030.eurprd08.prod.outlook.com (2603:10a6:102:342::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.18; Thu, 23 Feb 2023 15:11:40 +0000 Received: from PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::dc17:8fa2:cce5:3573]) by PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::dc17:8fa2:cce5:3573%8]) with mapi id 15.20.6134.021; Thu, 23 Feb 2023 15:11:40 +0000 To: GCC Patches CC: Richard Sandiford , Kyrylo Tkachov Subject: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Topic: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Index: AQHZR5de3lyELmNywka8opZDiKy6FQ== Date: Thu, 23 Feb 2023 15:11:39 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAWPR08MB8982:EE_|PAWPR08MB9030:EE_|AM7EUR03FT042:EE_|GV2PR08MB9206:EE_ X-MS-Office365-Filtering-Correlation-Id: e9780355-8682-4fae-77f0-08db15b04972 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: zfC2SuR1o//vKpOsW+k8S0MH2AiQsSTh/fegFIk+nChExNmLmAH47amR1eYOV/sN9z7nxdUtMHsiPHsz6ZGaajKxQJbh9QidWbKvqFNQjoZEkcqD34V1uESLZLSMtpaTBuGIK0GUTTPiBI44wBer6qxwmfDrzufk7fdxFJpACb8dKqwSu2PuYcL2wegNhsm5lrdq6+tO9KWlY1fppoHRWZcQmtqw93WO0FfTTSAp0CbK7RWIgJGLZzCBLg590W9H6OHybMuLEjALWhJmBQq/SNOC8AqGUamyzpU+Odoex0VcBT6VwTQ8n15mDbD3iMQVjD3m/v7oTnpreAFBzJ1qpDM9aYA0jVfC55JOfirMBwdBKkf5mm8aT1XFrbaeCBr6qbQUECTzeLCxqkEhzv99rDZzGWR4rwfMmonEb2kdt42npZQ/YHtKKaZ3t+j7MgH1dBET1jFhmwG4Ig75XqqaJFS8twaZA9U3lum7vXiC/cLj33aKYdJCnWs4WTMBAynfnXTBNXz3qUfopc9tKzu6Skrvp4UC0nLbchOuth5NSA3IepHsqlVFwziLd/GgUT5dJjNHuRXCPgOVCruCuYLNkGVDecB83jhmkOWSDz4YnVgFt77BetbssUbnoF/DBDjcWgPzo0F2/icEtEHgmKw5p9z5eXdqEePvvlxUMBkgmogzb7mdpK5owGTI/BH3O7Tq9C0SJlgClpBbEylcy3qPJA== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAWPR08MB8982.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(346002)(376002)(39860400002)(396003)(136003)(366004)(451199018)(26005)(76116006)(41300700001)(66476007)(55016003)(7696005)(9686003)(6506007)(4326008)(66556008)(66446008)(316002)(86362001)(33656002)(52536014)(186003)(8676002)(5660300002)(66946007)(83380400001)(64756008)(91956017)(122000001)(478600001)(54906003)(6916009)(8936002)(71200400001)(38100700002)(38070700005)(2906002); DIR:OUT; SFP:1101; MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB9030 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: e12d8d18-1207-4e4f-c451-08db15b04399 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RJb9//cz6m9KLXG1eYS9vX2Lv8U4mgR8ZgHsLLqvZ+EZf4NUOfZOknmdp0dFb4EWaepcXWEP6Vq2cb5NTBWExV8xqNXyWwQuH6MpZPwM5HrL73G+fUBoOPgrWgIReqpvtQOdisft7FB4kibO4zErN9u9dHuxLs00ew611G4lwfJQ3q14HmxWiaIdoz2DUMzN4qhmg/9KybB6cv2k8+S71xBap2zRfkir2BbRuFJyYNy4gBIGpYn4YMHFps2GlqSfRRzafFDLSpZ+WC+RXICDrhfFnL9qquVt4bZktkUmRC40byiYyG/S9G1MEgHT/zZQIAfq8FW6SaeGoFlsg0YUPeVh6F0eVXNFMFoBe6sEylHa4ClFFEfLlkr0r6yNxJKDgFKi+YM50hF0TZqmDBYf5qou9+GE66Ukoi4SR1sEmNUiT9nrjV7lk/gH3GL/eZkD6Jh+Vp3Y4s+WE8bXQSdTqHtaXlGq+fE92sez+V7/zGJTqwuF2SdKOCSSLdcEbKasFyNEOtNZht1U5jvYBnmhSTufDLyrUXhh82h5AjzzCmxpAsJ4J698NEGL3tcvUcEHgXjDiwGKRIMko4bIcEb809ewQD2vfSyKDAyUwGAityuapsMWNzkoTDBCoO/PUK6NGcA5ph3Su2syCeOPesQFDg5JU7CVq4g/dA5MOg5wHUx5zsZZ+33pAozK7/Je/CQiULksm+oZBDHC9ypIO4h3Cw== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(39860400002)(346002)(136003)(396003)(451199018)(46966006)(40470700004)(36840700001)(82310400005)(83380400001)(36860700001)(47076005)(6506007)(7696005)(33656002)(40460700003)(81166007)(54906003)(82740400003)(316002)(186003)(336012)(40480700001)(478600001)(356005)(26005)(9686003)(2906002)(8936002)(52536014)(5660300002)(41300700001)(4326008)(86362001)(70586007)(55016003)(6916009)(70206006)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2023 15:11:49.2941 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e9780355-8682-4fae-77f0-08db15b04972 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV2PR08MB9206 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SCC_10_SHORT_WORD_LINES, SCC_20_SHORT_WORD_LINES, SCC_5_SHORT_WORD_LINES, SPF_HELO_PASS, SPF_NONE, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Wilco Dijkstra via Gcc-patches From: Wilco Dijkstra Reply-To: Wilco Dijkstra Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758635137545176836?= X-GMAIL-MSGID: =?utf-8?q?1758635137545176836?= The LSE2 ifunc for 16-byte atomic load requires a barrier before the LDP - without it, it effectively has Load-AcquirePC semantics similar to LDAPR, which is less restrictive than what __ATOMIC_SEQ_CST requires. This patch fixes this and adds comments to make it easier to see which sequence is used for each case. Use a load/store exclusive loop for store to simplify testing memory ordering is correct (it is slightly faster too). Passes regress, OK for commit? libatomic/ PR libgcc/108891 config/linux/aarch64/atomic_16.S: Fix libat_load_16_i1. Add comments describing the memory order. diff --git a/libatomic/config/linux/aarch64/atomic_16.S b/libatomic/config/linux/aarch64/atomic_16.S index 732c3534a06678664a252bdbc53652eeab0af506..05439ce394b9653c9bcb582761ff7aaa7c8f9643 100644 --- a/libatomic/config/linux/aarch64/atomic_16.S +++ b/libatomic/config/linux/aarch64/atomic_16.S @@ -72,33 +72,38 @@ name: \ ENTRY (libat_load_16_i1) cbnz w1, 1f + + /* RELAXED. */ ldp res0, res1, [x0] ret 1: - cmp w1, ACQUIRE - b.hi 2f + cmp w1, SEQ_CST + b.eq 2f + + /* ACQUIRE/CONSUME (Load-AcquirePC semantics). */ ldp res0, res1, [x0] dmb ishld ret -2: + + /* SEQ_CST. */ +2: ldar tmp0, [x0] /* Block reordering with Store-Release instr. */ ldp res0, res1, [x0] - dmb ish + dmb ishld ret END (libat_load_16_i1) ENTRY (libat_store_16_i1) cbnz w4, 1f + + /* RELAXED. */ stp in0, in1, [x0] ret -1: - dmb ish - stp in0, in1, [x0] - cmp w4, SEQ_CST - beq 2f - ret -2: - dmb ish + + /* RELEASE/SEQ_CST. */ +1: ldaxp xzr, tmp0, [x0] + stlxp w4, in0, in1, [x0] + cbnz w4, 1b ret END (libat_store_16_i1) @@ -106,29 +111,33 @@ END (libat_store_16_i1) ENTRY (libat_exchange_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] stxp w4, in0, in1, [x5] cbnz w4, 1b ret 2: cmp w4, ACQUIRE b.hi 4f -3: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME. */ +3: ldaxp res0, res1, [x5] stxp w4, in0, in1, [x5] cbnz w4, 3b ret 4: cmp w4, RELEASE b.ne 6f -5: - ldxp res0, res1, [x5] + + /* RELEASE. */ +5: ldxp res0, res1, [x5] stlxp w4, in0, in1, [x5] cbnz w4, 5b ret -6: - ldaxp res0, res1, [x5] + + /* ACQ_REL/SEQ_CST. */ +6: ldaxp res0, res1, [x5] stlxp w4, in0, in1, [x5] cbnz w4, 6b ret @@ -142,6 +151,8 @@ ENTRY (libat_compare_exchange_16_i1) cbz w4, 2f cmp w4, RELEASE b.hs 3f + + /* ACQUIRE/CONSUME. */ caspa exp0, exp1, in0, in1, [x0] 0: cmp exp0, tmp0 @@ -153,15 +164,18 @@ ENTRY (libat_compare_exchange_16_i1) stp exp0, exp1, [x1] mov x0, 0 ret -2: - casp exp0, exp1, in0, in1, [x0] + + /* RELAXED. */ +2: casp exp0, exp1, in0, in1, [x0] b 0b -3: - b.hi 4f + + /* RELEASE. */ +3: b.hi 4f caspl exp0, exp1, in0, in1, [x0] b 0b -4: - caspal exp0, exp1, in0, in1, [x0] + + /* ACQ_REL/SEQ_CST. */ +4: caspal exp0, exp1, in0, in1, [x0] b 0b END (libat_compare_exchange_16_i1) @@ -169,15 +183,17 @@ END (libat_compare_exchange_16_i1) ENTRY (libat_fetch_add_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] adds tmplo, reslo, inlo adc tmphi, reshi, inhi stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] adds tmplo, reslo, inlo adc tmphi, reshi, inhi stlxp w4, tmp0, tmp1, [x5] @@ -189,15 +205,17 @@ END (libat_fetch_add_16_i1) ENTRY (libat_add_fetch_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] adds reslo, reslo, inlo adc reshi, reshi, inhi stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] adds reslo, reslo, inlo adc reshi, reshi, inhi stlxp w4, res0, res1, [x5] @@ -209,15 +227,17 @@ END (libat_add_fetch_16_i1) ENTRY (libat_fetch_sub_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] subs tmplo, reslo, inlo sbc tmphi, reshi, inhi stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] subs tmplo, reslo, inlo sbc tmphi, reshi, inhi stlxp w4, tmp0, tmp1, [x5] @@ -229,15 +249,17 @@ END (libat_fetch_sub_16_i1) ENTRY (libat_sub_fetch_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] subs reslo, reslo, inlo sbc reshi, reshi, inhi stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] subs reslo, reslo, inlo sbc reshi, reshi, inhi stlxp w4, res0, res1, [x5] @@ -249,15 +271,17 @@ END (libat_sub_fetch_16_i1) ENTRY (libat_fetch_or_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] orr tmp0, res0, in0 orr tmp1, res1, in1 stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] orr tmp0, res0, in0 orr tmp1, res1, in1 stlxp w4, tmp0, tmp1, [x5] @@ -269,15 +293,17 @@ END (libat_fetch_or_16_i1) ENTRY (libat_or_fetch_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] orr res0, res0, in0 orr res1, res1, in1 stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] orr res0, res0, in0 orr res1, res1, in1 stlxp w4, res0, res1, [x5] @@ -289,15 +315,17 @@ END (libat_or_fetch_16_i1) ENTRY (libat_fetch_and_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] and tmp0, res0, in0 and tmp1, res1, in1 stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] and tmp0, res0, in0 and tmp1, res1, in1 stlxp w4, tmp0, tmp1, [x5] @@ -309,15 +337,17 @@ END (libat_fetch_and_16_i1) ENTRY (libat_and_fetch_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] and res0, res0, in0 and res1, res1, in1 stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] and res0, res0, in0 and res1, res1, in1 stlxp w4, res0, res1, [x5] @@ -329,15 +359,17 @@ END (libat_and_fetch_16_i1) ENTRY (libat_fetch_xor_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] eor tmp0, res0, in0 eor tmp1, res1, in1 stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] eor tmp0, res0, in0 eor tmp1, res1, in1 stlxp w4, tmp0, tmp1, [x5] @@ -349,15 +381,17 @@ END (libat_fetch_xor_16_i1) ENTRY (libat_xor_fetch_16_i1) mov x5, x0 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] eor res0, res0, in0 eor res1, res1, in1 stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] eor res0, res0, in0 eor res1, res1, in1 stlxp w4, res0, res1, [x5] @@ -371,15 +405,17 @@ ENTRY (libat_fetch_nand_16_i1) mvn in0, in0 mvn in1, in1 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] orn tmp0, in0, res0 orn tmp1, in1, res1 stxp w4, tmp0, tmp1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] orn tmp0, in0, res0 orn tmp1, in1, res1 stlxp w4, tmp0, tmp1, [x5] @@ -393,15 +429,17 @@ ENTRY (libat_nand_fetch_16_i1) mvn in0, in0 mvn in1, in1 cbnz w4, 2f -1: - ldxp res0, res1, [x5] + + /* RELAXED. */ +1: ldxp res0, res1, [x5] orn res0, in0, res0 orn res1, in1, res1 stxp w4, res0, res1, [x5] cbnz w4, 1b ret -2: - ldaxp res0, res1, [x5] + + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ +2: ldaxp res0, res1, [x5] orn res0, in0, res0 orn res1, in1, res1 stlxp w4, res0, res1, [x5] @@ -413,9 +451,12 @@ END (libat_nand_fetch_16_i1) ENTRY (libat_test_and_set_16_i1) mov w2, 1 cbnz w1, 2f + + /* RELAXED. */ swpb w0, w2, [x0] ret + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ 2: swpalb w0, w2, [x0] ret END (libat_test_and_set_16_i1)