Message ID | 20240124105816.612670-1-ubizjak@gmail.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2553:b0:103:945f:af90 with SMTP id p19csp906980dyi; Wed, 24 Jan 2024 02:59:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IHr11kwDy3o5T/5jVjCrjc9Jhfgqe9hxDqK5UZGQIMpRC44u4yGoj8FF4frvdUB1ygxpw5P X-Received: by 2002:a05:6a20:4c85:b0:19c:34e1:4673 with SMTP id fq5-20020a056a204c8500b0019c34e14673mr608653pzb.107.1706093978454; Wed, 24 Jan 2024 02:59:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706093978; cv=pass; d=google.com; s=arc-20160816; b=Ts0R23osOA0hXgfk5iW3ltblRO6433le2fM2kfHWgtJuIVvhoLBIwQIWdFSrDGCxMD o61jFvxvIfDBJ14sntU4oV/X/haO1oms9CF15pLiyveVr0Wc1MTCFjJIvafjpxKRtd0U A1nC2CBpO6AeFg9/Xu1hZyH6yyrDs5AeNAE1uxeaUspbvmR5Pw2WNbK/cOB0TXukFnqB v4wGy7VCszuYdnrSwgTFVIWkO4yGK+6qul063dxAPDetRPm7kgfcb6zpxD91551m64SC bRH4tXEW7lPidTXmW3xOpOFljcj9ow7AqEOFFs4mA2fxUrKmdWWen026oXkH5unO+9MS E9Kg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=Ev+MvDWxtLh9tCM9Z9MMVPgHcCFJtvGcjwdf7MijC/Y=; fh=BZuqF0bxLOoHpmsorgAowQln80CBe9e5Ji9LrGNnUlw=; b=NgCRjS2BKSoBoKd0wyKLJuVP0fEZj5nnpViwu74ODDkgb2hBpJ7xghBKoB5bzTZsRB 5W6qlk5lb8HvxHGKfoyy4mwcRYxh2AkW8n9I75nqBuRu7T9mtg+ZmHOmoympovxw/te6 eHyAG9Hmu/4E0klSaea9Jqlj4FTPUY9nIrSvz9yAqTWJDsT93+v385OqdFFY1130fMBU WgVrXKZuezEpBuYt9J2Ae5hF+fbhdVLDaa97DIGRXin2wVL77c2okSnad4m+5avEWPL5 a8M4PapSHwGJJ9ealP0Z9MNZGRLpyKCxBInBs5Mu3dzmEagvyHb2qCLuiUOQ/wiwMSNQ GbFg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=eLSjG8AI; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id i8-20020a17090a974800b002901ded8651si11166612pjw.72.2024.01.24.02.59.38 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 02:59:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=eLSjG8AI; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-36821-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 780C528B018 for <ouuuleilei@gmail.com>; Wed, 24 Jan 2024 10:59:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E8CA620DE1; Wed, 24 Jan 2024 10:58:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eLSjG8AI" Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 985EB1119C for <linux-kernel@vger.kernel.org>; Wed, 24 Jan 2024 10:58:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706093903; cv=none; b=aPDZKTO9kMiqNuxiTKXg4zf782Oh6y29Pa9GCzPfCtC4b4mS0aLZhHHfWkdqThWGu737t8WLJRW05w1o6ZjvTz1gYAjQSt01OQZXLWgPtnBeHh1fgRxHUJpYw4pYYE7p1nM13WQMKVyx6peK/jOFNxAVFUeN7t4qId7i6SWt5zo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706093903; c=relaxed/simple; bh=wOa1a15URAxU8n05sFB6jneNgvxjEAxEfI+1gpcKVHc=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=tExNfytaLBFlPOeE81gGy/+XHrLsIo+0meDuxyUfDFFlqot3Se3OgW+3IYcEdJR4Hshjp5q4rc/jkOHrl86Lr4/sptdoUkrZaZCRpW3DgadnogD2P0ZjWS5eUzoVFX3iJ0bwRtQaoJT30ikf2aI8Jr6i/UxYxLFAKdSWb3xtw9Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eLSjG8AI; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a3106f5aac8so88388166b.0 for <linux-kernel@vger.kernel.org>; Wed, 24 Jan 2024 02:58:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706093900; x=1706698700; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Ev+MvDWxtLh9tCM9Z9MMVPgHcCFJtvGcjwdf7MijC/Y=; b=eLSjG8AIJITQb05QAftDulO2tRZqhwgic4zqDMmqcTWEfxoOrMLZA85HAsOp7fhFzR zwcnP1x0eHDFTUu7Uq82NM/+y3sKkLURv1nDxFRgbKHjLIp6KT94BTDEbBlh980Oenxi GDyJ/1+aVwRRwDf6eCBniokvu/OwpcWO4uAftNxavLg1LOesYo+MupbdFCp4EbMYdgds /xgGTxVXfNmXbSvT7r5RTvKEUv+3jbTaZfbzRnyJxH9QjWxIC3/SCoWpfwFNp1fWw9Pp +sJNAwTFMxUGC2ijSWq3WtaqtGwBv0eoVEcSMhovvhNcySGace7kogkSlX1azqQw+6Sc Lb5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706093900; x=1706698700; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ev+MvDWxtLh9tCM9Z9MMVPgHcCFJtvGcjwdf7MijC/Y=; b=lt1FWt92QcAPW4gdCp6zoQT5x1xYTFSh1FkbaDj7+x9bXY/Z/ftUFoPc6BvH5Bg0mX /mgZjkzVxiglgaVecJvYpl8Ul1P//SeLbmwGkaTd+K67WZ/bxFWUfvu2+jTIr08srxwe IQnEd6IHtAJw5JQvQAli/mIaBZlOyusqyaTQ0VTzZem3Q9esMcUpUUSvjiG7Dk52Al8M UJXHhLvU/ggFCye0Qrp+0G2tairUDZdlstlgoBgd7MVZWffznIZZNfNhh3oiYocKfYCo /E9c6bv6xutAbD3lIqRQs45Fb3USBZjlRcOYVnRfjajXTu6quX8YChP6hjPU8/05vezT q+vQ== X-Gm-Message-State: AOJu0YyM2hWtjGt7LZUuPHMn8mnJdeFVpYpEQ19yne6RAV7jsjW3avAO Uvl5n20eoN1O7O56o53YU03LV6CLe4IUWjw351rHspeo+Szy/RqZ X-Received: by 2002:a17:907:c243:b0:a2d:a6b7:cce with SMTP id tj3-20020a170907c24300b00a2da6b70ccemr973001ejc.122.1706093899520; Wed, 24 Jan 2024 02:58:19 -0800 (PST) Received: from localhost.si (84-255-245-182.static.t-2.net. [84.255.245.182]) by smtp.gmail.com with ESMTPSA id tj3-20020a170907c24300b00a310f2b4b91sm646377ejc.48.2024.01.24.02.58.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 02:58:19 -0800 (PST) From: Uros Bizjak <ubizjak@gmail.com> To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak <ubizjak@gmail.com>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, "H. Peter Anvin" <hpa@zytor.com> Subject: [PATCH] x86/asm: Implement local_xchg using CMPXCHG without lock prefix Date: Wed, 24 Jan 2024 11:58:16 +0100 Message-Id: <20240124105816.612670-1-ubizjak@gmail.com> X-Mailer: git-send-email 2.31.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: <linux-kernel.vger.kernel.org> List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org> List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788969199705367877 X-GMAIL-MSGID: 1788969199705367877 |
Series |
x86/asm: Implement local_xchg using CMPXCHG without lock prefix
|
|
Commit Message
Uros Bizjak
Jan. 24, 2024, 10:58 a.m. UTC
Implement local_xchg using CMPXCHG instruction without lock prefix.
XCHG is expensive due to the implied lock prefix. The processor
cannot prefetch cachelines if XCHG is used.
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
---
arch/x86/include/asm/local.h | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/local.h b/arch/x86/include/asm/local.h index 73dba8b94443..f9af6908aa2f 100644 --- a/arch/x86/include/asm/local.h +++ b/arch/x86/include/asm/local.h @@ -131,8 +131,20 @@ static inline bool local_try_cmpxchg(local_t *l, long *old, long new) (typeof(l->a.counter) *) old, new); } -/* Always has a lock prefix */ -#define local_xchg(l, n) (xchg(&((l)->a.counter), (n))) +/* + * Implement local_xchg using CMPXCHG instruction without lock prefix. + * XCHG is expensive due to the implied lock prefix. The processor + * cannot prefetch cachelines if XCHG is used. + */ +static __always_inline long +local_xchg(local_t *l, long n) +{ + long c = local_read(l); + + do { } while (!local_try_cmpxchg(l, &c, n)); + + return c; +} /** * local_add_unless - add unless the number is already a given value