From patchwork Sun Dec 31 21:54:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Laight X-Patchwork-Id: 184146 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp3698567dyb; Sun, 31 Dec 2023 13:55:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IHohmJHII9UGgpodp5CtZK1bGnK+fAeotl/l1BajU2cqhiR4BKxT6OJXHr/I/tz7qH+39Kg X-Received: by 2002:a05:6808:3c46:b0:3bb:bfa4:e6a3 with SMTP id gl6-20020a0568083c4600b003bbbfa4e6a3mr11397769oib.13.1704059741440; Sun, 31 Dec 2023 13:55:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704059741; cv=none; d=google.com; s=arc-20160816; b=po/kaqFs+w7g4N6Rub/g/b0a+DHTpb566UFSHB+FBMlh5lrRcibcOzQNUn7JHuCGUD YXtt3f9GR1M9pns92+f2Y297MgYF3g3u2C1pIuA90IxzjEL2iznqt1Lz2Jk8EfO7SuBC d/K1cQ0r+BO3Ry0/LqB0fLV4QHtU+zaQUvmxcewshEaZirFzZqMYgY3p3EP+AP/eV2pi /jc53o0belwbhVBBrR65OuctyMk1VU9/t6Mq6inEt9D8qeuEO+I/KN0HIBRtHEQ7tp49 mHIxpiIX92psWvVw95z3VqtXB4RJqZmVRqkLPLNLmAGDX5S18/XlQ7/+0vr7vfosNXwZ gzgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:accept-language :in-reply-to:references:message-id:date:thread-index:thread-topic :subject:cc:to:from; bh=S3V7uFQw1DfLwXObTeBHxkc0OVlMDI0mQz0HxIBqovs=; fh=rK8F4zm15J6Jc7pqIJYnBw4d8zHrqB7RdoX7pamyocA=; b=biQrb8crTkFexurE5zK9XRLnOV5ApeLjVlm6ULyJcm9ISKNEvr0Dp+sBEntMCdYVRA IWMNYPyEvOAg39qhUfkPO4CBfYy4eUKUHMQwcdvDrwAZo4qaEDaMkZTSxui8mcUevQjS AYFlsPFKt9Beu2NDfIrdxB65eQs4zoLgyHMt+AC9dVXGj16uyWdYnm+478MvpMBXui1d 27PCpMRgxn6z47CRA4XBkLgD0r2kimp0fRf09HquTp0O5D4WlNAuOdij7gz86CX0UBFj 6DTq4zy+dYCE4rNFb8b4vnarWkwNuUxYY/H7eWJn8Y4zoMdgWR7xSMv05AuaPP+xq9PW xpmg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13793-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13793-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id eb6-20020a056a004c8600b006da24ee5340si5788134pfb.59.2023.12.31.13.55.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Dec 2023 13:55:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-13793-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13793-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13793-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 35B062826A0 for ; Sun, 31 Dec 2023 21:55:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4B270FBED; Sun, 31 Dec 2023 21:55:27 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.85.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48BC1F9CC for ; Sun, 31 Dec 2023 21:55:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-44-6gjeuhImOKioWY8extJLBA-1; Sun, 31 Dec 2023 21:55:20 +0000 X-MC-Unique: 6gjeuhImOKioWY8extJLBA-1 Received: from AcuMS.Aculab.com (10.202.163.6) by AcuMS.aculab.com (10.202.163.6) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Sun, 31 Dec 2023 21:54:59 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Sun, 31 Dec 2023 21:54:59 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next v2 4/5] locking/osq_lock: Avoid writing to node->next in the osq_lock() fast path. Thread-Topic: [PATCH next v2 4/5] locking/osq_lock: Avoid writing to node->next in the osq_lock() fast path. Thread-Index: Ado8NAjQtRL812H3R1Kc4G+FOscjCQ== Date: Sun, 31 Dec 2023 21:54:59 +0000 Message-ID: <06a11b2c7d784f2d80dc8e81c7175c57@AcuMS.aculab.com> References: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> In-Reply-To: <2b4e8a5816a742d2bd23fdbaa8498e80@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786836147450490119 X-GMAIL-MSGID: 1786836147450490119 When osq_lock() returns false or osq_unlock() returns static analysis shows that node->next should always be NULL. This means that it isn't necessary to explicitly set it to NULL prior to atomic_xchg(&lock->tail, curr) on extry to osq_lock(). Just in case there a non-obvious race condition that can leave it non-NULL check with WARN_ON_ONCE() and NULL if set. Note that without this check the fast path (adding at the list head) doesn't need to to access the per-cpu osq_node at all. Signed-off-by: David Laight Reviewed-by: Waiman Long --- kernel/locking/osq_lock.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 27324b509f68..35bb99e96697 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -87,12 +87,17 @@ osq_wait_next(struct optimistic_spin_queue *lock, bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node = this_cpu_ptr(&osq_node); - struct optimistic_spin_node *prev, *next; + struct optimistic_spin_node *node, *prev, *next; int curr = encode_cpu(smp_processor_id()); int prev_cpu; - node->next = NULL; + /* + * node->next should be NULL on entry. + * Check just in case there is a race somewhere. + * Note that this is probably an unnecessary cache miss in the fast path. + */ + if (WARN_ON_ONCE(raw_cpu_read(osq_node.next) != NULL)) + raw_cpu_write(osq_node.next, NULL); /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -104,8 +109,9 @@ bool osq_lock(struct optimistic_spin_queue *lock) if (prev_cpu == OSQ_UNLOCKED_VAL) return true; - node->prev_cpu = prev_cpu; + node = this_cpu_ptr(&osq_node); prev = decode_cpu(prev_cpu); + node->prev_cpu = prev_cpu; node->locked = 0; /*