From patchwork Fri Dec 29 22:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Laight X-Patchwork-Id: 183954 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp2820199dyb; Fri, 29 Dec 2023 14:12:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IEGWIqDFutHyNoZch2dP6RDdPbv6JjmEbymbqAGCNwpkP2Qh8WtTOZo1HCqe2OCMo1XAxIn X-Received: by 2002:a50:d59b:0:b0:553:2dc2:4b9b with SMTP id v27-20020a50d59b000000b005532dc24b9bmr7955943edi.14.1703887936594; Fri, 29 Dec 2023 14:12:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703887936; cv=none; d=google.com; s=arc-20160816; b=qMTT43OOfbdm+NUmTYUpMEmR41hZX6iFSYMBihk0uvGnLPHEy00CVZMVbQGAheJgo3 3sJZeZksuSP0wcVs5RzuN+uufUbzZj3idmzFqzwRW2xyi0EWc0gqcu4+WuV2TvwW3UmX ii6BUzc3R6QjOCEqzyunvFh6+mLyPMIGAfPElX3gdl4hSwbwGV5W9wxWYJZLyP5+Eq9p GYbpW3fXmwptsRtSg0JQt/I4dTt+Tm1LCBCAysyPVEJn8OE9sJcOneLvcsL3zebYE69G UZoiWZrJYVy6zIC1A01YZ0uuMAdGx2XsCfV9iye/0KgOtwmdW434+2dSMRe48zG42E8g yQlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:accept-language :in-reply-to:references:message-id:date:thread-index:thread-topic :subject:cc:to:from; bh=Zxoj+Whcj6wwFWOInbOmrLUCPKc+AX+1M+bKnwz8TIE=; fh=9mjHhe+fO0FRmfh1SiQmAZ16aLlCaXA7AOG78MgCddY=; b=p+JTrZ4F3nv98aSJ6BOHGxZbE8Eh/MkFPPOtHwa354ERCSH14dvc9krQnYabHgBfRa +7zu2p1ugG6kRaPIpCJF6HAAlTDWfUjaPcWI1Skis8bc8q+N0aZt9qsK5Om7f8GhW0+7 YWTvWOkh9RNhBTBRnpKXSY0FxJcFOGaZgPi31YpNpJ2DX6gd7vD72fz5l2XrV9vUFCU3 QxyL6i2K7f6tyz0IQOKzK16dlMvwHOmT3cPGmXiS9M+4wmm11c7v6NEzrGFtGsPULqPK puJXzfszHYMJ+lzUeu4XAKMvIwrBKwHCauyNt+4BBfIXzRCtpKApDM/o0uh0iJZWwk7t vo0w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13259-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13259-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id m13-20020a056402510d00b00553b47279b0si7981978edd.220.2023.12.29.14.12.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 14:12:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-13259-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13259-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13259-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 37CF11F22E5D for ; Fri, 29 Dec 2023 22:12:16 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 03E8C14AB8; Fri, 29 Dec 2023 22:12:01 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CE6E14A97 for ; Fri, 29 Dec 2023 22:11:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-227-QXrAE9vJN0uhbMdRrMv0sA-1; Fri, 29 Dec 2023 22:11:54 +0000 X-MC-Unique: QXrAE9vJN0uhbMdRrMv0sA-1 Received: from AcuMS.Aculab.com (10.202.163.6) by AcuMS.aculab.com (10.202.163.6) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 22:11:40 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 22:11:40 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Topic: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Index: Ado6o/zBb4T2uRNuSTy7E9ZX7SDa6w== Date: Fri, 29 Dec 2023 22:11:40 +0000 Message-ID: References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786651511658470658 X-GMAIL-MSGID: 1786655997269097234 osq_lock() starts by setting node->next to NULL and node->locked to 0. Careful analysis shows that node->next is always NULL on entry. node->locked is set non-zero by another cpu to force a wakeup. This can only happen after the 'prev->next = node' assignment, so locked can be set to zero just before that (along with the assignment to node->prev). Only initialise node->cpu once, after that use its value instead of smp_processor_id() - which is probably a real function call. Should reduce cache-line bouncing a little. Signed-off-by: David Laight --- Re-send without the 'RE:' on the subject line. kernel/locking/osq_lock.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index d414eef4bec6..55f5db896c02 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -51,7 +51,7 @@ osq_wait_next(struct optimistic_spin_queue *lock, struct optimistic_spin_node *prev) { struct optimistic_spin_node *next = NULL; - int curr = encode_cpu(smp_processor_id()); + int curr = node->cpu; int old; /* @@ -98,12 +98,10 @@ bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node = this_cpu_ptr(&osq_node); struct optimistic_spin_node *prev, *next; - int curr = encode_cpu(smp_processor_id()); int old; - node->locked = 0; - node->next = NULL; - node->cpu = curr; + if (unlikely(node->cpu == OSQ_UNLOCKED_VAL)) + node->cpu = encode_cpu(smp_processor_id()); /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -111,12 +109,13 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - old = atomic_xchg(&lock->tail, curr); + old = atomic_xchg(&lock->tail, node->cpu); if (old == OSQ_UNLOCKED_VAL) return true; prev = decode_cpu(old); node->prev = prev; + node->locked = 0; /* * osq_lock() unqueue @@ -214,7 +213,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) void osq_unlock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node, *next; - int curr = encode_cpu(smp_processor_id()); + int curr = raw_cpu_read(osq_node.cpu); /* * Fast path for the uncontended case.