From patchwork Fri Dec 29 20:57:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Laight X-Patchwork-Id: 183937 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp2793481dyb; Fri, 29 Dec 2023 12:57:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IEgRuyBA8EMrNFi875v6GCPHSDVHhQFcZ3GQTwTVgCopJan95hFrrrmIJTaS6nEaydv0FUQ X-Received: by 2002:a05:6358:2786:b0:174:f648:f9c0 with SMTP id l6-20020a056358278600b00174f648f9c0mr7445801rwb.27.1703883471931; Fri, 29 Dec 2023 12:57:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703883471; cv=none; d=google.com; s=arc-20160816; b=jUttCQZFYeC1mA4TlG9RDi0Cp8zVTjNHEZz4R+7rPqoUeyvta2sdrZAI2XOQ1z2zR8 TjDK5hAa0pBtQtrKkUBCYiNPrjMTDi1oFTdP0sB6eH/EVSdSqlR5Knmo76ZirfGrZaZV 3HLDTwNO2S9A7XMiyf9S1Vw0S6evMaajm+jDcRPwor/3d3hLeYotfgeidfvx3zU4MrsD GSfFCxiAdTI0IbvFanMNUtqS6Sl4WVSrcIKiZpePm/vtSVhd0YRYGdeKLvLemdT7oFJT VMN3F0syUdXUR59wj6JAz8f7jD+70HXkOBs+2AWVs9iTS4z66GigSedXy9kTtF4cYS7J 7A+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:accept-language :in-reply-to:references:message-id:date:thread-index:thread-topic :subject:cc:to:from; bh=DWsG1Ubi+miEftDxDAFqhr19zEgnKNFv58uQtnm0+JY=; fh=9mjHhe+fO0FRmfh1SiQmAZ16aLlCaXA7AOG78MgCddY=; b=v0nFOu+/wGXUeUkPxoY1uO0tlbFXRJkDet8ykcdD2jCWhL12qAW3MKqAAFxTuIQD5O qjXjb2eTuqrSqCMQtcpL58Q3VoolKGoG4RG363wlZHQOdcpzsEzPAXyEFgR1M+/eZnXy L+0SANGOf/6bEazGBzROcUErm4xg9hts1AjidcknvkluWyNJPrIqYwYW1PlC1NV6gCXZ uc5dnY1VYuJe02rHRZIn8izjqEfaop33ec7HcN4RWTMfaCIcxn7U7l0lUlmnnvGWKvHh Q+8K6oyXCtSAwPLQJEmW5AVHXb2szt/Tu1ZM3F10KlTisXDMrtw2LwAZVIZnqq+uP3Vt 6uuw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13232-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13232-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id x26-20020a62fb1a000000b006d9b327f2d8si4642193pfm.74.2023.12.29.12.57.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Dec 2023 12:57:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-13232-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-13232-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-13232-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=aculab.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id AD4BD284275 for ; Fri, 29 Dec 2023 20:57:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4CCF814AA9; Fri, 29 Dec 2023 20:57:39 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C13B914A83 for ; Fri, 29 Dec 2023 20:57:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-311-yCoQCupGOBSlWEcLQ3ukbA-1; Fri, 29 Dec 2023 20:57:31 +0000 X-MC-Unique: yCoQCupGOBSlWEcLQ3ukbA-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:57:14 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:57:14 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Topic: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Index: Ado6mZUJWFdx4PkETd+mn/PWVjPd0A== Date: Fri, 29 Dec 2023 20:57:13 +0000 Message-ID: References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786651315777103321 X-GMAIL-MSGID: 1786651315777103321 this_cpu_ptr() is rather more expensive than raw_cpu_read() since the latter can use an 'offset from register' (%gs for x86-84). Add a 'self' field to 'struct optimistic_spin_node' that can be read with raw_cpu_read(), initialise on first call. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 9bb3a077ba92..b60b0add0161 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -13,7 +13,7 @@ */ struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; + struct optimistic_spin_node *self, *next, *prev; int locked; /* 1 if lock acquired */ int cpu; /* encoded CPU # + 1 value */ }; @@ -93,12 +93,16 @@ osq_wait_next(struct optimistic_spin_queue *lock, bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node = this_cpu_ptr(&osq_node); + struct optimistic_spin_node *node = raw_cpu_read(osq_node.self); struct optimistic_spin_node *prev, *next; int old; - if (unlikely(node->cpu == OSQ_UNLOCKED_VAL)) - node->cpu = encode_cpu(smp_processor_id()); + if (unlikely(!node)) { + int cpu = encode_cpu(smp_processor_id()); + node = decode_cpu(cpu); + node->self = node; + node->cpu = cpu; + } /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -222,7 +226,7 @@ void osq_unlock(struct optimistic_spin_queue *lock) /* * Second most likely case. */ - node = this_cpu_ptr(&osq_node); + node = raw_cpu_read(osq_node.self); next = xchg(&node->next, NULL); if (next) { WRITE_ONCE(next->locked, 1);