From patchwork Mon Oct 17 21:13:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 3746 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1647100wrs; Mon, 17 Oct 2022 14:21:52 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6ewMidRoP3EZrSrjoUmKlXH4H930I2ZWMcrvKAiAJmAjkdBeTJWHNZNdcdc3+wPvsxb7jD X-Received: by 2002:a17:902:f7c4:b0:182:25d6:fc4b with SMTP id h4-20020a170902f7c400b0018225d6fc4bmr13846527plw.63.1666041701171; Mon, 17 Oct 2022 14:21:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666041701; cv=none; d=google.com; s=arc-20160816; b=NqeYyGWNCw8BuSkbm4gJ4YQQlNnF8vqoHJodot4MWl+XQWnm+YRcUsJJlT6jjXSoDo QoEIZndGBSC6Mjo2I1OuUkCQGko8uXhAAe7Yezgh9IObVzEGToPwBmnkFQZ4uS3UqfFf ogUcfXqpeOdbtglgsKOgjnmekJk3RbExkTidAVVsPKpBvU0oi2f54lnWEmnmPIF2Re8n i8WjZWlN9v1OfY4jC1haKtjYOcRcwWwF8Xyfa3VFOUpy0i9NyAwaLeJIQQOgNWp40Gpx phxzIjyLT8qzs7WlE/QESj+LsPFgqUluvLj6oWwWFlGiJrSPVbeUEoh6luKg0oHkw4cY SgIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FOjkxhhELa1U9D/ByzteSY8pSFjeQrEmDncUbTIS7+c=; b=GLz+M6OD36ZPMpaFp4AtGwg1lOabN7pJvbdFuwezSl376gLlp5/5cgfz08SBcjiPEx vsnzPeV5/aWlJnydOU8b3dWqdbN2Hz6KFmy0Fbmf23fpP+mBOE64KJ7uMFTiOjUADNql JAjlUpgQSEEXocyc+Kd+V5FgrLgEqGXxkw6yLRU3fKWKNasBJGrEyeifY6hbMcBeQ/u9 MkPi0d4YPF4u86/HrHDcUJL7f1OFJFGmtDz9s/3vn6JWIyDOboIArt+xiy10M49hsTuk YnME1HDKsLHeUzv2QzhGIFCLN5glPnM0BvadFq9xjhADmK+hWVC2WAUgFVJZqnus+G5r fZQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=A5M3h+0E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g4-20020a1709026b4400b0017f8ca45f3dsi12048291plt.131.2022.10.17.14.21.12; Mon, 17 Oct 2022 14:21:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=A5M3h+0E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230345AbiJQVPE (ORCPT + 99 others); Mon, 17 Oct 2022 17:15:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229848AbiJQVOo (ORCPT ); Mon, 17 Oct 2022 17:14:44 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8447B12AF0 for ; Mon, 17 Oct 2022 14:14:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666041280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FOjkxhhELa1U9D/ByzteSY8pSFjeQrEmDncUbTIS7+c=; b=A5M3h+0EGbEnjr+QC5kiWmCuJlld+7OKwkyoooYOHgIAPMjCuX5bmFH5YTVRfK8p3YTV93 /zcbicelfkI0PrsYNVYfBgkP1ufmYc2GO5CbuTN5TxvQ+NmuqcJNPChXktCToc9wLdi1DZ n3/FjHDpppJhrvydl80UG9ERlACUo38= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-594-7Iole4m3PgibqziWY8sfPQ-1; Mon, 17 Oct 2022 17:14:36 -0400 X-MC-Unique: 7Iole4m3PgibqziWY8sfPQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 41BFE29AA2F0; Mon, 17 Oct 2022 21:14:36 +0000 (UTC) Received: from llong.com (unknown [10.22.33.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id D3AAD40C206B; Mon, 17 Oct 2022 21:14:35 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng Cc: linux-kernel@vger.kernel.org, john.p.donnelly@oracle.com, Hillf Danton , Mukesh Ojha , =?utf-8?b?VGluZzExIFdhbmcg546L5am3?= , Waiman Long Subject: [PATCH v3 1/5] locking/rwsem: Prevent non-first waiter from spinning in down_write() slowpath Date: Mon, 17 Oct 2022 17:13:52 -0400 Message-Id: <20221017211356.333862-2-longman@redhat.com> In-Reply-To: <20221017211356.333862-1-longman@redhat.com> References: <20221017211356.333862-1-longman@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746971342966287978?= X-GMAIL-MSGID: =?utf-8?q?1746971342966287978?= A non-first waiter can potentially spin in the for loop of rwsem_down_write_slowpath() without sleeping but fail to acquire the lock even if the rwsem is free if the following sequence happens: Non-first waiter First waiter Lock holder ---------------- ------------ ----------- Acquire wait_lock rwsem_try_write_lock(): Set handoff bit if RT or wait too long Set waiter->handoff_set Release wait_lock Acquire wait_lock Inherit waiter->handoff_set Release wait_lock Clear owner Release lock if (waiter.handoff_set) { rwsem_spin_on_owner((); if (OWNER_NULL) goto trylock_again; } trylock_again: Acquire wait_lock rwsem_try_write_lock(): if (first->handoff_set && (waiter != first)) return false; Release wait_lock It is especially problematic if the non-first waiter is an RT task and it is running on the same CPU as the first waiter as this can lead to live lock. Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more consistent") Reviewed-and-tested-by: Mukesh Ojha Signed-off-by: Waiman Long --- kernel/locking/rwsem.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index 44873594de03..be2df9ea7c30 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -624,18 +624,16 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, */ if (first->handoff_set && (waiter != first)) return false; - - /* - * First waiter can inherit a previously set handoff - * bit and spin on rwsem if lock acquisition fails. - */ - if (waiter == first) - waiter->handoff_set = true; } new = count; if (count & RWSEM_LOCK_MASK) { + /* + * A waiter (first or not) can set the handoff bit + * if it is an RT task or wait in the wait queue + * for too long. + */ if (has_handoff || (!rt_task(waiter->task) && !time_after(jiffies, waiter->timeout))) return false; @@ -651,11 +649,12 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, } while (!atomic_long_try_cmpxchg_acquire(&sem->count, &count, new)); /* - * We have either acquired the lock with handoff bit cleared or - * set the handoff bit. + * We have either acquired the lock with handoff bit cleared or set + * the handoff bit. Only the first waiter can have its handoff_set + * set here to enable optimistic spinning in slowpath loop. */ if (new & RWSEM_FLAG_HANDOFF) { - waiter->handoff_set = true; + first->handoff_set = true; lockevent_inc(rwsem_wlock_handoff); return false; } From patchwork Mon Oct 17 21:13:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 3748 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1649527wrs; Mon, 17 Oct 2022 14:30:24 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6s/wkbmPCggHwpskKtmIxrC5pxrITM662owhZJ3Q1Mm360OHxma2Ip2xfAT7hcSxfM80BO X-Received: by 2002:a65:4585:0:b0:464:4538:c0bc with SMTP id o5-20020a654585000000b004644538c0bcmr12576855pgq.455.1666042223826; Mon, 17 Oct 2022 14:30:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666042223; cv=none; d=google.com; s=arc-20160816; b=tylqLPM8LoFFFBsOCBFRZ8JMRx+Gt4la3w++NAtVzd7MvirYAYLut8ij+bHk/4xi+D ob3w9ouMmcTkcOpSVY+c1xK5tBfZh+WU4xInXaSSo+YSSQ8JzPDfne4ERL5on12bGkM8 AA6rchBz9TwVnYe+tcB04p5QTUIntRuw9vmSZvuR+QIiKCMIbyI++PlVGVTmkNDv1CBd g69AJM0rKDRSyyIXh1dJJhNu/Y9dBDuhc4JvgUN1n7wvYKW4UUvgxIf47r7m14AKN8RP AmsscNm0tNAieZXUFptyYmSeHVVeL+Y9eKNnc4VbilPzac83Anb3t0jKv4sniLaVcAeW fKsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5ELOXUp+yBwsB57Mhpd3zpRxtPtGhdnF8W6vkkIQT1w=; b=POuD6ZjpB/koRhImUCp1xaXGuUeOAeIvTln5cIsZLNSpujAMjLUSsry5vzfyYjOmaH 18wQmRJF1q2aDjswBkKjKKaQXgCloAz8kJQemtoXvfNo2uvqazxPPVJd+u+JADGZ49V/ njKHfEh4uXxhH6BVjrQyuuGhLe056ljGD5ldI6KP3mTrA/nCcJbWAkB9p0x3r1kqacEW KFm4DdMuYTygZ04liG6gZa0zn0UmrP8UpWxVfbish0f3nA/+amuNgxa28sQJ/ytjtkau Q3SFBs9vRr3idm84ILFsE5fiIUy3FPsB7fUI1kIRPzqb18uym7vw/q7317ayj/6Io6Oj 6vZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iQat6eIt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n15-20020a170902f60f00b0017f8290fcf0si12911309plg.272.2022.10.17.14.30.10; Mon, 17 Oct 2022 14:30:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iQat6eIt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230230AbiJQVO5 (ORCPT + 99 others); Mon, 17 Oct 2022 17:14:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230014AbiJQVOp (ORCPT ); Mon, 17 Oct 2022 17:14:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E053115A11 for ; Mon, 17 Oct 2022 14:14:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666041280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5ELOXUp+yBwsB57Mhpd3zpRxtPtGhdnF8W6vkkIQT1w=; b=iQat6eItQC468daxI9mFTRkb5xSl6yAHaQ2+bnJSiMXNuFwJLuMl7e6HDZiQAZNxqisD/f LWerrIV7bh4C/nuUUnk6+TTyGz3/3kydEYdPFwcSOg6ySVNn1u/cuZS6sUyBY8N+nLk7HN KGkYqWxeV5Dl56hxsqQmQdpHM9lPukI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-596-o2yZrGTSN0m2IO1NOahIbA-1; Mon, 17 Oct 2022 17:14:37 -0400 X-MC-Unique: o2yZrGTSN0m2IO1NOahIbA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AD3FA101A528; Mon, 17 Oct 2022 21:14:36 +0000 (UTC) Received: from llong.com (unknown [10.22.33.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D66740C206B; Mon, 17 Oct 2022 21:14:36 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng Cc: linux-kernel@vger.kernel.org, john.p.donnelly@oracle.com, Hillf Danton , Mukesh Ojha , =?utf-8?b?VGluZzExIFdhbmcg546L5am3?= , Waiman Long Subject: [PATCH v3 2/5] locking/rwsem: Limit # of null owner retries for handoff writer Date: Mon, 17 Oct 2022 17:13:53 -0400 Message-Id: <20221017211356.333862-3-longman@redhat.com> In-Reply-To: <20221017211356.333862-1-longman@redhat.com> References: <20221017211356.333862-1-longman@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746971890843441641?= X-GMAIL-MSGID: =?utf-8?q?1746971890843441641?= Commit 91d2a812dfb9 ("locking/rwsem: Make handoff writer optimistically spin on owner") assumes that when the owner field is changed to NULL, the lock will become free soon. That assumption may not be correct especially if the handoff writer doing the spinning is a RT task which may preempt another task from completing its action of either freeing the rwsem or properly setting up owner. To prevent this live lock scenario, we have to limit the number of trylock attempts without sleeping. The current limit is now set to 8 to allow enough time for the other task to hopefully complete its action. By adding new lock events to track the number of NULL owner retries with handoff flag set before a successful trylock when running a 96 threads locking microbenchmark with equal number of readers and writers running on a 2-core 96-thread system for 15 seconds, the following stats are obtained. Note that none of locking threads are RT tasks. Retries of successful trylock Count ----------------------------- ----- 1 1738 2 19 3 11 4 2 5 1 6 1 7 1 8 0 X 1 The last row is the one failed attempt that needs more than 8 retries. So a retry count maximum of 8 should capture most of them if no RT task is in the mix. Fixes: 91d2a812dfb9 ("locking/rwsem: Make handoff writer optimistically spin on owner") Reported-by: Mukesh Ojha Signed-off-by: Waiman Long Reviewed-and-tested-by: Mukesh Ojha --- kernel/locking/rwsem.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index be2df9ea7c30..c68d76fc8c68 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -1115,6 +1115,7 @@ static struct rw_semaphore __sched * rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) { struct rwsem_waiter waiter; + int null_owner_retries; DEFINE_WAKE_Q(wake_q); /* do optimistic spinning and steal lock if possible */ @@ -1156,7 +1157,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) set_current_state(state); trace_contention_begin(sem, LCB_F_WRITE); - for (;;) { + for (null_owner_retries = 0;;) { if (rwsem_try_write_lock(sem, &waiter)) { /* rwsem_try_write_lock() implies ACQUIRE on success */ break; @@ -1182,8 +1183,21 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) owner_state = rwsem_spin_on_owner(sem); preempt_enable(); - if (owner_state == OWNER_NULL) + /* + * owner is NULL doesn't guarantee the lock is free. + * An incoming reader will temporarily increment the + * reader count without changing owner and the + * rwsem_try_write_lock() will fails if the reader + * is not able to decrement it in time. Allow 8 + * trylock attempts when hitting a NULL owner before + * going to sleep. + */ + if ((owner_state == OWNER_NULL) && + (null_owner_retries < 8)) { + null_owner_retries++; goto trylock_again; + } + null_owner_retries = 0; } schedule(); From patchwork Mon Oct 17 21:13:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 3749 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1650781wrs; Mon, 17 Oct 2022 14:33:38 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4p+oLYd6Q2mxtoAowJVKnDfQ+fHg8fNteVqwr8D9lTogNFqYJHbtFNAnBSxgLLd1ne8yoq X-Received: by 2002:a05:6402:40c4:b0:45c:c718:54d1 with SMTP id z4-20020a05640240c400b0045cc71854d1mr12149524edb.319.1666042418371; Mon, 17 Oct 2022 14:33:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666042418; cv=none; d=google.com; s=arc-20160816; b=X8YdJeqndQz6B15S/5hLHli3S4p2EtISCztAQiF9ZQDUP94LCWqUU6GIX6WNwWqziE aWioExGCVabs11JOCcQebetY8T1EabZEGeWIdGNSVL7tMDrorvOJ0qQgx+WAHcPGIr8Z DJJ7592YGgh3+3ELBRZ7DLoLRLJdtBUh6e2xjHkJJQKf0f4lWrCn1bMKie3LFsH1w1M4 4FdeTdGgp0pteC37v5PbI/7rzEphWJS9M6pLa/hkw5KB7U3iBHTfSWgYRbjprTO3HIiI ZPzyNPaGKYb94HmPj4aRvtDeflaMTPH4aVMUNamoAnUSmmP1pt+3tu5P0A+ztP/eg2ks l7lA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+XFNqhe/9TyVt9WobmDUsN79a+94gmZaId1xsUvIpII=; b=f0CLrCgNgYH9lBCibbLNwn2NDjCNb60ly1SHrG9uGF0wWDInfPycyXJ2NY/WJw3Gj+ +0C65yNdApP4dIA16fRZXdyxyb5R5IEf67E9DSM7F1fNJJUDrYgrAZwocNZ6WgYwzSaB DUWYvEoHglzchCiZNsy2V9OG/bI08SZI6+8daf65mbGTLfIzIMlMMKRVA7zKKuzYpl/d sUXY+BQlLcGg5VnoCw5e/MCAuGjOcUBf5yS8uSSpEK5NneNXXCR4KEEQ/RCgLpy9oDk0 JiKd+PreJo8jh4u31P1wy1fzC6tVaxSV5BbD5ouSUvUefqQ1zKwLUBE+sp5Lu+tDOar6 LM4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Alip3GSi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f10-20020a0564021e8a00b0045d29efdb5esi10216235edf.381.2022.10.17.14.33.13; Mon, 17 Oct 2022 14:33:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Alip3GSi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230282AbiJQVOx (ORCPT + 99 others); Mon, 17 Oct 2022 17:14:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230107AbiJQVOp (ORCPT ); Mon, 17 Oct 2022 17:14:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5534718394 for ; Mon, 17 Oct 2022 14:14:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666041280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+XFNqhe/9TyVt9WobmDUsN79a+94gmZaId1xsUvIpII=; b=Alip3GSijY6mb6XN91sTWXw/TyMxJTAYRB1BtnnXvkzZ+tVPEkwyJ0/psI4BZC0hfRh8DC T/f7XFKlrsqnDVRCabfPquF2fSFdbXjjtB4kUuI5Tmtk8VS89FgC6tz5M5jvo4NVvoRPWV vyBycyjWflYEUB+FLnSE1GLylClF+Tg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-651-OigZaHQSMiuMzwfrjZ3cYw-1; Mon, 17 Oct 2022 17:14:37 -0400 X-MC-Unique: OigZaHQSMiuMzwfrjZ3cYw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 275C0811E67; Mon, 17 Oct 2022 21:14:37 +0000 (UTC) Received: from llong.com (unknown [10.22.33.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id B8B9E40C206B; Mon, 17 Oct 2022 21:14:36 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng Cc: linux-kernel@vger.kernel.org, john.p.donnelly@oracle.com, Hillf Danton , Mukesh Ojha , =?utf-8?b?VGluZzExIFdhbmcg546L5am3?= , Waiman Long Subject: [PATCH v3 3/5] locking/rwsem: Change waiter->hanodff_set to a handoff_state enum Date: Mon, 17 Oct 2022 17:13:54 -0400 Message-Id: <20221017211356.333862-4-longman@redhat.com> In-Reply-To: <20221017211356.333862-1-longman@redhat.com> References: <20221017211356.333862-1-longman@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746972094910330118?= X-GMAIL-MSGID: =?utf-8?q?1746972094910330118?= Change the boolean waiter->handoff_set to an enum type so that we can have more states in some later patches. Also use READ_ONCE() outside wait_lock critical sections for read and WRITE_ONCE() inside wait_lock critical sections for write for proper synchronization. There is no functional change. Signed-off-by: Waiman Long --- kernel/locking/rwsem.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index c68d76fc8c68..a8bfc905637a 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -338,12 +338,17 @@ enum rwsem_waiter_type { RWSEM_WAITING_FOR_READ }; +enum rwsem_handoff_state { + HANDOFF_NONE = 0, + HANDOFF_REQUESTED, +}; + struct rwsem_waiter { struct list_head list; struct task_struct *task; enum rwsem_waiter_type type; + enum rwsem_handoff_state handoff_state; unsigned long timeout; - bool handoff_set; }; #define rwsem_first_waiter(sem) \ list_first_entry(&sem->wait_list, struct rwsem_waiter, list) @@ -470,7 +475,7 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, adjustment -= RWSEM_FLAG_HANDOFF; lockevent_inc(rwsem_rlock_handoff); } - waiter->handoff_set = true; + WRITE_ONCE(waiter->handoff_state, HANDOFF_REQUESTED); } atomic_long_add(-adjustment, &sem->count); @@ -622,7 +627,7 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, * waiter is the one that set it. Otherwisee, we * still try to acquire the rwsem. */ - if (first->handoff_set && (waiter != first)) + if (first->handoff_state && (waiter != first)) return false; } @@ -650,11 +655,11 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, /* * We have either acquired the lock with handoff bit cleared or set - * the handoff bit. Only the first waiter can have its handoff_set + * the handoff bit. Only the first waiter can have its handoff_state * set here to enable optimistic spinning in slowpath loop. */ if (new & RWSEM_FLAG_HANDOFF) { - first->handoff_set = true; + WRITE_ONCE(first->handoff_state, HANDOFF_REQUESTED); lockevent_inc(rwsem_wlock_handoff); return false; } @@ -1043,7 +1048,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat waiter.task = current; waiter.type = RWSEM_WAITING_FOR_READ; waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT; - waiter.handoff_set = false; + waiter.handoff_state = HANDOFF_NONE; raw_spin_lock_irq(&sem->wait_lock); if (list_empty(&sem->wait_list)) { @@ -1131,7 +1136,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) waiter.task = current; waiter.type = RWSEM_WAITING_FOR_WRITE; waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT; - waiter.handoff_set = false; + waiter.handoff_state = HANDOFF_NONE; raw_spin_lock_irq(&sem->wait_lock); rwsem_add_waiter(sem, &waiter); @@ -1176,7 +1181,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) * In this case, we attempt to acquire the lock again * without sleeping. */ - if (waiter.handoff_set) { + if (READ_ONCE(waiter.handoff_state)) { enum owner_state owner_state; preempt_disable(); From patchwork Mon Oct 17 21:13:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 3752 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1654972wrs; Mon, 17 Oct 2022 14:46:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5zciRJKSgpG0D715NCtfXIY6s5ucRDvOJumZgKtd20vnsmweQVYQp2v8LOyTU1bqG1CKdZ X-Received: by 2002:a17:906:9b93:b0:78d:eb36:1ce7 with SMTP id dd19-20020a1709069b9300b0078deb361ce7mr10481880ejc.621.1666043194439; Mon, 17 Oct 2022 14:46:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666043194; cv=none; d=google.com; s=arc-20160816; b=umdhBTIDJxV0qmI+Ssut5lSgI/9NjE/87Qg0tP8g9CYm2oIV7GnDvavvcuR8/y3uEc YQj9atMKpLmMHhgX1IujTvn5DrHgn0kXKQW+O0sfEvvCY1uXYdAn7+4jFe22k77iw3T0 whv8xxaoyPJvkbJa0Te+f8WpNHmdtJmJA9cBkJ7aedvpTpxSWmuqThJKAentMpKSTDRR TCovPNOn2nefgvkurWASqa3YIS8xK0yE95lp+SA/GfnqSbmGVkyh2MeS9Puer9hL2/kD oqrmP04uREwnVotWd1o+sFBcGzgQd6/mNsgU4AbQTW4CMnIKGQyRmYGj7aqiv/BoVnh4 8qjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dj+K0QnmZOTaKDE9gNQSNFlKIJ5weGyKvZBo3KJQByY=; b=OV92l33A0DDNOFVxuYsZQi52PzImna8W044VrJPvXzE4Nuq0EPJYSurOP09S1sMVgx t87RcfLn++12VabmZYsZGwmyx3Ez35QCMPA/g/MzqyNgzI7g1EJvBdMm1QRHcbikCrJJ THungvkBXz+CDe0KpCETBDxiUii6y7HjsHPDqDvPeXjpOjA3Kjo9ahLVXaqwM3U1LFvd zdaq/1aKmVzbKfm3mw2WBIPfa5N+WWuvHdAFBo7wey9lhW23C7WJ1/an0yPE4ZwtuRr5 reUQ+yr5j/43+deZGKXST1JABebYG0oHsxu+IWMCCId1IUeIxuJH7gox/z0Hi5xCq+jo aR4A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="FqS4l/GY"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y11-20020a056402358b00b0045d22bc81a0si9642858edc.231.2022.10.17.14.46.09; Mon, 17 Oct 2022 14:46:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="FqS4l/GY"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230115AbiJQVPO (ORCPT + 99 others); Mon, 17 Oct 2022 17:15:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229520AbiJQVOq (ORCPT ); Mon, 17 Oct 2022 17:14:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5571818B09 for ; Mon, 17 Oct 2022 14:14:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666041281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dj+K0QnmZOTaKDE9gNQSNFlKIJ5weGyKvZBo3KJQByY=; b=FqS4l/GYf3xXLm0UDdMgY0X3nAj66jqkFEnldcMmyckzt99g5RlUO+8xTKUMm+V9xPlt5M aEfLOyqzczi+hqc6lexkhorcOq+3/KPDarBJcte7Rk/u7XYtKoBaVFKj0AN600EJl1h1RG OBJY4xAafZViaidje4iwW5CGvMjHxoI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-613-S1Sg4_zePQGqnwmuxf3Xiw-1; Mon, 17 Oct 2022 17:14:38 -0400 X-MC-Unique: S1Sg4_zePQGqnwmuxf3Xiw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 95DB73C025C8; Mon, 17 Oct 2022 21:14:37 +0000 (UTC) Received: from llong.com (unknown [10.22.33.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3472340C206B; Mon, 17 Oct 2022 21:14:37 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng Cc: linux-kernel@vger.kernel.org, john.p.donnelly@oracle.com, Hillf Danton , Mukesh Ojha , =?utf-8?b?VGluZzExIFdhbmcg546L5am3?= , Waiman Long Subject: [PATCH v3 4/5] locking/rwsem: Enable direct rwsem lock handoff Date: Mon, 17 Oct 2022 17:13:55 -0400 Message-Id: <20221017211356.333862-5-longman@redhat.com> In-Reply-To: <20221017211356.333862-1-longman@redhat.com> References: <20221017211356.333862-1-longman@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746972909091761415?= X-GMAIL-MSGID: =?utf-8?q?1746972909091761415?= The lock handoff provided in rwsem isn't a true handoff like that in the mutex. Instead, it is more like a quiescent state where optimistic spinning and lock stealing are disabled to make it easier for the first waiter to acquire the lock. Reworking the code to enable a true lock handoff is more complex due to the following facts: 1) The RWSEM_FLAG_HANDOFF bit is protected by the wait_lock and it is too expensive to always take the wait_lock in the unlock path to prevent racing. 2) The reader lock fast path may add a RWSEM_READER_BIAS at the wrong time to prevent a proper lock handoff from a reader owned rwsem. A lock handoff can only be initiated when the following conditions are true: 1) The RWSEM_FLAG_HANDOFF bit is set. 2) The task to do the handoff don't see any other active lock excluding the lock that it might have held. The new handoff mechanism performs handoff in rwsem_wakeup() to minimize overhead. The rwsem count will be known at that point to determine if handoff should be done. However, there is a small time gap between the rwsem becomes free and the wait_lock is taken where a reader can come in and add a RWSEM_READER_BIAS to the count or the current first waiter can take the rwsem and clear RWSEM_FLAG_HANDOFF in the interim. That will fail the handoff operation. To handle the former case, a secondary handoff will also be done in the rwsem_down_read_slowpath() to catch it. With true lock handoff, there is no need to do a NULL owner spinning anymore as wakeup will be performed if handoff is possible. So it is likely that the first waiter won't actually go to sleep even when schedule() is called in this case. Signed-off-by: Waiman Long --- kernel/locking/rwsem.c | 141 +++++++++++++++++++++++++++++++---------- 1 file changed, 109 insertions(+), 32 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index a8bfc905637a..287606aee0e6 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -341,6 +341,7 @@ enum rwsem_waiter_type { enum rwsem_handoff_state { HANDOFF_NONE = 0, HANDOFF_REQUESTED, + HANDOFF_GRANTED, }; struct rwsem_waiter { @@ -489,6 +490,12 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, */ owner = waiter->task; __rwsem_set_reader_owned(sem, owner); + } else if (waiter->handoff_state == HANDOFF_GRANTED) { + /* + * rwsem_handoff() has added to count RWSEM_READER_BIAS of + * the first waiter. + */ + adjustment = RWSEM_READER_BIAS; } /* @@ -586,7 +593,7 @@ rwsem_del_wake_waiter(struct rw_semaphore *sem, struct rwsem_waiter *waiter, struct wake_q_head *wake_q) __releases(&sem->wait_lock) { - bool first = rwsem_first_waiter(sem) == waiter; + struct rwsem_waiter *first = rwsem_first_waiter(sem); wake_q_init(wake_q); @@ -595,8 +602,21 @@ rwsem_del_wake_waiter(struct rw_semaphore *sem, struct rwsem_waiter *waiter, * the first waiter, we wake up the remaining waiters as they may * be eligible to acquire or spin on the lock. */ - if (rwsem_del_waiter(sem, waiter) && first) + if (rwsem_del_waiter(sem, waiter) && (waiter == first)) { + switch (waiter->handoff_state) { + case HANDOFF_GRANTED: + raw_spin_unlock_irq(&sem->wait_lock); + return; + case HANDOFF_REQUESTED: + /* Pass handoff state to the new first waiter */ + first = rwsem_first_waiter(sem); + WRITE_ONCE(first->handoff_state, HANDOFF_REQUESTED); + fallthrough; + default: + break; + } rwsem_mark_wake(sem, RWSEM_WAKE_ANY, wake_q); + } raw_spin_unlock_irq(&sem->wait_lock); if (!wake_q_empty(wake_q)) wake_up_q(wake_q); @@ -764,6 +784,11 @@ rwsem_spin_on_owner(struct rw_semaphore *sem) owner = rwsem_owner_flags(sem, &flags); state = rwsem_owner_state(owner, flags); + + /* A handoff may have been granted */ + if (!flags && (owner == current)) + return OWNER_NONSPINNABLE; + if (state != OWNER_WRITER) return state; @@ -977,6 +1002,32 @@ rwsem_spin_on_owner(struct rw_semaphore *sem) } #endif +/* + * Hand off the lock to the first waiter + */ +static void rwsem_handoff(struct rw_semaphore *sem, long adj, + struct wake_q_head *wake_q) +{ + struct rwsem_waiter *waiter; + enum rwsem_wake_type wake_type; + + lockdep_assert_held(&sem->wait_lock); + adj -= RWSEM_FLAG_HANDOFF; + waiter = rwsem_first_waiter(sem); + WRITE_ONCE(waiter->handoff_state, HANDOFF_GRANTED); + if (waiter->type == RWSEM_WAITING_FOR_WRITE) { + wake_type = RWSEM_WAKE_ANY; + adj += RWSEM_WRITER_LOCKED; + atomic_long_set(&sem->owner, (long)waiter->task); + } else { + wake_type = RWSEM_WAKE_READ_OWNED; + adj += RWSEM_READER_BIAS; + __rwsem_set_reader_owned(sem, waiter->task); + } + atomic_long_add(adj, &sem->count); + rwsem_mark_wake(sem, wake_type, wake_q); +} + /* * Prepare to wake up waiter(s) in the wait queue by putting them into the * given wake_q if the rwsem lock owner isn't a writer. If rwsem is likely @@ -1051,6 +1102,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat waiter.handoff_state = HANDOFF_NONE; raw_spin_lock_irq(&sem->wait_lock); + count = atomic_long_read(&sem->count); if (list_empty(&sem->wait_list)) { /* * In case the wait queue is empty and the lock isn't owned @@ -1058,7 +1110,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat * immediately as its RWSEM_READER_BIAS has already been set * in the count. */ - if (!(atomic_long_read(&sem->count) & RWSEM_WRITER_MASK)) { + if (!(count & RWSEM_WRITER_MASK)) { /* Provide lock ACQUIRE */ smp_acquire__after_ctrl_dep(); raw_spin_unlock_irq(&sem->wait_lock); @@ -1067,13 +1119,33 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat return sem; } adjustment += RWSEM_FLAG_WAITERS; + } else if ((count & RWSEM_FLAG_HANDOFF) && + ((count & RWSEM_LOCK_MASK) == RWSEM_READER_BIAS)) { + /* + * If the waiter to be handed off is a reader, this reader + * can piggyback on top of top of that. + */ + if (rwsem_first_waiter(sem)->type == RWSEM_WAITING_FOR_READ) + adjustment = 0; + rwsem_handoff(sem, adjustment, &wake_q); + + if (!adjustment) { + raw_spin_unlock_irq(&sem->wait_lock); + wake_up_q(&wake_q); + return sem; + } + adjustment = 0; } rwsem_add_waiter(sem, &waiter); - /* we're now waiting on the lock, but no longer actively locking */ - count = atomic_long_add_return(adjustment, &sem->count); - - rwsem_cond_wake_waiter(sem, count, &wake_q); + if (adjustment) { + /* + * We are now waiting on the lock with no handoff, but no + * longer actively locking. + */ + count = atomic_long_add_return(adjustment, &sem->count); + rwsem_cond_wake_waiter(sem, count, &wake_q); + } raw_spin_unlock_irq(&sem->wait_lock); if (!wake_q_empty(&wake_q)) @@ -1120,7 +1192,6 @@ static struct rw_semaphore __sched * rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) { struct rwsem_waiter waiter; - int null_owner_retries; DEFINE_WAKE_Q(wake_q); /* do optimistic spinning and steal lock if possible */ @@ -1162,7 +1233,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) set_current_state(state); trace_contention_begin(sem, LCB_F_WRITE); - for (null_owner_retries = 0;;) { + for (;;) { if (rwsem_try_write_lock(sem, &waiter)) { /* rwsem_try_write_lock() implies ACQUIRE on success */ break; @@ -1182,37 +1253,28 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) * without sleeping. */ if (READ_ONCE(waiter.handoff_state)) { - enum owner_state owner_state; - - preempt_disable(); - owner_state = rwsem_spin_on_owner(sem); - preempt_enable(); - - /* - * owner is NULL doesn't guarantee the lock is free. - * An incoming reader will temporarily increment the - * reader count without changing owner and the - * rwsem_try_write_lock() will fails if the reader - * is not able to decrement it in time. Allow 8 - * trylock attempts when hitting a NULL owner before - * going to sleep. - */ - if ((owner_state == OWNER_NULL) && - (null_owner_retries < 8)) { - null_owner_retries++; - goto trylock_again; + if (READ_ONCE(waiter.handoff_state) == HANDOFF_REQUESTED) { + preempt_disable(); + rwsem_spin_on_owner(sem); + preempt_enable(); } - null_owner_retries = 0; + if (READ_ONCE(waiter.handoff_state) == HANDOFF_GRANTED) + goto skip_sleep; } schedule(); lockevent_inc(rwsem_sleep_writer); set_current_state(state); -trylock_again: +skip_sleep: raw_spin_lock_irq(&sem->wait_lock); + if (waiter.handoff_state == HANDOFF_GRANTED) { + rwsem_del_waiter(sem, &waiter); + break; + } } __set_current_state(TASK_RUNNING); raw_spin_unlock_irq(&sem->wait_lock); +out_lock: lockevent_inc(rwsem_wlock); trace_contention_end(sem, 0); return sem; @@ -1221,6 +1283,9 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) __set_current_state(TASK_RUNNING); raw_spin_lock_irq(&sem->wait_lock); rwsem_del_wake_waiter(sem, &waiter, &wake_q); + if (unlikely(READ_ONCE(waiter.handoff_state) == HANDOFF_GRANTED)) + goto out_lock; + lockevent_inc(rwsem_wlock_fail); trace_contention_end(sem, -EINTR); return ERR_PTR(-EINTR); @@ -1232,12 +1297,24 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) */ static struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) { - unsigned long flags; DEFINE_WAKE_Q(wake_q); + unsigned long flags; + long count; raw_spin_lock_irqsave(&sem->wait_lock, flags); - if (!list_empty(&sem->wait_list)) + if (list_empty(&sem->wait_list)) { + raw_spin_unlock_irqrestore(&sem->wait_lock, flags); + return sem; + } + /* + * If the rwsem is free and handoff flag is set with wait_lock held, + * no other CPUs can take an active lock. + */ + count = atomic_long_read(&sem->count); + if (!(count & RWSEM_LOCK_MASK) && (count & RWSEM_FLAG_HANDOFF)) + rwsem_handoff(sem, 0, &wake_q); + else rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q); raw_spin_unlock_irqrestore(&sem->wait_lock, flags); From patchwork Mon Oct 17 21:13:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 3747 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1648947wrs; Mon, 17 Oct 2022 14:28:21 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7XGoKBWE5qKw6A9hb9qdlE+MqzT4T2IMFg/ICyIqN2kIarwSceQ/v3CXYNDVyYHe9h/bS2 X-Received: by 2002:a17:902:b598:b0:184:e49d:3042 with SMTP id a24-20020a170902b59800b00184e49d3042mr14168062pls.16.1666042101181; Mon, 17 Oct 2022 14:28:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666042101; cv=none; d=google.com; s=arc-20160816; b=LTPm5C1gpVV8GRoUOn6un1JRpYKVa0+Y6uJE+gYwy2Ld4r+iQs7PkcrR5MCalwg0wf zQv1HUjbgih3DfvhVOFc/ghl3yIu/HRh2JKsfYwEYqhewuTMzRC19hifnd65THSJCXZb RL8b88TB9GK/Ehcdrp7AqdGHTrKg/MdHn2Diiw3W+UBRQ9oc6rVoEzhXq5/p7OsHQ9FU PgXouj/Qyjexhr0n4FFGRIaOfI9eeOFxW2UK3Y0ByFfyj3w0P1xQnrrVtPtkkTK2xnzb OBTCE6zoNV6oQ2Fj4CAd6t8PhcgeQKV0224YIfTbMO1oppuAuKJrM1ARyX8tHi8U/AB1 KBGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fCvbSgxKkVngETIOIh3AmCdsye2c6FCNzZDR+LWaBok=; b=ru3nYpC46ltgRCd853T/upccVpQfho72hcUJJf5WP2xuH+In43lCwsy9syuDk6+aep /SgrFOKrvjEk4R03mM7FQjzxkoZ352qAVoO7W4flqVp5GxpSL/LjIku7J81KRxC2we8l NrCT4nCnEtiqA2hj85DidsMA/MLVYkSA1mP6Uuclar823uBo/w8gUTn3fFHIrepsAfak XNplqmz9msdpbZplA2E5zLuonZb/3wOWJSbBNKy0jmD7tW7GCo3EvHikElHaseWItyME CV4+wcmKWsg+xuvO87+j4xn5nkvOuCZ2XYw1DNIhYkpQM+Ph2M8wLNmr18wGhf//dd0/ sdxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UMfZL2Qi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y3-20020a1709029b8300b0017c2a42fc8csi11614485plp.270.2022.10.17.14.28.07; Mon, 17 Oct 2022 14:28:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UMfZL2Qi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230046AbiJQVPK (ORCPT + 99 others); Mon, 17 Oct 2022 17:15:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230048AbiJQVOq (ORCPT ); Mon, 17 Oct 2022 17:14:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2381218B21 for ; Mon, 17 Oct 2022 14:14:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666041282; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fCvbSgxKkVngETIOIh3AmCdsye2c6FCNzZDR+LWaBok=; b=UMfZL2Qi/lymbGAOceqPy6NnN2NAa8lr+VotqEqS88En9eNaKG/TQtUk7YdutZZS8x7UE1 MRj8QFY2CVpIJ1xW0cbyzgHm1JpxwMHoqbkuLZdzKFsxy+4ZGryhlm0lEB/xXL50V+0sRn b3FPFQ5rwM0EJBXSBW1+WydaJsJgMv0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-661-r6imPVvxMNqheLdEQw-wFg-1; Mon, 17 Oct 2022 17:14:38 -0400 X-MC-Unique: r6imPVvxMNqheLdEQw-wFg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0DD0C86F12F; Mon, 17 Oct 2022 21:14:38 +0000 (UTC) Received: from llong.com (unknown [10.22.33.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id A325240C206B; Mon, 17 Oct 2022 21:14:37 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng Cc: linux-kernel@vger.kernel.org, john.p.donnelly@oracle.com, Hillf Danton , Mukesh Ojha , =?utf-8?b?VGluZzExIFdhbmcg546L5am3?= , Waiman Long Subject: [PATCH v3 5/5] locking/rwsem: Update handoff lock events tracking Date: Mon, 17 Oct 2022 17:13:56 -0400 Message-Id: <20221017211356.333862-6-longman@redhat.com> In-Reply-To: <20221017211356.333862-1-longman@redhat.com> References: <20221017211356.333862-1-longman@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746971762197315855?= X-GMAIL-MSGID: =?utf-8?q?1746971762197315855?= With the new direct rwsem lock handoff, the corresponding handoff lock events are updated to also track the number of secondary lock handoffs in rwsem_down_read_slowpath() to see how prevalent those handoff events are. The number of primary lock handoffs in the unlock paths is (rwsem_handoff_read + rwsem_handoff_write - rwsem_handoff_rslow). After running a 96-thread rwsem microbenchmark with equal number of readers and writers on a 2-socket 96-thread system for 40s, the following handoff stats were obtained: rwsem_handoff_read=189 rwsem_handoff_rslow=1 rwsem_handoff_write=6678 rwsem_handoff_wspin=6681 The number of primary handoffs was 6866, whereas there was only one secondary handoff for this test run. Signed-off-by: Waiman Long --- kernel/locking/lock_events_list.h | 6 ++++-- kernel/locking/rwsem.c | 9 +++++---- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/kernel/locking/lock_events_list.h b/kernel/locking/lock_events_list.h index 97fb6f3f840a..04d101767c2c 100644 --- a/kernel/locking/lock_events_list.h +++ b/kernel/locking/lock_events_list.h @@ -63,7 +63,9 @@ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ LOCK_EVENT(rwsem_rlock_steal) /* # of read locks by lock stealing */ LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */ -LOCK_EVENT(rwsem_rlock_handoff) /* # of read lock handoffs */ LOCK_EVENT(rwsem_wlock) /* # of write locks acquired */ LOCK_EVENT(rwsem_wlock_fail) /* # of failed write lock acquisitions */ -LOCK_EVENT(rwsem_wlock_handoff) /* # of write lock handoffs */ +LOCK_EVENT(rwsem_handoff_read) /* # of read lock handoffs */ +LOCK_EVENT(rwsem_handoff_write) /* # of write lock handoffs */ +LOCK_EVENT(rwsem_handoff_rslow) /* # of handoffs in read slowpath */ +LOCK_EVENT(rwsem_handoff_wspin) /* # of handoff spins in write slowpath */ diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index 287606aee0e6..46aea1994bf8 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -472,10 +472,8 @@ static void rwsem_mark_wake(struct rw_semaphore *sem, * force the issue. */ if (time_after(jiffies, waiter->timeout)) { - if (!(oldcount & RWSEM_FLAG_HANDOFF)) { + if (!(oldcount & RWSEM_FLAG_HANDOFF)) adjustment -= RWSEM_FLAG_HANDOFF; - lockevent_inc(rwsem_rlock_handoff); - } WRITE_ONCE(waiter->handoff_state, HANDOFF_REQUESTED); } @@ -680,7 +678,6 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, */ if (new & RWSEM_FLAG_HANDOFF) { WRITE_ONCE(first->handoff_state, HANDOFF_REQUESTED); - lockevent_inc(rwsem_wlock_handoff); return false; } @@ -1019,10 +1016,12 @@ static void rwsem_handoff(struct rw_semaphore *sem, long adj, wake_type = RWSEM_WAKE_ANY; adj += RWSEM_WRITER_LOCKED; atomic_long_set(&sem->owner, (long)waiter->task); + lockevent_inc(rwsem_handoff_write); } else { wake_type = RWSEM_WAKE_READ_OWNED; adj += RWSEM_READER_BIAS; __rwsem_set_reader_owned(sem, waiter->task); + lockevent_inc(rwsem_handoff_read); } atomic_long_add(adj, &sem->count); rwsem_mark_wake(sem, wake_type, wake_q); @@ -1128,6 +1127,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat if (rwsem_first_waiter(sem)->type == RWSEM_WAITING_FOR_READ) adjustment = 0; rwsem_handoff(sem, adjustment, &wake_q); + lockevent_inc(rwsem_handoff_rslow); if (!adjustment) { raw_spin_unlock_irq(&sem->wait_lock); @@ -1257,6 +1257,7 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) preempt_disable(); rwsem_spin_on_owner(sem); preempt_enable(); + lockevent_inc(rwsem_handoff_wspin); } if (READ_ONCE(waiter.handoff_state) == HANDOFF_GRANTED) goto skip_sleep;