From patchwork Thu Jan 26 11:38:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 48604 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp225099wrn; Thu, 26 Jan 2023 03:41:31 -0800 (PST) X-Google-Smtp-Source: AMrXdXs0yWWUhjZbJhaXiZgkr9bv6GNvIwrLQjabLejbEDTpgJEH3ZW8Em9oi+CcyADDFB0tUsju X-Received: by 2002:a17:907:cf48:b0:86b:914a:571 with SMTP id uv8-20020a170907cf4800b0086b914a0571mr37279140ejc.5.1674733291548; Thu, 26 Jan 2023 03:41:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674733291; cv=none; d=google.com; s=arc-20160816; b=js9nUJuGqftBKC4CbjbsnAv/nLTxNIBTX1EG27iWJR6YvCtjnxtZ4IDwzQrUPm5tfW Ex21gC1++3Q6E4uDGVp3VsRAKtg8ivLk6XG0iPbdP9yt7RLR1H+7u1Msjvqs5ai9tsvV CxSbffhrHFp7n80Y8z9uhOzArzHuYwNJJq0AxRTlQAoHh8Iry4KLd1/Ho1HpznK5E8Qq BMyZcMIvKvviNBXBAySxaPRapi/kk+l46BdG0jPgliICPDmKSkM/BbsoL6261jkJ9Rj5 Fvk9/hzmBkYEtf4ZYMWvECV714WNERKz0BHXMbI7G16r7VXH5UFWqVUwMa+SPdOktS3+ CGJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=UxJDEu63Ijg3+35/HItpaVY6j0/2RXO+OkssGdF1C3Y=; b=cVinBhdCf8JLl76QffM33zcMePOaAIzZNqlX68a/AoG3kQcBKe189x+RPWgHTHuA3p yR4zo/pZR55IPL9iHbOhmob/sYrYJdEaG0jCMyEiHWlHHLejDdFVtA8r0GanQAFq7ieA fwm2k5LYXmqBDZZknnWssWze+GOsEniJF3zqQq3Vd6Jdti583S6CEuDK42Frxr7VtgLt Sz/AQk14M8BdVTl5DvkUCIV1kzgIWB324dt/EEtfOC51Ls/UxN17SHJxDLKrX+O8fgay 24DC/lcay2V2IdzJDzFCjPYSZk6CEX7Z4P66rYr7NjUY9DLqhDRzwCgVaMfOb9FRcOy+ VYSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="c/mOJxWC"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GDx8aomo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wj15-20020a170907050f00b00878543ce0e3si989449ejb.364.2023.01.26.03.41.07; Thu, 26 Jan 2023 03:41:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b="c/mOJxWC"; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=GDx8aomo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237337AbjAZLi7 (ORCPT + 99 others); Thu, 26 Jan 2023 06:38:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237307AbjAZLis (ORCPT ); Thu, 26 Jan 2023 06:38:48 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B1043E095; Thu, 26 Jan 2023 03:38:47 -0800 (PST) Date: Thu, 26 Jan 2023 11:38:45 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1674733126; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UxJDEu63Ijg3+35/HItpaVY6j0/2RXO+OkssGdF1C3Y=; b=c/mOJxWCoUpJg9oohAhNcAFZi9hFkxYlhSwEZefLcSfN1rlqSmK4REy4jT8TQyynWdptTT NkLh8RaI2ovvRgxn9bYgc+YjVKo3apc3ttAC4LdpmB6o1PiTWd6rFnjoSl5iIPjkJXFJlq ZPiS0s55876kxbu6I4ORwqK77YYHV2XpCyxS0vhM7Lbp/VxtTulBRcrRgFQK61nHhAAbtH L6WVUliUY4C+LKJ4TyYeuNoxtUeKYWCUfeTmuqMauiJeTchqkqzT7iwK1+hySekEWxRi/+ ayMAVJziVZWAk48ZzjkTxhbBA+Q3Vtt34pOgu8fmCXj177JGI2Cf/6bYbmP6tQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1674733126; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UxJDEu63Ijg3+35/HItpaVY6j0/2RXO+OkssGdF1C3Y=; b=GDx8aomo2ahGQ/EWR5iBq0/On2IzF3hZ8BCjgbSd7csPC2HtzcdIAQ/qTRipsnGEz6mxiB UaK3xPq0va+jxyBg== From: "tip-bot2 for Waiman Long" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] locking/rwsem: Prevent non-first waiter from spinning in down_write() slowpath Cc: Waiman Long , Ingo Molnar , Mukesh Ojha , stable@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230126003628.365092-2-longman@redhat.com> References: <20230126003628.365092-2-longman@redhat.com> MIME-Version: 1.0 Message-ID: <167473312559.4906.10064248025918048573.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756044291357203590?= X-GMAIL-MSGID: =?utf-8?q?1756085136255459134?= The following commit has been merged into the locking/core branch of tip: Commit-ID: b613c7f31476c44316bfac1af7cac714b7d6bef9 Gitweb: https://git.kernel.org/tip/b613c7f31476c44316bfac1af7cac714b7d6bef9 Author: Waiman Long AuthorDate: Wed, 25 Jan 2023 19:36:25 -05:00 Committer: Ingo Molnar CommitterDate: Thu, 26 Jan 2023 11:46:45 +01:00 locking/rwsem: Prevent non-first waiter from spinning in down_write() slowpath A non-first waiter can potentially spin in the for loop of rwsem_down_write_slowpath() without sleeping but fail to acquire the lock even if the rwsem is free if the following sequence happens: Non-first RT waiter First waiter Lock holder ------------------- ------------ ----------- Acquire wait_lock rwsem_try_write_lock(): Set handoff bit if RT or wait too long Set waiter->handoff_set Release wait_lock Acquire wait_lock Inherit waiter->handoff_set Release wait_lock Clear owner Release lock if (waiter.handoff_set) { rwsem_spin_on_owner((); if (OWNER_NULL) goto trylock_again; } trylock_again: Acquire wait_lock rwsem_try_write_lock(): if (first->handoff_set && (waiter != first)) return false; Release wait_lock A non-first waiter cannot really acquire the rwsem even if it mistakenly believes that it can spin on OWNER_NULL value. If that waiter happens to be an RT task running on the same CPU as the first waiter, it can block the first waiter from acquiring the rwsem leading to live lock. Fix this problem by making sure that a non-first waiter cannot spin in the slowpath loop without sleeping. Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more consistent") Signed-off-by: Waiman Long Signed-off-by: Ingo Molnar Tested-by: Mukesh Ojha Reviewed-by: Mukesh Ojha Cc: stable@vger.kernel.org Link: https://lore.kernel.org/r/20230126003628.365092-2-longman@redhat.com --- kernel/locking/rwsem.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index 4487359..be2df9e 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -624,18 +624,16 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, */ if (first->handoff_set && (waiter != first)) return false; - - /* - * First waiter can inherit a previously set handoff - * bit and spin on rwsem if lock acquisition fails. - */ - if (waiter == first) - waiter->handoff_set = true; } new = count; if (count & RWSEM_LOCK_MASK) { + /* + * A waiter (first or not) can set the handoff bit + * if it is an RT task or wait in the wait queue + * for too long. + */ if (has_handoff || (!rt_task(waiter->task) && !time_after(jiffies, waiter->timeout))) return false; @@ -651,11 +649,12 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem, } while (!atomic_long_try_cmpxchg_acquire(&sem->count, &count, new)); /* - * We have either acquired the lock with handoff bit cleared or - * set the handoff bit. + * We have either acquired the lock with handoff bit cleared or set + * the handoff bit. Only the first waiter can have its handoff_set + * set here to enable optimistic spinning in slowpath loop. */ if (new & RWSEM_FLAG_HANDOFF) { - waiter->handoff_set = true; + first->handoff_set = true; lockevent_inc(rwsem_wlock_handoff); return false; }