From patchwork Sat Feb 10 00:23:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 199168 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1215964dyd; Fri, 9 Feb 2024 16:29:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IHX4s4NlMeaXJkInrKTA/+CtDagkPfbva66HfdeJqQbvaS1C0kXxCQKliovMFRpI3SwJLzf X-Received: by 2002:a05:6808:2382:b0:3bf:f4bb:7630 with SMTP id bp2-20020a056808238200b003bff4bb7630mr769223oib.45.1707524966706; Fri, 09 Feb 2024 16:29:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707524966; cv=pass; d=google.com; s=arc-20160816; b=NRJF+jR3ohGCZJrRZjgpwS4Jws7sOjS2x3vZzBcb2KWD+9PnjOGYEyc324t0WmwCPT r8IUHScCvYrFpQ1QvnIU7ogvg/mrNFi1AVTG345R/7lxLMQZcm2sZ32FHgcxyK/sLdWq bOKL9tMCnD+36Q5x5/Xx+TkCixjLTJm5VzujRGLc4LugKdhZPeF8LnzHvllNF1k/3v3R rPVZG3jmSI3ukhh+FxZtZ6g3O7YFA9+eLSZ00m99wCm4inStYMiI5r97ULBgv2UW+cq9 Thz8amrrkeX02R0xj5MKYO4o7mG9XwH4yXEBq+ootjHnPMJYdVTRKdFEqBfi2MJhAmfJ RIsw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=XAy3C0x/5qzXDxILuLg04WzqVd2kbP9Lyu9Y6MgUWYE=; fh=Hxcs6laAEn+xRs6Ww7fwmUqLehFT/mdk2eFXHGwn8O0=; b=oZfdGXiKKz/sEDVEp7YNtGRfWmggcQ6VGVSLNcn9+wYtZtO0WpTkU5xwjQMn3HrBvA T4jiK3Yy/T0YRnul/pdumMCLWmWXRndawZqAkI6VeWiNAsWCvM3kSZH9ABd8vNRAIByf LKQjBY6TCnaJhAVrHFcm6yUNQpmsF9Vi6V5FBwhv2Tpjn09QM3oFSgOuEy9MnObgDD5L tgFfJ3YcEIKA7FeiJqOl0p2f61xfjxvKlI1zmyQICi4fqcWspi+yp43iaD/+ZaspsQip 49zj+wJuW8z17naEFXqK0rYRF8rPawH4rKy726kbOYsSGfc5+gkTabHCyLaUrsZXwnLX YgdQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=ayXl80aV; arc=pass (i=1 spf=pass spfdomain=flex--jstultz.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-60180-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60180-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCWux8T8mHNXhVbJ8D1pIrsDsdJ3VrD+YBD6px4Zs+2OqfZKCopMhAzjYvswAxXT3zsWBqiZrNCnWWOt0r1F4+78HoEn5g== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id y184-20020a638ac1000000b005dc3dcc360asi2467133pgd.674.2024.02.09.16.29.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 16:29:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60180-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=ayXl80aV; arc=pass (i=1 spf=pass spfdomain=flex--jstultz.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-60180-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60180-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 6765BB23C1A for ; Sat, 10 Feb 2024 00:24:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0FE1E5234; Sat, 10 Feb 2024 00:23:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ayXl80aV" Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D30BA40 for ; Sat, 10 Feb 2024 00:23:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707524621; cv=none; b=csOp3/KdhcsAZup2t2e0Hzv0GckYMFVDshYvP1TZZvOz/T3K2YhTUVfM0+sy5lt0YiWvU9+YyNnaCOKP9I+j3KOdrhOqfeTlEcmxAQKV27l2xNqcpoNChCCBJ9vnqGqk3tp+O9X9lbATQ0I4+DqKY13NT7q0dgBQj4HfsUpcOKI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707524621; c=relaxed/simple; bh=IRUqYFGJOvMutAvsPxcmasMWOBYCkWCHI8SeBldW878=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pFye0zEgMDkuO1Zc8QVArfi3ddrdeGmsD1x8sdmbKncmJog4Um1h14kg2IgpGrTguLmlfSpng420iGLxH2EcRm9wWEh0s+fsVjdJ132Mx65ULyONLbsXzpYGpnFhbDDRz5U+9rCW4xv1/q9nJgfVOsq0tl8uipYW99uga9t9wXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ayXl80aV; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-5cec090b2bdso1401195a12.0 for ; Fri, 09 Feb 2024 16:23:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707524619; x=1708129419; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XAy3C0x/5qzXDxILuLg04WzqVd2kbP9Lyu9Y6MgUWYE=; b=ayXl80aVsGuFPR9Dcfm6nav31J5xPJwkud6iDgLhW8Y3YfPIFERe/eui3XxhwxaGjW lAly9LDf9RaMMoFcyLeqygzvEn0qwSh16A6Xvr8wWgqFvWgHwdQUeaxvtDIctovefUTR QqpkPRHWi+yjMSeSl2frfrna1D/vHoUYWGGdf7uPqVQ3nsju7h10lSBDPtx0hukOSmo+ daK7gl38m1x8+YF4gXs6H6+INhynUW0IRenACqqMlYWyIqC/dM1hp7h+3kF1xY+ms/d0 evsrARwaB1S/R6W8dV4BbxM/WtkEMq9zqwY0RGVnzGHpfW1FkcMREj0GUjxRiDN5iRg/ v79Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707524619; x=1708129419; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XAy3C0x/5qzXDxILuLg04WzqVd2kbP9Lyu9Y6MgUWYE=; b=sncym+C2a9XCaiFCulcKIaTiOB3MCipng+qkkVyNmOoFKags1AnF/h7laXCWCQ4tkq 7uLSMnzIWjMo8s1J3+TTES+bRmgats7C007dGadKDLhtwG1wxkXnZqYu+4+CCDRF1ugW Up8j5J3FxNzyg9+/5A3hOa3hydXPewOi8ZYag07CyIF3FkukhDqh4/ERH65Lwjls7w6v FudMgKNUtuCY6xJGWdq0LLGbTLTjWxT4jcwXkKHOSwkh/jeuzUp4Ay1PT3uNgM9LGH3Y JqUJxAKQtNAVlxan/oeW2bNvR48PrA323wL/O3DturpUSNl+/F+AphPQj15+Cu4FhzJq cnqQ== X-Gm-Message-State: AOJu0YxSKe9HyXh+x36rf4zHdDAYqawjBKVP3xBajvgEjVAAZVjdYWkM agFcP8bbhHjF8yzo7bFTfNcfp5TpPmA8AjaFB7ZY8hiaZ/Xq/alW+hQZ7kvJ17KLwsgZ59nZHCi fgwpq0hcpzu+B7NYU7gov+0KUtLSrEhXGVc1cCPR79xMe+qNrndzk9lX47qMHnDg4SIAJclJIQu gszQujexYfpG+YYo6yYwypEZU7us2gVABld6HI5r2E/JW5 X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a65:6756:0:b0:5ce:8c6:c370 with SMTP id c22-20020a656756000000b005ce08c6c370mr1418pgu.10.1707524619373; Fri, 09 Feb 2024 16:23:39 -0800 (PST) Date: Fri, 9 Feb 2024 16:23:11 -0800 In-Reply-To: <20240210002328.4126422-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240210002328.4126422-1-jstultz@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240210002328.4126422-3-jstultz@google.com> Subject: [PATCH v8 2/7] locking/mutex: Make mutex::wait_lock irq safe From: John Stultz To: LKML Cc: Juri Lelli , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Youssef Esmat , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , kernel-team@android.com, "Connor O'Brien" , John Stultz X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790469699343395397 X-GMAIL-MSGID: 1790469699343395397 From: Juri Lelli mutex::wait_lock might be nested under rq->lock. Make it irq safe then. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Youssef Esmat Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: Juri Lelli Signed-off-by: Peter Zijlstra (Intel) [rebase & fix {un,}lock_wait_lock helpers in ww_mutex.h] Signed-off-by: Connor O'Brien Signed-off-by: John Stultz --- v3: * Re-added this patch after it was dropped in v2 which caused lockdep warnings to trip. v7: * Fix function definition for PREEMPT_RT case, as pointed out by Metin Kaya. * Fix incorrect flags handling in PREEMPT_RT case as found by Metin Kaya --- kernel/locking/mutex.c | 18 ++++++++++-------- kernel/locking/ww_mutex.h | 22 +++++++++++----------- 2 files changed, 21 insertions(+), 19 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 980ce630232c..7de72c610c65 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -578,6 +578,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas DEFINE_WAKE_Q(wake_q); struct mutex_waiter waiter; struct ww_mutex *ww; + unsigned long flags; int ret; if (!use_ww_ctx) @@ -620,7 +621,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas return 0; } - raw_spin_lock(&lock->wait_lock); + raw_spin_lock_irqsave(&lock->wait_lock, flags); /* * After waiting to acquire the wait_lock, try again. */ @@ -681,7 +682,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err; } - raw_spin_unlock(&lock->wait_lock); + raw_spin_unlock_irqrestore(&lock->wait_lock, flags); /* Make sure we do wakeups before calling schedule */ if (!wake_q_empty(&wake_q)) { wake_up_q(&wake_q); @@ -707,9 +708,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas trace_contention_begin(lock, LCB_F_MUTEX); } - raw_spin_lock(&lock->wait_lock); + raw_spin_lock_irqsave(&lock->wait_lock, flags); } - raw_spin_lock(&lock->wait_lock); + raw_spin_lock_irqsave(&lock->wait_lock, flags); acquired: __set_current_state(TASK_RUNNING); @@ -735,7 +736,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas if (ww_ctx) ww_mutex_lock_acquired(ww, ww_ctx); - raw_spin_unlock(&lock->wait_lock); + raw_spin_unlock_irqrestore(&lock->wait_lock, flags); wake_up_q(&wake_q); preempt_enable(); return 0; @@ -745,7 +746,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas __mutex_remove_waiter(lock, &waiter); err_early_kill: trace_contention_end(lock, ret); - raw_spin_unlock(&lock->wait_lock); + raw_spin_unlock_irqrestore(&lock->wait_lock, flags); debug_mutex_free_waiter(&waiter); mutex_release(&lock->dep_map, ip); wake_up_q(&wake_q); @@ -916,6 +917,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne struct task_struct *next = NULL; DEFINE_WAKE_Q(wake_q); unsigned long owner; + unsigned long flags; mutex_release(&lock->dep_map, ip); @@ -943,7 +945,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne } preempt_disable(); - raw_spin_lock(&lock->wait_lock); + raw_spin_lock_irqsave(&lock->wait_lock, flags); debug_mutex_unlock(lock); if (!list_empty(&lock->wait_list)) { /* get the first entry from the wait-list: */ @@ -960,7 +962,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); - raw_spin_unlock(&lock->wait_lock); + raw_spin_unlock_irqrestore(&lock->wait_lock, flags); wake_up_q(&wake_q); preempt_enable(); } diff --git a/kernel/locking/ww_mutex.h b/kernel/locking/ww_mutex.h index 7189c6631d90..9facc0ddfdd3 100644 --- a/kernel/locking/ww_mutex.h +++ b/kernel/locking/ww_mutex.h @@ -70,14 +70,14 @@ __ww_mutex_has_waiters(struct mutex *lock) return atomic_long_read(&lock->owner) & MUTEX_FLAG_WAITERS; } -static inline void lock_wait_lock(struct mutex *lock) +static inline void lock_wait_lock(struct mutex *lock, unsigned long *flags) { - raw_spin_lock(&lock->wait_lock); + raw_spin_lock_irqsave(&lock->wait_lock, *flags); } -static inline void unlock_wait_lock(struct mutex *lock) +static inline void unlock_wait_lock(struct mutex *lock, unsigned long *flags) { - raw_spin_unlock(&lock->wait_lock); + raw_spin_unlock_irqrestore(&lock->wait_lock, *flags); } static inline void lockdep_assert_wait_lock_held(struct mutex *lock) @@ -144,14 +144,14 @@ __ww_mutex_has_waiters(struct rt_mutex *lock) return rt_mutex_has_waiters(&lock->rtmutex); } -static inline void lock_wait_lock(struct rt_mutex *lock) +static inline void lock_wait_lock(struct rt_mutex *lock, unsigned long *flags) { - raw_spin_lock(&lock->rtmutex.wait_lock); + raw_spin_lock_irqsave(&lock->rtmutex.wait_lock, *flags); } -static inline void unlock_wait_lock(struct rt_mutex *lock) +static inline void unlock_wait_lock(struct rt_mutex *lock, unsigned long *flags) { - raw_spin_unlock(&lock->rtmutex.wait_lock); + raw_spin_unlock_irqrestore(&lock->rtmutex.wait_lock, *flags); } static inline void lockdep_assert_wait_lock_held(struct rt_mutex *lock) @@ -380,6 +380,7 @@ static __always_inline void ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) { DEFINE_WAKE_Q(wake_q); + unsigned long flags; ww_mutex_lock_acquired(lock, ctx); @@ -408,10 +409,9 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) * Uh oh, we raced in fastpath, check if any of the waiters need to * die or wound us. */ - lock_wait_lock(&lock->base); + lock_wait_lock(&lock->base, &flags); __ww_mutex_check_waiters(&lock->base, ctx, &wake_q); - unlock_wait_lock(&lock->base); - + unlock_wait_lock(&lock->base, &flags); wake_up_q(&wake_q); }