From patchwork Mon Mar 20 23:37:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 72526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1491841wrt; Mon, 20 Mar 2023 16:38:29 -0700 (PDT) X-Google-Smtp-Source: AK7set/2wRFklUMFQDxFkIAYbyrSMsNTV2wPccJOlq1xx3jaLAmj6vGTac1Pn5hOXPjE2z0NV7sV X-Received: by 2002:a05:6a20:8b07:b0:d9:840f:79c2 with SMTP id l7-20020a056a208b0700b000d9840f79c2mr281022pzh.2.1679355509659; Mon, 20 Mar 2023 16:38:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679355509; cv=none; d=google.com; s=arc-20160816; b=uE0PxYumBMbrPlxQG8emXyaYVFsWHQMQJk9DD8+uP8lFh6P23vxDMjLBfdN8DTaJ8b 4QpboLviKuYqoTtXxOkrky10EKJClLUioCbBNMjBlzuL9RKHNVSaS8FUq7NjhQDXJQeQ 9mHizxXNsM6XzeXv4gJUutLUhZ4qIFWT+qvTWnHXl9gs8ePPf0Cxhpgw6WzPCi340m2Y lHzA7reo/Wjjg8kdysTejGEBuUEuplcZfy0165Ng3pOZ6EOKhLDzJf91SCegemGemTzm trDx5JUwtmN5KMAqm8go7ZNu3JfNMmE01uyRH5gB2pG3zK8N7ENSYsV27SSERD9FzexG A9ZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=MCz4IwNxm3CO2/v3LkVI4pj4nAJZg4E6BpR/zMapMlw=; b=EqZ14Ul8w99+5WfXuf9TMmPayTA06GcODlTB3cpfqAzzlIV6LgYNQarObD4bngGrka hn7G76Cja85Q3IC0IfO+/173jwPgUaqHHl3sL9BksicM/LbyTtkMmJqQ2TqNG5NXsbpI x1OZAryqALYC9YDgALuE0EdiAMmL69H4mtXmXJPwgb9SIq00Ai4ioQ8XELYe97jzLSJM ymPBMPrZRrzyTJjDuQr14rEaxhgr5xrlTejVU1DK2GpOuNIwgBcSum7fN35Odl46HS8h BDjBppglIrpdz3LRXaFWi/J743dOkftIzOKRRC0iVrZK6bOJITFqH/LPq5V+RakQjZVT B0Cg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KSsEJIPR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m19-20020a056a00081300b00627f72304ebsi5016053pfk.88.2023.03.20.16.38.15; Mon, 20 Mar 2023 16:38:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KSsEJIPR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229734AbjCTXhl (ORCPT + 99 others); Mon, 20 Mar 2023 19:37:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229700AbjCTXhg (ORCPT ); Mon, 20 Mar 2023 19:37:36 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ED6331E38 for ; Mon, 20 Mar 2023 16:37:33 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id s3-20020a632c03000000b0050300a8089aso3013254pgs.0 for ; Mon, 20 Mar 2023 16:37:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679355453; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MCz4IwNxm3CO2/v3LkVI4pj4nAJZg4E6BpR/zMapMlw=; b=KSsEJIPRq2Y9zTLgQrB8V4MwfgeziPfrZbdI1uJi7RicmTnJDekbv634If36SOi9Fo x+DdA+GfDL990k9c0eiK9MGe/TIkJc3RAwQMrpUO0MFQPYE187AWWcdztx4ApZmX9AUh j6l174cqyrHdiX3ExY91cys7HZdEGq0zA8NbVl/whxI6pqnj61NoulDZhfDL1vkuEyll bgVmQxLY+zpmuu6elJzCiGkmTSHZPq+dVBkkfo/EY03Q8hCV1t6/b5P0fITjF274Qm9f RQaeMqGFQkumru9qiLh0eLaotwdbMM7/k4EvLwqiSfBCbaMqSGK6R97X1BKoshl4+zuL wIjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679355453; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MCz4IwNxm3CO2/v3LkVI4pj4nAJZg4E6BpR/zMapMlw=; b=Rh8SLgJjx8MYNjt55DcLz+sQA12LvNoPL5N0IXEav9AigXZyaB4UmuIJZAdBFD/nvE h6S9MnXpPacPBskgUugV1lZA7JNPpYH+4juNexRequTCHTlJrkH9gaW0onM8xEn2S+5q hvTdeuW+xDE0+m2u8sELxWMeZYFGpkmE/Ig4e25UsFwtOwXSelVBm+ZDlpyWl0u/SAaG lu/6BkknrSs1uC/osnUowsdCHx8c6hqKtPV1ekVXTF0gQ3eBNpgGTCrOQ9XqqVjGFD7i /OotXY/Sf0uumT+376OsXf4XbGbXBRAQpZ5cSldVdCN0LSB58BHdOjQOJFimGKquNq/f BJ8g== X-Gm-Message-State: AO0yUKX5cqPJ2uNpwcxNHDZC+lXFqLDIuIg1FEckvmj5boOHQYC5s5EM 1TzqOOhOwoQglO8RUYfHqDGNiqZZIWgj21rE9llJgLN6Dai4BfXHwyM7SJlQyjGP/cbmKB6uhEU HkYfeydfy1iPAan3ggcFaSTnzlxWyTbeawNdkrCMb9+2c9G2/h+fW+oYht5kZKEKMpUOi78w= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a65:5205:0:b0:503:7cc9:3f8d with SMTP id o5-20020a655205000000b005037cc93f8dmr108548pgp.9.1679355452949; Mon, 20 Mar 2023 16:37:32 -0700 (PDT) Date: Mon, 20 Mar 2023 23:37:09 +0000 In-Reply-To: <20230320233720.3488453-1-jstultz@google.com> Mime-Version: 1.0 References: <20230320233720.3488453-1-jstultz@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230320233720.3488453-2-jstultz@google.com> Subject: [PATCH v2 01/12] locking/ww_mutex: Remove wakeups from under mutex::wait_lock From: John Stultz To: LKML Cc: Peter Zijlstra , Joel Fernandes , Qais Yousef , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E . McKenney" , kernel-team@android.com, "Connor O'Brien" , John Stultz X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760931882757107167?= X-GMAIL-MSGID: =?utf-8?q?1760931882757107167?= From: Peter Zijlstra In preparation to nest mutex::wait_lock under rq::lock we need to remove wakeups from under it. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E . McKenney" Cc: kernel-team@android.com Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Connor O'Brien Signed-off-by: John Stultz --- v2: * Move wake_q_init() as suggested by Waiman Long --- include/linux/ww_mutex.h | 3 +++ kernel/locking/mutex.c | 8 ++++++++ kernel/locking/ww_mutex.h | 10 ++++++++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/include/linux/ww_mutex.h b/include/linux/ww_mutex.h index bb763085479a..9335b2202017 100644 --- a/include/linux/ww_mutex.h +++ b/include/linux/ww_mutex.h @@ -19,6 +19,7 @@ #include #include +#include #if defined(CONFIG_DEBUG_MUTEXES) || \ (defined(CONFIG_PREEMPT_RT) && defined(CONFIG_DEBUG_RT_MUTEXES)) @@ -58,6 +59,7 @@ struct ww_acquire_ctx { unsigned int acquired; unsigned short wounded; unsigned short is_wait_die; + struct wake_q_head wake_q; #ifdef DEBUG_WW_MUTEXES unsigned int done_acquire; struct ww_class *ww_class; @@ -137,6 +139,7 @@ static inline void ww_acquire_init(struct ww_acquire_ctx *ctx, ctx->acquired = 0; ctx->wounded = false; ctx->is_wait_die = ww_class->is_wait_die; + wake_q_init(&ctx->wake_q); #ifdef DEBUG_WW_MUTEXES ctx->ww_class = ww_class; ctx->done_acquire = 0; diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index d973fe6041bf..1582756914df 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -676,6 +676,8 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas } raw_spin_unlock(&lock->wait_lock); + if (ww_ctx) + ww_ctx_wake(ww_ctx); schedule_preempt_disabled(); first = __mutex_waiter_is_first(lock, &waiter); @@ -725,6 +727,8 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas ww_mutex_lock_acquired(ww, ww_ctx); raw_spin_unlock(&lock->wait_lock); + if (ww_ctx) + ww_ctx_wake(ww_ctx); preempt_enable(); return 0; @@ -736,6 +740,8 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas raw_spin_unlock(&lock->wait_lock); debug_mutex_free_waiter(&waiter); mutex_release(&lock->dep_map, ip); + if (ww_ctx) + ww_ctx_wake(ww_ctx); preempt_enable(); return ret; } @@ -946,9 +952,11 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); + preempt_disable(); raw_spin_unlock(&lock->wait_lock); wake_up_q(&wake_q); + preempt_enable(); } #ifndef CONFIG_DEBUG_LOCK_ALLOC diff --git a/kernel/locking/ww_mutex.h b/kernel/locking/ww_mutex.h index 56f139201f24..e49ea5336473 100644 --- a/kernel/locking/ww_mutex.h +++ b/kernel/locking/ww_mutex.h @@ -161,6 +161,12 @@ static inline void lockdep_assert_wait_lock_held(struct rt_mutex *lock) #endif /* WW_RT */ +void ww_ctx_wake(struct ww_acquire_ctx *ww_ctx) +{ + wake_up_q(&ww_ctx->wake_q); + wake_q_init(&ww_ctx->wake_q); +} + /* * Wait-Die: * The newer transactions are killed when: @@ -284,7 +290,7 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter, #ifndef WW_RT debug_mutex_wake_waiter(lock, waiter); #endif - wake_up_process(waiter->task); + wake_q_add(&ww_ctx->wake_q, waiter->task); } return true; @@ -331,7 +337,7 @@ static bool __ww_mutex_wound(struct MUTEX *lock, * wakeup pending to re-read the wounded state. */ if (owner != current) - wake_up_process(owner); + wake_q_add(&ww_ctx->wake_q, owner); return true; }