From patchwork Wed Sep 20 07:36:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 142432 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp4190938vqi; Wed, 20 Sep 2023 07:41:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEaJZu2z7VjmK9edCfztO7j9El5AD+Mt/as/xcxQRpDFxzVdsSNM559pZSRDtfyewXrtsl5 X-Received: by 2002:a17:902:bd43:b0:1b3:9d13:34b3 with SMTP id b3-20020a170902bd4300b001b39d1334b3mr2230177plx.37.1695220881609; Wed, 20 Sep 2023 07:41:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695220881; cv=none; d=google.com; s=arc-20160816; b=n0xzMx6PFhkgfThb3mt3vT/xO7hVDIfucQAYzUSML8a1BiqrN9Ji15qXLlf+OGMYbN eFEJ2gpKHLS0+ufoI1cxtTCbFg/f8TCDJ9dWXvLQ9DU0Qs8Mw+R971Wp4xyHtJGaUJhU kzvhz3X3xVfOq+4up95HzXLh8fE2jFiFiPjU8M4ITO0arj9o9knnYKLneu+zLNeqb1bF XgwJZRHOVTjzwa5s0b6Z3AhMHsluV0lh40Mysd+oNNg3yQBzxMZLbS/IxyOg18k0udtA KIVZUh7zwzpu/DbqOJaP4ADg+bf9aJaOHY4Gvss6vYewF7pA9zzaz7soSXZce8BHLSqt fUJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=IfjCkGSUzZIAS5GF7Q7Yq7fEldgU3hvl4AditqhqRTk=; fh=ZN+aWwXugoXtzuAOoWq1wNWhT1zZLE7GgNhaWQwLmjk=; b=wB2O/XSAS572wSJsNJcYdQ/vZVraz38C+A6fY4EdumVdbFcHBqu8kEY2khZ0z+XbLa L/arwzjDpkyHd1oN4uRzkQQ2vD1dCLGMhBY1kfUhe1xDZofnw4WUf7qwmxjXvUusnbvf LTKe/ORm+g9jVyAW8DroDXyI56CooWGWw+xXIFoN1oZ2rTIpWAXpttkQ1sxCfpMoG/R+ VJBoUsi/aZjvyYs8od4mKaDL6jVFjkj0HGFtLQbP+mUUJzAc8DvZst2qeeN/tR0sISQu 1cv0pRpOIoM4daiaNom8+imED/R0TQw+32UZGXypAgCnRMotO9RUXrHr5K/z71/vpG35 nQqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=XJG3bTna; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id x10-20020a170902ec8a00b001c3ea2bbebcsi12318135plg.322.2023.09.20.07.41.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Sep 2023 07:41:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=XJG3bTna; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 31E4C804C20C; Wed, 20 Sep 2023 00:37:01 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233803AbjITHgd (ORCPT + 26 others); Wed, 20 Sep 2023 03:36:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233785AbjITHgS (ORCPT ); Wed, 20 Sep 2023 03:36:18 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0E729E; Wed, 20 Sep 2023 00:36:12 -0700 (PDT) Date: Wed, 20 Sep 2023 07:36:10 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1695195371; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IfjCkGSUzZIAS5GF7Q7Yq7fEldgU3hvl4AditqhqRTk=; b=XJG3bTnasZIAcGpZrUrL+xSyDMeYWnqp5NfP3B1mxzRqCSA/XomvZA9+2pHzWdK8Pm96u9 NfhEhBlbiXiNg88ng3uZFg3VF4llLHkbM8tN+AepSBe1ntFa2hZTDu7t3DSRAkZW1ZpeY5 cWNbYMh/3ZXhUvIDIBJLJxekyrNzjtdyCQuwaIw1X4YEGjTbcMWbB0W85FVgJRAUle0McO 6D2XJq7VlXQGWcoAH1feo+spoUuuMYBLc5CQW6F3JsdNnQ1xqJLurh0NLzvfQtlPd6/lwZ wlU9I+PIY1QVEZLa4Tm0n1c3/vpf//Gb8NFFrrj5JKz8dKX5xUTxzrO2XQUWSQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1695195371; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IfjCkGSUzZIAS5GF7Q7Yq7fEldgU3hvl4AditqhqRTk=; b=Nf1exM4X1NFZGzqjtuwbUCpKiFPT+oxsby/Q3KsNzLAWyrEmZ8Lx8tDccdXHmtc40OmWEW o2KIgDPnCH0VWoAg== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] sched: Provide rt_mutex specific scheduler helpers Cc: Thomas Gleixner , "Peter Zijlstra (Intel)" , Sebastian Andrzej Siewior , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230908162254.999499-5-bigeasy@linutronix.de> References: <20230908162254.999499-5-bigeasy@linutronix.de> MIME-Version: 1.0 Message-ID: <169519537073.27769.17755473188215858233.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 20 Sep 2023 00:37:01 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777567931109879867 X-GMAIL-MSGID: 1777567931109879867 The following commit has been merged into the locking/core branch of tip: Commit-ID: 6b596e62ed9f90c4a97e68ae1f7b1af5beeb3c05 Gitweb: https://git.kernel.org/tip/6b596e62ed9f90c4a97e68ae1f7b1af5beeb3c05 Author: Peter Zijlstra AuthorDate: Fri, 08 Sep 2023 18:22:51 +02:00 Committer: Peter Zijlstra CommitterDate: Wed, 20 Sep 2023 09:31:12 +02:00 sched: Provide rt_mutex specific scheduler helpers With PREEMPT_RT there is a rt_mutex recursion problem where sched_submit_work() can use an rtlock (aka spinlock_t). More specifically what happens is: mutex_lock() /* really rt_mutex */ ... __rt_mutex_slowlock_locked() task_blocks_on_rt_mutex() // enqueue current task as waiter // do PI chain walk rt_mutex_slowlock_block() schedule() sched_submit_work() ... spin_lock() /* really rtlock */ ... __rt_mutex_slowlock_locked() task_blocks_on_rt_mutex() // enqueue current task as waiter *AGAIN* // *CONFUSION* Fix this by making rt_mutex do the sched_submit_work() early, before it enqueues itself as a waiter -- before it even knows *if* it will wait. [[ basically Thomas' patch but with different naming and a few asserts added ]] Originally-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20230908162254.999499-5-bigeasy@linutronix.de --- include/linux/sched.h | 3 +++ include/linux/sched/rt.h | 4 ++++ kernel/sched/core.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac..67623ff 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -911,6 +911,9 @@ struct task_struct { * ->sched_remote_wakeup gets used, so it can be in this word. */ unsigned sched_remote_wakeup:1; +#ifdef CONFIG_RT_MUTEXES + unsigned sched_rt_mutex:1; +#endif /* Bit to tell LSMs we're in execve(): */ unsigned in_execve:1; diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h index 994c256..b2b9e6e 100644 --- a/include/linux/sched/rt.h +++ b/include/linux/sched/rt.h @@ -30,6 +30,10 @@ static inline bool task_is_realtime(struct task_struct *tsk) } #ifdef CONFIG_RT_MUTEXES +extern void rt_mutex_pre_schedule(void); +extern void rt_mutex_schedule(void); +extern void rt_mutex_post_schedule(void); + /* * Must hold either p->pi_lock or task_rq(p)->lock. */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1ea7ba5..58d0346 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6723,9 +6723,6 @@ static inline void sched_submit_work(struct task_struct *tsk) static DEFINE_WAIT_OVERRIDE_MAP(sched_map, LD_WAIT_CONFIG); unsigned int task_flags; - if (task_is_running(tsk)) - return; - /* * Establish LD_WAIT_CONFIG context to ensure none of the code called * will use a blocking primitive -- which would lead to recursion. @@ -6783,7 +6780,12 @@ asmlinkage __visible void __sched schedule(void) { struct task_struct *tsk = current; - sched_submit_work(tsk); +#ifdef CONFIG_RT_MUTEXES + lockdep_assert(!tsk->sched_rt_mutex); +#endif + + if (!task_is_running(tsk)) + sched_submit_work(tsk); __schedule_loop(SM_NONE); sched_update_worker(tsk); } @@ -7044,6 +7046,32 @@ static void __setscheduler_prio(struct task_struct *p, int prio) #ifdef CONFIG_RT_MUTEXES +/* + * Would be more useful with typeof()/auto_type but they don't mix with + * bit-fields. Since it's a local thing, use int. Keep the generic sounding + * name such that if someone were to implement this function we get to compare + * notes. + */ +#define fetch_and_set(x, v) ({ int _x = (x); (x) = (v); _x; }) + +void rt_mutex_pre_schedule(void) +{ + lockdep_assert(!fetch_and_set(current->sched_rt_mutex, 1)); + sched_submit_work(current); +} + +void rt_mutex_schedule(void) +{ + lockdep_assert(current->sched_rt_mutex); + __schedule_loop(SM_NONE); +} + +void rt_mutex_post_schedule(void) +{ + sched_update_worker(current); + lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0)); +} + static inline int __rt_effective_prio(struct task_struct *pi_task, int prio) { if (pi_task)