From patchwork Thu Feb 9 19:17:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 55067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp526892wrn; Thu, 9 Feb 2023 11:23:55 -0800 (PST) X-Google-Smtp-Source: AK7set/qkUjP1SlG4XtHEsj9b+tT5nXJJPhqzN8Uqv6vuQqxVrmgFL+uUkEVkopWaS9x2JCchr01 X-Received: by 2002:a17:90b:1d08:b0:230:c247:731b with SMTP id on8-20020a17090b1d0800b00230c247731bmr13935480pjb.7.1675970634984; Thu, 09 Feb 2023 11:23:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675970634; cv=none; d=google.com; s=arc-20160816; b=WTcrRp/M88lAYAkTWxw/WhNDqwd7G5BxDx+BxZ6q3QI6iyeM6RuqXtyzOq1cJBU2yj sYiXtUElZ+Jddn+evBchTDR4QvaNpZK1L7+Ft2ndHqn6SNjtsvTgFnxQGsnn4Afi4qlu eDdX/0uBFU/VfN+xHQt5GY8EvqwsxkiJTyKspoK5IGG1Ooj8xC4cQZDy6bvmTC//n0gO jyXKP+dbqIva61/AZdM6U+juk01JWZEpNpFEAigd4FsAcqH8pu/lHL1aaV6eqaIaQ796 U41YW+9QjOjyBkkWIWsAzgP+jkgljNzkRFN3cF/NJCa/kpyAPm1fGOceOo2b94s38XqT sttQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eIfe60XJJIiDJTKhVwgb+glJla9ENtL00sq047lFBiY=; b=c+cn0G+HUpE8Ns83sre/sWkNU7dy6XpQOUW45Vs19f30apIuFti2uYc8Gzi8aAhqoX 1S8GessnL4t52YY60W8/bOSZ7Ae72TxcgnUIBOb8O1b4QugYkqZPLZZXExcHY03V0Upl m1tnl5mBcWWTu4hLTsJd5yquFIZ7BgJhIIS9FQVl/1EeV3csEFZE6lAWbJRHQTUOhRn8 Ab7hZEO00QVyxEPdO0Ca24+k4StceOHdDfRsO9qSsFVci3NaKnzk1LGbQ+ZBMKkdIP0N Pn1Q1FmWW3CJHuIi4vglTylzOo9R6W3HAZCmxrbNHgvwMNdTanwBg+4bdNKV+1HY/bKw vMgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EJ+0bCl0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kk15-20020a17090b4a0f00b0021a1261c317si6513093pjb.126.2023.02.09.11.23.40; Thu, 09 Feb 2023 11:23:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EJ+0bCl0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230029AbjBITS2 (ORCPT + 99 others); Thu, 9 Feb 2023 14:18:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230016AbjBITS0 (ORCPT ); Thu, 9 Feb 2023 14:18:26 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F898458A1; Thu, 9 Feb 2023 11:18:26 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C8574B8202F; Thu, 9 Feb 2023 19:18:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07908C433EF; Thu, 9 Feb 2023 19:18:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675970303; bh=2N2cNIU1XW5YmMOreRcBKEoSVcSbJ0JNBK/ngcNtt9Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EJ+0bCl0DYiLH3J2h2I74scxWV26SXxTDm1arHtllsr8w1Bq1EuE6aoePIvqs6ryi CfWcMTuzcZQCO3ceinvRXyadBcBNfkzYhtrdAnqVWVpAenvUbPrt7o97lP71TOgneU A8qBJefAmcCCMWG8D9fOfo2n8Fqis275rcLismrol8xdsrD11GJ0qmzqc74v06XFeV 1ngvEc6mfBrKZeKCAl2ZvA2LV+l8vfFAkFMA93hyPUFH9JkybLkldcyVeSP0MmqHM+ JBWKklr8/bso6Ry+CjEaHttL2xa7Geu5OzAgO6W8W/PCzHGE8tINlDYZyeqII2Og47 atGwj4GGGuLSQ== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar Subject: [PATCH 1/3] livepatch: Skip task_call_func() for current task Date: Thu, 9 Feb 2023 11:17:47 -0800 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757382584608197108?= X-GMAIL-MSGID: =?utf-8?q?1757382584608197108?= The current task doesn't need the scheduler's protection to unwind its own stack. Signed-off-by: Josh Poimboeuf Reviewed-by: Petr Mladek --- kernel/livepatch/transition.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c index f1b25ec581e0..4d1f443778f7 100644 --- a/kernel/livepatch/transition.c +++ b/kernel/livepatch/transition.c @@ -307,7 +307,11 @@ static bool klp_try_switch_task(struct task_struct *task) * functions. If all goes well, switch the task to the target patch * state. */ - ret = task_call_func(task, klp_check_and_switch_task, &old_name); + if (task == current) + ret = klp_check_and_switch_task(current, &old_name); + else + ret = task_call_func(task, klp_check_and_switch_task, &old_name); + switch (ret) { case 0: /* success */ break; From patchwork Thu Feb 9 19:17:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 55068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp526937wrn; Thu, 9 Feb 2023 11:24:02 -0800 (PST) X-Google-Smtp-Source: AK7set/M31qQu5kyLtjJ2xWlmgqLlk1+6fiZH/SrOk7HokuVF6RFoZ8uvkj/0CmccD7sVjlwC0SK X-Received: by 2002:a62:1c13:0:b0:593:2289:f01c with SMTP id c19-20020a621c13000000b005932289f01cmr10281087pfc.25.1675970641741; Thu, 09 Feb 2023 11:24:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675970641; cv=none; d=google.com; s=arc-20160816; b=uj99sM+++T5ujIu7q9VhB+iUgTjBJ2Rx03hcBWuCT1ahC5dGKNZjKHYnhHjHLMTNFK c9A+1ZL0SVzhXpj0cqhAvb1OJVNzZ3k5PN4ocgY+tyWEtQTklm8J+SsX3LBO+E/XVTyN zM+uhtYhuS1zDZzvzZfZEiR6QY1TGrwibp/GJq17xsRh3bbjS/Pk9DUdxcUlnsL5zs4e F/oi8sfB3BfATbpxViDTYcq58M4fhXzd/50DAfkqHcPSihzEavcFPdwV6BDEonDryPti 3EwAkmnZB1H5Ys5eKA7vee6CKs3WfDSEHklesvqo30yq3ikPSUGi0YJAwrbHaqRsPABh Dg1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ivOrwm1iX8p0PQ59kd0MctawwE6EPVIUWJRCxoVCuRw=; b=D9aQlg+1ALSy+WF5eIINFC/bVFW77tT9KOvAMC72Osiqr8F7JQxBgq3WVspmHvKgRH rJ2MvTQCMZGKPq2SEmhNJ55NO+8tki7/jFQDwyGtQss5qq2jqmWmR7VlkBLdBrI3pWlf DHoXeprEGbPIWDpu2fpGey8qPGtJO3ZujTgqNK7Z7zZFIf4jYU81AhCYxJvq/oDyCiac wERlOGuUlOzgrZsTUt60kFNDQ7Z/nWT7ZUJBXDeqyYu2mJw+wYsFzpr8hYeFU7jkGL5Q rwNYeGrc5qF/dNvYQBr45JmbRDShWvEpILUrjUVrXUmzQ/ZMt4muiz+yq8obfDsgOIGS RfGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=svNJRaRp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v129-20020a622f87000000b0058e01069f69si2322908pfv.115.2023.02.09.11.23.49; Thu, 09 Feb 2023 11:24:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=svNJRaRp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230044AbjBITSa (ORCPT + 99 others); Thu, 9 Feb 2023 14:18:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbjBITS2 (ORCPT ); Thu, 9 Feb 2023 14:18:28 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C37BC4740B; Thu, 9 Feb 2023 11:18:26 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 62C32B822DD; Thu, 9 Feb 2023 19:18:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C18BC433A1; Thu, 9 Feb 2023 19:18:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675970304; bh=o8e86Gia9Kjg1hPpDyFE9nyrHV0UHPjzeybmAA0GzcY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=svNJRaRpYd1UGnt0oC85+oWqbza9F2b2trGpQdXIslSjOWQlnE04Q4xjsl9h+H9qa KM2AyGTPz1YeFAtbu48UguGDvnV5dZsBwOVw/fwBXueFRaIe444GCjgK12ypjkDw4G zAN0q4FziMg0fIoC9BqEcHn2pQ9Kjy1XK5MuhTO3rE4TiVkAi1DiTIJf4MSmtEvCfI duT9Ow0acOC8FpTOxuL/ShvIGUoipkZ7BDj7m9b8Eq2YoNKN0E0Ni9XCqGW1MEjKCo AXGiH6x0f7GwAPPFDMWE0DRVaiR6TbMTSMAENXG7CrWoAAGr+nuKqLhJ24JjEEUJle e35d9GYRpZ8mg== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar Subject: [PATCH 2/3] livepatch,sched: Add livepatch task switching to cond_resched() Date: Thu, 9 Feb 2023 11:17:48 -0800 Message-Id: <58cd637f7557e829142a2eb255daa91fa57e6321.1675969869.git.jpoimboe@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757382584780525957?= X-GMAIL-MSGID: =?utf-8?q?1757382591702943942?= There have been reports [1][2] of live patches failing to complete within a reasonable amount of time due to CPU-bound kthreads. Fix it by patching tasks in cond_resched(). There are four different flavors of cond_resched(), depending on the kernel configuration. Hook into all of them. A more elegant solution might be to use a preempt notifier. However, non-ORC unwinders can't unwind a preempted task reliably. [1] https://lore.kernel.org/lkml/20220507174628.2086373-1-song@kernel.org/ [2] https://lkml.kernel.org/lkml/20230120-vhost-klp-switching-v1-0-7c2b65519c43@kernel.org Signed-off-by: Josh Poimboeuf --- include/linux/livepatch.h | 1 + include/linux/livepatch_sched.h | 29 +++++++++++++++ include/linux/sched.h | 20 ++++++++--- kernel/livepatch/transition.c | 54 ++++++++++++++++++++++++++++ kernel/sched/core.c | 64 ++++++++++++++++++++++++++++----- 5 files changed, 155 insertions(+), 13 deletions(-) create mode 100644 include/linux/livepatch_sched.h diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 293e29960c6e..9b9b38e89563 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -13,6 +13,7 @@ #include #include #include +#include #if IS_ENABLED(CONFIG_LIVEPATCH) diff --git a/include/linux/livepatch_sched.h b/include/linux/livepatch_sched.h new file mode 100644 index 000000000000..013794fb5da0 --- /dev/null +++ b/include/linux/livepatch_sched.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_LIVEPATCH_SCHED_H_ +#define _LINUX_LIVEPATCH_SCHED_H_ + +#include +#include + +#ifdef CONFIG_LIVEPATCH + +void __klp_sched_try_switch(void); + +#if !defined(CONFIG_PREEMPT_DYNAMIC) || !defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) + +DECLARE_STATIC_KEY_FALSE(klp_sched_try_switch_key); + +static __always_inline void klp_sched_try_switch(void) +{ + if (static_branch_unlikely(&klp_sched_try_switch_key)) + __klp_sched_try_switch(); +} + +#endif /* !CONFIG_PREEMPT_DYNAMIC || !CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + +#else /* !CONFIG_LIVEPATCH */ +static inline void klp_sched_try_switch(void) {} +static inline void __klp_sched_try_switch(void) {} +#endif /* CONFIG_LIVEPATCH */ + +#endif /* _LINUX_LIVEPATCH_SCHED_H_ */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 4df2b3e76b30..2bc5e925a6d8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -36,6 +36,7 @@ #include #include #include +#include #include /* task_struct member predeclarations (sorted alphabetically): */ @@ -2070,6 +2071,9 @@ extern int __cond_resched(void); #if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) +void sched_dynamic_klp_enable(void); +void sched_dynamic_klp_disable(void); + DECLARE_STATIC_CALL(cond_resched, __cond_resched); static __always_inline int _cond_resched(void) @@ -2078,6 +2082,7 @@ static __always_inline int _cond_resched(void) } #elif defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) + extern int dynamic_cond_resched(void); static __always_inline int _cond_resched(void) @@ -2085,20 +2090,25 @@ static __always_inline int _cond_resched(void) return dynamic_cond_resched(); } -#else +#else /* !CONFIG_PREEMPTION */ static inline int _cond_resched(void) { + klp_sched_try_switch(); return __cond_resched(); } -#endif /* CONFIG_PREEMPT_DYNAMIC */ +#endif /* PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ -#else +#else /* CONFIG_PREEMPTION && !CONFIG_PREEMPT_DYNAMIC */ -static inline int _cond_resched(void) { return 0; } +static inline int _cond_resched(void) +{ + klp_sched_try_switch(); + return 0; +} -#endif /* !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC) */ +#endif /* !CONFIG_PREEMPTION || CONFIG_PREEMPT_DYNAMIC */ #define cond_resched() ({ \ __might_resched(__FILE__, __LINE__, 0); \ diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c index 4d1f443778f7..e629966c6fbe 100644 --- a/kernel/livepatch/transition.c +++ b/kernel/livepatch/transition.c @@ -9,6 +9,7 @@ #include #include +#include #include "core.h" #include "patch.h" #include "transition.h" @@ -24,6 +25,25 @@ static int klp_target_state = KLP_UNDEFINED; static unsigned int klp_signals_cnt; +/* + * When a livepatch is in progress, enable klp stack checking in + * cond_resched(). This helps CPU-bound kthreads get patched. + */ +#if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) + +#define klp_cond_resched_enable() sched_dynamic_klp_enable() +#define klp_cond_resched_disable() sched_dynamic_klp_disable() + +#else /* !CONFIG_PREEMPT_DYNAMIC || !CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + +DEFINE_STATIC_KEY_FALSE(klp_sched_try_switch_key); +EXPORT_SYMBOL(klp_sched_try_switch_key); + +#define klp_cond_resched_enable() static_branch_enable(&klp_sched_try_switch_key) +#define klp_cond_resched_disable() static_branch_disable(&klp_sched_try_switch_key) + +#endif /* CONFIG_PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + /* * This work can be performed periodically to finish patching or unpatching any * "straggler" tasks which failed to transition in the first attempt. @@ -76,6 +96,8 @@ static void klp_complete_transition(void) klp_transition_patch->mod->name, klp_target_state == KLP_PATCHED ? "patching" : "unpatching"); + klp_cond_resched_disable(); + if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) { klp_unpatch_replaced_patches(klp_transition_patch); klp_discard_nops(klp_transition_patch); @@ -338,6 +360,36 @@ static bool klp_try_switch_task(struct task_struct *task) return !ret; } +void __klp_sched_try_switch(void) +{ + if (likely(!klp_patch_pending(current))) + return; + + /* + * This function is called from cond_resched() which is called in many + * places throughout the kernel. Using the klp_mutex here might + * deadlock. + * + * Instead, disable preemption to prevent racing with other callers of + * klp_try_switch_task(). Thanks to task_call_func() they won't be + * able to switch this task while it's running. + */ + preempt_disable(); + + /* + * Make sure current didn't get patched between the above check and + * preempt_disable(). + */ + if (unlikely(!klp_patch_pending(current))) + goto out; + + klp_try_switch_task(current); + +out: + preempt_enable(); +} +EXPORT_SYMBOL(__klp_sched_try_switch); + /* * Sends a fake signal to all non-kthread tasks with TIF_PATCH_PENDING set. * Kthreads with TIF_PATCH_PENDING set are woken up. @@ -496,6 +548,8 @@ void klp_start_transition(void) set_tsk_thread_flag(task, TIF_PATCH_PENDING); } + klp_cond_resched_enable(); + klp_signals_cnt = 0; } diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3a0ef2fefbd5..ed5bf5ec0477 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8500,6 +8500,7 @@ EXPORT_STATIC_CALL_TRAMP(might_resched); static DEFINE_STATIC_KEY_FALSE(sk_dynamic_cond_resched); int __sched dynamic_cond_resched(void) { + klp_sched_try_switch(); if (!static_branch_unlikely(&sk_dynamic_cond_resched)) return 0; return __cond_resched(); @@ -8648,13 +8649,17 @@ int sched_dynamic_mode(const char *str) #error "Unsupported PREEMPT_DYNAMIC mechanism" #endif -void sched_dynamic_update(int mode) +DEFINE_MUTEX(sched_dynamic_mutex); +static bool klp_override; + +static void __sched_dynamic_update(int mode) { /* * Avoid {NONE,VOLUNTARY} -> FULL transitions from ever ending up in * the ZERO state, which is invalid. */ - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_enable(might_resched); preempt_dynamic_enable(preempt_schedule); preempt_dynamic_enable(preempt_schedule_notrace); @@ -8662,36 +8667,79 @@ void sched_dynamic_update(int mode) switch (mode) { case preempt_dynamic_none: - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_disable(might_resched); preempt_dynamic_disable(preempt_schedule); preempt_dynamic_disable(preempt_schedule_notrace); preempt_dynamic_disable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: none\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: none\n"); break; case preempt_dynamic_voluntary: - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_enable(might_resched); preempt_dynamic_disable(preempt_schedule); preempt_dynamic_disable(preempt_schedule_notrace); preempt_dynamic_disable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: voluntary\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: voluntary\n"); break; case preempt_dynamic_full: - preempt_dynamic_disable(cond_resched); + if (!klp_override) + preempt_dynamic_disable(cond_resched); preempt_dynamic_disable(might_resched); preempt_dynamic_enable(preempt_schedule); preempt_dynamic_enable(preempt_schedule_notrace); preempt_dynamic_enable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: full\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: full\n"); break; } preempt_dynamic_mode = mode; } +void sched_dynamic_update(int mode) +{ + mutex_lock(&sched_dynamic_mutex); + __sched_dynamic_update(mode); + mutex_unlock(&sched_dynamic_mutex); +} + +#ifdef CONFIG_HAVE_PREEMPT_DYNAMIC_CALL + +static int klp_cond_resched(void) +{ + __klp_sched_try_switch(); + return __cond_resched(); +} + +void sched_dynamic_klp_enable(void) +{ + mutex_lock(&sched_dynamic_mutex); + + klp_override = true; + static_call_update(cond_resched, klp_cond_resched); + + mutex_unlock(&sched_dynamic_mutex); +} + +void sched_dynamic_klp_disable(void) +{ + mutex_lock(&sched_dynamic_mutex); + + klp_override = false; + __sched_dynamic_update(preempt_dynamic_mode); + + mutex_unlock(&sched_dynamic_mutex); +} + +#endif /* CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + static int __init setup_preempt_mode(char *str) { int mode = sched_dynamic_mode(str); From patchwork Thu Feb 9 19:17:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 55069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp527013wrn; Thu, 9 Feb 2023 11:24:13 -0800 (PST) X-Google-Smtp-Source: AK7set+YG+1XYmg/I4VqxX0lx55iQ/8d506QsVIYcdg+UX4EAaerrsKvFK1D+wSMXPVSxJR1+Zd+ X-Received: by 2002:a50:9554:0:b0:4ab:25d7:8cec with SMTP id v20-20020a509554000000b004ab25d78cecmr1270093eda.30.1675970653209; Thu, 09 Feb 2023 11:24:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675970653; cv=none; d=google.com; s=arc-20160816; b=QHSIfofbEagne94rTA7xZuG2jSD0QQxCNIlshPpwtpf1ahyehGwGVx52Zf+UG0w3h3 4a3b7n1k7c84H6GVrW5LFiW5e3riYmKwQNU2CGe26szyYl2GBp3pbeTgLCJuWe9bvoum fCk0VFMYVfBKcbt/udH6Xj2aMKj+0yHcUqeptYMxbZ1ilffoBEgKS5EuB279s6sd0VHc 3xMrwUNOl5mbf4uihfqXSsDT50nT2RwSRLrY+btKoEJfXLYTJy5S4b22e/VCmqP8xc3v KE4MghP+w0PB4xE7b3BSBkqLKb9mpLovgExmMkTbXxuspqabosbHtc4hl5nh71TUUSE1 ELaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1YwVVMwfkejb2PKVgntMKS/xxiJWK+Thf4NDrYceLI8=; b=VecjC7RBh0y5E+HAFUK2Es2AvGH0zl9QK3dwRMzSdPnaZIXViGYAbnHw4nlcK3F81v kXlPQojH2mN2ElS+DX154RZE7TbtidqEIbLW63bhoXSNYJc7NLBomvPw5akwkiETsSqU C2xNrUWO28ipTGG9DcClNamvFUcAWx7e8l4n82Bk7On7MRfNq+XBHDXuwsktf/A/3b+b MrhtNrFsE1lCV5k1pUlUtiaAITJcdWqJLkp/F7/hw/zZrMCwcrqVrX6eEb+mw//Elpxq XSvOPJe6/+skA18ZDrrAmM5pri7j0q4Dd9Tzi2qnWjyExDZJs8tE+BoaF5XZx0ITeb+q dttw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="HhkyRF/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l10-20020aa7d94a000000b004a23153d992si3103057eds.238.2023.02.09.11.23.49; Thu, 09 Feb 2023 11:24:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="HhkyRF/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230074AbjBITSd (ORCPT + 99 others); Thu, 9 Feb 2023 14:18:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230032AbjBITS2 (ORCPT ); Thu, 9 Feb 2023 14:18:28 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67CED6A70A; Thu, 9 Feb 2023 11:18:27 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F1D84B82292; Thu, 9 Feb 2023 19:18:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37C81C4339C; Thu, 9 Feb 2023 19:18:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675970304; bh=NKdf/DmfCqhfyz+hBsxyGBbopa789ZG35qd6K94RXng=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HhkyRF/zpOvo8qUME/yQOqPMB6OJmc0Y3wrOPwQq1Vr/2TcutOLZE3GpTVo62y4lO 2gvB4xNDXS2ufFFu5kXwzZNoOiQS/i9vMVtPDplVuPySNv9x+R47NjGWT0+1wXpQok aMFqCqjmZQ5f9fvJATem7WCQgWTDMfNQzUCS95WFr8LOtYhZ3m162G1Y661kqo9tBR F14sKAf4/89QjrMiQR3lfp8hdrWL5mBMywnjI854U/NXvcLJ6kl6JZoEHH8L4x1ReU 4uD1foRL8a9CDRhcg3d9BIUdU0btJTfSohGG6D5VhDEMNxmUIZMvgA4jNofd5vrdlo AzeyI4X9Xs/dA== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar Subject: [PATCH 3/3] vhost: Fix livepatch timeouts in vhost_worker() Date: Thu, 9 Feb 2023 11:17:49 -0800 Message-Id: <76622971662bbd42abc9a1b25a73a8c3ac929ca3.1675969869.git.jpoimboe@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757382604004396718?= X-GMAIL-MSGID: =?utf-8?q?1757382604004396718?= Livepatch timeouts were reported due to busy vhost_worker() kthreads. Now that cond_resched() can do livepatch task switching, use cond_resched() in vhost_worker(). That's the better way to conditionally call schedule() anyway. Reported-by: Seth Forshee (DigitalOcean) Link: https://lkml.kernel.org/lkml/20230120-vhost-klp-switching-v1-0-7c2b65519c43@kernel.org Signed-off-by: Josh Poimboeuf Reviewed-by: Petr Mladek --- drivers/vhost/vhost.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index cbe72bfd2f1f..424c0c939f57 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -363,8 +363,7 @@ static int vhost_worker(void *data) kcov_remote_start_common(dev->kcov_handle); work->fn(work); kcov_remote_stop(); - if (need_resched()) - schedule(); + cond_resched(); } } kthread_unuse_mm(dev->mm);