From patchwork Fri Feb 17 22:22:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 58785 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp99289wrn; Fri, 17 Feb 2023 14:30:51 -0800 (PST) X-Google-Smtp-Source: AK7set96eg52BOf2lqe2/ry2vJmOxA8t0EqjFoVqu+dgaWF7X/I0Zb6SkUFVQ4ny4tsiyB4raTwe X-Received: by 2002:a17:906:4910:b0:8b1:7de0:c854 with SMTP id b16-20020a170906491000b008b17de0c854mr4703022ejq.41.1676673051422; Fri, 17 Feb 2023 14:30:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676673051; cv=none; d=google.com; s=arc-20160816; b=JNZ2Zl70CWUIRIGHJ/h1ZJGTEeHmuKlu2VdJZBA+/UL20yAV9lOHa0sK1h2RxvQCew gj2xP4ucwc2CU9Lb8posUUBkS5TxEMb3H9LFFLAE2pZERtM5N8qYd7J31zdFmwJFL1t1 1evut0nCUEWZ6ON+QNibG1oiHEQflbfQgmkZv1p9KQV9k6CDZwIQtCuPNwgsNy4fkQzE /bPu0oeVqcSlDQ1tNPqhO8EVQvTxtuUSycveX3uCr3qjPHkFSYdEDW8IcvoevLOTxyHW 1r6B3wm+wUKT/NRysZUSRGA/hHc5raIEFShJZvsC1pqtuJ/PVOCXL7HipTj+DZo9iKTQ veKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4jU+SGCwX400pg3X4OgbqE9w71eYcYGFHVV8AI/hOxc=; b=sH3ncamc5t7EQ+FJeqGlBveRtb8rlth+GruFBETRkxbuH28AT5rSRdGxGT7qL5sN+x atu4e8sJuaIOiGOd+DyMw94YawAuLGN7Abq2Xwsp/NN5EtcmzVSczCdfiObXIvUqacCv +Eg1ISt+eHFKsHdkzMUt4rlLYcnU6RiKJ33uWFb9smV5sAFL8l4tWesSvf45R4rmQcyD 6ILdXBNhwHx0+MRDV9AfEux8Sik7ToDEq9G9R5BPoA48WlATGThteRTyndx7Y0OvqkoD C2Ujwb5XI4M6Uv8Q1HgmNYc8o6is7p5FVMyMkzxtFrRVHTOuwX0mPXRqfO3LGNHfoV31 hvHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HvaqSr5P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vl10-20020a17090730ca00b008b1484584dfsi6124488ejb.4.2023.02.17.14.30.28; Fri, 17 Feb 2023 14:30:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HvaqSr5P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229848AbjBQWXG (ORCPT + 99 others); Fri, 17 Feb 2023 17:23:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229509AbjBQWXC (ORCPT ); Fri, 17 Feb 2023 17:23:02 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B80345D3EB; Fri, 17 Feb 2023 14:23:01 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6EF30B82DD0; Fri, 17 Feb 2023 22:23:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC44DC433EF; Fri, 17 Feb 2023 22:22:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676672579; bh=nDYIubJTBPWH5ktJId6VhvPlAhhjbmackYfSKNUBnA8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HvaqSr5PbTHNs8mAMaAmaYbTfmj4JGX1vO3kz55UvEw93l5BToEp5+PskKaCtMUeS OUMKsxV+tVXyKFXE+8+csdac7DIDBeylDt7DWhvsWVljlU4UlctSUoRrOmFMPlYBpv zgOZhnks24H2u7Ie0dgDdmYZftBRMVkj0epeKHM4sLSs17K8xg3btcsVxQ2K3C5h7u sva5ly/+lj6K1pMbn2gwTWJBSx0JCIp/u7aSiSH3zae9c7968ZiY97Coz7SedTHhNz q31EckXsm5CLqqo9HEQ7WAcVB4TnJPv7Wpy4g6/QRkUk09ZQ7s5cUGDjCVS1gHXIfJ Lmn9k+xGyVk3Q== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar , Rik van Riel Subject: [PATCH v2 1/3] livepatch: Skip task_call_func() for current task Date: Fri, 17 Feb 2023 14:22:54 -0800 Message-Id: <4b92e793462d532a05f03767151fa29db3e68e13.1676672328.git.jpoimboe@kernel.org> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758119121778812375?= X-GMAIL-MSGID: =?utf-8?q?1758119121778812375?= The current task doesn't need the scheduler's protection to unwind its own stack. Tested-by: Seth Forshee (DigitalOcean) Reviewed-by: Petr Mladek Signed-off-by: Josh Poimboeuf --- kernel/livepatch/transition.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c index f1b25ec581e0..4d1f443778f7 100644 --- a/kernel/livepatch/transition.c +++ b/kernel/livepatch/transition.c @@ -307,7 +307,11 @@ static bool klp_try_switch_task(struct task_struct *task) * functions. If all goes well, switch the task to the target patch * state. */ - ret = task_call_func(task, klp_check_and_switch_task, &old_name); + if (task == current) + ret = klp_check_and_switch_task(current, &old_name); + else + ret = task_call_func(task, klp_check_and_switch_task, &old_name); + switch (ret) { case 0: /* success */ break; From patchwork Fri Feb 17 22:22:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 58786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp100232wrn; Fri, 17 Feb 2023 14:33:12 -0800 (PST) X-Google-Smtp-Source: AK7set+dx0R4GI6QSgMqj378kZHbzlXstPbVxhxmutpLqVBScrIxDI/6UFpG7P0xA43dE2negWr6 X-Received: by 2002:a17:906:55d2:b0:878:7a0e:5730 with SMTP id z18-20020a17090655d200b008787a0e5730mr1860199ejp.56.1676673192032; Fri, 17 Feb 2023 14:33:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676673192; cv=none; d=google.com; s=arc-20160816; b=UJaLh1tw501nSdUs8reGHwt/zbEapPDq0gf9/TFpfbGHIghloLgJs1CubjaEdE1y4t 3ch152wtJU9Jxt1FLQnXHsmEw4YOvPikVkW/hxk6kfMWYuS2yVOQ7KVB7VvYmGemdvFo /ShsyuCNs81aqVyWgQkSt2qF7/nN/O+7D0pzGrvwWzKkpe2fPjwz6AprMCNouzQjftbp f2gvyhmD4WXWt1VMv4z8CPGE7BG8CdsS0fE0E+iEsb8dh14IJkvWpK5FSs5ct48qKkAx Suj5A5QiA7pH/gO/9R0hgtJwKc4ZrFYsj91yMpkCwEXO5UpDBtT8kwh3DC7p88QkJLLA 08tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Z1VFKsPRh6UzpbR3qmjSpXpHB2EN394E3/CdgfKhnT8=; b=xFjjEWP4d13xM+QxjNkcsYdFMGp5+Mt0rOEH9C1Bj4EH6NYOwcqCF5Bo3d18l5r2SC P1WBgDbYRW+JQ196NZtlOki06Il2IjUnRWK8LLHdM1NAgqAybPcUf1I/3IOanoltTVvj 84GEcpgHOpt7CWFbOra3sN8G638Waf69P7HRTvHQM0xRMdJdZNdUib2vAMaiOg2bqZ8G 5SPOXZhQgcG6LCueyusRXJE0R7Ji9EFBQFsnKOnLnW/s2zdLZkUUGrjTgb4tPVElfBcG 3HRuMYc5+QanxDW78xfYhwiLIJunIUgRY5HzWdoiYx1/G1vHPXA5NRkZtUvFFSHML/Uq nX3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GbX8duAX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fx17-20020a1709069e9100b00877da25cb31si6152106ejc.869.2023.02.17.14.32.48; Fri, 17 Feb 2023 14:33:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GbX8duAX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbjBQWXL (ORCPT + 99 others); Fri, 17 Feb 2023 17:23:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229780AbjBQWXE (ORCPT ); Fri, 17 Feb 2023 17:23:04 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93C9E5D3ED; Fri, 17 Feb 2023 14:23:02 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 24F09B82E60; Fri, 17 Feb 2023 22:23:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 585ADC433AA; Fri, 17 Feb 2023 22:22:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676672579; bh=frILY3UK6CEIkeNVu4J9ZM0FyGlw0e5p8pr6LM4L8HQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GbX8duAXVXhM9FXOSJEecIPBcSioMYZ7uogUH9EnFvIRXZWyHUONEY1oJ1hNunUqn QoOXyet8BR5TUmQ5ubbbnRblo+qgHScZ5lVgvgIqnSEZP57zwmr8Q9NTHQdtUMlWci pjdXYZXRnV1Q3v+Tz90AZ8CK1cg9mo5t7w/XVlIPt9HCwZ0ihbbXD6DRR5CGX2FZwX mmcY6Dp9qrIJCsGdbctVafWxlhJ76lxWxH3QxgG5hpAXvxneFSc7gggQ0Od1UPVN1e ihfUymM4Z1WtmcpWMr7SNHZFUYwveDMy76g2RjYOEe08PrwiJbFvIZh3apf1k/jiUG 8anXXEzYvMtAA== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar , Rik van Riel Subject: [PATCH v2 2/3] livepatch,sched: Add livepatch task switching to cond_resched() Date: Fri, 17 Feb 2023 14:22:55 -0800 Message-Id: <9f09bff809fc026618108e8bbaac67ef2f8e6d3d.1676672328.git.jpoimboe@kernel.org> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758119120101326884?= X-GMAIL-MSGID: =?utf-8?q?1758119269332647900?= There have been reports [1][2] of live patches failing to complete within a reasonable amount of time due to CPU-bound kthreads. Fix it by patching tasks in cond_resched(). There are four different flavors of cond_resched(), depending on the kernel configuration. Hook into all of them. A more elegant solution might be to use a preempt notifier. However, non-ORC unwinders can't unwind a preempted task reliably. [1] https://lore.kernel.org/lkml/20220507174628.2086373-1-song@kernel.org/ [2] https://lkml.kernel.org/lkml/20230120-vhost-klp-switching-v1-0-7c2b65519c43@kernel.org Tested-by: Seth Forshee (DigitalOcean) Signed-off-by: Josh Poimboeuf Reviewed-by: Petr Mladek --- include/linux/livepatch.h | 1 + include/linux/livepatch_sched.h | 29 ++++++++++++ include/linux/sched.h | 20 ++++++--- kernel/livepatch/transition.c | 78 ++++++++++++++++++++++++++++----- kernel/sched/core.c | 64 +++++++++++++++++++++++---- 5 files changed, 168 insertions(+), 24 deletions(-) create mode 100644 include/linux/livepatch_sched.h diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 293e29960c6e..9b9b38e89563 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -13,6 +13,7 @@ #include #include #include +#include #if IS_ENABLED(CONFIG_LIVEPATCH) diff --git a/include/linux/livepatch_sched.h b/include/linux/livepatch_sched.h new file mode 100644 index 000000000000..013794fb5da0 --- /dev/null +++ b/include/linux/livepatch_sched.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_LIVEPATCH_SCHED_H_ +#define _LINUX_LIVEPATCH_SCHED_H_ + +#include +#include + +#ifdef CONFIG_LIVEPATCH + +void __klp_sched_try_switch(void); + +#if !defined(CONFIG_PREEMPT_DYNAMIC) || !defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) + +DECLARE_STATIC_KEY_FALSE(klp_sched_try_switch_key); + +static __always_inline void klp_sched_try_switch(void) +{ + if (static_branch_unlikely(&klp_sched_try_switch_key)) + __klp_sched_try_switch(); +} + +#endif /* !CONFIG_PREEMPT_DYNAMIC || !CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + +#else /* !CONFIG_LIVEPATCH */ +static inline void klp_sched_try_switch(void) {} +static inline void __klp_sched_try_switch(void) {} +#endif /* CONFIG_LIVEPATCH */ + +#endif /* _LINUX_LIVEPATCH_SCHED_H_ */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 853d08f7562b..bd1e6f02facb 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -36,6 +36,7 @@ #include #include #include +#include #include /* task_struct member predeclarations (sorted alphabetically): */ @@ -2064,6 +2065,9 @@ extern int __cond_resched(void); #if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) +void sched_dynamic_klp_enable(void); +void sched_dynamic_klp_disable(void); + DECLARE_STATIC_CALL(cond_resched, __cond_resched); static __always_inline int _cond_resched(void) @@ -2072,6 +2076,7 @@ static __always_inline int _cond_resched(void) } #elif defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) + extern int dynamic_cond_resched(void); static __always_inline int _cond_resched(void) @@ -2079,20 +2084,25 @@ static __always_inline int _cond_resched(void) return dynamic_cond_resched(); } -#else +#else /* !CONFIG_PREEMPTION */ static inline int _cond_resched(void) { + klp_sched_try_switch(); return __cond_resched(); } -#endif /* CONFIG_PREEMPT_DYNAMIC */ +#endif /* PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ -#else +#else /* CONFIG_PREEMPTION && !CONFIG_PREEMPT_DYNAMIC */ -static inline int _cond_resched(void) { return 0; } +static inline int _cond_resched(void) +{ + klp_sched_try_switch(); + return 0; +} -#endif /* !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC) */ +#endif /* !CONFIG_PREEMPTION || CONFIG_PREEMPT_DYNAMIC */ #define cond_resched() ({ \ __might_resched(__FILE__, __LINE__, 0); \ diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c index 4d1f443778f7..b9e006632124 100644 --- a/kernel/livepatch/transition.c +++ b/kernel/livepatch/transition.c @@ -9,6 +9,7 @@ #include #include +#include #include "core.h" #include "patch.h" #include "transition.h" @@ -24,6 +25,25 @@ static int klp_target_state = KLP_UNDEFINED; static unsigned int klp_signals_cnt; +/* + * When a livepatch is in progress, enable klp stack checking in + * cond_resched(). This helps CPU-bound kthreads get patched. + */ +#if defined(CONFIG_PREEMPT_DYNAMIC) && defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) + +#define klp_cond_resched_enable() sched_dynamic_klp_enable() +#define klp_cond_resched_disable() sched_dynamic_klp_disable() + +#else /* !CONFIG_PREEMPT_DYNAMIC || !CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + +DEFINE_STATIC_KEY_FALSE(klp_sched_try_switch_key); +EXPORT_SYMBOL(klp_sched_try_switch_key); + +#define klp_cond_resched_enable() static_branch_enable(&klp_sched_try_switch_key) +#define klp_cond_resched_disable() static_branch_disable(&klp_sched_try_switch_key) + +#endif /* CONFIG_PREEMPT_DYNAMIC && CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + /* * This work can be performed periodically to finish patching or unpatching any * "straggler" tasks which failed to transition in the first attempt. @@ -338,6 +358,36 @@ static bool klp_try_switch_task(struct task_struct *task) return !ret; } +void __klp_sched_try_switch(void) +{ + if (likely(!klp_patch_pending(current))) + return; + + /* + * This function is called from cond_resched() which is called in many + * places throughout the kernel. Using the klp_mutex here might + * deadlock. + * + * Instead, disable preemption to prevent racing with other callers of + * klp_try_switch_task(). Thanks to task_call_func() they won't be + * able to switch this task while it's running. + */ + preempt_disable(); + + /* + * Make sure current didn't get patched between the above check and + * preempt_disable(). + */ + if (unlikely(!klp_patch_pending(current))) + goto out; + + klp_try_switch_task(current); + +out: + preempt_enable(); +} +EXPORT_SYMBOL(__klp_sched_try_switch); + /* * Sends a fake signal to all non-kthread tasks with TIF_PATCH_PENDING set. * Kthreads with TIF_PATCH_PENDING set are woken up. @@ -444,7 +494,8 @@ void klp_try_complete_transition(void) return; } - /* we're done, now cleanup the data structures */ + /* Done! Now cleanup the data structures. */ + klp_cond_resched_disable(); patch = klp_transition_patch; klp_complete_transition(); @@ -496,6 +547,8 @@ void klp_start_transition(void) set_tsk_thread_flag(task, TIF_PATCH_PENDING); } + klp_cond_resched_enable(); + klp_signals_cnt = 0; } @@ -588,14 +641,10 @@ void klp_reverse_transition(void) klp_target_state == KLP_PATCHED ? "patching to unpatching" : "unpatching to patching"); - klp_transition_patch->enabled = !klp_transition_patch->enabled; - - klp_target_state = !klp_target_state; - /* * Clear all TIF_PATCH_PENDING flags to prevent races caused by - * klp_update_patch_state() running in parallel with - * klp_start_transition(). + * klp_update_patch_state() or __klp_sched_try_switch() running in + * parallel with the reverse transition. */ read_lock(&tasklist_lock); for_each_process_thread(g, task) @@ -605,9 +654,16 @@ void klp_reverse_transition(void) for_each_possible_cpu(cpu) clear_tsk_thread_flag(idle_task(cpu), TIF_PATCH_PENDING); - /* Let any remaining calls to klp_update_patch_state() complete */ + /* + * Make sure all existing invocations of klp_update_patch_state() and + * __klp_sched_try_switch() see the cleared TIF_PATCH_PENDING before + * starting the reverse transition. + */ klp_synchronize_transition(); + /* All patching has stopped, now start the reverse transition. */ + klp_transition_patch->enabled = !klp_transition_patch->enabled; + klp_target_state = !klp_target_state; klp_start_transition(); } @@ -621,9 +677,9 @@ void klp_copy_process(struct task_struct *child) * the task flag up to date with the parent here. * * The operation is serialized against all klp_*_transition() - * operations by the tasklist_lock. The only exception is - * klp_update_patch_state(current), but we cannot race with - * that because we are current. + * operations by the tasklist_lock. The only exceptions are + * klp_update_patch_state(current) and __klp_sched_try_switch(), but we + * cannot race with them because we are current. */ if (test_tsk_thread_flag(current, TIF_PATCH_PENDING)) set_tsk_thread_flag(child, TIF_PATCH_PENDING); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e838feb6adc5..895d2a1fdcb3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8487,6 +8487,7 @@ EXPORT_STATIC_CALL_TRAMP(might_resched); static DEFINE_STATIC_KEY_FALSE(sk_dynamic_cond_resched); int __sched dynamic_cond_resched(void) { + klp_sched_try_switch(); if (!static_branch_unlikely(&sk_dynamic_cond_resched)) return 0; return __cond_resched(); @@ -8635,13 +8636,17 @@ int sched_dynamic_mode(const char *str) #error "Unsupported PREEMPT_DYNAMIC mechanism" #endif -void sched_dynamic_update(int mode) +DEFINE_MUTEX(sched_dynamic_mutex); +static bool klp_override; + +static void __sched_dynamic_update(int mode) { /* * Avoid {NONE,VOLUNTARY} -> FULL transitions from ever ending up in * the ZERO state, which is invalid. */ - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_enable(might_resched); preempt_dynamic_enable(preempt_schedule); preempt_dynamic_enable(preempt_schedule_notrace); @@ -8649,36 +8654,79 @@ void sched_dynamic_update(int mode) switch (mode) { case preempt_dynamic_none: - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_disable(might_resched); preempt_dynamic_disable(preempt_schedule); preempt_dynamic_disable(preempt_schedule_notrace); preempt_dynamic_disable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: none\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: none\n"); break; case preempt_dynamic_voluntary: - preempt_dynamic_enable(cond_resched); + if (!klp_override) + preempt_dynamic_enable(cond_resched); preempt_dynamic_enable(might_resched); preempt_dynamic_disable(preempt_schedule); preempt_dynamic_disable(preempt_schedule_notrace); preempt_dynamic_disable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: voluntary\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: voluntary\n"); break; case preempt_dynamic_full: - preempt_dynamic_disable(cond_resched); + if (!klp_override) + preempt_dynamic_disable(cond_resched); preempt_dynamic_disable(might_resched); preempt_dynamic_enable(preempt_schedule); preempt_dynamic_enable(preempt_schedule_notrace); preempt_dynamic_enable(irqentry_exit_cond_resched); - pr_info("Dynamic Preempt: full\n"); + if (mode != preempt_dynamic_mode) + pr_info("Dynamic Preempt: full\n"); break; } preempt_dynamic_mode = mode; } +void sched_dynamic_update(int mode) +{ + mutex_lock(&sched_dynamic_mutex); + __sched_dynamic_update(mode); + mutex_unlock(&sched_dynamic_mutex); +} + +#ifdef CONFIG_HAVE_PREEMPT_DYNAMIC_CALL + +static int klp_cond_resched(void) +{ + __klp_sched_try_switch(); + return __cond_resched(); +} + +void sched_dynamic_klp_enable(void) +{ + mutex_lock(&sched_dynamic_mutex); + + klp_override = true; + static_call_update(cond_resched, klp_cond_resched); + + mutex_unlock(&sched_dynamic_mutex); +} + +void sched_dynamic_klp_disable(void) +{ + mutex_lock(&sched_dynamic_mutex); + + klp_override = false; + __sched_dynamic_update(preempt_dynamic_mode); + + mutex_unlock(&sched_dynamic_mutex); +} + +#endif /* CONFIG_HAVE_PREEMPT_DYNAMIC_CALL */ + static int __init setup_preempt_mode(char *str) { int mode = sched_dynamic_mode(str); From patchwork Fri Feb 17 22:22:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Poimboeuf X-Patchwork-Id: 58787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp100510wrn; Fri, 17 Feb 2023 14:33:59 -0800 (PST) X-Google-Smtp-Source: AK7set8EbBfRCtkH80ejaxOFLuqOPivPTa4OdVB8vwSAAMWO8H0CK/TnRfQ9IlNobZwF02wRHMh8 X-Received: by 2002:aa7:de83:0:b0:4ab:ec2:3cd1 with SMTP id j3-20020aa7de83000000b004ab0ec23cd1mr2086286edv.25.1676673239244; Fri, 17 Feb 2023 14:33:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676673239; cv=none; d=google.com; s=arc-20160816; b=T8C0vXQO6XewN5Z0rzM9Nqs6yhBoPMymnKlDeHRKWdhRkeNM8HpPgCuAOGIsmumYKk X6HpCECVJ9GFmhAG/j1Yot+zZf4nHFSkNl9kFQSwIxtMgfOop054rgrAFkz9M5o0nvnl 4I+yD0BUyQ9D6bEKhuFql0qWWWfukvK5JyRT1Q2aZcNDkoFF4FgetsYCh6wisgZ6yyo5 JPfa4mMVGiaqBPo0SQ+mTMm5Z+DeYuPPJ2yl1yAMfTlsLHNt3ywmu3lRLJDO9/BGbFtf F7sgWY+6UrZSTWEYISntmuB7sH5wcdLtx44I2b16+2D7ZbeYHF6JMZEKcJ7vI6eaofCE FKqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+f41pgVe5xvkAd/Ki0t6NEZtUsqf2xiwTtTa3TntQds=; b=Riw74iHC/jr/GyKk0Lc7+vJuX2acTkeB1EIZz3fXIAVFFFwD5F4ur8DwLwVDCpcppn aZHZS/ShlDttUy8lODpVdF2EKH93BhVxYeIqPn8icrdnywDELEm+RhdXfrAf2G+79DWx xzuIilgbSqe70GFNOurdA37uy2KqbfjYXItXj1qfLKies/3fd8bVDH5/ZcQ45NxA5ZSD w20FT1ktnN5EC193ztkZvrMBJ6lbGL3TZ9TYiUw3+bMqSeX6Z22QaWvx9VEkUfCu3tSW CfwkCB1sSoV9SSBcpnteTFkV9cY8BzXBupJUNWOotEi16ogsPmMf+A9elId6m0MBNgby R+uQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=S4bH2giI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e16-20020a50fb90000000b004acbebbb1c1si6183695edq.262.2023.02.17.14.33.36; Fri, 17 Feb 2023 14:33:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=S4bH2giI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229862AbjBQWXJ (ORCPT + 99 others); Fri, 17 Feb 2023 17:23:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229553AbjBQWXC (ORCPT ); Fri, 17 Feb 2023 17:23:02 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6292D5CF03; Fri, 17 Feb 2023 14:23:01 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id ED2DB6207F; Fri, 17 Feb 2023 22:23:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EA130C433AE; Fri, 17 Feb 2023 22:22:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676672580; bh=/+ANatHz1yxjGMA7ryG0c/Km7p/0KZLeF6+BAXFBfTI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S4bH2giIvuXGjgMxOlCrWfG7eHYRcyJ+RLkxqnp/b8gRM2nelDloLr4IUaRWuBpmW FXqUXTghpwLpCOd47jUXygi6IkGoRxW7BUtPgGElpBc3G6U/Pw5T4j59f4CHtXxYPN vDcHN8ePhyBF3Qfuu4QtqTU4gh/xe4JsMCqkXt8T1PR0L+U5RgbgYqDRkFp4+KdacG HAzPLIYts1Ss9b/VMDz3PaxrIF0ZeXgQg8dFvo17MsHPdsBm/UMSHIDN1By2JxIVow mw4KOjUpBl87uJroVqF3v4zXuhCsvc1THfNy4V+BC9i2j/0piMEd2aVCCc84NA0sM6 FuEbu1N7FQxzw== From: Josh Poimboeuf To: live-patching@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Seth Forshee , Peter Zijlstra , Song Liu , Mark Rutland , Petr Mladek , Joe Lawrence , Miroslav Benes , Jiri Kosina , Ingo Molnar , Rik van Riel Subject: [PATCH v2 3/3] vhost: Fix livepatch timeouts in vhost_worker() Date: Fri, 17 Feb 2023 14:22:56 -0800 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Content-type: text/plain X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758119318314133361?= X-GMAIL-MSGID: =?utf-8?q?1758119318314133361?= Livepatch timeouts were reported due to busy vhost_worker() kthreads. Now that cond_resched() can do livepatch task switching, use cond_resched() in vhost_worker(). That's the better way to conditionally call schedule() anyway. Reported-by: Seth Forshee (DigitalOcean) Link: https://lkml.kernel.org/lkml/20230120-vhost-klp-switching-v1-0-7c2b65519c43@kernel.org Tested-by: Seth Forshee (DigitalOcean) Reviewed-by: Petr Mladek Signed-off-by: Josh Poimboeuf --- drivers/vhost/vhost.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 43c9770b86e5..87e3cf12da1c 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -363,8 +363,7 @@ static int vhost_worker(void *data) kcov_remote_start_common(dev->kcov_handle); work->fn(work); kcov_remote_stop(); - if (need_resched()) - schedule(); + cond_resched(); } } kthread_unuse_mm(dev->mm);