From patchwork Mon Oct 17 14:53:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 3508 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1493417wrs; Mon, 17 Oct 2022 07:56:20 -0700 (PDT) X-Google-Smtp-Source: AMsMyM493FyC4pLZ//eDIxWurWlza5L0E8ju6e36B/AXoJ2TeUpavSBq4+3f9/XKeJmLg3bxD4wt X-Received: by 2002:a17:907:70b:b0:740:ef93:2ffb with SMTP id xb11-20020a170907070b00b00740ef932ffbmr8802005ejb.93.1666018580361; Mon, 17 Oct 2022 07:56:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666018580; cv=none; d=google.com; s=arc-20160816; b=SDq3nYdmBebkmVNcPuUAse5GFxmuUTonjxwHWsd/R9Qpr5K4ycQGXj7UBqnUvDZZ0G 0RnmAY74oz32uYgCuJbdHJ+gLvQETH9vQsfq8ZAWCgb34HbjlCEKPBr+vqARkRtzHCkM xxl5VXWGraSlU1kzhEM0d+Z/bKdvZEwlYF3YFGI0U0x/Xp5mhPG1prT1tLOrPtNE1z7a UoOUD70YSZpN7erTi8V3yPPsgqRYPRfVuPedWu6IXeWW6LQbX7kVE1hxivE04ig4ONkb gkNcvQavHXlOPgLb7OxHNDA19r09e5x/lr9pwJc8QTY8Khr7tK/R2d3TY0ccFeFI4fAo AwLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=1rbeIB8LitiEsAb/IgJkIxwr56dTJtOb8kqOCz0x2uk=; b=VgSi05kXyJ2wGp9KccMWkmAOZcAWCbOLCk384Wg7IueCruSux2oxXtKEfOzXzDVBXz lQB7Dn/qXOWAOb10/G1gOqAHC5Eh+GmiqXYKeB02frX+bcemKoVfiD2D5LZWHdiF9KCa 3wiqVRU1VNGFolM5rngXbgRAhQY7xLZVt7SuqTGAw+ruqbzKbR0yuDqxeHmpRuF8GkIW QvtmoXCOLko8hYmO7WiT1IQOE5F9oflCQBcvfdzkRb/DiP6CGZHU0yf6r3LCvTaiviDj nPDhvI6RnzLYN92KKDc0TSPg9sUXVudU4/E+daq9QrjAei4/olQVX/DQhf1jrw+2KBiO jgsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=mFEXRUeb; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 30-20020a508e1e000000b00458b204f294si7857026edw.298.2022.10.17.07.55.55; Mon, 17 Oct 2022 07:56:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=mFEXRUeb; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231254AbiJQOzF (ORCPT + 99 others); Mon, 17 Oct 2022 10:55:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230334AbiJQOyB (ORCPT ); Mon, 17 Oct 2022 10:54:01 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0CD2691A8; Mon, 17 Oct 2022 07:53:45 -0700 (PDT) Date: Mon, 17 Oct 2022 14:53:36 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1666018417; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1rbeIB8LitiEsAb/IgJkIxwr56dTJtOb8kqOCz0x2uk=; b=mFEXRUebwB0ceuQ8qbbEG6P86tI7rAGvb1nTJwNiKdnMD2DHpQpjdZ0gNFiHru6Gwa9otq nFuL0EpqmloSqJOlHp6wFl/zS55X65XwVaDJwZObl5p6CL6pRM6qcyNNNVIx3VoGHYoPpQ XadnJW3KrC9GU5RQWo6F84l+fCqqE8NNupK+wDi3Yf6F8HPxMZwus91S7hVHvhjf3xql8i cAjmKMdgwzdULuBp0okZLjIPKl2BPSO0lGNmx7YbHdlpJ0+H3UrNc6Fdw4sa48XSJD1gTM c55QFMx2/xX+D67CEyUnBXrlwo2jGksoA4h0RNY3CPfVQhxRc2l6/bbc+lGVbg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1666018417; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1rbeIB8LitiEsAb/IgJkIxwr56dTJtOb8kqOCz0x2uk=; b=dLINRy4T0JG/DJrkQuY3IcoXkIPVWp9tBssgWEzWthATzXSTkarB6xLqsvePtuSQrP9qm1 YLsrJevnj/oknRCg== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/core] static_call: Add call depth tracking support Cc: "Peter Zijlstra (Intel)" , Thomas Gleixner , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20220915111148.306100465@infradead.org> References: <20220915111148.306100465@infradead.org> MIME-Version: 1.0 Message-ID: <166601841630.401.8138685499148178070.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746947098654963489?= X-GMAIL-MSGID: =?utf-8?q?1746947098654963489?= The following commit has been merged into the x86/core branch of tip: Commit-ID: 7825451fa4dc04660f1f53d236e4302161d0ebd1 Gitweb: https://git.kernel.org/tip/7825451fa4dc04660f1f53d236e4302161d0ebd1 Author: Peter Zijlstra AuthorDate: Thu, 15 Sep 2022 13:11:31 +02:00 Committer: Peter Zijlstra CommitterDate: Mon, 17 Oct 2022 16:41:16 +02:00 static_call: Add call depth tracking support When indirect calls are switched to direct calls then it has to be ensured that the call target is not the function, but the call thunk when call depth tracking is enabled. But static calls are available before call thunks have been set up. Ensure a second run through the static call patching code after call thunks have been created. When call thunks are not enabled this has no side effects. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20220915111148.306100465@infradead.org --- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 18 ++++++++++++++++++ arch/x86/kernel/static_call.c | 1 + include/linux/static_call.h | 2 ++ kernel/static_call_inline.c | 23 ++++++++++++++++++----- 5 files changed, 44 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h index 4c416b2..07ac257 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -91,11 +91,16 @@ struct callthunk_sites { extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod); +extern void *callthunks_translate_call_dest(void *dest); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod) {} +static __always_inline void *callthunks_translate_call_dest(void *dest) +{ + return dest; +} #endif #ifdef CONFIG_SMP diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index dfe7fff..0710036 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -271,10 +272,27 @@ void __init callthunks_patch_builtin_calls(void) pr_info("Setting up call depth tracking\n"); mutex_lock(&text_mutex); callthunks_setup(&cs, &builtin_coretext); + static_call_force_reinit(); thunks_initialized = true; mutex_unlock(&text_mutex); } +void *callthunks_translate_call_dest(void *dest) +{ + void *target; + + lockdep_assert_held(&text_mutex); + + if (!thunks_initialized || skip_addr(dest)) + return dest; + + if (!is_coretext(NULL, dest)) + return dest; + + target = patch_dest(dest, false); + return target ? : dest; +} + #ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, struct module *mod) diff --git a/arch/x86/kernel/static_call.c b/arch/x86/kernel/static_call.c index 5d3844a..2ebc338 100644 --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -34,6 +34,7 @@ static void __ref __static_call_transform(void *insn, enum insn_type type, switch (type) { case CALL: + func = callthunks_translate_call_dest(func); code = text_gen_insn(CALL_INSN_OPCODE, insn, func); if (func == &__static_call_return0) { emulate = code; diff --git a/include/linux/static_call.h b/include/linux/static_call.h index df53bed..141e6b1 100644 --- a/include/linux/static_call.h +++ b/include/linux/static_call.h @@ -162,6 +162,8 @@ extern void arch_static_call_transform(void *site, void *tramp, void *func, bool extern int __init static_call_init(void); +extern void static_call_force_reinit(void); + struct static_call_mod { struct static_call_mod *next; struct module *mod; /* for vmlinux, mod == NULL */ diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c index dc5665b..639397b 100644 --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -15,7 +15,18 @@ extern struct static_call_site __start_static_call_sites[], extern struct static_call_tramp_key __start_static_call_tramp_key[], __stop_static_call_tramp_key[]; -static bool static_call_initialized; +static int static_call_initialized; + +/* + * Must be called before early_initcall() to be effective. + */ +void static_call_force_reinit(void) +{ + if (WARN_ON_ONCE(!static_call_initialized)) + return; + + static_call_initialized++; +} /* mutex to protect key modules/sites */ static DEFINE_MUTEX(static_call_mutex); @@ -475,7 +486,8 @@ int __init static_call_init(void) { int ret; - if (static_call_initialized) + /* See static_call_force_reinit(). */ + if (static_call_initialized == 1) return 0; cpus_read_lock(); @@ -490,11 +502,12 @@ int __init static_call_init(void) BUG(); } - static_call_initialized = true; - #ifdef CONFIG_MODULES - register_module_notifier(&static_call_module_nb); + if (!static_call_initialized) + register_module_notifier(&static_call_module_nb); #endif + + static_call_initialized = 1; return 0; } early_initcall(static_call_init);