From patchwork Mon Jan 9 13:58:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40879 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2169576wrt; Mon, 9 Jan 2023 05:59:27 -0800 (PST) X-Google-Smtp-Source: AMrXdXtmgj/CwQHt+ZVL6ceWvkDYFTrcsfQZoHaDKqJ5z4PnTCDLAWeoUJI3XcpQs0y3NotkZHoa X-Received: by 2002:a62:ed08:0:b0:577:272f:fdb with SMTP id u8-20020a62ed08000000b00577272f0fdbmr53043501pfh.29.1673272767000; Mon, 09 Jan 2023 05:59:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272766; cv=none; d=google.com; s=arc-20160816; b=RPGlojGwxufbk9mWzmxWttRAJuU6UcdrfQaWcStgcBofdzyzzgEN78qszZlijGgB7P JmjIGQNmjW8+Ew+JxzIzpezqi3fyJwy8l2TLNsA4gL22K5UEHlN4VLlu32PKcl72m/lU /dPoCth6/3bACs9EkwjQpCvX9VVmc8eLouDk3SwBWDT3bdNWNKJ0H4i3+5UHhQ0cCaY0 jDapRC0Aet2OfZ8wItPN2bWPW2nNYPFnS9r0+gbZ9w85E6N0EMyBoJso6UsMqfIJsA63 bW3LxMcmfMuanwO5QjmkJnAjVLPp3Vlu4/AgUMmatYVgeFd5eo+PRD+HblOMNRp1cyue p+6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=QC6IVpFz0PA6Zbil2AxcWuJx+5gzCmPeehhx4+03dC4=; b=yV8dlPTa+HnavkVhiJVb695nki72oz6qc7+UTjC3LwKA+VANk3SR5lcDaGkvFsHOcG 7uERgGn2jyC3fogsb1pNJNEnLPG3E++NgL5S3on2vqQSbINAo5Y+RNaxzb7+BuauXGyV 2Th0ErbpNer5DpMTZyX7/pX0pzmwg44oVGu4uGa+WM0d7ZdXgkhzcDaJplWkrLxQhEbc N9LVN0lpWaGJPqiDx7J2g1E8vpeU+L11lNdk7lKlQCxSoFV/rGzRXgIIRQg634vmxsU5 FB5wkpHalBJ2sypSqA9mEwupiIdAI08K2/ZbTpcGPmkUPlnONoOM4s4AsFavga1XYw6i /P0g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f7-20020a056a001ac700b005818631179asi10400600pfv.95.2023.01.09.05.59.13; Mon, 09 Jan 2023 05:59:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234481AbjAIN6z (ORCPT + 99 others); Mon, 9 Jan 2023 08:58:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237241AbjAIN6l (ORCPT ); Mon, 9 Jan 2023 08:58:41 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id DE339FCA; Mon, 9 Jan 2023 05:58:40 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9B4601515; Mon, 9 Jan 2023 05:59:22 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C43293F23F; Mon, 9 Jan 2023 05:58:38 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 1/8] Compiler attributes: GCC function alignment workarounds Date: Mon, 9 Jan 2023 13:58:21 +0000 Message-Id: <20230109135828.879136-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553665263390474?= X-GMAIL-MSGID: =?utf-8?q?1754553665263390474?= From local testing, contemporary versions of of GCC (e.g. GCC 12.2.0) don't respect '-falign-functions=N' in all cases. This is unfortunate, as (for non-zero values of N) CONFIG_FUNCTION_ALIGNMENT=N will set '-falign-functions=N', but this won't take effect for all functions. LLVM appears to respect '-falign-functions=N' in call cases. Today, for x86 this turns out to be functionally benign, though it does somewhat undermine the CONFIG_FUNCTION_ALIGNMENT option, and it means that CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B is not as robust as we'd like. On arm64 we'd like to use CONFIG_FUNCTION_ALIGNMENT to implement ftrace functionality, and we'll require function alignment to be respected for functional reasons. As far as I can tell, GCC doesn't respect '-falign-functions=N': * When the __weak__ attribute is used GCC seems to forget the alignment specified by '-falign-functions=N', but will respect the '__aligned__(N)' function attribute. Thus, we can work around this by explciitly setting the alignment for weak functions. * When the __cold__ attribute is used GCC seems to forget the alignment specified by '-falign-functions=N', and also doesn't seem to respect the '__aligned__(N)' function attribute. The only way to work around this is to not use the __cold__ attibute. This patch implements workarounds for these two cases, using a function attribute to set the alignment of __weak__ functions, and preventing the use of the __cold__ attribute when CONFIG_FUNCTION_ALIGNMENT is non-zero. I've tested this by selecting CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B=y, building and booting a kernel, and looking for misaligned text symbols: * arm64: Before: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 4939 After: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 908 * x86_64: Before: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 7969 After: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 2057 With the patch applied, the remaining unaligned text labels are a combination of static call trampolines, non-function labels in assembly, and ACPICA functions, which will be dealt with in subsequent patches. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon Cc: Miguel Ojeda Cc: Nick Desaulniers --- include/linux/compiler_attributes.h | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h index 898b3458b24a0..dcb7ac67b764f 100644 --- a/include/linux/compiler_attributes.h +++ b/include/linux/compiler_attributes.h @@ -33,6 +33,17 @@ #define __aligned(x) __attribute__((__aligned__(x))) #define __aligned_largest __attribute__((__aligned__)) +/* + * Contemporary versions of GCC (e.g. 12.2.0) don't always respect + * '-falign-functions=N', and require alignment to be specificed via a function + * attribute in some cases. + */ +#if CONFIG_FUNCTION_ALIGNMENT > 0 +#define __function_aligned __aligned(CONFIG_FUNCTION_ALIGNMENT) +#else +#define __function_aligned +#endif + /* * Note: do not use this directly. Instead, use __alloc_size() since it is conditionally * available and includes other attributes. For GCC < 9.1, __alloc_size__ gets undefined @@ -78,8 +89,15 @@ /* * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-cold-function-attribute * gcc: https://gcc.gnu.org/onlinedocs/gcc/Label-Attributes.html#index-cold-label-attribute + * + * GCC drops function alignment when the __cold__ attribute is used. Avoid the + * __cold__ attribute if function alignment is required. */ +#if !defined(CONFIG_CC_IS_GCC) || (CONFIG_FUNCTION_ALIGNMENT == 0) #define __cold __attribute__((__cold__)) +#else +#define __cold +#endif /* * Note the long name. @@ -369,8 +387,11 @@ /* * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-weak-function-attribute * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html#index-weak-variable-attribute + * + * GCC drops function alignment when the __weak__ attribute is used. This can + * be restored with function attributes. */ -#define __weak __attribute__((__weak__)) +#define __weak __attribute__((__weak__)) __function_aligned /* * Used by functions that use '__builtin_return_address'. These function From patchwork Mon Jan 9 13:58:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40880 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2169700wrt; Mon, 9 Jan 2023 05:59:41 -0800 (PST) X-Google-Smtp-Source: AMrXdXtmq4+0XK28vupSJ0VxFLYMMnyO+A99JgL03eFNfKfzQeRBAKarIQMCnv+r9R3PkHpiZ+v3 X-Received: by 2002:a05:6a20:9b87:b0:ac:44ab:be3b with SMTP id mr7-20020a056a209b8700b000ac44abbe3bmr77466553pzb.60.1673272781605; Mon, 09 Jan 2023 05:59:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272781; cv=none; d=google.com; s=arc-20160816; b=m3XfB3M0KCt3LcpKX4if4rezoyTpC6Kmc/l8OEBGBobmEbPT7qS0Nu9HdxAa/yazmf TPB3RqVx/5j0bPdvk8DEezhwilusQY1R5K+7GgXR+SVivh7WbRvsbqGhcsIIyW/GFt1x OpTB5PPaOucfuvRLk71X0t6WXa3N9f+2mhziF0wfEnlGa/QCIEDBq7CiQR+ouatwPMVP rZWABtz/YJa5+76B3+XSktUb+V04B3+8eDiZ0++Lqh3SnTzWOy36axc0VHhBOqQPj3bM o1A1cAyOOnqLgKsxh+wblIU9AurWrbvurqyc6BAvS1NI9Nd06NGBKx2gs05L8ctUYLQP dqQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=B/d9Pij30xbOyFawkCQpqEZ9o8SJzgaQZIB7fSTKZOo=; b=h2vxt4CdnnuPBqUxzWoMgaSxLw/QiTzj3zYuJxXQwujbuPRnpsyNt4vWPxvwEMmfpX SDH1KVrHQHNe+H/o1Si1PCeHMFA7mkz3xLf3PLD7MyOH/q2PBG21pLk4BsltOHFJVbu0 7DEDfwmJ4t35qya4AOCbCiO8qwx8zq+qBH+jqutXtTos7B40S/r9wZaRjahq/wqmMB66 Lmucgswhpy4CwXBt0ky55pG72V8cuIhMHWtVUf8WuXsskasxo1EbK2+9kACTn12ylqAy NV830bn7c5f59+CpurjhgDm4X49GtdBc1W9k1yrH29xyJABc7WpL3jijakonwUcN4tsD z0hg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h17-20020a635311000000b0046ef23ae9a6si9643195pgb.848.2023.01.09.05.59.29; Mon, 09 Jan 2023 05:59:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237083AbjAIN7H (ORCPT + 99 others); Mon, 9 Jan 2023 08:59:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237251AbjAIN6o (ORCPT ); Mon, 9 Jan 2023 08:58:44 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9879525FE; Mon, 9 Jan 2023 05:58:43 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5ECC71650; Mon, 9 Jan 2023 05:59:25 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 872733F23F; Mon, 9 Jan 2023 05:58:41 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Len Brown , "Rafael J . Wysocki" , Robert Moore Cc: catalin.marinas@arm.com, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, revest@chromium.org, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 2/8] ACPI: Don't build ACPICA with '-Os' Date: Mon, 9 Jan 2023 13:58:22 +0000 Message-Id: <20230109135828.879136-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553680140498277?= X-GMAIL-MSGID: =?utf-8?q?1754553680140498277?= The ACPICA code has been built with '-Os' since the beginning of git history, though there's no explanatory comment as to why. This is unfortunate as building with '-Os' overrides -falign-functions, which prevents CONFIG_FUNCITON_ALIGNMENT and CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B from having their expected effect on the ACPICA code. This is doubly unfortunate as in subsequent patches arm64 will depend upon CONFIG_FUNCTION_ALIGNMENT for its ftrace implementation. Drop the '-Os' flag when building the ACPICA code. With this removed, the code builds cleanly and works correctly in testing so far. I've tested this by selecting CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B=y, building and booting a kernel using ACPI, and looking for misaligned text symbols: * arm64: Before: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 908 # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | grep acpi | wc -l 576 After: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 322 # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | grep acpi | wc -l 0 * x86_64: Before: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 2057 # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | grep acpi | wc -l 706 After: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 1351 # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | grep acpi | wc -l 0 With the patch applied, the remaining unaligned text labels are a combination of static call trampolines and labels in assembly, which will be dealt with in subsequent patches. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Len Brown Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Rafael J. Wysocki Cc: Robert Moore Cc: Steven Rostedt Cc: Will Deacon Cc: linux-acpi@vger.kernel.org Acked-by: Rafael J. Wysocki --- drivers/acpi/acpica/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/acpi/acpica/Makefile b/drivers/acpi/acpica/Makefile index 9e0d95d76fff7..30f3fc13c29d1 100644 --- a/drivers/acpi/acpica/Makefile +++ b/drivers/acpi/acpica/Makefile @@ -3,7 +3,7 @@ # Makefile for ACPICA Core interpreter # -ccflags-y := -Os -D_LINUX -DBUILDING_ACPICA +ccflags-y := -D_LINUX -DBUILDING_ACPICA ccflags-$(CONFIG_ACPI_DEBUG) += -DACPI_DEBUG_OUTPUT # use acpi.o to put all files here into acpi.o modparam namespace From patchwork Mon Jan 9 13:58:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40886 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2171028wrt; Mon, 9 Jan 2023 06:01:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXvwc9gn4cjP0p+7Ma7EwwgWzMCvF0YMMt48obH2P2Sn5cSBY+Vs629vIOjdLHsEDi4Qo361 X-Received: by 2002:a17:90a:ebc6:b0:226:5900:f2f4 with SMTP id cf6-20020a17090aebc600b002265900f2f4mr34849887pjb.4.1673272891999; Mon, 09 Jan 2023 06:01:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272891; cv=none; d=google.com; s=arc-20160816; b=gV6r5aB9hBnwEerYEoWMeHr6c3zIOAAwG/Se8hjR6OQN94a+TzL14RZkeUC+vYTJ6Q +MkLWRyi6f6rr3YqYqbay/I+4AOvQffRAtll96E/5Kcqrvbims14NUAZJeL1lzMGAF7k 4vDq+264YMhv3QM20RYQqKQnOdyt7QV9jyOphjL4NVzC+Y6ZbhHbFnWmNEUdc8iIoHN+ cTHWj7CseSKEkHvs7Eb6r2yxt7YLp9vrtEyYRW1j5PxF8bZxKg6NZVBcJrGQkmPWoOLK pT/d6bx65R+ZoqZ9/+kKBBSgWqGdAkEAXXsTzHHQGOVJ4wXEnI+BImD2F7I+b3+82dk8 Nf2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=uS6ssPWGHEZJFSKX4hxjdcKxWoVnBVtrpoOuY5hYNNg=; b=C8xTiSxjmCgoWBpPUHZ6+69DKR9SnpZ9FfBFJ0peAAX320K/Inl5BewtnaftyQsH9n QSSozgDtzEpoP7RivOAx/Qd74QRV9MeoLcQ2GJCodTqnjIjmUPYazMCq1p1UvgZl3HUN FSp8rurwVcWS1IlkH+ilzpUQEqtYvu9wnFZzV6uDxC79ildh3XhZaIToqn7FKM+VuGYO mNINF+wZmzuPlOJAfZKsEhI+zgYY434ZkIxd2NDN1ZJAkIaLe5ewro/28VpG29HieUvV 1xb7iKN7vPRP1bBC2s8UipRaPVh2ky2DxKk39DKL/22AucrqEJcphDul9RjyfhenCgCu 4kOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mn20-20020a17090b189400b00226902ba17csi11609294pjb.109.2023.01.09.06.01.19; Mon, 09 Jan 2023 06:01:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237240AbjAIOAc (ORCPT + 99 others); Mon, 9 Jan 2023 09:00:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237264AbjAIN6r (ORCPT ); Mon, 9 Jan 2023 08:58:47 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 419D3F6F; Mon, 9 Jan 2023 05:58:46 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 05F2B176B; Mon, 9 Jan 2023 05:59:28 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 308803F23F; Mon, 9 Jan 2023 05:58:44 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 3/8] arm64: Extend support for CONFIG_FUNCTION_ALIGNMENT Date: Mon, 9 Jan 2023 13:58:23 +0000 Message-Id: <20230109135828.879136-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553795916692472?= X-GMAIL-MSGID: =?utf-8?q?1754553795916692472?= On arm64 we don't align assembly funciton in the same way as C functions. This somewhat limits the utility of CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B for testing. Follow the example of x86, and align assembly functions in the same way as C functions. I've tested this by selecting CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B=y, building and booting a kernel, and looking for misaligned text symbols: Before: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 322 After: # grep ' [Tt] ' /proc/kallsyms | grep -iv '[048c]0 [Tt] ' | wc -l 115 The remaining unaligned text symbols are a combination of non-function labels in assembly and early position-independent code which is built with '-Os'. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- arch/arm64/include/asm/linkage.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h index 1436fa1cde24d..df18a3446ce82 100644 --- a/arch/arm64/include/asm/linkage.h +++ b/arch/arm64/include/asm/linkage.h @@ -5,8 +5,14 @@ #include #endif -#define __ALIGN .align 2 -#define __ALIGN_STR ".align 2" +#if CONFIG_FUNCTION_ALIGNMENT > 0 +#define ARM64_FUNCTION_ALIGNMENT CONFIG_FUNCTION_ALIGNMENT +#else +#define ARM64_FUNCTION_ALIGNMENT 4 +#endif + +#define __ALIGN .balign ARM64_FUNCTION_ALIGNMENT +#define __ALIGN_STR ".balign " #ARM64_FUNCTION_ALIGNMENT /* * When using in-kernel BTI we need to ensure that PCS-conformant From patchwork Mon Jan 9 13:58:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40887 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2171272wrt; Mon, 9 Jan 2023 06:01:47 -0800 (PST) X-Google-Smtp-Source: AMrXdXso39i7mJYYVM94q8LNpKtD75hsx4RiQX2DCDLlQKM5eoZAYfatr0TrW5yaNaDRdetiTbL1 X-Received: by 2002:a05:6a21:998b:b0:a6:713d:4343 with SMTP id ve11-20020a056a21998b00b000a6713d4343mr105673004pzb.5.1673272906721; Mon, 09 Jan 2023 06:01:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272906; cv=none; d=google.com; s=arc-20160816; b=bjC16NqN+VwYjXJSkEpIQ3f0BKWJsH0ZwV/gg6MaW66foBqofmFRZIhjN7E+XKm/Ba UAMkZLJWwFgPCaqZ4TTJVfP5CLh4RzB3foSWmi2TGHlzi6NFVLDliG/SwXWA/yMmmUpJ 8ijW6I2o38G/mkXlhOIjH/tnSmQXRttrS7XoEwc1es/jx+t6IPRPsohSkPbImhQ6PFDZ 6UFUkk+vSuC945IC1+qEw9vjPz0KaE9vTG7J0pr/na6IWWdMquU4grv/t412NNpHKUsk JjjirFzJQ8iBIW6Xsy8dXO5wZOR0PgwMcoFkmCZ/XSrS1kimoWlUaP2OuwxxakHsgEqz MG+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=oqepHMY6ohgv6X8AjaxfHTldYFuqO97QSiR9jkczgI8=; b=VnHJSeua3eT8QWxpFn/nXNth3ob+vLHfwlPYL3wm5n9IKVI/4bAtKQwS2461qjhO5X GDdJX+pyamzk5rt607IivS6NIrurbCwbZAqvUhy9fjYIyeii8tC1ZoCMoQyXEmH+z4RD n3EFVfGV0Xb+3lHrqeGVGdvET0As8visfVlI3DDM5ir4WSgl9iJkgAtXvkwxh5q69v9Q PhIE09iZGKYwGx8J/MH/Ae3LfPvj1nSA/oSxGcI4VGlcDurcDzGtHdXHuql+Cn4l+sq8 ybww2i09NdyDMyDQg54tM7ldaCeNT4pLFqKCK3fnnIKU6CfXt0gfUJEBsVSqyKsrfOBC 2Gxw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u6-20020a654c06000000b0046ef21af717si10144261pgq.683.2023.01.09.06.01.33; Mon, 09 Jan 2023 06:01:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234203AbjAIOAl (ORCPT + 99 others); Mon, 9 Jan 2023 09:00:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231886AbjAIN6u (ORCPT ); Mon, 9 Jan 2023 08:58:50 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id CF601BCC; Mon, 9 Jan 2023 05:58:48 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 93076176C; Mon, 9 Jan 2023 05:59:30 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BDC393F23F; Mon, 9 Jan 2023 05:58:46 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 4/8] ftrace: Add DYNAMIC_FTRACE_WITH_CALL_OPS Date: Mon, 9 Jan 2023 13:58:24 +0000 Message-Id: <20230109135828.879136-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553810976200231?= X-GMAIL-MSGID: =?utf-8?q?1754553810976200231?= Architectures without dynamic ftrace trampolines incur an overhead when multiple ftrace_ops are enabled with distinct filters. in these cases, each call site calls a common trampoline which uses ftrace_ops_list_func() to iterate over all enabled ftrace functions, and so incurs an overhead relative to the size of this list (including RCU protection overhead). Architectures with dynamic ftrace trampolines avoid this overhead for call sites which have a single associated ftrace_ops. In these cases, the dynamic trampoline is customized to branch directly to the relevant ftrace function, avoiding the list overhead. On some architectures it's impractical and/or undesirable to implement dynamic ftrace trampolines. For example, arm64 has limited branch ranges and cannot always directly branch from a call site to an arbitrary address (e.g. from a kernel text address to an arbitrary module address). Calls from modules to core kernel text can be indirected via PLTs (allocated at module load time) to address this, but the same is not possible from calls from core kernel text. Using an indirect branch from a call site to an arbitrary trampoline is possible, but requires several more instructions in the function prologue (or immediately before it), and/or comes with far more complex requirements for patching. Instead, this patch adds a new option, where an architecture can associate each call site with a pointer to an ftrace_ops, placed at a fixed offset from the call site. A shared trampoline can recover this pointer and call ftrace_ops::func() without needing to go via ftrace_ops_list_func(), avoiding the associated overhead. This avoids issues with branch range limitations, and avoids the need to allocate and manipulate dynamic trampolines, making it far simpler to implement and maintain, while having similar performance characteristics. Note that this allows for dynamic ftrace_ops to be invoked directly from an architecture's ftrace_caller trampoline, whereas existing code forces the use of ftrace_ops_get_list_func(), which is in part necessary to permit the ftrace_ops to be freed once unregistereed *and* to avoid branch/address-generation range limitation on some architectures (e.g. where ops->func is a module address, and may be outside of the direct branch range for callsites within the main kernel image). The CALL_OPS approach avoids this problems and is safe as: * The existing synchronization in ftrace_shutdown() using ftrace_shutdown() using synchronize_rcu_tasks_rude() (and synchronize_rcu_tasks()) ensures that no tasks hold a stale reference to an ftrace_ops (e.g. in the middle of the ftrace_caller trampoline, or while invoking ftrace_ops::func), when that ftrace_ops is unregistered. Arguably this could also be relied upon for the existing scheme, permitting dynamic ftrace_ops to be invoked directly when ops->func is in range, but this will require additional logic to handle branch range limitations, and is not handled by this patch. * Each callsite's ftrace_ops pointer literal can hold any valid kernel address, and is updated atomically. As an architecture's ftrace_caller trampoline will atomically load the ops pointer then derefrence ops->func, there is no risk of invoking ops->func with a mismatches ops pointer, and updates to the ops pointer do not require special care. A subsequent patch will implement architectures support for arm64. There should be no functional change as a result of this patch alone. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- include/linux/ftrace.h | 15 +++++- kernel/trace/Kconfig | 7 +++ kernel/trace/ftrace.c | 109 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 124 insertions(+), 7 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 99f1146614c04..5eeb2776124c5 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -57,6 +57,9 @@ void arch_ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip); void arch_ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); #endif +extern const struct ftrace_ops ftrace_nop_ops; +extern const struct ftrace_ops ftrace_list_ops; +struct ftrace_ops *ftrace_find_unique_ops(struct dyn_ftrace *rec); #endif /* CONFIG_FUNCTION_TRACER */ /* Main tracing buffer and events set up */ @@ -563,6 +566,8 @@ bool is_ftrace_trampoline(unsigned long addr); * IPMODIFY - the record allows for the IP address to be changed. * DISABLED - the record is not ready to be touched yet * DIRECT - there is a direct function to call + * CALL_OPS - the record can use callsite-specific ops + * CALL_OPS_EN - the function is set up to use callsite-specific ops * * When a new ftrace_ops is registered and wants a function to save * pt_regs, the rec->flags REGS is set. When the function has been @@ -580,9 +585,11 @@ enum { FTRACE_FL_DISABLED = (1UL << 25), FTRACE_FL_DIRECT = (1UL << 24), FTRACE_FL_DIRECT_EN = (1UL << 23), + FTRACE_FL_CALL_OPS = (1UL << 22), + FTRACE_FL_CALL_OPS_EN = (1UL << 21), }; -#define FTRACE_REF_MAX_SHIFT 23 +#define FTRACE_REF_MAX_SHIFT 21 #define FTRACE_REF_MAX ((1UL << FTRACE_REF_MAX_SHIFT) - 1) #define ftrace_rec_count(rec) ((rec)->flags & FTRACE_REF_MAX) @@ -820,7 +827,8 @@ static inline int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) */ extern int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr); -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) || \ + defined(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) /** * ftrace_modify_call - convert from one addr to another (no nop) * @rec: the call site record (e.g. mcount/fentry) @@ -833,6 +841,9 @@ extern int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr); * what we expect it to be, and then on success of the compare, * it should write to the location. * + * When using call ops, this is called when the associated ops change, even + * when (addr == old_addr). + * * The code segment at @rec->ip should be a caller to @old_addr * * Return must be: diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 197545241ab83..5df427a2321df 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -42,6 +42,9 @@ config HAVE_DYNAMIC_FTRACE_WITH_REGS config HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS bool +config HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS + bool + config HAVE_DYNAMIC_FTRACE_WITH_ARGS bool help @@ -257,6 +260,10 @@ config DYNAMIC_FTRACE_WITH_DIRECT_CALLS depends on DYNAMIC_FTRACE_WITH_REGS depends on HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS +config DYNAMIC_FTRACE_WITH_CALL_OPS + def_bool y + depends on HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS + config DYNAMIC_FTRACE_WITH_ARGS def_bool y depends on DYNAMIC_FTRACE diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 442438b93fe98..c0b219ab89d2f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -125,6 +125,33 @@ struct ftrace_ops global_ops; void ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +/* + * Stub used to invoke the list ops without requiring a separate trampoline. + */ +const struct ftrace_ops ftrace_list_ops = { + .func = ftrace_ops_list_func, + .flags = FTRACE_OPS_FL_STUB, +}; + +static void ftrace_ops_nop_func(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *op, + struct ftrace_regs *fregs) +{ + /* do nothing */ +} + +/* + * Stub used when a call site is disabled. May be called transiently by threads + * which have made it into ftrace_caller but haven't yet recovered the ops at + * the point the call site is disabled. + */ +const struct ftrace_ops ftrace_nop_ops = { + .func = ftrace_ops_nop_func, + .flags = FTRACE_OPS_FL_STUB, +}; +#endif + static inline void ftrace_ops_init(struct ftrace_ops *ops) { #ifdef CONFIG_DYNAMIC_FTRACE @@ -1814,6 +1841,18 @@ static bool __ftrace_hash_rec_update(struct ftrace_ops *ops, * if rec count is zero. */ } + + /* + * If the rec has a single associated ops, and ops->func can be + * called directly, allow the call site to call via the ops. + */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) && + ftrace_rec_count(rec) == 1 && + ftrace_ops_get_func(ops) == ops->func) + rec->flags |= FTRACE_FL_CALL_OPS; + else + rec->flags &= ~FTRACE_FL_CALL_OPS; + count++; /* Must match FTRACE_UPDATE_CALLS in ftrace_modify_all_code() */ @@ -2108,8 +2147,9 @@ void ftrace_bug(int failed, struct dyn_ftrace *rec) struct ftrace_ops *ops = NULL; pr_info("ftrace record flags: %lx\n", rec->flags); - pr_cont(" (%ld)%s", ftrace_rec_count(rec), - rec->flags & FTRACE_FL_REGS ? " R" : " "); + pr_cont(" (%ld)%s%s", ftrace_rec_count(rec), + rec->flags & FTRACE_FL_REGS ? " R" : " ", + rec->flags & FTRACE_FL_CALL_OPS ? " O" : " "); if (rec->flags & FTRACE_FL_TRAMP_EN) { ops = ftrace_find_tramp_ops_any(rec); if (ops) { @@ -2177,6 +2217,7 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) * want the direct enabled (it will be done via the * direct helper). But if DIRECT_EN is set, and * the count is not one, we need to clear it. + * */ if (ftrace_rec_count(rec) == 1) { if (!(rec->flags & FTRACE_FL_DIRECT) != @@ -2185,6 +2226,19 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) } else if (rec->flags & FTRACE_FL_DIRECT_EN) { flag |= FTRACE_FL_DIRECT; } + + /* + * Ops calls are special, as count matters. + * As with direct calls, they must only be enabled when count + * is one, otherwise they'll be handled via the list ops. + */ + if (ftrace_rec_count(rec) == 1) { + if (!(rec->flags & FTRACE_FL_CALL_OPS) != + !(rec->flags & FTRACE_FL_CALL_OPS_EN)) + flag |= FTRACE_FL_CALL_OPS; + } else if (rec->flags & FTRACE_FL_CALL_OPS_EN) { + flag |= FTRACE_FL_CALL_OPS; + } } /* If the state of this record hasn't changed, then do nothing */ @@ -2229,6 +2283,21 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) rec->flags &= ~FTRACE_FL_DIRECT_EN; } } + + if (flag & FTRACE_FL_CALL_OPS) { + if (ftrace_rec_count(rec) == 1) { + if (rec->flags & FTRACE_FL_CALL_OPS) + rec->flags |= FTRACE_FL_CALL_OPS_EN; + else + rec->flags &= ~FTRACE_FL_CALL_OPS_EN; + } else { + /* + * Can only call directly if there's + * only one sets of associated ops. + */ + rec->flags &= ~FTRACE_FL_CALL_OPS_EN; + } + } } /* @@ -2258,7 +2327,8 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) * and REGS states. The _EN flags must be disabled though. */ rec->flags &= ~(FTRACE_FL_ENABLED | FTRACE_FL_TRAMP_EN | - FTRACE_FL_REGS_EN | FTRACE_FL_DIRECT_EN); + FTRACE_FL_REGS_EN | FTRACE_FL_DIRECT_EN | + FTRACE_FL_CALL_OPS_EN); } ftrace_bug_type = FTRACE_BUG_NOP; @@ -2431,6 +2501,25 @@ ftrace_find_tramp_ops_new(struct dyn_ftrace *rec) return NULL; } +struct ftrace_ops * +ftrace_find_unique_ops(struct dyn_ftrace *rec) +{ + struct ftrace_ops *op, *found = NULL; + unsigned long ip = rec->ip; + + do_for_each_ftrace_op(op, ftrace_ops_list) { + + if (hash_contains_ip(ip, op->func_hash)) { + if (found) + return NULL; + found = op; + } + + } while_for_each_ftrace_op(op); + + return found; +} + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* Protected by rcu_tasks for reading, and direct_mutex for writing */ static struct ftrace_hash *direct_functions = EMPTY_HASH; @@ -3780,11 +3869,12 @@ static int t_show(struct seq_file *m, void *v) if (iter->flags & FTRACE_ITER_ENABLED) { struct ftrace_ops *ops; - seq_printf(m, " (%ld)%s%s%s", + seq_printf(m, " (%ld)%s%s%s%s", ftrace_rec_count(rec), rec->flags & FTRACE_FL_REGS ? " R" : " ", rec->flags & FTRACE_FL_IPMODIFY ? " I" : " ", - rec->flags & FTRACE_FL_DIRECT ? " D" : " "); + rec->flags & FTRACE_FL_DIRECT ? " D" : " ", + rec->flags & FTRACE_FL_CALL_OPS ? " O" : " "); if (rec->flags & FTRACE_FL_TRAMP_EN) { ops = ftrace_find_tramp_ops_any(rec); if (ops) { @@ -3800,6 +3890,15 @@ static int t_show(struct seq_file *m, void *v) } else { add_trampoline_func(m, NULL, rec); } + if (rec->flags & FTRACE_FL_CALL_OPS_EN) { + ops = ftrace_find_unique_ops(rec); + if (ops) { + seq_printf(m, "\tops: %pS (%pS)", + ops, ops->func); + } else { + seq_puts(m, "\tops: ERROR!"); + } + } if (rec->flags & FTRACE_FL_DIRECT) { unsigned long direct; From patchwork Mon Jan 9 13:58:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40885 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2170950wrt; Mon, 9 Jan 2023 06:01:27 -0800 (PST) X-Google-Smtp-Source: AMrXdXvFuMczKdU6yrC8MT8+65R+BkwPpZBwiZIA65XFvcJKuMKIK3XYwZEzjPhTY7eAgZI1Wguh X-Received: by 2002:a62:3081:0:b0:578:5e94:22d0 with SMTP id w123-20020a623081000000b005785e9422d0mr65352589pfw.19.1673272887276; Mon, 09 Jan 2023 06:01:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272887; cv=none; d=google.com; s=arc-20160816; b=tLnQvUq5XYnJluSk5OUi8Q41HBf4/BWiI4ZG2hTJN2Z/jeYxcN5obitbLQifbSOc3m kZJOg5/g8CpBgI+XIWZoXJP9hHqIOFEcU+eEdAA1ANvgcorHxO9gXox3FUWQh2cD0QIx xLcsZnxsCWzAPzW3JzmHUtERtJTPrRmXhoOaKcjrzg8GJS8pyBjoPx6wTUOQY5/Y3/14 vN+OlxHexirqWUHwYUEXi731ByiBJIf/MSZqnBl5H2AXO59jdFqz/lRyAG4BE7mB5tH9 Z0wkWZnbQhRl45E1K4iwe8QtqOfRs3oYYssa38oHLk44ChvjfhIrClP1DmijNnXJkz23 zwsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=9TIHg1TnCtgCXxCOCjEq3lp9b9Ah1KeMTk1kU7HR+Fo=; b=PZC/WF2QHRUHzZJczYYY18ertXdqYnheXyYbguTDCKBqff+rng+IQuCqWmH4R0KXRW A0szEzRR1gd8NNjfdhUrgz1Djv2Yw9OnHMTibI933f6ztM7JqIm+Llfi9RE/9rp4bPv0 bivYqWpX+WeGnN9nrn88ZTijEkvrSrvUIX6qZrcm1PlbBaZSnxv/TIHy4KQdiifEZUhl TCM6f/ONrGyJ3NcZde8mqW227QxjKT0i+mJSAI6Pf71Txr7bpN4y2eLxHHrqCDbX9Hli vUVx7I5QaaqMAuUSjAJrr1ZAysAsbg7dbQBseQhWgcKRRemorcB9JYnodXztCUxBLVOl MuxQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n8-20020a654888000000b004707dd3280asi10401906pgs.204.2023.01.09.06.01.12; Mon, 09 Jan 2023 06:01:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237247AbjAIOAg (ORCPT + 99 others); Mon, 9 Jan 2023 09:00:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237271AbjAIN6w (ORCPT ); Mon, 9 Jan 2023 08:58:52 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7514FBCC; Mon, 9 Jan 2023 05:58:51 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3A8C519F0; Mon, 9 Jan 2023 05:59:33 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5B5FC3F23F; Mon, 9 Jan 2023 05:58:49 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 5/8] arm64: insn: Add helpers for BTI Date: Mon, 9 Jan 2023 13:58:25 +0000 Message-Id: <20230109135828.879136-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553790890842728?= X-GMAIL-MSGID: =?utf-8?q?1754553790890842728?= In subsequent patches we'd like to check whether an instruction is a BTI. In preparation for this, add basic instruction helpers for BTI instructions. Per ARM DDI 0487H.a section C6.2.41, BTI is encoded in binary as follows, MSB to LSB: 1101 0101 000 0011 0010 0100 xx01 1111 Where the `xx` bits encode J/C/JC: 00 : (omitted) 01 : C 10 : J 11 : JC Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- arch/arm64/include/asm/insn.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index aaf1f52fbf3e0..139a88e4e8522 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -420,6 +420,7 @@ __AARCH64_INSN_FUNCS(sb, 0xFFFFFFFF, 0xD50330FF) __AARCH64_INSN_FUNCS(clrex, 0xFFFFF0FF, 0xD503305F) __AARCH64_INSN_FUNCS(ssbb, 0xFFFFFFFF, 0xD503309F) __AARCH64_INSN_FUNCS(pssbb, 0xFFFFFFFF, 0xD503349F) +__AARCH64_INSN_FUNCS(bti, 0xFFFFFF3F, 0xD503241f) #undef __AARCH64_INSN_FUNCS From patchwork Mon Jan 9 13:58:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40882 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2169805wrt; Mon, 9 Jan 2023 05:59:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXvSAI5XudxapN5KUJ3k3ABYunHLhtJs9piiqpTZqNfRyjPiUJ3E+7T1jxh8ozLNfcCs2J4t X-Received: by 2002:a17:90b:354b:b0:225:ce98:4287 with SMTP id lt11-20020a17090b354b00b00225ce984287mr56599433pjb.23.1673272797307; Mon, 09 Jan 2023 05:59:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272797; cv=none; d=google.com; s=arc-20160816; b=JK3LLr8uVBDcQFxYYqm0ARjLTJe0Pd0MYECTkmWpjvVLqAHjWvUPlDWqPAF/6UlwBi EQ+qP/fYf6R3nnz8AcPG2SE13TZ4ieV642Yee9k/cGYLpPNxVdWBEUybJmE6UtBvNMQu LFF10GfXYb8B99aSFFgxAy6skJfz7YnoPiW8wbTOQ1JVHpxqqoHL/TzR3p6+YjcgcX6s FQ9Uq4HeT7T5BLs2csyoU2PADycStnT7y9I9++USGoPAaIldyVKIFGlyTUO7cPg3DGSK iGETTqp9AY+UtBRSj6PmHSMSu4kp4kFNMgfQP0oRVVEtKkF6qHPitXlm51mJv1yzSgyQ 7C/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=jAVtm1X18b1/w3mMJOfsOghN5u5sIMQe95YxgQLTtJU=; b=n4HPpxho5jdMWd2I+QylLSrzcTUcjMBRUCCXLrLFzVhC0nzeUkwel8Y3Okjgw1hWjj YenA/NJFDvr3Y+XUIPBVWeF3vXOnRwQGXJbiVsZb1l8REejlxg1sjPQ77opt8HzUV6ty iqhnsor6ZN+NqUa1JOixIY3wDbSqotkCfezzZapMxJrzsQ4qCuR3Xa2vyYXnnRBNCvgf /h7X2TkqZOY6Mr2eoYR8evsBcii/G5eLz+noHdfEA1KEp7wYfZN3MjMKy0plRATdoHRB JbWev+XMKR5SOXGtboWHK4QCryVQN+CeONn+iQwwmd2w9AdXjglafLL4hlIytORpqAJw zbug== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ce13-20020a17090aff0d00b0022726215f21si394838pjb.179.2023.01.09.05.59.45; Mon, 09 Jan 2023 05:59:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237123AbjAIN7N (ORCPT + 99 others); Mon, 9 Jan 2023 08:59:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234186AbjAIN6y (ORCPT ); Mon, 9 Jan 2023 08:58:54 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0B24BF0F; Mon, 9 Jan 2023 05:58:54 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C307F1A00; Mon, 9 Jan 2023 05:59:35 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id ED59C3F23F; Mon, 9 Jan 2023 05:58:51 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 6/8] arm64: patching: Add aarch64_insn_write_literal_u64() Date: Mon, 9 Jan 2023 13:58:26 +0000 Message-Id: <20230109135828.879136-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553696638754821?= X-GMAIL-MSGID: =?utf-8?q?1754553696638754821?= In subsequent patches we'll need to atomically write to a naturally-aligned 64-bit literal embedded within the kernel text. Add a helper for this. For consistency with other text patching code we use copy_to_kernel_nofault(), which is atomic for naturally-aligned accesses up to 64-bits. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- arch/arm64/include/asm/patching.h | 2 ++ arch/arm64/kernel/patching.c | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/patching.h b/arch/arm64/include/asm/patching.h index 6bf5adc562950..68908b82b168f 100644 --- a/arch/arm64/include/asm/patching.h +++ b/arch/arm64/include/asm/patching.h @@ -7,6 +7,8 @@ int aarch64_insn_read(void *addr, u32 *insnp); int aarch64_insn_write(void *addr, u32 insn); +int aarch64_insn_write_literal_u64(void *addr, u64 val); + int aarch64_insn_patch_text_nosync(void *addr, u32 insn); int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt); diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c index 33e0fabc0b79b..b4835f6d594bc 100644 --- a/arch/arm64/kernel/patching.c +++ b/arch/arm64/kernel/patching.c @@ -88,6 +88,23 @@ int __kprobes aarch64_insn_write(void *addr, u32 insn) return __aarch64_insn_write(addr, cpu_to_le32(insn)); } +noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val) +{ + u64 *waddr; + unsigned long flags; + int ret; + + raw_spin_lock_irqsave(&patch_lock, flags); + waddr = patch_map(addr, FIX_TEXT_POKE0); + + ret = copy_to_kernel_nofault(waddr, &val, sizeof(val)); + + patch_unmap(FIX_TEXT_POKE0); + raw_spin_unlock_irqrestore(&patch_lock, flags); + + return ret; +} + int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) { u32 *tp = addr; From patchwork Mon Jan 9 13:58:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40883 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2170454wrt; Mon, 9 Jan 2023 06:00:48 -0800 (PST) X-Google-Smtp-Source: AMrXdXuhOR7dKtYA5AoQ3lA8AjdcosmarK9mDCRCwfDrCbxb0kmRWuSFALyTZpoTyb+dDLR5Cj1r X-Received: by 2002:a17:902:7c0f:b0:193:213:9bf0 with SMTP id x15-20020a1709027c0f00b0019302139bf0mr12607744pll.49.1673272847801; Mon, 09 Jan 2023 06:00:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272847; cv=none; d=google.com; s=arc-20160816; b=UZsRBxvNHH/It/2BGdIoY7C/fEx8dSTgVMWOwCN8+AHRA6oHxXF6pDo36uXuE68Nxv O9RScSVBJvVryQHEoSmdnCDhlv91PBSYdrqypwVzGG4uNBT9UHn239Tj9GptdI0UYX8P EDD5YtRNeX7ju7Zbbde+bLjdm3Ot5HNZfp+TNpXCjWWlde1jqFgFYIwGJlUCwiNoPDWO bdoYAkSbP/6/xs9oCMJb89mvNaNZIF6p9T9AgJDiGqSd4aqahSYR6tkwoSCY+KdZCtO1 AqdSxkJwSFg2nIirFQ6HuOZHg/G2YF5O4CPHXKCi0mkofIEstM85uJXuWrgXGveB2n6y UR5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=hlvV2LLN1KwIHT8cIfouPzKE00Mk5FgVfeJmB18QM5c=; b=QqfY5rvxwK4RVHdFu4Pber60PadBYBJEzBNBaBUUpr0IBXC0q5RIBGuCMdWxTlpTBT yIYKa2yva9mexfIxp6ioN5yfzzZhLz323Q4woaWS45P95S5BVG64odITVegry+9zwkD6 Xd5i1guQvZKBmwUu2FSF3KDC0I17mqNfLPJzEJYnvD1p+syPgQQkBMYIBtfjQkfItwqW i1bdJKCEBwIEKpyP6LoR4KYt2dVgapjzEX9dpWB8zXcJ7KroBnDjRBOBQU7w/8uQWC1S sXPhvvgMZyRrS3qQMQB/6yRN5u41KVr9sLInphQFMg3Prc44VqMpyOKUURAO92Q7N79p tYfw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j3-20020a170903028300b00188b9b4139asi6480210plr.210.2023.01.09.06.00.32; Mon, 09 Jan 2023 06:00:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233715AbjAIN7R (ORCPT + 99 others); Mon, 9 Jan 2023 08:59:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236492AbjAIN65 (ORCPT ); Mon, 9 Jan 2023 08:58:57 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9090DF6F; Mon, 9 Jan 2023 05:58:56 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5558C1A25; Mon, 9 Jan 2023 05:59:38 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7FEEF3F23F; Mon, 9 Jan 2023 05:58:54 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 7/8] arm64: ftrace: Update stale comment Date: Mon, 9 Jan 2023 13:58:27 +0000 Message-Id: <20230109135828.879136-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553749577475464?= X-GMAIL-MSGID: =?utf-8?q?1754553749577475464?= In commit: 26299b3f6ba26bfc ("ftrace: arm64: move from REGS to ARGS") ... we folded ftrace_regs_entry into ftrace_caller, and ftrace_regs_entry no longer exists. Update the comment accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- arch/arm64/kernel/ftrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index b30b955a89211..38ebdf063255b 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -209,7 +209,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) * | NOP | MOV X9, LR | MOV X9, LR | * | NOP | NOP | BL | * - * The LR value will be recovered by ftrace_regs_entry, and restored into LR + * The LR value will be recovered by ftrace_caller, and restored into LR * before returning to the regular function prologue. When a function is not * being traced, the MOV is not harmful given x9 is not live per the AAPCS. * From patchwork Mon Jan 9 13:58:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 40884 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp2170830wrt; Mon, 9 Jan 2023 06:01:19 -0800 (PST) X-Google-Smtp-Source: AMrXdXtzwJfYstM9tVWrA8bw+v+inQcgO/vz3JRDxyvXd2PO/VJzavZ21TCU1EdWzOkmu3i/oe7Y X-Received: by 2002:a62:31c3:0:b0:574:c62c:2f55 with SMTP id x186-20020a6231c3000000b00574c62c2f55mr71591971pfx.3.1673272879405; Mon, 09 Jan 2023 06:01:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673272879; cv=none; d=google.com; s=arc-20160816; b=g1uAXRX5NHzJiNh1NyC5KaDWSlqa+dxntIgL3RIa4PnV2Kk2AQeSoYl398o1qpom1D 5x0GoVouhX3/zqwOJMaYPagippFH2d63a/OuR3TBQaMB5p5nGCkm8i+4h4RpIfeecDGX I2WafJ3O3iV67rcHXnxtod38lRauiInkawSFMSmygb58CM3ijdCeqvSupBDrBbQlRWM/ 5qQaJVRnY4jlOlzqg5qMBiORYbdmGw0dOSVuatdmJxNAzQ+Rnd707yWTu1LiRfRNNRDh fOInEpRi92BnCNG4ec+SnLr2GCLcNsd21DeYMrtt8gTEYCRLD43BVtZlQTOQxPEAb3Up 0g9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=GBVcgQybsciQ1xlrR5/UFR3fDU13efZ3vdgrWeRNlgM=; b=TuzDdgAMO+vU/CNOswhKVFMYshRu4WjpHqKNBqfb3YaCSUDwzqPOFwPxit36w2VUPL MgRPIa38Ui8BRBjMp9qU0rMZyE6MTfCfR5ZoMeyCLlkcnMKK9qK3zaezwemHUY1CIGYd Vr5WCDi9CPDQnkGvINQdTg7ucD7IoEjaYk/MvM8f7G/wsWuI5S19qzX8zMpCyPpM5+L/ SsC5stioPU1CIw1PW4YthDDs/wttvQo0sUMyVJblSYamfxOmD1BYssk40uBXOfVxOLmp oDPMOp0hUnFXYAoisq3yM9h15TcAiF05u3rPJvoecvoYEnGNt1LoAzK3F4ne09/LQ9PT PZgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g7-20020a056a0023c700b005786897552esi10397686pfc.33.2023.01.09.06.00.36; Mon, 09 Jan 2023 06:01:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237119AbjAIN73 (ORCPT + 99 others); Mon, 9 Jan 2023 08:59:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237054AbjAIN7A (ORCPT ); Mon, 9 Jan 2023 08:59:00 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5ABBC25CD; Mon, 9 Jan 2023 05:58:59 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 208AC1A2D; Mon, 9 Jan 2023 05:59:41 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 30F1C3F23F; Mon, 9 Jan 2023 05:58:57 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, lenb@kernel.org, linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, ndesaulniers@google.com, ojeda@kernel.org, peterz@infradead.org, rafael.j.wysocki@intel.com, revest@chromium.org, robert.moore@intel.com, rostedt@goodmis.org, will@kernel.org Subject: [PATCH 8/8] arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS Date: Mon, 9 Jan 2023 13:58:28 +0000 Message-Id: <20230109135828.879136-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230109135828.879136-1-mark.rutland@arm.com> References: <20230109135828.879136-1-mark.rutland@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754553782573336816?= X-GMAIL-MSGID: =?utf-8?q?1754553782573336816?= This patch enables support for DYNAMIC_FTRACE_WITH_CALL_OPS on arm64. This allows each ftrace callsite to provide an ftrace_ops to the common ftrace trampoline, allowing each callsite to invoke distinct tracer functions without the need to fall back to list processing or to allocate custom trampoliens for each callsite. This significantly speeds up cases where multiple distinct trace functions are used and callsites are mostly traced by a single tracer. The main idea is to place a pointer to the ftrace_ops as a literal at a fixed offset from the function entry point, which can be recovered by the common ftrace trampoline. Using a 64-bit literal avoids branch range limitations, and permits the ops to be swapped atomically without special considerations that apply to code-patching. In future this will also allow for the implementation of DYNAMIC_FTRACE_WITH_DIRECT_CALLS without branch range limitations by using additional fields in struct ftrace_ops. As noted in the core patch adding support for DYNAMIC_FTRACE_WITH_CALL_OPS, this approach allows for directly invoking ftrace_ops::func even for ftrace_ops which are dynamically-allocated (or part of a module), without going via ftrace_ops_list_func. Currently, this approach is not compatible with CLANG_CFI, as the presence/absence of pre-function NOPs changes the offset of the pre-function type hash, and there's no existing mechanism to ensure a consistent offset for instrumented and uninstrumented functions. When CLANG_CFI is enabled, the existing scheme with a global ops->func pointer is used, and there should be no functional change. I am currently working with others to allow the two to work together in future (though this will liekly require updated compiler support). I've benchamrked this with the ftrace_ops sample module [1], which is not currently upstream, but available at: https://lore.kernel.org/lkml/20230103124912.2948963-1-mark.rutland@arm.com git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git ftrace-ops-sample-20230109 Using that module I measured the total time taken for 100,000 calls to a trivial instrumented function, with a number of tracers enabled with relevant filters (which would apply to the instrumented function) and a number of tracers enabled with irrelevant filters (which would not apply to the instrumented function). I tested on an M1 MacBook Pro, running under a HVF-accelerated QEMU VM (i.e. on real hardware). Before this patch: Number of tracers || Total time | Per-call average time (ns) Relevant | Irrelevant || (ns) | Total | Overhead =========+============++=============+==============+============ 0 | 0 || 94,583 | 0.95 | - 0 | 1 || 93,709 | 0.94 | - 0 | 2 || 93,666 | 0.94 | - 0 | 10 || 93,709 | 0.94 | - 0 | 100 || 93,792 | 0.94 | - ---------+------------++-------------+--------------+------------ 1 | 1 || 6,467,833 | 64.68 | 63.73 1 | 2 || 7,509,708 | 75.10 | 74.15 1 | 10 || 23,786,792 | 237.87 | 236.92 1 | 100 || 106,432,500 | 1,064.43 | 1063.38 ---------+------------++-------------+--------------+------------ 1 | 0 || 1,431,875 | 14.32 | 13.37 2 | 0 || 6,456,334 | 64.56 | 63.62 10 | 0 || 22,717,000 | 227.17 | 226.22 100 | 0 || 103,293,667 | 1032.94 | 1031.99 ---------+------------++-------------+--------------+-------------- Note: per-call overhead is estiamated relative to the baseline case with 0 relevant tracers and 0 irrelevant tracers. After this patch Number of tracers || Total time | Per-call average time (ns) Relevant | Irrelevant || (ns) | Total | Overhead =========+============++=============+==============+============ 0 | 0 || 94,541 | 0.95 | - 0 | 1 || 93,666 | 0.94 | - 0 | 2 || 93,709 | 0.94 | - 0 | 10 || 93,667 | 0.94 | - 0 | 100 || 93,792 | 0.94 | - ---------+------------++-------------+--------------+------------ 1 | 1 || 281,000 | 2.81 | 1.86 1 | 2 || 281,042 | 2.81 | 1.87 1 | 10 || 280,958 | 2.81 | 1.86 1 | 100 || 281,250 | 2.81 | 1.87 ---------+------------++-------------+--------------+------------ 1 | 0 || 280,959 | 2.81 | 1.86 2 | 0 || 6,502,708 | 65.03 | 64.08 10 | 0 || 18,681,209 | 186.81 | 185.87 100 | 0 || 103,550,458 | 1,035.50 | 1034.56 ---------+------------++-------------+--------------+------------ Note: per-call overhead is estiamated relative to the baseline case with 0 relevant tracers and 0 irrelevant tracers. As can be seen from the above: a) Whenever there is a single relevant tracer function associated with a tracee, the overhead of invoking the tracer is constant, and does not scale with the number of tracers which are *not* associated with that tracee. b) The overhead for a single relevant tracer has dropped to ~1/7 of the overhead prior to this series (from 13.37ns to 1.86ns). This is largely due to permitting calls to dynamically-allocated ftrace_ops without going through ftrace_ops_list_func. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Florent Revest Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Will Deacon --- arch/arm64/Kconfig | 3 + arch/arm64/Makefile | 5 +- arch/arm64/include/asm/ftrace.h | 15 +-- arch/arm64/kernel/asm-offsets.c | 4 + arch/arm64/kernel/entry-ftrace.S | 32 ++++++- arch/arm64/kernel/ftrace.c | 156 +++++++++++++++++++++++++++++++ 6 files changed, 195 insertions(+), 20 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 03934808b2ed0..7838f568fa158 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -123,6 +123,7 @@ config ARM64 select DMA_DIRECT_REMAP select EDAC_SUPPORT select FRAME_POINTER + select FUNCTION_ALIGNMENT_8B if DYNAMIC_FTRACE_WITH_CALL_OPS select GENERIC_ALLOCATOR select GENERIC_ARCH_TOPOLOGY select GENERIC_CLOCKEVENTS_BROADCAST @@ -186,6 +187,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ + if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG) select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \ if DYNAMIC_FTRACE_WITH_ARGS select HAVE_EFFICIENT_UNALIGNED_ACCESS diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index d62bd221828f7..4c3be442fbb35 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -139,7 +139,10 @@ endif CHECKFLAGS += -D__aarch64__ -ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y) +ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS),y) + KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY + CC_FLAGS_FTRACE := -fpatchable-function-entry=4,2 +else ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y) KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY CC_FLAGS_FTRACE := -fpatchable-function-entry=2 endif diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 5664729800ae1..1c2672bbbf379 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -62,20 +62,7 @@ extern unsigned long ftrace_graph_call; extern void return_to_handler(void); -static inline unsigned long ftrace_call_adjust(unsigned long addr) -{ - /* - * Adjust addr to point at the BL in the callsite. - * See ftrace_init_nop() for the callsite sequence. - */ - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS)) - return addr + AARCH64_INSN_SIZE; - /* - * addr is the address of the mcount call instruction. - * recordmcount does the necessary offset calculation. - */ - return addr; -} +unsigned long ftrace_call_adjust(unsigned long addr); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct dyn_ftrace; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 2234624536d95..ae345b06e9f7e 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -193,6 +194,9 @@ int main(void) DEFINE(KIMAGE_HEAD, offsetof(struct kimage, head)); DEFINE(KIMAGE_START, offsetof(struct kimage, start)); BLANK(); +#endif +#ifdef CONFIG_FUNCTION_TRACER + DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func)); #endif return 0; } diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 3b625f76ffbae..350ed81324ace 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -65,13 +65,35 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE - sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) - mov x1, x9 // parent_ip (callsite's LR) - ldr_l x2, function_trace_op // op - mov x3, sp // regs + /* Prepare arguments for the the tracer func */ + sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) + mov x1, x9 // parent_ip (callsite's LR) + mov x3, sp // regs + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS + /* + * The literal pointer to the ops is at an 8-byte aligned boundary + * which is either 12 or 16 bytes before the BL instruction in the call + * site. See ftrace_call_adjust() for details. + * + * Therefore here the LR points at `literal + 16` or `literal + 20`, + * and we can find the address of the literal in either case by + * aligning to an 8-byte boundary and subtracting 16. We do the + * alignment first as this allows us to fold the subtraction into the + * LDR. + */ + bic x2, x30, 0x7 + ldr x2, [x2, #-16] // op + + ldr x4, [x2, #FTRACE_OPS_FUNC] // op->func + blr x4 // op->func(ip, parent_ip, op, regs) + +#else + ldr_l x2, function_trace_op // op SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) - bl ftrace_stub + bl ftrace_stub // func(ip, parent_ip, op, regs) +#endif /* * At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 38ebdf063255b..5545fe1a90125 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -60,6 +60,89 @@ int ftrace_regs_query_register_offset(const char *name) } #endif +unsigned long ftrace_call_adjust(unsigned long addr) +{ + /* + * When using mcount, addr is the address of the mcount call + * instruction, and no adjustment is necessary. + */ + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS)) + return addr; + + /* + * When using patchable-function-entry without pre-function NOPS, addr + * is the address of the first NOP after the function entry point. + * + * The compiler has either generated: + * + * addr+00: func: NOP // To be patched to MOV X9, LR + * addr+04: NOP // To be patched to BL + * + * Or: + * + * addr-04: BTI C + * addr+00: func: NOP // To be patched to MOV X9, LR + * addr+04: NOP // To be patched to BL + * + * We must adjust addr to the address of the NOP which will be patched + * to `BL `, which is at `addr + 4` bytes in either case. + * + */ + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) + return addr + AARCH64_INSN_SIZE; + + /* + * When using patchable-function-entry with pre-function NOPs, addr is + * the address of the first pre-function NOP. + * + * Starting from an 8-byte aligned base, the compiler has either + * generated: + * + * addr+00: NOP // Literal (first 32 bits) + * addr+04: NOP // Literal (last 32 bits) + * addr+08: func: NOP // To be patched to MOV X9, LR + * addr+12: NOP // To be patched to BL + * + * Or: + * + * addr+00: NOP // Literal (first 32 bits) + * addr+04: NOP // Literal (last 32 bits) + * addr+08: func: BTI C + * addr+12: NOP // To be patched to MOV X9, LR + * addr+16: NOP // To be patched to BL + * + * We must adjust addr to the address of the NOP which will be patched + * to `BL `, which is at either addr+12 or addr+16 depending on + * whether there is a BTI. + */ + + if (!IS_ALIGNED(addr, sizeof(unsigned long))) { + WARN_RATELIMIT(1, "Misaligned patch-site %pS\n", + (void *)(addr + 8)); + return 0; + } + + /* Skip the NOPs placed before the function entry point */ + addr += 2 * AARCH64_INSN_SIZE; + + /* Skip any BTI */ + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { + u32 insn = le32_to_cpu(*(__le32 *)addr); + + if (aarch64_insn_is_bti(insn)) { + addr += AARCH64_INSN_SIZE; + } else if (insn != aarch64_insn_gen_nop()) { + WARN_RATELIMIT(1, "unexpected insn in patch-site %pS: 0x%08x\n", + (void *)addr, insn); + } + } + + /* Skip the first NOP after function entry */ + addr += AARCH64_INSN_SIZE; + + return addr; +} + /* * Replace a single instruction, which may be a branch or NOP. * If @validate == true, a replaced instruction is checked against 'old'. @@ -98,6 +181,13 @@ int ftrace_update_ftrace_func(ftrace_func_t func) unsigned long pc; u32 new; + /* + * When using CALL_OPS, the function to call is associated with the + * call site, and we don't have a global function pointer to update. + */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) + return 0; + pc = (unsigned long)ftrace_call; new = aarch64_insn_gen_branch_imm(pc, (unsigned long)func, AARCH64_INSN_BRANCH_LINK); @@ -176,6 +266,44 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec, return true; } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +static const struct ftrace_ops *arm64_rec_get_ops(struct dyn_ftrace *rec) +{ + const struct ftrace_ops *ops = NULL; + + if (rec->flags & FTRACE_FL_CALL_OPS_EN) { + ops = ftrace_find_unique_ops(rec); + WARN_ON_ONCE(!ops); + } + + if (!ops) + ops = &ftrace_list_ops; + + return ops; +} + +static int ftrace_rec_set_ops(const struct dyn_ftrace *rec, + const struct ftrace_ops *ops) +{ + unsigned long literal = ALIGN_DOWN(rec->ip - 12, 8); + return aarch64_insn_write_literal_u64((void *)literal, + (unsigned long)ops); +} + +static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec) +{ + return ftrace_rec_set_ops(rec, &ftrace_nop_ops); +} + +static int ftrace_rec_update_ops(struct dyn_ftrace *rec) +{ + return ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec)); +} +#else +static int ftrace_rec_set_nop_ops(struct dyn_ftrace *rec) { return 0; } +static int ftrace_rec_update_ops(struct dyn_ftrace *rec) { return 0; } +#endif + /* * Turn on the call to ftrace_caller() in instrumented function */ @@ -183,6 +311,11 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned long pc = rec->ip; u32 old, new; + int ret; + + ret = ftrace_rec_update_ops(rec); + if (ret) + return ret; if (!ftrace_find_callable_addr(rec, NULL, &addr)) return -EINVAL; @@ -193,6 +326,19 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) return ftrace_modify_code(pc, old, new, true); } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) +{ + if (WARN_ON_ONCE(old_addr != (unsigned long)ftrace_caller)) + return -EINVAL; + if (WARN_ON_ONCE(addr != (unsigned long)ftrace_caller)) + return -EINVAL; + + return ftrace_rec_update_ops(rec); +} +#endif + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS /* * The compiler has inserted two NOPs before the regular function prologue. @@ -220,6 +366,11 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) { unsigned long pc = rec->ip - AARCH64_INSN_SIZE; u32 old, new; + int ret; + + ret = ftrace_rec_set_nop_ops(rec); + if (ret) + return ret; old = aarch64_insn_gen_nop(); new = aarch64_insn_gen_move_reg(AARCH64_INSN_REG_9, @@ -237,9 +388,14 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, { unsigned long pc = rec->ip; u32 old = 0, new; + int ret; new = aarch64_insn_gen_nop(); + ret = ftrace_rec_set_nop_ops(rec); + if (ret) + return ret; + /* * When using mcount, callsites in modules may have been initalized to * call an arbitrary module PLT (which redirects to the _mcount stub)