From patchwork Sat Dec 10 10:09:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 32003 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1234644wrr; Sat, 10 Dec 2022 02:23:40 -0800 (PST) X-Google-Smtp-Source: AA0mqf4A2/xZtlKKs1TRmPzvvD2q7w0nRPSeTnF6pD4Ec9lvnYhro5HPDTjof3EDuG/4KwX2FXHK X-Received: by 2002:a05:6a20:9595:b0:a3:ff0b:d1bc with SMTP id iu21-20020a056a20959500b000a3ff0bd1bcmr10763329pzb.62.1670667820492; Sat, 10 Dec 2022 02:23:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670667820; cv=none; d=google.com; s=arc-20160816; b=BMKZew727w/TtDY6IWgOxBvHA+lh3KGz/E6TduOxUNuNMiCfwLLpvKtQ7x3QFdLfWO LEQmzW/J6RM4C11YKDUxZJhGG5Sf2nEyNR/uDF4unjhK05zF3xSYNQ7TwUVGpZ/IFp1c 7SCjasXSEouT1Cd7Vp/8IfRSj843KomLHLF+PxIjMb1sgeb+mtVwHizbo5Da2Uaiuim7 1OVE8ZthEtZJDlRVjK4dODEG+qiP2M8k1jhsr9p4/rnYGSsAFhUwCGwJiFqcadCHVMW6 OwYa/Jiu2akyU6W+ejwl6qnrk5rz1Jp5dkAYXSRHAzZi7jV9+SXe6hbg0sZHsAdrrieR hb8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=53ee5kpbob8xjq2MlIoujoXPih5GwSwaNtTdJf4XEZ4=; b=GjQhSvmoOhp5y5Y6oVBpRhjc15XXo6IfryZDFBuJz5LCQ02+y2lEq5dJohxOtJrBkt 2MbPh4c45X55PfJwVZvx4DyzwqhrgUC4q7SG8887ozl+2EExLIH303vA+2XZq0hvQG2O phjxpdGgJEhq2HIAtTVO7LBWbr5aqJqcmVVsL2NtjzQbn8RaCMrXVcRagR+7KzK6fvKI XSSB1QDiKGe/pQQfz+NlM7vrmVIHaNIZXETXCktZSbtRJ0ulT+qvSVwra3Zbo67U9GV5 FsxfFLGAPZvlEPlOlWY+67vA4K9Pg8IKxlUA9kG4/Nhgpx0SiCXAxpVPWWds2WBLvoxl RaAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=c1gonYSl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b196-20020a6334cd000000b004785b66be7bsi4304523pga.89.2022.12.10.02.23.27; Sat, 10 Dec 2022 02:23:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=c1gonYSl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229854AbiLJKJu (ORCPT + 99 others); Sat, 10 Dec 2022 05:09:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229851AbiLJKJs (ORCPT ); Sat, 10 Dec 2022 05:09:48 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E549205D1 for ; Sat, 10 Dec 2022 02:09:47 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id AD0F7B82A53 for ; Sat, 10 Dec 2022 10:09:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D155FC433F1; Sat, 10 Dec 2022 10:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670666984; bh=K/60DtPvcZdUEbYVaryyiSYpmJ4q6k25bcoqeghcm6Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=c1gonYSltysNYBdWo1OiUPJAX6nQDNh2VBFXTKSWFvM4XLcSZWi9uw4vkUCo33oHp 5ZwvePgedJRPeibXPM0DauEBUHlHvkKFI1hkkSiaoA4CI8NYysWcKH0GeIzNJzMH0K eehfNuQZYYOYTqErRq0nT6cIuz/rX7499UcL5Pj+t5WNJngiN+R4ZPAvpEpWWaWZla Rpimacv6DCkQwRj/sAXAJi9linKUxlj0lBc90p56MGv2xWgZREJX/UbY3YlYobrN7u 5HG5IF2K70DLgBga9GBvDLq/WLcwif82wrH8b8mCXyuTmFrC8aAKyZvSITaCVzKR8I kZDPj/QyGxU0Q== From: guoren@kernel.org To: palmer@rivosinc.com, conor.Dooley@microchip.com, guoren@kernel.org, andy.chiu@sifive.com, greentime.hu@sifive.com, paul.walmsley@sifive.com, peterz@infradead.org, rostedt@goodmis.org, jrtc27@jrtc27.com Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH -next V2 1/2] riscv: jump_label: Fixup unaligned arch_static_branch function Date: Sat, 10 Dec 2022 05:09:26 -0500 Message-Id: <20221210100927.835145-2-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20221210100927.835145-1-guoren@kernel.org> References: <20221210100927.835145-1-guoren@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751822180249324275?= X-GMAIL-MSGID: =?utf-8?q?1751822180249324275?= From: Andy Chiu Runtime code patching must be done at a naturally aligned address, or we may execute on a partial instruction. We have encountered problems traced back to static jump functions during the test. We switched the tracer randomly for every 1~5 seconds on a dual-core QEMU setup and found the kernel sucking at a static branch where it jumps to itself. The reason is that the static branch was 2-byte but not 4-byte aligned. Then, the kernel would patch the instruction, either J or NOP, with two half-word stores if the machine does not have efficient unaligned accesses. Thus, moments exist where half of the NOP mixes with the other half of the J when transitioning the branch. In our particular case, on a little-endian machine, the upper half of the NOP was mixed with the lower part of the J when enabling the branch, resulting in a jump that jumped to itself. Conversely, it would result in a HINT instruction when disabling the branch, but it might not be observable. ARM64 does not have this problem since all instructions must be 4-byte aligned. Fixes: ebc00dde8a97 ("riscv: Add jump-label implementation") Link: https://lore.kernel.org/linux-riscv/20220913094252.3555240-6-andy.chiu@sifive.com/ Signed-off-by: Andy Chiu Reviewed-by: Greentime Hu Signed-off-by: Guo Ren --- arch/riscv/include/asm/jump_label.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/jump_label.h b/arch/riscv/include/asm/jump_label.h index 38af2ec7b9bf..729991e8f782 100644 --- a/arch/riscv/include/asm/jump_label.h +++ b/arch/riscv/include/asm/jump_label.h @@ -18,6 +18,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool branch) { asm_volatile_goto( + " .align 2 \n\t" " .option push \n\t" " .option norelax \n\t" " .option norvc \n\t" @@ -39,6 +40,7 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) { asm_volatile_goto( + " .align 2 \n\t" " .option push \n\t" " .option norelax \n\t" " .option norvc \n\t" From patchwork Sat Dec 10 10:09:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 32002 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1234204wrr; Sat, 10 Dec 2022 02:22:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf5+L7Z4e0BEeyk5HkZWEAFL+2VaZgMrhOFrnkPvDQGQWrp9uqS9KBeFGvzBia0M299zM0Vi X-Received: by 2002:a17:90a:c985:b0:219:56eb:6c1b with SMTP id w5-20020a17090ac98500b0021956eb6c1bmr10822183pjt.47.1670667724521; Sat, 10 Dec 2022 02:22:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670667724; cv=none; d=google.com; s=arc-20160816; b=etkOqnwRl887KXfzqdGHWTcD7XWsA+2ochZs544GJdkPVWvZFsnS+y0UsnZI1Z+761 Znjefy49fgyih8UhG8TU7XEaq+E8ecwJml3ZEqRL4BlVg3WsEtPnC3zKvjMisFQHfhBc 88uXk4eCigi6uDqcGsIQdpeSjV17myWxYGfzNySW55XoaNcFMFdfCuACIBG9nIQOAG43 h5+p2ZKQ9ysmQJGNRR4aUlXG62ybgXeZ+SqmwSsweZ0pHL3H0dQtj83bpDKN0Acanh/O dfljpshDgyOYo+dadv5c0hVHt2myg5TbUUUNqmrmXJ1KkfF+m3WuqHDmHinUIQpi7ZlE /sSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9zn6UjeaZkA9vY8s7FzdaWMQjKxtdac/K0ymMpe66bQ=; b=ZKogJIKcr+OSZLofAqMx5JHd0tWPjYMcj/Lzh//D5JQ67OtNhREB+bwC/0jL4R+EaM ht3xpq8pAsxVz/HEEFUpvirPoGz+KLxLRbLnt29uzOUNfzfVhSIsHZgMkQpTay3CKy7z ReF8TGcCxyZUD7xkB9DSawGnii1EhxsfsdEVtVyA0ddQe3H0781BV7Csp2OQ9yh8q6/U NXVGz3Wj+yMyvXweE8H505ZYCWu7A9eL5z8OPmiVWMJ8YoiB4g9yBF4GrdhnEqR9Lm0e p5O/jfvoj8K8XxCY+ks6ZLgOt+GKhHJv6IaUYdNE21BXd0xUZ5DDNrSzpCgOvSC6Fdhq qEAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iJ0pBgLJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lw1-20020a17090b180100b00202827b6f5asi4516891pjb.162.2022.12.10.02.21.45; Sat, 10 Dec 2022 02:22:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iJ0pBgLJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229877AbiLJKJw (ORCPT + 99 others); Sat, 10 Dec 2022 05:09:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbiLJKJu (ORCPT ); Sat, 10 Dec 2022 05:09:50 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 803D3205F9 for ; Sat, 10 Dec 2022 02:09:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1A2FE60A24 for ; Sat, 10 Dec 2022 10:09:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D610AC433F0; Sat, 10 Dec 2022 10:09:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670666988; bh=5ghN9JW2HFtpVtwiWAlI7UySwVA3lyKrsQIFRBo/g8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iJ0pBgLJTq5BcrKXlsghFrcJv0XBGPXuSeLbIAYvi4wXGi06KeGKZqj7UecDjHLSU Jaw0wt3GoUQ29OWIobpCjxTM+aXOQwesa0QnvBfqG5gwF9b4W0kIW/HHsW+DEEB9rs +phKyCzJVYxUoMYfieAekbg/Lbs9xY5zttYMPrUHQxyrdPq26Dael08jIStJKKE1Sh SFwQYQGaJG/y8EHGaA/gZTG4h5j79CCuws7spiqSKNks63EYkzQMXfYezT0LH3dvSc NvU2OvGLsGiZvofWJaxvAJJebE9XLJdtz2cEZuqy7okQdKGUvbqn34XikhAZCSaCq7 ZwB0hmfStFpag== From: guoren@kernel.org To: palmer@rivosinc.com, conor.Dooley@microchip.com, guoren@kernel.org, andy.chiu@sifive.com, greentime.hu@sifive.com, paul.walmsley@sifive.com, peterz@infradead.org, rostedt@goodmis.org, jrtc27@jrtc27.com Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Guo Ren Subject: [PATCH -next V2 2/2] riscv: jump_label: Using c.j for CONFIG_RISCV_ISA_C Date: Sat, 10 Dec 2022 05:09:27 -0500 Message-Id: <20221210100927.835145-3-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20221210100927.835145-1-guoren@kernel.org> References: <20221210100927.835145-1-guoren@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751822080049042743?= X-GMAIL-MSGID: =?utf-8?q?1751822080049042743?= From: Guo Ren Reduce the size of the static branch instruction and prevent atomic update problems. It would also reduce the range from 1MB to 4KB. The range of 4K is enough for static branch. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Cc: Jessica Clarke --- arch/riscv/include/asm/jump_label.h | 16 +++++++++++---- arch/riscv/kernel/jump_label.c | 30 +++++++++++++++++++++++++++-- 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/jump_label.h b/arch/riscv/include/asm/jump_label.h index 729991e8f782..e5f772ad5887 100644 --- a/arch/riscv/include/asm/jump_label.h +++ b/arch/riscv/include/asm/jump_label.h @@ -12,17 +12,23 @@ #include #include +#ifdef CONFIG_RISCV_ISA_C +#define JUMP_LABEL_NOP_SIZE 2 +#else #define JUMP_LABEL_NOP_SIZE 4 +#endif static __always_inline bool arch_static_branch(struct static_key *key, bool branch) { asm_volatile_goto( - " .align 2 \n\t" " .option push \n\t" " .option norelax \n\t" - " .option norvc \n\t" +#ifdef CONFIG_RISCV_ISA_C + "1: c.nop \n\t" +#else "1: nop \n\t" +#endif " .option pop \n\t" " .pushsection __jump_table, \"aw\" \n\t" " .align " RISCV_LGPTR " \n\t" @@ -40,11 +46,13 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) { asm_volatile_goto( - " .align 2 \n\t" " .option push \n\t" " .option norelax \n\t" - " .option norvc \n\t" +#ifdef CONFIG_RISCV_ISA_C + "1: c.j %l[label] \n\t" +#else "1: jal zero, %l[label] \n\t" +#endif " .option pop \n\t" " .pushsection __jump_table, \"aw\" \n\t" " .align " RISCV_LGPTR " \n\t" diff --git a/arch/riscv/kernel/jump_label.c b/arch/riscv/kernel/jump_label.c index e6694759dbd0..08f42c49e3a0 100644 --- a/arch/riscv/kernel/jump_label.c +++ b/arch/riscv/kernel/jump_label.c @@ -11,26 +11,52 @@ #include #include +#ifdef CONFIG_RISCV_ISA_C +#define RISCV_INSN_NOP 0x0001U +#define RISCV_INSN_C_J 0xa001U +#else #define RISCV_INSN_NOP 0x00000013U #define RISCV_INSN_JAL 0x0000006fU +#endif void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { void *addr = (void *)jump_entry_code(entry); +#ifdef CONFIG_RISCV_ISA_C + u16 insn; +#else u32 insn; +#endif if (type == JUMP_LABEL_JMP) { long offset = jump_entry_target(entry) - jump_entry_code(entry); - - if (WARN_ON(offset & 1 || offset < -524288 || offset >= 524288)) + if (WARN_ON(offset & 1 || offset < -2048 || offset >= 2048)) return; +#ifdef CONFIG_RISCV_ISA_C + /* + * 001 | imm[11|4|9:8|10|6|7|3:1|5] 01 - C.J + */ + insn = RISCV_INSN_C_J | + (((u16)offset & GENMASK(5, 5)) >> (5 - 2)) | + (((u16)offset & GENMASK(3, 1)) << (3 - 1)) | + (((u16)offset & GENMASK(7, 7)) >> (7 - 6)) | + (((u16)offset & GENMASK(6, 6)) << (7 - 6)) | + (((u16)offset & GENMASK(10, 10)) >> (10 - 8)) | + (((u16)offset & GENMASK(9, 8)) << (9 - 8)) | + (((u16)offset & GENMASK(4, 4)) << (11 - 4)) | + (((u16)offset & GENMASK(11, 11)) << (12 - 11)); +#else + /* + * imm[20|10:1|11|19:12] | rd | 1101111 - JAL + */ insn = RISCV_INSN_JAL | (((u32)offset & GENMASK(19, 12)) << (12 - 12)) | (((u32)offset & GENMASK(11, 11)) << (20 - 11)) | (((u32)offset & GENMASK(10, 1)) << (21 - 1)) | (((u32)offset & GENMASK(20, 20)) << (31 - 20)); +#endif } else { insn = RISCV_INSN_NOP; }