From patchwork Wed Feb 8 17:16:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xi Ruoyao X-Patchwork-Id: 54505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3579836wrn; Wed, 8 Feb 2023 09:18:12 -0800 (PST) X-Google-Smtp-Source: AK7set+6ukRlNnPMorVKgITJrPzTzKu17JjDqYI2BLL8kpI6cDUWJr9XhaWXD1SLtCDcA5ikikx3 X-Received: by 2002:a17:902:c946:b0:189:f460:d24b with SMTP id i6-20020a170902c94600b00189f460d24bmr8382273pla.5.1675876691825; Wed, 08 Feb 2023 09:18:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675876691; cv=none; d=google.com; s=arc-20160816; b=AgwVNmB5gV89B/KYVHNBDYrmFw+Ucj7fXmHLXWNqUgjelHzTFDjnxPnGbHaNr62COt vrOVU8qK6GHmvarbnOVLhiTkLi3H4nWggzmO9TYfXPhP4aM9vbt8BrAp8BJPb/7W7DYK RQOFpAi5m2aNqo2sQziykWVETfeGqVZQclu0XesHfgSkfIW/QT431H6T0eg5eF6MlLjI +41brCU1t57yEUY+WfMvJRsnCYmIGF4IJkK+Vml/1JqA7qOBJhUgGCoQFEl/gDsahNFt TxhTf9bQcUAKnyMjOAfBLmKlgF58xgTCcwhbFfTVkMfJ0D4GePjcRf/tYvF1EjFHrQNt zyrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eMyBJaDypypFT8uaN+vrKnKGXXH7Q926RrCo5TpdUv0=; b=u+KyAKYCkhAsFrutXLrcFBeXd5+4tAkAWDSyCcIR/H7vCUzjiLF8GfD7hIfMsNYARN ArzNg83pHZp/VgCWbiUp/+BxAlGnXwsq8Dvy0Fii7xPgVpgW59nCsHiv5ApmaAUVqV/Z 5JPbvXAnhN+QAh3bcQW0N2F+pINoKqQzZBDa8TmBqqivoisHFE4H2K5hAD5YxhSjK99D Cu+qILxs5B+sl940r1bKvkDfHWwBg56HoJ3x+x8/zzbluwc2m/VkNLAeO1YP65Ce1zTX bJDBFQrRvGYJjNAhLwdSnJuw/Ncik3YmmgbtHz4heikqQG0wDA3Jwfp+E4SAB9yCrifk EX2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="MVL/l6kU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id be3-20020a170902aa0300b0019648e516dfsi16941849plb.352.2023.02.08.09.17.58; Wed, 08 Feb 2023 09:18:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="MVL/l6kU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231549AbjBHRQd (ORCPT + 99 others); Wed, 8 Feb 2023 12:16:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231292AbjBHRQc (ORCPT ); Wed, 8 Feb 2023 12:16:32 -0500 Received: from xry111.site (xry111.site [89.208.246.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DE5429436 for ; Wed, 8 Feb 2023 09:16:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xry111.site; s=default; t=1675876590; bh=KqQlLM65Y+42Fzaj+u5LF59N7vKKvEZNbihd+mJCsx8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MVL/l6kUiZfPLh/gLfrTpJEPAlvyM5VsrmcLwFxYDTWntpd3sTe5QN4PvCLenOlI8 DlKthMbYgk6B+W59CxJyI3H+bpTny2zaHdYXaUz2F7wfmwEvxxyMoXsDG4gZHBECJL pwuH06A0mO7Kl+Md9pLTJNAZPmhFPhJJeAjiYUl8= Received: from xry111-x57s1.. (unknown [IPv6:240e:358:1174:d100:dc73:854d:832e:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) (Authenticated sender: xry111@xry111.site) by xry111.site (Postfix) with ESMTPSA id BAFBA65C52; Wed, 8 Feb 2023 12:16:26 -0500 (EST) From: Xi Ruoyao To: Youling Tang , Huacai Chen , Jinyang He Cc: Xuerui Wang , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, Xi Ruoyao Subject: [PATCH v3 1/5] LoongArch: Use la.pcrel instead of la.abs when it's trivially possible Date: Thu, 9 Feb 2023 01:16:03 +0800 Message-Id: <20230208171607.391107-2-xry111@xry111.site> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230208171607.391107-1-xry111@xry111.site> References: <20230208171607.391107-1-xry111@xry111.site> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757284078154079264?= X-GMAIL-MSGID: =?utf-8?q?1757284078154079264?= Let's start to kill la.abs inpreparation for the subsequent support of the PIE kernel. Signed-off-by: Xi Ruoyao --- arch/loongarch/include/asm/stackframe.h | 2 +- arch/loongarch/include/asm/uaccess.h | 1 - arch/loongarch/kernel/entry.S | 2 +- arch/loongarch/kernel/head.S | 2 +- arch/loongarch/mm/tlbex.S | 3 +-- 5 files changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index 4ca953062b5b..7deb043ce387 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -90,7 +90,7 @@ .endm .macro set_saved_sp stackp temp temp2 - la.abs \temp, kernelsp + la.pcrel \temp, kernelsp #ifdef CONFIG_SMP LONG_ADD \temp, \temp, u0 #endif diff --git a/arch/loongarch/include/asm/uaccess.h b/arch/loongarch/include/asm/uaccess.h index 255899d4a7c3..0d22991ae430 100644 --- a/arch/loongarch/include/asm/uaccess.h +++ b/arch/loongarch/include/asm/uaccess.h @@ -22,7 +22,6 @@ extern u64 __ua_limit; #define __UA_ADDR ".dword" -#define __UA_LA "la.abs" #define __UA_LIMIT __ua_limit /* diff --git a/arch/loongarch/kernel/entry.S b/arch/loongarch/kernel/entry.S index d53b631c9022..2566977f2f68 100644 --- a/arch/loongarch/kernel/entry.S +++ b/arch/loongarch/kernel/entry.S @@ -20,7 +20,7 @@ .align 5 SYM_FUNC_START(handle_syscall) csrrd t0, PERCPU_BASE_KS - la.abs t1, kernelsp + la.pcrel t1, kernelsp add.d t1, t1, t0 move t2, sp ld.d sp, t1, 0 diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index 57bada6b4e93..aa6181714ec3 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -117,7 +117,7 @@ SYM_CODE_START(smpboot_entry) li.w t0, 0x00 # FPE=0, SXE=0, ASXE=0, BTE=0 csrwr t0, LOONGARCH_CSR_EUEN - la.abs t0, cpuboot_data + la.pcrel t0, cpuboot_data ld.d sp, t0, CPU_BOOT_STACK ld.d tp, t0, CPU_BOOT_TINFO diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S index 58781c6e4191..3dd2a9615cd9 100644 --- a/arch/loongarch/mm/tlbex.S +++ b/arch/loongarch/mm/tlbex.S @@ -24,8 +24,7 @@ move a0, sp REG_S a2, sp, PT_BVADDR li.w a1, \write - la.abs t0, do_page_fault - jirl ra, t0, 0 + bl do_page_fault RESTORE_ALL_AND_RET SYM_FUNC_END(tlb_do_page_fault_\write) .endm From patchwork Wed Feb 8 17:16:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xi Ruoyao X-Patchwork-Id: 54509 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3580348wrn; Wed, 8 Feb 2023 09:19:05 -0800 (PST) X-Google-Smtp-Source: AK7set/mGg67aKYeZp6uAvuuIT1/LE31aIlRUVfE123Ux471hlSquzBgx2c8Kk89T1ShEwEls3ER X-Received: by 2002:a05:6402:510f:b0:4aa:a515:58ad with SMTP id m15-20020a056402510f00b004aaa51558admr10693556edd.0.1675876744818; Wed, 08 Feb 2023 09:19:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675876744; cv=none; d=google.com; s=arc-20160816; b=eePJjOGRIjdCLf3Gaod/OtJhSot6JsjjfV5LFbSdGCdi152Lxa/yeDhEhiK/mcrhfa 1gnshG9jRac1l5eU2DMKt1acQ9S9lqicl76WccBqLJUEPNoULenNc2FJJZwKJJ0TR6Vx Q8dBZ5eq/l64GZqhy/r2UbYuM1crKiI6HMLPjjBE6aNWWkbE0wQPqx4r4eZp9rkMLzAM ioXDbV7aHPYHV4yzA4cKzdxZvHadQk2J7e9XW+XFJXOEhwNigaliqF2o60jYSbz1QuNJ u8T6sDtjFSSD8HJzIsANXG4fE8SOl7Tx1xgDEFpfj7Pj6lQlFmu7qUMKsICP9E7/cGta xHlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MQjIGAz5ihyv+TSenErPOiZ6YB4wggArlKQAbcxYAAI=; b=0LClPHPjA43Ey4nriIrblBbkOhSrHAKK48kzDXE6KAwv31tj/8Pb5oCOz4yo8prHD2 kiK8hb/9AneerGts+F3Lj4ttmG+p/Lb/TfnW92AWv9n700WEWCzfMzy6NoRQdBOmLTW+ uDGovnlPXSqdJKna02s6bZ7hjPxPJlKMRCQfI3C3PhqvsWDDZfs+Q29taVRofY9k3r3z 7vvebhmksrt+CV2HtftMuGEOvyqyk1JPCkqC/Ehz/ZY2ikVvOyZ8VsdBF2bidQMDILzj uM9dnp9QT6mx+eVQFJ7QWGMeL97KcvB9mhnVGuQMmsez/v8wrUM5SfMVw+Ts8Bgu53WI 8isQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="Ie3gUh/M"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id er22-20020a056402449600b004a21cd9fef7si19723705edb.323.2023.02.08.09.18.41; Wed, 08 Feb 2023 09:19:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="Ie3gUh/M"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231614AbjBHRQl (ORCPT + 99 others); Wed, 8 Feb 2023 12:16:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231346AbjBHRQh (ORCPT ); Wed, 8 Feb 2023 12:16:37 -0500 Received: from xry111.site (xry111.site [89.208.246.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7E174ED19 for ; Wed, 8 Feb 2023 09:16:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xry111.site; s=default; t=1675876595; bh=FjG2q/BqmWHWGDdVBl1Ir4njZnUEulypldGQfN1H+3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ie3gUh/MbhglAnDqK2TkQcTdZtmi8xGA6iq/3hKaKWtRK3uN1fk3I4+Y3btes9xrV Xn6luu77wXK8Fz1WJOGc6nilw/DldlP13iE4QuKpxBdiAAwLtvDGbGO7WyA/W2KeB/ HNRg/ea0KRLUY6Z3necLzeV2aq5uY0o7tdZB8hdI= Received: from xry111-x57s1.. (unknown [IPv6:240e:358:1174:d100:dc73:854d:832e:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) (Authenticated sender: xry111@xry111.site) by xry111.site (Postfix) with ESMTPSA id 6C55465C53; Wed, 8 Feb 2023 12:16:31 -0500 (EST) From: Xi Ruoyao To: Youling Tang , Huacai Chen , Jinyang He Cc: Xuerui Wang , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, Xi Ruoyao Subject: [PATCH v3 2/5] LoongArch: Use la.pcrel instead of la.abs for exception handlers Date: Thu, 9 Feb 2023 01:16:04 +0800 Message-Id: <20230208171607.391107-3-xry111@xry111.site> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230208171607.391107-1-xry111@xry111.site> References: <20230208171607.391107-1-xry111@xry111.site> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757284133919843758?= X-GMAIL-MSGID: =?utf-8?q?1757284133919843758?= It's needed to build the kernel as a PIE, or the linker will complain. For the consideration about performance, we copy the exception handlers to a dedicated 64 KB area for each CPU. So, the PC-relative offset calculated at link time will be incorrect and we need to relocate the exception handlers after copying them. For the simplicity, we don't use the ELF R_LARCH_* relocations, but code an relocation entry as simply (offset_in_the_handler, symbol_addr). For each exception handler, we also code the number of relocation entries. Then we can use the relocation information to fix up the handlers after copying them. Signed-off-by: Xi Ruoyao --- arch/loongarch/include/asm/inst.h | 1 + arch/loongarch/include/asm/setup.h | 6 +- arch/loongarch/include/asm/stackframe.h | 3 +- arch/loongarch/kernel/genex.S | 40 +++++- arch/loongarch/kernel/traps.c | 158 ++++++++++++++++++++---- arch/loongarch/mm/tlb.c | 23 ++-- arch/loongarch/mm/tlbex.S | 69 +++++++++-- 7 files changed, 255 insertions(+), 45 deletions(-) diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h index 7eedd83fd0d7..426054518a3d 100644 --- a/arch/loongarch/include/asm/inst.h +++ b/arch/loongarch/include/asm/inst.h @@ -32,6 +32,7 @@ enum reg1i20_op { lu12iw_op = 0x0a, lu32id_op = 0x0b, pcaddi_op = 0x0c, + pcalau12i_op = 0x0d, pcaddu12i_op = 0x0e, pcaddu18i_op = 0x0f, }; diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h index 72ead58039f3..f0a2b34365f1 100644 --- a/arch/loongarch/include/asm/setup.h +++ b/arch/loongarch/include/asm/setup.h @@ -11,6 +11,9 @@ #define VECSIZE 0x200 +struct handler_reloc; + +extern struct handler_reloc *eentry_reloc[]; extern unsigned long eentry; extern unsigned long tlbrentry; extern char init_command_line[COMMAND_LINE_SIZE]; @@ -18,7 +21,8 @@ extern void tlb_init(int cpu); extern void cpu_cache_init(void); extern void cache_error_setup(void); extern void per_cpu_trap_init(int cpu); -extern void set_handler(unsigned long offset, void *addr, unsigned long len); +extern void set_handler(unsigned long exccode, void *addr); extern void set_merr_handler(unsigned long offset, void *addr, unsigned long len); +extern void reloc_handler(unsigned long handler, struct handler_reloc *rel); #endif /* __SETUP_H */ diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index 7deb043ce387..bbec1e56b61b 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -77,7 +77,8 @@ * new value in sp. */ .macro get_saved_sp docfi=0 - la.abs t1, kernelsp + /* The label is used for generating reloc tables for handlers */ +514: la.pcrel t1, t0, kernelsp #ifdef CONFIG_SMP csrrd t0, PERCPU_BASE_KS LONG_ADD t1, t1, t0 diff --git a/arch/loongarch/kernel/genex.S b/arch/loongarch/kernel/genex.S index 7e5c293ed89f..005a10fe5a50 100644 --- a/arch/loongarch/kernel/genex.S +++ b/arch/loongarch/kernel/genex.S @@ -34,7 +34,7 @@ SYM_FUNC_END(__arch_cpu_idle) SYM_FUNC_START(handle_vint) BACKUP_T0T1 SAVE_ALL - la.abs t1, __arch_cpu_idle +0: la.pcrel t1, t2, __arch_cpu_idle LONG_L t0, sp, PT_ERA /* 32 byte rollback region */ ori t0, t0, 0x1f @@ -43,11 +43,25 @@ SYM_FUNC_START(handle_vint) LONG_S t0, sp, PT_ERA 1: move a0, sp move a1, sp - la.abs t0, do_vint +2: la.pcrel t0, t2, do_vint jirl ra, t0, 0 RESTORE_ALL_AND_RET SYM_FUNC_END(handle_vint) +SYM_DATA_START(rel_handle_vint) +LONG 3 + +LONG 514b - handle_vint +LONG kernelsp + +LONG 0b - handle_vint +LONG __arch_cpu_idle + +LONG 2b - handle_vint +LONG do_vint + +SYM_DATA_END(rel_handle_vint) + SYM_FUNC_START(except_vec_cex) b cache_parity_error SYM_FUNC_END(except_vec_cex) @@ -72,12 +86,24 @@ SYM_FUNC_END(except_vec_cex) SAVE_ALL build_prep_\prep move a0, sp - la.abs t0, do_\handler + 667: + la.pcrel t0, t1, do_\handler jirl ra, t0, 0 668: RESTORE_ALL_AND_RET SYM_FUNC_END(handle_\exception) SYM_DATA(unwind_hint_\exception, .word 668b - 666b) + + SYM_DATA_START(rel_handle_\exception) + LONG 2 + + LONG 514b - 666b + LONG kernelsp + + LONG 667b - 666b + LONG do_\handler + + SYM_DATA_END(rel_handle_\exception) .endm BUILD_HANDLER ade ade badv @@ -93,6 +119,12 @@ SYM_FUNC_END(except_vec_cex) BUILD_HANDLER reserved reserved none /* others */ SYM_FUNC_START(handle_sys) - la.abs t0, handle_syscall + la.pcrel t0, t1, handle_syscall jr t0 SYM_FUNC_END(handle_sys) + +SYM_DATA_START(rel_handle_sys) +LONG 1 +LONG 0 +LONG handle_syscall +SYM_DATA_END(rel_handle_sys) diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c index c38a146a973b..7e073854f493 100644 --- a/arch/loongarch/kernel/traps.c +++ b/arch/loongarch/kernel/traps.c @@ -62,6 +62,127 @@ extern asmlinkage void handle_reserved(void); extern asmlinkage void handle_watch(void); extern asmlinkage void handle_vint(void); +struct handler_reloc_entry { + unsigned long offset; + unsigned long sym; +}; + +struct handler_reloc { + unsigned long cnt; + struct handler_reloc_entry entries[]; +}; + +extern struct handler_reloc rel_handle_tlb_load; +extern struct handler_reloc rel_handle_tlb_store; +extern struct handler_reloc rel_handle_tlb_modify; +extern struct handler_reloc rel_handle_tlb_protect; +extern struct handler_reloc rel_handle_ade; +extern struct handler_reloc rel_handle_ale; +extern struct handler_reloc rel_handle_sys; +extern struct handler_reloc rel_handle_bp; +extern struct handler_reloc rel_handle_ri; +extern struct handler_reloc rel_handle_fpu; +extern struct handler_reloc rel_handle_lsx; +extern struct handler_reloc rel_handle_lasx; +extern struct handler_reloc rel_handle_fpe; +extern struct handler_reloc rel_handle_lbt; +extern struct handler_reloc rel_handle_watch; +extern struct handler_reloc rel_handle_reserved; +extern struct handler_reloc rel_handle_vint; + +struct handler_reloc *eentry_reloc[128] = { + [0] = NULL, /* merr handler */ + [EXCCODE_TLBL] = &rel_handle_tlb_load, + [EXCCODE_TLBS] = &rel_handle_tlb_store, + [EXCCODE_TLBI] = &rel_handle_tlb_load, + [EXCCODE_TLBM] = &rel_handle_tlb_modify, + [EXCCODE_TLBNR] = &rel_handle_tlb_protect, + [EXCCODE_TLBNX] = &rel_handle_tlb_protect, + [EXCCODE_TLBPE] = &rel_handle_tlb_protect, + [EXCCODE_ADE] = &rel_handle_ade, + [EXCCODE_ALE] = &rel_handle_ale, + [EXCCODE_SYS] = &rel_handle_sys, + [EXCCODE_BP] = &rel_handle_bp, + [EXCCODE_INE] = &rel_handle_ri, + [EXCCODE_IPE] = &rel_handle_ri, + [EXCCODE_FPDIS] = &rel_handle_fpu, + [EXCCODE_LSXDIS] = &rel_handle_lsx, + [EXCCODE_LASXDIS] = &rel_handle_lasx, + [EXCCODE_FPE] = &rel_handle_fpe, + [EXCCODE_BTDIS] = &rel_handle_lbt, + [EXCCODE_WATCH] = &rel_handle_watch, + [(EXCCODE_WATCH + 1) ... (EXCCODE_INT_START - 1)] = &rel_handle_reserved, + [EXCCODE_INT_START ... (EXCCODE_INT_END - 1)] = &rel_handle_vint, +}; + +void reloc_handler(unsigned long handler, struct handler_reloc *rel) +{ + if (!rel) + return; + + for (unsigned long i = 0; i < rel->cnt; i++) { + unsigned long pc = handler + rel->entries[i].offset; + union loongarch_instruction *insn = + (union loongarch_instruction *)pc; + u32 imm[4]; + unsigned long v = rel->entries[i].sym; + + /* GNU as >= 2.40 uses pcalau12i for la.pcrel, but GNU ld <= 2.39 + * uses pcaddu12i. + */ + if (insn->reg1i20_format.opcode == pcalau12i_op) { + /* Use s32 deliberately for sign extension. */ + s32 offset_hi20 = ((v + 0x800) & ~0xfff) - + (pc & ~0xfff); + unsigned long anchor = (pc & ~0xfff) + offset_hi20; + unsigned long offset_rem = v - anchor; + + imm[0] = (offset_hi20 >> 12) & 0xfffff; + imm[1] = v & 0xfff; + imm[2] = (offset_rem >> 32) & 0xfffff; + imm[3] = offset_rem >> 52; + } else if (insn->reg1i20_format.opcode == pcaddu12i_op) { + /* Use s32 deliberately for sign extension. */ + s32 offset_lo = v - pc; + unsigned long offset_hi = v - pc - offset_lo; + + imm[0] = (offset_lo >> 12) & 0xfffff; + imm[1] = offset_lo & 0xfff; + imm[2] = (offset_hi >> 32) & 0xfffff; + imm[3] = offset_hi >> 52; + } else + panic("Cannot fixup la.pcrel for exception handler at %lu: unexpected instruction %d!", + pc, insn->word); + + insn[0].reg1i20_format.immediate = imm[0]; + insn[1].reg2i12_format.immediate = imm[1]; + insn[2].reg1i20_format.immediate = imm[2]; + insn[3].reg2i12_format.immediate = imm[3]; + } +} + +/* Install CPU exception handler */ +static void do_set_handler(unsigned long exccode, void *addr, + struct handler_reloc *rel) +{ + unsigned long dest_addr = eentry + exccode * VECSIZE; + + memcpy((void *)dest_addr, addr, VECSIZE); + reloc_handler(dest_addr, rel); + local_flush_icache_range(dest_addr, dest_addr + VECSIZE); +} + +/* Install CPU exception handler, with the reloc table from eentry_reloc */ +void set_handler(unsigned long exccode, void *addr) +{ + do_set_handler(exccode, addr, eentry_reloc[exccode]); +} + +static void set_handler_reserved(unsigned long exccode) +{ + do_set_handler(exccode, handle_reserved, &rel_handle_reserved); +} + static void show_backtrace(struct task_struct *task, const struct pt_regs *regs, const char *loglvl, bool user) { @@ -704,19 +825,12 @@ void per_cpu_trap_init(int cpu) /* Initialise exception handlers */ if (cpu == 0) for (i = 0; i < 64; i++) - set_handler(i * VECSIZE, handle_reserved, VECSIZE); + set_handler_reserved(i); tlb_init(cpu); cpu_cache_init(); } -/* Install CPU exception handler */ -void set_handler(unsigned long offset, void *addr, unsigned long size) -{ - memcpy((void *)(eentry + offset), addr, size); - local_flush_icache_range(eentry + offset, eentry + offset + size); -} - static const char panic_null_cerr[] = "Trying to set NULL cache error exception handler\n"; @@ -741,20 +855,20 @@ void __init trap_init(void) /* Set interrupt vector handler */ for (i = EXCCODE_INT_START; i < EXCCODE_INT_END; i++) - set_handler(i * VECSIZE, handle_vint, VECSIZE); - - set_handler(EXCCODE_ADE * VECSIZE, handle_ade, VECSIZE); - set_handler(EXCCODE_ALE * VECSIZE, handle_ale, VECSIZE); - set_handler(EXCCODE_SYS * VECSIZE, handle_sys, VECSIZE); - set_handler(EXCCODE_BP * VECSIZE, handle_bp, VECSIZE); - set_handler(EXCCODE_INE * VECSIZE, handle_ri, VECSIZE); - set_handler(EXCCODE_IPE * VECSIZE, handle_ri, VECSIZE); - set_handler(EXCCODE_FPDIS * VECSIZE, handle_fpu, VECSIZE); - set_handler(EXCCODE_LSXDIS * VECSIZE, handle_lsx, VECSIZE); - set_handler(EXCCODE_LASXDIS * VECSIZE, handle_lasx, VECSIZE); - set_handler(EXCCODE_FPE * VECSIZE, handle_fpe, VECSIZE); - set_handler(EXCCODE_BTDIS * VECSIZE, handle_lbt, VECSIZE); - set_handler(EXCCODE_WATCH * VECSIZE, handle_watch, VECSIZE); + set_handler(i, handle_vint); + + set_handler(EXCCODE_ADE, handle_ade); + set_handler(EXCCODE_ALE, handle_ale); + set_handler(EXCCODE_SYS, handle_sys); + set_handler(EXCCODE_BP, handle_bp); + set_handler(EXCCODE_INE, handle_ri); + set_handler(EXCCODE_IPE, handle_ri); + set_handler(EXCCODE_FPDIS, handle_fpu); + set_handler(EXCCODE_LSXDIS, handle_lsx); + set_handler(EXCCODE_LASXDIS, handle_lasx); + set_handler(EXCCODE_FPE, handle_fpe); + set_handler(EXCCODE_BTDIS, handle_lbt); + set_handler(EXCCODE_WATCH, handle_watch); cache_error_setup(); diff --git a/arch/loongarch/mm/tlb.c b/arch/loongarch/mm/tlb.c index 8bad6b0cff59..6f70aab7202a 100644 --- a/arch/loongarch/mm/tlb.c +++ b/arch/loongarch/mm/tlb.c @@ -253,7 +253,6 @@ static void output_pgtable_bits_defines(void) #ifdef CONFIG_NUMA unsigned long pcpu_handlers[NR_CPUS]; #endif -extern long exception_handlers[VECSIZE * 128 / sizeof(long)]; void setup_tlb_handler(int cpu) { @@ -264,19 +263,20 @@ void setup_tlb_handler(int cpu) if (cpu == 0) { memcpy((void *)tlbrentry, handle_tlb_refill, 0x80); local_flush_icache_range(tlbrentry, tlbrentry + 0x80); - set_handler(EXCCODE_TLBI * VECSIZE, handle_tlb_load, VECSIZE); - set_handler(EXCCODE_TLBL * VECSIZE, handle_tlb_load, VECSIZE); - set_handler(EXCCODE_TLBS * VECSIZE, handle_tlb_store, VECSIZE); - set_handler(EXCCODE_TLBM * VECSIZE, handle_tlb_modify, VECSIZE); - set_handler(EXCCODE_TLBNR * VECSIZE, handle_tlb_protect, VECSIZE); - set_handler(EXCCODE_TLBNX * VECSIZE, handle_tlb_protect, VECSIZE); - set_handler(EXCCODE_TLBPE * VECSIZE, handle_tlb_protect, VECSIZE); + set_handler(EXCCODE_TLBI, handle_tlb_load); + set_handler(EXCCODE_TLBL, handle_tlb_load); + set_handler(EXCCODE_TLBS, handle_tlb_store); + set_handler(EXCCODE_TLBM, handle_tlb_modify); + set_handler(EXCCODE_TLBNR, handle_tlb_protect); + set_handler(EXCCODE_TLBNX, handle_tlb_protect); + set_handler(EXCCODE_TLBPE, handle_tlb_protect); } #ifdef CONFIG_NUMA else { void *addr; + unsigned long addr_ul; struct page *page; - const int vec_sz = sizeof(exception_handlers); + const int vec_sz = VECSIZE * 128; if (pcpu_handlers[cpu]) return; @@ -286,8 +286,11 @@ void setup_tlb_handler(int cpu) return; addr = page_address(page); + addr_ul = (unsigned long)addr; pcpu_handlers[cpu] = (unsigned long)addr; - memcpy((void *)addr, (void *)eentry, vec_sz); + memcpy(addr, (void *)eentry, vec_sz); + for (unsigned long i = 0; i < 128; i++) + reloc_handler(addr_ul + i * VECSIZE, eentry_reloc[i]); local_flush_icache_range((unsigned long)addr, (unsigned long)addr + vec_sz); csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_EENTRY); csr_write64(pcpu_handlers[cpu], LOONGARCH_CSR_MERRENTRY); diff --git a/arch/loongarch/mm/tlbex.S b/arch/loongarch/mm/tlbex.S index 3dd2a9615cd9..044c2190771a 100644 --- a/arch/loongarch/mm/tlbex.S +++ b/arch/loongarch/mm/tlbex.S @@ -39,11 +39,21 @@ SYM_FUNC_START(handle_tlb_protect) move a1, zero csrrd a2, LOONGARCH_CSR_BADV REG_S a2, sp, PT_BVADDR - la.abs t0, do_page_fault +1: la.pcrel t0, t1, do_page_fault jirl ra, t0, 0 RESTORE_ALL_AND_RET SYM_FUNC_END(handle_tlb_protect) +SYM_DATA_START(rel_handle_tlb_protect) + LONG 2 + + LONG 514b - handle_tlb_protect + LONG kernelsp + + LONG 1b - handle_tlb_protect + LONG do_page_fault +SYM_DATA_END(rel_handle_tlb_protect) + SYM_FUNC_START(handle_tlb_load) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -115,7 +125,8 @@ smp_pgtable_change_load: #ifdef CONFIG_64BIT vmalloc_load: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_load overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_load #endif @@ -186,10 +197,24 @@ tlb_huge_update_load: nopage_tlb_load: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_0 +2: la.pcrel t0, t1, tlb_do_page_fault_0 jr t0 SYM_FUNC_END(handle_tlb_load) +SYM_DATA_START(rel_handle_tlb_load) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_load + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_load + LONG tlb_do_page_fault_0 +SYM_DATA_END(rel_handle_tlb_load) + SYM_FUNC_START(handle_tlb_store) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -262,7 +287,8 @@ smp_pgtable_change_store: #ifdef CONFIG_64BIT vmalloc_store: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_store overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_store #endif @@ -335,10 +361,24 @@ tlb_huge_update_store: nopage_tlb_store: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_1 +2: la.pcrel t0, t1, tlb_do_page_fault_1 jr t0 SYM_FUNC_END(handle_tlb_store) +SYM_DATA_START(rel_handle_tlb_store) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_store + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_store + LONG tlb_do_page_fault_1 +SYM_DATA_END(rel_handle_tlb_store) + SYM_FUNC_START(handle_tlb_modify) csrwr t0, EXCEPTION_KS0 csrwr t1, EXCEPTION_KS1 @@ -410,7 +450,8 @@ smp_pgtable_change_modify: #ifdef CONFIG_64BIT vmalloc_modify: - la.abs t1, swapper_pg_dir + /* The first insn of vmalloc_done_modify overwrites ra */ +1: la.pcrel t1, ra, swapper_pg_dir b vmalloc_done_modify #endif @@ -482,10 +523,24 @@ tlb_huge_update_modify: nopage_tlb_modify: dbar 0 csrrd ra, EXCEPTION_KS2 - la.abs t0, tlb_do_page_fault_1 +2: la.pcrel t0, t1, tlb_do_page_fault_1 jr t0 SYM_FUNC_END(handle_tlb_modify) +SYM_DATA_START(rel_handle_tlb_modify) +#ifdef CONFIG_64BIT + LONG 2 + + LONG 1b - handle_tlb_modify + LONG swapper_pg_dir +#else + LONG 1 +#endif + + LONG 2b - handle_tlb_modify + LONG tlb_do_page_fault_1 +SYM_DATA_END(rel_handle_tlb_modify) + SYM_FUNC_START(handle_tlb_refill) csrwr t0, LOONGARCH_CSR_TLBRSAVE csrrd t0, LOONGARCH_CSR_PGD From patchwork Wed Feb 8 17:16:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xi Ruoyao X-Patchwork-Id: 54508 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3580092wrn; Wed, 8 Feb 2023 09:18:40 -0800 (PST) X-Google-Smtp-Source: AK7set8lw2zuEsRfVjVqXDo+u+GCWSndnIen6ZJZrtLkxshG4fL2zxwFtxfUFELjG66itR78B1kg X-Received: by 2002:a05:6402:278a:b0:4aa:b394:7b28 with SMTP id b10-20020a056402278a00b004aab3947b28mr10154703ede.3.1675876719916; Wed, 08 Feb 2023 09:18:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675876719; cv=none; d=google.com; s=arc-20160816; b=swA3zc5oEyOqmlExFXES/ch3aTRUZ/YJNdu7W/oQDTFPaguEefzMIcNomVvE/+7hWW oE35d8UO//0M3r2gPHZcltQG7Wv/ciDMYHw01opsEacB3sXqx7G17F7dvZUC+ESw7xpG 4/Vi57qynAjT2bBOgEKRoELwVID+pLOFUgw20tFyGLPf7Gme029gPk0xyRzNgv4EhNCE 87IOT2BzQRNvz2iiHx4bynP5yGdkFBefeJX9ozVcQ99CpdPsfBDOLMRSBS41KcLcVJ8f 6NYPPm/0waoLe1n8R2Lxa3XLE2THfO1bdzsamwEL+iuM+j2ApBM0k0jTNT1GJ3h8tJRh KMAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y1P9spuCbkWHfrrcSMf4sCzvfete5jIqJkvDtoI5CMk=; b=fzobL+r0pdmP1TrNaOwUJ4Wp84a+AlUzImG3o42oCPgdOeLiYW9XheyOwRO4+XdJtx U32b+uRtr5sUZwWYI92FCbKK8vyfr7tWLlZ/7885zTb1zgEZqFUCaqbk9IGl17RwrYz3 N/YWpraSvGoAHIl/GlXrT4X6hJ+ZtTeihPYyeaYuPHQAi/nhEbjLjpDNN5FhQGJKFSp1 rGKHPf9XmPAOat68T6Mph/UqgKzEB0RC3boT8qSrxDnPTqxi3LRPTn2YDtb0piYs71qZ OTZMiuhW5DOXZFCVnTreP2eBe86whBujfNi3MSTcvBMoMvYb9V25aFigeC7KsGDXnYmH PDpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b=bTJCWvXW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p19-20020aa7d313000000b004aadd73ded4si3191280edq.605.2023.02.08.09.18.16; Wed, 08 Feb 2023 09:18:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b=bTJCWvXW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231697AbjBHRQo (ORCPT + 99 others); Wed, 8 Feb 2023 12:16:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231626AbjBHRQl (ORCPT ); Wed, 8 Feb 2023 12:16:41 -0500 Received: from xry111.site (xry111.site [89.208.246.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 848DB4FAC0 for ; Wed, 8 Feb 2023 09:16:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xry111.site; s=default; t=1675876600; bh=h1F/YrWW3XcqAQ47zVGYpE7dCV+4pws3M6qqm11X04s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bTJCWvXWM8Q6dno6KlrLRbNe2ft4qV9CVD5bVpswhtMpAjddknkD7Agan6hxb7ezR HamGmkv3iExs4uewZorU53ThtWzwcQWOn0W64nrX7uR8rHTvxhftkj9ztbzxUuXt01 oqszZ07bC5I2ofKd3M1gcj18kdGs1QM4vZmbG9J8= Received: from xry111-x57s1.. (unknown [IPv6:240e:358:1174:d100:dc73:854d:832e:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) (Authenticated sender: xry111@xry111.site) by xry111.site (Postfix) with ESMTPSA id 9184265C4F; Wed, 8 Feb 2023 12:16:36 -0500 (EST) From: Xi Ruoyao To: Youling Tang , Huacai Chen , Jinyang He Cc: Xuerui Wang , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH v3 3/5] LoongArch: Add JUMP_LINK_ADDR macro implementation to avoid using la.abs Date: Thu, 9 Feb 2023 01:16:05 +0800 Message-Id: <20230208171607.391107-4-xry111@xry111.site> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230208171607.391107-1-xry111@xry111.site> References: <20230208171607.391107-1-xry111@xry111.site> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757284107549152160?= X-GMAIL-MSGID: =?utf-8?q?1757284107549152160?= From: Youling Tang Add JUMP_LINK_ADDR macro implementation to avoid using la.abs. Signed-off-by: Youling Tang --- arch/loongarch/include/asm/stackframe.h | 8 ++++++++ arch/loongarch/kernel/head.S | 10 ++++------ arch/loongarch/power/suspend_asm.S | 5 ++--- 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index bbec1e56b61b..cd8240858599 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -217,4 +217,12 @@ RESTORE_SP_AND_RET \docfi .endm +/* Jump to the virtual address of the link. */ + .macro JUMP_LINK_ADDR + li.d t0, CACHE_BASE + pcaddi t1, 0 + or t0, t0, t1 + jirl zero, t0, 0xc + .endm + #endif /* _ASM_STACKFRAME_H */ diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index aa6181714ec3..e8a4bf9d7599 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -52,9 +52,8 @@ SYM_CODE_START(kernel_entry) # kernel entry point /* We might not get launched at the address the kernel is linked to, so we jump there. */ - la.abs t0, 0f - jr t0 -0: + JUMP_LINK_ADDR + /* Enable PG */ li.w t0, 0xb0 # PLV=0, IE=0, PG=1 csrwr t0, LOONGARCH_CSR_CRMD @@ -106,9 +105,8 @@ SYM_CODE_START(smpboot_entry) li.d t0, CSR_DMW1_INIT # CA, PLV0 csrwr t0, LOONGARCH_CSR_DMWIN1 - la.abs t0, 0f - jr t0 -0: + JUMP_LINK_ADDR + /* Enable PG */ li.w t0, 0xb0 # PLV=0, IE=0, PG=1 csrwr t0, LOONGARCH_CSR_CRMD diff --git a/arch/loongarch/power/suspend_asm.S b/arch/loongarch/power/suspend_asm.S index eb2675642f9f..596a682a7924 100644 --- a/arch/loongarch/power/suspend_asm.S +++ b/arch/loongarch/power/suspend_asm.S @@ -78,9 +78,8 @@ SYM_INNER_LABEL(loongarch_wakeup_start, SYM_L_GLOBAL) li.d t0, CSR_DMW1_INIT # CA, PLV0 csrwr t0, LOONGARCH_CSR_DMWIN1 - la.abs t0, 0f - jr t0 -0: + JUMP_LINK_ADDR + la.pcrel t0, acpi_saved_sp ld.d sp, t0, 0 SETUP_WAKEUP From patchwork Wed Feb 8 17:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xi Ruoyao X-Patchwork-Id: 54507 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3580068wrn; Wed, 8 Feb 2023 09:18:37 -0800 (PST) X-Google-Smtp-Source: AK7set+euTLy9tx3KXmwSnIoqTbFF+zwF6uEt3lttI0BAdO+GAQc/aq3zz/KvNosRR7upb6RyY9H X-Received: by 2002:a05:6a20:7d83:b0:be:adc8:96c8 with SMTP id v3-20020a056a207d8300b000beadc896c8mr10151698pzj.3.1675876717292; Wed, 08 Feb 2023 09:18:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675876717; cv=none; d=google.com; s=arc-20160816; b=BYFe43gGKqgAaxKG/Qjwga7qyh054t1VEzYolYXu3c3Nf5K73Em8Y9lai5bWu0b0zN 6hyXdDVAjf7van5XkBfYVZRWg1EKDbODAqcjypazmEiu0kGGSYVadoVjIxEHwSqo/KGR p4mLFmHlM+USDglbflnbeO6t2qLabY5hv5emFq0NWrtdvM7BXfnreFF3/pURaJY3J80c zTa+nyBchYxX9ffxXfoskOyBK8g9vDz/06TpiLVFcw8wh2by+L+QpQY6rCS1tlnB+XjW Ui/hr3ec/xKiFpQHnz2+/qkqmtkKm+puHoGMWjOQOwkdYRfBNJX9ShRg0huseAgCwNK0 eOmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n6JiozY0jn02s5KZIkH1L/ouJQdLUL6Jbgvb1L/wCyU=; b=UYdUbTolrB52g9NcRdMq4xYASdbY4wXvylOKtI43+J2negNLbMVW8lPSQS0DF9qgi9 a2XJzFWZgAocOCOSr91U65lOGsso6mlDQvI5puTdCLbNy7i/2kSSajF+Zalt6ybzOO/Y KPK5QjIrkHE+W1JSRXvU/0L6N3yt9ma8gdEe1FR5b4zk0yL+QSNl8afvw4iMyZ05j9xk GeQMAbSST3cEqOBrDrEq0moC9trWzG9CM6S1vrjNS1y2FojcYS85uHyUApjfAxFuIJvF JMKaPF3MZmsB9QeyZ+MVBL0zN2q4VQjmx1wNI0GoCYvZw09qk63J1EB8LkUlvTzAbZA6 a98g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b=PGcbUbIX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s28-20020a63925c000000b004f1e98f79aesi19555692pgn.868.2023.02.08.09.18.24; Wed, 08 Feb 2023 09:18:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b=PGcbUbIX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231626AbjBHRRI (ORCPT + 99 others); Wed, 8 Feb 2023 12:17:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231691AbjBHRRC (ORCPT ); Wed, 8 Feb 2023 12:17:02 -0500 Received: from xry111.site (xry111.site [IPv6:2001:470:683e::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E68049421 for ; Wed, 8 Feb 2023 09:16:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xry111.site; s=default; t=1675876607; bh=sYO9Eb5bFjdOGiuca613wuxLe3kLk565FA8I9/Mll+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PGcbUbIXGNH7OgrUI02nWRGUucyb+FkY0yzQ4eBwIoGBg3seYqs0Bho7f5N1UyTgV b7lO+PWEwv/sMdUX6kCTar8vHBJPKiCfKJ8otiyIFH5GFu1PVzyVyIZvTtVfbPp4lF bKzkYezoRTps59bVRpZ1sYXdgXnB3GnKNRdXu+rU= Received: from xry111-x57s1.. (unknown [IPv6:240e:358:1174:d100:dc73:854d:832e:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) (Authenticated sender: xry111@xry111.site) by xry111.site (Postfix) with ESMTPSA id 7F8AA65C52; Wed, 8 Feb 2023 12:16:40 -0500 (EST) From: Xi Ruoyao To: Youling Tang , Huacai Chen , Jinyang He Cc: Xuerui Wang , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, Xi Ruoyao Subject: [PATCH v3 4/5] LoongArch: Add support for kernel relocation Date: Thu, 9 Feb 2023 01:16:06 +0800 Message-Id: <20230208171607.391107-5-xry111@xry111.site> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230208171607.391107-1-xry111@xry111.site> References: <20230208171607.391107-1-xry111@xry111.site> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757284104841950826?= X-GMAIL-MSGID: =?utf-8?q?1757284104841950826?= From: Youling Tang arch/loongarch/kernel/relocate.c contains the functions necessary to relocate the kernel elsewhere in memory. The kernel makes a copy of itself at the new address. It uses the relocation table inserted by the relocs tool to fix symbol references within the new image. If copy/relocation is successful then the entry point of the new kernel is returned, otherwise fall back to starting the kernel in place. Signed-off-by: Youling Tang Signed-off-by: Xi Ruoyao # Fix compiler warnings --- arch/loongarch/Kconfig | 15 +++++ arch/loongarch/Makefile | 5 ++ arch/loongarch/kernel/Makefile | 2 + arch/loongarch/kernel/head.S | 18 ++++++ arch/loongarch/kernel/relocate.c | 96 +++++++++++++++++++++++++++++ arch/loongarch/kernel/vmlinux.lds.S | 11 +++- 6 files changed, 145 insertions(+), 2 deletions(-) create mode 100644 arch/loongarch/kernel/relocate.c diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 9cc8b84f7eb0..089a4695b1b3 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -48,6 +48,7 @@ config LOONGARCH select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING + select SYS_SUPPORTS_RELOCATABLE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF select ARCH_USE_QUEUED_RWLOCKS @@ -229,6 +230,11 @@ config SCHED_OMIT_FRAME_POINTER config AS_HAS_EXPLICIT_RELOCS def_bool $(as-instr,x:pcalau12i \$t0$(comma)%pc_hi20(x)) +config SYS_SUPPORTS_RELOCATABLE + bool + help + Selected if the platform supports relocating the kernel. + menu "Kernel type and options" source "kernel/Kconfig.hz" @@ -474,6 +480,15 @@ config PHYSICAL_START specified in the "crashkernel=YM@XM" command line boot parameter passed to the panic-ed kernel). +config RELOCATABLE + bool "Relocatable kernel" + depends on SYS_SUPPORTS_RELOCATABLE + help + This builds the kernel as a Position Independent Executable (PIE), + which retains all relocation metadata required to relocate the + kernel binary at runtime to a different virtual address than the + address it was linked at. + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" depends on PROC_FS diff --git a/arch/loongarch/Makefile b/arch/loongarch/Makefile index 4402387d2755..27b5a70ff31c 100644 --- a/arch/loongarch/Makefile +++ b/arch/loongarch/Makefile @@ -71,6 +71,11 @@ KBUILD_AFLAGS_MODULE += -Wa,-mla-global-with-abs KBUILD_CFLAGS_MODULE += -fplt -Wa,-mla-global-with-abs,-mla-local-with-abs endif +ifeq ($(CONFIG_RELOCATABLE),y) +LDFLAGS_vmlinux += -static -pie --no-dynamic-linker -z notext +KBUILD_CFLAGS_KERNEL += -fPIE +endif + cflags-y += -ffreestanding cflags-y += $(call cc-option, -mno-check-zero-division) diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile index c8cfbd562921..3341dd5f0926 100644 --- a/arch/loongarch/kernel/Makefile +++ b/arch/loongarch/kernel/Makefile @@ -31,6 +31,8 @@ endif obj-$(CONFIG_MODULES) += module.o module-sections.o obj-$(CONFIG_STACKTRACE) += stacktrace.o +obj-$(CONFIG_RELOCATABLE) += relocate.o + obj-$(CONFIG_PROC_FS) += proc.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index e8a4bf9d7599..6db1549177ad 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -88,7 +88,25 @@ SYM_CODE_START(kernel_entry) # kernel entry point PTR_ADD sp, sp, tp set_saved_sp sp, t0, t1 +#ifdef CONFIG_RELOCATABLE + /* Copy kernel and apply the relocations */ + bl relocate_kernel + + /* Repoint the sp into the new kernel image */ + PTR_LI sp, (_THREAD_SIZE - 32 - PT_SIZE) + PTR_ADD sp, sp, tp + set_saved_sp sp, t0, t1 + PTR_ADDI sp, sp, -4 * SZREG # init stack pointer + + /* + * relocate_kernel returns the entry point either + * in the relocated kernel or the original if for + * some reason relocation failed. + */ + jr a0 +#else bl start_kernel +#endif ASM_BUG() SYM_CODE_END(kernel_entry) diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c new file mode 100644 index 000000000000..9c61e03d3e91 --- /dev/null +++ b/arch/loongarch/kernel/relocate.c @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Support for Kernel relocation at boot time + * + * Copyright (C) 2023 Loongson Technology Corporation Limited + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#define RELOCATED(x) ((void *)((long)x + offset)) + +extern long __rela_dyn_start; +extern long __rela_dyn_end; + +/* + * Choose a new address for the kernel, for now we'll hard + * code the destination. + */ +static inline void __init *determine_relocation_address(void) +{ + return (void *)(CACHE_BASE + 0x02000000); +} + +static inline int __init relocation_addr_valid(void *loc_new) +{ + if ((unsigned long)loc_new & 0x0000ffff) { + /* Inappropriately aligned new location */ + return 0; + } + if ((unsigned long)loc_new < (unsigned long)_end) { + /* New location overlaps original kernel */ + return 0; + } + return 1; +} + +void *__init relocate_kernel(void) +{ + Elf64_Rela *rela, *rela_end; + void *loc_new; + unsigned long kernel_length; + long offset = 0; + /* Default to original kernel entry point */ + void *kernel_entry = start_kernel; + + kernel_length = (long)(_end) - (long)(_text); + + loc_new = determine_relocation_address(); + + /* Sanity check relocation address */ + if (relocation_addr_valid(loc_new)) + offset = (unsigned long)loc_new - (unsigned long)(_text); + + if (offset) { + /* Copy the kernel to it's new location */ + memcpy(loc_new, _text, kernel_length); + + /* Sync the caches ready for execution of new kernel */ + __asm__ __volatile__ ( + "ibar 0 \t\n" + "dbar 0 \t\n"); + + rela = (Elf64_Rela *)RELOCATED(&__rela_dyn_start); + rela_end = (Elf64_Rela *)RELOCATED(&__rela_dyn_end); + + for ( ; rela < rela_end; rela++) { + Elf64_Addr addr = rela->r_offset; + Elf64_Addr relocated_addr = rela->r_addend; + + if (rela->r_info != R_LARCH_RELATIVE) + continue; + + if (relocated_addr >= VMLINUX_LOAD_ADDRESS) + relocated_addr = + (Elf64_Addr)RELOCATED(relocated_addr); + + *(Elf64_Addr *)RELOCATED(addr) = relocated_addr; + + } + + /* The current thread is now within the relocated image */ + __current_thread_info = RELOCATED(__current_thread_info); + + /* Return the new kernel's entry point */ + kernel_entry = RELOCATED(start_kernel); + } + + return kernel_entry; +} diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S index 733b16e8d55d..aec0b6567d24 100644 --- a/arch/loongarch/kernel/vmlinux.lds.S +++ b/arch/loongarch/kernel/vmlinux.lds.S @@ -70,6 +70,8 @@ SECTIONS .plt : ALIGN(16) { *(.plt) } .got.plt : ALIGN(16) { *(.got.plt) } + .data.rel : { *(.data.rel*) } + . = ALIGN(PECOFF_SEGMENT_ALIGN); __init_begin = .; __inittext_begin = .; @@ -93,8 +95,6 @@ SECTIONS PERCPU_SECTION(1 << CONFIG_L1_CACHE_SHIFT) #endif - .rela.dyn : ALIGN(8) { *(.rela.dyn) *(.rela*) } - .init.bss : { *(.init.bss) } @@ -107,6 +107,12 @@ SECTIONS RO_DATA(4096) RW_DATA(1 << CONFIG_L1_CACHE_SHIFT, PAGE_SIZE, THREAD_SIZE) + .rela.dyn : ALIGN(8) { + __rela_dyn_start = .; + *(.rela.dyn) *(.rela*) + __rela_dyn_end = .; + } + .sdata : { *(.sdata) } @@ -133,6 +139,7 @@ SECTIONS DISCARDS /DISCARD/ : { + *(.dynamic .dynsym .dynstr .hash .gnu.hash) *(.gnu.attributes) *(.options) *(.eh_frame) From patchwork Wed Feb 8 17:16:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xi Ruoyao X-Patchwork-Id: 54514 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3585763wrn; Wed, 8 Feb 2023 09:30:28 -0800 (PST) X-Google-Smtp-Source: AK7set9eQt2a4kFDx/RcXn/VlyBOb/nuDVZyD25HH2+RAQzYu476nL40GubGbcAPGJY1zh4uotNb X-Received: by 2002:a17:90a:ea83:b0:229:8e05:d59d with SMTP id h3-20020a17090aea8300b002298e05d59dmr7432425pjz.3.1675877428291; Wed, 08 Feb 2023 09:30:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675877428; cv=none; d=google.com; s=arc-20160816; b=b6rECaoO0GrourqcMNLMkKhjhZN853QXbQCSBs2FcAvTGRMe9+SmcZO+5RM4WghIOU esnCGPhvDMaMi16wNRG+Jtv3gH2ZRv65yFd0ExxSAi52oJvYpZjokUNjsXZsd1v7/ZmZ YLS9dcoF0oXmNoy1yXn+hdGoNGUmDUlvHDWIcBfkMBDc67y1nEhxgXHhyVsq4emRYVAc XT9cJzJi4C/lTGTl9QF5l7RXK17Y1pL1rjJQ1TMqXY5QWU7MdH4eL0Gb+dPfobn1ysnV 0Y2AAVVAdjyzik1z7zPi6ftnoFxM5GthA6uUeEGqmr2A2k4F1zYdXT2YHKYoMtfw6AxQ yFYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n2Y/CcM5/IUX6Gkiom5/CbpJn2tqOmafTIfiM5pWoK4=; b=M4WKagb4b89o9Rxa2jVkv8CHh9dC+WhBSwJlH49dWGfJf4RCtGIAxHzlJhumn5ACCy IWU1hEheu99lhzGa6ecA42qf12EsZ8kMWqOUDwo8TG6apn/uQQUVidBnVKPmkESDIenc lj165tZfZsTFl65AdGU7MHATB2tH2QJ2aVGxiSGNUJ/yTKISIynva4wPyRjETmIvNsYa JULLalj8LKChG69778wQ62YuHXu9lYNy2hQDaUKa/qYG7d7RLZ44+/Pe1jlkKYuu21u7 UjlSuHZcGZ7Wswowi807ZrvOXkuueNkX9xGJIeTfnqJiKZDDeXrb9FQbNvCGy8eLz6IL m9Og== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="Q/t03KGy"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ie3-20020a17090b400300b00230c8e3ed4bsi2698767pjb.24.2023.02.08.09.30.16; Wed, 08 Feb 2023 09:30:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@xry111.site header.s=default header.b="Q/t03KGy"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=xry111.site Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231671AbjBHRRM (ORCPT + 99 others); Wed, 8 Feb 2023 12:17:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230516AbjBHRRH (ORCPT ); Wed, 8 Feb 2023 12:17:07 -0500 Received: from xry111.site (xry111.site [89.208.246.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B63359EEC for ; Wed, 8 Feb 2023 09:16:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=xry111.site; s=default; t=1675876612; bh=uc+/2NQViYQRzOBEKp2Pmt1zBZRq+boXO32ftDdN+ZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q/t03KGyTJcfNXXG+B8HT3W1NVOajpdtNpfk/XppJkwDKDkFJtss6rO+nNQfIVGUa irHhfrgBj0HB63eIq8Y5jpsraNcFz2UQkGzhksaVOnzyMkvIj7vZuae77RrP/UU4va 5ICoxGZteD3q0O6yUjrFJMPLNZ5ljQ84Ffct+9nI= Received: from xry111-x57s1.. (unknown [IPv6:240e:358:1174:d100:dc73:854d:832e:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) (Authenticated sender: xry111@xry111.site) by xry111.site (Postfix) with ESMTPSA id D1F6665C4F; Wed, 8 Feb 2023 12:16:47 -0500 (EST) From: Xi Ruoyao To: Youling Tang , Huacai Chen , Jinyang He Cc: Xuerui Wang , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, Xi Ruoyao Subject: [PATCH v3 5/5] LoongArch: Add support for kernel address space layout randomization (KASLR) Date: Thu, 9 Feb 2023 01:16:07 +0800 Message-Id: <20230208171607.391107-6-xry111@xry111.site> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230208171607.391107-1-xry111@xry111.site> References: <20230208171607.391107-1-xry111@xry111.site> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757284850539287686?= X-GMAIL-MSGID: =?utf-8?q?1757284850539287686?= From: Youling Tang This patch adds support for relocating the kernel to a random address. Entropy is derived from the banner, which will change every build and random_get_entropy() which should provide additional runtime entropy. The kernel is relocated by up to RANDOMIZE_BASE_MAX_OFFSET bytes from its link address. Because relocation happens so early in the kernel boot, the amount of physical memory has not yet been determined. This means the only way to limit relocation within the available memory is via Kconfig. Limit the maximum value of RANDOMIZE_BASE_MAX_OFFSET to 256M(0x10000000) because our memory layout has many holes. KERNELOFFSET (kaslr_offset) is added to vmcoreinfo in the future, for crash --kaslr support. Signed-off-by: Youling Tang Signed-off-by: Xi Ruoyao # Use arch_initcall --- arch/loongarch/Kconfig | 22 ++++++ arch/loongarch/include/asm/page.h | 6 ++ arch/loongarch/kernel/relocate.c | 115 ++++++++++++++++++++++++++++++ arch/loongarch/kernel/setup.c | 3 + 4 files changed, 146 insertions(+) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 089a4695b1b3..cdf0535764ae 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -489,6 +489,28 @@ config RELOCATABLE kernel binary at runtime to a different virtual address than the address it was linked at. +config RANDOMIZE_BASE + bool "Randomize the address of the kernel image (KASLR)" + depends on RELOCATABLE + help + Randomizes the physical and virtual address at which the + kernel image is loaded, as a security feature that + deters exploit attempts relying on knowledge of the location + of kernel internals. + + The kernel will be offset by up to RANDOMIZE_BASE_MAX_OFFSET. + + If unsure, say N. + +config RANDOMIZE_BASE_MAX_OFFSET + hex "Maximum KASLR offset" if EXPERT + depends on RANDOMIZE_BASE + range 0x0 0x10000000 if 64BIT + default "0x01000000" + help + When KASLR is active, this provides the maximum offset that will + be applied to the kernel image. + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" depends on PROC_FS diff --git a/arch/loongarch/include/asm/page.h b/arch/loongarch/include/asm/page.h index 53f284a96182..6dda0d6271ca 100644 --- a/arch/loongarch/include/asm/page.h +++ b/arch/loongarch/include/asm/page.h @@ -106,6 +106,12 @@ extern int __virt_addr_valid(volatile void *kaddr); ((current->personality & READ_IMPLIES_EXEC) ? VM_EXEC : 0) | \ VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC) +extern unsigned long __kaslr_offset; +static inline unsigned long kaslr_offset(void) +{ + return __kaslr_offset; +} + #include #include diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c index 9c61e03d3e91..351168b70b7a 100644 --- a/arch/loongarch/kernel/relocate.c +++ b/arch/loongarch/kernel/relocate.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -19,6 +20,70 @@ extern long __rela_dyn_start; extern long __rela_dyn_end; +#ifdef CONFIG_RANDOMIZE_BASE + +static inline __init unsigned long rotate_xor(unsigned long hash, + const void *area, size_t size) +{ + size_t i; + unsigned long *ptr = (unsigned long *)area; + + for (i = 0; i < size / sizeof(hash); i++) { + /* Rotate by odd number of bits and XOR. */ + hash = (hash << ((sizeof(hash) * 8) - 7)) | (hash >> 7); + hash ^= ptr[i]; + } + + return hash; +} + +static inline __init unsigned long get_random_boot(void) +{ + unsigned long entropy = random_get_entropy(); + unsigned long hash = 0; + + /* Attempt to create a simple but unpredictable starting entropy. */ + hash = rotate_xor(hash, linux_banner, strlen(linux_banner)); + + /* Add in any runtime entropy we can get */ + hash = rotate_xor(hash, &entropy, sizeof(entropy)); + + return hash; +} + +static inline __init bool kaslr_disabled(void) +{ + char *str; + + str = strstr(boot_command_line, "nokaslr"); + if (str == boot_command_line || (str > boot_command_line && *(str - 1) == ' ')) + return true; + + return false; +} + +/* Choose a new address for the kernel */ +static inline void __init *determine_relocation_address(void) +{ + unsigned long kernel_length; + void *dest = _text; + unsigned long offset; + + if (kaslr_disabled()) + return dest; + + kernel_length = (long)_end - (long)_text; + + offset = get_random_boot() << 16; + offset &= (CONFIG_RANDOMIZE_BASE_MAX_OFFSET - 1); + if (offset < kernel_length) + offset += ALIGN(kernel_length, 0xffff); + + return RELOCATED(dest); +} + +#else + /* * Choose a new address for the kernel, for now we'll hard * code the destination. @@ -28,6 +93,8 @@ static inline void __init *determine_relocation_address(void) return (void *)(CACHE_BASE + 0x02000000); } +#endif + static inline int __init relocation_addr_valid(void *loc_new) { if ((unsigned long)loc_new & 0x0000ffff) { @@ -41,6 +108,13 @@ static inline int __init relocation_addr_valid(void *loc_new) return 1; } +static inline void __init update_kaslr_offset(unsigned long *addr, long offset) +{ + unsigned long *new_addr = (unsigned long *)RELOCATED(addr); + + *new_addr = (unsigned long)offset; +} + void *__init relocate_kernel(void) { Elf64_Rela *rela, *rela_end; @@ -49,6 +123,10 @@ void *__init relocate_kernel(void) long offset = 0; /* Default to original kernel entry point */ void *kernel_entry = start_kernel; + char *cmdline = early_ioremap(fw_arg1, COMMAND_LINE_SIZE); + + /* Boot command line was passed in fw_arg1 */ + strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE); kernel_length = (long)(_end) - (long)(_text); @@ -90,7 +168,44 @@ void *__init relocate_kernel(void) /* Return the new kernel's entry point */ kernel_entry = RELOCATED(start_kernel); + + /* Error may occur before, so keep it at last */ + update_kaslr_offset(&__kaslr_offset, offset); } return kernel_entry; } + +/* + * Show relocation information on panic. + */ +static void show_kernel_relocation(const char *level) +{ + if (__kaslr_offset > 0) { + printk(level); + pr_cont("Kernel relocated offset @ 0x%lx\n", __kaslr_offset); + pr_cont(" .text @ 0x%lx\n", (unsigned long)&_text); + pr_cont(" .data @ 0x%lx\n", (unsigned long)&_sdata); + pr_cont(" .bss @ 0x%lx\n", (unsigned long)&__bss_start); + } +} + +static int kernel_location_notifier_fn(struct notifier_block *self, + unsigned long v, void *p) +{ + show_kernel_relocation(KERN_EMERG); + return NOTIFY_DONE; +} + +static struct notifier_block kernel_location_notifier = { + .notifier_call = kernel_location_notifier_fn +}; + +static int __init register_kernel_offset_dumper(void) +{ + atomic_notifier_chain_register(&panic_notifier_list, + &kernel_location_notifier); + return 0; +} + +arch_initcall(register_kernel_offset_dumper); diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c index 4344502c0b31..b2ba2741945c 100644 --- a/arch/loongarch/kernel/setup.c +++ b/arch/loongarch/kernel/setup.c @@ -82,6 +82,9 @@ static struct resource code_resource = { .name = "Kernel code", }; static struct resource data_resource = { .name = "Kernel data", }; static struct resource bss_resource = { .name = "Kernel bss", }; +unsigned long __kaslr_offset __ro_after_init; +EXPORT_SYMBOL(__kaslr_offset); + const char *get_system_type(void) { return "generic-loongson-machine";