From patchwork Mon Dec 25 04:42:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 183120 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp233289dyb; Sun, 24 Dec 2023 20:55:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IGv7L43lsk7SVFIj2cKXjT5ckXGrQCXygNllTHu6nTnS+QXIvK/wAfHVZbggGg0COq5G4n+ X-Received: by 2002:a17:907:3d8e:b0:a23:6244:8370 with SMTP id he14-20020a1709073d8e00b00a2362448370mr2667937ejc.142.1703480118570; Sun, 24 Dec 2023 20:55:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703480118; cv=none; d=google.com; s=arc-20160816; b=LDh3AfJsnakykqCn+sbj3d7Jqs5PsjVN+Wf0nMtPIFCFdWcjEl6MPjl0qkN0HI5E3M QT3p4J+8+7JlVkgnEOx4RvjBbBW06a4dgd65r8MDwomcv2zmtspawUeB6SNF1eTr3rI2 +2TuURr4VIee7Y+yxldlz1kaM3F/DpY9I/3/MwkLT6kR0YY0dcxUHPGAMY+2my8Vog4t mxEtc4iySXDBFHW4Tf45a+SypFQejPqXufUBb7Cnv3pJXsfG0z0X6pUp6z+md2Lwq4xT 5w0oAVXhX+xJPJqa5LlyTHRU4tKebhlXHjMUvGeIsTJUOIasCKiS7qG1uCYJ+WVSBKjr FdVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=iT8PHEDIhr2+mfpmXGSy+ZxGtwsywYpmP3I4Y4PpOwE=; fh=UNuO+lQlip0GI5UaM6CATOxbnP1aMuk9nV/bmz8XnUw=; b=g0QjSMsuZLAil3HWc8mgQm+ntfBiqBiFYWsr69nTW2dYcgU2VRZSivmNudGIONFvvm i/xMAH3hACe6x77OpA0fwQZ0Qu8qGmqMGBdPp1eiwOjngJTllVletqGKVXFnG+KBui0k 4sK0TpomLE71pIrsH9wWeCBD0T/dubMc4msK9HIfVoqbH72frpz+wNI6HWRPwP31o4Eq YuofbhoZBXHrwMYYQUZTcTPZ1OczstXGD4kjv68krjAa94r+UAH8bTsHsSe21yq5DTSr yKBuwUYVR8jN0gI3Sot5nmcxzx2s9o/9ScTH3lAuyJD+FFhvxSFKwh9u/e05CCMjx/Cg 6DKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ApSX89aV; spf=pass (google.com: domain of linux-kernel+bounces-10892-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10892-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id h22-20020a17090791d600b00a26aa93ff28si3633027ejz.639.2023.12.24.20.55.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Dec 2023 20:55:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10892-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ApSX89aV; spf=pass (google.com: domain of linux-kernel+bounces-10892-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10892-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 34EE31F21359 for ; Mon, 25 Dec 2023 04:55:18 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F25C22908; Mon, 25 Dec 2023 04:54:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ApSX89aV" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5ECF017FE for ; Mon, 25 Dec 2023 04:54:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D88DC433C9; Mon, 25 Dec 2023 04:54:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703480094; bh=joYODQQgM8OQwgZYoqPwgCmKzSHGbsW2er2Qt7q5rp8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ApSX89aVjmgqbdkXyq2u5pLTi4jiZQPcbWieu08zEKDV+Dph9ImzxtwcsGt5gy8wr /TPfe/Cqs8k7g5QxPltEkToGICEHusdBwH+BW/P4QGyTWlikiQ+KV91UxOhnGwtY3c 62T/Ms8cCCjpthMdr1c7uRPwAwhn6ga8kJFq/mbyljdf+t9cOOAkO4wdcsp6ehWHg1 eLDPnILHE/41GJwESFN4SoIiwr26rVqsjG9I4d/50ZVfoarK04qEswNR7OvVSUPl3x ORk0R8hDtobKogJ4ISI9Ei8kZKGWfBcjKtMXq0H6NRCViLC1l6L9fSyImDeQ/TOicW AbSrAHslZGDZA== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Eric Biggers , Conor Dooley , Qingfang DENG , Charlie Jenkins Subject: [PATCH v4 1/2] riscv: introduce RISCV_EFFICIENT_UNALIGNED_ACCESS Date: Mon, 25 Dec 2023 12:42:06 +0800 Message-Id: <20231225044207.3821-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231225044207.3821-1-jszhang@kernel.org> References: <20231225044207.3821-1-jszhang@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786228368507814400 X-GMAIL-MSGID: 1786228368507814400 Some riscv implementations such as T-HEAD's C906, C908, C910 and C920 support efficient unaligned access, for performance reason we want to enable HAVE_EFFICIENT_UNALIGNED_ACCESS on these platforms. To avoid performance regressions on other non efficient unaligned access platforms, HAVE_EFFICIENT_UNALIGNED_ACCESS can't be globally selected. To solve this problem, runtime code patching based on the detected speed is a good solution. But that's not easy, it involves lots of work to modify vairous subsystems such as net, mm, lib and so on. This can be done step by step. So let's take an easier solution: add support to efficient unaligned access and hide the support under NONPORTABLE. Now let's introduce RISCV_EFFICIENT_UNALIGNED_ACCESS which depends on NONPORTABLE, if users know during config time that the kernel will be only run on those efficient unaligned access hw platforms, they can enable it. Obviously, generic unified kernel Image shouldn't enable it. Signed-off-by: Jisheng Zhang Reviewed-by: Charlie Jenkins Reviewed-by: Eric Biggers --- arch/riscv/Kconfig | 13 +++++++++++++ arch/riscv/Makefile | 2 ++ 2 files changed, 15 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 24c1799e2ec4..afcc5fdc16f7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -651,6 +651,19 @@ config RISCV_MISALIGNED load/store for both kernel and userspace. When disable, misaligned accesses will generate SIGBUS in userspace and panic in kernel. +config RISCV_EFFICIENT_UNALIGNED_ACCESS + bool "Assume the CPU supports fast unaligned memory accesses" + depends on NONPORTABLE + select HAVE_EFFICIENT_UNALIGNED_ACCESS + help + Say Y here if you want the kernel to assume that the CPU supports + efficient unaligned memory accesses. When enabled, this option + improves the performance of the kernel on such CPUs. However, the + kernel will run much more slowly, or will not be able to run at all, + on CPUs that do not support efficient unaligned memory accesses. + + If unsure what to do here, say N. + endmenu # "Platform type" menu "Kernel features" diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index a74be78678eb..ebbe02628a27 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -108,7 +108,9 @@ KBUILD_AFLAGS_MODULE += $(call as-option,-Wa$(comma)-mno-relax) # unaligned accesses. While unaligned accesses are explicitly allowed in the # RISC-V ISA, they're emulated by machine mode traps on all extant # architectures. It's faster to have GCC emit only aligned accesses. +ifneq ($(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS),y) KBUILD_CFLAGS += $(call cc-option,-mstrict-align) +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare From patchwork Mon Dec 25 04:42:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 183121 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp233344dyb; Sun, 24 Dec 2023 20:55:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IGApZ30EYthHJeZFMGDY5A+hPgpCLJegrq171GUfERaAakrgGN6usk+oWUiilzrtsxXYRQj X-Received: by 2002:a05:6808:ec2:b0:3b8:4ced:40c with SMTP id q2-20020a0568080ec200b003b84ced040cmr5741041oiv.59.1703480132610; Sun, 24 Dec 2023 20:55:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703480132; cv=none; d=google.com; s=arc-20160816; b=XpB/fJOoMun0gRK7G0XomKTXLrFI8RzTJLSa3tUW7F1arhKDMzt8V8MRbFcZBTGYX6 NsRdQ74zciU46gwV5CrR+FICj900zCz3DlM+A2IQJ0WXrjYwbz8q53ElarB1A92t2nJ4 cwP8R4f+yDdl0Sjx/ZGRVIzBm+3OKSSK3Ves2kNvex8wFYt6dYmc3WeN59WIIsS75lo0 O6hPxuTJIqrI9BtvrWTxCBjarbRm/9jw4beIdWr8nnWBcJZoOHVdiCMjQ312WbYvDIyU xoHv19pD5K2uGCRYb+BXuE6NOrpe0OUxv++QI2vERVCZOinyeHlDnR4BVNVNiUupdU6m YOcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=ZUIX+kM4M42dQ2aqBRnvaD09zhlvU97u7PRQtXGt3p4=; fh=ZTT3Py7m/X4mSvw6SK6+TRZhmqxsZi0TneX42+FLUx4=; b=HG1+5tMr8BbtqC/oJHlS6AJyoiPNeUFB5kqxfoUu5a7qivVH0qcPgDpNDcN0w16sRh V71IdABwS0VkEJ7VERnPg8UbrLT9PChZHkhAlCimRKTc6ciWInto59mhC3Xd98iivvnb SNyVSYVNGLTnxpMABKUX5kC5yZt7OV03tOeXyLZ468r2PJ4EASK02GUozpK0cUywWb0E 4BMOFi0kDj6dmM9MlL6DwyhaZqCUlX2qX4c0bSqgx/RqPY+efnZL0pzWXP3VG22ETCwD IdKgz6yzsFg77Wb1BYJtSXE/mIiulMb91WpMs1YxBkQEEZe3KFGWVpkqMZuARLz20zsU j7HA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Jozbum0Q; spf=pass (google.com: domain of linux-kernel+bounces-10893-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10893-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id ei31-20020a056a0080df00b006d9aef87840si2416193pfb.300.2023.12.24.20.55.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Dec 2023 20:55:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10893-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Jozbum0Q; spf=pass (google.com: domain of linux-kernel+bounces-10893-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10893-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 750D428194F for ; Mon, 25 Dec 2023 04:55:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C4AC353A2; Mon, 25 Dec 2023 04:55:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Jozbum0Q" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 327DB29B4 for ; Mon, 25 Dec 2023 04:54:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77FAFC433CC; Mon, 25 Dec 2023 04:54:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703480097; bh=6XpfAVFOAhuOEVj65LCeIoXPY0Gh1KiNb/Z3XmVwtL0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jozbum0Q/fdcDqQiB55TcCxgyo/ld5j3NiVIyavjUoZCjmDITUYvfSoktcEfqZDYm mkC0SQ4pQdcN/ov8EfaafmTTO/k/2HGyVDc/5pTkpdtHezaUNRwez/H9DX5kgQYTmM F+p4cBLepUCvQK+Xr95SD6HfEaB0S5n6KhAj3EkQoaE12Y2MkTv/Zsx91T/kccZdAb xQat64yxzEgNYd16DF2dUQPW2UHhBuUPyVhlysL49m6z0FPVZ8TcNRwBR9RRMP9d9k wuezVnDoaRokWyF/r27e2kVZy6KIOgqu5H+HF83G0BjnBA8QoLH3a94LfTcetypgy2 ehU3rhFvFjLbw== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Eric Biggers , Conor Dooley , Qingfang DENG Subject: [PATCH v4 2/2] riscv: select DCACHE_WORD_ACCESS for efficient unaligned access HW Date: Mon, 25 Dec 2023 12:42:07 +0800 Message-Id: <20231225044207.3821-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231225044207.3821-1-jszhang@kernel.org> References: <20231225044207.3821-1-jszhang@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786228383315028311 X-GMAIL-MSGID: 1786228383315028311 DCACHE_WORD_ACCESS uses the word-at-a-time API for optimised string comparisons in the vfs layer. This patch implements support for load_unaligned_zeropad in much the same way as has been done for arm64. Here is the test program and step: $ cat tt.c #include #include #include #define ITERATIONS 1000000 #define PATH "123456781234567812345678123456781" int main(void) { unsigned long i; struct stat buf; for (i = 0; i < ITERATIONS; i++) stat(PATH, &buf); return 0; } $ gcc -O2 tt.c $ touch 123456781234567812345678123456781 $ time ./a.out Per my test on T-HEAD C910 platforms, the above test performance is improved by about 7.5%. Signed-off-by: Jisheng Zhang --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/asm-extable.h | 15 ++++++++++++ arch/riscv/include/asm/word-at-a-time.h | 27 +++++++++++++++++++++ arch/riscv/mm/extable.c | 31 +++++++++++++++++++++++++ 4 files changed, 74 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index afcc5fdc16f7..e34863c5a8ed 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -654,6 +654,7 @@ config RISCV_MISALIGNED config RISCV_EFFICIENT_UNALIGNED_ACCESS bool "Assume the CPU supports fast unaligned memory accesses" depends on NONPORTABLE + select DCACHE_WORD_ACCESS if MMU select HAVE_EFFICIENT_UNALIGNED_ACCESS help Say Y here if you want the kernel to assume that the CPU supports diff --git a/arch/riscv/include/asm/asm-extable.h b/arch/riscv/include/asm/asm-extable.h index 00a96e7a9664..0c8bfd54fc4e 100644 --- a/arch/riscv/include/asm/asm-extable.h +++ b/arch/riscv/include/asm/asm-extable.h @@ -6,6 +6,7 @@ #define EX_TYPE_FIXUP 1 #define EX_TYPE_BPF 2 #define EX_TYPE_UACCESS_ERR_ZERO 3 +#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4 #ifdef CONFIG_MMU @@ -47,6 +48,11 @@ #define EX_DATA_REG_ZERO_SHIFT 5 #define EX_DATA_REG_ZERO GENMASK(9, 5) +#define EX_DATA_REG_DATA_SHIFT 0 +#define EX_DATA_REG_DATA GENMASK(4, 0) +#define EX_DATA_REG_ADDR_SHIFT 5 +#define EX_DATA_REG_ADDR GENMASK(9, 5) + #define EX_DATA_REG(reg, gpr) \ "((.L__gpr_num_" #gpr ") << " __stringify(EX_DATA_REG_##reg##_SHIFT) ")" @@ -62,6 +68,15 @@ #define _ASM_EXTABLE_UACCESS_ERR(insn, fixup, err) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) +#define _ASM_EXTABLE_LOAD_UNALIGNED_ZEROPAD(insn, fixup, data, addr) \ + __DEFINE_ASM_GPR_NUMS \ + __ASM_EXTABLE_RAW(#insn, #fixup, \ + __stringify(EX_TYPE_LOAD_UNALIGNED_ZEROPAD), \ + "(" \ + EX_DATA_REG(DATA, data) " | " \ + EX_DATA_REG(ADDR, addr) \ + ")") + #endif /* __ASSEMBLY__ */ #else /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/word-at-a-time.h b/arch/riscv/include/asm/word-at-a-time.h index 7c086ac6ecd4..f3f031e34191 100644 --- a/arch/riscv/include/asm/word-at-a-time.h +++ b/arch/riscv/include/asm/word-at-a-time.h @@ -9,6 +9,7 @@ #define _ASM_RISCV_WORD_AT_A_TIME_H +#include #include struct word_at_a_time { @@ -45,4 +46,30 @@ static inline unsigned long find_zero(unsigned long mask) /* The mask we created is directly usable as a bytemask */ #define zero_bytemask(mask) (mask) +#ifdef CONFIG_DCACHE_WORD_ACCESS + +/* + * Load an unaligned word from kernel space. + * + * In the (very unlikely) case of the word being a page-crosser + * and the next page not being mapped, take the exception and + * return zeroes in the non-existing part. + */ +static inline unsigned long load_unaligned_zeropad(const void *addr) +{ + unsigned long ret; + + /* Load word from unaligned pointer addr */ + asm( + "1: " REG_L " %0, %2\n" + "2:\n" + _ASM_EXTABLE_LOAD_UNALIGNED_ZEROPAD(1b, 2b, %0, %1) + : "=&r" (ret) + : "r" (addr), "m" (*(unsigned long *)addr)); + + return ret; +} + +#endif /* CONFIG_DCACHE_WORD_ACCESS */ + #endif /* _ASM_RISCV_WORD_AT_A_TIME_H */ diff --git a/arch/riscv/mm/extable.c b/arch/riscv/mm/extable.c index 35484d830fd6..dd1530af3ef1 100644 --- a/arch/riscv/mm/extable.c +++ b/arch/riscv/mm/extable.c @@ -27,6 +27,14 @@ static bool ex_handler_fixup(const struct exception_table_entry *ex, return true; } +static inline unsigned long regs_get_gpr(struct pt_regs *regs, unsigned int offset) +{ + if (unlikely(!offset || offset > MAX_REG_OFFSET)) + return 0; + + return *(unsigned long *)((unsigned long)regs + offset); +} + static inline void regs_set_gpr(struct pt_regs *regs, unsigned int offset, unsigned long val) { @@ -50,6 +58,27 @@ static bool ex_handler_uaccess_err_zero(const struct exception_table_entry *ex, return true; } +static bool +ex_handler_load_unaligned_zeropad(const struct exception_table_entry *ex, + struct pt_regs *regs) +{ + int reg_data = FIELD_GET(EX_DATA_REG_DATA, ex->data); + int reg_addr = FIELD_GET(EX_DATA_REG_ADDR, ex->data); + unsigned long data, addr, offset; + + addr = regs_get_gpr(regs, reg_addr * sizeof(unsigned long)); + + offset = addr & 0x7UL; + addr &= ~0x7UL; + + data = *(unsigned long *)addr >> (offset * 8); + + regs_set_gpr(regs, reg_data * sizeof(unsigned long), data); + + regs->epc = get_ex_fixup(ex); + return true; +} + bool fixup_exception(struct pt_regs *regs) { const struct exception_table_entry *ex; @@ -65,6 +94,8 @@ bool fixup_exception(struct pt_regs *regs) return ex_handler_bpf(ex, regs); case EX_TYPE_UACCESS_ERR_ZERO: return ex_handler_uaccess_err_zero(ex, regs); + case EX_TYPE_LOAD_UNALIGNED_ZEROPAD: + return ex_handler_load_unaligned_zeropad(ex, regs); } BUG();