From patchwork Sat Dec 23 15:52:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 182974 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp1748584dyi; Sat, 23 Dec 2023 08:05:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IHOfjJVt6RzrJmuNlAmVd/Du3YcSrR4JNlo6cmznzL8VKTcvcbFh1nYNr8Qxg2zrjoOrjXf X-Received: by 2002:ac8:5f0c:0:b0:427:7ee1:b2cf with SMTP id x12-20020ac85f0c000000b004277ee1b2cfmr3458692qta.71.1703347535626; Sat, 23 Dec 2023 08:05:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703347535; cv=none; d=google.com; s=arc-20160816; b=bW8kixy6iOE34PKdsNGWglbPsFdLcMiFn/b5aWHJzX2h5wHujp6nR/iogmDeWWpNVZ vaE9wXsZ6skRzVMsGFGQX0EnggvFIt4VWrmdIJZNUY+4gBEo9uD1cLK8p6wxSWhGabQo 1IXuNAalkUvdjd4Ayg3AV/yEeDdiomf4i5ddW1zBv7Oi9KxujeV1V2MGuUHZblu0JD4r uIOgnqZtmpa2XKJjHysuRrxY7PbhGVai6srfnh6mcaqzylNabZXFyIVPvNxLmpQ++Qam pq6lTXHhFypqfe4pv8tzYdIYcLrgavMfqnjGHAQIqbFJWwhSuk6GnU3bQcorkptcxkn1 Z8Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=iPgvgf80kziCLDMRSFlc9mzwq2EN646UxWAf2fL+s5U=; fh=hWCudT2NEUPWqYS2EES6uXIvcbr87NeXzPCLDcfDTro=; b=wtP0HRx+I4owYpA/+/GuZP8oEA0yXFcamRuIkMBb2qF+ab+SmRiMqD3FPyUs+jdTKM svdJT/eLgMfNuOTZDSwfAID3zAbQxi6R72JUBkeyTfkNJ28BqkcKCe7wn9BCDmTng1FY W1KJH7kfeWBwgS5EjnwppmeYcyUJNHjMn3R6Ff1LrojVf1DS8iS5PgX2Ka1qrxaVAQQA qryC6CPUmg7PajALyIKRFb96GkGae00IXZMNpQ+5DC9wsRkiPq+JJWH6vf5yK0Mq9LOd eF+s3prFrF5jucULRND1QffhSFudroJtyqYAQduOAL/MRGchykZZNt1PN9ptes2GrbKi xtGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QTsd1xdh; spf=pass (google.com: domain of linux-kernel+bounces-10484-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10484-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id d13-20020a05622a15cd00b0042784efd303si6949347qty.162.2023.12.23.08.05.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Dec 2023 08:05:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10484-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QTsd1xdh; spf=pass (google.com: domain of linux-kernel+bounces-10484-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10484-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 6AF5D1C211E1 for ; Sat, 23 Dec 2023 16:05:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 861CB12E6D; Sat, 23 Dec 2023 16:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QTsd1xdh" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F371510979 for ; Sat, 23 Dec 2023 16:05:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BAA3C433C8; Sat, 23 Dec 2023 16:05:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703347510; bh=9PwrcihFF5i7bY070ZxIROnCJMVZyvc9x7nf1yDV/Tc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QTsd1xdhj0LqRZ2MgmYoGjWEEqLJbmKVgowayVJ83RFEE38PjtAEf+44jZP8EzkQG Jjb9v+dvlONiF5uYgot39N5ERitQlH/cXWEdABAPR2sz8pudoLyzuxJtoOm6HnIPQk a5W7QqkC+8oX/fjaJESoJeEHnv9X6osE2Mu9gIDMjFw7GUX0q9xVaqkuabOgH+Vd8z lx1DXV3tTIDLOLBi7/jVN1TyRhOLgO6AOqBPqbJZgiXn6FIkKVAn7DnsgfsD4vwEOa j0U/SW8P9Vuy+EvLOfUnfFFN0+6sOVyRBbStdXZc9Vffw7cdbbJln8FEd6vPeJE76b OqctVbxiijVKA== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: Conor Dooley , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Qingfang DENG , Eric Biggers , Charlie Jenkins Subject: [PATCH v3 1/2] riscv: introduce RISCV_EFFICIENT_UNALIGNED_ACCESS Date: Sat, 23 Dec 2023 23:52:25 +0800 Message-Id: <20231223155226.4050-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231223155226.4050-1-jszhang@kernel.org> References: <20231223155226.4050-1-jszhang@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786089345545051172 X-GMAIL-MSGID: 1786089345545051172 Some riscv implementations such as T-HEAD's C906, C908, C910 and C920 support efficient unaligned access, for performance reason we want to enable HAVE_EFFICIENT_UNALIGNED_ACCESS on these platforms. To avoid performance regressions on other non efficient unaligned access platforms, HAVE_EFFICIENT_UNALIGNED_ACCESS can't be globally selected. To solve this problem, runtime code patching based on the detected speed is a good solution. But that's not easy, it involves lots of work to modify vairous subsystems such as net, mm, lib and so on. This can be done step by step. So let's take an easier solution: add support to efficient unaligned access and hide the support under NONPORTABLE. Now let's introduce RISCV_EFFICIENT_UNALIGNED_ACCESS which depends on NONPORTABLE, if users know during config time that the kernel will be only run on those efficient unaligned access hw platforms, they can enable it. Obviously, generic unified kernel Image shouldn't enable it. Signed-off-by: Jisheng Zhang Reviewed-by: Charlie Jenkins --- arch/riscv/Kconfig | 12 ++++++++++++ arch/riscv/Makefile | 2 ++ 2 files changed, 14 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 24c1799e2ec4..b91094ea53b7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -651,6 +651,18 @@ config RISCV_MISALIGNED load/store for both kernel and userspace. When disable, misaligned accesses will generate SIGBUS in userspace and panic in kernel. +config RISCV_EFFICIENT_UNALIGNED_ACCESS + bool "Use unaligned access for some functions" + depends on NONPORTABLE + select HAVE_EFFICIENT_UNALIGNED_ACCESS + default n + help + Say Y here if you want the kernel only run on hardware platforms which + support efficient unaligned access, then unaligned access will be used + in some functions for optimized performance. + + If unsure what to do here, say N. + endmenu # "Platform type" menu "Kernel features" diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index a74be78678eb..ebbe02628a27 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -108,7 +108,9 @@ KBUILD_AFLAGS_MODULE += $(call as-option,-Wa$(comma)-mno-relax) # unaligned accesses. While unaligned accesses are explicitly allowed in the # RISC-V ISA, they're emulated by machine mode traps on all extant # architectures. It's faster to have GCC emit only aligned accesses. +ifneq ($(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS),y) KBUILD_CFLAGS += $(call cc-option,-mstrict-align) +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare From patchwork Sat Dec 23 15:52:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 182975 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp1748707dyi; Sat, 23 Dec 2023 08:05:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IE525Doht2HoFBWHJl26FUWjrwWXKMmoboNMfTeuoNRJjprPHE+fcoaTA6SfMM0Uvupn/Dn X-Received: by 2002:a17:902:784c:b0:1d0:4cb8:4510 with SMTP id e12-20020a170902784c00b001d04cb84510mr1329709pln.50.1703347550420; Sat, 23 Dec 2023 08:05:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703347550; cv=none; d=google.com; s=arc-20160816; b=zRFekrpMQx3pohe5OP33MnbkYNlsskJBzVlwLWoLH12llz0/+QLAoJTSIzujMBwL2a DcQSXHaJjv2dxvch7kwFjm8eYuZqx4cFBM/nAwwdNzM4+TRo7pJVl+CDNaNHWbzFb+Sx Ktr3TbIrEmCKE9XmZAEshRDeFH/fb1JMX1ujaHlv5q4dE1mVOxFxzbLPewjFOWnCghHO L6xopAYuG8Z7lN9HscDAz6Yhirwov0kjGvdyfB9MsRyOUxinhrHfZSTfmgTe0r802+FK hVuLCoHWxTKcmAzSRs6Ab8//XHOkfRL9MeCbaQjg1F/5MNGWAdlPny/wiZhxSn+XF2jK R2qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=071BX8kdPIsao85/uJnCWSDyL4Vk8WztgP4W5kmYUcI=; fh=D9r4Sizq2FdKSK6sy9TLYUFhmy1mZUM4RB+eGjJiKEU=; b=xx5ButHi0U8/SC9ZwrEMqSL0X9krihIIhIBAuu6D/PiEQ6dDHE6xzXYSvLCR4KAWYf oYp0tGL9dMlDA9iS80FMtbG6DOpueI+RXzH9EQJ2uVBSTzLy9y9i6FacrcgT2Uk8GDTR cNWphzYbAIyKiNRX7hlJ7OG486fYwGC4jgwHKxDvBVGKnH3dEraH8LRNT8BxmZM+LWql dIhUYhR26kux6P9/mrQGqgMAYp5zeSd1M+clWjRkEd454eILbruJxCs7TPSyudjFS9mR qXC9jpDVzfC0a8xqdpdnuaLeZd5Dt6yRTh2KCmg3QtpJfIAvsQ6CYlDQ83wO9jdm9c7A bJug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UO+6uySz; spf=pass (google.com: domain of linux-kernel+bounces-10485-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10485-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id e10-20020a170902b78a00b001d002eeb4a1si2526489pls.15.2023.12.23.08.05.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Dec 2023 08:05:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10485-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UO+6uySz; spf=pass (google.com: domain of linux-kernel+bounces-10485-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10485-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 22FC4B22408 for ; Sat, 23 Dec 2023 16:05:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AAABA14F62; Sat, 23 Dec 2023 16:05:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UO+6uySz" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1EBD412E5E for ; Sat, 23 Dec 2023 16:05:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F00A5C433CD; Sat, 23 Dec 2023 16:05:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703347512; bh=v6WGCTMiMOCTlpt4SkvLg03xjdq5zQMjhktOr8ueBgk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UO+6uySzueLe3eDdS9Quk61zCkY7sorf45mCzxqWKL9Eh2d49xqiE6ftJ11ZKKPWO lblt+nBlAQ6rmJ/gbTba+Un/KKX5iJ+lpvxRjpupaiFI3VsG8pB0Ig/gCqVdPKA4zJ BAPGT+5Xwlxy0nXtW882nZ6p3P+aJ4qpA69T1gyPXInakkVgrXhy+lpxi8ErdfIJPZ 6y7GGgQqH8FOKqZ6ZNi3D+CiULUULMdOxFIRChDD+IeLfLd0X8ldKHXpOF7mQI02EL 5JMO6SElA7uGNbfiDY4MbOICSI4VSs1Vhc2lnXoHw/Vb1+aRDXRKNBj+i+mATFHpq/ xNcDB3PzXVJmw== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: Conor Dooley , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Qingfang DENG , Eric Biggers Subject: [PATCH v3 2/2] riscv: select DCACHE_WORD_ACCESS for efficient unaligned access HW Date: Sat, 23 Dec 2023 23:52:26 +0800 Message-Id: <20231223155226.4050-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231223155226.4050-1-jszhang@kernel.org> References: <20231223155226.4050-1-jszhang@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786089360811700153 X-GMAIL-MSGID: 1786089360811700153 DCACHE_WORD_ACCESS uses the word-at-a-time API for optimised string comparisons in the vfs layer. This patch implements support for load_unaligned_zeropad in much the same way as has been done for arm64. Here is the test program and step: $ cat tt.c #include #include #include #define ITERATIONS 1000000 #define PATH "123456781234567812345678123456781" int main(void) { unsigned long i; struct stat buf; for (i = 0; i < ITERATIONS; i++) stat(PATH, &buf); return 0; } $ gcc -O2 tt.c $ touch 123456781234567812345678123456781 $ time ./a.out Per my test on T-HEAD C910 platforms, the above test performance is improved by about 7.5%. Signed-off-by: Jisheng Zhang --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/asm-extable.h | 15 ++++++++++++ arch/riscv/include/asm/word-at-a-time.h | 27 +++++++++++++++++++++ arch/riscv/mm/extable.c | 31 +++++++++++++++++++++++++ 4 files changed, 74 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index b91094ea53b7..52e45760863c 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -654,6 +654,7 @@ config RISCV_MISALIGNED config RISCV_EFFICIENT_UNALIGNED_ACCESS bool "Use unaligned access for some functions" depends on NONPORTABLE + select DCACHE_WORD_ACCESS if MMU select HAVE_EFFICIENT_UNALIGNED_ACCESS default n help diff --git a/arch/riscv/include/asm/asm-extable.h b/arch/riscv/include/asm/asm-extable.h index 00a96e7a9664..0c8bfd54fc4e 100644 --- a/arch/riscv/include/asm/asm-extable.h +++ b/arch/riscv/include/asm/asm-extable.h @@ -6,6 +6,7 @@ #define EX_TYPE_FIXUP 1 #define EX_TYPE_BPF 2 #define EX_TYPE_UACCESS_ERR_ZERO 3 +#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4 #ifdef CONFIG_MMU @@ -47,6 +48,11 @@ #define EX_DATA_REG_ZERO_SHIFT 5 #define EX_DATA_REG_ZERO GENMASK(9, 5) +#define EX_DATA_REG_DATA_SHIFT 0 +#define EX_DATA_REG_DATA GENMASK(4, 0) +#define EX_DATA_REG_ADDR_SHIFT 5 +#define EX_DATA_REG_ADDR GENMASK(9, 5) + #define EX_DATA_REG(reg, gpr) \ "((.L__gpr_num_" #gpr ") << " __stringify(EX_DATA_REG_##reg##_SHIFT) ")" @@ -62,6 +68,15 @@ #define _ASM_EXTABLE_UACCESS_ERR(insn, fixup, err) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) +#define _ASM_EXTABLE_LOAD_UNALIGNED_ZEROPAD(insn, fixup, data, addr) \ + __DEFINE_ASM_GPR_NUMS \ + __ASM_EXTABLE_RAW(#insn, #fixup, \ + __stringify(EX_TYPE_LOAD_UNALIGNED_ZEROPAD), \ + "(" \ + EX_DATA_REG(DATA, data) " | " \ + EX_DATA_REG(ADDR, addr) \ + ")") + #endif /* __ASSEMBLY__ */ #else /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/word-at-a-time.h b/arch/riscv/include/asm/word-at-a-time.h index 7c086ac6ecd4..f3f031e34191 100644 --- a/arch/riscv/include/asm/word-at-a-time.h +++ b/arch/riscv/include/asm/word-at-a-time.h @@ -9,6 +9,7 @@ #define _ASM_RISCV_WORD_AT_A_TIME_H +#include #include struct word_at_a_time { @@ -45,4 +46,30 @@ static inline unsigned long find_zero(unsigned long mask) /* The mask we created is directly usable as a bytemask */ #define zero_bytemask(mask) (mask) +#ifdef CONFIG_DCACHE_WORD_ACCESS + +/* + * Load an unaligned word from kernel space. + * + * In the (very unlikely) case of the word being a page-crosser + * and the next page not being mapped, take the exception and + * return zeroes in the non-existing part. + */ +static inline unsigned long load_unaligned_zeropad(const void *addr) +{ + unsigned long ret; + + /* Load word from unaligned pointer addr */ + asm( + "1: " REG_L " %0, %2\n" + "2:\n" + _ASM_EXTABLE_LOAD_UNALIGNED_ZEROPAD(1b, 2b, %0, %1) + : "=&r" (ret) + : "r" (addr), "m" (*(unsigned long *)addr)); + + return ret; +} + +#endif /* CONFIG_DCACHE_WORD_ACCESS */ + #endif /* _ASM_RISCV_WORD_AT_A_TIME_H */ diff --git a/arch/riscv/mm/extable.c b/arch/riscv/mm/extable.c index 35484d830fd6..dd1530af3ef1 100644 --- a/arch/riscv/mm/extable.c +++ b/arch/riscv/mm/extable.c @@ -27,6 +27,14 @@ static bool ex_handler_fixup(const struct exception_table_entry *ex, return true; } +static inline unsigned long regs_get_gpr(struct pt_regs *regs, unsigned int offset) +{ + if (unlikely(!offset || offset > MAX_REG_OFFSET)) + return 0; + + return *(unsigned long *)((unsigned long)regs + offset); +} + static inline void regs_set_gpr(struct pt_regs *regs, unsigned int offset, unsigned long val) { @@ -50,6 +58,27 @@ static bool ex_handler_uaccess_err_zero(const struct exception_table_entry *ex, return true; } +static bool +ex_handler_load_unaligned_zeropad(const struct exception_table_entry *ex, + struct pt_regs *regs) +{ + int reg_data = FIELD_GET(EX_DATA_REG_DATA, ex->data); + int reg_addr = FIELD_GET(EX_DATA_REG_ADDR, ex->data); + unsigned long data, addr, offset; + + addr = regs_get_gpr(regs, reg_addr * sizeof(unsigned long)); + + offset = addr & 0x7UL; + addr &= ~0x7UL; + + data = *(unsigned long *)addr >> (offset * 8); + + regs_set_gpr(regs, reg_data * sizeof(unsigned long), data); + + regs->epc = get_ex_fixup(ex); + return true; +} + bool fixup_exception(struct pt_regs *regs) { const struct exception_table_entry *ex; @@ -65,6 +94,8 @@ bool fixup_exception(struct pt_regs *regs) return ex_handler_bpf(ex, regs); case EX_TYPE_UACCESS_ERR_ZERO: return ex_handler_uaccess_err_zero(ex, regs); + case EX_TYPE_LOAD_UNALIGNED_ZEROPAD: + return ex_handler_load_unaligned_zeropad(ex, regs); } BUG();