From patchwork Thu Feb 16 16:00:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kristina Martsenko X-Patchwork-Id: 58157 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp383599wrn; Thu, 16 Feb 2023 08:06:07 -0800 (PST) X-Google-Smtp-Source: AK7set9ewhBnz88FNudLqmYPDiGS2UtgFBK6dRwms5CeT9GKu4zuSe6LdbQ1XqfJe+MVqlQPaLpU X-Received: by 2002:a17:906:a84a:b0:8af:370b:da59 with SMTP id dx10-20020a170906a84a00b008af370bda59mr7183962ejb.17.1676563567848; Thu, 16 Feb 2023 08:06:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676563567; cv=none; d=google.com; s=arc-20160816; b=LjXif0srijQY7K86iI1eq+IovTY3JLK8vxq7BO0nYIq7JDJw8tsXNvslQHdNAgHRJ2 AsuNS1cSNePElBpYox0qim38WUbiMpzjhHJgGP3tbQgMs08M+GGbF4ZFPY/924qxwQrh KukSg8Mut7WcaIWNkDXm7xGGRHFun8buD9wP9XuwBv2HlRt7wOClro1o4a8/5LxAOOas ZYCCq2o6E5vSsoBXNeUHfInFo4oLVhTVdc+8LZFryTcYWexRl1BlhWZIcUo4r6LoVVPO FuWVR7tEwXsQ8NHdLdE51eGUBbNLrk++n1Zzurb0jg9IXlAzi9+hg47Q0BoQF18f1qOw VaEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=8tCVCZmK2IYEpD2Zvc6s5ajUqZ5YZqtqCzMIhDVvdds=; b=aXPT1PW3yPTULkV+ZoFtQYBjzPnP+961pUYIh47Wl26d48UwITSUiXy+T81RyJEmf2 8amTm+tYYEe72TMX4+0K1KJQ0R5b5ExBnh3HpduSagbYW8Sz698uJgJBhGIX4a8yhwZs EybhqmPm5Mi1eu50V/va5wP98Z3JDjakJ/0VRW3Fgo7WKVrMKBhmsRR2SJ2WYA7LQLK6 YumbJwD1IjC8Z3sMO4r0p0Nrj01d1uM7E+tgCXsS9HwA7DdgcFHSfMPMgSJJe0i7nSoS +nqmlC46LNCDnxBQykfO32S/qSlieGepqqDoaohcGhVPg++VrU7SlTZuYXJ6LggnhU9B CXUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mt23-20020a170907619700b008b178b41ec1si635875ejc.284.2023.02.16.08.05.13; Thu, 16 Feb 2023 08:06:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229723AbjBPQCR (ORCPT + 99 others); Thu, 16 Feb 2023 11:02:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjBPQCD (ORCPT ); Thu, 16 Feb 2023 11:02:03 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 700BF56EFE for ; Thu, 16 Feb 2023 08:01:54 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E7996113E; Thu, 16 Feb 2023 08:02:36 -0800 (PST) Received: from e126864.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 25CE73F881; Thu, 16 Feb 2023 08:01:51 -0800 (PST) From: Kristina Martsenko To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Mark Rutland , Mark Brown , Luis Machado , Vladimir Murzin , linux-kernel@vger.kernel.org Subject: [PATCH 07/10] arm64: mops: handle MOPS exceptions Date: Thu, 16 Feb 2023 16:00:09 +0000 Message-Id: <20230216160012.272345-8-kristina.martsenko@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230216160012.272345-1-kristina.martsenko@arm.com> References: <20230216160012.272345-1-kristina.martsenko@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758004319741800571?= X-GMAIL-MSGID: =?utf-8?q?1758004319741800571?= The memory copy/set instructions added as part of FEAT_MOPS can take an exception part-way through their execution and resume execution afterwards. If however the task is re-scheduled and execution resumes on a different CPU, then the CPU may take a new type of exception to indicate this. In this case the OS has to reset the registers and restart execution from the prologue instruction. The algorithm for doing this is provided as part of the Arm ARM. Add an exception handler for the new exception and wire it up for userspace tasks. Signed-off-by: Kristina Martsenko Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/esr.h | 11 +++++++- arch/arm64/include/asm/exception.h | 1 + arch/arm64/kernel/entry-common.c | 11 ++++++++ arch/arm64/kernel/traps.c | 45 ++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index c9f15b9e3c71..96caaaee97a3 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -47,7 +47,7 @@ #define ESR_ELx_EC_DABT_LOW (0x24) #define ESR_ELx_EC_DABT_CUR (0x25) #define ESR_ELx_EC_SP_ALIGN (0x26) -/* Unallocated EC: 0x27 */ +#define ESR_ELx_EC_MOPS (0x27) #define ESR_ELx_EC_FP_EXC32 (0x28) /* Unallocated EC: 0x29 - 0x2B */ #define ESR_ELx_EC_FP_EXC64 (0x2C) @@ -352,6 +352,15 @@ #define ESR_ELx_SME_ISS_ZA_DISABLED 3 #define ESR_ELx_SME_ISS_ZT_DISABLED 4 +/* ISS field definitions for MOPS exceptions */ +#define ESR_ELx_MOPS_ISS_MEM_INST (UL(1) << 24) +#define ESR_ELx_MOPS_ISS_FROM_EPILOGUE (UL(1) << 18) +#define ESR_ELx_MOPS_ISS_WRONG_OPTION (UL(1) << 17) +#define ESR_ELx_MOPS_ISS_OPTION_A (UL(1) << 16) +#define ESR_ELx_MOPS_ISS_DESTREG(esr) (((esr) & (UL(0x1f) << 10)) >> 10) +#define ESR_ELx_MOPS_ISS_SRCREG(esr) (((esr) & (UL(0x1f) << 5)) >> 5) +#define ESR_ELx_MOPS_ISS_SIZEREG(esr) (((esr) & (UL(0x1f) << 0)) >> 0) + #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index 92963f98afec..5a6dc3643e9b 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -77,6 +77,7 @@ void do_el0_svc(struct pt_regs *regs); void do_el0_svc_compat(struct pt_regs *regs); void do_el0_fpac(struct pt_regs *regs, unsigned long esr); void do_el1_fpac(struct pt_regs *regs, unsigned long esr); +void do_el0_mops(struct pt_regs *regs, unsigned long esr); void do_serror(struct pt_regs *regs, unsigned long esr); void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index cce1167199e3..2ef3ab5d7555 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -611,6 +611,14 @@ static void noinstr el0_bti(struct pt_regs *regs) exit_to_user_mode(regs); } +static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr) +{ + enter_from_user_mode(regs); + local_daif_restore(DAIF_PROCCTX); + do_el0_mops(regs, esr); + exit_to_user_mode(regs); +} + static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr) { enter_from_user_mode(regs); @@ -688,6 +696,9 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) case ESR_ELx_EC_BTI: el0_bti(regs); break; + case ESR_ELx_EC_MOPS: + el0_mops(regs, esr); + break; case ESR_ELx_EC_BREAKPT_LOW: case ESR_ELx_EC_SOFTSTP_LOW: case ESR_ELx_EC_WATCHPT_LOW: diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 0ccc063daccb..689188712909 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -507,6 +507,50 @@ void do_el1_fpac(struct pt_regs *regs, unsigned long esr) die("Oops - FPAC", regs, esr); } +void do_el0_mops(struct pt_regs *regs, unsigned long esr) +{ + bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION; + bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A; + int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr); + int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr); + int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr); + unsigned long dst, src, size; + + dst = pt_regs_read_reg(regs, dstreg); + src = pt_regs_read_reg(regs, srcreg); + size = pt_regs_read_reg(regs, sizereg); + + /* + * Put the registers back in the original format suitable for a + * prologue instruction, using the generic return routine from the + * Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH. + */ + if (esr & ESR_ELx_MOPS_ISS_MEM_INST) { + if ((!option_a && wrong_option) || (option_a && !wrong_option)) { + pt_regs_write_reg(regs, dstreg, dst + size); + pt_regs_write_reg(regs, sizereg, -size); + } + } else { + if ((option_a && wrong_option) || (!option_a && !wrong_option)) { + if (regs->pstate & PSR_N_BIT) { + pt_regs_write_reg(regs, dstreg, dst - size); + pt_regs_write_reg(regs, srcreg, src - size); + } + } else { + if (size & BIT(63)) { + pt_regs_write_reg(regs, dstreg, dst + size); + pt_regs_write_reg(regs, srcreg, src + size); + pt_regs_write_reg(regs, sizereg, -size); + } + } + } + + if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE) + regs->pc -= 8; + else + regs->pc -= 4; +} + #define __user_cache_maint(insn, address, res) \ if (address >= TASK_SIZE_MAX) { \ res = -EFAULT; \ @@ -817,6 +861,7 @@ static const char *esr_class_str[] = { [ESR_ELx_EC_DABT_LOW] = "DABT (lower EL)", [ESR_ELx_EC_DABT_CUR] = "DABT (current EL)", [ESR_ELx_EC_SP_ALIGN] = "SP Alignment", + [ESR_ELx_EC_MOPS] = "MOPS", [ESR_ELx_EC_FP_EXC32] = "FP (AArch32)", [ESR_ELx_EC_FP_EXC64] = "FP (AArch64)", [ESR_ELx_EC_SERROR] = "SError",