From patchwork Tue Sep 12 11:57:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138128 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp355876vqx; Tue, 12 Sep 2023 05:05:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFCRzLNlGGAEr3ylN8QOEU09ZOlEW+u4U7yuZe7/1eIBaMteGaPTvIKDvrK5QtI3/PutHwh X-Received: by 2002:a17:90b:1956:b0:268:314f:8f40 with SMTP id nk22-20020a17090b195600b00268314f8f40mr11053877pjb.16.1694520345787; Tue, 12 Sep 2023 05:05:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694520345; cv=none; d=google.com; s=arc-20160816; b=LFrzq5CajPRNVjvKJYQ94Ivx7qkWBYHQAyYrSrYLPFHQyc1hJjgihNsRuu1Aq+jDb7 fEW/OJItYIcfmiyCQDdbkOmsVkQiXLmEKxqoTq1ROZIPY4P28gmdZhNxKTF4op4P38Vo 3TPh8nFsSQgl8XpgDL0OTiilvtnBfnxKOhPsM9gZopXP085QG6ZRAmltgXmj1S3YXkOT D9QexBwbn6Aak3LPTRmBObIzSyYfP1VX/RQw0LBf8CZ95MRpuuoo/OX3xk8eCAiKuo8X MrufVOzH2uJ+yGU6YITQyAuw2ib1/zmHqdJ8VMeTQ1VssHWwNzCxDMyh5sJcBm6ZDPp5 ZzqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qQEa2caJqCZ1yZ2MHd+d/zVbc9q8FEM+3Tl2koyy8nc=; fh=bbkmuS+FOavf7ws39/zvRk36cN7w9Vj1/3qjZj/v/e4=; b=qLnqny/6Ll9kPxrGXSuwFD8lkSV03tV3OxlgegAWE4tMFtWne0KHGu4ues+6ce8t3G DT8Az89K47evQBVOhGB3ySBf+zANjwz3lG2CA6CgUd0J769byZqb6LaJ/zJrInx0z45u 8wxpGKLVbgd7TiTQezjrqe01xjb2raomWI8TAm+p3hn0ekeVg1Rxj50unyTVlS2sSZz5 kKjGnv5Q1j/ej7jEcgyyMp2FlNh8HQWJUVjoEMWpo1MrGicUaQI5cYvAfVN+molvLCBG +j9r+6nPZ6FLCA5iJJfms5gnK1iVRlglF0VbUmhne9o/PljtR8jcozo0u1itAXl6QoJI JkRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=OWSuIkta; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id na18-20020a17090b4c1200b002680c21ca6csi10142105pjb.95.2023.09.12.05.05.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 05:05:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=OWSuIkta; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id CADF780F7293; Tue, 12 Sep 2023 04:58:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234920AbjILL6A (ORCPT + 37 others); Tue, 12 Sep 2023 07:58:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234896AbjILL5u (ORCPT ); Tue, 12 Sep 2023 07:57:50 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E962110D2; Tue, 12 Sep 2023 04:57:46 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55D39C433C7; Tue, 12 Sep 2023 11:57:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519866; bh=Ea39ZA0/3RZcsItSpEiJ7oucesvvZLA878PUm8prZII=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OWSuIktaGwuOcxkYUaHG9dbixPw8+Fmca3NGIJ8npY33w861samcAJr8dlErVkZQZ ogAUnfuw/wBiSsJU7CrCWNYmnkTRayDwb5plTR0zOBrJGGAFkarcUMOS6MfvxFV1TQ W8Yr6CwRmua46FInyLDTl5TzVypnX9AGXkiqlulVBkoyKURC8DnXWllWLlXh8QFav/ TE7svf8wPhemHjBbEDEJ8ArE7MVqd5DG07kVK/Otkfz25G3EWK5ZPgH5Hcl+Y5IvUu nphOf8uAlggGwAR/0XPoM2AFjY4/XkoGJz13gNVswScu3YRugHFqINclexQWtcKA48 v+7F6PN3cggOw== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Conor Dooley , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= Subject: [RFC PATCH 1/6] riscv: sched: defer restoring Vector context for user Date: Tue, 12 Sep 2023 13:57:23 +0200 Message-Id: <20230912115728.172982-2-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:58:11 -0700 (PDT) X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776833366350822058 X-GMAIL-MSGID: 1776833366350822058 From: Andy Chiu User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches. Signed-off-by: Andy Chiu Acked-by: Conor Dooley Reviewed-by: Björn Töpel --- arch/riscv/include/asm/entry-common.h | 13 +++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/signal.c | 2 +- arch/riscv/kernel/vector.c | 2 +- 6 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 6e4dee49d84b..52926f4d8d7c 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,19 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..b182f2d03e25 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -93,12 +93,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index c5ee07b3df07..531873986a15 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -184,6 +184,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -191,7 +200,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index e32d737e039f..ec89e7edb6fd 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -153,6 +153,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 180d951d3624..0fca2c128b5f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 8d92fb6c522c..9d583b760db4 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Tue Sep 12 11:57:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138438 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp683187vqx; Tue, 12 Sep 2023 14:13:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFhzeTNUz4TiCgCl0+VgZ+PPdM4nqdg+xy58A/lkd7rxPMqKzVP97D/23KxFAlXXZa9yAlZ X-Received: by 2002:a05:6a00:1896:b0:68f:a92a:8509 with SMTP id x22-20020a056a00189600b0068fa92a8509mr5835763pfh.7.1694553183795; Tue, 12 Sep 2023 14:13:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694553183; cv=none; d=google.com; s=arc-20160816; b=X0Jk1uouv43uqzC8ZY0kHbUiikHsygVm34RF7B7bFakJ25pQVUZqMQpcFNqflImqie YBq6gfGTOStQ6U5mIOBVP0UDr+cXniyv0whuFF39XWHwl3EOBftAXWXs6JPzqjoXYcJl DbRFIIzbc0dW1vpkKWDsRFAXCKTrUqngXpghXblkjVOR5MP+Yb/YWt2zzzFnYIctPFg+ bUEDfcw5SgbIQovDaJZ5LLizOyRtaoKmL5ql/J4zbma8tLCuMFHl+Rs8Ehs1wuEGYQf5 JBFKBG35XBih5qq7N4J2OC8JCRSOToEDEowbc115SmykXnGS3MWZaoAWCo9E2WplIfj7 6pNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y819uLKh53epEKj/V0Yh6y5gNWD5Pel5wKj2jMGhTbc=; fh=y0acXNvN9AYKGBQ8KBV/edyZXsDtHTynT/GAJFlltSY=; b=rts8GNhiyEUtCZHQuDtfXM0D/FYRrVcEM9mbdhfjHJ3hTH8Lc9U3x7hlv7/9CzirAq xxdzaqDUw15s8uaecvy8RWCzKSWZyxmtxGoIK6uJDcRdB5z9kkg37K2DKz7nqUc/N1cU cQ05D8QLsMGdH22mPzHRSxcpXBXU2Zm+MMxvLNH0sN6imFx85+Ziieo3lzhVQbCF5KVu 1pzZVOW1rtPqwLqvOK7X9ri4ov1rHIKsvTf+oPkdq7AJdIMOKprsbzSxTc1/RVb9eErb 0/DVs4D3vVSmnMF8WvsSDKqAI8VNTbOlEW2BE9jSHTFfhYQA6jLVmaQ1JGKKuIZeZwY2 7k2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=uJQMU5Kp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id h190-20020a6383c7000000b00577743e7edasi4058055pge.205.2023.09.12.14.13.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 14:13:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=uJQMU5Kp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 3C39F8452EE0; Tue, 12 Sep 2023 04:58:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234933AbjILL6T (ORCPT + 37 others); Tue, 12 Sep 2023 07:58:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234883AbjILL6L (ORCPT ); Tue, 12 Sep 2023 07:58:11 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0A5210D5; Tue, 12 Sep 2023 04:57:50 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 03115C433C8; Tue, 12 Sep 2023 11:57:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519870; bh=Jm4l0sbOk6Ms1fXIEvExUKNazEpiinXNjMDaBl0+dio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uJQMU5KpIKu4PCBZchwGTlBMX7IENnzJyElrdYCvTtIKQkPONMCvt6kRbahyfGvwY bTm2rJRSAq2A/vLpBEPb8bWmW0+NgG15rXRjfom+lF6MvKuNtOg7JMkwYmOxwcHVD3 kgzxeK9dgmEunco6QNKZOk06YqcFIzaoXCT1d5yeeZ1NwPsE8HZvtyDWWk/pqiNbvF gTTfJJjhL21p8Y+5z5JGTWDBpDY8RS1N8jEtsWZVr6EVaIgeICyIODavG8R2kM/kJx bOnVmMx6aoleYapblYH92RR+MCLKIuB84RKNFqHiBt20763Dy0/FeELGZ86dmHwGxy jQOijpTQKUbtQ== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Vincent Chen Subject: [RFC PATCH 2/6] riscv: Add support for kernel mode vector Date: Tue, 12 Sep 2023 13:57:24 +0200 Message-Id: <20230912115728.172982-3-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:58:39 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776867799449221665 X-GMAIL-MSGID: 1776867799449221665 From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- arch/riscv/include/asm/simd.h | 50 ++++++++++++ arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++ 4 files changed, 154 insertions(+) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..ef70af78005d --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#ifdef CONFIG_RISCV_ISA_V + +DECLARE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 531873986a15..768acd517414 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,8 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 95cf25d48405..0597bb668b6e 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -62,6 +62,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..1c3b32d2b340 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + get_cpu_vector_context(); + + riscv_v_enable(); + + return 0; +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_set_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); From patchwork Tue Sep 12 11:57:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138437 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp682219vqx; Tue, 12 Sep 2023 14:10:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGZB5s0IVthb2JbozmrDKBStNQai3bS3gcmCnCIX+JjORGxKmlcJjoAYux3mxXxgHhuWvHa X-Received: by 2002:a05:6830:1e6b:b0:6b7:4a86:f038 with SMTP id m11-20020a0568301e6b00b006b74a86f038mr887750otr.15.1694553039056; Tue, 12 Sep 2023 14:10:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694553038; cv=none; d=google.com; s=arc-20160816; b=Qnd0C4NsnpyqEbz5fiA+noYsTJZNebuDpy++OUZbOm+XIBKEr47JYwRZwiiPzpzQ3q cORu/cEx/Fzh/EJal7ss+i8Ic5zgJy9MsW2AoWlUF4XSGc8hqHcP05inCX6+OE8Xo9aT Q3pkG4AKIvNvysy2XaMJ8PgfLiWqT59mVLMcy8xJ98yuSEEvS5hdIBd2KClyJD9SElxG Mxi4dh65o7TL1nRwBj1CZV0HnDJ1rM3oZYpETlOvtZ+Fpp/DJ5m8USJc8UWSHFnsomEj 0TcM1FiJ9GfglIAfvUOsZ3EP2CG12d/fmoYPmH2mg7bCpC71qb0yaSG4meEq4nXpB6Te 1vKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vB3tBBbp39GWDT+lu59LRTBFrUwVNDbAhUtqYeqqtj8=; fh=7ac1WITHXs70wMZD5cN37stWlaqYMESEmjDrX6h2Nb8=; b=w9yYu7ENOshICs2egHa5PK05H07t+gYevt+iODk87L1b8XHFmLqyGaLaRnCW9f4ihB ir2UTxeIgIp4UjTCzsIsUZWIfhnVkS3T/rBbGC97LWz5mVtzIkeX2R2jcCnAWPvD+8xw J6KXFdKOFqXSN6V3aHeMxvMQAal+KK0Ybabs1HhWEBZjl4W49+PTrdusTDhIEygKKuqy LYnz9IDjwOC48H3CDcIrfyV3tArKLQN3byyZePrELkKKeubLiAmQBHFBnHfojUje671N oOkGhfIYFFEQvV4hFP9VodOIqOScqdZcj5GEBmp3DXOVmIs2p/COvZaNS4GB+NPFA1X8 YAfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=dVMV6Pcc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id r10-20020a632b0a000000b0056baff5c55esi8441203pgr.74.2023.09.12.14.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 14:10:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=dVMV6Pcc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A5C198489E41; Tue, 12 Sep 2023 04:58:46 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234663AbjILL6g (ORCPT + 37 others); Tue, 12 Sep 2023 07:58:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234939AbjILL6R (ORCPT ); Tue, 12 Sep 2023 07:58:17 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDE26170A; Tue, 12 Sep 2023 04:57:54 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CADE2C433C7; Tue, 12 Sep 2023 11:57:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519874; bh=1ADMXMRwZkpdzFukU2GAMQEjbVj/Leav+1IcY00RlEw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dVMV6Pccozr1DmAoiXckxsMasYBCGNzRqYFffPsr6ND06gImbEZbAjJBeqNujL1nP HvzKuliJG53cdzoINehbgOT8cBE4SuQOydEN9H9vh9ARVG38rKPJ0Uv7yi71AeX3gz bnqWoGQWURSrVno/EXIIsIeXQ5E872EUYCGCArYmr7WQo4fxG8cOdbCIlJC+LTSrty F4AFyqW+CSincnMlFM/hA+bEGsPy5In6aoWvXlJYn2lHIPD4lUvx1J9fhXdWO8b8Bl /1mJ8Ut4yH4iKYxJDI3eYCKHrgWohMwPlx8t2p/VsRy4v0/0Ba4JGZt3B+XKYNSG5S huDDZuwK3IPKg== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Han-Kuan Chen , Conor Dooley Subject: [RFC PATCH 3/6] riscv: Add vector extension XOR implementation Date: Tue, 12 Sep 2023 13:57:25 +0200 Message-Id: <20230912115728.172982-4-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:58:46 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776867647370209704 X-GMAIL-MSGID: 1776867647370209704 From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..903c3275f8d0 --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..3bc059e18171 --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +ENTRY(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +ENTRY(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +ENTRY(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +ENTRY(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Tue Sep 12 11:57:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp357902vqx; Tue, 12 Sep 2023 05:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFepQfY4CllzY9EMJHi/FgPmksKFWkGg+4kLhn+OnRJXoyhRKTFLH+Tk7wnmChTJxngzO6M X-Received: by 2002:a17:902:eec6:b0:1c3:a2ea:64d3 with SMTP id h6-20020a170902eec600b001c3a2ea64d3mr5778358plb.41.1694520508937; Tue, 12 Sep 2023 05:08:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694520508; cv=none; d=google.com; s=arc-20160816; b=ldu0qq7Gxg3VNSBNMEyMxmY2GIrgTqizJ+ECt1mFZ92AiqdeamEKEzN+zHwy3XrsuF tgP1BTzVUxY96WlB896iABNVIlgADtFW2BdVf6kRNZmnXNCi72uFnZZetZ+Qdhqgw0/R fu/ZgH7Ho7Cl8zxrKZKuUxFNcypnvYx4dxTkplA/jIpoRVAF+7x+MfLv5hIEPKtm9JCI TIZ0UfzuhbqkAP6voodlCVLor+kvZaT70sZ4gO+F2zBzxSYBtpbn50YUX+iJ0DtWpfa4 SqXaexTWUraEGjnNTtlt/ZZUtcdK1FgBbRXHcP935y0f/EoDIuRmd9kB2pBEWAQurrCl v0nQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=juDFR4Ar9TgxKJrkiWqJEz8fXc4Ud+MPnp/iMr2Y9h0=; fh=C8mCC3ypj3OyIBqw5eZrikOox5/LznLR6aqIotMtVSY=; b=N2FZKILYhInMRGXp6GKIOzWiES6QOz0RJFR8Zy1YU3i5XI0YJgoe2pRYX0nRxUy8f3 zR/l3vi5nojXKq0TYKJDXLFpWPuXJDN6nDgkSw8hToCqp3PT9B6wZ9A6+Arr+ZVrqrHq iCi0ceaopsNvubiCjcbl/lFkvJEaBhFIimuboYXUGRjBa6rSnNGphmcOCKtFj0YBrJxi iTw+R3iqZbcYhjAmKDuedX4v2CJpo2Fbrc5waDErCmEd8wIkzBg4CN69cvYQtfCguxNf eoFUAEHAxXbDakkQvve1wHuSOEFHJ1LcHBDRrDmpXbcFzhUVoQzChBrCYcbgm8LcqHni sskA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iEbhIbao; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id h14-20020a170902680e00b001c38ae70497si6561399plk.553.2023.09.12.05.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 05:08:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iEbhIbao; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 198348148BB5; Tue, 12 Sep 2023 04:59:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234958AbjILL64 (ORCPT + 37 others); Tue, 12 Sep 2023 07:58:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234910AbjILL6T (ORCPT ); Tue, 12 Sep 2023 07:58:19 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 116D31722; Tue, 12 Sep 2023 04:57:58 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF5D4C433C8; Tue, 12 Sep 2023 11:57:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519877; bh=4Xylyg4Vp6jVguzMuCezz1iiCS344DbnSDbckTx6MjQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iEbhIbaoqPi47AUJVl/lKVeL5CpiDVMTgRXXH7bT2lk4AdvIgqSUZfGesMXkkTcA6 ZLQRGQOZO/3Ksh9YoKQEzahqLPYG6aWiQVWaQoWl+aM4Wv6NWs9WR3HVCz0bDe6O5G 42DzL4K5JEM8E9Fq0ceeYA2YnzHsazVTwmoX3f3xZpDrevRZwIHSDA6fUa3Mko6mG5 g/aYl+eA6C6enOca9zJvdeqAUocV8tdD+gM9UR7J5a1IY0Bvqy6rQHomMIkYNCkjiT RHAftuDsGNBOAU1eFaybqPUXwujUvM9GpMslSiVZvuyc2UaYQIUNEAhghspWSj+Fpq BoP5jUfhPcN3g== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 4/6] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Tue, 12 Sep 2023 13:57:26 +0200 Message-Id: <20230912115728.172982-5-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:59:10 -0700 (PDT) X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776833537079543469 X-GMAIL-MSGID: 1776833537079543469 From: Andy Chiu riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 52926f4d8d7c..aa1b9e50d6c8 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, { if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 768acd517414..9b818aac8a94 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -164,23 +164,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -201,7 +197,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -219,8 +215,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 1c3b32d2b340..d9e097e68937 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -68,7 +68,7 @@ void kernel_vector_begin(void) BUG_ON(!may_use_simd()); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); get_cpu_vector_context(); diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..2e7e00f4f8e1 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -100,7 +100,7 @@ static int riscv_vr_get(struct task_struct *target, * copying them to membuf. */ if (target == current) - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 0fca2c128b5f..75fd8cc05e10 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); From patchwork Tue Sep 12 11:57:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138127 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp355702vqx; Tue, 12 Sep 2023 05:05:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHYtE+UiFAVFm8q9zkt4AMk/e4paw81SFR1dFSEcd6YZ2M5BDbpZNcsGZnGeaNNskQ0LyDi X-Received: by 2002:a05:6a20:748b:b0:141:cc5f:7d28 with SMTP id p11-20020a056a20748b00b00141cc5f7d28mr11129741pzd.50.1694520333116; Tue, 12 Sep 2023 05:05:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694520333; cv=none; d=google.com; s=arc-20160816; b=v6NxIz8YyVaFbotu70xm8L4PbOHO5v8aBMBpggn5betFm7iWR/RIh6cstvcKe+R1OT o0SaVI5l5ueayIA80M2Wzzs1cG/aDO/Yp/B49OYO/MfeHLP1EJp+svz/RpeiMZ+eER3b dGrP7D1OJzFdd1gWLLYxkUE7PRoqRyLSKc/yIBe7lsqeSbVP3M+J52PfNI3YGlwWukHJ C6p2RE2Ni/2ouBl0QIe77MqYSaG/aTRgOBhHQ5xJ5XMd8Rp6V3vcuNRpHickxWFhSVrQ NC2HzX9myV+ONUWmkKdM03dBSOs9TO9xvOJfXmBEBnceLD6tx3IiD5uOiqbFTRORhdPK d27A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=r7sHk7BzTgrc9LRivvKaHx6bCJ4W90JKrm0Ch+E1hdg=; fh=C8mCC3ypj3OyIBqw5eZrikOox5/LznLR6aqIotMtVSY=; b=qbnbahzYvhqv0GwLhR8C0UMlauk6Yu5ynWKhRbi4EgzJ2WmakBPG0lMjirgzUCaMcm FVHegcoukq9qYsEffL+qnuPxb89LzNdfFxyh8wMN6YYJr3aksEF8Il1PiPPHNls0gToE 24my2ACpCwy4nIoouZOkvgA8bR0KLMbOEtWvoIEDXEamz0/9gmd0VEIqi+6gRlF11Fis mMvRjxuh7lDCs2lpNo73PwxTYnSQGe9gkQ7fdSev1Y9IbP68WYv8Bl1q/PR9rgAL6lRT ythDG1MfuLU3SuGlm7PsE6UUkwDCFflCpVzKloozCKR0zmL/cz1iYMgNy0LJ+tfksMi4 IGHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N9waNLu4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id o185-20020a6341c2000000b0056b83e0ae92si7852923pga.465.2023.09.12.05.05.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 05:05:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N9waNLu4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 55D4B8148BBD; Tue, 12 Sep 2023 04:59:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235065AbjILL7O (ORCPT + 37 others); Tue, 12 Sep 2023 07:59:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234978AbjILL6U (ORCPT ); Tue, 12 Sep 2023 07:58:20 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C66131981; Tue, 12 Sep 2023 04:58:01 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30E0BC433C7; Tue, 12 Sep 2023 11:57:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519881; bh=G3ZQqiwe5gyTFzZ+daR1X2GrYG5m5yMSRJlV8uyElWE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N9waNLu43RFFaehAC2n7+UH532/KbktpyyUI5X4q9mkeuMBcnRYayjmZetCZt/gbv KbY+tNF613mLp/gYnys/+D2vN13XmCS4e3aDnOjY9Xy7pgvUUz3UcCdSGnoNWNFgEN zSqTfJ56Y4/gIhE22VFdx3QfCxyDAvZQpdSAoK1ymsPfxcjdZrW/wsmmKy9r3AoTlT dPyljxaiQO0uLPQGsB6zioThmZOXDLPjtSTbOhOiUN20Xx9Hq9QnNTDgIwiyNBP6Br KSLoRFPUZuqT/zd5r06UTfaV3WUjBdKoowgmgFXuW3qoLl4trKIz+ZzguGO7bpsOqA 2PqjCXdRT2fzA== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 5/6] riscv: vector: allow kernel-mode Vector with preemption Date: Tue, 12 Sep 2023 13:57:27 +0200 Message-Id: <20230912115728.172982-6-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:59:37 -0700 (PDT) X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776833352898521100 X-GMAIL-MSGID: 1776833352898521100 From: Andy Chiu Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set. Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead. Signed-off-by: Andy Chiu --- arch/riscv/Kconfig | 10 +++++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/simd.h | 4 +- arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 +++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 45 ++++++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 10 files changed, 148 insertions(+), 10 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d607ab0f7c6d..dc51164b8fd4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -520,6 +520,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding memory on demand for tracking kernel's V-context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 3e23e1786d05..f9b85e37e624 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -82,6 +82,8 @@ struct thread_struct { unsigned long bad_cause; unsigned long vstate_ctrl; struct __riscv_v_ext_state vstate; + struct pt_regs *trap_pt_regs; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef70af78005d..a54a0ce58f4d 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_RISCV_ISA_V @@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void) * where it is set. */ return !in_irq() && !irqs_disabled() && !in_nmi() && - !this_cpu_read(vector_context_busy); + !this_cpu_read(vector_context_busy) && + !test_thread_flag(TIF_RISCV_V_KERNEL_MODE); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index b182f2d03e25..8797d520e8ef 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ +#define TIF_RISCV_V_KERNEL_MODE 13 /* kernel-mode Vector run with preemption-on */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KERNEL_MODE (1 << TIF_RISCV_V_KERNEL_MODE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE 0x20 + #endif /* _ASM_RISCV_THREAD_INFO_H */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 9b818aac8a94..9fb2dea66abd 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -196,9 +196,24 @@ static inline void __switch_to_vector(struct task_struct *prev, { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && + test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) { + regs = prev->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_save(&prev->thread.kernel_vstate, regs); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && + test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) { + regs = next->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_restore(&next->thread.kernel_vstate, regs); + } else { + riscv_v_vstate_set_restore(next, task_pt_regs(next)); + } } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -223,4 +238,10 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #endif /* CONFIG_RISCV_ISA_V */ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +void kernel_vector_allow_preemption(void); +#else +#define kernel_vector_allow_preemption() do {} while (0) +#endif + #endif /* ! __ASM_RISCV_VECTOR_H */ diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..4b062f7741b2 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,8 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_THREAD_TRAP_REGP, task_struct, thread.trap_pt_regs); + OFFSET(TASK_THREAD_VSTATE_CTRL, task_struct, thread.vstate_ctrl); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..b6a7d4e9f526 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -66,6 +66,29 @@ _save_context: REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + /* + * Record the register set at the frame where in-kernel V registers are + * last alive. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KERNEL_MODE + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bnez s1, 1f + REG_S sp, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + li s2, ~RISCV_V_VSTATE_CTRL_PREEMPTIBLE + and s1, s1, s2 + REG_S s1, (s0) +1: +#endif + /* * Set the scratch register to 0, so that if a recursive exception * occurs, the exception vector knows it came from the kernel @@ -129,6 +152,28 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + /* + * Clear tracking of the trap registers when we return to the frame + * that uses kernel mode Vector. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KERNEL_MODE + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bne s1, sp, 1f + REG_S x0, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + ori s1, s1, RISCV_V_VSTATE_CTRL_PREEMPTIBLE + REG_S s1, (s0) +1: +#endif + REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index d9e097e68937..5c64f2034cdc 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -48,6 +49,44 @@ static void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +void kernel_vector_allow_preemption(void) +{ + current->thread.vstate_ctrl |= RISCV_V_VSTATE_CTRL_PREEMPTIBLE; +} + +static bool kernel_vector_preemptible(void) +{ + return !!(current->thread.vstate_ctrl & RISCV_V_VSTATE_CTRL_PREEMPTIBLE); +} + +static int riscv_v_start_kernel_context(void) +{ + struct __riscv_v_ext_state *vstate; + + vstate = ¤t->thread.kernel_vstate; + if (!vstate->datap) { + vstate->datap = kmalloc(riscv_v_vsize, GFP_KERNEL); + if (!vstate->datap) + return -ENOMEM; + } + + current->thread.trap_pt_regs = NULL; + WARN_ON(test_and_set_thread_flag(TIF_RISCV_V_KERNEL_MODE)); + return 0; +} + +static void riscv_v_stop_kernel_context(void) +{ + WARN_ON(!test_and_clear_thread_flag(TIF_RISCV_V_KERNEL_MODE)); + current->thread.trap_pt_regs = NULL; +} +#else +#define kernel_vector_preemptible() (false) +#define riscv_v_start_kernel_context() (0) +#define riscv_v_stop_kernel_context() do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -70,11 +109,14 @@ void kernel_vector_begin(void) riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); - get_cpu_vector_context(); + if (!preemptible() || !kernel_vector_preemptible()) { + get_cpu_vector_context(); + } else { + if (riscv_v_start_kernel_context()) + get_cpu_vector_context(); + } riscv_v_enable(); - - return 0; } EXPORT_SYMBOL_GPL(kernel_vector_begin); @@ -96,6 +138,9 @@ void kernel_vector_end(void) riscv_v_disable(); - put_cpu_vector_context(); + if (!test_thread_flag(TIF_RISCV_V_KERNEL_MODE)) + put_cpu_vector_context(); + else + riscv_v_stop_kernel_context(); } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index ec89e7edb6fd..18cb37c305ab 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -160,8 +160,11 @@ void flush_thread(void) void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ - if (has_vector()) + if (has_vector()) { kfree(tsk->thread.vstate.datap); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE)) + kfree(tsk->thread.kernel_vstate.datap); + } } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); + clear_tsk_thread_flag(dst, TIF_RISCV_V_KERNEL_MODE); return 0; } @@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + kernel_vector_allow_preemption(); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 9d583b760db4..42f227077ee5 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Tue Sep 12 11:57:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 138152 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp377968vqx; Tue, 12 Sep 2023 05:44:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH6fiHxc5QLLWS1p1ZNeoHBQstjpFw6KZIb9fv838TSLhzadZ0jGnRNb+HkK1CiTxdav88Y X-Received: by 2002:a05:6808:1495:b0:3a4:25ab:eecb with SMTP id e21-20020a056808149500b003a425abeecbmr15129119oiw.51.1694522663897; Tue, 12 Sep 2023 05:44:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694522663; cv=none; d=google.com; s=arc-20160816; b=ovDun/4ys9c90y0VatbOCCpEVV1W6zoSwcweTNbvhQTJq1aS3b5vEiLqAWPBfhEhxp Jimxy+pVAz8/+Tic5U44fAJVm+WbUXZ8Rqb9KsDEOIuLPtVLuWwrDRw5OJvDFJjHh2g6 wEM/xMr4AcB9jNs/w59O+OBzRG50KnBUsKfp4RfSNVGGHjU8WBJDL55/qxpPdNr8QSDW 9JU05L8dtFdYGn6iicNarA4GcG5hbWFMMmjvYJQAsA/xnllqjF7bVVzqXw7jhRXOFRF3 xHHMF3LF5bUPq1YA7qF8kKP9lia4U6U0AW3mObC/nwiuh0UM7K2yBx6iLvE372cPSTJG qUAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MxlNboNPvMcHzGrn//XaoIv/SPeZtda7ptCbuhSCKoo=; fh=LLe8Ph/5vJdOAAtnZSTKJFI0/S74xLKiNByYIJ9D7E4=; b=hN4ykHQuH4wkZLPjj5utlpX6TeLSaccX4GTLx2kFeTYbXXFaQltLrbtAl9ZIZS3KRy AXGpI/sqGYvQcIQu/49K3m7M5GZTJm2HmuL1XvqqGhsuUaXRtONc0xcJXtWQBFtm3yAK Zb6OKvLdetnaKlFGbyVy4f08pcLi0Gd7jGCOJXr7q+BccHLfD+S9WDdAbnplnvnr0jiq FtDVIOExaztCfEcfSciQ4qGODzEoNatLRBQFD0TNSrWhZanOkU3cKU5+HRpPtfqU8YX4 nyOBq5mPsBpIIXDJVGtXppHtEDu2VvHcOb3pmpMWyJc8h0wmdEpHciDLtO7UVIJ+3H3Z m4Ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MoYuiZ1H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id c5-20020a6566c5000000b00573f9a427d6si2814145pgw.450.2023.09.12.05.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 05:44:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MoYuiZ1H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 0961F8108BEE; Tue, 12 Sep 2023 04:59:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.8 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234958AbjILL7Y (ORCPT + 37 others); Tue, 12 Sep 2023 07:59:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234931AbjILL6s (ORCPT ); Tue, 12 Sep 2023 07:58:48 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34AF61995; Tue, 12 Sep 2023 04:58:04 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A657FC433C9; Tue, 12 Sep 2023 11:58:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694519884; bh=FNGfbJ4BeAojKOcc8XZoe5o2N5XosASh69jYqRn8nm0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MoYuiZ1Hv5aNzyMT9Y3OkcWL6CW/YbKqWh5x7U/f5bq07dWG6nAcBBIECn52cd03r zwDfkwxY8yoHhmQzPBH2CL5KzMDJKbMDNtFoe2QI6VjQWXgJ2jTGvhUepnN5iOn58e XH1CPPpeBOIft+G5++iqN0k/h3alC774X/l+dboqElEAHFzegWZ+6rJkEB/DTRPfyl MTfAnvEQxzvC6upk11CAo/eQQzlvk1ZjNRkuUzVcbgSqkd9u4LJ7g6XF7QWKw+RD1A KsY28Hc/2oJ0zqRyeMa11tV3J5uXN38h6qJl44TqKRACyd/50HG2RXJZFeOyi+VRBf sBwt0qyN4NJqQ== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org, Andy Chiu , Greentime Hu , "Jason A . Donenfeld" , Samuel Neves Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Heiko Stuebner , Herbert Xu , "David S. Miller" , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 6/6] riscv: Add BLAKE2s V implementation Date: Tue, 12 Sep 2023 13:57:28 +0200 Message-Id: <20230912115728.172982-7-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230912115728.172982-1-bjorn@kernel.org> References: <20230912115728.172982-1-bjorn@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 12 Sep 2023 04:59:37 -0700 (PDT) X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776835796364716909 X-GMAIL-MSGID: 1776835796364716909 From: Björn Töpel BLAKE2s is used in various places, most notably in Wireguard, and as of v5.17 in drivers/char/random. Add a BLAKE2s implementation using the V-extension. This is a translation of the x86-64 AVX512 variant (arch/x86/crypto/blake2s-core.S) to the RISC-V V-extension. The AVX512 variant requires registers >= 256b (ymm*), and hence this implementation requires a VLEN of >=256. The implementation passes the kernel BLAKE2s selftest, and has been tested on spike and qemu. Instruction-wise, the V-variant uses 60% less instructions than the generic, C-based, implementation. Signed-off-by: Björn Töpel --- arch/riscv/Kbuild | 2 +- arch/riscv/crypto/Kconfig | 16 +++ arch/riscv/crypto/Makefile | 6 ++ arch/riscv/crypto/blake2s-glue.c | 39 ++++++++ arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++++++++ crypto/Kconfig | 3 + drivers/net/Kconfig | 1 + 7 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/crypto/Kconfig create mode 100644 arch/riscv/crypto/Makefile create mode 100644 arch/riscv/crypto/blake2s-glue.c create mode 100644 arch/riscv/crypto/blake2s-v.S diff --git a/arch/riscv/Kbuild b/arch/riscv/Kbuild index d25ad1c19f88..1a5d89a18920 100644 --- a/arch/riscv/Kbuild +++ b/arch/riscv/Kbuild @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-y += kernel/ mm/ net/ +obj-y += kernel/ mm/ net/ crypto/ obj-$(CONFIG_BUILTIN_DTB) += boot/dts/ obj-y += errata/ obj-$(CONFIG_KVM) += kvm/ diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig new file mode 100644 index 000000000000..e072400eb456 --- /dev/null +++ b/arch/riscv/crypto/Kconfig @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0 + +menu "Accelerated Cryptographic Algorithms for CPU (RISC-V)" + +config CRYPTO_BLAKE2S_RISCV + bool "Hash functions: BLAKE2s (V)" + depends on 64BIT && MMU && RISCV_ISA_V + select CRYPTO_LIB_BLAKE2S_GENERIC + select CRYPTO_ARCH_HAVE_LIB_BLAKE2S + help + BLAKE2s cryptographic hash function (RFC 7693) + + Architecture: riscv64 using: + - V extension (VLEN >= 256) + +endmenu diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile new file mode 100644 index 000000000000..1768025c13ae --- /dev/null +++ b/arch/riscv/crypto/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# RISC-V crypto algorithms + +obj-$(CONFIG_CRYPTO_BLAKE2S_RISCV) += libblake2s-riscv.o +libblake2s-riscv-y := blake2s-v.o blake2s-glue.o diff --git a/arch/riscv/crypto/blake2s-glue.c b/arch/riscv/crypto/blake2s-glue.c new file mode 100644 index 000000000000..d3de0e8e5872 --- /dev/null +++ b/arch/riscv/crypto/blake2s-glue.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* + * Copyright (C) 2022 Rivos, Inc. All Rights Reserved. + * Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. + */ + +#include + +#include +#include +#include + +#include +#include +#include +#include + +asmlinkage void blake2s_compress_vector(struct blake2s_state *state, const u8 *block, + const size_t nblocks, const u32 inc); + +void blake2s_compress(struct blake2s_state *state, const u8 *block, size_t nblocks, const u32 inc) +{ + if (!(has_vector() && riscv_v_vsize >= 256 * 32 / 8) || !may_use_simd()) { + blake2s_compress_generic(state, block, nblocks, inc); + return; + } + + do { + const size_t blocks = min_t(size_t, nblocks, SZ_4K / BLAKE2S_BLOCK_SIZE); + + kernel_vector_begin(); + blake2s_compress_vector(state, block, blocks, inc); + kernel_vector_end(); + + nblocks -= blocks; + block += blocks * BLAKE2S_BLOCK_SIZE; + } while (nblocks); +} +EXPORT_SYMBOL(blake2s_compress); diff --git a/arch/riscv/crypto/blake2s-v.S b/arch/riscv/crypto/blake2s-v.S new file mode 100644 index 000000000000..9653b7a19127 --- /dev/null +++ b/arch/riscv/crypto/blake2s-v.S @@ -0,0 +1,164 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * BLAKE2s RISC-V V implementation + * + * Copyright (C) 2022 Rivos, Inc. All Rights Reserved. + * Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. + * Copyright (C) 2017-2019 Samuel Neves . All Rights Reserved. + */ + +#include +#include + +.section .rodata, "a", @progbits + +.align 8 +IV: +.octa 0xA54FF53A3C6EF372BB67AE856A09E667 +.octa 0x5BE0CD191F83D9AB9B05688C510E527F + +.align 8 +SIGMA2: +.long 0, 2, 4, 6, 1, 3, 5, 7, 14, 8, 10, 12, 15, 9, 11, 13 +.long 8, 2, 13, 15, 10, 9, 12, 3, 6, 4, 0, 14, 5, 11, 1, 7 +.long 11, 13, 8, 6, 5, 10, 14, 3, 2, 4, 12, 15, 1, 0, 7, 9 +.long 11, 10, 7, 0, 8, 15, 1, 13, 3, 6, 2, 12, 4, 14, 9, 5 +.long 4, 10, 9, 14, 15, 0, 11, 8, 1, 7, 3, 13, 2, 5, 6, 12 +.long 2, 11, 4, 15, 14, 3, 10, 8, 13, 6, 5, 7, 0, 12, 1, 9 +.long 4, 8, 15, 9, 14, 11, 13, 5, 3, 2, 1, 12, 6, 10, 7, 0 +.long 6, 13, 0, 14, 12, 2, 1, 11, 15, 4, 5, 8, 7, 9, 3, 10 +.long 15, 5, 4, 13, 10, 7, 3, 11, 12, 2, 0, 6, 9, 8, 1, 14 +.long 8, 7, 14, 11, 13, 15, 0, 12, 10, 4, 5, 6, 3, 2, 1, 9 + +.align 8 +GATHER_DIAG1: +.long 3, 0, 1, 2 +.align 8 +GATHER_DIAG2: +.long 2, 3, 0, 1 +.align 8 +GATHER_DIAG3: +.long 1, 2, 3, 0 + +.section .text + +.macro VROR REG BITS TMPREG + vmv.v.v \TMPREG, \REG + vsrl.vi \REG, \REG, \BITS + vsll.vi \TMPREG, \TMPREG, 32-\BITS + vor.vv \REG, \REG, \TMPREG +.endm + +/* + * void blake2s_compress_vector(struct blake2s_state *state, + * const u8 *block, size_t nblocks, const u32 inc) + * + * blake2s_compress_vector is a translation of the AVX512 variant + * (arch/x86/blake2s-core.S) to RISC-V V. The implementation requires + * VLEN>=256b. + * + */ +SYM_FUNC_START(blake2s_compress_vector) + vsetivli t0, 1, e32, m1, ta, ma + vmv.v.x v5, a3 + vsetivli t0, 4, e32, m1, ta, ma + vle32.v v0, (a0) + addi t0, a0, 0x10 + vle32.v v1, (t0) + addi t0, t0, 0x10 + vle32.v v4, (t0) + la t0, IV + vle32.v v14, (t0) + la t0, IV+16 + vle32.v v15, (t0) + la t0, GATHER_DIAG1 + vle32.v v16, (t0) + la t0, GATHER_DIAG2 + vle32.v v18, (t0) + la t0, GATHER_DIAG3 + vle32.v v19, (t0) +.Lblake2s_compress_vector_mainloop: + vmv.v.v v10, v0 + vmv.v.v v11, v1 + vadd.vv v4, v5, v4 + vmv.v.v v2, v14 + vxor.vv v3, v15, v4 + vsetivli t0, 8, e32, m1, ta, ma + vle32.v v6, (a1) + addi a1, a1, 0x20 + vle32.v v7, (a1) + addi a1, a1, 0x20 + la t1, SIGMA2 + li t2, 0xa +.Lblake2s_compress_vector_roundloop: + vsetivli t0, 8, e32, m1, ta, ma + vle32.v v12, (t1) + addi t1, t1, 0x20 + vle32.v v13, (t1) + addi t1, t1, 0x20 + vsetivli t0, 16, e32, m2, ta, ma + vrgather.vv v8, v6, v12 + vmv.v.v v6, v8 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v8, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 16 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 12 v17 + vsetivli t0, 8, e32, m1, ta, ma + vslidedown.vi v8, v8, 4 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v8, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 8 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1, 7, v17 + vmv.v.v v17, v0 + vrgather.vv v0, v17, v16 + vmv.v.v v17, v3 + vrgather.vv v3, v17, v18 + vmv.v.v v17, v2 + vrgather.vv v2, v17, v19 + vadd.vv v0, v9, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 16 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 12 v17 + vsetivli t0, 8, e32, m1, ta, ma + vslidedown.vi v9, v9, 4 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v9, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3, 8, v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 7 v17 + vmv.v.v v17, v0 + vrgather.vv v0, v17, v19 + vmv.v.v v17, v3 + vrgather.vv v3, v17, v18 + vmv.v.v v17, v2 + vrgather.vv v2, v17, v16 + addi t2, t2, -1 + bne t2, x0, .Lblake2s_compress_vector_roundloop + vxor.vv v0, v10, v0 + vxor.vv v1, v11, v1 + vxor.vv v0, v2, v0 + vxor.vv v1, v3, v1 + addi a2, a2, -1 + bne a2, x0, .Lblake2s_compress_vector_mainloop + vse32.v v0, (a0) + addi a0, a0, 0x10 + vse32.v v1, (a0) + addi a0, a0, 0x10 + vse32.v v4, (a0) + ret +SYM_FUNC_END(blake2s_compress_vector) + diff --git a/crypto/Kconfig b/crypto/Kconfig index 650b1b3620d8..c7b23d2c58e4 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1436,6 +1436,9 @@ endif if PPC source "arch/powerpc/crypto/Kconfig" endif +if RISCV +source "arch/riscv/crypto/Kconfig" +endif if S390 source "arch/s390/crypto/Kconfig" endif diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index 44eeb5d61ba9..f7fb168a3944 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -94,6 +94,7 @@ config WIREGUARD select CRYPTO_CHACHA_MIPS if CPU_MIPS32_R2 select CRYPTO_POLY1305_MIPS if MIPS select CRYPTO_CHACHA_S390 if S390 + select CRYPTO_BLAKE2S_RISCV if RISCV && 64BIT help WireGuard is a secure, fast, and easy to use replacement for IPSec that uses modern cryptography and clever networking tricks. It's