From patchwork Tue Feb 21 19:08:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 60232 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp178589wrd; Tue, 21 Feb 2023 11:12:47 -0800 (PST) X-Google-Smtp-Source: AK7set8/vuixCU5Zd35WgE+fwfuyGwuhA2p6qp5A9awHjZapPe8ZnOMjF6F75pYnazf9Atvc96Zu X-Received: by 2002:a17:906:9f25:b0:8b1:7569:ad58 with SMTP id fy37-20020a1709069f2500b008b17569ad58mr10099908ejc.2.1677006767264; Tue, 21 Feb 2023 11:12:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677006767; cv=none; d=google.com; s=arc-20160816; b=l1TByseU5m6oVSaTZhEWHyW1lFZfRZ3uaZzrkpvV0hdvBhTsI5Xxlz9eN+COPWsFJ4 oehtVs1moaSTA5Yw3RcVWl7NGleuSJafwqo7r0n7G6KsFuG8c0pW+cnbMRksAbb4i4lA lIIOqCf8RTJLe8M6OrR/kDH2Djqfii5sjftA3Fgua0WV873Izn5HVBq8WYQk7lXc2H4+ tXo0KXaA5fF/Em6ntpxH1XkNj6JIPxDJVxrIdWi0THM6+A0FBk1ZkJoxuv2+eFhCpueq nCgnmv/t1nqDDMH6pjQrl3mD2IOO3FWUFfDsiP0wuE2tpKWcNJzTgVwjO+4Ex4pd2ZEg nkgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KbdXcY/cyfsn3vuj+G9y8VbBmqDYFOwE9fF3Bt0phIY=; b=MgEL/n51V/qmN+I6eJX/GSMlRHMPfE5EhzaAIbBFhmMuGLpFAhTOOvZHKhDWWyBqaT bcWbWaqTVxaww+uk4AVFmYHMjAOX4FztCLBB+BZ1WDzrm91830cA0pIZhvdH/5Gi6ju0 O6q6epvrjEKQuKq+qcTCr7SsY19oD/y2qkAIOkgVaZSWCigvdPljN0rhSVxioxwsI8X4 UvcKQ175c4uoSeCDkhxPNrFDqtq4FYEd/OCdPzjjeXo502jbP0Ag1E6kOaAspS1gyzGT VGQ8CQwXAjYY1QU8U/EwTrKGqUaGil1fVqXrNZ6OPiiM1B9atIqMKNPI5SYGEezVxXUI o6HQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=csHRX7en; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id uk19-20020a170907ca1300b008e23e890059si1678738ejc.268.2023.02.21.11.12.23; Tue, 21 Feb 2023 11:12:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=csHRX7en; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230020AbjBUTKQ (ORCPT + 99 others); Tue, 21 Feb 2023 14:10:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbjBUTKA (ORCPT ); Tue, 21 Feb 2023 14:10:00 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5951F305C8 for ; Tue, 21 Feb 2023 11:09:45 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id x34so2718673pjj.0 for ; Tue, 21 Feb 2023 11:09:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KbdXcY/cyfsn3vuj+G9y8VbBmqDYFOwE9fF3Bt0phIY=; b=csHRX7enYMnjCMTP6woBc7E2NOR8kXQNGrxRlxwRA8LYFRm+dEXDW8b5XJRYkt2T2i Ahe5pRPhESqNItcsWgXWfbKdy+PLMavIffxoF9BJBL/P2snpZ3JDeACBsw5ajcZKn+ac 78mpOZYXwyOEEQxOJrhICXEKjksghXtqu8vtE5kq2ubGy9KRD/HNngW+RedQK7TS43Z4 i8Q1lrLQhnDsHVqQvyOtaCFPL+PBFup/ns3XMrGJYw84HO49GX1ZI5I9tglYnSfoj2HF WzKyhZK2MY4ut9Nxd1bPPkfXLI0Sm1k/aBGYKAfqaS7ZhYgK7U+D4M1x3szK7S6lD3q3 0Big== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KbdXcY/cyfsn3vuj+G9y8VbBmqDYFOwE9fF3Bt0phIY=; b=JnlJn7+5jCBZVowHKW5UD8ZW5vzE93vQywY7hBuvyeNbT3qS+eTdZ52bj1qNtK8hzA h2li2954gCvUWhe0DYOKKWhYpREC9WrDhoJhA8BtnOduDVaNU4pW+2nsu4F7GLGBwqlJ /SSC61G1gnUIKpO7pi5/e5Tq+Dhtk6Vbl9kFCuhYRa+KWcqGzk2yVYYnUvHU2NTyVNP5 6uKQ+RKaG+WilPvmFowCK48l9fUdbsE+f5mn80einMJwsNR/o59Jij6yPcf/l57m/MQf pPltKrHIoDlMjI074jNYd616Y2F1+bWqhgdgzmBBD7EzE+G2nwk6B45Z2aODVkHql1uz Xj0Q== X-Gm-Message-State: AO0yUKXukwrMQc+9ltf9eJJbrFhnxJIAN66Zv+n/rR5I7zGj9zcOII9J 416OEajIxCal/GIX+UF66Y97vQ== X-Received: by 2002:a17:903:24e:b0:198:b99c:f6ee with SMTP id j14-20020a170903024e00b00198b99cf6eemr8941360plh.65.1677006584803; Tue, 21 Feb 2023 11:09:44 -0800 (PST) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id b16-20020a170902b61000b0019602b2c00csm1583863pls.175.2023.02.21.11.09.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Feb 2023 11:09:44 -0800 (PST) From: Evan Green To: Palmer Dabbelt Cc: heiko@sntech.de, Conor Dooley , slewis@rivosinc.com, vineetg@rivosinc.com, Evan Green , Albert Ou , Andrew Bresticker , Arnd Bergmann , Celeste Liu , Guo Ren , Jisheng Zhang , Nathan Chancellor , Palmer Dabbelt , Paul Walmsley , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v3 7/7] RISC-V: Add hwprobe vDSO function and data Date: Tue, 21 Feb 2023 11:08:58 -0800 Message-Id: <20230221190858.3159617-8-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230221190858.3159617-1-evan@rivosinc.com> References: <20230221190858.3159617-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758469047861473983?= X-GMAIL-MSGID: =?utf-8?q?1758469047861473983?= Add a vDSO function __vdso_riscv_hwprobe, which can sit in front of the riscv_hwprobe syscall and answer common queries. We stash a copy of static answers for the "all CPUs" case in the vDSO data page. This data is private to the vDSO, so we can decide later to change what's stored there or under what conditions we defer to the syscall. Currently all data can be discovered at boot, so the vDSO function answers all queries when the cpumask is set to the "all CPUs" hint. There's also a boolean in the data that lets the vDSO function know that all CPUs are the same. In that case, the vDSO will also answer queries for arbitrary CPU masks in addition to the "all CPUs" hint. Signed-off-by: Evan Green --- Changes in v3: - Introduced vDSO function One aspect of this that's less than perfect is that there are two copies of the arch_vdso_data in the data page. This stems from the fact that vdso_data is arrayed by CS_BASES. Since we're very far away from filling up the page, and this is not ABI, I left it for now. If things get full we can move this data out to its own page, or refactor vdso_data tree-wide to allow for non-arrayed data. --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/vdso/data.h | 17 ++++++++++ arch/riscv/kernel/sys_riscv.c | 41 ++++++++++++++++++++++++ arch/riscv/kernel/vdso/Makefile | 2 ++ arch/riscv/kernel/vdso/hwprobe.c | 47 ++++++++++++++++++++++++++++ arch/riscv/kernel/vdso/sys_hwprobe.S | 15 +++++++++ arch/riscv/kernel/vdso/vdso.lds.S | 1 + 7 files changed, 124 insertions(+) create mode 100644 arch/riscv/include/asm/vdso/data.h create mode 100644 arch/riscv/kernel/vdso/hwprobe.c create mode 100644 arch/riscv/kernel/vdso/sys_hwprobe.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index e2b656043abf..218e75975a15 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -33,6 +33,7 @@ config RISCV select ARCH_HAS_STRICT_MODULE_RWX if MMU && !XIP_KERNEL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAS_UBSAN_SANITIZE_ALL + select ARCH_HAS_VDSO_DATA select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT select ARCH_STACKWALK diff --git a/arch/riscv/include/asm/vdso/data.h b/arch/riscv/include/asm/vdso/data.h new file mode 100644 index 000000000000..dc2f76f58b76 --- /dev/null +++ b/arch/riscv/include/asm/vdso/data.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __RISCV_ASM_VDSO_DATA_H +#define __RISCV_ASM_VDSO_DATA_H + +#include +#include +#include + +struct arch_vdso_data { + /* Stash static answers to the hwprobe queries when all CPUs are selected. */ + __u64 all_cpu_hwprobe_values[RISCV_HWPROBE_MAX_KEY + 1]; + + /* Boolean indicating all CPUs have the same static hwprobe values. */ + __u8 homogeneous_cpus; +}; + +#endif /* __RISCV_ASM_VDSO_DATA_H */ diff --git a/arch/riscv/kernel/sys_riscv.c b/arch/riscv/kernel/sys_riscv.c index afc1b96d5c1a..113b3d97c8cc 100644 --- a/arch/riscv/kernel/sys_riscv.c +++ b/arch/riscv/kernel/sys_riscv.c @@ -14,6 +14,7 @@ #include #include #include +#include static long riscv_sys_mmap(unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, @@ -242,6 +243,46 @@ int do_riscv_hwprobe(struct riscv_hwprobe __user *pairs, long pair_count, } +static int __init init_hwprobe_vdso_data(void) +{ + struct vdso_data *vd = __arch_get_k_vdso_data(); + struct arch_vdso_data *avd = &vd->arch_data; + u64 id_bitsmash = 0; + struct riscv_hwprobe pair; + int key; + + /* + * Initialize vDSO data with the answers for the "all CPUs" case, to + * save a syscall in the common case. + */ + for (key = 0; key <= RISCV_HWPROBE_MAX_KEY; key++) { + pair.key = key; + hwprobe_one_pair(&pair, cpu_online_mask); + + WARN_ON_ONCE(pair.key < 0); + + avd->all_cpu_hwprobe_values[key] = pair.value; + /* + * Smash together the vendor, arch, and impl IDs to see if + * they're all 0 or any negative. + */ + if (key <= RISCV_HWPROBE_KEY_MIMPID) + id_bitsmash |= pair.value; + } + + /* + * If the arch, vendor, and implementation ID are all the same across + * all harts, then assume all CPUs are the same, and allow the vDSO to + * answer queries for arbitrary masks. However if all values are 0 (not + * populated) or any value returns -1 (varies across CPUs), then the + * vDSO should defer to the kernel for exotic cpu masks. + */ + avd->homogeneous_cpus = (id_bitsmash > 0); + return 0; +} + +arch_initcall_sync(init_hwprobe_vdso_data); + SYSCALL_DEFINE5(riscv_hwprobe, uintptr_t, pairs, uintptr_t, pair_count, uintptr_t, cpu_count, uintptr_t, cpus, uintptr_t, flags) { diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makefile index 06e6b27f3bcc..107fd2162cdf 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -12,6 +12,8 @@ vdso-syms += vgettimeofday endif vdso-syms += getcpu vdso-syms += flush_icache +vdso-syms += hwprobe +vdso-syms += sys_hwprobe # Files to link into the vdso obj-vdso = $(patsubst %, %.o, $(vdso-syms)) note.o diff --git a/arch/riscv/kernel/vdso/hwprobe.c b/arch/riscv/kernel/vdso/hwprobe.c new file mode 100644 index 000000000000..1792d6c30c97 --- /dev/null +++ b/arch/riscv/kernel/vdso/hwprobe.c @@ -0,0 +1,47 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2023 Rivos, Inc + */ + +#include +#include +#include + +extern int riscv_hwprobe(struct riscv_hwprobe *pairs, long pair_count, + long cpu_count, unsigned long *cpus, + unsigned long flags); + +int __vdso_riscv_hwprobe(struct riscv_hwprobe *pairs, long pair_count, + long cpu_count, unsigned long *cpus, + unsigned long flags) +{ + const struct vdso_data *vd = __arch_get_vdso_data(); + const struct arch_vdso_data *avd = &vd->arch_data; + bool all_cpus = !cpu_count && !cpus; + struct riscv_hwprobe *p = pairs; + struct riscv_hwprobe *end = pairs + pair_count; + + /* + * Defer to the syscall for exotic requests. The vdso has answers + * stashed away only for the "all cpus" case. If all CPUs are + * homogeneous, then this function can handle requests for arbitrary + * masks. + */ + if ((flags != 0) || (!all_cpus && !avd->homogeneous_cpus)) + return riscv_hwprobe(pairs, pair_count, cpu_count, cpus, flags); + + /* This is something we can handle, fill out the pairs. */ + while (p < end) { + if (p->key <= RISCV_HWPROBE_MAX_KEY) { + p->value = avd->all_cpu_hwprobe_values[p->key]; + + } else { + p->key = -1; + p->value = 0; + } + + p++; + } + + return 0; +} diff --git a/arch/riscv/kernel/vdso/sys_hwprobe.S b/arch/riscv/kernel/vdso/sys_hwprobe.S new file mode 100644 index 000000000000..2511c8ba8f86 --- /dev/null +++ b/arch/riscv/kernel/vdso/sys_hwprobe.S @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2022 Rivos, Inc */ + +#include +#include + +.text +ENTRY(riscv_hwprobe) + .cfi_startproc + li a7, __NR_riscv_hwprobe + ecall + ret + + .cfi_endproc +ENDPROC(riscv_hwprobe) diff --git a/arch/riscv/kernel/vdso/vdso.lds.S b/arch/riscv/kernel/vdso/vdso.lds.S index 150b1a572e61..dfe63fb71041 100644 --- a/arch/riscv/kernel/vdso/vdso.lds.S +++ b/arch/riscv/kernel/vdso/vdso.lds.S @@ -75,6 +75,7 @@ VERSION #endif __vdso_getcpu; __vdso_flush_icache; + __vdso_riscv_hwprobe; local: *; }; }