From patchwork Sat Nov 19 12:09:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 23262 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp680346wrr; Sat, 19 Nov 2022 04:13:06 -0800 (PST) X-Google-Smtp-Source: AA0mqf7fwe1HZ1FZ02LPci9lyjwUzUyjSmY8YxYE+zQZYDIZ532gri8kLT2mJxxq6uehgqHCINo+ X-Received: by 2002:a17:902:f7cf:b0:188:d452:2ca3 with SMTP id h15-20020a170902f7cf00b00188d4522ca3mr3794568plw.35.1668859985809; Sat, 19 Nov 2022 04:13:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668859985; cv=none; d=google.com; s=arc-20160816; b=Nt5Q1OBw/pI3OT5SGcbBQU1+m7oW8ljqZubD/Wmp4tiggi9BVcy+bD65zbP33ytqc1 sxnJZqE8EEWCv9rREGDNcsipLzsbcArzpa+c7WYmBcvEOaMqn5YZGIU/Pm4FpFHeJKr+ rsrB2JvdS8DUO9gcSrxWBtYJTjOwCQ/8lh7clwCHqJiAnmLU+Sfc3uhsLzI7BxbVVdmx 7KRIteL63xITW4vSRuUyN2wjMbFZR+I4uVvAm5D+pmH40MQYSMu4A+HNWsUQxGrUCWQD fWOJTxAy1cvAXFUOoSwhHGg+vRMLcMSh/t7VB620YRgDvRZnGa9poO09lgEreJgSnasN JTXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QX2ONF7SqIyngPkDRzIiroDZkPeJ27gDqN4LMfbgmZw=; b=K5ZRd3Iq1vUZOEhRf42fEU3Hwq1zSKVXIN9nvwvPTBr9RbJu9JeaktWvTHdL6QVbEe n6O0G2V/xuK47G2oJAO/Ail+CvZJZO6kDB7sU9GhUuOSy1L7DkI448ZTaRILPHtkcsCK qO/PdLSRq2BAC5zzYL9t//noY/7BjRTmcl8iSVCMR4E48nL//DBPhzxhoYIZ45QhzRAu ryS7rxkNDj/G0IFIDYK6TKbY0+il5g9xHEe5xO7y8t4vP8/MTBciGaoM/BVi+mxHYDgH 0j2qJM++aMUvNFdrWCchYhdpxE27721u3Y6TCI1QyZ/8Mtz7/qXIKo4JMWLBMeYyFKI4 vmhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=kAHikCwV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z18-20020a63e552000000b00476a08c5d68si6377410pgj.822.2022.11.19.04.12.53; Sat, 19 Nov 2022 04:13:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=kAHikCwV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233016AbiKSMJ4 (ORCPT + 99 others); Sat, 19 Nov 2022 07:09:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231683AbiKSMJq (ORCPT ); Sat, 19 Nov 2022 07:09:46 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F186075D9E; Sat, 19 Nov 2022 04:09:44 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8DAEB60B72; Sat, 19 Nov 2022 12:09:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E80EDC433D6; Sat, 19 Nov 2022 12:09:42 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="kAHikCwV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668859780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QX2ONF7SqIyngPkDRzIiroDZkPeJ27gDqN4LMfbgmZw=; b=kAHikCwVnj7KvsAVAr/EkYTZn9T4DLGzFikojNlPDuklgz8/B9JPl2iCFk0GZW+5Yx5p9/ 6By1IsmDecvZqvBTTz+7hRuOpy8JW0jf8Xijy+LrkV/ki5cyFCKWs8iC807bFH2R4uIGIl 9YunD5rj1UR0Y13OIUmR7eRslWSOEpo= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id cc601153 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sat, 19 Nov 2022 12:09:40 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v5 1/3] random: add vgetrandom_alloc() syscall Date: Sat, 19 Nov 2022 13:09:27 +0100 Message-Id: <20221119120929.2963813-2-Jason@zx2c4.com> In-Reply-To: <20221119120929.2963813-1-Jason@zx2c4.com> References: <20221119120929.2963813-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749926528183053581?= X-GMAIL-MSGID: =?utf-8?q?1749926528183053581?= The vDSO getrandom() works over an opaque per-thread state of an unexported size, which must be marked as MADV_WIPEONFORK and be mlock()'d for proper operation. Over time, the nuances of these allocations may change or grow or even differ based on architectural features. The syscall has the signature: void *vgetrandom_alloc([inout] size_t *num, [out] size_t *size_per_each, unsigned int flags); This takes the desired number of opaque states in `num`, and returns a pointer to an array of opaque states, the number actually allocated back in `num`, and the size in bytes of each one in `size_per_each`, enabling a libc to slice up the returned array into a state per each thread. (The `flags` argument is always zero for now.) Libc is expected to allocate a chunk of these on first use, and then dole them out to threads as they're created, allocating more when needed. The following commit shows an example of this, being used in conjunction with the getrandom() vDSO function. We very intentionally do *not* leave state allocation for vDSO getrandom() up to userspace itself, but rather provide this new syscall for such allocations. vDSO getrandom() must not store its state in just any old memory address, but rather just ones that the kernel specially allocates for it, leaving the particularities of those allocations up to the kernel. Signed-off-by: Jason A. Donenfeld --- MAINTAINERS | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/unistd.h | 1 + drivers/char/random.c | 59 +++++++++++++++++++++++++ include/uapi/asm-generic/unistd.h | 7 ++- kernel/sys_ni.c | 3 ++ lib/vdso/getrandom.h | 23 ++++++++++ scripts/checksyscalls.sh | 4 ++ tools/include/uapi/asm-generic/unistd.h | 7 ++- 10 files changed, 105 insertions(+), 2 deletions(-) create mode 100644 lib/vdso/getrandom.h diff --git a/MAINTAINERS b/MAINTAINERS index 256f03904987..843dd6a49538 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17287,6 +17287,7 @@ T: git https://git.kernel.org/pub/scm/linux/kernel/git/crng/random.git S: Maintained F: drivers/char/random.c F: drivers/virt/vmgenid.c +F: lib/vdso/getrandom.h RAPIDIO SUBSYSTEM M: Matt Porter diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl index 320480a8db4f..ea0fbc2ded5e 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -455,3 +455,4 @@ 448 i386 process_mrelease sys_process_mrelease 449 i386 futex_waitv sys_futex_waitv 450 i386 set_mempolicy_home_node sys_set_mempolicy_home_node +451 i386 vgetrandom_alloc sys_vgetrandom_alloc diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index c84d12608cd2..0186f173f0e8 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 common vgetrandom_alloc sys_vgetrandom_alloc # # Due to a historical design error, certain syscalls are numbered differently diff --git a/arch/x86/include/asm/unistd.h b/arch/x86/include/asm/unistd.h index 761173ccc33c..f9673293cd95 100644 --- a/arch/x86/include/asm/unistd.h +++ b/arch/x86/include/asm/unistd.h @@ -57,5 +57,6 @@ # define __ARCH_WANT_SYS_VFORK # define __ARCH_WANT_SYS_CLONE # define __ARCH_WANT_SYS_CLONE3 +# define __ARCH_WANT_VGETRANDOM_ALLOC #endif /* _ASM_X86_UNISTD_H */ diff --git a/drivers/char/random.c b/drivers/char/random.c index 65ee69896967..ab6e02efa432 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -8,6 +8,7 @@ * into roughly six sections, each with a section header: * * - Initialization and readiness waiting. + * - vDSO support helpers. * - Fast key erasure RNG, the "crng". * - Entropy accumulation and extraction routines. * - Entropy collection routines. @@ -39,6 +40,7 @@ #include #include #include +#include #include #include #include @@ -59,6 +61,7 @@ #include #include #include +#include "../../lib/vdso/getrandom.h" /********************************************************************* * @@ -146,6 +149,62 @@ EXPORT_SYMBOL(wait_for_random_bytes); __func__, (void *)_RET_IP_, crng_init) + +/******************************************************************** + * + * vDSO support helpers. + * + * The actual vDSO function is defined over in lib/vdso/getrandom.c, + * but this section contains the kernel-mode helpers to support that. + * + ********************************************************************/ + +#ifdef __ARCH_WANT_VGETRANDOM_ALLOC +/* + * The vgetrandom() function in userspace requires an opaque state, which this + * function provides to userspace. The result is that it maps a certain + * number of special pages into the calling process and returns the address. + */ +SYSCALL_DEFINE3(vgetrandom_alloc, unsigned long __user *, num, + unsigned long __user *, size_per_each, unsigned int, flags) +{ + unsigned long alloc_size; + unsigned long num_states; + unsigned long pages_addr; + int ret; + + if (flags) + return -EINVAL; + + if (get_user(num_states, num)) + return -EFAULT; + + alloc_size = size_mul(num_states, sizeof(struct vgetrandom_state)); + if (alloc_size == SIZE_MAX) + return -EOVERFLOW; + alloc_size = roundup(alloc_size, PAGE_SIZE); + + if (put_user(alloc_size / sizeof(struct vgetrandom_state), num) || + put_user(sizeof(struct vgetrandom_state), size_per_each)) + return -EFAULT; + + pages_addr = vm_mmap(NULL, 0, alloc_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_LOCKED, 0); + if (IS_ERR_VALUE(pages_addr)) + return pages_addr; + + ret = do_madvise(current->mm, pages_addr, alloc_size, MADV_WIPEONFORK); + if (ret < 0) + goto err_unmap; + + return pages_addr; + +err_unmap: + vm_munmap(pages_addr, alloc_size); + return ret; +} +#endif + /********************************************************************* * * Fast key erasure RNG, the "crng". diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 45fa180cc56a..77b6debe7e18 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -886,8 +886,13 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) #define __NR_set_mempolicy_home_node 450 __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) +#ifdef __ARCH_WANT_VGETRANDOM_ALLOC +#define __NR_vgetrandom_alloc 451 +__SYSCALL(__NR_vgetrandom_alloc, sys_vgetrandom_alloc) +#endif + #undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 860b2dcf3ac4..f28196cb919b 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -360,6 +360,9 @@ COND_SYSCALL(pkey_free); /* memfd_secret */ COND_SYSCALL(memfd_secret); +/* random */ +COND_SYSCALL(vgetrandom_alloc); + /* * Architecture specific weak syscall entries. */ diff --git a/lib/vdso/getrandom.h b/lib/vdso/getrandom.h new file mode 100644 index 000000000000..85c2f62c0f5f --- /dev/null +++ b/lib/vdso/getrandom.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ + +#ifndef _VDSO_LIB_GETRANDOM_H +#define _VDSO_LIB_GETRANDOM_H + +#include + +struct vgetrandom_state { + unsigned long generation; + union { + struct { + u8 key[CHACHA_KEY_SIZE]; + u8 batch[CHACHA_BLOCK_SIZE * 3 / 2]; + }; + u8 key_batch[CHACHA_BLOCK_SIZE * 2]; + }; + u8 pos; +}; + +#endif /* _VDSO_LIB_GETRANDOM_H */ diff --git a/scripts/checksyscalls.sh b/scripts/checksyscalls.sh index f33e61aca93d..7f7928c6487f 100755 --- a/scripts/checksyscalls.sh +++ b/scripts/checksyscalls.sh @@ -44,6 +44,10 @@ cat << EOF #define __IGNORE_memfd_secret #endif +#ifndef __ARCH_WANT_VGETRANDOM_ALLOC +#define __IGNORE_vgetrandom_alloc +#endif + /* Missing flags argument */ #define __IGNORE_renameat /* renameat2 */ diff --git a/tools/include/uapi/asm-generic/unistd.h b/tools/include/uapi/asm-generic/unistd.h index 45fa180cc56a..77b6debe7e18 100644 --- a/tools/include/uapi/asm-generic/unistd.h +++ b/tools/include/uapi/asm-generic/unistd.h @@ -886,8 +886,13 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) #define __NR_set_mempolicy_home_node 450 __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) +#ifdef __ARCH_WANT_VGETRANDOM_ALLOC +#define __NR_vgetrandom_alloc 451 +__SYSCALL(__NR_vgetrandom_alloc, sys_vgetrandom_alloc) +#endif + #undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452 /* * 32 bit systems traditionally used different From patchwork Sat Nov 19 12:09:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 23261 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp680334wrr; Sat, 19 Nov 2022 04:13:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf6JuhAOuiRDNOktaiExGQGkYupwgNWuoshZrJ+UXiPImclY5D+umssNtPghdqb/D4rhtOnM X-Received: by 2002:a17:902:ed93:b0:188:a281:1b16 with SMTP id e19-20020a170902ed9300b00188a2811b16mr3787241plj.103.1668859984073; Sat, 19 Nov 2022 04:13:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668859984; cv=none; d=google.com; s=arc-20160816; b=Hx/eoR9tbzkuEQ1Cf1ht97GXkI6pfF6BNyeAm/T80K3XKsZa7NW9IH+/zvCkpPBWhJ 36qzdsua/sY/6tjILXhpu3vFw/ZWthEO2YFNxNCY8g5W4g8Zcw3CKdctDXNcaDfUrjDa EuQl0b4ECtaUD2yWG27Rr9YCdRJypzYq6mF1SnqtZD8mwkVQ9ymQjtSWYv/NiO3UgDA5 +LW4buxfmoVOW/jid1F/mR90XYEYnzzs5rzaQC/DuDgSrBmkb8gE3AJjCi3dPn3o17+R NTtsCjEQSlXwm4KUncp6Egu17rSZJDGzbrWnOebQEn7QDDu5G+4HkNLlpR7DODWGC25C e6Yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lphMbEIVa3jmpq1teHjf8K2TbEwzWR3uJsGcb+bX3WI=; b=EsXJYKqE4+u0LIqac3gIDqZwBZjluNtQuNfW0zCfogvTd+47BphJWWgEA49XL6NS6L vJzk7s26XyVoUphRoIsW1ZxDN1ngihHnKedhVK9g9ky/LtODJR4lHJb44W2Nnmbo8tR3 s4hFbRqfX4IA+nrgllEnMLr6iDyKaX2UhDGK+dnQ5KMU2y99bjSteRGjkN6czrQN/2Sx DTzSC80slHEBUrCE+DpU7ImX8SrOJrh0Zd9czsN0mmymWnknHNk3fZCMENNSqcSsXKAR HqmyMk8i4LB4hOyFrB+mRlNZ6POEY0qiznu2ctof2Lh+7SscNZJAcurswC2AxCOA1B37 kPYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=fcT5H43A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l6-20020a170903244600b00186b3cb49basi7540373pls.202.2022.11.19.04.12.51; Sat, 19 Nov 2022 04:13:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=fcT5H43A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233217AbiKSMJ6 (ORCPT + 99 others); Sat, 19 Nov 2022 07:09:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232969AbiKSMJy (ORCPT ); Sat, 19 Nov 2022 07:09:54 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA157776D0; Sat, 19 Nov 2022 04:09:48 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 79DBFB80AB4; Sat, 19 Nov 2022 12:09:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BB46C433C1; Sat, 19 Nov 2022 12:09:45 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="fcT5H43A" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668859784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lphMbEIVa3jmpq1teHjf8K2TbEwzWR3uJsGcb+bX3WI=; b=fcT5H43AArKQRgeCTtHarL/xRX+XfcZ28557vAe4ALQ0UbYVrccvjTbMcrICFaLeqNYj6x 9xKp+7cPQpgqTExfECc97L+un2nB98w8BrJjrZW0EOvQRuuNHRYU3As1HZZGmmNskM05GU a7MIBTFeXsl8y3OcQylbH38/1/TbXrg= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 69deb66a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sat, 19 Nov 2022 12:09:44 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v5 2/3] random: introduce generic vDSO getrandom() implementation Date: Sat, 19 Nov 2022 13:09:28 +0100 Message-Id: <20221119120929.2963813-3-Jason@zx2c4.com> In-Reply-To: <20221119120929.2963813-1-Jason@zx2c4.com> References: <20221119120929.2963813-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS,T_FILL_THIS_FORM_SHORT autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749926526567848860?= X-GMAIL-MSGID: =?utf-8?q?1749926526567848860?= Provide a generic C vDSO getrandom() implementation, which operates on an opaque state returned by vgetrandom_alloc() and produces random bytes the same way as getrandom(). This has a the API signature: ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); The return value and the first 3 arguments are the same as ordinary getrandom(), while the last argument is a pointer to the opaque allocated state. Were all four arguments passed to the getrandom() syscall, nothing different would happen, and the functions would have the exact same behavior. The actual vDSO RNG algorithm implemented is the same one implemented by drivers/char/random.c, using the same fast-erasure techniques as that. Should the in-kernel implementation change, so too will the vDSO one. Initially, the state is keyless, and so the first call makes a getrandom() syscall to generate that key, and then uses it for subsequent calls. By keeping track of a generation counter, it knows when its key is invalidated and it should fetch a new one using the syscall. Later, more than just a generation counter might be used. Since MADV_WIPEONFORK is set on the opaque state, the key and related state is wiped during a fork(), so secrets don't roll over into new processes, and the same state doesn't accidentally generate the same random stream. The generation counter, as well, is always >0, so that the 0 counter is a useful indication of a fork() or otherwise uninitialized state. If the kernel RNG is not yet initialized, then the vDSO always calls the syscall, because that behavior cannot be emulated in userspace, but fortunately that state is short lived and only during early boot. If it has been initialized, then there is no need to inspect the `flags` argument, because the behavior does not change post-initialization regardless of the `flags` value. Together with the previous commit that introduces vgetrandom_alloc(), this functionality is intended to be integrated into libc's thread management. As an illustrative example, the following code might be used to do the same outside of libc. All of the static functions are to be considered implementation private, including the vgetrandom_alloc() syscall wrapper, which generally shouldn't be exposed outside of libc, with the non-static vgetrandom() function at the end being the exported interface. The various pthread-isms are expected to be elided into libc internals. This per-thread allocation scheme is very naive and does not shrink; other implementations may choose to be more complex. static void *vgetrandom_alloc(size_t *num, size_t *size_per_each, unsigned int flags) { unsigned long ret = syscall(__NR_vgetrandom_alloc, num, size_per_each, flags); return ret > -4096UL ? NULL : (void *)ret; } static struct { pthread_mutex_t lock; void **states; size_t len, cap; } grnd_allocator = { .lock = PTHREAD_MUTEX_INITIALIZER }; static void *vgetrandom_get_state(void) { void *state = NULL; pthread_mutex_lock(&grnd_allocator.lock); if (!grnd_allocator.len) { size_t new_cap, size_per_each, num = 16; /* Just a hint. */ void *new_block = vgetrandom_alloc(&num, &size_per_each, 0), *new_states; if (!new_block) goto out; new_cap = grnd_allocator.cap + num; new_states = reallocarray(grnd_allocator.states, new_cap, sizeof(*grnd_allocator.states)); if (!new_states) { munmap(new_block, num * size_per_each); goto out; } grnd_allocator.cap = new_cap; grnd_allocator.states = new_states; for (size_t i = 0; i < num; ++i) { grnd_allocator.states[i] = new_block; new_block += size_per_each; } grnd_allocator.len = num; } state = grnd_allocator.states[--grnd_allocator.len]; out: pthread_mutex_unlock(&grnd_allocator.lock); return state; } static void vgetrandom_put_state(void *state) { if (!state) return; pthread_mutex_lock(&grnd_allocator.lock); grnd_allocator.states[grnd_allocator.len++] = state; pthread_mutex_unlock(&grnd_allocator.lock); } static struct { ssize_t(*fn)(void *buf, size_t len, unsigned long flags, void *state); pthread_key_t key; pthread_once_t initialized; } grnd_ctx = { .initialized = PTHREAD_ONCE_INIT }; static void vgetrandom_init(void) { if (pthread_key_create(&grnd_ctx.key, vgetrandom_put_state) != 0) return; grnd_ctx.fn = __vdsosym("LINUX_2.6", "__vdso_getrandom"); } ssize_t vgetrandom(void *buf, size_t len, unsigned long flags) { void *state; pthread_once(&grnd_ctx.initialized, vgetrandom_init); if (!grnd_ctx.fn) return getrandom(buf, len, flags); state = pthread_getspecific(grnd_ctx.key); if (!state) { state = vgetrandom_get_state(); if (pthread_setspecific(grnd_ctx.key, state) != 0) { vgetrandom_put_state(state); state = NULL; } if (!state) return getrandom(buf, len, flags); } return grnd_ctx.fn(buf, len, flags, state); } Signed-off-by: Jason A. Donenfeld --- MAINTAINERS | 1 + drivers/char/random.c | 5 ++ include/vdso/datapage.h | 6 +++ lib/crypto/chacha.c | 4 ++ lib/vdso/Kconfig | 5 ++ lib/vdso/getrandom.c | 115 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 136 insertions(+) create mode 100644 lib/vdso/getrandom.c diff --git a/MAINTAINERS b/MAINTAINERS index 843dd6a49538..e0aa33f54c57 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17287,6 +17287,7 @@ T: git https://git.kernel.org/pub/scm/linux/kernel/git/crng/random.git S: Maintained F: drivers/char/random.c F: drivers/virt/vmgenid.c +F: lib/vdso/getrandom.c F: lib/vdso/getrandom.h RAPIDIO SUBSYSTEM diff --git a/drivers/char/random.c b/drivers/char/random.c index ab6e02efa432..7dfdbf424c92 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -61,6 +61,7 @@ #include #include #include +#include #include "../../lib/vdso/getrandom.h" /********************************************************************* @@ -307,6 +308,8 @@ static void crng_reseed(struct work_struct *work) if (next_gen == ULONG_MAX) ++next_gen; WRITE_ONCE(base_crng.generation, next_gen); + if (IS_ENABLED(CONFIG_HAVE_VDSO_GETRANDOM)) + smp_store_release(&_vdso_rng_data.generation, next_gen + 1); if (!static_branch_likely(&crng_is_ready)) crng_init = CRNG_READY; spin_unlock_irqrestore(&base_crng.lock, flags); @@ -756,6 +759,8 @@ static void __cold _credit_init_bits(size_t bits) crng_reseed(NULL); /* Sets crng_init to CRNG_READY under base_crng.lock. */ if (static_key_initialized) execute_in_process_context(crng_set_ready, &set_ready); + if (IS_ENABLED(CONFIG_HAVE_VDSO_GETRANDOM)) + smp_store_release(&_vdso_rng_data.is_ready, true); wake_up_interruptible(&crng_init_wait); kill_fasync(&fasync, SIGIO, POLL_IN); pr_notice("crng init done\n"); diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 73eb622e7663..cbacfd923a5c 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -109,6 +109,11 @@ struct vdso_data { struct arch_vdso_data arch_data; }; +struct vdso_rng_data { + unsigned long generation; + bool is_ready; +}; + /* * We use the hidden visibility to prevent the compiler from generating a GOT * relocation. Not only is going through a GOT useless (the entry couldn't and @@ -120,6 +125,7 @@ struct vdso_data { */ extern struct vdso_data _vdso_data[CS_BASES] __attribute__((visibility("hidden"))); extern struct vdso_data _timens_data[CS_BASES] __attribute__((visibility("hidden"))); +extern struct vdso_rng_data _vdso_rng_data __attribute__((visibility("hidden"))); /* * The generic vDSO implementation requires that gettimeofday.h diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c index b748fd3d256e..944991bb36c7 100644 --- a/lib/crypto/chacha.c +++ b/lib/crypto/chacha.c @@ -17,8 +17,10 @@ static void chacha_permute(u32 *x, int nrounds) { int i; +#ifndef CHACHA_FOR_VDSO_INCLUDE /* whitelist the allowed round counts */ WARN_ON_ONCE(nrounds != 20 && nrounds != 12); +#endif for (i = 0; i < nrounds; i += 2) { x[0] += x[4]; x[12] = rol32(x[12] ^ x[0], 16); @@ -87,6 +89,7 @@ void chacha_block_generic(u32 *state, u8 *stream, int nrounds) state[12]++; } +#ifndef CHACHA_FOR_VDSO_INCLUDE EXPORT_SYMBOL(chacha_block_generic); /** @@ -112,3 +115,4 @@ void hchacha_block_generic(const u32 *state, u32 *stream, int nrounds) memcpy(&stream[4], &x[12], 16); } EXPORT_SYMBOL(hchacha_block_generic); +#endif diff --git a/lib/vdso/Kconfig b/lib/vdso/Kconfig index d883ac299508..c35fac664574 100644 --- a/lib/vdso/Kconfig +++ b/lib/vdso/Kconfig @@ -30,4 +30,9 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO +config HAVE_VDSO_GETRANDOM + bool + help + Selected by architectures that support vDSO getrandom(). + endif diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c new file mode 100644 index 000000000000..8bef1e92a79d --- /dev/null +++ b/lib/vdso/getrandom.c @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ + +#include +#include +#include +#include +#include +#include +#include "getrandom.h" + +#undef memcpy +#define memcpy(d,s,l) __builtin_memcpy(d,s,l) +#undef memset +#define memset(d,c,l) __builtin_memset(d,c,l) + +#define CHACHA_FOR_VDSO_INCLUDE +#include "../crypto/chacha.c" + +static void memcpy_and_zero(void *dst, void *src, size_t len) +{ +#define CASCADE(type) \ + while (len >= sizeof(type)) { \ + *(type *)dst = *(type *)src; \ + *(type *)src = 0; \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS +#if BITS_PER_LONG == 64 + CASCADE(u64); +#endif + CASCADE(u32); + CASCADE(u16); +#endif + CASCADE(u8); +#undef CASCADE +} + +static __always_inline ssize_t +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, + unsigned int flags, void *opaque_state) +{ + ssize_t ret = min_t(size_t, MAX_RW_COUNT, len); + struct vgetrandom_state *state = opaque_state; + u32 chacha_state[CHACHA_STATE_WORDS]; + unsigned long current_generation; + size_t batch_len; + + if (unlikely(!rng_info->is_ready)) + return getrandom_syscall(buffer, len, flags); + + if (unlikely(!len)) + return 0; + +retry_generation: + current_generation = READ_ONCE(rng_info->generation); + if (unlikely(state->generation != current_generation)) { + if (getrandom_syscall(state->key, sizeof(state->key), 0) != sizeof(state->key)) + return getrandom_syscall(buffer, len, flags); + WRITE_ONCE(state->generation, current_generation); + state->pos = sizeof(state->batch); + } + + len = ret; +more_batch: + batch_len = min_t(size_t, sizeof(state->batch) - state->pos, len); + if (batch_len) { + memcpy_and_zero(buffer, state->batch + state->pos, batch_len); + state->pos += batch_len; + buffer += batch_len; + len -= batch_len; + } + if (!len) { + /* + * Since rng_info->generation will never be 0, we re-read state->generation, + * rather than using the local current_generation variable, to learn whether + * we forked. + */ + if (unlikely(READ_ONCE(state->generation) != READ_ONCE(rng_info->generation))) { + buffer -= ret; + goto retry_generation; + } + return ret; + } + + chacha_init_consts(chacha_state); + memcpy(&chacha_state[4], state->key, CHACHA_KEY_SIZE); + memset(&chacha_state[12], 0, sizeof(u32) * 4); + + while (len >= CHACHA_BLOCK_SIZE) { + chacha20_block(chacha_state, buffer); + if (unlikely(chacha_state[12] == 0)) + ++chacha_state[13]; + buffer += CHACHA_BLOCK_SIZE; + len -= CHACHA_BLOCK_SIZE; + } + + chacha20_block(chacha_state, state->key_batch); + if (unlikely(chacha_state[12] == 0)) + ++chacha_state[13]; + chacha20_block(chacha_state, state->key_batch + CHACHA_BLOCK_SIZE); + state->pos = 0; + memzero_explicit(chacha_state, sizeof(chacha_state)); + goto more_batch; +} + +static __always_inline ssize_t +__cvdso_getrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state) +{ + return __cvdso_getrandom_data(__arch_get_vdso_rng_data(), buffer, len, flags, opaque_state); +} From patchwork Sat Nov 19 12:09:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 23264 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp681552wrr; Sat, 19 Nov 2022 04:15:36 -0800 (PST) X-Google-Smtp-Source: AA0mqf4IHZIj7Lsh8N0FDPmLvp2H1OBFJ5rDMPeZFVU93Crnb+WxYF5PFnS28iHKs8cFe7ZMWNCz X-Received: by 2002:a17:903:2013:b0:188:f7d0:3952 with SMTP id s19-20020a170903201300b00188f7d03952mr3548585pla.164.1668860136580; Sat, 19 Nov 2022 04:15:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668860136; cv=none; d=google.com; s=arc-20160816; b=l/d0YJlNlsnH1qTHOFLBU3h660w0lw0O+snp6YYch/yVnccZkt8Fri66wbDqz/ZBE4 hNRxX6I0oIhUxBUDT9qLcOzii8RlsN+qToJ0/Kngfzub/ShIDUm3DMXS7YD8g+B05xf2 h6ZdY5cC/JZuasQcT+Ko37+WZ7WBWXFNlVZvDYIBkhxZQzdBX/HD+472G/FNPExe4Cbo ukF/25unFQ1kSz0k4F2OsxAPxfsGNy7Nau0VKpS8EIRhyWAqHIt54uZsaQGOzV4FIq7n 1+/+kaM3WP1zY7J24P158f/l0/9XCvVlozpgLF8m5hbb9eyuvYs0Uod6RvQKp4YKNnFI tYOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GyKOGkofkPCvMPD0/UwJzzsEw9ZOaxhI19IJoJsqMz4=; b=FiYTFK+nB72wvTf0DH0lAqKVftyTbH4FgA1hsT9wvdmxoY9gGLz5KrvJIpsRtsdoia NtGBzljFT8IUn7Hkera3qCC3mw7ktfs2uZOK9KakijJhkCq+5iisj6ASWfFx5MSXQcTI PxlehHklIlkPy3SVWWpTeolsVM6CBzJO058BzTPoflX0Z1UVWyZ+yNiNnr/OWmQv7WgH eh+uPhL5+p18ey41XW60rzfa6U5JDZcDN9hrLrb1r/Os2yXqIwH3iG5GAIkf+Ab8G+db HVNYQ8o2toxjWBD2GLKxhdHn2R3Cxd0LumdZwK1PPqzoqN42IPpgw0wFi1C989DWDhtQ 6/KQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=bnbraqRk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q15-20020a170902eb8f00b001786f5de8f2si6198470plg.95.2022.11.19.04.15.22; Sat, 19 Nov 2022 04:15:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=bnbraqRk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233576AbiKSMKC (ORCPT + 99 others); Sat, 19 Nov 2022 07:10:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233066AbiKSMJy (ORCPT ); Sat, 19 Nov 2022 07:09:54 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC22C75DAE; Sat, 19 Nov 2022 04:09:50 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A73DAB80AB8; Sat, 19 Nov 2022 12:09:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6B5DDC43152; Sat, 19 Nov 2022 12:09:47 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="bnbraqRk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668859786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GyKOGkofkPCvMPD0/UwJzzsEw9ZOaxhI19IJoJsqMz4=; b=bnbraqRkItNbvFNMs0FemNZhrh46NikwIlxJrbXkdmgDihDg7YTuOYPHxTg9iCVC5tegwW KTHOze/FEslAi3HQFTm56CNdOsIQyq1LrKVKAS2z17l5cLcptlfWo9nrwfou5h+k6nmjJO ofk/he6/o+fGvkBaSojMLm/UwBRocqk= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 48ac3dcc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sat, 19 Nov 2022 12:09:46 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v5 3/3] x86: vdso: Wire up getrandom() vDSO implementation Date: Sat, 19 Nov 2022 13:09:29 +0100 Message-Id: <20221119120929.2963813-4-Jason@zx2c4.com> In-Reply-To: <20221119120929.2963813-1-Jason@zx2c4.com> References: <20221119120929.2963813-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749926686738506024?= X-GMAIL-MSGID: =?utf-8?q?1749926686738506024?= Hook up the generic vDSO implementation to the x86 vDSO data page. Since the existing vDSO infrastructure is heavily based on the timekeeping functionality, which works over arrays of bases, a new macro is introduced for vvars that are not arrays. Signed-off-by: Jason A. Donenfeld --- arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 3 ++- arch/x86/entry/vdso/vdso.lds.S | 2 ++ arch/x86/entry/vdso/vgetrandom.c | 16 ++++++++++++ arch/x86/include/asm/vdso/getrandom.h | 37 +++++++++++++++++++++++++++ arch/x86/include/asm/vdso/vsyscall.h | 2 ++ arch/x86/include/asm/vvar.h | 16 ++++++++++++ 7 files changed, 76 insertions(+), 1 deletion(-) create mode 100644 arch/x86/entry/vdso/vgetrandom.c create mode 100644 arch/x86/include/asm/vdso/getrandom.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 67745ceab0db..210318da7505 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -269,6 +269,7 @@ config X86 select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_USER_RETURN_NOTIFIER select HAVE_GENERIC_VDSO + select HAVE_VDSO_GETRANDOM select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING select NEED_PER_CPU_EMBED_FIRST_CHUNK diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index 3e88b9df8c8f..adc3792dbbac 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -27,7 +27,7 @@ VDSO32-$(CONFIG_X86_32) := y VDSO32-$(CONFIG_IA32_EMULATION) := y # files to link into the vdso -vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o +vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o vgetrandom.o vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) += vsgx.o @@ -104,6 +104,7 @@ CFLAGS_REMOVE_vclock_gettime.o = -pg CFLAGS_REMOVE_vdso32/vclock_gettime.o = -pg CFLAGS_REMOVE_vgetcpu.o = -pg CFLAGS_REMOVE_vsgx.o = -pg +CFLAGS_REMOVE_vgetrandom.o = -pg # # X32 processes use x32 vDSO to access 64bit kernel data. diff --git a/arch/x86/entry/vdso/vdso.lds.S b/arch/x86/entry/vdso/vdso.lds.S index 4bf48462fca7..1919cc39277e 100644 --- a/arch/x86/entry/vdso/vdso.lds.S +++ b/arch/x86/entry/vdso/vdso.lds.S @@ -28,6 +28,8 @@ VERSION { clock_getres; __vdso_clock_getres; __vdso_sgx_enter_enclave; + getrandom; + __vdso_getrandom; local: *; }; } diff --git a/arch/x86/entry/vdso/vgetrandom.c b/arch/x86/entry/vdso/vgetrandom.c new file mode 100644 index 000000000000..0a0c0ad93cd0 --- /dev/null +++ b/arch/x86/entry/vdso/vgetrandom.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#include +#include + +#include "../../../../lib/vdso/getrandom.c" + +ssize_t __vdso_getrandom(void *buffer, size_t len, unsigned int flags, void *state) +{ + return __cvdso_getrandom(buffer, len, flags, state); +} + +ssize_t getrandom(void *, size_t, unsigned int, void *) + __attribute__((weak, alias("__vdso_getrandom"))); diff --git a/arch/x86/include/asm/vdso/getrandom.h b/arch/x86/include/asm/vdso/getrandom.h new file mode 100644 index 000000000000..c414043e975d --- /dev/null +++ b/arch/x86/include/asm/vdso/getrandom.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#ifndef __ASM_VDSO_GETRANDOM_H +#define __ASM_VDSO_GETRANDOM_H + +#ifndef __ASSEMBLY__ + +#include +#include + +static __always_inline ssize_t +getrandom_syscall(void *buffer, size_t len, unsigned int flags) +{ + long ret; + + asm ("syscall" : "=a" (ret) : + "0" (__NR_getrandom), "D" (buffer), "S" (len), "d" (flags) : + "rcx", "r11", "memory"); + + return ret; +} + +#define __vdso_rng_data (VVAR(_vdso_rng_data)) + +static __always_inline const struct vdso_rng_data *__arch_get_vdso_rng_data(void) +{ + if (__vdso_data->clock_mode == VDSO_CLOCKMODE_TIMENS) + return (void *)&__vdso_rng_data + + ((void *)&__timens_vdso_data - (void *)&__vdso_data); + return &__vdso_rng_data; +} + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_VDSO_GETRANDOM_H */ diff --git a/arch/x86/include/asm/vdso/vsyscall.h b/arch/x86/include/asm/vdso/vsyscall.h index be199a9b2676..71c56586a22f 100644 --- a/arch/x86/include/asm/vdso/vsyscall.h +++ b/arch/x86/include/asm/vdso/vsyscall.h @@ -11,6 +11,8 @@ #include DEFINE_VVAR(struct vdso_data, _vdso_data); +DEFINE_VVAR_SINGLE(struct vdso_rng_data, _vdso_rng_data); + /* * Update the vDSO data page to keep in sync with kernel timekeeping. */ diff --git a/arch/x86/include/asm/vvar.h b/arch/x86/include/asm/vvar.h index 183e98e49ab9..9d9af37f7cab 100644 --- a/arch/x86/include/asm/vvar.h +++ b/arch/x86/include/asm/vvar.h @@ -26,6 +26,8 @@ */ #define DECLARE_VVAR(offset, type, name) \ EMIT_VVAR(name, offset) +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + EMIT_VVAR(name, offset) #else @@ -37,6 +39,10 @@ extern char __vvar_page; extern type timens_ ## name[CS_BASES] \ __attribute__((visibility("hidden"))); \ +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + extern type vvar_ ## name \ + __attribute__((visibility("hidden"))); \ + #define VVAR(name) (vvar_ ## name) #define TIMENS(name) (timens_ ## name) @@ -44,12 +50,22 @@ extern char __vvar_page; type name[CS_BASES] \ __attribute__((section(".vvar_" #name), aligned(16))) __visible +#define DEFINE_VVAR_SINGLE(type, name) \ + type name \ + __attribute__((section(".vvar_" #name), aligned(16))) __visible + #endif /* DECLARE_VVAR(offset, type, name) */ DECLARE_VVAR(128, struct vdso_data, _vdso_data) +#if !defined(_SINGLE_DATA) +#define _SINGLE_DATA +DECLARE_VVAR_SINGLE(640, struct vdso_rng_data, _vdso_rng_data) +#endif + #undef DECLARE_VVAR +#undef DECLARE_VVAR_SINGLE #endif