From patchwork Fri Nov 18 17:28:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 22408 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp326671wrr; Fri, 18 Nov 2022 09:45:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf4K4CjMRRrDQm850DcXWFxK5nXQj+bLwrXLfSj4BcwEURZRHqgumo9SlhTEacavRsVK4JBW X-Received: by 2002:a63:5548:0:b0:476:c39b:ab5b with SMTP id f8-20020a635548000000b00476c39bab5bmr7258067pgm.565.1668793549858; Fri, 18 Nov 2022 09:45:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668793549; cv=none; d=google.com; s=arc-20160816; b=Ru6K3+auaMLX4SfqRcoiOdPlFs183CHbANueqWzEBVF30RX2q+mp6/rZrWmpqKt2G8 zWAJhkodHmg3dCo4A+JYbeP9IU9ylgSn05y1oT6AE4PBbR3rNsaws/A45QymXNO9xja6 ID7A9PLbOYZehM3ibXhQ6JtSs8FXsvOp0/V4wi9Lp+2W/DwrYC6CNsFpgNaCOsbECwMr ylBzu3BGoGgZdaK8S1R27AwM2G/MffLdcqaRjpLxuxPQlr2O6VYzQNJg4Lg3VXUPb9sv GWTLSF5RySK843kB2pFLNAYTIz0FZOiBV1k8yHXriwbkvVpAVKiTvniAFk2ye2orXugs 5+6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T6L3jtW2SoWaDL/fJUN0DeobDZjSMEBLN32jobVlzYg=; b=rLadNz72Pfi/IFNmjGC78mwA1upAzhugz/KTc6P4hiA3ByPcXmWDcC7UPfOVkETFRF 2LU6mGiDcon7R81PCwx/FD5R45GrqbTrDDyaf/3tLoKHUn63FjTkP6U8o4lYTvd00q2l SKfMsMVsq1CQJLUegJaSNE2+II6pQKDfBaPX9Q6aqs+Y8xe5sGIiO4EZcGvJFwOPvvuM SRWAU4Cftfo84v4LCn8537hBOJYGsg6oW3ZfBuj++hUwSmoNYawCu//9bhYYoRMNpFxJ lLO4h+wDfch2y0sYxP4BFpNmWkOR/fxCMw6fMhNZfCuLEJK67Jufq4YGbtut/MHyvncz MN3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=SUnZUV4t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b14-20020a17090acc0e00b00213120e0c85si7387627pju.156.2022.11.18.09.45.35; Fri, 18 Nov 2022 09:45:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=SUnZUV4t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241538AbiKRRcg (ORCPT + 99 others); Fri, 18 Nov 2022 12:32:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242329AbiKRRb7 (ORCPT ); Fri, 18 Nov 2022 12:31:59 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E36B592093; Fri, 18 Nov 2022 09:31:58 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 74F34626B5; Fri, 18 Nov 2022 17:31:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D71F4C433C1; Fri, 18 Nov 2022 17:31:56 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="SUnZUV4t" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668792716; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T6L3jtW2SoWaDL/fJUN0DeobDZjSMEBLN32jobVlzYg=; b=SUnZUV4tCzfyqJ5Ujl3gQLBqgYbONv32LXgkKjFgokeZQ8I9ZovXyhM4BorxGHmLcWu4XK ATkD0NnG1qQqNJ8xqW2ojeYjzt6LKUf1i38U4xtWYCcY6Jcr5gnuPjYXQmCSIfLYmcbSM3 Rgj+DcrYPVguSTID1t1Lr18fEZFpGoI= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id a931288d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 18 Nov 2022 17:31:55 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v4 1/3] random: add vgetrandom_alloc() syscall Date: Fri, 18 Nov 2022 18:28:37 +0100 Message-Id: <20221118172839.2653829-2-Jason@zx2c4.com> In-Reply-To: <20221118172839.2653829-1-Jason@zx2c4.com> References: <20221118172839.2653829-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749856865830823009?= X-GMAIL-MSGID: =?utf-8?q?1749856865830823009?= The vDSO getrandom() works over an opaque per-thread state of an unexported size, which must be marked as MADV_WIPEONFORK and be mlock()'d for proper operation. Over time, the nuances of these allocations may change or grow or even differ based on architectural features. The syscall has the signature: void *vgetrandom_alloc([inout] size_t *num, [out] size_t *size_per_each, unsigned int flags); This takes the desired number of opaque states in `num`, and returns a pointer to an array of opaque states, the number actually allocated back in `num`, and the size in bytes of each one in `size_per_each`, enabling a libc to slice up the returned array into a state per each thread. (The `flags` argument is always zero for now.) Libc is expected to allocate a chunk of these on first use, and then dole them out to threads as they're created, allocating more when needed. We very intentionally do *not* leave state allocation for vDSO getrandom() up to userspace itself, but rather provide this new syscall for such allocations. vDSO getrandom() must not store its state in just any old memory address, but rather just ones that the kernel specially allocates for it, leaving the particularities of those allocations up to the kernel. Signed-off-by: Jason A. Donenfeld --- MAINTAINERS | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + drivers/char/random.c | 57 ++++++++++++++++++++++++++ lib/vdso/getrandom.h | 25 +++++++++++ 5 files changed, 85 insertions(+) create mode 100644 lib/vdso/getrandom.h diff --git a/MAINTAINERS b/MAINTAINERS index 256f03904987..843dd6a49538 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17287,6 +17287,7 @@ T: git https://git.kernel.org/pub/scm/linux/kernel/git/crng/random.git S: Maintained F: drivers/char/random.c F: drivers/virt/vmgenid.c +F: lib/vdso/getrandom.h RAPIDIO SUBSYSTEM M: Matt Porter diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl index 320480a8db4f..ea0fbc2ded5e 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -455,3 +455,4 @@ 448 i386 process_mrelease sys_process_mrelease 449 i386 futex_waitv sys_futex_waitv 450 i386 set_mempolicy_home_node sys_set_mempolicy_home_node +451 i386 vgetrandom_alloc sys_vgetrandom_alloc diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index c84d12608cd2..0186f173f0e8 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 common vgetrandom_alloc sys_vgetrandom_alloc # # Due to a historical design error, certain syscalls are numbered differently diff --git a/drivers/char/random.c b/drivers/char/random.c index 65ee69896967..b83481eb00a9 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -8,6 +8,7 @@ * into roughly six sections, each with a section header: * * - Initialization and readiness waiting. + * - VDSO support helpers. * - Fast key erasure RNG, the "crng". * - Entropy accumulation and extraction routines. * - Entropy collection routines. @@ -39,6 +40,7 @@ #include #include #include +#include #include #include #include @@ -59,6 +61,7 @@ #include #include #include +#include "../../lib/vdso/getrandom.h" /********************************************************************* * @@ -146,6 +149,60 @@ EXPORT_SYMBOL(wait_for_random_bytes); __func__, (void *)_RET_IP_, crng_init) + +/******************************************************************** + * + * VDSO support helpers. + * + * The actual vDSO function is defined over in lib/vdso/getrandom.c, + * but this section contains the kernel-mode helpers to support that. + * + ********************************************************************/ + +/* + * The vgetrandom() function in userspace requires an opaque state, which this + * function provides to userspace. The result is that it maps a certain + * number of special pages into the calling process and returns the address. + */ +SYSCALL_DEFINE3(vgetrandom_alloc, unsigned long __user *, num, + unsigned long __user *, size_per_each, unsigned int, flags) +{ + unsigned long alloc_size; + unsigned long num_states; + unsigned long pages_addr; + int ret; + + if (flags) + return -EINVAL; + + if (get_user(num_states, num)) + return -EFAULT; + + alloc_size = size_mul(num_states, sizeof(struct vgetrandom_state)); + if (alloc_size == SIZE_MAX) + return -EOVERFLOW; + alloc_size = roundup(alloc_size, PAGE_SIZE); + + if (put_user(alloc_size / sizeof(struct vgetrandom_state), num) || + put_user(sizeof(struct vgetrandom_state), size_per_each)) + return -EFAULT; + + pages_addr = vm_mmap(NULL, 0, alloc_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_LOCKED, 0); + if (IS_ERR_VALUE(pages_addr)) + return pages_addr; + + ret = do_madvise(current->mm, pages_addr, alloc_size, MADV_WIPEONFORK); + if (ret < 0) + goto err_unmap; + + return pages_addr; + +err_unmap: + vm_munmap(pages_addr, alloc_size); + return ret; +} + /********************************************************************* * * Fast key erasure RNG, the "crng". diff --git a/lib/vdso/getrandom.h b/lib/vdso/getrandom.h new file mode 100644 index 000000000000..fef7c0261e2b --- /dev/null +++ b/lib/vdso/getrandom.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ + +#ifndef _VDSO_LIB_GETRANDOM_H +#define _VDSO_LIB_GETRANDOM_H + +#include + +struct vgetrandom_state { + u64 last_reseed; + unsigned long generation; + union { + struct { + u8 key[CHACHA_KEY_SIZE]; + u8 batch[CHACHA_BLOCK_SIZE * 3 / 2]; + }; + u8 key_batch[CHACHA_BLOCK_SIZE * 2]; + }; + u8 pos; + bool not_forked; +}; + +#endif /* _VDSO_LIB_GETRANDOM_H */ From patchwork Fri Nov 18 17:28:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 22406 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp325588wrr; Fri, 18 Nov 2022 09:43:05 -0800 (PST) X-Google-Smtp-Source: AA0mqf4VSSMV+ytx8FTPVGKOljQ+wBE+9wvbsi8w37nfqy6qXr6huRJtuMqw7NqFKEdioJuxCD5K X-Received: by 2002:a17:902:e48b:b0:17f:799b:297d with SMTP id i11-20020a170902e48b00b0017f799b297dmr534468ple.72.1668793385245; Fri, 18 Nov 2022 09:43:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668793385; cv=none; d=google.com; s=arc-20160816; b=vVUU/T8X+B8vR5noIC4mdY4Pu4Wx+4A7e2R4ft96dJVvyBZCreXpWsi150Cyhb223Y Nh83wFilKgeG2L9C/srRYf6NPH4UosxLVzWqVlS0UVGbKNsSk4gEq8cI91y3oNIyXSY1 eF9NVsF1qHQqXDWo5FGT96JuZLCgcDFlIbvpCOEp/NAYiKXZMFVkkg0m1W7GNEgzIxRT mB3w6ZVHjp2/Ikwx1Nrzg5XrEwxEXC1eIBtlK3U9S+wWo7UAvyvnm4oTtiVZ8/ut4bvM Sbu+itBJt2DF+2sZKvCEkURA9D1FmInNVXnNhtV/qm0erGSa27FEcIhTJbdVxfUZPIsG wd9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9p1UIdcHRziM7m9qcPYp6vWPtjBY0MybeVn0TVZVSIU=; b=LBqwwci7gk5QP45CA52/FpWYE31U12s8ZalX4JIEn7VhFiaoyatZoot41QLyy+bgmx y5tu06CkrXkyzZzt8ytZRhaDsHIrq5dyU1A/IOKaiY86Xd14fwQGAyX92tCzjlZ3ZXMo tNCeyIZokeo62AfulnGJdNStRrF07NKZkD3tPRUjPw8ooIAVmvU5DBGFbYl8FaUZshAz nBnBT1zSd9dkPgY35OydQr0XPmnBQRrxyS52TvpVTjtCRstgPvGkXBl7PCNPDinyl8ty REM+xqU8+Nm/VO9zsibosq/3hfo1ZhxKCW0apKGiMXA3NtvgzIeSiau/KFUm8qwwfKnt XbPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=CA4UvkNZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a5-20020a170902ecc500b0017a7506e0dcsi4574003plh.231.2022.11.18.09.42.51; Fri, 18 Nov 2022 09:43:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=CA4UvkNZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235364AbiKRRcO (ORCPT + 99 others); Fri, 18 Nov 2022 12:32:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242396AbiKRRcE (ORCPT ); Fri, 18 Nov 2022 12:32:04 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 280108B13D; Fri, 18 Nov 2022 09:32:03 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A59A46268B; Fri, 18 Nov 2022 17:32:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 267DBC433D7; Fri, 18 Nov 2022 17:32:01 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="CA4UvkNZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668792720; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9p1UIdcHRziM7m9qcPYp6vWPtjBY0MybeVn0TVZVSIU=; b=CA4UvkNZOFiaNqpDzRS53bJWuvypQSw1i9RuqmYM4MNPUbZy5P0j0bfJZxeE9eSLg4IaZM /62QGDuhxJy+hPb0XOCQ6zLTo6xgvYw+pY1OBrKOVzavS1vVDVkK1D7At0IlZmtWxL+82D SN5HA8YTwsOblwTRcmy4jgr8rZjWmlA= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 7ec53f12 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 18 Nov 2022 17:31:59 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v4 2/3] random: introduce generic vDSO getrandom() implementation Date: Fri, 18 Nov 2022 18:28:38 +0100 Message-Id: <20221118172839.2653829-3-Jason@zx2c4.com> In-Reply-To: <20221118172839.2653829-1-Jason@zx2c4.com> References: <20221118172839.2653829-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749856692737343234?= X-GMAIL-MSGID: =?utf-8?q?1749856692737343234?= Provide a generic C vDSO getrandom() implementation, which operates on an opaque state returned by vgetrandom_alloc() and produces random bytes the same way as getrandom(). This has a the API signature: ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); The return value and the first 3 arguments are the same as ordinary getrandom(), while the last argument is a pointer to the opaque allocated state. Were all four arguments passed to the getrandom() syscall, nothing different would happen, and the functions would have the exact same behavior. The actual vDSO RNG algorithm implemented is the same one implemented by drivers/char/random.c, using the same fast-erasure techniques as that. Should the in-kernel implementation change, so too will the vDSO one. Initially, the state is keyless, and so the first call makes a getrandom() syscall to generate that key, and then uses it for subsequent calls. By keeping track of a generation counter, it knows when its key is invalidated and it should fetch a new one using the syscall. Later, more than just a generation counter might be used. Since MADV_WIPEONFORK is set on the opaque state, the key and related state is wiped during a fork(), so secrets don't roll over into new processes, and the same state doesn't accidentally generate the same random stream. The generation counter, as well, is always >0, so that the 0 counter is a useful indication of a fork() or otherwise uninitialized state. If the kernel RNG is not yet initialized, then the vDSO always calls the syscall, because that behavior cannot be emulated in userspace, but fortunately that state is short lived and only during early boot. If it has been initialized, then there is no need to inspect the `flags` argument, because the behavior does not change post-initialization regardless of the `flags` value. Signed-off-by: Jason A. Donenfeld --- MAINTAINERS | 1 + drivers/char/random.c | 5 ++ include/vdso/datapage.h | 6 +++ lib/crypto/chacha.c | 4 ++ lib/vdso/Kconfig | 5 ++ lib/vdso/getrandom.c | 109 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 130 insertions(+) create mode 100644 lib/vdso/getrandom.c diff --git a/MAINTAINERS b/MAINTAINERS index 843dd6a49538..e0aa33f54c57 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17287,6 +17287,7 @@ T: git https://git.kernel.org/pub/scm/linux/kernel/git/crng/random.git S: Maintained F: drivers/char/random.c F: drivers/virt/vmgenid.c +F: lib/vdso/getrandom.c F: lib/vdso/getrandom.h RAPIDIO SUBSYSTEM diff --git a/drivers/char/random.c b/drivers/char/random.c index b83481eb00a9..a62255d852f9 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -61,6 +61,7 @@ #include #include #include +#include #include "../../lib/vdso/getrandom.h" /********************************************************************* @@ -305,6 +306,8 @@ static void crng_reseed(struct work_struct *work) if (next_gen == ULONG_MAX) ++next_gen; WRITE_ONCE(base_crng.generation, next_gen); + if (IS_ENABLED(CONFIG_HAVE_VDSO_GETRANDOM)) + smp_store_release(&_vdso_rng_data.generation, next_gen + 1); if (!static_branch_likely(&crng_is_ready)) crng_init = CRNG_READY; spin_unlock_irqrestore(&base_crng.lock, flags); @@ -754,6 +757,8 @@ static void __cold _credit_init_bits(size_t bits) crng_reseed(NULL); /* Sets crng_init to CRNG_READY under base_crng.lock. */ if (static_key_initialized) execute_in_process_context(crng_set_ready, &set_ready); + if (IS_ENABLED(CONFIG_HAVE_VDSO_GETRANDOM)) + smp_store_release(&_vdso_rng_data.is_ready, true); wake_up_interruptible(&crng_init_wait); kill_fasync(&fasync, SIGIO, POLL_IN); pr_notice("crng init done\n"); diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 73eb622e7663..cbacfd923a5c 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -109,6 +109,11 @@ struct vdso_data { struct arch_vdso_data arch_data; }; +struct vdso_rng_data { + unsigned long generation; + bool is_ready; +}; + /* * We use the hidden visibility to prevent the compiler from generating a GOT * relocation. Not only is going through a GOT useless (the entry couldn't and @@ -120,6 +125,7 @@ struct vdso_data { */ extern struct vdso_data _vdso_data[CS_BASES] __attribute__((visibility("hidden"))); extern struct vdso_data _timens_data[CS_BASES] __attribute__((visibility("hidden"))); +extern struct vdso_rng_data _vdso_rng_data __attribute__((visibility("hidden"))); /* * The generic vDSO implementation requires that gettimeofday.h diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c index b748fd3d256e..944991bb36c7 100644 --- a/lib/crypto/chacha.c +++ b/lib/crypto/chacha.c @@ -17,8 +17,10 @@ static void chacha_permute(u32 *x, int nrounds) { int i; +#ifndef CHACHA_FOR_VDSO_INCLUDE /* whitelist the allowed round counts */ WARN_ON_ONCE(nrounds != 20 && nrounds != 12); +#endif for (i = 0; i < nrounds; i += 2) { x[0] += x[4]; x[12] = rol32(x[12] ^ x[0], 16); @@ -87,6 +89,7 @@ void chacha_block_generic(u32 *state, u8 *stream, int nrounds) state[12]++; } +#ifndef CHACHA_FOR_VDSO_INCLUDE EXPORT_SYMBOL(chacha_block_generic); /** @@ -112,3 +115,4 @@ void hchacha_block_generic(const u32 *state, u32 *stream, int nrounds) memcpy(&stream[4], &x[12], 16); } EXPORT_SYMBOL(hchacha_block_generic); +#endif diff --git a/lib/vdso/Kconfig b/lib/vdso/Kconfig index d883ac299508..c35fac664574 100644 --- a/lib/vdso/Kconfig +++ b/lib/vdso/Kconfig @@ -30,4 +30,9 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO +config HAVE_VDSO_GETRANDOM + bool + help + Selected by architectures that support vDSO getrandom(). + endif diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c new file mode 100644 index 000000000000..b253e9247706 --- /dev/null +++ b/lib/vdso/getrandom.c @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ + +#include +#include +#include +#include +#include +#include +#include "getrandom.h" + +#undef memcpy +#define memcpy(d,s,l) __builtin_memcpy(d,s,l) +#undef memset +#define memset(d,c,l) __builtin_memset(d,c,l) + +#define CHACHA_FOR_VDSO_INCLUDE +#include "../crypto/chacha.c" + +static void memcpy_and_zero(void *dst, void *src, size_t len) +{ +#define CASCADE(type) \ + while (len >= sizeof(type)) { \ + *(type *)dst = *(type *)src; \ + *(type *)src = 0; \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS +#if BITS_PER_LONG == 64 + CASCADE(u64); +#endif + CASCADE(u32); + CASCADE(u16); +#endif + CASCADE(u8); +#undef CASCADE +} + +static __always_inline ssize_t +__cvdso_getrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state) +{ + struct vgetrandom_state *state = opaque_state; + const struct vdso_rng_data *rng_info = __arch_get_vdso_rng_data(); + u32 chacha_state[CHACHA_STATE_WORDS]; + ssize_t ret = min_t(size_t, MAX_RW_COUNT, len); + unsigned long current_generation; + size_t batch_len; + + if (unlikely(!rng_info->is_ready)) + return getrandom_syscall(buffer, len, flags); + + if (unlikely(!len)) + return 0; + + if (unlikely(!READ_ONCE(state->not_forked))) + state->not_forked = true; + +retry_generation: + current_generation = READ_ONCE(rng_info->generation); + if (unlikely(state->generation != current_generation)) { + if (getrandom_syscall(state->key, sizeof(state->key), 0) != sizeof(state->key)) + return getrandom_syscall(buffer, len, flags); + state->generation = current_generation; + state->pos = sizeof(state->batch); + } + + len = ret; +more_batch: + batch_len = min_t(size_t, sizeof(state->batch) - state->pos, len); + if (batch_len) { + memcpy_and_zero(buffer, state->batch + state->pos, batch_len); + state->pos += batch_len; + buffer += batch_len; + len -= batch_len; + } + if (!len) { + if (unlikely(current_generation != READ_ONCE(rng_info->generation))) + goto retry_generation; + if (unlikely(!READ_ONCE(state->not_forked))) { + state->not_forked = true; + goto retry_generation; + } + return ret; + } + + chacha_init_consts(chacha_state); + memcpy(&chacha_state[4], state->key, CHACHA_KEY_SIZE); + memset(&chacha_state[12], 0, sizeof(u32) * 4); + + while (len >= CHACHA_BLOCK_SIZE) { + chacha20_block(chacha_state, buffer); + if (unlikely(chacha_state[12] == 0)) + ++chacha_state[13]; + buffer += CHACHA_BLOCK_SIZE; + len -= CHACHA_BLOCK_SIZE; + } + + chacha20_block(chacha_state, state->key_batch); + if (unlikely(chacha_state[12] == 0)) + ++chacha_state[13]; + chacha20_block(chacha_state, state->key_batch + CHACHA_BLOCK_SIZE); + state->pos = 0; + memzero_explicit(chacha_state, sizeof(chacha_state)); + goto more_batch; +} From patchwork Fri Nov 18 17:28:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jason A. Donenfeld" X-Patchwork-Id: 22407 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp326454wrr; Fri, 18 Nov 2022 09:45:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf5VotdsHukI4l/wsDh0pUcqd9DTgCyIpX8RwLssG49Pz+lZcfbhHX4ZBAdVDsRbTrI3s4nl X-Received: by 2002:a17:90a:6007:b0:200:8a2b:d120 with SMTP id y7-20020a17090a600700b002008a2bd120mr14297628pji.208.1668793519354; Fri, 18 Nov 2022 09:45:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668793519; cv=none; d=google.com; s=arc-20160816; b=T1E4U2tLb6hv6iU4rEabHkM8IaoF9igECMe/maBaWFFP6rdNiLLUQbP3CniQ/09L9t c3Yggm1mJCHww74i/HVwDZWtz8G9gYEmlnMMu5dd8LAbe3TWRybkBObpjYPDg7e7Sm3M SIeQnDNAQu5SwWpqTWz7tXUdsv6nTKR2sZ+G/5Kaq/sz5JXBgPQNeITBH1Av/RCUYCqr BkaTT/MQctDZGTx1OlaTQ++ng9sasfYpcBdc7YtisSoXveHVvjOAhTN8dqWMhk15RU2V shkzj+ystAmXTbIf4agWoUhkGD4LzHhR28HZZviLyuJvNt95wKDd9zrdZWJk/w3xc6eQ 5/XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GyKOGkofkPCvMPD0/UwJzzsEw9ZOaxhI19IJoJsqMz4=; b=YpWnQcXkgeNgb+HsUKs/BHCq3RaMt4didSb7h42q/knICe4drltsKZktVtCs2WSkg9 iFmu4A8RJWbj+DtehXDG6+haXjhPqtp4mISv+fg5apYbAxYbc4Al0ESjoKTCXzuufDfS ihRKuAJIJgjao06xxl6yzdR0oZUuyFeb0blO3Cch3jGqyPt1qRc0zpcPa6KApi1NHVKo VDGvVyyw8n5DqJx9weqa7Bqk68/6wDtX1lOAjljNwRjYy4+csYB/FMY2V/W9demhTkDy zL14e3nxrSsq4kDh4JJFlDlRyfjACt6SK68jOCAWjSxfPM+OOXOJkFTgTX2s6633Gc/Z YrBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=Ck10xJJF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ob6-20020a17090b390600b0020a6302b4f2si8924115pjb.75.2022.11.18.09.45.03; Fri, 18 Nov 2022 09:45:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@zx2c4.com header.s=20210105 header.b=Ck10xJJF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=zx2c4.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235528AbiKRRc1 (ORCPT + 99 others); Fri, 18 Nov 2022 12:32:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242530AbiKRRcL (ORCPT ); Fri, 18 Nov 2022 12:32:11 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59A32186EC; Fri, 18 Nov 2022 09:32:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 6253FCE221A; Fri, 18 Nov 2022 17:32:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5560BC433C1; Fri, 18 Nov 2022 17:32:05 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="Ck10xJJF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1668792724; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GyKOGkofkPCvMPD0/UwJzzsEw9ZOaxhI19IJoJsqMz4=; b=Ck10xJJFyJtJu2syfKbZVRObrnnQWQQl4uCW82Ixw9QTcO0pvDWSSDNrWQpF28KzVirC8E 94cCuokeZvQBA8UdWGqX3ec1roFifnyxFkjv+f9eQTeKIdo8EsDKQCId6MnKR5czXDdwCh O2e9onhFhYQMa1O39oHan0T3zGUDSvY= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 70c6e225 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 18 Nov 2022 17:32:04 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, patches@lists.linux.dev Cc: "Jason A. Donenfeld" , linux-crypto@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Greg Kroah-Hartman , Adhemerval Zanella Netto , Carlos O'Donell Subject: [PATCH v4 3/3] x86: vdso: Wire up getrandom() vDSO implementation Date: Fri, 18 Nov 2022 18:28:39 +0100 Message-Id: <20221118172839.2653829-4-Jason@zx2c4.com> In-Reply-To: <20221118172839.2653829-1-Jason@zx2c4.com> References: <20221118172839.2653829-1-Jason@zx2c4.com> MIME-Version: 1.0 X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749856833525663339?= X-GMAIL-MSGID: =?utf-8?q?1749856833525663339?= Hook up the generic vDSO implementation to the x86 vDSO data page. Since the existing vDSO infrastructure is heavily based on the timekeeping functionality, which works over arrays of bases, a new macro is introduced for vvars that are not arrays. Signed-off-by: Jason A. Donenfeld --- arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 3 ++- arch/x86/entry/vdso/vdso.lds.S | 2 ++ arch/x86/entry/vdso/vgetrandom.c | 16 ++++++++++++ arch/x86/include/asm/vdso/getrandom.h | 37 +++++++++++++++++++++++++++ arch/x86/include/asm/vdso/vsyscall.h | 2 ++ arch/x86/include/asm/vvar.h | 16 ++++++++++++ 7 files changed, 76 insertions(+), 1 deletion(-) create mode 100644 arch/x86/entry/vdso/vgetrandom.c create mode 100644 arch/x86/include/asm/vdso/getrandom.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 67745ceab0db..210318da7505 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -269,6 +269,7 @@ config X86 select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_USER_RETURN_NOTIFIER select HAVE_GENERIC_VDSO + select HAVE_VDSO_GETRANDOM select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING select NEED_PER_CPU_EMBED_FIRST_CHUNK diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index 3e88b9df8c8f..adc3792dbbac 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -27,7 +27,7 @@ VDSO32-$(CONFIG_X86_32) := y VDSO32-$(CONFIG_IA32_EMULATION) := y # files to link into the vdso -vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o +vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o vgetrandom.o vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) += vsgx.o @@ -104,6 +104,7 @@ CFLAGS_REMOVE_vclock_gettime.o = -pg CFLAGS_REMOVE_vdso32/vclock_gettime.o = -pg CFLAGS_REMOVE_vgetcpu.o = -pg CFLAGS_REMOVE_vsgx.o = -pg +CFLAGS_REMOVE_vgetrandom.o = -pg # # X32 processes use x32 vDSO to access 64bit kernel data. diff --git a/arch/x86/entry/vdso/vdso.lds.S b/arch/x86/entry/vdso/vdso.lds.S index 4bf48462fca7..1919cc39277e 100644 --- a/arch/x86/entry/vdso/vdso.lds.S +++ b/arch/x86/entry/vdso/vdso.lds.S @@ -28,6 +28,8 @@ VERSION { clock_getres; __vdso_clock_getres; __vdso_sgx_enter_enclave; + getrandom; + __vdso_getrandom; local: *; }; } diff --git a/arch/x86/entry/vdso/vgetrandom.c b/arch/x86/entry/vdso/vgetrandom.c new file mode 100644 index 000000000000..0a0c0ad93cd0 --- /dev/null +++ b/arch/x86/entry/vdso/vgetrandom.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#include +#include + +#include "../../../../lib/vdso/getrandom.c" + +ssize_t __vdso_getrandom(void *buffer, size_t len, unsigned int flags, void *state) +{ + return __cvdso_getrandom(buffer, len, flags, state); +} + +ssize_t getrandom(void *, size_t, unsigned int, void *) + __attribute__((weak, alias("__vdso_getrandom"))); diff --git a/arch/x86/include/asm/vdso/getrandom.h b/arch/x86/include/asm/vdso/getrandom.h new file mode 100644 index 000000000000..c414043e975d --- /dev/null +++ b/arch/x86/include/asm/vdso/getrandom.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#ifndef __ASM_VDSO_GETRANDOM_H +#define __ASM_VDSO_GETRANDOM_H + +#ifndef __ASSEMBLY__ + +#include +#include + +static __always_inline ssize_t +getrandom_syscall(void *buffer, size_t len, unsigned int flags) +{ + long ret; + + asm ("syscall" : "=a" (ret) : + "0" (__NR_getrandom), "D" (buffer), "S" (len), "d" (flags) : + "rcx", "r11", "memory"); + + return ret; +} + +#define __vdso_rng_data (VVAR(_vdso_rng_data)) + +static __always_inline const struct vdso_rng_data *__arch_get_vdso_rng_data(void) +{ + if (__vdso_data->clock_mode == VDSO_CLOCKMODE_TIMENS) + return (void *)&__vdso_rng_data + + ((void *)&__timens_vdso_data - (void *)&__vdso_data); + return &__vdso_rng_data; +} + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_VDSO_GETRANDOM_H */ diff --git a/arch/x86/include/asm/vdso/vsyscall.h b/arch/x86/include/asm/vdso/vsyscall.h index be199a9b2676..71c56586a22f 100644 --- a/arch/x86/include/asm/vdso/vsyscall.h +++ b/arch/x86/include/asm/vdso/vsyscall.h @@ -11,6 +11,8 @@ #include DEFINE_VVAR(struct vdso_data, _vdso_data); +DEFINE_VVAR_SINGLE(struct vdso_rng_data, _vdso_rng_data); + /* * Update the vDSO data page to keep in sync with kernel timekeeping. */ diff --git a/arch/x86/include/asm/vvar.h b/arch/x86/include/asm/vvar.h index 183e98e49ab9..9d9af37f7cab 100644 --- a/arch/x86/include/asm/vvar.h +++ b/arch/x86/include/asm/vvar.h @@ -26,6 +26,8 @@ */ #define DECLARE_VVAR(offset, type, name) \ EMIT_VVAR(name, offset) +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + EMIT_VVAR(name, offset) #else @@ -37,6 +39,10 @@ extern char __vvar_page; extern type timens_ ## name[CS_BASES] \ __attribute__((visibility("hidden"))); \ +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + extern type vvar_ ## name \ + __attribute__((visibility("hidden"))); \ + #define VVAR(name) (vvar_ ## name) #define TIMENS(name) (timens_ ## name) @@ -44,12 +50,22 @@ extern char __vvar_page; type name[CS_BASES] \ __attribute__((section(".vvar_" #name), aligned(16))) __visible +#define DEFINE_VVAR_SINGLE(type, name) \ + type name \ + __attribute__((section(".vvar_" #name), aligned(16))) __visible + #endif /* DECLARE_VVAR(offset, type, name) */ DECLARE_VVAR(128, struct vdso_data, _vdso_data) +#if !defined(_SINGLE_DATA) +#define _SINGLE_DATA +DECLARE_VVAR_SINGLE(640, struct vdso_rng_data, _vdso_rng_data) +#endif + #undef DECLARE_VVAR +#undef DECLARE_VVAR_SINGLE #endif