From patchwork Fri Oct 21 22:25:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 7018 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4242:0:0:0:0:0 with SMTP id s2csp930645wrr; Fri, 21 Oct 2022 15:30:16 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4oYa7lVmYL1X0z4y6v/miTp0T8Kqkf8WbY+REPJenfAvl1oiVdl6sco38mLyGdeMJiN2TM X-Received: by 2002:a17:903:11c3:b0:178:aec1:189c with SMTP id q3-20020a17090311c300b00178aec1189cmr21259817plh.136.1666391415985; Fri, 21 Oct 2022 15:30:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666391415; cv=none; d=google.com; s=arc-20160816; b=ITJqpEJMvQFjCGUIK2X+3lAT4EJuHddABEq5eKHT3/n3sQgdnWRvltKc2LcsxneG0N 31D/UO+jaofQYZpkjk1yy2F/sZ0jkhGD3fCJjx7/2rUscuk7ODGeBwQxiiHWM44pzVun tagmgQee5Ygek77hRbOkmZ1O999rH5ryEuBziZPvnrFINut+/hMjOq2E1rdpDrl9j/yb MEn3RrcGyGhAB4cytBBEbYCRaTLCZoBHUcR/99DSRCSxtrsclJG/L6is8xCauZB2hARN 0MowRbzKNZ/25GGe8/L30yylV6QNpWPxcLBCdiRVHVVh8FN6UUcSj1n8R2jkWmpsU9iX 9AQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:cc:subject:to:reply-to:sender:from :dkim-signature:dkim-signature:date; bh=WWVRtK1oSdCMD8tZ02jUK7li0uyMogbugXnIeZb0lpI=; b=bmPfZQfH7WCZVP6xBu5rKPOssarx6jO8Fy/9jKhzggIgMfErK+c81LGflsdN9J6O4L 9d6b2QolMj2OTd613dxtjSCeFRHlE37xxxGllp2hV7l5MCthCqLYiOGASeuR7DXxFN6I owMb7Jc+OugUjwsVAQetKtlrZm0TN5PmupPGJKmcrkPWK/2k8zm1kB9rHGJ1ctExZRed /cffjLfPgVMDB3brVqgaK66pwOk1JDMhqPCPINtRo+xyuZ4v3T5/I7ANBLqz/kUCbma0 EOscux41ZlBQepkkyORrpI9ilpGZ59bZo0TuyCkbdNL3ePe0FwI+XL8NsZmt5CCCeiC3 H7gQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MOXaMPag; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y11-20020a056a001c8b00b00562c5e2a99bsi20986522pfw.163.2022.10.21.15.30.02; Fri, 21 Oct 2022 15:30:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MOXaMPag; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229832AbiJUW0E (ORCPT + 99 others); Fri, 21 Oct 2022 18:26:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229838AbiJUWZw (ORCPT ); Fri, 21 Oct 2022 18:25:52 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA08C18E2B6; Fri, 21 Oct 2022 15:25:50 -0700 (PDT) Date: Fri, 21 Oct 2022 22:25:45 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1666391147; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WWVRtK1oSdCMD8tZ02jUK7li0uyMogbugXnIeZb0lpI=; b=MOXaMPagHMJyaT7UFqU6r5YeyKdgx/3/fTdRwcfJgOK3OY865Gz8aoka70HiYpKGu0d/P1 B3LW5JPT2xKddtUMT8R9Ro4VjBuWpRVJYwXwyfS7yjsL7QrOC2vzK5ggjBcIFWRpAJO1LE hnuRODhRrwIFNQtA9AUqqweo0nYY8nANvIT53NazAf7r3ACutC+H4BqffrkQ3L4wgqwGGO nQIrqYi1vF0Le8a08NTGpPRT2WtxBOAUKGN0UYMS3f1fERsz3Bgxela1o645LfX1Qq0end ESC1KR2P9REUDeZ20+NBIhcEzIl9Ac+PFNmbE5X3KLgEfKKKkFDnTv8Htm44hA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1666391147; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WWVRtK1oSdCMD8tZ02jUK7li0uyMogbugXnIeZb0lpI=; b=xV5NvZMPujhZ7Bix4/2jqP1TNwg0tQvyn+Ugx933Iut6LwL1ZLcQi6yvNOb4w+e8192Zpq zHaZtm/lWjV6fUBw== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/mm] x86/mm: Randomize per-cpu entry area Cc: Seth Jenkins , "Peter Zijlstra (Intel)" , Dave Hansen , Kees Cook , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <166639114548.401.707105259028399157.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747338045487595087?= X-GMAIL-MSGID: =?utf-8?q?1747338045487595087?= The following commit has been merged into the x86/mm branch of tip: Commit-ID: 11a4f78908cb8a6cccbb49dd7d0455a94741e959 Gitweb: https://git.kernel.org/tip/11a4f78908cb8a6cccbb49dd7d0455a94741e959 Author: Peter Zijlstra AuthorDate: Fri, 07 Oct 2022 10:42:36 +02:00 Committer: Dave Hansen CommitterDate: Fri, 21 Oct 2022 09:48:58 -07:00 x86/mm: Randomize per-cpu entry area Seth found that the CPU-entry-area; the piece of per-cpu data that is mapped into the userspace page-tables for kPTI is not subject to any randomization -- irrespective of kASLR settings. On x86_64 a whole P4D (512 GB) of virtual address space is reserved for this structure, which is plenty large enough to randomize things a little. As such, use a straightforward randomization scheme that avoids duplicates to spread the existing CPUs over the available space. This makes it harder to find the addresses of important structures in the cpu entry areas like the entry stacks. [ dhansen: add minor comment in "sodding terrible" loop ] Reported-by: Seth Jenkins Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Dave Hansen Reviewed-by: Kees Cook --- arch/x86/include/asm/cpu_entry_area.h | 4 +-- arch/x86/include/asm/pgtable_areas.h | 8 +++- arch/x86/kernel/hw_breakpoint.c | 2 +- arch/x86/mm/cpu_entry_area.c | 47 +++++++++++++++++++++++--- 4 files changed, 51 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h index 75efc4c..462fc34 100644 --- a/arch/x86/include/asm/cpu_entry_area.h +++ b/arch/x86/include/asm/cpu_entry_area.h @@ -130,10 +130,6 @@ struct cpu_entry_area { }; #define CPU_ENTRY_AREA_SIZE (sizeof(struct cpu_entry_area)) -#define CPU_ENTRY_AREA_ARRAY_SIZE (CPU_ENTRY_AREA_SIZE * NR_CPUS) - -/* Total size includes the readonly IDT mapping page as well: */ -#define CPU_ENTRY_AREA_TOTAL_SIZE (CPU_ENTRY_AREA_ARRAY_SIZE + PAGE_SIZE) DECLARE_PER_CPU(struct cpu_entry_area *, cpu_entry_area); DECLARE_PER_CPU(struct cea_exception_stacks *, cea_exception_stacks); diff --git a/arch/x86/include/asm/pgtable_areas.h b/arch/x86/include/asm/pgtable_areas.h index d34cce1..62e5ede 100644 --- a/arch/x86/include/asm/pgtable_areas.h +++ b/arch/x86/include/asm/pgtable_areas.h @@ -11,6 +11,12 @@ #define CPU_ENTRY_AREA_RO_IDT_VADDR ((void *)CPU_ENTRY_AREA_RO_IDT) -#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + CPU_ENTRY_AREA_ARRAY_SIZE - CPU_ENTRY_AREA_BASE) +#ifdef CONFIG_X86_32 +#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + \ + (CPU_ENTRY_AREA_SIZE * NR_CPUS) - \ + CPU_ENTRY_AREA_BASE +#else +#define CPU_ENTRY_AREA_MAP_SIZE P4D_SIZE +#endif #endif /* _ASM_X86_PGTABLE_AREAS_H */ diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c index 668a4a6..bbb0f73 100644 --- a/arch/x86/kernel/hw_breakpoint.c +++ b/arch/x86/kernel/hw_breakpoint.c @@ -266,7 +266,7 @@ static inline bool within_cpu_entry(unsigned long addr, unsigned long end) /* CPU entry erea is always used for CPU entry */ if (within_area(addr, end, CPU_ENTRY_AREA_BASE, - CPU_ENTRY_AREA_TOTAL_SIZE)) + CPU_ENTRY_AREA_MAP_SIZE)) return true; /* diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c index 6c2f1b7..ad1f750 100644 --- a/arch/x86/mm/cpu_entry_area.c +++ b/arch/x86/mm/cpu_entry_area.c @@ -15,16 +15,54 @@ static DEFINE_PER_CPU_PAGE_ALIGNED(struct entry_stack_page, entry_stack_storage) #ifdef CONFIG_X86_64 static DEFINE_PER_CPU_PAGE_ALIGNED(struct exception_stacks, exception_stacks); DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks); -#endif -#ifdef CONFIG_X86_32 +static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, _cea_offset); + +static inline unsigned int cea_offset(unsigned int cpu) +{ + return per_cpu(_cea_offset, cpu); +} + +static __init void init_cea_offsets(void) +{ + unsigned int max_cea; + unsigned int i, j; + + max_cea = (CPU_ENTRY_AREA_MAP_SIZE - PAGE_SIZE) / CPU_ENTRY_AREA_SIZE; + + /* O(sodding terrible) */ + for_each_possible_cpu(i) { + unsigned int cea; + +again: + cea = prandom_u32_max(max_cea); + + /* Make sure that no previous CPU shares the offset: */ + for_each_possible_cpu(j) { + if (cea_offset(j) == cea) + goto again; + + if (i == j) + break; + } + + per_cpu(_cea_offset, i) = cea; + } +} +#else /* !X86_64 */ DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack); + +static inline unsigned int cea_offset(unsigned int cpu) +{ + return cpu; +} +static inline void init_cea_offsets(void) { } #endif /* Is called from entry code, so must be noinstr */ noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu) { - unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE; + unsigned long va = CPU_ENTRY_AREA_PER_CPU + cea_offset(cpu) * CPU_ENTRY_AREA_SIZE; BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0); return (struct cpu_entry_area *) va; @@ -205,7 +243,6 @@ static __init void setup_cpu_entry_area_ptes(void) /* The +1 is for the readonly IDT: */ BUILD_BUG_ON((CPU_ENTRY_AREA_PAGES+1)*PAGE_SIZE != CPU_ENTRY_AREA_MAP_SIZE); - BUILD_BUG_ON(CPU_ENTRY_AREA_TOTAL_SIZE != CPU_ENTRY_AREA_MAP_SIZE); BUG_ON(CPU_ENTRY_AREA_BASE & ~PMD_MASK); start = CPU_ENTRY_AREA_BASE; @@ -221,6 +258,8 @@ void __init setup_cpu_entry_areas(void) { unsigned int cpu; + init_cea_offsets(); + setup_cpu_entry_area_ptes(); for_each_possible_cpu(cpu)