From patchwork Fri Oct 28 06:43:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 12119 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp659921wru; Thu, 27 Oct 2022 23:47:26 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5c8/Rl1IieYLg6k6Df2XiNpYWvTu+cq3DaPHLJIEjIHtDu8XpgW4ZlbFaIBq+OWG/a9VfP X-Received: by 2002:a17:907:31c4:b0:78d:9b2f:4e1a with SMTP id xf4-20020a17090731c400b0078d9b2f4e1amr45567456ejb.306.1666939645687; Thu, 27 Oct 2022 23:47:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666939645; cv=none; d=google.com; s=arc-20160816; b=oWLxPgwGZHHXrcr4U1zpzp44BlepoDaQO0OQ483CHsic+SfWK0Uhow4pHKQktfFowY DS/PjqgqL7NUs0gUtIpEdFfozRarvG3L/2QizncCDO8YxYWIzetOB2DBVcJBi6951i51 BhZ5teeEeCzpN7dIKqBfDePACJmVITz6/bu+fjy6IghNmPPIZJtR/5NpNRKsBrTy6ox3 P1pOQEddODwquM9n5ZroDouMndBp+kj/2UXocJfjv4X8u6RZPT4ii/BHKdOQlclhWeFa r3BoZ+jKLjrXngMvlO+ZqLPJJ+AMrZg39NzsU0LPc0AAPMk8hP0VhmqpvHvnJA3BVd4T GrNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:cc:subject:to:reply-to:sender:from :dkim-signature:dkim-signature:date; bh=Ix3l74K5a6QQMFq/Prx0vqbj1SroRjSG3cNNGwkaFQ4=; b=eiYXAv25FFW+tftfFSckSLaFt+ItcHx+q4vsLZUb4B4mm21rwrfcwqBBta81O88Yuc 1KL0QkvgsdpNORAKdZcqc6esheAaraC+4cHDMl3m4dWnZpwwkwfU88gocT5aozyzKkjo dQ/s++wb37WIH1+BVM2KVfwP1pOJYjNA8sH2/oLfAyEMj7QIdcsRi8Nti5lKHTeH3DEy jGcWqAyxeZzZzC+k6Lv6q7nmc0xhPqIPneS8Oc04Pd4vRDMEKu3E/djy0zrPAL8eYr5L SNsU6rK5Rh/kFrGPCUHE+rMFOjMaz3XBvbJqxDqus6vLB8amO22jVYadUCbhqVq+cuL+ FdPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=xOqcRyke; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s17-20020a170906bc5100b0079b40c09982si1223682ejv.340.2022.10.27.23.47.02; Thu, 27 Oct 2022 23:47:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=xOqcRyke; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229941AbiJ1Gqr (ORCPT + 99 others); Fri, 28 Oct 2022 02:46:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230047AbiJ1GqK (ORCPT ); Fri, 28 Oct 2022 02:46:10 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 081DE1BE90B; Thu, 27 Oct 2022 23:43:07 -0700 (PDT) Date: Fri, 28 Oct 2022 06:43:03 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1666939385; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Ix3l74K5a6QQMFq/Prx0vqbj1SroRjSG3cNNGwkaFQ4=; b=xOqcRykepM9FExepY5+Qd4xaXGQ4261ZGHHHyhn8Nhql8fkSlZVhD6S6ETgE3j3hRWwDsI mRNEZXL1yPgbswbhNC4cec//pLZkoBVKqOYSxHUR6ygseigu8MPFaKkoC7KMD/tQFwZdt1 MsaIL5vKj+IHrersIxwkmXuks8zh5EH5m90dSv4HqhIa2/st6323vL20o4CBcFvwFXHx83 r9e0widtwdC/BKN7BIFc0dgKn6XL683xsw/Ex/8Ct64H6uj0U57DSDEkK/m/Az64vtkCpD KQMXsmLc6c0QuL+FEwWZHnodCaMJvQW3pL1OK5qM/9PZIqblrZMsv4SqxHuzvw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1666939385; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Ix3l74K5a6QQMFq/Prx0vqbj1SroRjSG3cNNGwkaFQ4=; b=RDv6HtAGdo1sca0TRnmeCXOB5VVTIKmnkObYe0FgG8VGo0CNp5X8NoSEjV6xEBoGMfTp7N NPF/CY80/pNjXoDg== From: "tip-bot2 for Peter Zijlstra" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/mm] x86/mm: Randomize per-cpu entry area Cc: Seth Jenkins , "Peter Zijlstra (Intel)" , Dave Hansen , Kees Cook , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <166693938356.29415.10672976989880224171.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747912905729315835?= X-GMAIL-MSGID: =?utf-8?q?1747912905729315835?= The following commit has been merged into the x86/mm branch of tip: Commit-ID: d6d1a3923799d04f56cd644a9cb7ecf19de67949 Gitweb: https://git.kernel.org/tip/d6d1a3923799d04f56cd644a9cb7ecf19de67949 Author: Peter Zijlstra AuthorDate: Thu, 27 Oct 2022 14:54:41 -07:00 Committer: Dave Hansen CommitterDate: Thu, 27 Oct 2022 15:01:53 -07:00 x86/mm: Randomize per-cpu entry area Seth found that the CPU-entry-area; the piece of per-cpu data that is mapped into the userspace page-tables for kPTI is not subject to any randomization -- irrespective of kASLR settings. On x86_64 a whole P4D (512 GB) of virtual address space is reserved for this structure, which is plenty large enough to randomize things a little. As such, use a straight forward randomization scheme that avoids duplicates to spread the existing CPUs over the available space. Reported-by: Seth Jenkins Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Dave Hansen Reviewed-by: Kees Cook --- arch/x86/include/asm/cpu_entry_area.h | 4 +-- arch/x86/include/asm/pgtable_areas.h | 8 ++++- arch/x86/kernel/hw_breakpoint.c | 2 +- arch/x86/mm/cpu_entry_area.c | 46 +++++++++++++++++++++++--- 4 files changed, 50 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h index 75efc4c..462fc34 100644 --- a/arch/x86/include/asm/cpu_entry_area.h +++ b/arch/x86/include/asm/cpu_entry_area.h @@ -130,10 +130,6 @@ struct cpu_entry_area { }; #define CPU_ENTRY_AREA_SIZE (sizeof(struct cpu_entry_area)) -#define CPU_ENTRY_AREA_ARRAY_SIZE (CPU_ENTRY_AREA_SIZE * NR_CPUS) - -/* Total size includes the readonly IDT mapping page as well: */ -#define CPU_ENTRY_AREA_TOTAL_SIZE (CPU_ENTRY_AREA_ARRAY_SIZE + PAGE_SIZE) DECLARE_PER_CPU(struct cpu_entry_area *, cpu_entry_area); DECLARE_PER_CPU(struct cea_exception_stacks *, cea_exception_stacks); diff --git a/arch/x86/include/asm/pgtable_areas.h b/arch/x86/include/asm/pgtable_areas.h index d34cce1..62e5ede 100644 --- a/arch/x86/include/asm/pgtable_areas.h +++ b/arch/x86/include/asm/pgtable_areas.h @@ -11,6 +11,12 @@ #define CPU_ENTRY_AREA_RO_IDT_VADDR ((void *)CPU_ENTRY_AREA_RO_IDT) -#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + CPU_ENTRY_AREA_ARRAY_SIZE - CPU_ENTRY_AREA_BASE) +#ifdef CONFIG_X86_32 +#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + \ + (CPU_ENTRY_AREA_SIZE * NR_CPUS) - \ + CPU_ENTRY_AREA_BASE +#else +#define CPU_ENTRY_AREA_MAP_SIZE P4D_SIZE +#endif #endif /* _ASM_X86_PGTABLE_AREAS_H */ diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c index 668a4a6..bbb0f73 100644 --- a/arch/x86/kernel/hw_breakpoint.c +++ b/arch/x86/kernel/hw_breakpoint.c @@ -266,7 +266,7 @@ static inline bool within_cpu_entry(unsigned long addr, unsigned long end) /* CPU entry erea is always used for CPU entry */ if (within_area(addr, end, CPU_ENTRY_AREA_BASE, - CPU_ENTRY_AREA_TOTAL_SIZE)) + CPU_ENTRY_AREA_MAP_SIZE)) return true; /* diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c index d7081b1..dff9001 100644 --- a/arch/x86/mm/cpu_entry_area.c +++ b/arch/x86/mm/cpu_entry_area.c @@ -16,16 +16,53 @@ static DEFINE_PER_CPU_PAGE_ALIGNED(struct entry_stack_page, entry_stack_storage) #ifdef CONFIG_X86_64 static DEFINE_PER_CPU_PAGE_ALIGNED(struct exception_stacks, exception_stacks); DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks); -#endif -#ifdef CONFIG_X86_32 +static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, _cea_offset); + +static __always_inline unsigned int cea_offset(unsigned int cpu) +{ + return per_cpu(_cea_offset, cpu); +} + +static __init void init_cea_offsets(void) +{ + unsigned int max_cea; + unsigned int i, j; + + max_cea = (CPU_ENTRY_AREA_MAP_SIZE - PAGE_SIZE) / CPU_ENTRY_AREA_SIZE; + + /* O(sodding terrible) */ + for_each_possible_cpu(i) { + unsigned int cea; + +again: + cea = prandom_u32_max(max_cea); + + for_each_possible_cpu(j) { + if (cea_offset(j) == cea) + goto again; + + if (i == j) + break; + } + + per_cpu(_cea_offset, i) = cea; + } +} +#else /* !X86_64 */ DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack); + +static __always_inline unsigned int cea_offset(unsigned int cpu) +{ + return cpu; +} +static inline void init_cea_offsets(void) { } #endif /* Is called from entry code, so must be noinstr */ noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu) { - unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE; + unsigned long va = CPU_ENTRY_AREA_PER_CPU + cea_offset(cpu) * CPU_ENTRY_AREA_SIZE; BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0); return (struct cpu_entry_area *) va; @@ -211,7 +248,6 @@ static __init void setup_cpu_entry_area_ptes(void) /* The +1 is for the readonly IDT: */ BUILD_BUG_ON((CPU_ENTRY_AREA_PAGES+1)*PAGE_SIZE != CPU_ENTRY_AREA_MAP_SIZE); - BUILD_BUG_ON(CPU_ENTRY_AREA_TOTAL_SIZE != CPU_ENTRY_AREA_MAP_SIZE); BUG_ON(CPU_ENTRY_AREA_BASE & ~PMD_MASK); start = CPU_ENTRY_AREA_BASE; @@ -227,6 +263,8 @@ void __init setup_cpu_entry_areas(void) { unsigned int cpu; + init_cea_offsets(); + setup_cpu_entry_area_ptes(); for_each_possible_cpu(cpu)