From patchwork Wed Nov 15 17:37:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Gerst X-Patchwork-Id: 165533 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2700769vqg; Wed, 15 Nov 2023 09:39:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IE5lXubxVQrMMgrthRabkn4YA2DUWQLyEcYpHrXslCsee7vomhqCtEen2BsuT6UVEzHVMp0 X-Received: by 2002:a54:4612:0:b0:3ac:b12b:9acc with SMTP id p18-20020a544612000000b003acb12b9accmr14654028oip.43.1700069965051; Wed, 15 Nov 2023 09:39:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700069965; cv=none; d=google.com; s=arc-20160816; b=UXMfmHbqfxwJwQNxBzoVlQuIHkANXSWWD0ZeFw0PZPwC0iOsuIedOCPZ4aCgQBeg0T RZridJv7cfM+JXUSTp8gAbpDJbLEOS1n1yVUM+1RSwsuyIGPp3QiEXa+mje855hr0ccO vflJYY5d2vRfPUpJaYAww1G+IRaMgciY1fg430+5Epw80WHFHyJMeW/nMuabH0MBKpIj 0lhGs46qULmEADtIhCB0wjLW7zic9gNSSdEt2eRnEZGMC9C4vq3FvFY+/vfgZpxUfUj0 3lrFw3F8tej3IFd1XdE1DJlgDz7QMjtq3/agfaKcsnnHT8aLULLC8qs7japFHZUj0nWv tZDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pxm7rj1nNs1AxdmMRh1p1RVuTse+b2SlzLkV0LVvDBU=; fh=O55jSVrVODK5Pp/8HguUJuz95L5ieTggTCPnkrmwi7I=; b=UMxx4OZvU9IA/gdGuDXQw/m6mcLhmnl9lXiAW6cwjCUzNaTtGe9BmmHRnWw4wDV67m AYCTLs9wS6ofHDja1FivhKET0yfakEub/VPEAQf+2kGTQIW5JfwcAwoBKBQ4+QiPNxU5 lmJLNd1mW1jatKojiq66skYyP8W8Comi9xFsPN5VYW1kxMZw2AA5w/2C04cTf/pWTzzg fGKg9eua3asuY+8AGh+yqomCDFsJt460jZNwk5I3lAzdJZcSfLe3QhPs86sFlUx/2Eua QrvHT7HTjv+Z7TQGm2iRtW//0+9FF0ys2G73sIxJSLSAqoes5yrvsQaudvmwGftnCt1P o3nQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=iDAvRoTy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bz41-20020a056a02062900b005be052b03a9si10733602pgb.354.2023.11.15.09.39.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 09:39:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=iDAvRoTy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 1F561821ADA8; Wed, 15 Nov 2023 09:38:23 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232109AbjKORh6 (ORCPT + 29 others); Wed, 15 Nov 2023 12:37:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232740AbjKORhn (ORCPT ); Wed, 15 Nov 2023 12:37:43 -0500 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3BB683 for ; Wed, 15 Nov 2023 09:37:38 -0800 (PST) Received: by mail-qk1-x72d.google.com with SMTP id af79cd13be357-7789a4c01ddso456314085a.1 for ; Wed, 15 Nov 2023 09:37:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700069857; x=1700674657; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pxm7rj1nNs1AxdmMRh1p1RVuTse+b2SlzLkV0LVvDBU=; b=iDAvRoTyGuyIGDfa3wfdJU3vhm8OjtJjhH8gZTj6uZXuZ5Z/C6cojYYCyjEObu243b rzt+Ay01W9KuZm4nl/iQCymlIuCtYrjx6aj+Z75kOWzquRRlcxusagEMa8+Yyodh1e/N O7XrhGxtjkStBDZvNADs9hQZ/bV0soX9heP5QBfkfBe3mgoCuKG2CT0vyBviI7Z8Ucu/ Ltt79ZnMFd1+T5wf3+X585c+cII6nJcl049CrmutnSPIDTAd+JcGfj4aqqldRCB2aqvG gnbcv86DKUE6E2n0zPa8vTvWsquxlxGyG4/BeekihPYE/9Out9+moOsoFpjRzuUDGvbC HisA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700069857; x=1700674657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pxm7rj1nNs1AxdmMRh1p1RVuTse+b2SlzLkV0LVvDBU=; b=lv2XfPd6nEQXgbjkeWC3z6MdZ8Hltdd1MQ8yaJpXyjomls2xKM+bF1SUl2uZA2sz85 0Y2eh/u0gSd4EcGwxj3vVDFx/qFgyEHPzgnlCDM2c14NBWV8AUzp6hmmq2Rz93v8UoDy yzW2U4PGyKYJYwTR+iG3/G5C4+ZPYa4METatoXDVMgj2BMwZhNzW/eoNY+Oa1rQs+A32 /SlxHEnMOK6Lx2eznwVvdEhYqDW6s/2icGQYgA+ltJZBgl0lRjEiHevbalyqdYTH9VWl hhxvrt16FS8/GnAsMGbeXtbDhoFk1nHMXQe/3iGnP74P1CXTxzJ8D09dKkcior2qjI/L uBfQ== X-Gm-Message-State: AOJu0Ywk141eJ+aRY0gyI5dXm9yqevMPgXs5KfvHbq4SNIA/IUlEWuFw 3ZQ6rTUFs9DDSdnJ72iq7BLYY5amZg== X-Received: by 2002:a05:620a:118e:b0:778:8da0:a6a2 with SMTP id b14-20020a05620a118e00b007788da0a6a2mr5872469qkk.28.1700069856945; Wed, 15 Nov 2023 09:37:36 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id x20-20020a05620a0b5400b0077bda014d8esm3260099qkg.87.2023.11.15.09.37.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 09:37:36 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , Thomas Gleixner , Borislav Petkov , "H . Peter Anvin" , Peter Zijlstra , Uros Bizjak , David.Laight@aculab.com, Brian Gerst Subject: [PATCH v3 09/14] x86/percpu/64: Use relative percpu offsets Date: Wed, 15 Nov 2023 12:37:03 -0500 Message-ID: <20231115173708.108316-10-brgerst@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231115173708.108316-1-brgerst@gmail.com> References: <20231115173708.108316-1-brgerst@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 09:38:23 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782652563798068108 X-GMAIL-MSGID: 1782652563798068108 The percpu section is currently linked at virtual address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section can be linked normally. This means that x86-64 will calculate percpu offsets like most other architectures, as the delta between the initial percpu address and the dynamically allocated memory. Signed-off-by: Brian Gerst --- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/head_64.S | 19 +++++++++---------- arch/x86/kernel/setup_percpu.c | 12 ++---------- arch/x86/kernel/vmlinux.lds.S | 29 +---------------------------- arch/x86/platform/pvh/head.S | 5 ++--- arch/x86/tools/relocs.c | 10 +++------- arch/x86/xen/xen-head.S | 9 ++++----- init/Kconfig | 2 +- 8 files changed, 27 insertions(+), 65 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 3ee091225904..73fa9d4d2e16 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -403,7 +403,11 @@ DECLARE_INIT_PER_CPU(fixed_percpu_data); static inline unsigned long cpu_kernelmode_gs_base(int cpu) { - return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu); +#ifdef CONFIG_SMP + return per_cpu_offset(cpu); +#else + return 0; +#endif } extern asmlinkage void entry_SYSCALL32_ignore(void); diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 0d94d2a091fe..fe73e1c4cc5d 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -70,11 +70,14 @@ SYM_CODE_START_NOALIGN(startup_64) leaq _text(%rip), %rdi - /* Setup GSBASE to allow stack canary access for C code */ + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx + xorl %eax, %eax + xorl %edx, %edx wrmsr call startup_64_setup_env @@ -343,16 +346,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl %eax,%fs movl %eax,%gs - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx -#endif movl %edx, %eax shrq $32, %rdx wrmsr diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 2c97bf7b56ae..8707dd07b9ce 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -23,18 +23,10 @@ #include #include -#ifdef CONFIG_X86_64 -#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) -#else -#define BOOT_PERCPU_OFFSET 0 -#endif - -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET; +DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); -unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init = { - [0 ... NR_CPUS-1] = BOOT_PERCPU_OFFSET, -}; +unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init; EXPORT_SYMBOL(__per_cpu_offset); /* diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 1239be7cc8d8..57a83fb2d8a0 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -103,12 +103,6 @@ const_pcpu_hot = pcpu_hot; PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ -#ifdef CONFIG_X86_64 -#ifdef CONFIG_SMP - percpu PT_LOAD FLAGS(6); /* RW_ */ -#endif - init PT_LOAD FLAGS(7); /* RWE */ -#endif note PT_NOTE FLAGS(0); /* ___ */ } @@ -224,21 +218,7 @@ SECTIONS __init_begin = .; /* paired with __init_end */ } -#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) - /* - * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the - * output PHDR, so the next output section - .init.text - should - * start another segment - init. - */ - PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) - ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, - "per-CPU data too large - increase CONFIG_PHYSICAL_START") -#endif - INIT_TEXT_SECTION(PAGE_SIZE) -#ifdef CONFIG_X86_64 - :init -#endif /* * Section for code used exclusively before alternatives are run. All @@ -368,9 +348,7 @@ SECTIONS EXIT_DATA } -#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP) PERCPU_SECTION(INTERNODE_CACHE_BYTES) -#endif . = ALIGN(PAGE_SIZE); @@ -508,16 +486,11 @@ SECTIONS * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load +#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); -#ifdef CONFIG_SMP -. = ASSERT((fixed_percpu_data == 0), - "fixed_percpu_data is not at start of per-cpu area"); -#endif - #ifdef CONFIG_CPU_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); #endif diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S index fab90368481f..2ce07dffc314 100644 --- a/arch/x86/platform/pvh/head.S +++ b/arch/x86/platform/pvh/head.S @@ -100,9 +100,8 @@ SYM_CODE_START_LOCAL(pvh_start_xen) * the per cpu areas are set up. */ mov $MSR_GS_BASE,%ecx - lea INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - mov %edx, %eax - shr $32, %rdx + xor %eax, %eax + xor %edx, %edx wrmsr call xen_prepare_pvh diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c index 24ad10c62840..ef355242a8d8 100644 --- a/arch/x86/tools/relocs.c +++ b/arch/x86/tools/relocs.c @@ -822,12 +822,7 @@ static void percpu_init(void) */ static int is_percpu_sym(ElfW(Sym) *sym, const char *symname) { - int shndx = sym_index(sym); - - return (shndx == per_cpu_shndx) && - strcmp(symname, "__init_begin") && - strcmp(symname, "__per_cpu_load") && - strncmp(symname, "init_per_cpu_", 13); + return 0; } @@ -1051,7 +1046,8 @@ static int cmp_relocs(const void *va, const void *vb) static void sort_relocs(struct relocs *r) { - qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs); + if (r->count) + qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs); } static int write32(uint32_t v, FILE *f) diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index 30f27e757354..7e8754c5fa1d 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -51,15 +51,14 @@ SYM_CODE_START(startup_xen) leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax - cdq + xorl %eax, %eax + xorl %edx, %edx wrmsr mov %rsi, %rdi diff --git a/init/Kconfig b/init/Kconfig index 9ffb103fc927..5f2c1f4a16aa 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1718,7 +1718,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default n config KALLSYMS_BASE_RELATIVE bool