From patchwork Thu Nov 2 11:25:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 160828 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp281398vqu; Thu, 2 Nov 2023 04:29:13 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEhxElEoESMre7vT6p6fwvSCYT1yvPaYO5FezC2NlOIJR+/73jXdi31UhqAjn6EUGC7nSOD X-Received: by 2002:a05:6a20:2447:b0:13d:1d14:6693 with SMTP id t7-20020a056a20244700b0013d1d146693mr17355223pzc.45.1698924553471; Thu, 02 Nov 2023 04:29:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698924553; cv=none; d=google.com; s=arc-20160816; b=rgvuuvJ3478tuayVU9PFg2S8lyxydV9EOUODO4p2twVDF+IzHaskmfBCvbJqL6BOr/ 1O4FIjBBi5rmqyFRxMsFo+IWhYcbluStnROa9Wa6ryYh5BnoW5r6G2kBQVTq6cBSsXAa dP3pN5DW1+RJbIto2A/p5Kd6BZ1xseFnu9zh6F9XgMckQ/9EHdhNbqgcF6z5hgAfhh08 qw55fzJNH4M8kjCxyB28cLSt7pcQltLrc8j8LQ+1GTjIp6hNiSvXyGnFuKfJKmaJnXvA acCsr60sNy1kikKYUH2tZ3OVtKb6Pf8Rehg1+QcC/CHeAN0ERdqBrsU4xc3dRAyEp7b2 4qFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nK9f/0DyWoovHMW6tPJeem6iX8A0uEcMjFuwkUDDCuM=; fh=sWK7YOurtkTyaE7aaZtytOmSvUDe8vhU+5hRY6IWLIo=; b=NGyLdQHyyKT/4r1cgrndvtByXS2swlnNYYa9j9FznSwDWH9Z3YM6xspYPaSm42JYw+ o4xBWAijlgZg3OqoIBhdpF8IAN5tZEi5vdxkBUht7e+y4TJZ6TQX3IE47/ibT5AE5z00 CbrpLygwZH840N8HOGCzFxDdp7l3PbGE+gwVisByAJjkxft/jNHTqrIwlE+2DWJlWf6h CQ3boo/n/hZjuS2pbNimWJnwtZ7D4gTeHa0uGDQVkAXj7wEBpaLsa+VZfGa80m4q62OI ZMJ2+jnfMJwgTFr5ShwhjmHGYgDDqP4DpeNSKl4fu2QTQf2J8ytt76pnU1rhAHgYqMty SYrQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="Ab/KOJpQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id s11-20020a170902ea0b00b001c62b51ac0fsi5106341plg.306.2023.11.02.04.29.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="Ab/KOJpQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7B2FC82159F8; Thu, 2 Nov 2023 04:29:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376269AbjKBL3J (ORCPT + 36 others); Thu, 2 Nov 2023 07:29:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbjKBL3H (ORCPT ); Thu, 2 Nov 2023 07:29:07 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93093131 for ; Thu, 2 Nov 2023 04:29:04 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-307d58b3efbso464807f8f.0 for ; Thu, 02 Nov 2023 04:29:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698924543; x=1699529343; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nK9f/0DyWoovHMW6tPJeem6iX8A0uEcMjFuwkUDDCuM=; b=Ab/KOJpQwGaxoUiUb8senjwYjA7Qkzd6BAtIQkeTbwMIGyayD9csK9X+QmFG1/H+D4 75v1KAFY/7WVX9isuTarnUdFIg0NyjJ16GcZHiA3dNi/jIZGPPeyR+A/mxEhvCUQDZm5 H3Kcy4091V9U3USTHPvVJE/zVC8ZqcIGclUHuujqwYT8WVUE3aOHA7mUknQGZYRzRZss epsw6wOx1Yzk9y2zZZr+f3UrTHS8zVl2X3bJSfwo1UOuaj493D9UAeMYl/vWPC98HJss yMlegsJbDFzc7Zf6O8rPvqQxS9d64ZOZQgvIuDsAo3wY1SfUTs3si0XAUprciEYT/Kxg v/nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698924543; x=1699529343; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nK9f/0DyWoovHMW6tPJeem6iX8A0uEcMjFuwkUDDCuM=; b=JGMn1A3XZ9zxIh8qCX8eQxZZb6bNihbz7yu8aHQkkN4k/TTeXEmDcgPB2UAkFbo0JQ ZO2IEmCd1mUGpWwel00hkom2VWtP0RkGENUUrDA8edLOANmS7L322xDFlLzZdVIVyW4R L072+vXRJN4V5+TNDniSgLlNBf6XI++STGthLgDpVuM+eCE90EEDDKxWPvcEWesZ+wxh wsquh8eCDQBv3BFM4uiDKt7PHW1ejZqIC7n4Ua/q2tQzehr5F1NGPN/Cs5xjqbn54ZoV yXzFgFfGhBtoSv7uPrkI9vlUE182eJaxCXyZFaz3hEpSonop9F6mzPbRoC4TB+POD7yR /pPQ== X-Gm-Message-State: AOJu0YxWvm9bK5xME8ydV90O4BIiWAdokisHSMI507fDjk1GvI/XK4Xl e24GbwiJhfhE+1YirwzuQ9A= X-Received: by 2002:adf:f646:0:b0:31f:db1b:7296 with SMTP id x6-20020adff646000000b0031fdb1b7296mr11670913wrp.21.1698924542495; Thu, 02 Nov 2023 04:29:02 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id d10-20020a5d4f8a000000b003232f167df5sm2168227wru.108.2023.11.02.04.29.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:02 -0700 (PDT) From: Uros Bizjak To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Peter Zijlstra Subject: [PATCH 1/3] x86/callthunks: Move call thunk template to .S file Date: Thu, 2 Nov 2023 12:25:46 +0100 Message-ID: <20231102112850.3448745-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231102112850.3448745-1-ubizjak@gmail.com> References: <20231102112850.3448745-1-ubizjak@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 02 Nov 2023 04:29:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781451512462109400 X-GMAIL-MSGID: 1781451512462109400 Currently INCREMENT_CALL_DEPTH explicitly defines %gs: segment register prefix for its percpu variable. This is not compatible with !CONFIG_SMP, which requires non-prefixed percpu variables. Move call thunk template to its own callthunks-tmpl.S assembly file where PER_CPU_VAR macro from percpu.h can be used to conditionally use %gs: segment register prefix, depending on CONFIG_SMP. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Peter Zijlstra Signed-off-by: Uros Bizjak --- arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/callthunks-tmpl.S | 11 +++++++++++ arch/x86/kernel/callthunks.c | 10 ---------- 3 files changed, 12 insertions(+), 11 deletions(-) create mode 100644 arch/x86/kernel/callthunks-tmpl.S diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 3269a0e23d3a..6b6b68ef4c3b 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -143,7 +143,7 @@ obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o obj-$(CONFIG_CFI_CLANG) += cfi.o -obj-$(CONFIG_CALL_THUNKS) += callthunks.o +obj-$(CONFIG_CALL_THUNKS) += callthunks.o callthunks-tmpl.o obj-$(CONFIG_X86_CET) += cet.o diff --git a/arch/x86/kernel/callthunks-tmpl.S b/arch/x86/kernel/callthunks-tmpl.S new file mode 100644 index 000000000000..e82c473bd1b1 --- /dev/null +++ b/arch/x86/kernel/callthunks-tmpl.S @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include + + .section .rodata + .global skl_call_thunk_template + .global skl_call_thunk_tail + +skl_call_thunk_template: + INCREMENT_CALL_DEPTH +skl_call_thunk_tail: diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index e9ad518a5003..d0922cf94c90 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -62,16 +62,6 @@ static const struct core_text builtin_coretext = { .name = "builtin", }; -asm ( - ".pushsection .rodata \n" - ".global skl_call_thunk_template \n" - "skl_call_thunk_template: \n" - __stringify(INCREMENT_CALL_DEPTH)" \n" - ".global skl_call_thunk_tail \n" - "skl_call_thunk_tail: \n" - ".popsection \n" -); - extern u8 skl_call_thunk_template[]; extern u8 skl_call_thunk_tail[]; From patchwork Thu Nov 2 11:25:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 160829 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp281440vqu; Thu, 2 Nov 2023 04:29:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IExnU2QfFQfOtrFI1kv0ARq3WqtEZy0hhWQkT3oXCMF93tUEH2IWN9fPyeytUv/dS14FobT X-Received: by 2002:a17:90a:f48d:b0:280:46ac:be71 with SMTP id bx13-20020a17090af48d00b0028046acbe71mr11307127pjb.15.1698924559208; Thu, 02 Nov 2023 04:29:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698924559; cv=none; d=google.com; s=arc-20160816; b=ApDyzig0qJArNckTZOLTWOxjV1w4f+RiGQNVDZ78b5jIVIGJWdY7TCvOlE14JsxvFq eVgSstse08ibZei+dMZZEMz2IoRxkmQo0qIRFjB+JfkPzvJJr9tmNsW3iHWsWNCPF7Fv dh3b9GzAB9gnEOntxt9hBoRF8YPd+zR1sTgXCp72fY+9541MeMfHWcWCV0+xqmU7nTeW PPlguRms7iYd9Xdt+8+j0scKb/58R6OXVKsBpvZB/fkI3H/TjcnhKEyKLEar2q8CNchA 2xLxQZAh3vINcXIn1Mz+4WIecTDPO43NBezyM2WjE7jX3bIq2xWRwXNEnq6FdB82Fwqd Bx5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ENp6FOnR5GB+Ukedmf+eZL7XOs80yvuA3UZNbpkC5yw=; fh=sWK7YOurtkTyaE7aaZtytOmSvUDe8vhU+5hRY6IWLIo=; b=ZFVBAAN40K2zbwzmpxMPYXWjvn0CSNzLZj/XS7aKJ0CF05KZNNq6B/7QZmEszThVhZ IZsfJD5Cpxp0ZFn3306owGe06AGorP/IaUJiGUp5qTv0hRKJjt7JPU7KJ9gJYcGJz4DU jc4BtekEWPN5dhmf532IgB4qeFH+xI+wcmctLbn/9fNt0WlPg361Que3KtoZNJTKLqE3 hNae+fsrRj9xsIolzWbJuPG3gkPay2UzCi/xUFtXuUvJkwo3TdOOfYp8ybA8m7RdiKF8 H24Beb9np5gTS6qjGZhiecewJ9Nfr1tpj08WpxGyv3SmQ03iCeDoNyMNHI8Lg3Zh9bVL /7fA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=G1tZ92eN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id e1-20020a17090a728100b0027d15abe599si2641520pjg.143.2023.11.02.04.29.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=G1tZ92eN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 6105682159EE; Thu, 2 Nov 2023 04:29:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376286AbjKBL3M (ORCPT + 36 others); Thu, 2 Nov 2023 07:29:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347360AbjKBL3I (ORCPT ); Thu, 2 Nov 2023 07:29:08 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19BD3132 for ; Thu, 2 Nov 2023 04:29:05 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-40838915cecso6059665e9.2 for ; Thu, 02 Nov 2023 04:29:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698924543; x=1699529343; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ENp6FOnR5GB+Ukedmf+eZL7XOs80yvuA3UZNbpkC5yw=; b=G1tZ92eN5IySeufq2ZBdtSRSw7mWs+skrMYclDOcRVZe4WsIwizZ5r65TSheopW/sp HXvIL9OYaFr30xNuv2Os3ESCHp2FTJx6UpIIuPrlvdotPMOYEh8FOOELKRwwIyvBA0o7 coLu3P+wbbOBhTdmtV1H+6V283ooP3/3Uxe0eVYAkJUimEuOIi3e30iEdCOEhb6DS6Wv j/1ha5z10KRtlLh8tbUEM6yrvw2VW43oo+75DdNeCDkLSL78YHnWFIUEbnFUhxLnyThc LktOqAB6avs5GESbdtpsHZxojtIwzldTkXDa6tLiNSL6iukB1K2dFFs967EdnSXYkOQ1 uiAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698924543; x=1699529343; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ENp6FOnR5GB+Ukedmf+eZL7XOs80yvuA3UZNbpkC5yw=; b=Rz2N3PSEXJX6emBREhE67BwCZv2hoaemZ8v/OXaSOfwRQ/COtSINQ0CPmCP9WUA3GC 1pNhuNHAcw3tWmx6HmwvNJ/Hcw85ZKZJBiCrGVfKT8cZvLHbCFdARyZiYaVwZnpfSffZ VnWOCcnu7nVi4KGBmdw3HoK4CO4GmlV4yWPX1L0kG0TeqKIVgibOKp+3ba4wSyCvLpqm bsvvADKhJu3IhSpNEanOE5XH/CCj7Kmh1DH3CtP9/R+8E1Jv3eRVHibZxXKE2Fcy8L17 jKrqF2Nkr1eXFgUZT/uRM7okCCdDAEdVyYX4TiETK3BTtSZbnzfKzRv0cClgkE+17+5K dtKQ== X-Gm-Message-State: AOJu0YyUaAkwtD6CG3xgA3kWcRdrFy4KNGrjklgdf3vBys9qtOlO4nrA 3FKi32NWK6ScJOo9plYHUcU= X-Received: by 2002:a05:6000:686:b0:32f:9ac4:6f29 with SMTP id bo6-20020a056000068600b0032f9ac46f29mr6281726wrb.70.1698924543282; Thu, 02 Nov 2023 04:29:03 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id d10-20020a5d4f8a000000b003232f167df5sm2168227wru.108.2023.11.02.04.29.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:03 -0700 (PDT) From: Uros Bizjak To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Peter Zijlstra Subject: [PATCH 2/3] x86/callthunks: Handle %rip-relative relocations in call thunk template Date: Thu, 2 Nov 2023 12:25:47 +0100 Message-ID: <20231102112850.3448745-3-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231102112850.3448745-1-ubizjak@gmail.com> References: <20231102112850.3448745-1-ubizjak@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 02 Nov 2023 04:29:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781451518685544680 X-GMAIL-MSGID: 1781451518685544680 Contrary to alternatives, relocations are currently not supported in call thunk templates. Implement minimal support for relocations when copying template from its storage location to handle %rip-relative addresses. Support for relocations will be needed when PER_CPU_VAR macro switches to %rip-relative addressing. The patch allows unification of ASM_INCREMENT_CALL_DEPTH, which already uses PER_CPU_VAR macro, with INCREMENT_CALL_DEPTH, used in call thunk template, which is currently limited to use absolute address. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Peter Zijlstra Signed-off-by: Uros Bizjak --- arch/x86/kernel/callthunks.c | 63 ++++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index d0922cf94c90..bda09d82bff7 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -24,6 +24,8 @@ static int __initdata_or_module debug_callthunks; +#define MAX_PATCH_LEN (255-1) + #define prdbg(fmt, args...) \ do { \ if (debug_callthunks) \ @@ -166,13 +168,51 @@ static const u8 nops[] = { 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, }; +#define apply_reloc_n(n_, p_, d_) \ + do { \ + s32 v = *(s##n_ *)(p_); \ + v += (d_); \ + BUG_ON((v >> 31) != (v >> (n_-1))); \ + *(s##n_ *)(p_) = (s##n_)v; \ + } while (0) + +static __always_inline +void apply_reloc(int n, void *ptr, uintptr_t diff) +{ + switch (n) { + case 4: apply_reloc_n(32, ptr, diff); break; + default: BUG(); + } +} + +static void apply_relocation(u8 *buf, size_t len, u8 *dest, u8 *src) +{ + for (int next, i = 0; i < len; i = next) { + struct insn insn; + + if (WARN_ON_ONCE(insn_decode_kernel(&insn, &buf[i]))) + return; + + next = i + insn.length; + + if (insn_rip_relative(&insn)) + apply_reloc(insn.displacement.nbytes, + buf + i + insn_offset_displacement(&insn), + src - dest); + } +} + static void *patch_dest(void *dest, bool direct) { unsigned int tsize = SKL_TMPL_SIZE; + u8 insn_buff[MAX_PATCH_LEN]; u8 *pad = dest - tsize; + memcpy(insn_buff, skl_call_thunk_template, tsize); + apply_relocation(insn_buff, tsize, pad, skl_call_thunk_template); + /* Already patched? */ - if (!bcmp(pad, skl_call_thunk_template, tsize)) + if (!bcmp(pad, insn_buff, tsize)) return pad; /* Ensure there are nops */ @@ -182,9 +222,9 @@ static void *patch_dest(void *dest, bool direct) } if (direct) - memcpy(pad, skl_call_thunk_template, tsize); + memcpy(pad, insn_buff, tsize); else - text_poke_copy_locked(pad, skl_call_thunk_template, tsize, true); + text_poke_copy_locked(pad, insn_buff, tsize, true); return pad; } @@ -281,20 +321,26 @@ void *callthunks_translate_call_dest(void *dest) static bool is_callthunk(void *addr) { unsigned int tmpl_size = SKL_TMPL_SIZE; - void *tmpl = skl_call_thunk_template; + u8 insn_buff[MAX_PATCH_LEN]; unsigned long dest; + u8 *pad; dest = roundup((unsigned long)addr, CONFIG_FUNCTION_ALIGNMENT); if (!thunks_initialized || skip_addr((void *)dest)) return false; - return !bcmp((void *)(dest - tmpl_size), tmpl, tmpl_size); + *pad = dest - tmpl_size; + + memcpy(insn_buff, skl_call_thunk_template, tmpl_size); + apply_relocation(insn_buff, tmpl_size, pad, skl_call_thunk_template); + + return !bcmp(pad, insn_buff, tmpl_size); } int x86_call_depth_emit_accounting(u8 **pprog, void *func) { unsigned int tmpl_size = SKL_TMPL_SIZE; - void *tmpl = skl_call_thunk_template; + u8 insn_buff[MAX_PATCH_LEN]; if (!thunks_initialized) return 0; @@ -303,7 +349,10 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *func) if (func && is_callthunk(func)) return 0; - memcpy(*pprog, tmpl, tmpl_size); + memcpy(insn_buff, skl_call_thunk_template, tmpl_size); + apply_relocation(insn_buff, tmpl_size, *pprog, skl_call_thunk_template); + + memcpy(*pprog, insn_buff, tmpl_size); *pprog += tmpl_size; return tmpl_size; } From patchwork Thu Nov 2 11:25:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 160830 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp281451vqu; Thu, 2 Nov 2023 04:29:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEMgJ03SbsbnxXhVfcv3T+axdzLFU/i0XpaqCkVshLUoq+wOxi+0gEcFcFo5+lhiiS2lqNW X-Received: by 2002:a05:6358:71b:b0:168:ff1b:8f59 with SMTP id e27-20020a056358071b00b00168ff1b8f59mr25222781rwj.4.1698924561992; Thu, 02 Nov 2023 04:29:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698924561; cv=none; d=google.com; s=arc-20160816; b=EYRb3Y6R27TZATsFFfXTJnjCFb81t6pQjU1yk70G7zMUWUvLNNip1fTCwj05alLkaY +7H9t4uGgG3Jj23CzG2A+TUG9KNpVljIU7ioK1gxygGRZUBuaXFwxRzFgz+DM+V8xIME r/98cpxGYj6eZZgXWV88ql7hjDhHL8+Ay5bo/AW5UroRmT45Z2Xf0eWPj7KIpzHvVvml D8d5zNPsHrG4VQY506bywaz91s6vhEC55ZO+jWXU36muibnGFLmxwF0/kCdAwdNRwUsV h+USptuP0czzJ1WlU16pK0c72Agx3fZd1sBZVHvOnD5NtTMDv7C88psdMOjZD89eyQ99 zOlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tgAqePGVhUCu3U56kTFLSJugpfg6qt/lSZ7uytS2Gf4=; fh=sWK7YOurtkTyaE7aaZtytOmSvUDe8vhU+5hRY6IWLIo=; b=ZxsU020KrxMPgv4FAn9yM2/A5+D+nhh49lV4SrHBpL93hq0rrertRizjxE01HLiJsG g/XCq0rDxMpY9ojiB0kbM2l4BHcJqKHUdV9NpsA/rg0z9zX5FhW5A/r+mabl5kfDRD4I GHg7TWJ7xsUv7f1KG4eu8ZL3wpJWKVclXr2DJ3XrzJPFvjdo8yZfb4RLI8Fs6YvI9Xu8 dXh7zeGDthXnOUovPFAr/bdY/Orp3n4kL52ixuISqBtnciumRP+GMe22rFFcgGbXUfdP QvrtVe+M5S9WTtzK9oZ2DIHcDXcfywYBCt47eATs+rxq4VzZFmVQa5IyZEn/JuXdgpRo piIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=BQb08mpb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id c4-20020a656744000000b005b8f0c8ddbasi1876414pgu.243.2023.11.02.04.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=BQb08mpb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 109EB82164CC; Thu, 2 Nov 2023 04:29:21 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376307AbjKBL3P (ORCPT + 36 others); Thu, 2 Nov 2023 07:29:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376270AbjKBL3J (ORCPT ); Thu, 2 Nov 2023 07:29:09 -0400 Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10578111 for ; Thu, 2 Nov 2023 04:29:07 -0700 (PDT) Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-507973f3b65so984015e87.3 for ; Thu, 02 Nov 2023 04:29:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698924545; x=1699529345; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tgAqePGVhUCu3U56kTFLSJugpfg6qt/lSZ7uytS2Gf4=; b=BQb08mpbYh4FwfaddGcZDs6LMGQnnXbzI0le3TBrHcv1i3VvxZ6K4o2jAR9Z81J//U OzR5vtBq+WdIuym23V3smYHtt+mrRKJ012WMHImhqZ9pqo3fUp8gOoiPeqbfS8tEdZOe YiJqlTlegHZkL2SYFi/jpdcvnU/cC26Tts7euW0wYEmE50bQ0KG1SXDdmnfttrjopwlu ZGpzCRQ6/e7a62wj19wkm6oHOxE0GhipaQwJ5w/OU9L8oCKxS/wanYGP8m/5XlTHFSb7 hxPzYr++/0wRXmTuHQ1hnV40jgLrq4qqZ0v7S578+ZiY8rRpe2qddCF4EkWybRak/Dyo WZow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698924545; x=1699529345; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tgAqePGVhUCu3U56kTFLSJugpfg6qt/lSZ7uytS2Gf4=; b=kVcUYyi3QCMHygkg8XU72NlXW8AmSc2iHX1cN1+LguvIifXgFNdUtBvZauT1QqLu6H Vv3kyfibAe0EH8IrKrEeyxWp3/CUvdiX79iTSRCKa6c9P+lyvNM5mXeSyKwYUpanwDaw c5a1lq0Te8cTXihda64h9h3lTcIxU9iRPYkkHyqUG4iyJksM4cso3BRRjIDrJLqMv7+z /wAMp1ULU4tH8C4W+cN3TAW70Kyq9He2/vAAB1fdSESq6nIJn2GV8ZVD08uEe7H5vnwN jhDOwfvTh6rblQJKKsK5cKzI63C2lEdc9sHhRyI24YBvuNljS09ka4tK9GTxiqj+VNDt PMxw== X-Gm-Message-State: AOJu0YxaiEzqDHN4BMupSiBiZamZwZWeIYqEN/rgod60DpBInJzdfztR efXaTBdXlr4UkQ6YMtPwapk= X-Received: by 2002:a05:6512:2248:b0:4fe:7e7f:1328 with SMTP id i8-20020a056512224800b004fe7e7f1328mr17608450lfu.16.1698924544906; Thu, 02 Nov 2023 04:29:04 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id d10-20020a5d4f8a000000b003232f167df5sm2168227wru.108.2023.11.02.04.29.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Nov 2023 04:29:04 -0700 (PDT) From: Uros Bizjak To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Peter Zijlstra Subject: [PATCH 3/3] x86/callthunks: Fix and unify call thunks assembly snippets Date: Thu, 2 Nov 2023 12:25:48 +0100 Message-ID: <20231102112850.3448745-4-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231102112850.3448745-1-ubizjak@gmail.com> References: <20231102112850.3448745-1-ubizjak@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 02 Nov 2023 04:29:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781451521862565292 X-GMAIL-MSGID: 1781451521862565292 Currently thunk debug macros explicitly define %gs: segment register prefix for their percpu variables. This is not compatible with !CONFIG_SMP, which requires non-prefixed percpu variables. Fix call thunks debug macros to use PER_CPU_VAR macro from percpu.h to conditionally use %gs: segment register prefix, depending on CONFIG_SMP. Finally, unify ASM_ prefixed assembly macros with their non-prefixed variants. With support of %rip-relative relocations in place, call thunk templates allow %rip-relative addressing, so unified assembly snippet can be used everywhere. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Peter Zijlstra Signed-off-by: Uros Bizjak --- arch/x86/include/asm/nospec-branch.h | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index f93e9b96927a..6f677be6bdb9 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -59,13 +59,13 @@ #ifdef CONFIG_CALL_THUNKS_DEBUG # define CALL_THUNKS_DEBUG_INC_CALLS \ - incq %gs:__x86_call_count; + incq PER_CPU_VAR(__x86_call_count); # define CALL_THUNKS_DEBUG_INC_RETS \ - incq %gs:__x86_ret_count; + incq PER_CPU_VAR(__x86_ret_count); # define CALL_THUNKS_DEBUG_INC_STUFFS \ - incq %gs:__x86_stuffs_count; + incq PER_CPU_VAR(__x86_stuffs_count); # define CALL_THUNKS_DEBUG_INC_CTXSW \ - incq %gs:__x86_ctxsw_count; + incq PER_CPU_VAR(__x86_ctxsw_count); #else # define CALL_THUNKS_DEBUG_INC_CALLS # define CALL_THUNKS_DEBUG_INC_RETS @@ -80,9 +80,6 @@ #define CREDIT_CALL_DEPTH \ movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); -#define ASM_CREDIT_CALL_DEPTH \ - movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); - #define RESET_CALL_DEPTH \ xor %eax, %eax; \ bts $63, %rax; \ @@ -95,20 +92,14 @@ CALL_THUNKS_DEBUG_INC_CALLS #define INCREMENT_CALL_DEPTH \ - sarq $5, %gs:pcpu_hot + X86_call_depth; \ - CALL_THUNKS_DEBUG_INC_CALLS - -#define ASM_INCREMENT_CALL_DEPTH \ sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS #else #define CREDIT_CALL_DEPTH -#define ASM_CREDIT_CALL_DEPTH #define RESET_CALL_DEPTH -#define INCREMENT_CALL_DEPTH -#define ASM_INCREMENT_CALL_DEPTH #define RESET_CALL_DEPTH_FROM_CALL +#define INCREMENT_CALL_DEPTH #endif /* @@ -158,7 +149,7 @@ jnz 771b; \ /* barrier for jnz misprediction */ \ lfence; \ - ASM_CREDIT_CALL_DEPTH \ + CREDIT_CALL_DEPTH \ CALL_THUNKS_DEBUG_INC_CTXSW #else /* @@ -311,7 +302,7 @@ .macro CALL_DEPTH_ACCOUNT #ifdef CONFIG_CALL_DEPTH_TRACKING ALTERNATIVE "", \ - __stringify(ASM_INCREMENT_CALL_DEPTH), X86_FEATURE_CALL_DEPTH + __stringify(INCREMENT_CALL_DEPTH), X86_FEATURE_CALL_DEPTH #endif .endm