From patchwork Tue Dec 13 06:09:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32640 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2657453wrr; Mon, 12 Dec 2022 22:10:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf4k+St3ND9xZOUW5qcS1FqEfU+xCxn97CNQZIEPmRGIEMQGWgoacQIra1j+3wOXIGD7nmEK X-Received: by 2002:a05:6402:1f85:b0:467:b851:6066 with SMTP id c5-20020a0564021f8500b00467b8516066mr23310201edc.6.1670911844127; Mon, 12 Dec 2022 22:10:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911844; cv=none; d=google.com; s=arc-20160816; b=FeXWNDrk7oZJwmQk9ewB/n1z9G7P72B6d6PHYycNNJaf8LtpYhRrLmQMLn6ACkqX6k HfjremMIyCZEk6UbCXHjDm1EOhXwLTVbVhBJ8z/oYuDlCCNslUS08CU7vmvDDMNO/Hj9 3WgVTAgNz2/nFLppz3gzpaxOeqnijKDjMyHJ9WiVcviohOceqoTeRjeU179HyvXTTVCm eJ2+3p8S6g30CGKIqXk/3bzgK2qXSUXDgyYcpPfWNrK4gV1l6a5WiYgsY+w5qTO0AtfO PibkjHxDGEwCUwPHyAVRu3V5udfyNd5R3W7GOxlekxY94XxnmNU15/qC26GqcI39AaUO SwGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=+xrIAhvGbtHzogIJ3aNRRLQUShQPCNp9dtUJo01jRd4=; b=ZgIydpURQzXWTWwUAqsWwJJqe6PZ/Qc6efMIHNI2bQdAJ/gfj9L+3ImoQ3wwaXUIhb XWqQCHSlVg5/MipgEe60eRfMC5HddN6lKAlpf0ZPjkQdjVbbL6aqB/cN0OqQgY/FKCL/ 6kdpDxKKBLG+VuqRv8KALb1PYOiZQ7FOGGEyVJssZW2d7jxK5dgA4mxfUmkkRc41s6v1 sfgrRZngxqOtC+Fu9YXcomY8/0BO7hT8WW/OS0OXnGQf73/i8nRE3xPmkD1tSIXGRdKB 9UTxbOvNFiB2P07VGd2TUi7ddokC/P91Q21fJ7otYuTTkG7bWcNgDDippuwAS1IS2vlB AvAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=TGHdJLGE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f22-20020aa7d856000000b00462c7bbc03esi8652087eds.114.2022.12.12.22.10.20; Mon, 12 Dec 2022 22:10:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=TGHdJLGE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234404AbiLMGJ1 (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234095AbiLMGJU (ORCPT ); Tue, 13 Dec 2022 01:09:20 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D8C91A832 for ; Mon, 12 Dec 2022 22:09:19 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-3b0af5bcbd3so157025607b3.0 for ; Mon, 12 Dec 2022 22:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+xrIAhvGbtHzogIJ3aNRRLQUShQPCNp9dtUJo01jRd4=; b=TGHdJLGEzqgYr9cAqGjdS3I4g33rP9ZZMIsLVLRpNw5QthQ5hNOCOIbR2FnxmU3emT BG0VoYoi8EQYjr9Ps/mEbzLbYgXxhMaaGFakXoyRnMg7exBdOpMv/P2grFeziK1ZZ+t8 krjqRIC9W5rOMjaeNVgM7X/8G/PrWxPeTiBjtTYuyrTCIYBLEHrpZ1RHKvivzqBnwNRl Pyc9obee24XNzNAwLDVLPjAq7t+STbiLwQvNSJJf/a4VdYdDZ+nos6y2MLX9w775lzYX ScnLoAgHVKPv4imTnqHRhUte42MDWnoAkRUPvRfADnr8YMZIWKMm2gQb4U4AXh2rJuyo chfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+xrIAhvGbtHzogIJ3aNRRLQUShQPCNp9dtUJo01jRd4=; b=KKNnPrKtg5byJ93e8fSbKk7cisiy/SjlYCKp+hVJJfhO15PdOgz54DrdeRgsj4syn5 cV4haJU0SsNc8/J9f8FoK+0HruNRSxpGgaKJRkmUvSyrT6KhO3PpOrsQVIjM3gz7fAxG mQGiIViuqXWgqz9YL4KCH4xH2ZH7jjniCDdGUhJEYL78PJNvVOKjfug/4Z6xQBpuedRI xyMfNUQwuAy+TbhVHPa/kOj+tn4IAaiXYWbFxEJpbiPXKyDb4JHWAmrdDXzPrgdBrD68 4tZhhkpSkH8gc+hEIEloUyY1dPYhzlwKqMB34CutIZBut9CkBL5dlFIbczyncfcuOKni GFbg== X-Gm-Message-State: ANoB5pm4XqW11CTGscsPHfWBTxzmT84FVpYtnpTZ+9vL5/QUuxkjrt3M nRYvq8iGaaCyUICFBU8Eqo51gfD1eZI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:1683:0:b0:6f5:6a39:978e with SMTP id 125-20020a251683000000b006f56a39978emr52138384ybw.6.1670911758316; Mon, 12 Dec 2022 22:09:18 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:06 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-2-seanjc@google.com> Subject: [PATCH 1/7] KVM: x86: Make vmx_get_exit_qual() and vmx_get_intr_info() noinstr-friendly From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078057940051963?= X-GMAIL-MSGID: =?utf-8?q?1752078057940051963?= Add an extra special noinstr-friendly helper to test+mark a "register" available and use it when caching vmcs.EXIT_QUALIFICATION and vmcs.VM_EXIT_INTR_INFO. Make the caching helpers __always_inline too so that they can be used in noinstr functions. A future fix will move VMX's handling of NMI exits into the noinstr vmx_vcpu_enter_exit() so that the NMI is processed before any kind of instrumentation can trigger a fault and thus IRET, i.e. so that KVM doesn't invoke the NMI handler with NMIs enabled. Cc: Peter Zijlstra Signed-off-by: Sean Christopherson --- arch/x86/kvm/kvm_cache_regs.h | 12 ++++++++++++ arch/x86/kvm/vmx/vmx.h | 14 ++++++-------- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index c09174f73a34..4c91f626c058 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -75,6 +75,18 @@ static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); } +/* + * kvm_register_test_and_mark_available() is a special snowflake that uses an + * arch bitop directly to avoid the explicit instrumentation that comes with + * the generic bitops. This allows code that cannot be instrumented (noinstr + * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registers. + */ +static __always_inline bool kvm_register_test_and_mark_available(struct kvm_vcpu *vcpu, + enum kvm_reg reg) +{ + return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); +} + /* * The "raw" register helpers are only for cases where the full 64 bits of a * register are read/written irrespective of current vCPU mode. In other words, diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index a3da84f4ea45..bb720a2f11ab 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -669,25 +669,23 @@ void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu); int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu); void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu); -static inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu) +static __always_inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); - if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1); + if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) vmx->exit_qualification = vmcs_readl(EXIT_QUALIFICATION); - } + return vmx->exit_qualification; } -static inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) +static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); - if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2); + if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO); - } + return vmx->exit_intr_info; } From patchwork Tue Dec 13 06:09:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32645 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2657923wrr; Mon, 12 Dec 2022 22:12:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf6IgOTt77zrjHEsB5wJ3VvZrQnGxiOgypSOEwJreKgJ2hXs/iHEVGdslPyur8UHY+2tdTfR X-Received: by 2002:a17:902:6901:b0:188:c395:1756 with SMTP id j1-20020a170902690100b00188c3951756mr19385093plk.41.1670911939530; Mon, 12 Dec 2022 22:12:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911939; cv=none; d=google.com; s=arc-20160816; b=jyMbWrvaQQ0tTT6UY9PaUzUlniuIZ8WY30ES5Mkzd3DhcA3bHzXciDy+muQN0piICH BkxXg1BeReqLe3kl9629q6Co6gIPEiOvxTQtKHmqHm5PFYk3j0Wyz9ATVeK17F6vFLYh fHqgMD0u4JjbS7lIGL62l8e3m5oyVa2kQJGJBIRLv1Cv8RcZlWprqXbbndoC25qbBq7g 7ozuR4qaYcJfC7s+TbEvqZGoJW8nT/kg4AzKbEEzho+TN2sbm4qjKwyOr0bAvWVwR49m vHSnUOqgquPuWgB6FKdFAWLO7yg+9UyXXjAB0CdcvO1A1R1XMwOua2S2pQtjO6TOWsy5 wTOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=3rYplRUH90lferx/1QrjPQLbygHUDtj5qeJE1KQCTqc=; b=OJ0zaG1rBLAphTQZDZQUpQHg1R2thLLNGPHjlX9zLoJb3Xisri5MTw1Mw/LfgRpwLl eg+iOCtrMmcjjPz/rcBdPO6FbSFjlX8k+F3AUvPIn15KAyrP6pCxKdsb2G8/UiJEoNVK Ac7MbLuyX1maOHIYJ0wNcAN6g6TwAoPjKzvyEApCPxoB53/KM5BxKyZbLk5o1oLeL1yr BpHQeYXY6qCFzjdrtmRDYcEARK+bYtTrbM7oDmOOmqTr201GcjUj2Gbt7FQrrmzsS7j0 Q9OVt8XerIEstaAXC7fYwecLtDguk3VGFNUWfb/nsNnX/k+t1+rUG+Bt5ZORBwGeuNNe kEFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="EI40B1d/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t10-20020a170902e84a00b00188b297f39bsi12827659plg.216.2022.12.12.22.12.05; Mon, 12 Dec 2022 22:12:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="EI40B1d/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234374AbiLMGJc (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232252AbiLMGJV (ORCPT ); Tue, 13 Dec 2022 01:09:21 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 332D619286 for ; Mon, 12 Dec 2022 22:09:20 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id ep17-20020a17090ae65100b00219702c495cso1339189pjb.2 for ; Mon, 12 Dec 2022 22:09:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3rYplRUH90lferx/1QrjPQLbygHUDtj5qeJE1KQCTqc=; b=EI40B1d/Mbv8m0VRGcWzxsd7VuKCZkwAPkWVNlQv9uYdsZdudkMdcUKtlj3sz3lzDa LUEvap9m5gVbRVLArHGjcPy1pJ1T0m1vx5tGFB7nzLVu6AbFptybI8XxoFizChEiw11b Xluwh7PxOVZthsV34BGOkV0j7R35fZic+oWBLpklUUJMaeKQh0ePUgTdiBi1QwYG9uQ/ Y8VI8cc/xhlGHau44SRNm3GSt4rYlXVCNL1nR2HMxr47eALh9q6z3n65fdaXu06LCVWV HngCvPGCyXTRA5NAp54D7J80f9dtcUdssrAsbngPyX1B17IdPjhXDZyFtP0PwjqGOjP/ 5dlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3rYplRUH90lferx/1QrjPQLbygHUDtj5qeJE1KQCTqc=; b=hvGRlDqTI6QGpjl01FPxoAWz0h4xA2OiI0eVCFDVWadu/uyajAcBmbiB6eWe8qFX4X bDwmG9tPYOzxs+KgOx76uY8VqIhITGwyySGVAik96KU3ZP6Zm+7NxUG/1MUvcSENxm9D 1y5bFdbrAXWSXAnjNaUC6hz1pfW4ZfjpS5e7v9GMZLVMy0RmWv6idAOhfGnOSzL/UzMB 0ueUedHNSAWsI7rU+LsM1HUi9PQl4g8QKe4crjmYtTahRlnLPZDHU3bSCokISRYtOl3X 7d1LqBX5D7SFw0zCTLq0GFnwqkMc0W/LJ6AFWTCg0rvJtDDSy475QVxNkvIp4+KCgd69 0Sqw== X-Gm-Message-State: ANoB5pngMIQVTDV/KItmC8iADC4yumxhV6RxKVZP3i1NPgBiZ+Lti8Oz sFKwNUDA8bUrKgk1qc8FI+33aFV9zQ0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a62:1991:0:b0:578:5559:d5a5 with SMTP id 139-20020a621991000000b005785559d5a5mr501316pfz.74.1670911759951; Mon, 12 Dec 2022 22:09:19 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:07 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-3-seanjc@google.com> Subject: [PATCH 2/7] KVM: VMX: Allow VM-Fail path of VMREAD helper to be instrumented From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078157713862516?= X-GMAIL-MSGID: =?utf-8?q?1752078157713862516?= Allow instrumentation in the VM-Fail path of __vmcs_readl() so that the helper can be used in noinstr functions, e.g. to get the exit reason in vmx_vcpu_enter_exit() in order to handle NMI VM-Exits in the noinstr section. While allowing instrumentation isn't technically safe, KVM has much bigger problems if VMREAD fails in a noinstr section. Note, all other VMX instructions also allow instrumentation in their VM-Fail paths for similar reasons, VMREAD was simply omitted by commit 3ebccdf373c2 ("x86/kvm/vmx: Move guest enter/exit into .noinstr.text") because VMREAD wasn't used in a noinstr section at the time. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx_ops.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx_ops.h b/arch/x86/kvm/vmx/vmx_ops.h index 842dc898c972..5838489e719b 100644 --- a/arch/x86/kvm/vmx/vmx_ops.h +++ b/arch/x86/kvm/vmx/vmx_ops.h @@ -100,8 +100,10 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) return value; do_fail: + instrumentation_begin(); WARN_ONCE(1, "kvm: vmread failed: field=%lx\n", field); pr_warn_ratelimited("kvm: vmread failed: field=%lx\n", field); + instrumentation_end(); return 0; do_exception: From patchwork Tue Dec 13 06:09:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32641 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2657588wrr; Mon, 12 Dec 2022 22:11:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Ai4j5fPtoeXFYHWqANRGABAQ42vLAEVUUA1AkE4aPBXFPqe/Ormz5pQ8hEu7We3/8/gT6 X-Received: by 2002:a05:6402:1f0a:b0:461:46c7:3617 with SMTP id b10-20020a0564021f0a00b0046146c73617mr13622489edb.30.1670911868409; Mon, 12 Dec 2022 22:11:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911868; cv=none; d=google.com; s=arc-20160816; b=o6GyA5I6XrgvpE5IxTWrnyM7prEkfoeqNoGbHaXnjG2bKV4BKJdxjy3NYnz+OQ8Djc cigWUqqvK86yEGaCepCKHFCunGwRDEo0NnHsgRDSrMV5imzDG9H8dJBSSY7jUHZ4MC8U fOp9442GuJp7SQisStdpoGRHkdbTQIAiDViw65I3BaUG+9ESe0/5AjmtadnAEvXm4AGZ JRsSniOmnS0h/4G6qYMCtigiQ7uD+jRGClsoN77owAviuLc3bj9HxglMsiK9jXB61c6H +y+QWNAP/KAbqDJNqQuqBzRY6yJZS4lUuv/m1URSrhJ/z41adVJF+Za+9z0ATRoqvtlW pgRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=vIlOHydaCh0ao3eFY0wKJPSpCdXFKH/dETpychmunOk=; b=i+R4bZ7kbuHGLM+2HA9JZuHdq3XxHr0wRV1/pSKhyCoJMuauuf5oEaUXJWjAzLg/D2 dNPgYIUv3l8Vg5fchCUw9JwWMOR9B0iZokWuI7VJT1xbM2SpUvXWRJXAPOExs9/gbDhD p4aCpAP3vk1gPo0mLefPB98i/f4fRYjrDTwdrHZDGkhHKHc8m21nUQIoiC00zGwh00Cu AORmm2H0/u1ZwNPKCf3upQpCDQHH26sxy9SK64Pew5RyKU6WuJKqser+JPjNFOmxhPKA xQhv87LlGXLizwmya2AHj0jO6aEQlBhSsDY7fP0I4ALgJXpPHvq9d5Zfwi6AO7/O1BaZ X7Lg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=VPMI9+ZU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hd39-20020a17090796a700b0078dcdbb3e87si9153949ejc.530.2022.12.12.22.10.44; Mon, 12 Dec 2022 22:11:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=VPMI9+ZU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234430AbiLMGJg (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234371AbiLMGJX (ORCPT ); Tue, 13 Dec 2022 01:09:23 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 171BC1AD80 for ; Mon, 12 Dec 2022 22:09:22 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id md9-20020a17090b23c900b00218fa3308a9so1317925pjb.8 for ; Mon, 12 Dec 2022 22:09:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=vIlOHydaCh0ao3eFY0wKJPSpCdXFKH/dETpychmunOk=; b=VPMI9+ZU1BsAAUUeJUhhb/vSZ0K6O7FxmmGujRXdbsYFdCthw2opRpUDUo05tgMl7/ UAXsgfaJIDtdQjv0gQWhKHu7t1Kb3ZKBGBmDp8R+/uaRObMKKI/qIHywslWfBcWugzQn X74V27+dpXTJKwaD3we+FcosDS3oA++Kx3Y5XTdoNLdjVoW+r5wWOm1m3KrNXUi79VnS pQXShGI/HjzdNdcN2abo9e0E6uGUbGF6MR8/oAW1BUnD3OnSR4SgSNWgY3ti+j7/SNLo ffA6rdN9rF8+xRcUjWLRIXuiBPJyLExE6DrezUnktN5GHRdKPbbIcG+J/VJARYXR7x58 lk0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vIlOHydaCh0ao3eFY0wKJPSpCdXFKH/dETpychmunOk=; b=YhVqOJYzqYYWlxBDH5SPtQ7L2gM88fwa7xpBySIQWv6CAvydWDYv6sHkjPMh4epIRs I4cXnOCD1TNgp+Vo08y7ncby5fcosK3FCv77bJM6lwE4t/HWuyy6ja/LUUxltPxOP16V vivV48+1GCjncFbgXVqdPtFOt8c8eJIbxhGDUKsl5vnFvc3h6aGhR0kNFQThRckqqmak hiUEaY9FPEhtD4aKcuti4+dDwfZx9RM3ut7ai9DXBGF+bFjauynlIZvblOVWl0nqUz/0 AXyGa9YrikIA8c9TxgbV8nI8CTMrGiTWSfoXRLIamVir+DPztaD8Kgg0o7WslS4Iws0w J/nw== X-Gm-Message-State: ANoB5plrYGcBWFS7DcMIqWQmmBNCTOvx5pTqqTkucbzA5wAiPHke1BWp iP8xA3lXDLbUFNKRr+1m5uSz17CxQ+4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:181:b0:189:8f11:f2f with SMTP id z1-20020a170903018100b001898f110f2fmr53434946plg.133.1670911761544; Mon, 12 Dec 2022 22:09:21 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:08 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-4-seanjc@google.com> Subject: [PATCH 3/7] KVM: VMX: Always inline eVMCS read/write helpers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078083188581400?= X-GMAIL-MSGID: =?utf-8?q?1752078083188581400?= Tag all evmcs_{read,write}() helpers __always_inline so that they can be freely used in noinstr sections, e.g. to get the VM-Exit reason in vcpu_vmx_enter_exit() (in a future patch). For consistency and to avoid more spot fixes in the future, e.g. see commit 010050a86393 ("x86/kvm: Always inline evmcs_write64()"), tag all accessors even though evmcs_read32() is the only anticipated use case in the near future. In practice, non-KASAN builds are all but guaranteed to inline the helpers anyways. vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x107: call to evmcs_read32() leaves .noinstr.text section Reported-by: kernel test robot Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/hyperv.h | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/vmx/hyperv.h b/arch/x86/kvm/vmx/hyperv.h index 571e7929d14e..3f4049e4f35d 100644 --- a/arch/x86/kvm/vmx/hyperv.h +++ b/arch/x86/kvm/vmx/hyperv.h @@ -136,7 +136,7 @@ static __always_inline void evmcs_write64(unsigned long field, u64 value) current_evmcs->hv_clean_fields &= ~clean_field; } -static inline void evmcs_write32(unsigned long field, u32 value) +static __always_inline void evmcs_write32(unsigned long field, u32 value) { u16 clean_field; int offset = get_evmcs_offset(field, &clean_field); @@ -148,7 +148,7 @@ static inline void evmcs_write32(unsigned long field, u32 value) current_evmcs->hv_clean_fields &= ~clean_field; } -static inline void evmcs_write16(unsigned long field, u16 value) +static __always_inline void evmcs_write16(unsigned long field, u16 value) { u16 clean_field; int offset = get_evmcs_offset(field, &clean_field); @@ -160,7 +160,7 @@ static inline void evmcs_write16(unsigned long field, u16 value) current_evmcs->hv_clean_fields &= ~clean_field; } -static inline u64 evmcs_read64(unsigned long field) +static __always_inline u64 evmcs_read64(unsigned long field) { int offset = get_evmcs_offset(field, NULL); @@ -170,7 +170,7 @@ static inline u64 evmcs_read64(unsigned long field) return *(u64 *)((char *)current_evmcs + offset); } -static inline u32 evmcs_read32(unsigned long field) +static __always_inline u32 evmcs_read32(unsigned long field) { int offset = get_evmcs_offset(field, NULL); @@ -180,7 +180,7 @@ static inline u32 evmcs_read32(unsigned long field) return *(u32 *)((char *)current_evmcs + offset); } -static inline u16 evmcs_read16(unsigned long field) +static __always_inline u16 evmcs_read16(unsigned long field) { int offset = get_evmcs_offset(field, NULL); @@ -213,11 +213,11 @@ static inline void evmcs_load(u64 phys_addr) #else /* !IS_ENABLED(CONFIG_HYPERV) */ static __always_inline void evmcs_write64(unsigned long field, u64 value) {} -static inline void evmcs_write32(unsigned long field, u32 value) {} -static inline void evmcs_write16(unsigned long field, u16 value) {} -static inline u64 evmcs_read64(unsigned long field) { return 0; } -static inline u32 evmcs_read32(unsigned long field) { return 0; } -static inline u16 evmcs_read16(unsigned long field) { return 0; } +static __always_inline void evmcs_write32(unsigned long field, u32 value) {} +static __always_inline void evmcs_write16(unsigned long field, u16 value) {} +static __always_inline u64 evmcs_read64(unsigned long field) { return 0; } +static __always_inline u32 evmcs_read32(unsigned long field) { return 0; } +static __always_inline u16 evmcs_read16(unsigned long field) { return 0; } static inline void evmcs_load(u64 phys_addr) {} static inline void evmcs_touch_msr_bitmap(void) {} #endif /* IS_ENABLED(CONFIG_HYPERV) */ From patchwork Tue Dec 13 06:09:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32644 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2657747wrr; Mon, 12 Dec 2022 22:11:42 -0800 (PST) X-Google-Smtp-Source: AA0mqf4JDYEj+YZrtNQGX3tmD5cehdBKRxDrlHILH9zaNRoLAymTMvQNRuDY/MNNuSkXKHcK6PRA X-Received: by 2002:a17:906:7116:b0:7c1:8450:f963 with SMTP id x22-20020a170906711600b007c18450f963mr3914977ejj.34.1670911902397; Mon, 12 Dec 2022 22:11:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911902; cv=none; d=google.com; s=arc-20160816; b=N153/CSS/O9l7WF+M3tDkJzljBBqy7l7O/U158lObZZ2JyO8U+QL/9OGj/E+xF28v8 r365jBTiHl3tFLTniJycAikWBprkJoLZwOwMwbM39mryvnZf5TKK7ZDHZEKgDpvRS58l LZVKqXzZs/jyKVfB99K0N1d8cC3ACBWADuiUqsLR+drrwk85q5YJxPsiwwpLBsOEzjOH bSvQ0KzGfSg8UVZ8H6AJ85O1IRNQGOyKQX/ARlYOquYv6H0aW+atojAsxYjmEGc+JGJj cLEo+apZkpxBGacW60sQpAxE/B0aV8WCGQns9dws/4dGVDi5A4VEaR/5YHedKxSsSBNB MOuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=NFEhFRkg0+nIHx94jx/VqF2L4sTiUmledWciOk7WPsw=; b=Dwlv+38A4ccV1uItj1OcYeuU18ncOoiYi+WkqhXkcmRtI3AkTrRNJiaEIlz6Wc/t/a V0wsSUDwffnN6Y+sjF5t2vdpvKNjbYCT2IKIxL5xpoxQuzJQeKKS/S0bc9gcqmKo/pDy RenocjuF7dXmOVe6ZsuC8kBzBvbLuSeCueR5wB1WbMYU7+q7F0Ju2DV6K0Se/tuxn/8Y o73ASMCwUFbFW9fQ08KnCAxN7kiSKY4mR3H/wUF4t2hrB2cJLSflJVx61yHP2bGdYHlK QS34Jp1VHMyGEbwjLLEvZ6Eb01J3Xhlb57AHrqfysDH6DzUX97GqLoQt9NUHjyZXbcnE 5bZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Mb6tzs3Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sh37-20020a1709076ea500b0078d2f0bf546si917270ejc.186.2022.12.12.22.11.19; Mon, 12 Dec 2022 22:11:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Mb6tzs3Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234444AbiLMGJk (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234391AbiLMGJY (ORCPT ); Tue, 13 Dec 2022 01:09:24 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EC781A809 for ; Mon, 12 Dec 2022 22:09:23 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id cp23-20020a056a00349700b005775c52dbceso1406303pfb.21 for ; Mon, 12 Dec 2022 22:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=NFEhFRkg0+nIHx94jx/VqF2L4sTiUmledWciOk7WPsw=; b=Mb6tzs3YFjNk5QMuKsNyQaWTpQK6Q7lsJPdTl4vbogpOKDC+ntZV7Jpct4rB9qejgS +TtDI14OD1/tbbyL0L1wge29Emn1TKCNDyUVIVW/9MyHaXI0nXK8IvBdIC6yg8guyoAF gfuWOj8X/Vo1tvZWFwPCHpGbpQmSokz4ohTqy8O5ywgy8xNUQdq25HZyfOn06xVPbeHz f1jN5vSQgRJo2tLg2dmUbtNrbiKUpHc6IEDkC93EvSlNUWgEFgWzh466hT2saqDJ4VPo 6z92FRVBWysyeOs5/s4MlzrzvuBOUfUzSekVTf7PEWy4nB84hbdH6GhBOxMWrJyxaqJ0 D0sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NFEhFRkg0+nIHx94jx/VqF2L4sTiUmledWciOk7WPsw=; b=OMdqhkjHrmc7VDOj4Rfj1Tyx1Z2pDX6f02PGC2NqgwtneMlyjHIgvUJ0QIC4dOgRVN 1P6WnAlCZlRr+zvNgdGz30/M74iQow8I7OMRDNvtz9MKjfTuhoswx7YM4jSqAcrMDWFB efF2nKkLwrCeZ7ot24HV/2SB2nw8jcrkQWOlZEBqxBsOPnASt+knCRRSDIFLVxcsf/Kl faT87OtPnuIeLUuy8SSRsslP+FKZmdznafTU/iHCiKmpCXIkeCZHeuBUjJnGbZFNBjv6 ojXX7BHsOvltZf6Eu/dVjrbzCE6iEc7U5jvJpVV25IxSf2lWUmzPlUr/N/llZghQ41Yp T8sw== X-Gm-Message-State: ANoB5plsP/e8HyOPdWrw6xyT/dUbLDhrmMEf3SZycRQZ27d3mphz2Jrr 3msOjD5YFAR3S99FhynPYWUh3pBSjeM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90b:2d90:b0:219:7904:6de with SMTP id sj16-20020a17090b2d9000b00219790406demr166821pjb.136.1670911763239; Mon, 12 Dec 2022 22:09:23 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:09 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-5-seanjc@google.com> Subject: [PATCH 4/7] KVM: VMX: Always inline to_vmx() and to_kvm_vmx() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078118774688145?= X-GMAIL-MSGID: =?utf-8?q?1752078118774688145?= Tag to_vmx() and to_kvm_vmx() __always_inline as they both just reflect the passed in pointer (the embedded struct is the first field in the container), and drop the @vmx param from vmx_vcpu_enter_exit(), which likely existed purely to make noinstr validation happy. Amusingly, when the compiler decides to not inline the helpers, e.g. for KASAN builds, to_vmx() and to_kvm_vmx() may end up pointing at the same symbol, which generates very confusing objtool warnings. E.g. the use of to_vmx() in a future patch led to objtool complaining about to_kvm_vmx(), and only once all use of to_kvm_vmx() was commented out did to_vmx() pop up in the obj tool report. vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x160: call to to_kvm_vmx() leaves .noinstr.text section Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 5 +++-- arch/x86/kvm/vmx/vmx.h | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fe5615fd8295..e2c96f204b82 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7096,9 +7096,10 @@ static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) } static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, - struct vcpu_vmx *vmx, unsigned long flags) { + struct vcpu_vmx *vmx = to_vmx(vcpu); + guest_state_enter_irqoff(); /* L1D Flush includes CPU buffer clear to mitigate MDS */ @@ -7216,7 +7217,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) kvm_wait_lapic_expire(vcpu); /* The actual VMENTER/EXIT is in the .noinstr.text section. */ - vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx)); + vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx)); /* All fields are clean at this point */ if (static_branch_unlikely(&enable_evmcs)) { diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index bb720a2f11ab..2acdc54bc34b 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -640,12 +640,12 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) (1 << VCPU_EXREG_EXIT_INFO_1) | \ (1 << VCPU_EXREG_EXIT_INFO_2)) -static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) +static __always_inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) { return container_of(kvm, struct kvm_vmx, kvm); } -static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu) +static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_vmx, vcpu); } From patchwork Tue Dec 13 06:09:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32643 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2657702wrr; Mon, 12 Dec 2022 22:11:32 -0800 (PST) X-Google-Smtp-Source: AA0mqf4TMcDBPakuKOtDaOd0Xd+3MhDyqOKQdHWSO3hh9tbCgG36LcWYtcyOTR6+bMzGqWBNJro9 X-Received: by 2002:a17:906:cc55:b0:7c1:6981:d062 with SMTP id mm21-20020a170906cc5500b007c16981d062mr7302541ejb.72.1670911892014; Mon, 12 Dec 2022 22:11:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911891; cv=none; d=google.com; s=arc-20160816; b=L5NIqp1tOH3tBRrOSxaFF/5teONOZfv2cM8XoxeuqTPSft46v6BMa3iIWKwj3mRBw2 m4I/R0kemI0Vqr5siBArvsEKt7HfV+xR6K90PULg5awmXqixL08evWUTOiS45Gx5SUo9 xnrNhIoec/mIyIQ54Ox8Nfw9gwTK0sy1KxNpaYwqD/Eo/P979lrYXa17f3aBdNASLfs3 SoWNWmIRPkU0MM07TvutjrIVfGH6wOGdW2L2UZt6DeRfrGXtAMICuvcZxRiN1d8dL4N3 ien0lZTCWHrNmU3XfazxT3idGVcsh6jez0Svb9f79bNufiJYGWqZVyyCQXuoii9wkWVM rSag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=3ygtdZJHpe2jAbhrmEODv8rggvGUoguv5qPo3aQqhPM=; b=UOyJptmB/cSL7FsbpAfTT8U5hHVP0sfPDpr/jR/ij9ULSfIh4ZDQG76rOfQ69lmMWb hpiKJsU++mv0NxcYr4j12vw5VoZD8QzDkSHqZYWi0o6ik5WR4tj6ho3l4iJfhwLuVABy 87MvTuPNg/LuSD3e6TUB7lTbBlwkXRuIemE/PlECYxk8j6bHTkA1aJH7hiP/lu69tsS/ kZqeeT15plnGolHTj+am0ZgLKOvQspE2bIMW/Qh85VKFz6Qq6YLaNdMGTvXUa5XBJECV MOtccjipUEIKuSPRuZyrE5q+MkW9hMVPgGMmJrJ/2RBtiebfyQKwJYQ+HL+NiVdmh8r3 pAxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=pcmVTu4J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds22-20020a170907725600b007c185c81f57si2096913ejc.326.2022.12.12.22.11.09; Mon, 12 Dec 2022 22:11:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=pcmVTu4J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234491AbiLMGJy (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234443AbiLMGJj (ORCPT ); Tue, 13 Dec 2022 01:09:39 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3975B1ADA3 for ; Mon, 12 Dec 2022 22:09:25 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id d7-20020a170902b70700b0018f4bf00569so5160600pls.4 for ; Mon, 12 Dec 2022 22:09:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3ygtdZJHpe2jAbhrmEODv8rggvGUoguv5qPo3aQqhPM=; b=pcmVTu4JD8o5BzVbE5RqsAHQF4iTjRKXTPoe+Wr1AmHVQrlVOOLTREcGLzBdX+CoDo X5MOObPom/RmvwSbNsO6KyVsAm0XRd0voNAY0/brtLjD+YSWb/xEvQB+xHexbwoeoPei 9jdk3IzS747RceXkfVCGzDvhLBGFC74oZOJdMmE48mLLaRx/wH091RT6FwBPkS/zbRf9 Oyb4HYwaW1FIGfBTJJj30F7YXbKC/w0okF46x5P04R3+nRZpvXvjuTWHfIssZ5QAoX7I 6ex6l/Gs54L1I9eeJHSnhDUtkWvSF4Whla4N2HvTnwSewDlW82hqZ8zQ+QeeiOZJ0UkE 4cag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3ygtdZJHpe2jAbhrmEODv8rggvGUoguv5qPo3aQqhPM=; b=XsGojyzMPaUTbuBvbDTvEMfB4C3LvEwuZMEjkK9cfQkJsp791KbxeTMo+uJ3He2vTc 1m0oahd+r+03yrbGcDm1Bp4CRRzj/iiZW4RsL5XLsP+z59YE7OrKuco7Cr5TLQFzQt/z NzdjCQ33N3xTO6NLYDVoLhDzsO9ZExiKm6WxUgmwOO7AErVHtc+fU7+9w59o4nULwt8p TBPH9f3nwkMzuZN7XS+N60m9WtuFBFBOcuarI/md/zwSafZqIdqg+xxpNsE37YXQy9IR 8me75pWcDNNr23ig6ttZ+k+77Dl66S9nOdlcJ0mCp7QI57LfewYQn2Z82i5pFd3AYOHC xQSw== X-Gm-Message-State: ANoB5pl1BWrSsFcSrYn1AlXCMUUAov9qXuWd4fp+ro/poLZeGZjr+rRH WshcVkLOEA4N47O+mOMNQA1iPrwaVe0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:f14b:b0:18f:5a3:6069 with SMTP id d11-20020a170902f14b00b0018f05a36069mr706521plb.46.1670911764799; Mon, 12 Dec 2022 22:09:24 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:10 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-6-seanjc@google.com> Subject: [PATCH 5/7] x86/entry: KVM: Use dedicated VMX NMI entry for 32-bit kernels too From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078108286348877?= X-GMAIL-MSGID: =?utf-8?q?1752078108286348877?= Use a dedicated entry for invoking the NMI handler from KVM VMX's VM-Exit path for 32-bit even though using a dedicated entry for 32-bit isn't strictly necessary. Exposing a single symbol will allow KVM to reference the entry point in assembly code without having to resort to more #ifdefs (or #defines). identry.h is intended to be included from asm files only once, and so simply including idtentry.h in KVM assembly isn't an option. Bypassing the ESP fixup and CR3 switching in the standard NMI entry code is safe as KVM always handles NMIs that occur in the guest on a kernel stack, with a kernel CR3. Cc: Andy Lutomirski Cc: Thomas Gleixner Signed-off-by: Sean Christopherson Reviewed-by: Lai Jiangshan --- arch/x86/include/asm/idtentry.h | 16 ++++++---------- arch/x86/kernel/nmi.c | 8 ++++---- arch/x86/kvm/vmx/vmx.c | 4 ++-- 3 files changed, 12 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 72184b0b2219..b241af4ce9b4 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -582,18 +582,14 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_check); /* NMI */ -#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL) +#if IS_ENABLED(CONFIG_KVM_INTEL) /* - * Special NOIST entry point for VMX which invokes this on the kernel - * stack. asm_exc_nmi() requires an IST to work correctly vs. the NMI - * 'executing' marker. - * - * On 32bit this just uses the regular NMI entry point because 32-bit does - * not have ISTs. + * Special entry point for VMX which invokes this on the kernel stack, even for + * 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to work + * correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as well + * to avoid more ifdeffery. */ -DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_noist); -#else -#define asm_exc_nmi_noist asm_exc_nmi +DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_kvm_vmx); #endif DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi); diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index cec0bfa3bc04..e37faba95bb5 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -527,14 +527,14 @@ DEFINE_IDTENTRY_RAW(exc_nmi) mds_user_clear_cpu_buffers(); } -#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL) -DEFINE_IDTENTRY_RAW(exc_nmi_noist) +#if IS_ENABLED(CONFIG_KVM_INTEL) +DEFINE_IDTENTRY_RAW(exc_nmi_kvm_vmx) { exc_nmi(regs); } -#endif #if IS_MODULE(CONFIG_KVM_INTEL) -EXPORT_SYMBOL_GPL(asm_exc_nmi_noist); +EXPORT_SYMBOL_GPL(asm_exc_nmi_kvm_vmx); +#endif #endif void stop_nmi(void) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index e2c96f204b82..7ace22ee240d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6791,7 +6791,7 @@ void vmx_do_interrupt_nmi_irqoff(unsigned long entry); static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu, unsigned long entry) { - bool is_nmi = entry == (unsigned long)asm_exc_nmi_noist; + bool is_nmi = entry == (unsigned long)asm_exc_nmi_kvm_vmx; kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ); vmx_do_interrupt_nmi_irqoff(entry); @@ -6820,7 +6820,7 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) { - const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_noist; + const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_kvm_vmx; u32 intr_info = vmx_get_intr_info(&vmx->vcpu); /* if exit due to PF check for async PF */ From patchwork Tue Dec 13 06:09:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32647 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2658146wrr; Mon, 12 Dec 2022 22:12:54 -0800 (PST) X-Google-Smtp-Source: AA0mqf4eNixcsjTaNs/p/TrvLh2S0CPdxwN3RJS4diRice4NaTqSF5hOQ5RlpGfzp5Kx+rmuA38w X-Received: by 2002:a17:906:c781:b0:78d:f455:b5d2 with SMTP id cw1-20020a170906c78100b0078df455b5d2mr16139841ejb.18.1670911973739; Mon, 12 Dec 2022 22:12:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911973; cv=none; d=google.com; s=arc-20160816; b=wYB1Ig46gIzRKbSm4VQi90/NN7RaAXQJ6g+8xwcUgTjvIUG3AihwqAvLxFdgVJaY0Q fzkZhvITE09mFzDRUM77byR0AGjeJIRj64dbXK153k4UvIltTGPM3hGQQOOlFiLMJPTC TZXyNd2jBUiwim+h+KdM2wAz9lsbP+aVdbZhbtJJkh5ZXBx9+mojPGhjremI3T1Y6IzX aCltfDSwLq7ufS7kJwOXpSU/9EJqTUJW/UnXvVtU5D1yEmDVEnOWYEgkyXXK2WBJtZsP yzNeDJWXEqBBMtT8iXOsfX1I2ApPJHuU0x3i17fTMRkUgEwthPquugaozZ6Y7H9l+TWU Jgzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=XJmCztab5I+GOPa0DQl6Gc8jULACLfl5WBPqqp47Nbo=; b=ap8g1Sdv+pyNwOmNVKRXidLgHOqYXvqMcoYdOTydTzBpvToINaFMjWZvb3QxMt+XMC ekGP/3VVPgUf4krEve5sQrGXhNxj1BNxGGksbo2LY9uPjQ9+XRaOBd9KIM/4XEbpN06B DQOWk75IsJl+rkaIZAkWShvqMWZqzEMT4T5j5fS7car/hoxcQ3RvV5nDyniy1HmuLHN/ WAdPDkW1h4hRIozsA/00f4inBH71We+iokxo++QtU0gRIlfeRUXIeoWyFWMRGkTbaW3P bXj48bms4BJKPaOKoeoQ8/D/ahH8t4Z4gfPKKf6JseGXBEVAd9Vr40GYL8Dyn4gUf17x qkDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=eOYAUekk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fq36-20020a1709069da400b007c0baa5000csi5149756ejc.340.2022.12.12.22.12.30; Mon, 12 Dec 2022 22:12:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=eOYAUekk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234049AbiLMGJ7 (ORCPT + 99 others); Tue, 13 Dec 2022 01:09:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229645AbiLMGJp (ORCPT ); Tue, 13 Dec 2022 01:09:45 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 125B61ADAA for ; Mon, 12 Dec 2022 22:09:27 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id o6-20020a17090ab88600b0021edf53917eso1336719pjr.3 for ; Mon, 12 Dec 2022 22:09:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=XJmCztab5I+GOPa0DQl6Gc8jULACLfl5WBPqqp47Nbo=; b=eOYAUekkYFKW32/sardr9LbkqefyMBx11vxUVkrH0ldjTntI2ZqZ12Z1YKA8CIJJll 5gJKypePk09DROtfI5aCEo66BA5V+NzqriBgU1mn2W2AI3R8kOtCMoF3y5solFD0HK9p p320W+0By14HNqdn0+hlsW1X/3EZmtWRJH2n3mdjsIYLGDt3ZRy7pdDm1JUce8nvs+U+ agGnzIMGnjy9CKnRAg/9Z18OiIYhxe/3PQZzB9sfpM59C3Qi4oSGs7XPTYOW3TOVjoiO e+gPF63js8NwFX6PxL4drafuNhkepvmEbblZQV/V0zrMJiMYs2C9kZeO/uUexIci8mmg Pl7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XJmCztab5I+GOPa0DQl6Gc8jULACLfl5WBPqqp47Nbo=; b=2dgXaa7LvRPq2vmKxbmVq3w10jepr2t9Uz4ZkcfuikbLuqnrYt48FqsuujmARyK2lh xJJSKbRtb3dl2BWBXNXZTsuxaykaFu4Pn8JTCtdKwRX/1gD8o5t/ChyMjB8mypla9M5q j2FPTXbuOW70Ol3Vw7H3ATRsNKSQGwCHCEK9Fd66JTUi537H39NUNnkEqfH1Bb+g/nJd In9m3OW1f6/pjFFfJkiAe9VNYKKpxtT99xoiFEMIIUqCmHyU9pjv+hDAK81QfiLpiGs0 hS8X4j+dNsco8d/Qe2NyQqbi2cwf1/yrSBnvAqTpek+yFuaCF6of2BB6ThDLeoohpWNY K+yg== X-Gm-Message-State: ANoB5plCFbRJUCjqoAvJ/yAgdVsbx/p6nATDMKALdwt6z8Q0vbBwpkaa dXgArlVmwoW67mnD5fjz4fy02t4cow4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:2ecb:b0:219:5b3b:2b9f with SMTP id h11-20020a17090a2ecb00b002195b3b2b9fmr22689pjs.2.1670911766390; Mon, 12 Dec 2022 22:09:26 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:11 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-7-seanjc@google.com> Subject: [PATCH 6/7] KVM: VMX: Provide separate subroutines for invoking NMI vs. IRQ handlers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078193851905024?= X-GMAIL-MSGID: =?utf-8?q?1752078193851905024?= Split the asm subroutines for handling NMIs versus IRQs that occur in the guest so that the NMI handler can be called from a noinstr section. As a bonus, the NMI path doesn't need an indirect branch. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmenter.S | 70 +++++++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.c | 26 ++++++-------- 2 files changed, 50 insertions(+), 46 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 766c6b3ef5ed..9d987e7e48c4 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -31,6 +31,39 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif +.macro VMX_DO_EVENT_IRQOFF call_insn call_target + /* + * Unconditionally create a stack frame, getting the correct RSP on the + * stack (for x86-64) would take two instructions anyways, and RBP can + * be used to restore RSP to make objtool happy (see below). + */ + push %_ASM_BP + mov %_ASM_SP, %_ASM_BP + +#ifdef CONFIG_X86_64 + /* + * Align RSP to a 16-byte boundary (to emulate CPU behavior) before + * creating the synthetic interrupt stack frame for the IRQ/NMI. + */ + and $-16, %rsp + push $__KERNEL_DS + push %rbp +#endif + pushf + push $__KERNEL_CS + \call_insn \call_target + + /* + * "Restore" RSP from RBP, even though IRET has already unwound RSP to + * the correct value. objtool doesn't know the callee will IRET and, + * without the explicit restore, thinks the stack is getting walloped. + * Using an unwind hint is problematic due to x86-64's dynamic alignment. + */ + mov %_ASM_BP, %_ASM_SP + pop %_ASM_BP + RET +.endm + .section .noinstr.text, "ax" /** @@ -320,35 +353,10 @@ SYM_FUNC_START(vmread_error_trampoline) SYM_FUNC_END(vmread_error_trampoline) #endif -SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) - /* - * Unconditionally create a stack frame, getting the correct RSP on the - * stack (for x86-64) would take two instructions anyways, and RBP can - * be used to restore RSP to make objtool happy (see below). - */ - push %_ASM_BP - mov %_ASM_SP, %_ASM_BP +SYM_FUNC_START(vmx_do_nmi_irqoff) + VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(vmx_do_nmi_irqoff) -#ifdef CONFIG_X86_64 - /* - * Align RSP to a 16-byte boundary (to emulate CPU behavior) before - * creating the synthetic interrupt stack frame for the IRQ/NMI. - */ - and $-16, %rsp - push $__KERNEL_DS - push %rbp -#endif - pushf - push $__KERNEL_CS - CALL_NOSPEC _ASM_ARG1 - - /* - * "Restore" RSP from RBP, even though IRET has already unwound RSP to - * the correct value. objtool doesn't know the callee will IRET and, - * without the explicit restore, thinks the stack is getting walloped. - * Using an unwind hint is problematic due to x86-64's dynamic alignment. - */ - mov %_ASM_BP, %_ASM_SP - pop %_ASM_BP - RET -SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff) +SYM_FUNC_START(vmx_do_interrupt_irqoff) + VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 +SYM_FUNC_END(vmx_do_interrupt_irqoff) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 7ace22ee240d..c242e2591896 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6786,17 +6786,8 @@ static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir)); } -void vmx_do_interrupt_nmi_irqoff(unsigned long entry); - -static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu, - unsigned long entry) -{ - bool is_nmi = entry == (unsigned long)asm_exc_nmi_kvm_vmx; - - kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ); - vmx_do_interrupt_nmi_irqoff(entry); - kvm_after_interrupt(vcpu); -} +void vmx_do_interrupt_irqoff(unsigned long entry); +void vmx_do_nmi_irqoff(void); static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) { @@ -6820,7 +6811,6 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) { - const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_kvm_vmx; u32 intr_info = vmx_get_intr_info(&vmx->vcpu); /* if exit due to PF check for async PF */ @@ -6833,8 +6823,11 @@ static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) else if (is_machine_check(intr_info)) kvm_machine_check(); /* We need to handle NMIs before interrupts are enabled */ - else if (is_nmi(intr_info)) - handle_interrupt_nmi_irqoff(&vmx->vcpu, nmi_entry); + else if (is_nmi(intr_info)) { + kvm_before_interrupt(&vmx->vcpu, KVM_HANDLING_NMI); + vmx_do_nmi_irqoff(); + kvm_after_interrupt(&vmx->vcpu); + } } static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) @@ -6847,7 +6840,10 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) "KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; - handle_interrupt_nmi_irqoff(vcpu, gate_offset(desc)); + kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ); + vmx_do_interrupt_irqoff(gate_offset(desc)); + kvm_after_interrupt(vcpu); + vcpu->arch.at_instruction_boundary = true; } From patchwork Tue Dec 13 06:09:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 32646 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp2658046wrr; Mon, 12 Dec 2022 22:12:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf4wVqw7JBstLwrBgJ9j0ZALyK6bSiDVXAxxEoecJZExTVU0Uct0J83o0poYTmk8e7pk0JPw X-Received: by 2002:a17:90a:f3c1:b0:21e:dec6:fc3d with SMTP id ha1-20020a17090af3c100b0021edec6fc3dmr15390201pjb.8.1670911957116; Mon, 12 Dec 2022 22:12:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670911957; cv=none; d=google.com; s=arc-20160816; b=JIZn6194tZgNRh6cJFTZjm9HaGYrw/uGvv2H5ycGnwFIZG7GwhsPtw4EwOm0cW3HCv dEbHeg+K3fKaS/OZ/Hu6yvbEgJKRh3VmKHdQvyR6At3KcCPjoD/IHrJa70CfZZAqJPWj nsA4K74bfcwBinXP1gF2ePAHDAvb3hTTabRcNAgDRDgjkLdHgKRMQa9umhFt8SDaqjzU uhGTsOVqAdEw3A/ESBjsDInt2Tw3jCNLjxx6/5jbLc2PlyMlPX3lGOcRjO0KsZusSg+D SVvsG7FbOgHAmJxmtwOHSW+MnjI8wgtinIxBtyUJVclW0SOo16x3myztkIwt+Ap+P+wb f4NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=FqdJjiwJ+3KBa9yJbuoyosmHEFIdRQ7rtSq2Gl3y3sA=; b=OuHyZhu6zbEjRwjhyVYPXC8YWnfCfUQjH8GdMDSB7v8q/PcpCPJhl78SfKdmkOCFBG KEZJ7JEJzqUG9jT5I/nvn9ywVe+4ugOPMmJXlwdTpkQTbG/vXH7/OZdP22fZ3L6qBoBx 0O+swPky/ZHQdO0f2H9berS1gZDi+8xOVW7eYtT8heNOKhoTp+oqjQENm4v4c2HjwC/o sTUJDti2bD/S6ntHMo7fKhnxV7Cr1juHL8SZ6nKXV0jhfNMfjM2iYWT8jixjK6H0hRZ2 h5QBJIDWqySRW8L3R8qQHXU8oxQ82k7YL7nCLXiAbPhmylxC7ZbkwO6eDHM4CI2HbxQt DCJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=L0uRFcqM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id my17-20020a17090b4c9100b0020d401a03b5si12339268pjb.180.2022.12.12.22.12.24; Mon, 12 Dec 2022 22:12:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=L0uRFcqM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234535AbiLMGKG (ORCPT + 99 others); Tue, 13 Dec 2022 01:10:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234461AbiLMGJu (ORCPT ); Tue, 13 Dec 2022 01:09:50 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BE721ADA8 for ; Mon, 12 Dec 2022 22:09:28 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id k7-20020a17090a39c700b002192c16f19aso1347144pjf.1 for ; Mon, 12 Dec 2022 22:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=FqdJjiwJ+3KBa9yJbuoyosmHEFIdRQ7rtSq2Gl3y3sA=; b=L0uRFcqMg4zxrXLXzS5agUTJtSHNHZaZB/Umstjib0KDLbzEMDkaIE5IYILiNh4ihO Ks61vZYMKGfYWtht5ScwwNerE+C+y6uWagJ0CcBgoMLA2MHnwrngn38QzloYlobOUjZ2 LyWEcAhE8EP0Ftp6XZJM/ey9/GhxkepiJvO7Zdo3G+MDxVA6RcDRZc9XqkSiJ0pa5KsN GjAVmtse3/tNYkngK2mewx7lLyeujrbP9aRghooZ2FUrMr++mTifKKsdKXJjGPNfl4VC EVTOIpaCOflVepNg538v55OTMIVlEOvHsRVJ0ZKCh92IxSJ3cgUee0q54Mfe68Bz0vwf ej5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FqdJjiwJ+3KBa9yJbuoyosmHEFIdRQ7rtSq2Gl3y3sA=; b=yA+JhqbwSL3/J4aIEaFlAPY+4aLnwsuaVBZj0qJrFcvEORq+HbiPRnCYrp16OY3WRk GNDRQeIkx4/NPqdEg5hETYZ7RjNUsyj2a1NhNR7nVB8DNJrp2VExduWPC5gGiELCNj3d j4+r3/1ju3cv+BxLn7o0eqc937Gm7U2mzCXpiu3xqXjzcinssccAlbgjcsc7xscpF3Rx ASQABG2fZBhPqTfPw1X6rwCpXKDE356H6il6JEorV5yUSYs8PoU0yV2s6jpFs9TZfUYd UcGvjb4pVzP/wFXv7EVICjK7KeGLiiyKthqr2o10VCTVlN4ZPwqyE9Nn8ODIQzRt6jdO KvUw== X-Gm-Message-State: ANoB5pkHxviwmmyW9y+wI53RfLVfwdn5IqIYEWRubppvQTT9+h3cCMrP e3zxQ11Mdv468qqp1sXPJ/6xqr1TsoQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:aa7:85d9:0:b0:577:53a9:836c with SMTP id z25-20020aa785d9000000b0057753a9836cmr10672135pfn.5.1670911768146; Mon, 12 Dec 2022 22:09:28 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:12 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-8-seanjc@google.com> Subject: [PATCH 7/7] KVM: VMX: Handle NMI VM-Exits in noinstr region From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752078176706724353?= X-GMAIL-MSGID: =?utf-8?q?1752078176706724353?= Move VMX's handling of NMI VM-Exits into vmx_vcpu_enter_exit() so that the NMI is handled prior to leaving the safety of noinstr. Handling the NMI after leaving noinstr exposes the kernel to potential ordering problems as an instrumentation-induced fault, e.g. #DB, #BP, #PF, etc. will unblock NMIs when IRETing back to the faulting instruction. Reported-by: Peter Zijlstra Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmcs.h | 4 ++-- arch/x86/kvm/vmx/vmenter.S | 8 ++++---- arch/x86/kvm/vmx/vmx.c | 34 +++++++++++++++++++++------------- arch/x86/kvm/x86.h | 6 +++--- 4 files changed, 30 insertions(+), 22 deletions(-) diff --git a/arch/x86/kvm/vmx/vmcs.h b/arch/x86/kvm/vmx/vmcs.h index ac290a44a693..7c1996b433e2 100644 --- a/arch/x86/kvm/vmx/vmcs.h +++ b/arch/x86/kvm/vmx/vmcs.h @@ -75,7 +75,7 @@ struct loaded_vmcs { struct vmcs_controls_shadow controls_shadow; }; -static inline bool is_intr_type(u32 intr_info, u32 type) +static __always_inline bool is_intr_type(u32 intr_info, u32 type) { const u32 mask = INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK; @@ -146,7 +146,7 @@ static inline bool is_icebp(u32 intr_info) return is_intr_type(intr_info, INTR_TYPE_PRIV_SW_EXCEPTION); } -static inline bool is_nmi(u32 intr_info) +static __always_inline bool is_nmi(u32 intr_info) { return is_intr_type(intr_info, INTR_TYPE_NMI_INTR); } diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 9d987e7e48c4..059243085211 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -299,6 +299,10 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) SYM_FUNC_END(__vmx_vcpu_run) +SYM_FUNC_START(vmx_do_nmi_irqoff) + VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(vmx_do_nmi_irqoff) + .section .text, "ax" @@ -353,10 +357,6 @@ SYM_FUNC_START(vmread_error_trampoline) SYM_FUNC_END(vmread_error_trampoline) #endif -SYM_FUNC_START(vmx_do_nmi_irqoff) - VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx -SYM_FUNC_END(vmx_do_nmi_irqoff) - SYM_FUNC_START(vmx_do_interrupt_irqoff) VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 SYM_FUNC_END(vmx_do_interrupt_irqoff) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c242e2591896..b03020ca1840 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5095,8 +5095,13 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) vect_info = vmx->idt_vectoring_info; intr_info = vmx_get_intr_info(vcpu); + /* + * Machine checks are handled by handle_exception_irqoff(), or by + * vmx_vcpu_run() if a #MC occurs on VM-Entry. NMIs are handled by + * vmx_vcpu_enter_exit(). + */ if (is_machine_check(intr_info) || is_nmi(intr_info)) - return 1; /* handled by handle_exception_nmi_irqoff() */ + return 1; /* * Queue the exception here instead of in handle_nm_fault_irqoff(). @@ -6809,7 +6814,7 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); } -static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) +static void handle_exception_irqoff(struct vcpu_vmx *vmx) { u32 intr_info = vmx_get_intr_info(&vmx->vcpu); @@ -6822,12 +6827,6 @@ static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) /* Handle machine checks before interrupts are enabled */ else if (is_machine_check(intr_info)) kvm_machine_check(); - /* We need to handle NMIs before interrupts are enabled */ - else if (is_nmi(intr_info)) { - kvm_before_interrupt(&vmx->vcpu, KVM_HANDLING_NMI); - vmx_do_nmi_irqoff(); - kvm_after_interrupt(&vmx->vcpu); - } } static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) @@ -6857,7 +6856,7 @@ static void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) if (vmx->exit_reason.basic == EXIT_REASON_EXTERNAL_INTERRUPT) handle_external_interrupt_irqoff(vcpu); else if (vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI) - handle_exception_nmi_irqoff(vmx); + handle_exception_irqoff(vmx); } /* @@ -7119,6 +7118,18 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx_enable_fb_clear(vmx); + if (unlikely(vmx->fail)) + vmx->exit_reason.full = 0xdead; + else + vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON); + + if ((u16)vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI && + is_nmi(vmx_get_intr_info(vcpu))) { + kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); + vmx_do_nmi_irqoff(); + kvm_after_interrupt(vcpu); + } + guest_state_exit_irqoff(); } @@ -7260,12 +7271,9 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) vmx->idt_vectoring_info = 0; - if (unlikely(vmx->fail)) { - vmx->exit_reason.full = 0xdead; + if (unlikely(vmx->fail)) return EXIT_FASTPATH_NONE; - } - vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON); if (unlikely((u16)vmx->exit_reason.basic == EXIT_REASON_MCE_DURING_VMENTRY)) kvm_machine_check(); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 9de72586f406..44d1827f0a30 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -382,13 +382,13 @@ enum kvm_intr_type { KVM_HANDLING_NMI, }; -static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu, - enum kvm_intr_type intr) +static __always_inline void kvm_before_interrupt(struct kvm_vcpu *vcpu, + enum kvm_intr_type intr) { WRITE_ONCE(vcpu->arch.handling_intr_from_guest, (u8)intr); } -static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu) +static __always_inline void kvm_after_interrupt(struct kvm_vcpu *vcpu) { WRITE_ONCE(vcpu->arch.handling_intr_from_guest, 0); }