From patchwork Wed Feb 1 19:45:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51535 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp477708wrn; Wed, 1 Feb 2023 11:51:53 -0800 (PST) X-Google-Smtp-Source: AK7set+RLHgZmHkFFskvU7Fct4fshxM0P2PJnVazZGwu9lFqMhDoY/wW/foAwEL76n24Nv5Al7vl X-Received: by 2002:a17:907:990b:b0:87b:dac1:bbe6 with SMTP id ka11-20020a170907990b00b0087bdac1bbe6mr4071529ejc.36.1675281113702; Wed, 01 Feb 2023 11:51:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281113; cv=none; d=google.com; s=arc-20160816; b=eD0Y84q7mAfErs2Er17ZroYPE0+ipXXBiycPKSuB4f/TnXJLU+O0hP1DM8T/BVrIJC xCDu/XiFD7DQjLgGDlUAg8jcKAPsa/UKAwoWPvi1XLP4jXQu7jJxuDa97DQ47FrEKo/P B1WiIsSNELJZWQXwv5HIVzZKkPnsFpyH0xXspqLWYSbz9kEmWtUWzIPxC4VxovcdfKGy dpp+jt5SXCGnL12QUS0bK7M7Uh4Y1Yn6GgIiDGGI+2SNmasdgLkVBsNP3Lf9+kkUpQBr ViikFI0bS5n+lz7snZBaEhsB9VCk7YPh8oR/W5cc07fxdssBjtK2tsVmQx31c256KJEu rZ5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HICwk+ZaGfDJCbBt6gOrUiuKx+JyONa1rsd399znELE=; b=c/dy+dfuklg/q8uOrcGXuP9rnk/1DX7ylb9af/bNZmCFmGRojWYWuQPqpexY0SjtG7 3vao9E5U0UvMLnTtXlsmQATkC56H5n6pbUkre7WfdrxS4x2QbXrtua48tUb64R3vTUh6 sGlIyfwzBdc8VeuPS5nEHMOBEpQDFxFsxzg2+BLqjbkabh99dxuHCSiMhIcgfl0GDt8L A5MdM1t/uBMQZoqhIz1ReukQ/jyWV9nL5Trnp6Q4rYVv2KMbghueT3Kwtdg+5s74w/YU Azuzsk1aoT9mXepsQ8AOybpER1jn5D1rtLCZ/D+TR7bYF7dVKkyBSFlfvXUkc/tTE2vQ ufgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=s+TkGtd0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eu15-20020a170907298f00b008880c90cfacsi4152908ejc.542.2023.02.01.11.51.30; Wed, 01 Feb 2023 11:51:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=s+TkGtd0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232168AbjBATob (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232081AbjBATo2 (ORCPT ); Wed, 1 Feb 2023 14:44:28 -0500 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A85B7A4B4 for ; Wed, 1 Feb 2023 11:44:27 -0800 (PST) Received: by mail-ej1-x62f.google.com with SMTP id gr7so30044349ejb.5 for ; Wed, 01 Feb 2023 11:44:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HICwk+ZaGfDJCbBt6gOrUiuKx+JyONa1rsd399znELE=; b=s+TkGtd0CJKn0fgzwiRNSQ0hciitR9Qd6pS4R55tq+1KVcodwIh7hzdgMhVzvVJQl/ Vogax6v8Gs19BrA4b+x/qdMusyTdBCEx4Y7dfR68DJXA7Iha1agsWxF5Bn6Tryp4+0o3 iA53H/yFXVCL79uMOWiCoCkgi/QGS9xr1UlpMZsSMFL4zjLwN6bRgumgIBMmy+uZ9w+8 ifMxfGaPOdhU2cgK1wRoX5Swx0HuIiG6bM/tkO/ugP/iLROL4hMcMxJvk8YxgtJq6CbA GsDlvTkmOpcp5+nRgvXWHg7dHGEhDVIvHPIUJ2wOvtewT6J+x5xjElC4R23hET6sLn+J yQ8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HICwk+ZaGfDJCbBt6gOrUiuKx+JyONa1rsd399znELE=; b=UDR0DEAFq7aPz7IdWGJe5B2SZmFGse3S9X0fhdeUI7zjJ+A2kfDN0OlH6/5hX/Y6tM /4ChEJbW+pJSQvgkQNCqFq4UF+9VQgAThFHFuGaBlUiJQ0Fa7B7FJWdMEk4YaG4UAKwX 8WR5HHW34nm7mobW9I9dUHVu9GD2K83UcUAd3hEYTi3cyncwHiS+nV16cNjeuc2cyt+v muumGwqADNG0IxNLkxf89dolL4mWyzqyU3lLBJc8raCuHfVpiEKTQvaYEluWa//8q1Im CP7+8LOEB8kLLOQaI+qk5cOJ/aJVaG2QR4tF7C2zgYgWsvVIXi39gTynij4IEGmWrYC8 01vQ== X-Gm-Message-State: AO0yUKUHK5k8uFc5+5Q0Hypa/qdSUn1x4Wez0jVTwGo9eQ/pg8iwVLkS +YpCmLDdOMrNWXzYas17REL1GA== X-Received: by 2002:a17:907:990b:b0:87b:dac1:bbe6 with SMTP id ka11-20020a170907990b00b0087bdac1bbe6mr4042757ejc.36.1675280665653; Wed, 01 Feb 2023 11:44:25 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:25 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 1/6] KVM: x86/mmu: Avoid indirect call for get_cr3 Date: Wed, 1 Feb 2023 20:45:59 +0100 Message-Id: <20230201194604.11135-2-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659569475844922?= X-GMAIL-MSGID: =?utf-8?q?1756659569475844922?= From: Paolo Bonzini Most of the time, calls to get_guest_pgd result in calling kvm_read_cr3 (the exception is only nested TDP). Hardcode the default instead of using the get_cr3 function, avoiding a retpoline if they are enabled. Signed-off-by: Paolo Bonzini Signed-off-by: Mathias Krause --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++++++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index aeb240b339f5..505768631614 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -241,6 +241,20 @@ static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) return regs; } +static unsigned long get_guest_cr3(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr3(vcpu); +} + +static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + if (IS_ENABLED(CONFIG_RETPOLINE) && mmu->get_guest_pgd == get_guest_cr3) + return kvm_read_cr3(vcpu); + + return mmu->get_guest_pgd(vcpu); +} + static inline bool kvm_available_flush_tlb_with_range(void) { return kvm_x86_ops.tlb_remote_flush_with_range; @@ -3722,7 +3736,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) int quadrant, i, r; hpa_t root; - root_pgd = mmu->get_guest_pgd(vcpu); + root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; if (mmu_check_root(vcpu, root_gfn)) @@ -4172,7 +4186,7 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, arch.token = alloc_apf_token(vcpu); arch.gfn = gfn; arch.direct_map = vcpu->arch.mmu->root_role.direct; - arch.cr3 = vcpu->arch.mmu->get_guest_pgd(vcpu); + arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); return kvm_setup_async_pf(vcpu, cr2_or_gpa, kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch); @@ -4191,7 +4205,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) return; if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 != vcpu->arch.mmu->get_guest_pgd(vcpu)) + work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) return; kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); @@ -4592,11 +4606,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); -static unsigned long get_cr3(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr3(vcpu); -} - static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { @@ -5147,7 +5156,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu, context->page_fault = kvm_tdp_page_fault; context->sync_page = nonpaging_sync_page; context->invlpg = NULL; - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; @@ -5297,7 +5306,7 @@ static void init_kvm_softmmu(struct kvm_vcpu *vcpu, kvm_init_shadow_mmu(vcpu, cpu_role); - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; } @@ -5311,7 +5320,7 @@ static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu, return; g_context->cpu_role.as_u64 = new_mode.as_u64; - g_context->get_guest_pgd = get_cr3; + g_context->get_guest_pgd = get_guest_cr3; g_context->get_pdptr = kvm_pdptr_read; g_context->inject_page_fault = kvm_inject_page_fault; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index e5662dbd519c..78448fb84bd6 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -324,7 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; - pte = mmu->get_guest_pgd(vcpu); + pte = kvm_mmu_get_guest_pgd(vcpu, mmu); have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); #if PTTYPE == 64 From patchwork Wed Feb 1 19:46:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51536 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp477849wrn; Wed, 1 Feb 2023 11:52:24 -0800 (PST) X-Google-Smtp-Source: AK7set8yopGMEx0emf9jByL7jEhJ0DruIMDpOQp01en5w+kk0b1QgPfnF9b/ho0pRlWRoJOoe8qg X-Received: by 2002:a05:6402:1772:b0:49f:da00:47a5 with SMTP id da18-20020a056402177200b0049fda0047a5mr2939063edb.25.1675281144294; Wed, 01 Feb 2023 11:52:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281144; cv=none; d=google.com; s=arc-20160816; b=UXRysXXxO1Uhd6ujPdsJpCXXYMlaTvd7igxRNXxtvbrcKmGzQUXKsNHnl/21DUwuOz 2sDAVGG6w25EpNUksXhtemOikcZzFy7/QQt5aJFms5XCHXS6WLUNCnebO9QsYO1u0NHU Z9n9lYl5oKVkv2wDoFRFgaw9f1CFyFzToxGUV/XDvhvIkuK7VW3ZHyRgBpYIp3uulzw7 WdOAJSFrSQZsxY12xhrkG7BN33TEIV3reHMVGPibDnoz6m7AakEp6eY27UnmPr1tUBEm RJUS5k4j0NliTCCGU4KYsc0has2Q25f5pubhgZJZVD8zzDm/XH1AK+VcN/rzjNWQot3K nOBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SwDZjz3R/xUrA9TRfuAPMHW2TTSVBtiLMqPJ61VaYAk=; b=siYJEsAYR5NOyUuc22wmI5oP3yESyslRgFxX3jsdDF0+2XFC3F/jMEA94IeQUu9NTd mSMKe3YTJAis92sRMOM6KsZyNHfGQgDbvPO36qRKjpAmdufgBKSs69ZHdqp3tqMMHCaD Mf0XbjFJzrTTqteBiApa91ggv00ObU50EtToJOnq3LASqDef33f7yaDAMbQknPqMqM/S +Xchaqt78xc0Ea8oSiRKuJfgWgBtvH7gkVAqdaD2MmZo7CZFJsqjl9H9WA6a9ELALM19 /WVen+lZU6ZSXXUWwYRJ/IbB2sKUNaWnhZwg4DAfYNMLsniHL8qLW+4jvIQP9EQk+5fD mrpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b="RaxIe/1D"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bp21-20020a170907919500b0084cd1ecf338si19491587ejb.705.2023.02.01.11.52.00; Wed, 01 Feb 2023 11:52:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b="RaxIe/1D"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232208AbjBAToe (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232094AbjBATo2 (ORCPT ); Wed, 1 Feb 2023 14:44:28 -0500 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C877A7BE6F for ; Wed, 1 Feb 2023 11:44:27 -0800 (PST) Received: by mail-ej1-x62b.google.com with SMTP id p26so43432882ejx.13 for ; Wed, 01 Feb 2023 11:44:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SwDZjz3R/xUrA9TRfuAPMHW2TTSVBtiLMqPJ61VaYAk=; b=RaxIe/1DMFnMimcNs7o86F/NH2QJAimOB2DuKkM8Rlp3M6jd8ZwN2g3mowK0Iz+Pqg bYC5ACF10ECbb0drnKJUP3eHL/RHsNmek0qc7K+fOTMcmg7/SFSpv2rJKMJEx3GdsV07 rT9ycN16bqmTAY+53MXGTPGDbAMHlLjodfIG0FT7QxHQpmyYi2igCaIjsIehkX2f4CCl 4qjCG11Xd6eatyCxEBzgvTJT5UBgwiW3lcZf46NgbwCU5LLkDkuZAUX0KTkWTcmt4Otn zwrh/hdBGwSjBdsza6opiEjbdZvqe+jcqP9ZK9WfeWU1nlFVG97Ldwg+a19E3iJeccod 1Fmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SwDZjz3R/xUrA9TRfuAPMHW2TTSVBtiLMqPJ61VaYAk=; b=7NWDBTGQ14qzJKNgSw7EqvXEkp8Nw7KgG56QRPfTgFCuZj4FG37J6okB5tVZZH50l/ 7ElhovdD/p0XSMpmotBx1DqO5XaH8iJoSqiE87n6JJqaddvM3i7qvyv6SCRCIDN3x6Xj E9Wxel5HvAYPTPZZrgfC3AXKlEcIHMpN1AfA8cBIRxk2oMH1zn+DQzeHj4IEDjTcAwKK 4unfPEkNG7qMnX5BB4y4g2aUAMJIXAYnPHevLEZimCx+yKHvY1NOIELixK3+CVu80W/7 aMQ3SUpfmyRKuBRyldvGJTswX1PG8HnPg/DDktM97qlD+gORGAQ66nApJ7axe7agvdXz d0Aw== X-Gm-Message-State: AO0yUKXWRrcgUcBAdzpLkjCclDb9PtlBEM3oxdo8lqS36Uja32kVw9BC J8m9SRvajz5l4eZveyc8Gwt4gQ== X-Received: by 2002:a17:906:3a91:b0:88b:a30:25f0 with SMTP id y17-20020a1709063a9100b0088b0a3025f0mr3397279ejd.32.1675280666358; Wed, 01 Feb 2023 11:44:26 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:26 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 2/6] KVM: VMX: Avoid retpoline call for control register caused exits Date: Wed, 1 Feb 2023 20:46:00 +0100 Message-Id: <20230201194604.11135-3-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659600770136687?= X-GMAIL-MSGID: =?utf-8?q?1756659600770136687?= Complement commit 4289d2728664 ("KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers") and avoid a retpoline call for control register accesses as well. This speeds up guests that make heavy use of it, like grsecurity kernels toggling CR0.WP to implement kernel W^X. Signed-off-by: Mathias Krause --- Meanwhile I got my hands on a AMD system and while doing a similar change for SVM gives a small measurable win (1.1% faster for grsecurity guests), it would provide nothing for other guests, as the change I was testing was specifically targeting CR0 caused exits. A more general approach would instead cover CR3 and, maybe, CR4 as well. However, that would require a lot more exit code compares, likely vanishing the gains in the general case. So this tweak is VMX only. arch/x86/kvm/vmx/vmx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c788aa382611..c8198c8a9b55 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6538,6 +6538,8 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return handle_external_interrupt(vcpu); else if (exit_reason.basic == EXIT_REASON_HLT) return kvm_emulate_halt(vcpu); + else if (exit_reason.basic == EXIT_REASON_CR_ACCESS) + return handle_cr(vcpu); else if (exit_reason.basic == EXIT_REASON_EPT_MISCONFIG) return handle_ept_misconfig(vcpu); #endif From patchwork Wed Feb 1 19:46:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51537 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp478407wrn; Wed, 1 Feb 2023 11:53:49 -0800 (PST) X-Google-Smtp-Source: AK7set/iMBVkS5bOuO6j4o7ZSfcInY6tyw366jrpReiIj3UuelLZrLl/1M2OjiUmrCA1G6dP3C18 X-Received: by 2002:a50:ab13:0:b0:46c:d2a3:76b3 with SMTP id s19-20020a50ab13000000b0046cd2a376b3mr3765148edc.14.1675281229542; Wed, 01 Feb 2023 11:53:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281229; cv=none; d=google.com; s=arc-20160816; b=f8ATyETGFGaw/DI0RA0GEHc0HlvoocyjhsudBiQiRe6H6c8mvnV7RSS2fekC2LxuJ5 AJ4CVxzlZsELtTOcr9vQ6X3BmDF9WY+dLB/aZYXgA55hdCAHhneUnSw/YKaX6AbeGJqX /+OQLudjfzK8fjf9veP8itC9N6gRFJ1F96ZC4YK3ix8CdpAoWsmiR2v5qaFxSEjGc1Ch 5vGEU8QPcxuh/Dmap4/LTZB9Y4yLUTQgpNalOSVB/G62wVEUPj83yh2scONl5seQqSoK T2WyIk5Ps1x29vG2DpuJpQcW/BJGrhNN4AY3dLc337XH40cy33ErpbSXB5SVl3381iQQ fVVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2E5YMmSOIqZe/jnf+n11QdKNSwd3QkONdPmR1/5wHHU=; b=F9lj80s6hZ1ZATBYdqisucIvMlKWgYzHSkuCD8+i4GV/oP3PJ1n13trST5k8FHtVM4 EFmJRoV0xfz1GO3kJprebmlZZtXock5I+6Mj/mu4anavLJLlEdmiXKt7hLuy1C0aTr3f 572xTXlXtapKDWrO+e83yGSmNuqssdIhsllnwgh8EBl+T+q3amGVTS4dc8mg6sbXVIkE w3lbepHv7dG0SE3uOxHTSt8HsFFGft4SatGPJYQC9HxMZ1Ug/4/o7EbKBfUffbzH3QGF LVDnwJSejuU5XfDQXmy2gJ/zqljsWA4yPrlcEkc5PxpAPu8sxa2wbWIfSpNggt6NyMB0 Yc1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=dN6uhAsK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c13-20020a0564021f8d00b0049e07ea3918si24645236edc.520.2023.02.01.11.53.25; Wed, 01 Feb 2023 11:53:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=dN6uhAsK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232280AbjBAToj (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232108AbjBATo3 (ORCPT ); Wed, 1 Feb 2023 14:44:29 -0500 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A49F78ACB for ; Wed, 1 Feb 2023 11:44:28 -0800 (PST) Received: by mail-ej1-x62d.google.com with SMTP id hx15so35095972ejc.11 for ; Wed, 01 Feb 2023 11:44:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2E5YMmSOIqZe/jnf+n11QdKNSwd3QkONdPmR1/5wHHU=; b=dN6uhAsKaM6VDdtUOahHFzOhYO0AzcwTuctDdbSBi1eK7+SJZo9eHgYPeHkXfTqYXM jFGUyi6ConRUKTp/znJ+WnsVXoR8yHvTEM9JJt/hz5C4/Hi16zD1lVck944gFlUofPyA X4LxeoB7DxASdjidq8sq/Aw9rQKVqp8uRZIspm/O2CSouFTY0go1AI3HK6vy4ZNjnfeh C3BKsRtassll4W9+PjLg6hklei4tWVPFFoChPdCsl8zxVIycdYQAQE6iZ5qxqgKsBpnS AoBvG6UClRQ3ZoCpCwcOwIxtk2ViC8B6QPuFmDUxCtBE4h9vlap6TJAXndDECJckgs75 ynKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2E5YMmSOIqZe/jnf+n11QdKNSwd3QkONdPmR1/5wHHU=; b=syiRBo/er9F6AI1EtwVzog00ZWlTLeOS/WSqvrr+O1aPiQ2jTThNlAjGQW/gOZDAxz Y+CsmYmYnPaqSFI7hm1LMMvC9t4p3P6OTrKbmZw62TCkZbQjEVWvj/TdyibuTpxpKjJE 89Uomo9TZq9zFPP9p9JmWBgAAYhchZfObeoeunrqOu2G4Wxr9o4+DGE/cOsFAHyQ8qF4 QZdm0gd1meabf7amlSBOMQdywau/EICZAvID0lFTXk6xplpv0j8N4T74xTerrtFJn28n lzAbLa+6Gn6dZ1SqelhiLvC5cSvasSheLNHmZadwYsl+MOAYPPB0mjOpdwz6SVGNZxX4 ie3w== X-Gm-Message-State: AO0yUKUD53frgayjwvmMgC6xYm0qHRjedZ5TZrQFsLr+WVCiHd36xVEy mutCdWZSjvyC0k1xy1K6QKnXxg== X-Received: by 2002:a17:907:2d28:b0:88e:682e:3a9e with SMTP id gs40-20020a1709072d2800b0088e682e3a9emr2765126ejc.61.1675280667157; Wed, 01 Feb 2023 11:44:27 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:26 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 3/6] KVM: x86: Do not unload MMU roots when only toggling CR0.WP Date: Wed, 1 Feb 2023 20:46:01 +0100 Message-Id: <20230201194604.11135-4-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659690817851628?= X-GMAIL-MSGID: =?utf-8?q?1756659690817851628?= There is no need to unload the MMU roots for a direct MMU role when only CR0.WP has changed -- the paging structures are still valid, only the permission bitmap needs to be updated. One heavy user of toggling CR0.WP is grsecurity's KERNEXEC feature to implement kernel W^X. The optimization brings a huge performance gain for this case as the following micro-benchmark running 'ssdd 10 50000' from rt-tests[1] on a grsecurity L1 VM shows (runtime in seconds, lower is better): legacy TDP shadow kvm.git/queue 11.55s 13.91s 75.2s kvm.git/queue+patch 7.32s 7.31s 74.6s For legacy MMU this is ~36% faster, for TTP MMU even ~47% faster. Also TDP and legacy MMU now both have around the same runtime which vanishes the need to disable TDP MMU for grsecurity. Shadow MMU sees no measurable difference and is still slow, as expected. [1] https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git Co-developed-by: Sean Christopherson Signed-off-by: Mathias Krause --- v2: handle the CR0.WP case directly in kvm_post_set_cr0() and only for the direct MMU role -- Sean I re-ran the benchmark and it's even faster than with my patch, as the critical path is now the first one handled and is now inline. Thanks a lot for the suggestion, Sean! arch/x86/kvm/x86.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 508074e47bc0..f09bfc0a3cc1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -902,6 +902,15 @@ EXPORT_SYMBOL_GPL(load_pdptrs); void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) { + /* + * Toggling just CR0.WP doesn't invalidate page tables per se, only the + * permission bits. + */ + if (vcpu->arch.mmu->root_role.direct && (cr0 ^ old_cr0) == X86_CR0_WP) { + kvm_init_mmu(vcpu); + return; + } + if ((cr0 ^ old_cr0) & X86_CR0_PG) { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu); From patchwork Wed Feb 1 19:46:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51540 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp479032wrn; Wed, 1 Feb 2023 11:55:26 -0800 (PST) X-Google-Smtp-Source: AK7set/WVmq7KrBLeX//s0suBJ3r7sapmYURa4jPq9201H7JKENWBZ46te5S+Wn2dfoexzxl6QUM X-Received: by 2002:a05:6402:294c:b0:49e:15ee:4f72 with SMTP id ed12-20020a056402294c00b0049e15ee4f72mr3580871edb.36.1675281326266; Wed, 01 Feb 2023 11:55:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281326; cv=none; d=google.com; s=arc-20160816; b=wCgP2hWAuxxukTILPOWk1Te1UHPOgMwktlJIc4V5oIs2OdWzL2NoTZ3sQiRIyupIOM LuhcjukuIbXt/Z9ko+Cak3jRxMK3ijEqvN69kBKNi1tM8GnCFLqzM4GXme0nu4DOK5/U XcVuwVwmsj22+txSukfoM7ZDidd21HE0QtquNqWRslibvMhi2fdjZVi3XJZk1MxWa2oa g55wfuYhRGQKdwSsKyF3tzU4ciOGD35dhvKJfvjNpmfGqXnh8XOGs113XxdIk2X76C1U QwMCdEtmnMlxlJzZ96scH97pE0spi/5G2EXX+yHBYLACetweDmId7RpWUXg8PGP6thDt Gr3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KunpYARU3Z8opuHOQvkXAppY0penEFLU7lOORGdMuTM=; b=vWHiHCw2qJlEyy1Qb1Knz9J5x+Achf0iFkGtLTyBCyHkCvf7Int+UMVgOJNCTw1zNH rAdcX26ezjQa/lGpvEDjR20cqKhas2AEtbBtUmJEPrAXOAZd1MpR+NF3cCzocqRJ59nZ DwCMHtGnSkfNeEv+qcl+KKv+nYJCdUTMsnmotMdkUlN8DNo+q15lAEed7zqc/Ch4FA1W IsXS8bz55bMCewogrCMjZy5kp9p9bR0BhEcXdJBH8qz38eG5lKriZJB5ISeksFbOdICA cozCcThVPn031Onn6mm4yfFRthBgZOUicU0RqNNkxNThQA6F5hXqGWHsj9usTYmTOL7+ 0uew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=AI1HafiS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x3-20020a056402414300b0049fe17759bfsi22980676eda.230.2023.02.01.11.55.02; Wed, 01 Feb 2023 11:55:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=AI1HafiS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229546AbjBAToh (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232141AbjBAToa (ORCPT ); Wed, 1 Feb 2023 14:44:30 -0500 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 777E37A4B4 for ; Wed, 1 Feb 2023 11:44:29 -0800 (PST) Received: by mail-ej1-x62c.google.com with SMTP id lu11so17314023ejb.3 for ; Wed, 01 Feb 2023 11:44:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KunpYARU3Z8opuHOQvkXAppY0penEFLU7lOORGdMuTM=; b=AI1HafiS7yvl+3Vfl6WZGwTbF4dqFNC7qE0AlRRxIcILQypAR3O1OWlF8mQVoGQ/aN mPFTcy+IcLwRC9EpggiAmx2I7s2RFGMAWK3OWFi8Qo36/dHwJnxDx4Z7q3dTJP35uKan EjiialoxgW7rIayXBv3gpS2QYrrVFaOnZZPeXZpVs3xdSEPsXpz91JLi40vUF26p5HK6 RImt/CnMpqcbH88FaixMQAmxzE6C57B+3hSyFkaCI3IvkQyGK2lqHBhFCWzhkNcK5FMJ sl68khinMx9LmNefwLC8hP5uKyf64IkbT9FRHCvBQB9EEDkQBfaB66dsVn88wIQwazFW GApw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KunpYARU3Z8opuHOQvkXAppY0penEFLU7lOORGdMuTM=; b=DjLcpr2CeTX3C20ifu060U5N3hWGjbIurXj1RGwWG/XtcAmvy7i2Zg4HZ7XeadykWL 1Nw0+yQOh5U1c4WExxnT5QaHeQB9hKRTRg5I/RVsF5A0qspYibh6uvTCfDjvqbFsgy3D 1OOmj2maUyT74IF5mEvcdBbD8/GTVC3heQImbFUuRqMwbz6Aqn+pOMKlJ9D8wtLiQzpQ G3N71S/QWQqc0fqNNOBQnpAVuoV34oy7sbQGP+5d1bwa1IEbJkqbMfmQ1qpzI7axYUma 3P+QM4B8VTS4/leQHML+ByG9XFkzJEHQtWqMXPwle2ak+tpMBLeROhU8rLOkQhyCHscO /QDw== X-Gm-Message-State: AO0yUKU8ZqmKdGeBLkkmCPnL/6+bWSpM4XUiZPaOjkLcZRQzDE94KjS2 fECzXx8gyMB7H/ZpY0KLSZoPWoTGrFKgGPJ1MNA= X-Received: by 2002:a17:906:dd4:b0:877:667b:f1e2 with SMTP id p20-20020a1709060dd400b00877667bf1e2mr3330879eji.11.1675280667922; Wed, 01 Feb 2023 11:44:27 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:27 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 4/6] KVM: x86: Make use of kvm_read_cr*_bits() when testing bits Date: Wed, 1 Feb 2023 20:46:02 +0100 Message-Id: <20230201194604.11135-5-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659791963470621?= X-GMAIL-MSGID: =?utf-8?q?1756659791963470621?= Make use of the kvm_read_cr{0,4}_bits() helper functions when we only want to know the state of certain bits instead of the whole register. This not only makes the intend cleaner, it also avoids a VMREAD in case the tested bits aren't guest owned. Signed-off-by: Mathias Krause --- arch/x86/kvm/pmu.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index d939d3b84e6f..d9922277df67 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -439,9 +439,9 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) if (!pmc) return 1; - if (!(kvm_read_cr4(vcpu) & X86_CR4_PCE) && + if (!(kvm_read_cr4_bits(vcpu, X86_CR4_PCE)) && (static_call(kvm_x86_get_cpl)(vcpu) != 0) && - (kvm_read_cr0(vcpu) & X86_CR0_PE)) + (kvm_read_cr0_bits(vcpu, X86_CR0_PE))) return 1; *data = pmc_read_counter(pmc) & mask; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c8198c8a9b55..d3b49e0b6c32 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5487,7 +5487,7 @@ static int handle_cr(struct kvm_vcpu *vcpu) break; case 3: /* lmsw */ val = (exit_qualification >> LMSW_SOURCE_DATA_SHIFT) & 0x0f; - trace_kvm_cr_write(0, (kvm_read_cr0(vcpu) & ~0xful) | val); + trace_kvm_cr_write(0, (kvm_read_cr0_bits(vcpu, ~0xful) | val)); kvm_lmsw(vcpu, val); return kvm_skip_emulated_instruction(vcpu); @@ -7547,7 +7547,7 @@ static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) if (!kvm_arch_has_noncoherent_dma(vcpu->kvm)) return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; - if (kvm_read_cr0(vcpu) & X86_CR0_CD) { + if (kvm_read_cr0_bits(vcpu, X86_CR0_CD)) { if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) cache = MTRR_TYPE_WRBACK; else From patchwork Wed Feb 1 19:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51538 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp478422wrn; Wed, 1 Feb 2023 11:53:51 -0800 (PST) X-Google-Smtp-Source: AK7set88PLa/WZ0lBDhPGZKsY8VDa44kG4CIyikU8I3DOqnEisxVtDhnL2uuYJsCgzbZV1R8BMPa X-Received: by 2002:a05:6402:320d:b0:48e:f679:e733 with SMTP id g13-20020a056402320d00b0048ef679e733mr4029657eda.36.1675281231210; Wed, 01 Feb 2023 11:53:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281231; cv=none; d=google.com; s=arc-20160816; b=Dj3pREn0nwUhP2PAuTTPDntCeeNOZnYN3RfRD6sWcLwibhCKzmTjHUmQ8p7m5UgBsz FmXECKoWNBXkc3sTEnIdu26IVGrbNbUdhrsmmMiiI7ZZtyxNy4+PVLPGhumgH9mIBea9 Y+iN23+KIyMDm6QHZsKrAinfxRYrtvQoLe2RoCR0vKZd2HMIKG8OwfPkWOpLaWqb9gA7 Qp6xqtu3HQ8+qv+LQyZD/6ca7k/v2cRX3P1lGxA2LL1SE/i1jmBoHSosSpUlJYIFGuDx g6ut8EXlW6b/2dy4Rpg0vJrymR+KCLKEgpogcnx+7/bENPdTtIxBGV6tLHW/SM0NnIEB 7JqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4WIfgFu7sWBxHubiDa704oZ6rZ4jmUPQhefB2XMq/DY=; b=XY2+SMWDGEA8vE86UIkUzN1VPT5oBzBykgRzrmnvCjz0OQ5vnDopAskdEHkrAgaVAj FMpXbEPGo6qWpGUkSjVcSz2MAWnxpeAp/6ZBNGMccN7S7ATU/WSXjmwxO31EcyfSwQ94 4jCiZ6qGevP9HtTX3lRuctU9sFPZrTyiTLnDatmEyt2GMMqxSCQvIoHuniPLYZtuNRrV 9HJzXzhFfzlxHUVlW0I24cX5aSsGAKiNzSFnfe/5itNiPaZfJS51B2xW4ELwK8dcxGyJ VF0F7DynKGnyDjdX7zPIPquIL0a6TfOVk1bEZUiaY7fpi7R82EOUbdYMooHdG0dDVlcw fe7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=HWSMoT32; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z41-20020a509e2c000000b0049e2dbaeb16si24345529ede.570.2023.02.01.11.53.27; Wed, 01 Feb 2023 11:53:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=HWSMoT32; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232272AbjBAToo (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232158AbjBATob (ORCPT ); Wed, 1 Feb 2023 14:44:31 -0500 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E6D17D6CF for ; Wed, 1 Feb 2023 11:44:30 -0800 (PST) Received: by mail-ej1-x62a.google.com with SMTP id m2so54177386ejb.8 for ; Wed, 01 Feb 2023 11:44:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4WIfgFu7sWBxHubiDa704oZ6rZ4jmUPQhefB2XMq/DY=; b=HWSMoT32VGWmMlCTrhxHivAlxKAAZNSvyoygy4yjRPBu0W4L15o7HIYq85B7iah+I/ Z8PNyxMMnBzbh7df65L7txwlSvM4GY0KOAyEkXIRwH8BLecSy6HwHBBKPhcgTgPAMoXz o8jExm4r5vV/LZ4W32jN8QUXeTT5lPMYKcMsECiA+wYAsgkZbKBlgdHyMu5TTxk7DODO dPsPPeEqIoqS9WC9kReJaacLryNx8eM9ZTa6zlD5QZvG1kHwckHUNEolVKTlKaSBPWzQ d7qoDWqz/ZNeISAZOGJnH1tCuM7GVVaOIFg7s2fTi1gNcWU4FkXs9x4IPupNRMhT4NDM b8Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4WIfgFu7sWBxHubiDa704oZ6rZ4jmUPQhefB2XMq/DY=; b=WH5u6vmhy0qkoMLSn0QAG6nH+K+I3LQBN/VtCzGjc4P6NoBAg6AuJFpvT3KKutnWmm ZZtsi+nSYv8qfIGvNQMwmA1IF+wTLKLkZF5A61dmcHkxSs7Tej4nUhYESbQI4HKOKCAc I37BFGANhcA1NND8msO70CWDxRlSsWlc5jLY1Juyq7TQSnvyVqqSDyA93USy6lAi4q5J 2LVX2odfUC4ItHyHCyXUwIrVKxIjf+aAWtyhEKXmX/1tO3N11Snr6+tuf6ewkJMJu04c Skm47qwPgzPFSWtB+vluxF/TMrnm807xKrbdSoAkbNV8zjvSXSWuChvrKVGeT8G+rTcX Iwyg== X-Gm-Message-State: AO0yUKVgjhVocWINxjrkw/igMHTZH89t7ZRBz7QRb6oLJ93yNej8AHhp sm7RSiymzTGj1A6R03QGNfIJdQ== X-Received: by 2002:a17:906:8281:b0:862:11f6:a082 with SMTP id h1-20020a170906828100b0086211f6a082mr3412157ejx.17.1675280668753; Wed, 01 Feb 2023 11:44:28 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:28 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 5/6] KVM: x86/mmu: Fix comment typo Date: Wed, 1 Feb 2023 20:46:03 +0100 Message-Id: <20230201194604.11135-6-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659692180273135?= X-GMAIL-MSGID: =?utf-8?q?1756659692180273135?= Fix a small comment typo in make_spte(). Signed-off-by: Mathias Krause --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index fce6f047399f..95441ffdccc8 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -164,7 +164,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, /* * For simplicity, enforce the NX huge page mitigation even if not * strictly necessary. KVM could ignore the mitigation if paging is - * disabled in the guest, as the guest doesn't have an page tables to + * disabled in the guest, as the guest doesn't have any page tables to * abuse. But to safely ignore the mitigation, KVM would have to * ensure a new MMU is loaded (or all shadow pages zapped) when CR0.PG * is toggled on, and that's a net negative for performance when TDP is From patchwork Wed Feb 1 19:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 51539 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp478447wrn; Wed, 1 Feb 2023 11:53:54 -0800 (PST) X-Google-Smtp-Source: AK7set+OjEfmnBN2QFunA+1kb/q16y15Cenri7PFytU9Di1Dsurst/2LDlloe+ks8wpZSphtJ9Fh X-Received: by 2002:aa7:d406:0:b0:4a2:5a66:f4a8 with SMTP id z6-20020aa7d406000000b004a25a66f4a8mr3463711edq.19.1675281233979; Wed, 01 Feb 2023 11:53:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675281233; cv=none; d=google.com; s=arc-20160816; b=CbeLzzfhA0hVYRcOi3aAFj2xmfaDqQOlfG3TxVesrnFWVuOKfi1Bmux8iJ/poykPGL 8Hr9khrQ4fMSADHqcxivj1ggHJVARnNN7VU3m8ySgxBLt1v0ZxsGxELeTuyHwOM7oOaG PAgxU8RObi2UyAbYA21rQnapKXRp5aaeSiP6Nu/49iGcBwVRyGrhlMHGLbcj0Tm2onZc NjtF74UFfcooOLx/ZRPpKXJ8hbZdda6qyLJpqo6JP2xVPH+J9aIqf6Oo02rqNd4jM2aK 5J6DnuWlL7rbgcsvqGt9+49DYBu15GVC/Vs6REKuEHFDUe2zwiiCcebd4zJuF68SzvQj 776Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Qsy2Uw5CdDxyHnHzJNf7tebm5waQcm3qgkQypIRD0TQ=; b=uIE1vDjSl2pxG3Dhk5kOdNFjASVr42ETQXrqwQlk4BetLeVo937S0W+UPlxgxHxTAN 7ffTZvJb7y5e3IpLAh1A55ExYpGGSB/qBBfIbgSHbQizw9++syNAFvH/iFHhCQKLnMUT ZUHBCb2csdS+0PZCZyoXNP/670XbYww6CLXQTigHjcqVSwxiT9aEni4/jkL6P4giT79Z pP603aGnE1PgTslGCJRMvND3xeKh9a3p5XOyutT6uQgVE9m6LCBSexPD0YQLtHBftxe5 AAIbNpVRGiUDNEltJ2Ypdo06WwVbZVa01z/udmJPZUKft5zSn8IOiYzXcvNXS8Dgq0cX 1xAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=u5nOIOiI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 9-20020a170906020900b0088a161c2342si10484131ejd.192.2023.02.01.11.53.30; Wed, 01 Feb 2023 11:53:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=u5nOIOiI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232301AbjBATos (ORCPT + 99 others); Wed, 1 Feb 2023 14:44:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232159AbjBATob (ORCPT ); Wed, 1 Feb 2023 14:44:31 -0500 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4CA074C3E for ; Wed, 1 Feb 2023 11:44:29 -0800 (PST) Received: by mail-ej1-x630.google.com with SMTP id ud5so54662157ejc.4 for ; Wed, 01 Feb 2023 11:44:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qsy2Uw5CdDxyHnHzJNf7tebm5waQcm3qgkQypIRD0TQ=; b=u5nOIOiIHGHsUUIO2RFWXjN11TEIX9e+ptL1+CFtVB/c9QeQUBW4CCMKhj+d+jZC9W o/pFuvwXHpvKU8vlW/cQMaNDWw9ok4uB7LnEfQZYqdJ4RbicM3mJwYF33qErrH0McdbG dol1+omltPpOKQY+gJs/XG9hjc4gUAUwj6uo+Uf29HCeDeyK9S7kBCmVzfj+WAdTaiQq GBlkXnew/QD5OUivpAPWqzpIL6dpx4aP3jDDVIlYTRhATKAcyEr1YPyss0/IcWNpOZ3C B7UvaRb7awdj/0U0aX0rRbASJ8Glxqm26OFAY87zPlrP9k5ra2Xo6PpZYUfrgVtaR6CQ zcrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qsy2Uw5CdDxyHnHzJNf7tebm5waQcm3qgkQypIRD0TQ=; b=fXNO8k/q1N45whpX3NDyaJLv0ua6vdFi+Rsy31AohRayj+U2HZLtnZT5WBWovSqfxK Hi6XWF5QBt3X455r5k+iD5NXJrov+nyJe/kl1HD5raWhpkYe9jNUVl9zKbBTOMqkzLwp aLkNeFmWbAgnSi64ySFMBBWEZXCE74WiJZ3a5+YJw1THcxcKnTT5pOApzTILscCudP3q iZcxehxfH3eC8qL/Wc9Z7MF8hu8ljVP7iKGJEzpflS+cCvEwG+gMvxA8t0BRWyc1W3MH Tl6Mo5qhjPXJ7LSr88D2F494otMHpiywUgU3znHWTi2TB8AZFBnaCzDuMEIWOB9XyEat g0JQ== X-Gm-Message-State: AO0yUKVZqG5N4urygbL2zp8Yw71voXxpjuZXwMCOON1aalxZiGmH9kLT P+Gan9ZAzoNWwLoMUMuWoPSIUg== X-Received: by 2002:a17:906:4c8a:b0:887:c44a:b8e9 with SMTP id q10-20020a1709064c8a00b00887c44ab8e9mr3727788eju.76.1675280669450; Wed, 01 Feb 2023 11:44:29 -0800 (PST) Received: from nuc.fritz.box (p200300f6af111a00277482c051eca183.dip0.t-ipconnect.de. [2003:f6:af11:1a00:2774:82c0:51ec:a183]) by smtp.gmail.com with ESMTPSA id c23-20020a170906155700b00869f2ca6a87sm10397579ejd.135.2023.02.01.11.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 11:44:29 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v3 6/6] KVM: VMX: Make CR0.WP a guest owned bit Date: Wed, 1 Feb 2023 20:46:04 +0100 Message-Id: <20230201194604.11135-7-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230201194604.11135-1-minipli@grsecurity.net> References: <20230201194604.11135-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756659695496606854?= X-GMAIL-MSGID: =?utf-8?q?1756659695496606854?= Guests like grsecurity that make heavy use of CR0.WP to implement kernel level W^X will suffer from the implied VMEXITs. For a direct MMU role there is no need to intercept a guest change of CR0.WP, so simply make it a guest owned bit if we can do so. This implies that a read of a guest's CR0.WP bit might need a VMREAD. However, the only potentially affected user seems to be kvm_init_mmu() which is a heavy operation to begin with. But also most callers already cache the full value of CR0 anyway, so no additional VMREAD is needed. The only exception is nested_vmx_load_cr3(). Add a new module parameter 'lazycr0' to allow users to revert back to the old behaviour by loading kvm-intel.ko with 'lazycr0=0'. This change is VMX-specific, as SVM has no such fine grained control register intercept control. Suggested-by: Sean Christopherson Signed-off-by: Mathias Krause --- Initially I wanted to implement the scheme Sean sketched[1]: having a threshold where we would switch from eager to lazy CR0.WP tracking after toggling the bit often enough, make the bit guest owned afterwards and VMREAD CR0 when needed. However, when starting to look for users that would be affected, I only found kvm_init_mmu() (via kvm_init_mmu() -> vcpu_to_role_regs() -> kvm_read_cr0_bits(KVM_MMU_CR0_ROLE_BITS)). It has only these three interesting callers: 1/ kvm_mmu_reset_context(), which isn't all that interesting, as it's a heavy weight operation anyway and many of the control flows leading to it already cache the value of CR0, so no additional VMREAD is needed, 2/ nested_vmx_load_cr3() and 3/ kvm_post_set_cr0(), only when CR0.WP was toggled and the MMU is in direct mode (optimization introduced by patch 3). The last case's most interesting caller is likely kvm_set_cr0(), which already caches the written CR0 value, thereby vanishes the need for another VMREAD in vcpu_to_role_regs(). That's why I went with the much simpler approach and always allow CR0.WP to be guest owned if EPT is enabled as well. There's nothing we can do for SVM, though :/ [1] https://lore.kernel.org/kvm/Y8cTMnyBzNdO5dY3@google.com/ --- arch/x86/kvm/kvm_cache_regs.h | 3 ++- arch/x86/kvm/vmx/capabilities.h | 1 + arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 9 ++++++++- arch/x86/kvm/vmx/vmx.h | 8 ++++++++ 5 files changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index c09174f73a34..495ae0204933 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -4,7 +4,8 @@ #include -#define KVM_POSSIBLE_CR0_GUEST_BITS X86_CR0_TS +#define KVM_LAZY_CR0_GUEST_BITS X86_CR0_WP +#define KVM_POSSIBLE_CR0_GUEST_BITS (X86_CR0_TS | KVM_LAZY_CR0_GUEST_BITS) #define KVM_POSSIBLE_CR4_GUEST_BITS \ (X86_CR4_PVI | X86_CR4_DE | X86_CR4_PCE | X86_CR4_OSFXSR \ | X86_CR4_OSXMMEXCPT | X86_CR4_PGE | X86_CR4_TSD | X86_CR4_FSGSBASE) diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index 45162c1bcd8f..41d48a3a651e 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -12,6 +12,7 @@ extern bool __read_mostly enable_vpid; extern bool __read_mostly flexpriority_enabled; extern bool __read_mostly enable_ept; +extern bool __read_mostly enable_lazy_cr0; extern bool __read_mostly enable_unrestricted_guest; extern bool __read_mostly enable_ept_ad_bits; extern bool __read_mostly enable_pml; diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 557b9c468734..2a0010ca7277 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4478,7 +4478,7 @@ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu, * CR0_GUEST_HOST_MASK is already set in the original vmcs01 * (KVM doesn't change it); */ - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs12->host_cr0); /* Same as above - no reason to call set_cr4_guest_host_mask(). */ @@ -4629,7 +4629,7 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) */ vmx_set_efer(vcpu, nested_vmx_get_vmcs01_guest_efer(vmx)); - vcpu->arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vcpu->arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmx_set_cr0(vcpu, vmcs_readl(CR0_READ_SHADOW)); vcpu->arch.cr4_guest_owned_bits = ~vmcs_readl(CR4_GUEST_HOST_MASK); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d3b49e0b6c32..1969360d2744 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -91,6 +91,9 @@ module_param_named(flexpriority, flexpriority_enabled, bool, S_IRUGO); bool __read_mostly enable_ept = 1; module_param_named(ept, enable_ept, bool, S_IRUGO); +bool __read_mostly enable_lazy_cr0 = 1; +module_param_named(lazycr0, enable_lazy_cr0, bool, S_IRUGO); + bool __read_mostly enable_unrestricted_guest = 1; module_param_named(unrestricted_guest, enable_unrestricted_guest, bool, S_IRUGO); @@ -4765,7 +4768,7 @@ static void init_vmcs(struct vcpu_vmx *vmx) /* 22.2.1, 20.8.1 */ vm_entry_controls_set(vmx, vmx_vmentry_ctrl()); - vmx->vcpu.arch.cr0_guest_owned_bits = KVM_POSSIBLE_CR0_GUEST_BITS; + vmx->vcpu.arch.cr0_guest_owned_bits = vmx_guest_owned_cr0_bits(); vmcs_writel(CR0_GUEST_HOST_MASK, ~vmx->vcpu.arch.cr0_guest_owned_bits); set_cr4_guest_host_mask(vmx); @@ -8370,6 +8373,10 @@ static __init int hardware_setup(void) return -EOPNOTSUPP; } + /* Need EPT for lazy CR0.WP synchronization. */ + if (!enable_ept) + enable_lazy_cr0 = 0; + if (!cpu_has_vmx_ept_ad_bits() || !enable_ept) enable_ept_ad_bits = 0; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index a3da84f4ea45..e899c2291a3f 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -640,6 +640,14 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) (1 << VCPU_EXREG_EXIT_INFO_1) | \ (1 << VCPU_EXREG_EXIT_INFO_2)) +static inline unsigned long vmx_guest_owned_cr0_bits(void) +{ + unsigned long bits = KVM_POSSIBLE_CR0_GUEST_BITS; + if (!enable_lazy_cr0) + bits &= ~KVM_LAZY_CR0_GUEST_BITS; + return bits; +} + static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) { return container_of(kvm, struct kvm_vmx, kvm);