From patchwork Wed Jan 18 14:50:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 45316 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2388148wrn; Wed, 18 Jan 2023 07:07:50 -0800 (PST) X-Google-Smtp-Source: AMrXdXuKTCz7D/EDb2VnHbiaFsZJF5rxziecpiDNs+L5cN365xCe5wurfWYMzNYKPpOs4fSHsufs X-Received: by 2002:a17:906:94d0:b0:7c4:f0b5:fedc with SMTP id d16-20020a17090694d000b007c4f0b5fedcmr7657023ejy.65.1674054470650; Wed, 18 Jan 2023 07:07:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674054470; cv=none; d=google.com; s=arc-20160816; b=PVGzQ6MajCX5QNVsycVEZyZodxv6hBpPE2Vij24Tq/rLEUfY2ME5+8t/eo+aDF9Mf9 buA6Vu5cKjcC60/GR+lGDaM7BoQce12Xnst/FylzEmM154MqoTpCnTZaK4/DS+irUpaj 4XcdaOSj4CPyO/uILoA03LT4HAiTf9MquDA9bADpAyH8kwJdz2yo1IlWYRctcW7H6CoK wZyWxqa5ImqxzbOQ2I+cZx8OCOhLa+OYdTaLfh8WvgM6oezB4tUBG/QosxNjfOb2/nd5 2Gl4yEbNDvmpqdJDwE1spLZy4ZnA0I2SVApQs4zu3f6bOAptWsAElzsKeJi8OpGerRif t36w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=joD6UNobMMls+6tKVhM1/laXbk6jjwsQnCEiOHpSVIA=; b=0OlssNxq9sEz05OqUQDbmzNTez5azJKhhmbse0uM916DBLIsMPSe2/oNkdQ/PipQj5 sqk5pB6h5civiBK7LD9zQGR6dgyBo/7t+b/9cM1/nbJhgSyzjFKR6ii32izbZykqk5OS X4xNH7pUEetKsi6JbQngUHO600cU/1ah+mfg5hW1jJM75BNvhU6y1qrctZM7qXjbDVLF ahJSEqd8+0eCexvqLdkjV84QbmGWFwSKXz4oOTooCeIlE7ebBbhKKG7OdLz302t3ljz0 SfepRvgdK+a5T+mfisBeGp3i/3JF1mNkS/1ObSeneS7A/TSR7T8aX1c1vhB4WhFs62vm ofDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=kcTDj7kf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ht13-20020a170907608d00b007ba713e241dsi42333695ejc.894.2023.01.18.07.07.26; Wed, 18 Jan 2023 07:07:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=kcTDj7kf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231562AbjAROyd (ORCPT + 99 others); Wed, 18 Jan 2023 09:54:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231590AbjAROyC (ORCPT ); Wed, 18 Jan 2023 09:54:02 -0500 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60C7FE3B8 for ; Wed, 18 Jan 2023 06:48:52 -0800 (PST) Received: by mail-ed1-x535.google.com with SMTP id s21so5717935edi.12 for ; Wed, 18 Jan 2023 06:48:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=joD6UNobMMls+6tKVhM1/laXbk6jjwsQnCEiOHpSVIA=; b=kcTDj7kf9QUJL+m+KQdNrIqCOkXZxTmuAvbzi81FXwYxVb+iwApEJJeoKQAwCF9lar nPAW2STRniXosa99/5v96Ok1DbvuMCIfuz0pHV+XG/DMBOVkt04a4+Hven9XxZaSX+B0 1jSd2+vjSESJDsW3PGJatIc4Yq7db3Lnuwc3FsW3jk4h6DhYWiwzIHvt8pR/lrR7ynap VkqfDr7Y9Hj5Lp7TEAi/G+0w6QjF5pwRUhwyODQxzl6HYsJ697XRNFHX0bgPAVy4ADi5 /ju1rCNHmd4/Mxa4rpPnm9DE5Gfe3QQZF5eJ6zVVpi9qitRuHPwxwKHBZET61/Lk0D3/ tJPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=joD6UNobMMls+6tKVhM1/laXbk6jjwsQnCEiOHpSVIA=; b=NwWWMu4nOhK1cz9Cbc/8nbl0jQx/57VCorN8bNMytdemSUjpRWrDLtMzevYZod6Qto dm/WnOfJK2GLxZXguGYk1wmHbbTqwveTuViuGW0y2Zm/gQyTGFa9YfytYe5JBe/jeafJ x1GfFKyjhyXSQvl5Qx59h2ztB9tKv6MAnTrkKg306cN4bvl3yoyEhID+aA728gREQVzF tf1mR/XUe2+DoqUVC7+6fVh2XTd5M56LF6ppMt4+imqozPu2aBGOVzO9qE61amC2KJNf 0XntrhfnDtYCDcZ90uGZxRakjZm0LkpzO57wIEKVUl+v2pHexIeIfijTzRUaOqpYq41d NXKg== X-Gm-Message-State: AFqh2kqid0np9NnXWwT36ir6KCGkfgwW2Wwyf8nTjZUeUVxOUr4jHDbC 6cbnL8N2sFWg/F2rrUhcJF+3jg== X-Received: by 2002:a05:6402:10c9:b0:49d:a87f:ba78 with SMTP id p9-20020a05640210c900b0049da87fba78mr7121605edu.35.1674053330954; Wed, 18 Jan 2023 06:48:50 -0800 (PST) Received: from nuc.fritz.box (p200300f6af03d2006e0fc0b921f9db5c.dip0.t-ipconnect.de. [2003:f6:af03:d200:6e0f:c0b9:21f9:db5c]) by smtp.gmail.com with ESMTPSA id p11-20020a05640243cb00b0049e19136c22sm3627509edc.95.2023.01.18.06.48.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 06:48:50 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v2 1/3] KVM: x86/mmu: avoid indirect call for get_cr3 Date: Wed, 18 Jan 2023 15:50:28 +0100 Message-Id: <20230118145030.40845-2-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118145030.40845-1-minipli@grsecurity.net> References: <20230118145030.40845-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755373340146089393?= X-GMAIL-MSGID: =?utf-8?q?1755373340146089393?= From: Paolo Bonzini Most of the time, calls to get_guest_pgd result in calling kvm_read_cr3 (the exception is only nested TDP). Hardcode the default instead of using the get_cr3 function, avoiding a retpoline if they are enabled. Signed-off-by: Paolo Bonzini Signed-off-by: Mathias Krause --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++++++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index aeb240b339f5..505768631614 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -241,6 +241,20 @@ static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) return regs; } +static unsigned long get_guest_cr3(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr3(vcpu); +} + +static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu) +{ + if (IS_ENABLED(CONFIG_RETPOLINE) && mmu->get_guest_pgd == get_guest_cr3) + return kvm_read_cr3(vcpu); + + return mmu->get_guest_pgd(vcpu); +} + static inline bool kvm_available_flush_tlb_with_range(void) { return kvm_x86_ops.tlb_remote_flush_with_range; @@ -3722,7 +3736,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) int quadrant, i, r; hpa_t root; - root_pgd = mmu->get_guest_pgd(vcpu); + root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; if (mmu_check_root(vcpu, root_gfn)) @@ -4172,7 +4186,7 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, arch.token = alloc_apf_token(vcpu); arch.gfn = gfn; arch.direct_map = vcpu->arch.mmu->root_role.direct; - arch.cr3 = vcpu->arch.mmu->get_guest_pgd(vcpu); + arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); return kvm_setup_async_pf(vcpu, cr2_or_gpa, kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch); @@ -4191,7 +4205,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) return; if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 != vcpu->arch.mmu->get_guest_pgd(vcpu)) + work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) return; kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); @@ -4592,11 +4606,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); -static unsigned long get_cr3(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr3(vcpu); -} - static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { @@ -5147,7 +5156,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu, context->page_fault = kvm_tdp_page_fault; context->sync_page = nonpaging_sync_page; context->invlpg = NULL; - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; @@ -5297,7 +5306,7 @@ static void init_kvm_softmmu(struct kvm_vcpu *vcpu, kvm_init_shadow_mmu(vcpu, cpu_role); - context->get_guest_pgd = get_cr3; + context->get_guest_pgd = get_guest_cr3; context->get_pdptr = kvm_pdptr_read; context->inject_page_fault = kvm_inject_page_fault; } @@ -5311,7 +5320,7 @@ static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu, return; g_context->cpu_role.as_u64 = new_mode.as_u64; - g_context->get_guest_pgd = get_cr3; + g_context->get_guest_pgd = get_guest_cr3; g_context->get_pdptr = kvm_pdptr_read; g_context->inject_page_fault = kvm_inject_page_fault; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index e5662dbd519c..78448fb84bd6 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -324,7 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; - pte = mmu->get_guest_pgd(vcpu); + pte = kvm_mmu_get_guest_pgd(vcpu, mmu); have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); #if PTTYPE == 64 From patchwork Wed Jan 18 14:50:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 45314 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2388056wrn; Wed, 18 Jan 2023 07:07:42 -0800 (PST) X-Google-Smtp-Source: AMrXdXtOtJJgcGlkJ7XBNDtD45hrwtaeXsmQFSeOOQbRtIWMoTAaxl6DVnirOYGkjeCOxYZE08XQ X-Received: by 2002:a62:15d2:0:b0:58d:be61:8bc5 with SMTP id 201-20020a6215d2000000b0058dbe618bc5mr8220914pfv.13.1674054462301; Wed, 18 Jan 2023 07:07:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674054462; cv=none; d=google.com; s=arc-20160816; b=tSNVqcKTI2hoyTWDgOgmJ+ydylnb60yT86LCNQWQVlGnXd/tnAJYzvgsBpZcgv+OFt rRVPgoEWYRRwsE3P1eAfCn7WqCBNHCi5eGkNYQIZdglxQK3fdhlK0ar3juYf4f0xXnl+ IgbxJNmKSyxWHFehw4vx3cXjQJhL53CjNp9BjlCl5xuKxU3F5A/p+nrbsJZs4P5ZA1/E duVcH8+iJYGo9bJYtRPhDnxH532T/ztBys5chIdDeDPSEon7tKaRO6Uk6vsS5qT6tfnV 5tZKOqOx46gEMLgBVdStkkRQEXKDT+AL8igjtjVqL/vrAVm5qB8o6BanKLIbgiBBKQxK 1ybQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b6W0pZJ6ERcXnvSOd8zNWpB3EIqXZWM69qX+jHbVTMc=; b=ZTf6QVlZ28nf+yqOtCeN0QY+i4XGRyH2y5+SlG24wZNsR/sNmHkwuiZYSeZOpkqdlf bgp2jGnaRpbGEiSfuce2aYaiYJogUoSgOx3OJG8THa/1FXtKYjQB2l97VTB5QUAu1RDi dq+ZD+gpBdAXyVlhzmx4FK+/PpMnYxwCclkGTZia26OPWC0nA/LtXN4xVlqx3HCbGnYX ERUWcUBx7kThympLdLukNE7hTYLJNGAC2rIS3r80C2JgZrg2UMM49C5z4QKXtY9NCn/p vvzZXqY3MEU2oXDGpq3zYwwCuwjiXK6r/Tgb79vRj9sTWFQIOuEaJ8ZTBvk+dWUkqLh3 4//w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b="IBT0qj/R"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g17-20020a625211000000b005769b34452fsi32702890pfb.203.2023.01.18.07.07.28; Wed, 18 Jan 2023 07:07:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b="IBT0qj/R"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229935AbjAROyk (ORCPT + 99 others); Wed, 18 Jan 2023 09:54:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230350AbjAROyD (ORCPT ); Wed, 18 Jan 2023 09:54:03 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E9CC17CEE for ; Wed, 18 Jan 2023 06:48:53 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id v10so48720914edi.8 for ; Wed, 18 Jan 2023 06:48:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b6W0pZJ6ERcXnvSOd8zNWpB3EIqXZWM69qX+jHbVTMc=; b=IBT0qj/RPtCvkxVkBIfHYd7QteXzIP67i647IqhTBlkd71McZh+opt3wap1cZCAih3 qyu/DTKBwBL6nJT2BGjW5K+gPq+wjvxNqGY4YPXwZLTySKFOF+VSovv5xJ7QsXwxK3y8 Q4c1c+jL2v1+8AQPHNvhgpgZInBCLehbKZ/7FKuYASFL00ghpeud/Qriwg/I3m+4SBAg AM57y1MJKZI7tIx14X5KraP3NJ0/u8reJ3vBvZhwAfMvVJ8PPRnegZh9Na9R6sWEuY0Y TiIys37zs94rkWlC81Vy9Hk1Sl/tH8OJSvrxfjuhAjZ2t0PaqTX3Ke3cW1I7Uw0NweEg E4Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b6W0pZJ6ERcXnvSOd8zNWpB3EIqXZWM69qX+jHbVTMc=; b=YLv3tJJQY2N3DfTDYrqQXUu6qeqAoz4LnLc8J0m6ESjH+sKGJ6YIF+RjpslFD2DYiG zBf6+MYHmaiTd1aQDV4tzc43PvrNtSWCNytJAO8FhJ5pBikK/Ep2KH2qTu037TxG84wD RLSW2xFWZQtABYSfbWJgQbwB6gqkbpVO0teSqB7xFH7Q/+6UFxUFV0sLR1vzcu73aaVi LwqA76DnoiX7xBUVKxgEIDAfF7B/goOTdDt4fOu8ZaKSGUAbVqaVLoigoov5lnjQRt7E TdrgboGg2gbuzBWNtHWsk0T5drVn2/j64gqynLKWgzGrfF2o+31Ovvzi4KaFzleCn8Ht I2/A== X-Gm-Message-State: AFqh2krGFqPocLidVNjBZIpCP2HyyaPy/MT8nrfSOSQe8dpMn0bshIH/ VT7/TttGOEioys6c31rtAiNYqw== X-Received: by 2002:aa7:cb01:0:b0:495:fa3d:1d72 with SMTP id s1-20020aa7cb01000000b00495fa3d1d72mr7477248edt.8.1674053332042; Wed, 18 Jan 2023 06:48:52 -0800 (PST) Received: from nuc.fritz.box (p200300f6af03d2006e0fc0b921f9db5c.dip0.t-ipconnect.de. [2003:f6:af03:d200:6e0f:c0b9:21f9:db5c]) by smtp.gmail.com with ESMTPSA id p11-20020a05640243cb00b0049e19136c22sm3627509edc.95.2023.01.18.06.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 06:48:51 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v2 2/3] KVM: VMX: avoid retpoline call for control register caused exits Date: Wed, 18 Jan 2023 15:50:29 +0100 Message-Id: <20230118145030.40845-3-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118145030.40845-1-minipli@grsecurity.net> References: <20230118145030.40845-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755373331694193015?= X-GMAIL-MSGID: =?utf-8?q?1755373331694193015?= Complement commit 4289d2728664 ("KVM: retpolines: x86: eliminate retpoline from vmx.c exit handlers") and avoid a retpoline call for control register accesses as well. This speeds up guests that make heavy use of it, like grsecurity kernels toggling CR0.WP to implement kernel W^X. Signed-off-by: Mathias Krause --- SVM may gain from a similar change as well, however, I've no AMD box to test this on. arch/x86/kvm/vmx/vmx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c788aa382611..c8198c8a9b55 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6538,6 +6538,8 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return handle_external_interrupt(vcpu); else if (exit_reason.basic == EXIT_REASON_HLT) return kvm_emulate_halt(vcpu); + else if (exit_reason.basic == EXIT_REASON_CR_ACCESS) + return handle_cr(vcpu); else if (exit_reason.basic == EXIT_REASON_EPT_MISCONFIG) return handle_ept_misconfig(vcpu); #endif From patchwork Wed Jan 18 14:50:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathias Krause X-Patchwork-Id: 45315 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2388070wrn; Wed, 18 Jan 2023 07:07:44 -0800 (PST) X-Google-Smtp-Source: AMrXdXs3nEwkP1pZv7q4ffIOl84xrLIslhqn3ySHfxlAZTbtiH93tcZqy0FM0XQgKLphrlRRPa4B X-Received: by 2002:a17:90a:d384:b0:229:a2:a273 with SMTP id q4-20020a17090ad38400b0022900a2a273mr5189801pju.2.1674054463923; Wed, 18 Jan 2023 07:07:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674054463; cv=none; d=google.com; s=arc-20160816; b=RtuhcyD6k2YObC65ZSdklCP6DYJK6nlGYUJ00+dvxjlqAAgxTMO1Kg1oqiCdCwB7Z2 DMBCTUYbXTbIPiktVOCnEG4cpiaF4myeb/noX6sBIidkQ7JK1eVbeewJLL8Z5NX/XgTt UJ1UDwTmYRXQULNh2QuKb5dsucFWHaFPNbkyx3qUMFG4miNngEdjwS0GwwLKZTm9dnGv bGSLV6n2R2weib4esGkv9gEbtTByeS1HFUEaAAJi9rkXpoXj5vEstx+yv5hAYdm04W2g 0cdHIMgD7qzOm5VRemHXzYz0IJaFUgLFvkyUme9Fg14m6eX4f85tmHHWH/0BaIMWhkrf u9gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1X9cKTEOns6mbPxiAJGzCLzAUmWNA4hlvu/w3TuUZPU=; b=iOOvQ4CGYwyqxhI8rh5hU+Qj+wNb/lfnLN7UW6Fb6kRHAvndrH/EN3JlUE3tUvnQrn AGaDQd9pSShy3+j2t4aBGHRUn/1ItZ9NorKFKRdvVbpKFs+eaYwstyo2+LkG/4O9nqNM cNWFFdJ9+C7CoXmUqwONkoTFeRlncWDsfQsFb9xv7v3zGosYoGSvO1AYp1IgeZ8qE0eX 6pV3PRTmCAIMLirZF+bdZTFnRBg9Pi5c0924IrEf20eB8qmALTyAju0mJiFV6hOnQiRO FpGvfFtewIxag3fxT2wow2DYm9zuBYPwc5itc2S/mUBmaKafgXKdGqdt8Ie9Cp2z9QVE W07Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=XIcJS96o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bm18-20020a656e92000000b004b4ee249a2dsi26519672pgb.479.2023.01.18.07.07.30; Wed, 18 Jan 2023 07:07:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@grsecurity.net header.s=grsec header.b=XIcJS96o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=grsecurity.net Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229982AbjAROyu (ORCPT + 99 others); Wed, 18 Jan 2023 09:54:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230360AbjAROyD (ORCPT ); Wed, 18 Jan 2023 09:54:03 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 648C021A0D for ; Wed, 18 Jan 2023 06:48:54 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id s3so11770786edd.4 for ; Wed, 18 Jan 2023 06:48:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grsecurity.net; s=grsec; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1X9cKTEOns6mbPxiAJGzCLzAUmWNA4hlvu/w3TuUZPU=; b=XIcJS96oA67Hn1nXDnoKQbHuH4zcVXwA50Mjy6FlCanCgJYtLRHrD8yqDVgbDh6z+p 1I8ZymV+22mLjRbNpwIbZt99snoyWphLUFE+CVd+7gbAinmHhJ+RBTbocHZ3XQiud8yP ukeMOSSnCGg3l//ciBp5SHyqmZ5CGHmlemY2A+SP54cAImPLvsUpH2jG3b2A6W4UjvHC lHXz1ZBsjSmZLeuJhMuO/xk2ztPc1Q4k6fLxSHMG4aW518ZIJjm46LqKi1a+SoNzpKvz hes3dNC0ovt5IEsubdBtgmMHyFnKhwLrIcv4/2pYj0uJgLxy8icT5CGlqtxmWUIKPFWb 6RZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1X9cKTEOns6mbPxiAJGzCLzAUmWNA4hlvu/w3TuUZPU=; b=aEoyy5DQ6amJKwGXFNC84QvF/fK5UnU1oxoDcZ/mnSzHjrBqQkcXROcK5muNzoKYiJ z74Ds5wyM3GLk78BAWm0eUG1a9A8tXVe+3R0PLp/YuqbcPizxnnkQhXIG85bhMQSmHD1 +a7sYG+D5PzL/VB4rHuFj2RfGyVIiuZCtocG/WcUZ04PcDdf5uJyh5WlvnGOkIpn0NJK iFVCXrTuUWZ5Kq41vtMvwkXy/4aqNxYXGNOysn9/NJUuljzsKeNDAhka7hLqVrUSnf1L DNDxO8bbMUcqyKT2ONx/J+8eWfhgJjugZvFg9vSqnbA99XkRDsQkyeGePpa0+cKaukBx uKlA== X-Gm-Message-State: AFqh2koWt3Va+/S3waTctmJtcnP8wDibkT+ac/b/yIAIxBj1aPvAFTAV J4KDERZLkg6Ee+o8Bw49oW92P6BmpbdzlmOa X-Received: by 2002:a50:fa8f:0:b0:49e:31d5:6769 with SMTP id w15-20020a50fa8f000000b0049e31d56769mr6624614edr.41.1674053332988; Wed, 18 Jan 2023 06:48:52 -0800 (PST) Received: from nuc.fritz.box (p200300f6af03d2006e0fc0b921f9db5c.dip0.t-ipconnect.de. [2003:f6:af03:d200:6e0f:c0b9:21f9:db5c]) by smtp.gmail.com with ESMTPSA id p11-20020a05640243cb00b0049e19136c22sm3627509edc.95.2023.01.18.06.48.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Jan 2023 06:48:52 -0800 (PST) From: Mathias Krause To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Sean Christopherson , Paolo Bonzini , Mathias Krause Subject: [PATCH v2 3/3] KVM: x86: do not unload MMU roots when only toggling CR0.WP Date: Wed, 18 Jan 2023 15:50:30 +0100 Message-Id: <20230118145030.40845-4-minipli@grsecurity.net> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230118145030.40845-1-minipli@grsecurity.net> References: <20230118145030.40845-1-minipli@grsecurity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755373333424886590?= X-GMAIL-MSGID: =?utf-8?q?1755373333424886590?= There is no need to unload the MMU roots for a direct MMU role when only CR0.WP has changed -- the paging structures are still valid, only the permission bitmap needs to be updated. One heavy user of toggling CR0.WP is grsecurity's KERNEXEC feature to implement kernel W^X. The optimization brings a huge performance gain for this case as the following micro-benchmark running 'ssdd 10 50000' from rt-tests[1] on a grsecurity L1 VM shows (runtime in seconds, lower is better): legacy TDP shadow kvm.git/queue 11.55s 13.91s 75.2s kvm.git/queue+patch 7.32s 7.31s 74.6s For legacy MMU this is ~36% faster, for TTP MMU even ~47% faster. Also TDP and legacy MMU now both have around the same runtime which vanishes the need to disable TDP MMU for grsecurity. Shadow MMU sees no measurable difference and is still slow, as expected. [1] https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git Co-developed-by: Sean Christopherson Signed-off-by: Mathias Krause --- v2: handle the CR0.WP case directly in kvm_post_set_cr0() and only for the direct MMU role -- Sean I re-ran the benchmark and it's even faster than with my patch, as the critical path is now the first one handled and is now inline. Thanks a lot for the suggestion, Sean! arch/x86/kvm/x86.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 508074e47bc0..f09bfc0a3cc1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -902,6 +902,15 @@ EXPORT_SYMBOL_GPL(load_pdptrs); void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) { + /* + * Toggling just CR0.WP doesn't invalidate page tables per se, only the + * permission bits. + */ + if (vcpu->arch.mmu->root_role.direct && (cr0 ^ old_cr0) == X86_CR0_WP) { + kvm_init_mmu(vcpu); + return; + } + if ((cr0 ^ old_cr0) & X86_CR0_PG) { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu);