From patchwork Sat Jul 29 00:51:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 127916 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp785913vqg; Fri, 28 Jul 2023 18:18:41 -0700 (PDT) X-Google-Smtp-Source: APBJJlHhVgO/0LaMaju9pe/QVBLAdvDTfMUyYqhLC0yqYfdwaMAKN6B6YYXALPT5L5XAQOGuV/oJ X-Received: by 2002:a17:902:b613:b0:1b5:1787:d23f with SMTP id b19-20020a170902b61300b001b51787d23fmr2299009pls.5.1690593520708; Fri, 28 Jul 2023 18:18:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690593520; cv=none; d=google.com; s=arc-20160816; b=Yh0zZ4WNKHje4soM3In4YrKKWYVt+tSt5sAvJyH4pbBNfGWQSloy+BehCw2m6N67US Nn9nbIvfx4+4We51d7ncAYnD/oWMtzeFNs1Jip+n1E8j7apASakxNZXKADZhLvxeLJ15 6s6NJ1Pqy8raxx7WtavR4/o/InpovBH1bvyV2g7cBAVFV43GAZQOefr2m9W45psu6fxZ iPex32R+SUtvQ0LVke0lIkdnWQjADky3enFQ0GZ/X8jZWjZwghZRqnlvpln9fgFnlANq zZcyP/MnrYmUS7/SCu0T3iB1jQfxQFWo5nqDHDKpRy4I1ascn4a+PvmhjDOEROSDoKoP s7gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; fh=5ox/zlAFQPnl383vszwIhsQYFuHPJFmQzI3udqQ+nUw=; b=mlS8zmeSlPH74DAekz81xrjlMhceGxkJKm9DXzbN4diJ0Yo/ptbe22KtcpnDHiCq5J 7A+BDgGlNmHlvV5V9n7UkWpBUGjgzVaury7nibIzrtS9jYIa/UxOjv4CkWi+jgPxTi0J ySvCUmfYTexPSKluuHRCPgokXAm9pFUDnA/A31R7kQPYYKMi1MybPwUmfucELEum+eG1 ZVG6kQxkHT8pwXp9mvJWd8qDLOf0RyZkIISyI1SP3H/8VZRAkF2v5zEvnR4QWH6njsB5 CoJ3TI10KrvwIGOp9Nf3eMk/jfLekJERwhy46KTq47MOA+SUW2feHXK3nZ9Cth5UdKvj MN7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=oP9v3B7e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g15-20020a17090a7d0f00b0026816382fdbsi3913699pjl.40.2023.07.28.18.18.28; Fri, 28 Jul 2023 18:18:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=oP9v3B7e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237286AbjG2AyV (ORCPT + 99 others); Fri, 28 Jul 2023 20:54:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237126AbjG2Axe (ORCPT ); Fri, 28 Jul 2023 20:53:34 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E310F49ED for ; Fri, 28 Jul 2023 17:53:04 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1bb962ada0dso17237275ad.2 for ; Fri, 28 Jul 2023 17:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591924; x=1691196724; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; b=oP9v3B7eglFdcVwjdKlFHcGD/0WUKiEfzvXaqLmMq67/t5PI5rmXtEhnqO+qWDCBXT 8FzhRhkE38yBETpK7GgAQZckPVoBi5pk10zSmcsgkwe862bauk7/NWdF59MvxE56Qw3G p/2FID9Ss66U/c6ARkkYrTXVscaZp3OJiSo9+6hL0XPXYl9UBGQm4g0vjbH7kSVlxB9c XvgHyvzGenaPkhH5gSxLYI3XGBTXWeldnWkwsPjQRm84nedB0avi3XH3KmVl7+8RjvUi RhVT7ILzlGGno1LYjXEfN7KbcEt7o+KxLGqP3UAoEE2ppbvEIMwxjcUaK5KqOLDH2coV WMiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591924; x=1691196724; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; b=T3dj1MNvJe38q8GjtWxkfTvIsM9MIp3GdxNAcMxUEfnO5tZfuxmkoi+Ga1WBk36jV9 91l/bYW5a8YP3yYG87trfLioeKRm38OIPT2a9VcmlD2EK6HUpUyrx6B/Rw+vKMZ48tJE f3xzTMFwDhK2U/OFeMs5c0vXlh9I4KoSru1A8GEkNgTRfUiO/SijnqpIjF4XGh6paCLj 2+6yUjOU2kGv8jNBZXlZZr2+JBeRl7w0UjpQuUhIc3RpHMGdAL8ibkXJ5G6tRXuUZlLK miV8rvwP+uhIDWqR2qJoHNLclvlMbDUsNwUClQ3Uxx8IQotpFoV0N2APJ9RU1tN8zpdx 9fvg== X-Gm-Message-State: ABy/qLbIfxoO45FLuKbog1w8h8fOxGlxgLd9BSjFvmo5HgIuC6cDhomN SR68eUPYqWqW5yF43mjuhB2M2BZvPKc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e892:b0:1b9:c027:b5b8 with SMTP id w18-20020a170902e89200b001b9c027b5b8mr12917plg.12.1690591924329; Fri, 28 Jul 2023 17:52:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 28 Jul 2023 17:51:56 -0700 In-Reply-To: <20230729005200.1057358-1-seanjc@google.com> Mime-Version: 1.0 References: <20230729005200.1057358-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729005200.1057358-2-seanjc@google.com> Subject: [PATCH v2 1/5] KVM: x86/mmu: Add helper to convert root hpa to shadow page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yu Zhang , Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772715791748957366 X-GMAIL-MSGID: 1772715791748957366 Add a dedicated helper for converting a root hpa to a shadow page in anticipation of using a "dummy" root to handle the scenario where KVM needs to load a valid shadow root (from hardware's perspective), but the guest doesn't have a visible root to shadow. Similar to PAE roots, the dummy root won't have an associated kvm_mmu_page and will need special handling when finding a shadow page given a root. Opportunistically retrieve the root shadow page in kvm_mmu_sync_roots() *after* verifying the root is unsync (the dummy root can never be unsync). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 28 +++++++++++++--------------- arch/x86/kvm/mmu/spte.h | 9 +++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 3 files changed, 23 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ec169f5c7dce..1eadfcde30be 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3574,11 +3574,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, if (!VALID_PAGE(*root_hpa)) return; - /* - * The "root" may be a special root, e.g. a PAE entry, treat it as a - * SPTE to ensure any non-PA bits are dropped. - */ - sp = spte_to_child_sp(*root_hpa); + sp = root_to_sp(*root_hpa); if (WARN_ON(!sp)) return; @@ -3624,7 +3620,7 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, &invalid_list); if (free_active_root) { - if (to_shadow_page(mmu->root.hpa)) { + if (root_to_sp(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) { @@ -3648,6 +3644,7 @@ EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) { unsigned long roots_to_free = 0; + struct kvm_mmu_page *sp; hpa_t root_hpa; int i; @@ -3662,8 +3659,8 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) if (!VALID_PAGE(root_hpa)) continue; - if (!to_shadow_page(root_hpa) || - to_shadow_page(root_hpa)->role.guest_mode) + sp = root_to_sp(root_hpa); + if (!sp || sp->role.guest_mode) roots_to_free |= KVM_MMU_ROOT_PREVIOUS(i); } @@ -4018,7 +4015,7 @@ static bool is_unsync_root(hpa_t root) * requirement isn't satisfied. */ smp_rmb(); - sp = to_shadow_page(root); + sp = root_to_sp(root); /* * PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the @@ -4048,11 +4045,12 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) if (vcpu->arch.mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { hpa_t root = vcpu->arch.mmu->root.hpa; - sp = to_shadow_page(root); if (!is_unsync_root(root)) return; + sp = root_to_sp(root); + write_lock(&vcpu->kvm->mmu_lock); mmu_sync_children(vcpu, sp, true); write_unlock(&vcpu->kvm->mmu_lock); @@ -4382,7 +4380,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, static bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { - struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa); + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); /* Special roots, e.g. pae_root, are not backed by shadow pages. */ if (sp && is_obsolete_sp(vcpu->kvm, sp)) @@ -4564,7 +4562,7 @@ static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd, { return (role.direct || pgd == root->pgd) && VALID_PAGE(root->hpa) && - role.word == to_shadow_page(root->hpa)->role.word; + role.word == root_to_sp(root->hpa)->role.word; } /* @@ -4638,7 +4636,7 @@ static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs * later if necessary. */ - if (VALID_PAGE(mmu->root.hpa) && !to_shadow_page(mmu->root.hpa)) + if (VALID_PAGE(mmu->root.hpa) && !root_to_sp(mmu->root.hpa)) kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); if (VALID_PAGE(mmu->root.hpa)) @@ -4686,7 +4684,7 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) */ if (!new_role.direct) __clear_sp_write_flooding_count( - to_shadow_page(vcpu->arch.mmu->root.hpa)); + root_to_sp(vcpu->arch.mmu->root.hpa)); } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); @@ -5555,7 +5553,7 @@ static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) * (c) KVM doesn't track previous roots for PAE paging, and the guest * is unlikely to zap an in-use PGD. */ - sp = to_shadow_page(root_hpa); + sp = root_to_sp(root_hpa); return !sp || is_obsolete_sp(kvm, sp); } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 1279db2eab44..9f8e8cda89e8 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -236,6 +236,15 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep) return to_shadow_page(__pa(sptep)); } +static inline struct kvm_mmu_page *root_to_sp(hpa_t root) +{ + /* + * The "root" may be a special root, e.g. a PAE entry, treat it as a + * SPTE to ensure any non-PA bits are dropped. + */ + return spte_to_child_sp(root); +} + static inline bool is_mmio_spte(u64 spte) { return (spte & shadow_mmio_mask) == shadow_mmio_value && diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 512163d52194..046ac2589611 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -689,7 +689,7 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter, else #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) + for_each_tdp_pte(_iter, root_to_sp(_mmu->root.hpa), _start, _end) /* * Yield if the MMU lock is contended or this thread needs to return control From patchwork Sat Jul 29 00:51:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 127913 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp785577vqg; Fri, 28 Jul 2023 18:17:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlGquFDjtLsOyNuIIW8AzvDmdVUpNsfGGB6hg5nXK7p0Z5N+LCMEfe1F+CKiL1kXA6cmRQ30 X-Received: by 2002:a05:6a00:2d23:b0:66d:514c:cb33 with SMTP id fa35-20020a056a002d2300b0066d514ccb33mr4232031pfb.6.1690593454304; Fri, 28 Jul 2023 18:17:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690593454; cv=none; d=google.com; s=arc-20160816; b=H/dWtM45xBcj5aqlSmtuBSLtget5+T+5gRZIUg3vX032WR6Umaz0GSYnhf8DI6/w/h tNf7/jHYrHZwEmASwutwwnsB742vQ9k1YGSc2RNGiXhLHkhksglyhReAq1PoQix9yLqV oRKdvGQ9GXcfvJuPNqDu5ExGYKq0gAj0fbXGoHh7tYRM4gE5DBle2cst8oP/bGbVITe3 yQuC3Sm6J6a1jiwyG9NMkEPqriOFPaTHTmdXmVdPM1/VFmWIr3Mgv0NEAQVKXWlOgF56 khavX25KEZREgof2OLwjVYZosqts+kcoFo9EQox+oi0xGf1gXSdyqkxawFQuMUAHKlQD oa0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; fh=5ox/zlAFQPnl383vszwIhsQYFuHPJFmQzI3udqQ+nUw=; b=kj0HDfdkUgspIQLJs2yOyjft7Gb1XFcHMdq9XN9QcdM6ERlOgbxMamiqEiQ1OcIy6R duAmjFbQ7xtzfacbfWWY3HyV81pNiDxuxTx1VV65K8rPbKZRvEGbD8Rnql6COx+SXEII 2cuRx1woP2qOFU/ZUiHgCySJM3rVT57IcvzgBsQMQqJDUihLwsuClDciOh6dfVyEpCYn EkpBrfoZGMDImgHCA3jq+ggCHO4mEJbmCiDveBBhYJI7OkI4NY9s9ETnhsDAoYe6ha5T H4Www3lcbCjSsqBvlwitqHdVwYxjkmAWc3V820pd9H02Nm25hmnIT/Qjv3+rckDyP6Hr kc4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EKB+UIYo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h123-20020a636c81000000b0056406432f69si3060729pgc.132.2023.07.28.18.17.21; Fri, 28 Jul 2023 18:17:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EKB+UIYo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237295AbjG2AyY (ORCPT + 99 others); Fri, 28 Jul 2023 20:54:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237127AbjG2Axe (ORCPT ); Fri, 28 Jul 2023 20:53:34 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C165326B2 for ; Fri, 28 Jul 2023 17:53:05 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1bb982d2572so17350195ad.0 for ; Fri, 28 Jul 2023 17:53:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591926; x=1691196726; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; b=EKB+UIYo3U8JrsxxkoCQHi5oERvy0PlpE5t8T6ECPd1S920laYGLkn5v+VUuZuVRNx TNdT8RcQ0UGfrMuKrJodH/v6mmbDBatABlksyCDAMRh43mT0KlZhqGmXtI3C5fpZpBNQ iw09XJROxq8kshDmgjI9DCbOP1qWMKVyiMTGDY+VVhxgCX25iLB0Pw1e2eevCGa4nI8D xPPeR7CawBIktWLbE6vQ55sQ47znxIYdnLKYz1NKvJ5CiqzCZpN9b1p/ZncxygvkZcDo hoRZVAMyaQb6fuI17sz8+UvDEkP3iIAMtdcF+6fpJ1UJnq4Bowf+GNAYZCbq85JAvBIK jztg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591926; x=1691196726; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; b=Idmmniw73h7yxUNWTwpWSb8fuhcT/gyqz1yjTuM9OsvF8fOtchXlDkA3CBy/xoWfcc I5EtXCZMo20sH21YVYQDrma/AlQUcmx6kE2kwkm2AZsD+8kiH3TC0CL7q/do+EgKn/nP YNLdJV30uGjyXdmdaICmuDUliv0cudmyGbrqerlG2KWfIUzhaza30lHVjP5eGi1WNEE+ gfSFRKDXO6zOq7BId9l0WXkPqLrtRnv+wum1XKbQ1g2kDWmWlOKztqWn4EvJhsZUtxmW kISAwv/f1gYQ6Jqf/2Xa3fGkAsMjccMIq/jN5xcZNkVniAaRoDOevaVoUv2CKHhXHqB3 CaHg== X-Gm-Message-State: ABy/qLaDpzRwzG/WDWqoLSuuILyW5jqh1qphAU4vQDUIChauqxtrcOxr wRyQ1PjUiWWO/EX5gbk8++hMkdM2uJs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:f990:b0:1bb:1ffd:5cc8 with SMTP id ky16-20020a170902f99000b001bb1ffd5cc8mr12038plb.11.1690591925918; Fri, 28 Jul 2023 17:52:05 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 28 Jul 2023 17:51:57 -0700 In-Reply-To: <20230729005200.1057358-1-seanjc@google.com> Mime-Version: 1.0 References: <20230729005200.1057358-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729005200.1057358-3-seanjc@google.com> Subject: [PATCH v2 2/5] KVM: x86/mmu: Harden new PGD against roots without shadow pages From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yu Zhang , Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772715721961846005 X-GMAIL-MSGID: 1772715721961846005 Harden kvm_mmu_new_pgd() against NULL pointer dereference bugs by sanity checking that the target root has an associated shadow page prior to dereferencing said shadow page. The code in question is guaranteed to only see roots with shadow pages as fast_pgd_switch() explicitly frees the current root if it doesn't have a shadow page, i.e. is a PAE root, and that in turn prevents valid roots from being cached, but that's all very subtle. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1eadfcde30be..dd8cc46551b2 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4560,9 +4560,19 @@ static void nonpaging_init_context(struct kvm_mmu *context) static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd, union kvm_mmu_page_role role) { - return (role.direct || pgd == root->pgd) && - VALID_PAGE(root->hpa) && - role.word == root_to_sp(root->hpa)->role.word; + struct kvm_mmu_page *sp; + + if (!VALID_PAGE(root->hpa)) + return false; + + if (!role.direct && pgd != root->pgd) + return false; + + sp = root_to_sp(root->hpa); + if (WARN_ON_ONCE(!sp)) + return false; + + return role.word == sp->role.word; } /* @@ -4682,9 +4692,12 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) * If this is a direct root page, it doesn't have a write flooding * count. Otherwise, clear the write flooding count. */ - if (!new_role.direct) - __clear_sp_write_flooding_count( - root_to_sp(vcpu->arch.mmu->root.hpa)); + if (!new_role.direct) { + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); + + if (!WARN_ON_ONCE(!sp)) + __clear_sp_write_flooding_count(sp); + } } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); From patchwork Sat Jul 29 00:51:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 127986 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp817363vqg; Fri, 28 Jul 2023 20:05:14 -0700 (PDT) X-Google-Smtp-Source: APBJJlGrVJ5YGpoqA8p0u/f9zyKZ7lJAjaeiEBHo8fPi8VRNjNcnD3qt6mQkoyiryLpNF6Jjh6ZS X-Received: by 2002:a17:903:11c7:b0:1b8:5aba:509d with SMTP id q7-20020a17090311c700b001b85aba509dmr3230604plh.21.1690599914378; Fri, 28 Jul 2023 20:05:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690599914; cv=none; d=google.com; s=arc-20160816; b=W6Rl9WFscdaIwVUGIW+XZ+5ftS5jKcrbuo/UUvwCADtpChjgg/SHGHSPif83hQlIoH UdyiS4+quouWERVihEq/8zLgwZ9/o1iIxwHJOrAAylOSv/r4ekr+ik+ZYC9aG2rRKaKW s0o7tSWD3S9cfd8CZmR0wErNrIiprgdjYgSEESv4/h7vq1sbzvCX/JylifjcLpTREPPf Flo33yGV+oA2GyANl2yGYGZN8eBEs81WD5ck+4u6f24Rv1oTIOM3K3iOPm6s0s8vf6+y i0zJ20rzco4+X9b2tIAKHuUdxmAKRe99yx7sO43VgBhkCk//6Pys+Ygw1Ygq9aGl8xc9 CYLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=c2hzKspWULGao7+7tlz/GXIKMGwiNk1PKHvUMvBGqxg=; fh=5ox/zlAFQPnl383vszwIhsQYFuHPJFmQzI3udqQ+nUw=; b=hSTRlyBZTNez//iExPCxiNinAdqWOuo90p2QHEmJsNfgez4JrhTyC0YqYcCk1xxufG FBeruolNZ5k8Kxde+unFxPhh1NPBRt8PWdjCy7jLbdJDPkcD40X8hjTDDuG4VFg+c53J 6EQAIWN6aHDMtSUHDz52+DpAoBP/rS8fN5wmXiAeX58hXZNyNVhEzGlmL1hz2n9mdj8J QxN5+9bKVFntLqH8LBwCBdbHnhn8NLQtDfUxVYix9HazANXCCZa+vfQJSuXrz/Cof+Fo 5jbStEMFPK2p36rjmCYnonjQbP71aPFG7oVB3/mZSiLniqAz4Hito12ejdTm/wvgS+A1 IJbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=MnK7365h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id le16-20020a170902fb1000b001b80b061e8asi2672022plb.341.2023.07.28.20.05.00; Fri, 28 Jul 2023 20:05:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=MnK7365h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237307AbjG2Ay0 (ORCPT + 99 others); Fri, 28 Jul 2023 20:54:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237143AbjG2Axe (ORCPT ); Fri, 28 Jul 2023 20:53:34 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6762649F1 for ; Fri, 28 Jul 2023 17:53:07 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1bba9a0da10so18263395ad.2 for ; Fri, 28 Jul 2023 17:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591928; x=1691196728; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=c2hzKspWULGao7+7tlz/GXIKMGwiNk1PKHvUMvBGqxg=; b=MnK7365hqYqDmIToHDHmfuIORMIpqw1AhEznKuxs+/8El6VtqwC9A5SoRAWJoYQ7y3 2j4AMTwoOFp67ycnoR7TGuJFzJ1J2rNVTU18ch/3mPlf7/aI5Nrr373YvoVnVqj3xQjN xLmMI0urjdys7Tp+u97p9GcuViNxd4ObxQG1Rd4a4C7R1jjkHzUds/GQuNYS4A1APfwG t+SIsYwzsJQ5gMgqOK4U+uPBJhenN3YxYLv7ZRcC1IlDtr7mSkgDmyqJmYU65bPHr1pk k3F4LRVQiIz/OvA0gxExBoecq+72mtcMLmTF1MD/bUjclRuDKcLO0e6akXWL6mK2XQwq ZFZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591928; x=1691196728; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=c2hzKspWULGao7+7tlz/GXIKMGwiNk1PKHvUMvBGqxg=; b=bV/eBjx3F3/vFSqe4Ue2liyra6Ub2eji+ORFti/gwPb3LFQShKkprcWAJRFpHX5uZx nt6X/X9uONtwLqdmlWXJjtzPYWcshRVPHbedJfWLIGpDtpXeJAcFfJfjjKC/qdqkgVdh zCFTqz7C0OFTsS9UZqaJ8ydvO+dsYnuq4QzxrulwIg1EIoo3XxXv++gKMlxRHYo3CNl5 auiVo+VbyEjBHsl6Shk1Ssg5Lit6NgjPbKmIalD9xEIFItza0VHupKeUrobXODGtcDMk T1Pc82ks+kwQkLWJsuXZFeysxUNtZXizMKinZt56gyX7HI7WZErURs9n6cxuv7NXs78m 10oA== X-Gm-Message-State: ABy/qLaUk2ORbcqn1L13ql60JRnsSjm5MwSEvM/VVG5QX+eeQ0lNKFEc auHBuvTzibfzOtaga0T/yCMM8V+WePo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e804:b0:1b8:9468:c04 with SMTP id u4-20020a170902e80400b001b894680c04mr12249plg.5.1690591927773; Fri, 28 Jul 2023 17:52:07 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 28 Jul 2023 17:51:58 -0700 In-Reply-To: <20230729005200.1057358-1-seanjc@google.com> Mime-Version: 1.0 References: <20230729005200.1057358-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729005200.1057358-4-seanjc@google.com> Subject: [PATCH v2 3/5] KVM: x86/mmu: Harden TDP MMU iteration against root w/o shadow page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yu Zhang , Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772722495710048710 X-GMAIL-MSGID: 1772722495710048710 Explicitly check that tdp_iter_start() is handed a valid shadow page to harden KVM against bugs, e.g. if KVM calls into the TDP MMU with an invalid or shadow MMU root (which would be a fatal KVM bug), the shadow page pointer will be NULL. Opportunistically stop the TDP MMU iteration instead of continuing on with garbage if the incoming root is bogus. Attempting to walk a garbage root is more likely to caused major problems than doing nothing. Cc: Yu Zhang Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_iter.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index d2eb0d4f8710..bd30ebfb2f2c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -39,13 +39,14 @@ void tdp_iter_restart(struct tdp_iter *iter) void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn) { - int root_level = root->role.level; - - WARN_ON(root_level < 1); - WARN_ON(root_level > PT64_ROOT_MAX_LEVEL); + if (WARN_ON_ONCE(!root || (root->role.level < 1) || + (root->role.level > PT64_ROOT_MAX_LEVEL))) { + iter->valid = false; + return; + } iter->next_last_level_gfn = next_last_level_gfn; - iter->root_level = root_level; + iter->root_level = root->role.level; iter->min_level = min_level; iter->pt_path[iter->root_level - 1] = (tdp_ptep_t)root->spt; iter->as_id = kvm_mmu_page_as_id(root); From patchwork Sat Jul 29 00:51:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 127930 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp791296vqg; Fri, 28 Jul 2023 18:36:39 -0700 (PDT) X-Google-Smtp-Source: APBJJlHymuT7aQvHvP741A/61+XMz3a7d0sJoQgw4ub6JaHPikrSB1IaIkUtCb+cLn7D9U+pEGv1 X-Received: by 2002:a17:907:75da:b0:997:beca:f9db with SMTP id jl26-20020a17090775da00b00997becaf9dbmr784213ejc.54.1690594599444; Fri, 28 Jul 2023 18:36:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690594599; cv=none; d=google.com; s=arc-20160816; b=0Mm7fDX+BjoHHPdgomAST8OUYqfvAoBCb5bOlfNdioqSXDHHtY53LKTS2FfgRIR9IF z2Owoh1L/VqdRMvtWI+lH0lAZ2VJG+ZyHE/g5N2P7TZWOeXbBBzT5xiIDm/7Yay9M1Y3 PXRUfUCLKKUbkFiYQ7vqx8JVvda2l3yV7DsfZOX+c8Za1G9KPEE9dVv1o73sn17fFQ2X OV7BVuei6Ex7TWPummUxN4uPuEe3UVDdkaOGlDBnbjRoDCKa7XlPZ8M9TXv6FDeSgIwX aUmay1s4NobNmBUiBjJyMkcutN8iGolPErceDqm8njNn5NosodPXPgYmtJe2krqOZ3rW x5ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; fh=5ox/zlAFQPnl383vszwIhsQYFuHPJFmQzI3udqQ+nUw=; b=ZpzxYBxjK+lyoF4GSdLubZ27LwEW3iqP4zrpeO5vh8cXpRzV7y7qlgcFeUyrGQjbd6 i9P8y4s0Xwey3N7SEjruBnm3jsJKVWcTKVg9u2cqTm8KB3wt+yAc+898bSPJnTmCxUc2 Ar0QEHC1+YdHErpo0VQyLAOUAJRaeJnP4WcbH46g/SEHg/WOQMbxWbU9pPGnaOyP/EHg a0WTuRGJZ7UzQEQS8Ltgfdea6FNs9uzpBdROdJg/VJFesrQjVT7MnRgzRWebtkpX2BK4 dwg1RCHrydV5rqnEWhuGR3GmyXxk5ij89GMZfK58whZko/bDkOiUY8wx373HYUSHYE7D O8Pg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=r48oWTbm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a26-20020a17090640da00b0098895f19707si3646655ejk.171.2023.07.28.18.36.13; Fri, 28 Jul 2023 18:36:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=r48oWTbm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236942AbjG2Ayb (ORCPT + 99 others); Fri, 28 Jul 2023 20:54:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237145AbjG2Axe (ORCPT ); Fri, 28 Jul 2023 20:53:34 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8770D49F5 for ; Fri, 28 Jul 2023 17:53:08 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1bb982d2572so17350305ad.0 for ; Fri, 28 Jul 2023 17:53:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591929; x=1691196729; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; b=r48oWTbmRTUWH3Z3twYcBmYDoGqgbAjjPqBmvui4nkkBegzbN6TfRQJYzbz5xJX0KK X9ZgKU6wFEwfylQNRownUXs6PJOB5Zz9LybqmE4Jq/UoiIhkKrzX5Dsf7A70vL1+afVG C4ifTpOrE/gja7r+0dRTQldVXd2mxVS+I4xcVPQgeGTjtGx0oky4JpuQpiVUt7s3EBid bPu5gqi8jKk1wp5vFmVU7LMTdXVrtPJ9Zbugk1qr8arQBbFqIvrye52e7QP0UQRIr+Rr NypmxslReFidTdyPqTe6dxxdhbgiOBtb20No55FnjNInlGL/tdCxerGILm33QsLb/W7A OUnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591929; x=1691196729; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; b=Y7yKZ6C8EHTlRucTEmCwykvgRKPrWe+ZVKmulNQQu2fANVW1r33dy+7IEE59sk2aor FEjEMx3udQQlP+VHNrHlCtCLCQHHZkeoU492w31EBID+62bJM1UMnoTOp2d9gZsUUULV iRZzONmKJUuC7hR5bRv9pJ2JOwn0lV8cqJvaM+IvuNtTI3jAbKFdP8dJB0PCNAWwnhf+ DvJtiRfPgAat6TMPAylrKc9NCmJaLcuPY07+nHyAiNoxhgGNL0G8pzBBrZ/aP0zhkXWv i+kmnxwZzWdHZr83OJ828YvktAl2eyheBCVYgY4I/1HhY2Gfh5vMIJ84r/QiuIBE5EAT i45g== X-Gm-Message-State: ABy/qLbQ6mhpSEgusxtle6ZZ4PNebiKB+U6NSot4OhxnaeLq1v6mKQCm sj/grqO7/daFO/6ogIUmILnxxvq4o/g= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:c951:b0:1ae:6895:cb96 with SMTP id i17-20020a170902c95100b001ae6895cb96mr13176pla.5.1690591929667; Fri, 28 Jul 2023 17:52:09 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 28 Jul 2023 17:51:59 -0700 In-Reply-To: <20230729005200.1057358-1-seanjc@google.com> Mime-Version: 1.0 References: <20230729005200.1057358-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729005200.1057358-5-seanjc@google.com> Subject: [PATCH v2 4/5] KVM: x86/mmu: Disallow guest from using !visible slots for page tables From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yu Zhang , Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772716922936811814 X-GMAIL-MSGID: 1772716922936811814 Explicitly inject a page fault if guest attempts to use a !visible gfn as a page table. kvm_vcpu_gfn_to_hva_prot() will naturally handle the case where there is no memslot, but doesn't catch the scenario where the gfn points at a KVM-internal memslot. Letting the guest backdoor its way into accessing KVM-internal memslots isn't dangerous on its own, e.g. at worst the guest can crash itself, but disallowing the behavior will simplify fixing how KVM handles !visible guest root gfns (immediately synthesizing a triple fault when loading the root is architecturally wrong). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/paging_tmpl.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0662e0278e70..122bfc0124d3 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -351,6 +351,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, ++walker->level; do { + struct kvm_memory_slot *slot; unsigned long host_addr; pt_access = pte_access; @@ -381,7 +382,11 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, if (unlikely(real_gpa == INVALID_GPA)) return 0; - host_addr = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(real_gpa), + slot = kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(real_gpa)); + if (!kvm_is_visible_memslot(slot)) + goto error; + + host_addr = gfn_to_hva_memslot_prot(slot, gpa_to_gfn(real_gpa), &walker->pte_writable[walker->level - 1]); if (unlikely(kvm_is_error_hva(host_addr))) goto error; From patchwork Sat Jul 29 00:52:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 127927 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp787972vqg; Fri, 28 Jul 2023 18:24:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlFv9MmAs5MH9n/KkOFPZd8vRRyzoSdcZgMCiHN5eFElLjdp9RfEdPA0W8dkIrJZWMFrXuUy X-Received: by 2002:a05:6a20:2447:b0:134:1c06:19bb with SMTP id t7-20020a056a20244700b001341c0619bbmr4356752pzc.22.1690593894546; Fri, 28 Jul 2023 18:24:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690593894; cv=none; d=google.com; s=arc-20160816; b=lw+XsHb8Ue0IRtyJ8XzJL917odx53ok37eQwi8lTO7/xEPWoMcH0bnQuOWdQvQLDx6 fQVYMU/4K4p01Jy55zg3VU8D1stgm/L/IKIkdrLSQzyUL5t/p8q6n31pXLbNWKUiylcL CWknPUyKDpvQ/zIVSScJHexvTcwa3xi7R49TfoLFS3s12k7IU38z3zpiODlRXmpu2Zrm 6W6Ntg30vTxTmU13PpG6q1Sc48YtOZgQpnm2KMuvKe3WgPvMeh6ILbcfsxzPiNRqQ38b JQ6EVZf80pWqnVu3KYwuAMU5F6C8+rXKevoejck+zidsjzjUAPm87Rn6OT6msdc6kpUc ecgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=scefrr3SIufw6sNZG0VjXNno2RViLSFvpYi41J7nSSE=; fh=5ox/zlAFQPnl383vszwIhsQYFuHPJFmQzI3udqQ+nUw=; b=cgpw80aImZ9W5YCV8hCF0FQalUZrUePvJSRd0zPI280qTw1LlLH5SHORsjlqHiXH6Z uAN33HV9COh2vShRnbQglf8zcR+6mzyMbsi3/f5HEbC7WV7GtGX7DGXKEonnGGSVvzvn W1166+ZozOBvr5yAjWau59qhrH5oMKtOi0yVSZdO0WLvp4QXLiklxYvnZmKBSfDsrC5H gxw9Ia2tsi85MBaCGS2UYqv2JqpDlyLBzxQI4RGnf+pyOnc2+sZsekJdW2D3O5MKAGDj l3AXvVuZgCvenhRC687dixP3EF1BVxxZhkZSp8sX3xiZdATrQZrjMDVKBIfPq1jpfXFt ZMLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=x5RzGi3s; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p12-20020a63fe0c000000b00553b9b18882si3723876pgh.574.2023.07.28.18.24.40; Fri, 28 Jul 2023 18:24:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=x5RzGi3s; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229889AbjG2Ayl (ORCPT + 99 others); Fri, 28 Jul 2023 20:54:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235282AbjG2AyB (ORCPT ); Fri, 28 Jul 2023 20:54:01 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A5364ED8 for ; Fri, 28 Jul 2023 17:53:12 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1bb98659f3cso18258615ad.3 for ; Fri, 28 Jul 2023 17:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591931; x=1691196731; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=scefrr3SIufw6sNZG0VjXNno2RViLSFvpYi41J7nSSE=; b=x5RzGi3sYi9BiTkBJxkYuke/eRiSqxxn6e8LLPx+jc0R1g87xw23xc1uTMjhHPOhUj 51SR01z5k7qEZgwqE7HuF/7KW+s8T8fdPQsjvQ7vgqnc7jblmNVQ5XVZSEeB1s2dZOtE mSuV919P1mcYTdRWKawEg7mkLNFRbUIAMMlAJjIiyEC6+Y4fwbaCJRCSqUD939LeJL3+ K06Khr/NfX9YrBf/pgpC1WLGKhooduG1EbwFzsoxeHbhUnt1T/Xg+MjLnVhYgGr0nOQ5 tY3nn+OuNMHlBrmhxoQ34+vkxNMy1DSh8xo1E1iaPsNl2Umkqub1UjtNMBdpuirs9YRA EcRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591931; x=1691196731; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=scefrr3SIufw6sNZG0VjXNno2RViLSFvpYi41J7nSSE=; b=TTZ7WS8BeLKC93xXAI0jjiQVeKXLjSMEzRxXDbGeE6TBxIGm9vknOaaeWcAgSkiB7n 6LtG+lMi3/ryb5rfSS+4i4+Dxm6ROVumnHnOGNHPFvpdATeonGQCSgIwJVAlqbqc/Ai7 l2GsrY1QnxGSbo63cRBQuOloXM4lmWxHqWilDaX9d/Hpxc9XEM4Ovs7s2XioUWECv510 rdKqh7y7fSBVEpD+xrTKID2cCDxbzsJnvHmJ6pVLEA0PAgINY2EK7SII3SUdu+CMZGC8 kvuoHOKON6VoYxeZJOAGI7gCUaXVvO0wk5uASbofnXq6M1/LS8MB2AsRdz4s3jz7MHIS eEaw== X-Gm-Message-State: ABy/qLZFE9QH+uXojiTdO0GLFl0UJVC2pZKhVivy45FpRN5D7xDMEQFi Olx9oWZVfmC+bXx26yu6DAQh9va50SQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e748:b0:1b5:61d3:dae5 with SMTP id p8-20020a170902e74800b001b561d3dae5mr14358plf.1.1690591931517; Fri, 28 Jul 2023 17:52:11 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 28 Jul 2023 17:52:00 -0700 In-Reply-To: <20230729005200.1057358-1-seanjc@google.com> Mime-Version: 1.0 References: <20230729005200.1057358-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729005200.1057358-6-seanjc@google.com> Subject: [PATCH v2 5/5] KVM: x86/mmu: Use dummy root, backed by zero page, for !visible guest roots From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yu Zhang , Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772716183656948417 X-GMAIL-MSGID: 1772716183656948417 When attempting to allocate a shadow root for a !visible guest root gfn, e.g. that resides in MMIO space, load a dummy root that is backed by the zero page instead of immediately synthesizing a triple fault shutdown (using the zero page ensures any attempt to translate memory will generate a !PRESENT fault and thus VM-Exit). Unless the vCPU is racing with memslot activity, KVM will inject a page fault due to not finding a visible slot in FNAME(walk_addr_generic), i.e. the end result is mostly same, but critically KVM will inject a fault only *after* KVM runs the vCPU with the bogus root. Waiting to inject a fault until after running the vCPU fixes a bug where KVM would bail from nested VM-Enter if L1 tried to run L2 with TDP enabled and a !visible root. Even though a bad root will *probably* lead to shutdown, (a) it's not guaranteed and (b) the CPU won't read the underlying memory until after VM-Enter succeeds. E.g. if L1 runs L2 with a VMX preemption timer value of '0', then architecturally the preemption timer VM-Exit is guaranteed to occur before the CPU executes any instruction, i.e. before the CPU needs to translate a GPA to a HPA (so long as there are no injected events with higher priority than the preemption timer). If KVM manages to get to FNAME(fetch) with a dummy root, e.g. because userspace created a memslot between installing the dummy root and handling the page fault, simply unload the MMU to allocate a new root and retry the instruction. Use KVM_REQ_MMU_FREE_OBSOLETE_ROOTS to drop the root, as invoking kvm_mmu_free_roots() while holding mmu_lock would deadlock, and conceptually the dummy root has indeeed become obsolete. The only difference versus existing usage of KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is that the root has become obsolete due to memslot *creation*, not memslot deletion or movement. Reported-by: Reima Ishii Cc Yu Zhang Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 47 ++++++++++++++++----------------- arch/x86/kvm/mmu/mmu_internal.h | 10 +++++++ arch/x86/kvm/mmu/paging_tmpl.h | 11 ++++++++ arch/x86/kvm/mmu/spte.h | 3 +++ 4 files changed, 47 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index dd8cc46551b2..565988c81b57 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3620,7 +3620,9 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, &invalid_list); if (free_active_root) { - if (root_to_sp(mmu->root.hpa)) { + if (kvm_mmu_is_dummy_root(mmu->root.hpa)) { + /* Nothing to cleanup for dummy roots. */ + } else if (root_to_sp(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) { @@ -3668,19 +3670,6 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) } EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots); - -static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn) -{ - int ret = 0; - - if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { - kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); - ret = 1; - } - - return ret; -} - static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) { @@ -3818,8 +3807,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; - if (mmu_check_root(vcpu, root_gfn)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { + mmu->root.hpa = kvm_mmu_get_dummy_root(); + return 0; + } /* * On SVM, reading PDPTRs might access guest memory, which might fault @@ -3831,8 +3822,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) if (!(pdptrs[i] & PT_PRESENT_MASK)) continue; - if (mmu_check_root(vcpu, pdptrs[i] >> PAGE_SHIFT)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, pdptrs[i] >> PAGE_SHIFT)) + pdptrs[i] = 0; } } @@ -3999,7 +3990,7 @@ static bool is_unsync_root(hpa_t root) { struct kvm_mmu_page *sp; - if (!VALID_PAGE(root)) + if (!VALID_PAGE(root) || kvm_mmu_is_dummy_root(root)) return false; /* @@ -4405,6 +4396,10 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault { int r; + /* Dummy roots are used only for shadowing bad guest roots. */ + if (WARN_ON_ONCE(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) + return RET_PF_RETRY; + if (page_fault_handle_page_track(vcpu, fault)) return RET_PF_EMULATE; @@ -4642,9 +4637,8 @@ static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, gpa_t new_pgd, union kvm_mmu_page_role new_role) { /* - * For now, limit the caching to 64-bit hosts+VMs in order to avoid - * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs - * later if necessary. + * Limit reuse to 64-bit hosts+VMs without "special" roots in order to + * avoid having to deal with PDPTEs and other complexities. */ if (VALID_PAGE(mmu->root.hpa) && !root_to_sp(mmu->root.hpa)) kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); @@ -5557,14 +5551,19 @@ static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) /* * When freeing obsolete roots, treat roots as obsolete if they don't - * have an associated shadow page. This does mean KVM will get false + * have an associated shadow page, as it's impossible to determine if + * such roots are fresh or stale. This does mean KVM will get false * positives and free roots that don't strictly need to be freed, but * such false positives are relatively rare: * - * (a) only PAE paging and nested NPT has roots without shadow pages + * (a) only PAE paging and nested NPT have roots without shadow pages + * (or any shadow paging flavor with a dummy root, see note below) * (b) remote reloads due to a memslot update obsoletes _all_ roots * (c) KVM doesn't track previous roots for PAE paging, and the guest * is unlikely to zap an in-use PGD. + * + * Note! Dummy roots are unique in that they are obsoleted by memslot + * _creation_! See also FNAME(fetch). */ sp = root_to_sp(root_hpa); return !sp || is_obsolete_sp(kvm, sp); diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index d39af5639ce9..3ca986450393 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -44,6 +44,16 @@ extern bool dbg; #define INVALID_PAE_ROOT 0 #define IS_VALID_PAE_ROOT(x) (!!(x)) +static inline hpa_t kvm_mmu_get_dummy_root(void) +{ + return my_zero_pfn(0) << PAGE_SHIFT; +} + +static inline bool kvm_mmu_is_dummy_root(hpa_t shadow_page) +{ + return is_zero_pfn(shadow_page >> PAGE_SHIFT); +} + typedef u64 __rcu *tdp_ptep_t; struct kvm_mmu_page { diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 122bfc0124d3..caf7623e5f91 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -646,6 +646,17 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) goto out_gpte_changed; + /* + * Load a new root and retry the faulting instruction in the extremely + * unlikely scenario that the guest root gfn became visible between + * loading a dummy root and handling the resulting page fault, e.g. if + * userspace create a memslot in the interim. + */ + if (unlikely(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) { + kvm_make_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu); + goto out_gpte_changed; + } + for_each_shadow_entry(vcpu, fault->addr, it) { gfn_t table_gfn; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 9f8e8cda89e8..ac8ad12f9698 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -238,6 +238,9 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep) static inline struct kvm_mmu_page *root_to_sp(hpa_t root) { + if (kvm_mmu_is_dummy_root(root)) + return NULL; + /* * The "root" may be a special root, e.g. a PAE entry, treat it as a * SPTE to ensure any non-PA bits are dropped.