From patchwork Sat Jul 22 01:23:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 124181 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp575962vqg; Fri, 21 Jul 2023 19:15:46 -0700 (PDT) X-Google-Smtp-Source: APBJJlGPSmiFxbwxNNI1mxxAGD96IklJglY2tpiHKGnQSrZTkjdmnRpQi7Re/5DczsuSMbLF7v0p X-Received: by 2002:a05:6a20:325b:b0:135:2b01:3737 with SMTP id hm27-20020a056a20325b00b001352b013737mr3095088pzc.38.1689992146139; Fri, 21 Jul 2023 19:15:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689992146; cv=none; d=google.com; s=arc-20160816; b=UqD+unxcbzVhv7hfozq0BqF8JCnjJMYT3J3TKqyo0qteI5EugqdmRunPntF9JMgXmm o3FDjpBy7kgXrqsPHntKcgFZpvTKtg//oTQE6Pw3UXxJvk6UixT5YKW5WHQiVBfShfoX 04RLy+ROcoatr6lKduZ765FeecFxi/AGe8XPrGmT8NAFobQLxa5ep1op0YLYapeeR+9N gOTJtUtgwlH9tOtqubNPpjGyV0NYYUX22rpah7AkQSscM+9UQWoiMBpWIRzc8drqu1sN Lpn+VTAztOwd+cKAmZCb7atHEHAzdL9KSLCCamqAIWlABYqfXqmQFpA/6msZT+3A5Fa5 f/WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; fh=+wFND6X2PweTsJwZ99iP2DHiwxyX/zx7C8kEbHFvmqg=; b=JYA3yXMfSFkJ10LGbWBGQyeTjAyoXXzMhkzjPbXuhRAQjyMQ0QqoxvlwrnoA+kk5y/ jZzHjZTmJKbsUXOKGDj4nIeiwZnTJuStiux7KNckOtwEagly24m5k+vW3L0W8382a55Z l09hnQmcRZ141FHBFs6K9M+a4aF1yikf6kMDp6UiGlL6t7xavHtBhRZvCcZYEl0RXLIN dUyw+BzBXcqmXzU9IwBSrQLguzwtFb23BvIUVE4XSQwtXE/iAmxkagBmLKLiqC8Wwm+L DG0zjiOqnPgV1ehMZAoYwbRhaW6di0u9VMKcw57PNqPXf4I4gAYL69y9GEqb6bNfCp48 93Ig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Y5QCdS11; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h70-20020a638349000000b0054fd504e80asi4298857pge.542.2023.07.21.19.15.33; Fri, 21 Jul 2023 19:15:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Y5QCdS11; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231374AbjGVBYB (ORCPT + 99 others); Fri, 21 Jul 2023 21:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230232AbjGVBX4 (ORCPT ); Fri, 21 Jul 2023 21:23:56 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 800753A84 for ; Fri, 21 Jul 2023 18:23:54 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1b895fa8929so13364345ad.0 for ; Fri, 21 Jul 2023 18:23:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689989034; x=1690593834; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; b=Y5QCdS11tnDnLCj/zoS5V7HwmN0Zn3zbcafntmhLGKG/jUIQ9ujkzt5nDgDd4lkun8 yoBt5rwD0PYKLsAefwjU+bqE15ubcmUUslyoUDprloZZdGWmoHXnMq0s28PhQuWA7Vcy VM1hBuXBFSVRZqygAKzVKUNwO4/9DWJxfj8TOJqmILGVyZhNifXEyWhsRRI5UllnoVlt 9z5I2rt/jpWI1ygnQdsbAWXPC9VQvXUOFNgmHlR+53q/Ky22/G6TH9d5NvcwpU6V+5Di 9J3t8p4C0MdbLeuKl1VlqeOxoQrTlJI6OOX0OoxF2Kdc0WEGVKjhQpwo+BZa/pQbmDjl RDXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689989034; x=1690593834; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PY67CjTOwz6KI4T0Lj9P5q66c/RjGwxRUJSEeS4GRlE=; b=TtvOEucW+DLujRBTySFxeEdVb00Polv9fdtuQvyLWYv88H+op3C/TZCNRWTwgUqddT t6tUUKfh07j/Szq5Nvf8VgsGgCpa1PdvGKfKe0sQtfxq72mz9ADZrT2Pw6+TdnQHvHZe 9RKQYNZXlewWjRHxPNH85JSt31MSAgalJS5HE2Q23k4XVHAumem9sONd/TDSRq4CwOoa VIpw3CoNzfMrWljUaGSDpytc7zG+rp3+1oDptL6YiH24nnZIFW8yZgPpKVEp5mKSsu9u QJlE8OA/u21ffHeUsn8v28yhxaA+rOAIHcqOuidaxCCCmqv9aPnZ0UAjCA0JB+hjp89W +IAw== X-Gm-Message-State: ABy/qLbKzdAHQ+V9EYEUCWMbhgktgZTaE0wN1t9A5xcViy8hSpcq6CCT G6aeJtfl90quO8qUMiuWjN7xKBs4ttA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:c410:b0:1ab:18eb:17c8 with SMTP id k16-20020a170902c41000b001ab18eb17c8mr17406plk.2.1689989033958; Fri, 21 Jul 2023 18:23:53 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 21 Jul 2023 18:23:46 -0700 In-Reply-To: <20230722012350.2371049-1-seanjc@google.com> Mime-Version: 1.0 References: <20230722012350.2371049-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230722012350.2371049-2-seanjc@google.com> Subject: [PATCH 1/5] KVM: x86/mmu: Add helper to convert root hpa to shadow page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772085204830065992 X-GMAIL-MSGID: 1772085204830065992 Add a dedicated helper for converting a root hpa to a shadow page in anticipation of using a "dummy" root to handle the scenario where KVM needs to load a valid shadow root (from hardware's perspective), but the guest doesn't have a visible root to shadow. Similar to PAE roots, the dummy root won't have an associated kvm_mmu_page and will need special handling when finding a shadow page given a root. Opportunistically retrieve the root shadow page in kvm_mmu_sync_roots() *after* verifying the root is unsync (the dummy root can never be unsync). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 28 +++++++++++++--------------- arch/x86/kvm/mmu/spte.h | 9 +++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 3 files changed, 23 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ec169f5c7dce..1eadfcde30be 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3574,11 +3574,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, if (!VALID_PAGE(*root_hpa)) return; - /* - * The "root" may be a special root, e.g. a PAE entry, treat it as a - * SPTE to ensure any non-PA bits are dropped. - */ - sp = spte_to_child_sp(*root_hpa); + sp = root_to_sp(*root_hpa); if (WARN_ON(!sp)) return; @@ -3624,7 +3620,7 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, &invalid_list); if (free_active_root) { - if (to_shadow_page(mmu->root.hpa)) { + if (root_to_sp(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) { @@ -3648,6 +3644,7 @@ EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) { unsigned long roots_to_free = 0; + struct kvm_mmu_page *sp; hpa_t root_hpa; int i; @@ -3662,8 +3659,8 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) if (!VALID_PAGE(root_hpa)) continue; - if (!to_shadow_page(root_hpa) || - to_shadow_page(root_hpa)->role.guest_mode) + sp = root_to_sp(root_hpa); + if (!sp || sp->role.guest_mode) roots_to_free |= KVM_MMU_ROOT_PREVIOUS(i); } @@ -4018,7 +4015,7 @@ static bool is_unsync_root(hpa_t root) * requirement isn't satisfied. */ smp_rmb(); - sp = to_shadow_page(root); + sp = root_to_sp(root); /* * PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the @@ -4048,11 +4045,12 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) if (vcpu->arch.mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { hpa_t root = vcpu->arch.mmu->root.hpa; - sp = to_shadow_page(root); if (!is_unsync_root(root)) return; + sp = root_to_sp(root); + write_lock(&vcpu->kvm->mmu_lock); mmu_sync_children(vcpu, sp, true); write_unlock(&vcpu->kvm->mmu_lock); @@ -4382,7 +4380,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, static bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { - struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa); + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); /* Special roots, e.g. pae_root, are not backed by shadow pages. */ if (sp && is_obsolete_sp(vcpu->kvm, sp)) @@ -4564,7 +4562,7 @@ static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd, { return (role.direct || pgd == root->pgd) && VALID_PAGE(root->hpa) && - role.word == to_shadow_page(root->hpa)->role.word; + role.word == root_to_sp(root->hpa)->role.word; } /* @@ -4638,7 +4636,7 @@ static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs * later if necessary. */ - if (VALID_PAGE(mmu->root.hpa) && !to_shadow_page(mmu->root.hpa)) + if (VALID_PAGE(mmu->root.hpa) && !root_to_sp(mmu->root.hpa)) kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); if (VALID_PAGE(mmu->root.hpa)) @@ -4686,7 +4684,7 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) */ if (!new_role.direct) __clear_sp_write_flooding_count( - to_shadow_page(vcpu->arch.mmu->root.hpa)); + root_to_sp(vcpu->arch.mmu->root.hpa)); } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); @@ -5555,7 +5553,7 @@ static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) * (c) KVM doesn't track previous roots for PAE paging, and the guest * is unlikely to zap an in-use PGD. */ - sp = to_shadow_page(root_hpa); + sp = root_to_sp(root_hpa); return !sp || is_obsolete_sp(kvm, sp); } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 1279db2eab44..9f8e8cda89e8 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -236,6 +236,15 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep) return to_shadow_page(__pa(sptep)); } +static inline struct kvm_mmu_page *root_to_sp(hpa_t root) +{ + /* + * The "root" may be a special root, e.g. a PAE entry, treat it as a + * SPTE to ensure any non-PA bits are dropped. + */ + return spte_to_child_sp(root); +} + static inline bool is_mmio_spte(u64 spte) { return (spte & shadow_mmio_mask) == shadow_mmio_value && diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 512163d52194..046ac2589611 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -689,7 +689,7 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter, else #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) + for_each_tdp_pte(_iter, root_to_sp(_mmu->root.hpa), _start, _end) /* * Yield if the MMU lock is contended or this thread needs to return control From patchwork Sat Jul 22 01:23:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 124180 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp574996vqg; Fri, 21 Jul 2023 19:12:51 -0700 (PDT) X-Google-Smtp-Source: APBJJlHJuL9XvZapIjzfF8ehWO+AT7fQxOK+OP59hx5eu5iR91UAVRZVZt8dYS1BWACic9Mji+Iz X-Received: by 2002:a05:6a00:2e1f:b0:668:74e9:8eea with SMTP id fc31-20020a056a002e1f00b0066874e98eeamr1916300pfb.33.1689991971339; Fri, 21 Jul 2023 19:12:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689991971; cv=none; d=google.com; s=arc-20160816; b=m5zOgxJ09XO8JztEQo/DavU13zUtCfuKDV9jXt6Q5GbCnN/7P2LxrcuAvwgep1vdjd rMxl6/31a4OdFn8jS7jTWZZELNjNlcVRC5ShIYmDJfbs4/3jSsU1BqUxhQ7pleTF46xs qd714P2zMRGXHGtd86TvINFFumo1D/GRfyD/kCZfue6opA7BoYf+aHrKfudo1x3evP5N iZskkqTz4mEdyUEvErvdKrQ2V2QPEMqSRVR16g6TfDwJy2EoFS2eLFNo0hOzEBwp6jgk y5C3SXzvsrycVRtRNoAg1RDJILokwiw1JhKNtjVXPsUHb9B+n9X4ttFDFeaWddjVL95V Lxmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; fh=+wFND6X2PweTsJwZ99iP2DHiwxyX/zx7C8kEbHFvmqg=; b=tQeWKlM7+UGGyHwftVuwYMmru294NYKGjtb33iEj+ixXPeGePmV/hfvL0k+08G6iyA vvmWgtPbXTMvrG8FNS/Z9d3h+6B1AtbfuLbqVt7xFe1h4SkFTdE9M7iqvZXuQgP/fBo/ aVOShOXpOFdxD8ciTedvnvARc5HuAhkswqmigB5XZnj7pAfeHabM1AE1O+hA+n6ZYyxP i/VOwYvUC64JokG605BnZ2LVjkWev1s0W9fqupPpxjmMJVXsbRjBEQcT4jjwmTZk9a21 WyA2wmlyux2LeLzy09fk2lCzALink2QgJMuw9z5HejIxy6rMNW4KrMjqziLOJCFHajVd T8zg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EHkFOyw4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k4-20020a056a00134400b006828af9aeb5si4394636pfu.352.2023.07.21.19.12.37; Fri, 21 Jul 2023 19:12:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EHkFOyw4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231325AbjGVBYE (ORCPT + 99 others); Fri, 21 Jul 2023 21:24:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231355AbjGVBX6 (ORCPT ); Fri, 21 Jul 2023 21:23:58 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A86B435B3 for ; Fri, 21 Jul 2023 18:23:56 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-c4f27858e4eso2879764276.1 for ; Fri, 21 Jul 2023 18:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689989036; x=1690593836; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; b=EHkFOyw454NLdiUH/cpx5RfGl2xPzYBb+VKZHOiimOclYEia0svEdqgFzm03OWmWHy dkKkpSnnOCoQ1HqQvKJpRCFvmh2C/8yf+6wTzsFjWnNxqevYbnFIFPdDioBUEswbDid4 ZAYMCNS9F5oQm+0L8WtSeoKYZY4X0tzo+rnD7LqEVf6XAAseeoeLDY4P23x2zDF3pObH ymTPh8gB9tnAWpVmBOGbmiXmRhXGor5RmoVM5F7FJPKv/fadii2NcrE4jSMH/lCb+XsX wPeZtSSD9IkSzVHAvYff1x/x7g4rqLjATIXIgT/39Jv1C8hSLdrzU5o7gxjmwMqdGlyE 2OEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689989036; x=1690593836; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wviPOsS6wxIVOzMjjv3mWVw+feSBMfs9pmoqRvrXCaY=; b=KS/l3nk4GMMv5dBF4tA8tAHTUepdRpJXKnAIQg6jbOKFhubqYfdn4O+HST3k9SRdKK PSPiIg2AFocBjMRrDsTuVeGXPJwyk8tOvUAhy+QfrRDENTcT4RdqB960tbqy7YywsQqA UDpg6jhjEwIH5bsUSfi8G4kwH4vtagzBUzxknQEOA0iS4eH+t/fvXZnklJMp4PVHh05w dBA46TELZqn9KUsTFAkMXqHA8Ry9E8JpZoOPXCH1qjdb8c3OlFT3NJujKJGo91/dKOuv x5RrIrAGnoQajmdIXXurvoVvFwguwjSbJshrRsx9VIC4XJb46LrrjVtawIgfmPfo+ACT 3d7g== X-Gm-Message-State: ABy/qLZ4PZGjH3I5TS/JMXtnPaQG9gkNDlKCwjnz1vNQQb1B4jGZM0lY mnY6D0CZ/1EAVzAs09kkZ8Wu6tvw/0U= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:2fd5:0:b0:cb0:e4d4:f4ff with SMTP id v204-20020a252fd5000000b00cb0e4d4f4ffmr21447ybv.3.1689989035982; Fri, 21 Jul 2023 18:23:55 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 21 Jul 2023 18:23:47 -0700 In-Reply-To: <20230722012350.2371049-1-seanjc@google.com> Mime-Version: 1.0 References: <20230722012350.2371049-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230722012350.2371049-3-seanjc@google.com> Subject: [PATCH 2/5] KVM: x86/mmu: Harden new PGD against roots without shadow pages From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772085021046741548 X-GMAIL-MSGID: 1772085021046741548 Harden kvm_mmu_new_pgd() against NULL pointer dereference bugs by sanity checking that the target root has an associated shadow page prior to dereferencing said shadow page. The code in question is guaranteed to only see roots with shadow pages as fast_pgd_switch() explicitly frees the current root if it doesn't have a shadow page, i.e. is a PAE root, and that in turn prevents valid roots from being cached, but that's all very subtle. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1eadfcde30be..dd8cc46551b2 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4560,9 +4560,19 @@ static void nonpaging_init_context(struct kvm_mmu *context) static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd, union kvm_mmu_page_role role) { - return (role.direct || pgd == root->pgd) && - VALID_PAGE(root->hpa) && - role.word == root_to_sp(root->hpa)->role.word; + struct kvm_mmu_page *sp; + + if (!VALID_PAGE(root->hpa)) + return false; + + if (!role.direct && pgd != root->pgd) + return false; + + sp = root_to_sp(root->hpa); + if (WARN_ON_ONCE(!sp)) + return false; + + return role.word == sp->role.word; } /* @@ -4682,9 +4692,12 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) * If this is a direct root page, it doesn't have a write flooding * count. Otherwise, clear the write flooding count. */ - if (!new_role.direct) - __clear_sp_write_flooding_count( - root_to_sp(vcpu->arch.mmu->root.hpa)); + if (!new_role.direct) { + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); + + if (!WARN_ON_ONCE(!sp)) + __clear_sp_write_flooding_count(sp); + } } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); From patchwork Sat Jul 22 01:23:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 124179 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp562796vqg; Fri, 21 Jul 2023 18:30:47 -0700 (PDT) X-Google-Smtp-Source: APBJJlET7qSbnX2s2XMMSlPnmsmcoyUsihXCiNPobWmKwLOzphUpqdMLRhI3cGFG2CAgUCtPHcOW X-Received: by 2002:a05:6402:1397:b0:51d:9e0c:1396 with SMTP id b23-20020a056402139700b0051d9e0c1396mr2439738edv.35.1689989446701; Fri, 21 Jul 2023 18:30:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689989446; cv=none; d=google.com; s=arc-20160816; b=ytGbtecu/iH3bHBPZ6dc0yWsyCRx5bzVfLJlo9axRY/lcL9PdTu6AGkziC1LgvEZlK j59yF0N5OYb/QNqy+RH+VWu8Y0jfcv2+WeuQVk4jC4QG5YcPCyvJYgxMTgrrZdoFFqni lF83qZuGohl+S174bb8IHRE3LviqvHe2fPNrQEcm6KJIjjynGW4vyqKw0o6fvoa/do3X qvXv3UVEdeCiwQTnJap5aqCdSo8uJGMUbFWor9p4ojMztn3ubYamqGcUmupiuEJUvnmt jUKckVfS4vFtmtEXAyjIozm/tVPQ0Abr6vpj7og335aZ4FEQUCRjl6nWr3Rp2MqQI0Dq D/KQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=K8g8ylIBioHABO4UYnvq+eRAgeeK2kAPLMh05XoHYU4=; fh=+wFND6X2PweTsJwZ99iP2DHiwxyX/zx7C8kEbHFvmqg=; b=tPTMqCwF511eyyOWLrmMokM8QETL0/uAv0REfH7W6Li8VLkF/skmzdstI4DUOHn9Ad +25GX5CklFSGRsXgdPoPQzcPyQiNnxPusDWO7TywHH+ma4tv3QTkJxGtdisUFvlLLIPD SL3LVCBkMjG3ZSeiEikvBj021azS+RaQ7+RA28UAd1uR2x7lihWwWiOluYph3ywW7Oft fU6EeVu0K4RUZYzDBpORtccJq/+SHnLwGU21mrQiZzX6/IWMcKPEXt1yEWJ/F/d4Lq2I XvWt262Xi11PLkEjHjO93kExraBnIeDUfBj5/80VD+IQFlrx4hRxXcqgVZ+SJzAU97J4 Utpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=d9n714xK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r15-20020a056402034f00b0051e4c4ac51esi3010920edw.467.2023.07.21.18.30.23; Fri, 21 Jul 2023 18:30:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=d9n714xK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230073AbjGVBYH (ORCPT + 99 others); Fri, 21 Jul 2023 21:24:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231228AbjGVBX7 (ORCPT ); Fri, 21 Jul 2023 21:23:59 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3840B35AD for ; Fri, 21 Jul 2023 18:23:58 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cab7304dcccso2457603276.3 for ; Fri, 21 Jul 2023 18:23:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689989037; x=1690593837; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=K8g8ylIBioHABO4UYnvq+eRAgeeK2kAPLMh05XoHYU4=; b=d9n714xKJqMwcsAZMI0NHTCtMswGC8ES6vvzsYorFNt0M9xXUeBRyv2a9UkMH2SDBb 1ScznqmzIh4kjUyE1rSGMt/RSAjUOgBLjmGeokgwc60pdNX308+hKXWH1xXe2eJ1E7UE 05Os0QeMBsYL+KYrO61vzAUcL8pSdwb+96Z18F1kECGsA94JjKWTeYHXZFmAJjnJ1zha vpdW1oLjoY/H5htKcV2jS7OfuDX/CAAY8O0JZDKKDltJ//M2/x0m9xTroHtPNR5sFrHa 9EfyrhBnD0vt7qK+t8Y0oFu8yMInPvL09a53JrtWtYnKRNKlsMgf/sCN6zvwRwW95iyM scGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689989037; x=1690593837; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=K8g8ylIBioHABO4UYnvq+eRAgeeK2kAPLMh05XoHYU4=; b=Kqk9lh2PGzgwMwRtswRqNhkDxZ8DqHm2LcE4xo0eDQUKUEZgCFg4aZGYqkiRWH9vUy /ki1HMmjV87ge/7zUuhB754iyluz60bFoMjRgQihl+Qqv2TqQq6CFN62ZNtYH1Jp/S6J tORReMJPInOvhHNpm0F7cGNlYQjVNyUklD+XiN0tLHUFOpL6H0KZIhXQAOFKcW8ha1K2 hi8K5YIpH5gyetH9E0CM2rNZC3LRzJ33OCRbeDHp6CAVgpOQoyxr5uI5R1+AXGpDhSnU s2DRQXVnN1Rwbu+JhAOblxqrzT9USig8TTusg+vdLOHp9OdBzNkNaDP9FG7uBRhOP8fp puWA== X-Gm-Message-State: ABy/qLYPqmtqMIxn2T5ixtZxCWyT4Jh6RVM74dDNIop/LgBIP4ZmEI0V ykquYx6yQlU3FynPfrAkAFGEYAeVX3o= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:245:0:b0:d06:cbd:1f3e with SMTP id 66-20020a250245000000b00d060cbd1f3emr13920ybc.3.1689989037474; Fri, 21 Jul 2023 18:23:57 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 21 Jul 2023 18:23:48 -0700 In-Reply-To: <20230722012350.2371049-1-seanjc@google.com> Mime-Version: 1.0 References: <20230722012350.2371049-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230722012350.2371049-4-seanjc@google.com> Subject: [PATCH 3/5] KVM: x86/mmu: Harden TDP MMU iteration against root w/o shadow page From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772082374281015058 X-GMAIL-MSGID: 1772082374281015058 Explicitly check that tdp_iter_start() is handed a valid shadow page to harden KVM against bugs where Opportunistically stop the TDP MMU iteration instead of continuing on with garbage if the incoming root is bogus. Attempting to walk a garbage root is more likely to caused major problems than doing nothing. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_iter.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index d2eb0d4f8710..bd30ebfb2f2c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -39,13 +39,14 @@ void tdp_iter_restart(struct tdp_iter *iter) void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn) { - int root_level = root->role.level; - - WARN_ON(root_level < 1); - WARN_ON(root_level > PT64_ROOT_MAX_LEVEL); + if (WARN_ON_ONCE(!root || (root->role.level < 1) || + (root->role.level > PT64_ROOT_MAX_LEVEL))) { + iter->valid = false; + return; + } iter->next_last_level_gfn = next_last_level_gfn; - iter->root_level = root_level; + iter->root_level = root->role.level; iter->min_level = min_level; iter->pt_path[iter->root_level - 1] = (tdp_ptep_t)root->spt; iter->as_id = kvm_mmu_page_as_id(root); From patchwork Sat Jul 22 01:23:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 124182 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp576963vqg; Fri, 21 Jul 2023 19:18:49 -0700 (PDT) X-Google-Smtp-Source: APBJJlFYuyhdlb9/ZOks6u7YRxGpSMi+MUmFrk5Nxj/61/0yRLKe/bawWiygMrrse64trHKG9GeB X-Received: by 2002:a05:6a20:734f:b0:137:57fc:4f9d with SMTP id v15-20020a056a20734f00b0013757fc4f9dmr3817579pzc.10.1689992328976; Fri, 21 Jul 2023 19:18:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689992328; cv=none; d=google.com; s=arc-20160816; b=d/+ZaI0CU1Zur9i/AHKtBtbt9dChJ7Z6JiWgxwajVUJkexE3Ihd+dWWsoJ1GRmdvhn 4yEg39AXMYWpTty6S/hwMqi0SVKdvCoZvEG+/HLLOMrO8ijiNPssIuiH4Um0sTcs+a/N +8ueao3b7tDj1GEgOGfaXjoPYl5hTeghXeoQXE6/j142EeoQW12GBYBo03qUw/OGNM0h SDSEJKWMyixafRdZXhhPxThJIWnHlhA4Dk1NubDX+sZlODvgQqPT1BHPLrJlfiU63euq QRycfFjOBYAs6QHHLaQqMntKgAyV5YUArNYA5jF+ujIIWSdF41XVj8M28yP8ofoRylp8 DFfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; fh=+wFND6X2PweTsJwZ99iP2DHiwxyX/zx7C8kEbHFvmqg=; b=mt/JKPi5sslBPETJbXA/QoK1Um+o+bh1d4V/keIXfRBifi7Pxr1L64On73yi+EUFGc AJcmucNwt0RpSfjFzCywuBG1ITMvjLdDdJfN6l4jgAuJnFCi9GtxsIXSZauVjxq0V9N1 ZiAXXO1sbxULNjY0z5yiHaoH5HySSoX3z+v2KLSVMQDA99oVB22O4ZTro/MEvcYnUDq3 GnRYsJpGa9a4MFQ7K74tuxVCBvQv38rsvFD4ZP1zdTi9QBo6rHC5uUL8o7K4NNvwvO/d RPBtdKBhHuls4noPzmuyFxPpUOZqtOt+f3rtTePCStN0KGcuR5S3giRflBE+10lRKnQ0 vjoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=jA6SOzCZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o17-20020a637e51000000b0053490e8df4dsi4290137pgn.104.2023.07.21.19.18.36; Fri, 21 Jul 2023 19:18:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=jA6SOzCZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231251AbjGVBYM (ORCPT + 99 others); Fri, 21 Jul 2023 21:24:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231373AbjGVBYB (ORCPT ); Fri, 21 Jul 2023 21:24:01 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2788530C4 for ; Fri, 21 Jul 2023 18:24:00 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d0737b86c45so381085276.2 for ; Fri, 21 Jul 2023 18:24:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689989039; x=1690593839; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; b=jA6SOzCZ5JV7gpezuMZaywqy77akcRhAVmKuF6oqtso7tzhrR5MVVK0jfJGHg7rfJb FaPJ742hNwflKbWuQ22IMhyWTrBH4002LLDcFa1S87vsGhPwjHQnYseUqYK6KJ7g2RMc XUGtCCZRZWaTSAPtYDlqRaGN7Dx4oGErykWf4zUhs9BZ46uJlFZh46ZN8QN1Up7cpWj0 W9zo7aSE/D5zytti/7d1f1srapwfpt/aGgQUKvId1yAi0P2UPz1QopjoYVUiZDViSRUZ kOqhG40It9Hg+4Hbnogmc7L/jM3hweiBdcpjXezcVzs4ENU0p4nyRFqVvowFX1ru6cuC TEFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689989039; x=1690593839; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=um0JUlmUESKdJimio+8DNUJWds66RsQougy5ghLTK+M=; b=DiG9oXEoLyBKWeROj51hPwxpUTg9/L/Q8xvegc0kHINtpkmZyOQlGqHpcNMNzR9+rM PtDlUeSrnwaYW1njN3DIcGbYWWaQi61DsL4oVwpIRBMy/LR2yhnTJ7EbzBxjioUHvO7I haGR/3RLUe7jKXCwg7dNZyOmz8TW4+3ESjLtisEBko5K4+EIH3KI8oRF03JjmZULhPH/ UahoQdkJs2OG6vl3XbYLHuaRlF0IYsW22M9nuS3MkJw2ZOZPJDiZv9gJF/rRfGDxSB+n 7zEcXLRer1NC/nV5QtrUvW4+M856GtWMuxZFeMU3EliJvfQN/58SpEymGxWLbK+lEJWJ Y7Cg== X-Gm-Message-State: ABy/qLYvEa9Cd8+nSTFfVexJ/nrX8V0uwunhho0yefgFwgP8gbUI06CN tw6w22S22HF/W9Vq8mJk/DHYqU5vvdA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:d6ce:0:b0:c6d:a342:99f1 with SMTP id n197-20020a25d6ce000000b00c6da34299f1mr21928ybg.13.1689989039471; Fri, 21 Jul 2023 18:23:59 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 21 Jul 2023 18:23:49 -0700 In-Reply-To: <20230722012350.2371049-1-seanjc@google.com> Mime-Version: 1.0 References: <20230722012350.2371049-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230722012350.2371049-5-seanjc@google.com> Subject: [PATCH 4/5] KVM: x86/mmu: Disallow guest from using !visible slots for page tables From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772085396428232359 X-GMAIL-MSGID: 1772085396428232359 Explicitly inject a page fault if guest attempts to use a !visible gfn as a page table. kvm_vcpu_gfn_to_hva_prot() will naturally handle the case where there is no memslot, but doesn't catch the scenario where the gfn points at a KVM-internal memslot. Letting the guest backdoor its way into accessing KVM-internal memslots isn't dangerous on its own, e.g. at worst the guest can crash itself, but disallowing the behavior will simplify fixing how KVM handles !visible guest root gfns (immediately synthesizing a triple fault when loading the root is architecturally wrong). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/paging_tmpl.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0662e0278e70..122bfc0124d3 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -351,6 +351,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, ++walker->level; do { + struct kvm_memory_slot *slot; unsigned long host_addr; pt_access = pte_access; @@ -381,7 +382,11 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, if (unlikely(real_gpa == INVALID_GPA)) return 0; - host_addr = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(real_gpa), + slot = kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(real_gpa)); + if (!kvm_is_visible_memslot(slot)) + goto error; + + host_addr = gfn_to_hva_memslot_prot(slot, gpa_to_gfn(real_gpa), &walker->pte_writable[walker->level - 1]); if (unlikely(kvm_is_error_hva(host_addr))) goto error; From patchwork Sat Jul 22 01:23:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 124183 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9010:0:b0:3e4:2afc:c1 with SMTP id l16csp578316vqg; Fri, 21 Jul 2023 19:23:25 -0700 (PDT) X-Google-Smtp-Source: APBJJlELPNT9YwcVbxc8ESLsIlVL8FfzOOFANT/wkO+l5y0YZCZnMvZmna/QwIMiR6V7bbOXgd2K X-Received: by 2002:a05:6358:c19:b0:129:c6d6:ce40 with SMTP id f25-20020a0563580c1900b00129c6d6ce40mr2022845rwj.15.1689992605161; Fri, 21 Jul 2023 19:23:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689992605; cv=none; d=google.com; s=arc-20160816; b=v8vTHWVfLG5glbT7W0zAwld7YqC/Elo4urcYdpNYbPVT+kLaXtR5MAA5v8Pwb1g/GR YyOR6PTlvuhjrbpfM8qYma3abm1c3AW9reyqApFH7yBF9zfBiMb7bWS8crGdHqKbmzTf pU1YFslTB06UVRcrLG4eT1jhLNfq72O6GLWh4CicH+r2/VGuC1gAQDYZHCw4qEE+8nNV CHbtkwvklyFXjFon+Y7TyyG+Qd5B6g1I11Wcz+M6kS7Wni46tZiFoJ7t1cTI4OBKmDl/ +pCuo6FEe66ITf35Lk7P1hyGhY+vHCQOUKP1XpPxcSKQjBsBBq0wz+eXUpmoV6tWh8v2 u8kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=UWMhkNgMbNQj7ZnQNio+vf6z7pD8eN3w4tJfm1Vde8c=; fh=+wFND6X2PweTsJwZ99iP2DHiwxyX/zx7C8kEbHFvmqg=; b=i0JKJX1il4JqbBRAYSY7tlvSPXAmkgTtGxn2iLc4Wy0lNEn2pYpblS2jkyOS5gZfaB GBWL6GUxqDIcqirIHEflvYycGEiOfUpokJL4TEHPWPnz80LsTTV5EbwhN6/GzrewPWzA HSQI38Yjur/QJP7zTJUgpWuwp3O5Kl/vogqCSlgT14AY4am6CTvbPiM3qNWn8GWRY5zk wmH7MvKZ1HsEnNHjEZBC1BMpnyU6SBRFNhdACV0glXUUix9Q9q8gNBRpXX6EEabPoEHm xyh7dR6xuqFNwyW9fyHr6WQAyttzkEmtHHOwqLUGbQlPBvkSAsiFbiqKFXyzjBle7PhQ 4vyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="OsfVm6o/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b5-20020a056a000a8500b006683174a49csi4418206pfl.198.2023.07.21.19.23.12; Fri, 21 Jul 2023 19:23:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="OsfVm6o/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231230AbjGVBYQ (ORCPT + 99 others); Fri, 21 Jul 2023 21:24:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231398AbjGVBYI (ORCPT ); Fri, 21 Jul 2023 21:24:08 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4EEE3A91 for ; Fri, 21 Jul 2023 18:24:01 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1b89e3715acso14554015ad.3 for ; Fri, 21 Jul 2023 18:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689989041; x=1690593841; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=UWMhkNgMbNQj7ZnQNio+vf6z7pD8eN3w4tJfm1Vde8c=; b=OsfVm6o/VAqNlcpylHC5doRDd0YXzKj4TvUCiPKNfITxCM+JS+jLmnU5A6Df/DNryR 4z6I+m0o9c8PgWqkhxYn8nybAEBlHRog0ZZ1biQJSMYPzXQhKX9r7rzBO7uQ8ps6Di9M VuEi5AR6Epj+ctL34uaLw4hNSNbY05UdkH4B5vFJXrlwPl90Iv+Bq9lfQ23XjamqbISt iALQlJe5HsAlKij8WucbkKqJBeOKhdDzexyWPffqu8e5xnpPZX3XWGYdQUhjIhfc0RDo rK4pyuMLZKqx4k1LdYhbiAkgtJdQPetJF0Kn+YVJ9Jh1dIV9vkIvCL76IbpoE7js7hSE GQ2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689989041; x=1690593841; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UWMhkNgMbNQj7ZnQNio+vf6z7pD8eN3w4tJfm1Vde8c=; b=BOkFE3ryB3r7pHS7H5Fs3Yv5i4KJV6r1anD8mcTvOVuxnZ0mytvgwmR+D+nP0GYYco LpnH2Ilzb7tVR+LAQEsHIDMLlKCKNbrb7H4uWB5+pRE897L1kvzXHGAkCDO0imeIlFYc lfnpP5QGsyzzeH0rJvp0ZCGdQEZ9oU218Tz80qHe8DuOkHPKG95wZloaqCsNR8nvgalw rjFbVy0Isneqwo9YW1vOe7K0KsEtcfzgeQXWQMQBUnvb87BSo+Ixy5e2X6KwJ17V0fQD buxusYS02GI+nT9PDhZpgmJD/kZ2bqlG9mTgG/+8+D9iGD+oKWvwBoN49wbuQfHaWfcc fslg== X-Gm-Message-State: ABy/qLbcBhHC5rclbXPh9oLU/PQlxgaxA27Ui3Tc8zTVu4HVZOL8Jifd 6H/x6Z7H7qrE46iiUHHBfU2+dBIP5rY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:1c9:b0:1b9:df8f:888c with SMTP id e9-20020a17090301c900b001b9df8f888cmr13094plh.8.1689989041203; Fri, 21 Jul 2023 18:24:01 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 21 Jul 2023 18:23:50 -0700 In-Reply-To: <20230722012350.2371049-1-seanjc@google.com> Mime-Version: 1.0 References: <20230722012350.2371049-1-seanjc@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230722012350.2371049-6-seanjc@google.com> Subject: [PATCH 5/5] KVM: x86/mmu: Use dummy root, backed by zero page, for !visible guest roots From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Reima Ishii X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772085685919771810 X-GMAIL-MSGID: 1772085685919771810 When attempting to allocate a shadow root for a !visible guest root gfn, e.g. that resides in MMIO space, load a dummy root that is backed by the zero page instead of immediately synthesizing a triple fault shutdown (using the zero page ensures any attempt to translate memory will generate a !PRESENT fault and thus VM-Exit). Unless the vCPU is racing with memslot activity, KVM will inject a page fault due to not finding a visible slot in FNAME(walk_addr_generic), i.e. the end result is mostly same, but critically KVM will inject a fault only *after* KVM runs the vCPU with the bogus root. Waiting to inject a fault until after running the vCPU fixes a bug where KVM would bail from nested VM-Enter if L1 tried to run L2 with TDP enabled and a !visible root. Even though a bad root will *probably* lead to shutdown, (a) it's not guaranteed and (b) the CPU won't read the underlying memory until after VM-Enter succeeds. E.g. if L1 runs L2 with a VMX preemption timer value of '0', then architecturally the preemption timer VM-Exit is guaranteed to occur before the CPU executes any instruction, i.e. before the CPU needs to translate a GPA to a HPA (so long as there are no injected events with higher priority than the preemption timer). If KVM manages to get to FNAME(fetch) with a dummy root, e.g. because userspace created a memslot between installing the dummy root and handling the page fault, simply unload the MMU to allocate a new root and retry the instruction. Reported-by: Reima Ishii Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 41 +++++++++++++++------------------ arch/x86/kvm/mmu/mmu_internal.h | 10 ++++++++ arch/x86/kvm/mmu/paging_tmpl.h | 11 +++++++++ arch/x86/kvm/mmu/spte.h | 3 +++ 4 files changed, 42 insertions(+), 23 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index dd8cc46551b2..20e289e872eb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3620,7 +3620,9 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, &invalid_list); if (free_active_root) { - if (root_to_sp(mmu->root.hpa)) { + if (kvm_mmu_is_dummy_root(mmu->root.hpa)) { + /* Nothing to cleanup for dummy roots. */ + } else if (root_to_sp(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) { @@ -3668,19 +3670,6 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) } EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots); - -static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn) -{ - int ret = 0; - - if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { - kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); - ret = 1; - } - - return ret; -} - static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) { @@ -3818,8 +3807,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; - if (mmu_check_root(vcpu, root_gfn)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { + mmu->root.hpa = kvm_mmu_get_dummy_root(); + return 0; + } /* * On SVM, reading PDPTRs might access guest memory, which might fault @@ -3831,8 +3822,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) if (!(pdptrs[i] & PT_PRESENT_MASK)) continue; - if (mmu_check_root(vcpu, pdptrs[i] >> PAGE_SHIFT)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, pdptrs[i] >> PAGE_SHIFT)) + pdptrs[i] = 0; } } @@ -3999,7 +3990,7 @@ static bool is_unsync_root(hpa_t root) { struct kvm_mmu_page *sp; - if (!VALID_PAGE(root)) + if (!VALID_PAGE(root) || kvm_mmu_is_dummy_root(root)) return false; /* @@ -4405,6 +4396,10 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault { int r; + /* Dummy roots are used only for shadowing bad guest roots. */ + if (WARN_ON_ONCE(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) + return RET_PF_RETRY; + if (page_fault_handle_page_track(vcpu, fault)) return RET_PF_EMULATE; @@ -4642,9 +4637,8 @@ static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, gpa_t new_pgd, union kvm_mmu_page_role new_role) { /* - * For now, limit the caching to 64-bit hosts+VMs in order to avoid - * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs - * later if necessary. + * Limit reuse to 64-bit hosts+VMs without "special" roots in order to + * avoid having to deal with PDPTEs and other complexities. */ if (VALID_PAGE(mmu->root.hpa) && !root_to_sp(mmu->root.hpa)) kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); @@ -5561,7 +5555,8 @@ static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) * positives and free roots that don't strictly need to be freed, but * such false positives are relatively rare: * - * (a) only PAE paging and nested NPT has roots without shadow pages + * (a) only PAE paging and nested NPT have roots without shadow pages + * (or any shadow paging flavor with a dummy root) * (b) remote reloads due to a memslot update obsoletes _all_ roots * (c) KVM doesn't track previous roots for PAE paging, and the guest * is unlikely to zap an in-use PGD. diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index d39af5639ce9..3ca986450393 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -44,6 +44,16 @@ extern bool dbg; #define INVALID_PAE_ROOT 0 #define IS_VALID_PAE_ROOT(x) (!!(x)) +static inline hpa_t kvm_mmu_get_dummy_root(void) +{ + return my_zero_pfn(0) << PAGE_SHIFT; +} + +static inline bool kvm_mmu_is_dummy_root(hpa_t shadow_page) +{ + return is_zero_pfn(shadow_page >> PAGE_SHIFT); +} + typedef u64 __rcu *tdp_ptep_t; struct kvm_mmu_page { diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 122bfc0124d3..e9d4d7b66111 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -646,6 +646,17 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) goto out_gpte_changed; + /* + * Load a new root and retry the faulting instruction in the extremely + * unlikely scenario that the guest root gfn became visible between + * loading a dummy root and handling the resulting page fault, e.g. if + * userspace create a memslot in the interim. + */ + if (unlikely(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) { + kvm_mmu_unload(vcpu); + goto out_gpte_changed; + } + for_each_shadow_entry(vcpu, fault->addr, it) { gfn_t table_gfn; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 9f8e8cda89e8..ac8ad12f9698 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -238,6 +238,9 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep) static inline struct kvm_mmu_page *root_to_sp(hpa_t root) { + if (kvm_mmu_is_dummy_root(root)) + return NULL; + /* * The "root" may be a special root, e.g. a PAE entry, treat it as a * SPTE to ensure any non-PA bits are dropped.