From patchwork Fri Apr 7 20:19:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp514035vqo; Fri, 7 Apr 2023 13:32:25 -0700 (PDT) X-Google-Smtp-Source: AKy350ZaqbqDPPzv+yjtZ3jGDuQ/W+HiBKi8tHgFtcyg0GMttYbz2jVVwO3KCR3nh3jenrk8xWjY X-Received: by 2002:aa7:d98e:0:b0:4fc:8642:ce56 with SMTP id u14-20020aa7d98e000000b004fc8642ce56mr2208830eds.25.1680899544863; Fri, 07 Apr 2023 13:32:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899544; cv=none; d=google.com; s=arc-20160816; b=ODk+F8s6jUvXJPkVZ1kjMxMSjrpO04J0aonpp/U8Q99/YzQaGtlXhpVC5PHYSCaEY6 9owA+K12fkFu4IVz+6LOaaaNsorwq2Llqu8lrMM4AU1mjP0uiO3hOAj3C6KHdld+3RRR ySBnGTuoV+1YZ1ZF6EjwzJT7N03gdqxK0HitdgyXby0xAd//KKHSzUvHVWLqjSkwbrPR FHjFXfvOgM9lTX2DK0Oip4TBXFd79RhgocEjiENEZ6McuU4Cquo0Z83zaEfWY5a15vgm OYYWp1X1VdStFlrWrHThuQMsK3e0q3aUvPMiXFCqAbjEjro69OkabQsYARSEDp0B2UXR DOtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=HNPF6cF55pSJrG4vKP61aGnpmnZAXtuohe9MtpkraqFMTMGEED+FtU733UX1Ka35cE O9oCoaJEBDENusrviKRak38f/C00G1nb0uaVmr6thgjqR/rqaS19LjTWfw/B1IfE2eJ7 G1NV10X38L/Znrx7AhBEzWAJI8EjCQJOKbmwUtjLC2wG24hY5SZH+aS2ldzpVc4T6zVv HbTlJ+fvRGwYSYvecgb/FaUufP19kMnb2QzmU7DbPVtlcPVFu7/9cIfRs4Z9UHODtvCM QcaYQradw+uJCVUhAP+w/FIoLbmw2jF88rl40/STjfpUrL5lJGLTJZwHAS3u3DSMWczj yAlg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=reXZKVgJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o12-20020a056402038c00b005028f35cbbesi3892874edv.200.2023.04.07.13.32.00; Fri, 07 Apr 2023 13:32:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=reXZKVgJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231126AbjDGUU2 (ORCPT + 99 others); Fri, 7 Apr 2023 16:20:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231231AbjDGUUS (ORCPT ); Fri, 7 Apr 2023 16:20:18 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D93FBC65F for ; Fri, 7 Apr 2023 13:20:07 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-54bfc4e0330so79098737b3.3 for ; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=reXZKVgJAcUhKBFqAhAfXZIMQul2LNsezKlEEoHtiLFTJvynT88LRPnZVM8H43my2O 0AdWQfw3AEOAoOIF7B7AWJxVFSFD/w3Q2pEfhk9TLSFsHjsMCNwDjt1nToC3Erq9lXvB g1LbxQEVzYJTdlQ5MYz5hDc9P8gRoRV3iaOFlg0OdPEkY2HKDJ8CNZ5ywg2KOO/iF5t6 lVplsBeH6uvx9ev0WzAiyQG3vF+n90o7T5CR+YbfiXxyMGTfkTZG4CFGpeaqYlO+IJif dO3yYbv3bKl2jMrlV2SrXMEB8JHo/vfYywtDCcW7wAKIKF7yN3CbU+qRkgehxk1sT0AX eQgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=7JnEeDrobtxT7XUksu4yoXsn8s+3qK9fTEQymgEIjbh3vbTWZvWX2Q9uihCi6oCBB8 oj5m7e9rU+aWgvxf2ThTUWE9Vwlda1XfO98XPUiBz5X489kG7lcsw7KYepgbWm1lOxW1 8AxMNnrxWf4Fdt9ljYWkvE2F143n8xHw/MnuSJRVEjK4aFhDIrhJfQozgFxt8gpr9zAP S7qa5KRkLAlDB+nGhIJLgKsfJ44E+zivoG6/zRVImGI9Nl6hiyADXqxox2xf0B9eGwqs DgnFmZ2wkHTRRzrdsyrkh7sNRLzYm592GlYoSZtPmBAbotBK789krz9u/kMGTBtMcBNS amEQ== X-Gm-Message-State: AAQBX9eaNlf3CYb4IAKUd3DnLvKeJdWBF8btMQxmnBD61pydccvdmeVf VoAxVi5czxKowqb+Lx6I1/OFjEGMkA== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a05:6902:909:b0:a27:3ecc:ffe7 with SMTP id bu9-20020a056902090900b00a273eccffe7mr5158014ybb.3.1680898807168; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:20 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-5-sagis@google.com> Subject: [RFC PATCH 4/5] KVM: TDX: Implement moving private pages between 2 TDs From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762550921073069923?= X-GMAIL-MSGID: =?utf-8?q?1762550921073069923?= Added functionality for moving the private EPT table from one TD to a new one. This function moves the root of the private EPT table and overwrite the root of the destination. Signed-off-by: Sagi Shahar --- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 77 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 3 ++ 4 files changed, 137 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index d10b08eeaefee..09bae7fe18a12 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -120,6 +120,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu); void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a35f2e7f9bc70..1acc9338323da 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3789,6 +3789,66 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm) return r; } +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_mmu *src_mmu = src_vcpu->arch.mmu; + gfn_t gfn_shared = kvm_gfn_shared_mask(vcpu->kvm); + hpa_t private_root_hpa, shared_root_hpa; + int r = -EINVAL; + + // Hold locks for both src and dst. Always take the src lock first. + write_lock(&src_vcpu->kvm->mmu_lock); + write_lock(&vcpu->kvm->mmu_lock); + + if (!gfn_shared) + goto out_unlock; + + WARN_ON_ONCE(!is_tdp_mmu_active(vcpu)); + WARN_ON_ONCE(!is_tdp_mmu_active(src_vcpu)); + + r = mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->root_role.direct); + if (r) + goto out_unlock; + + /* + * The private root is moved from the src to the dst and is marked as + * invalid in the src. + */ + private_root_hpa = kvm_tdp_mmu_move_private_pages_from(vcpu, src_vcpu); + if (private_root_hpa == INVALID_PAGE) { + /* + * This likely means that the private root was already moved by + * another vCPU. + */ + private_root_hpa = kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(vcpu, true); + if (private_root_hpa == INVALID_PAGE) { + r = -EINVAL; + goto out_unlock; + } + } + + mmu->private_root_hpa = private_root_hpa; + src_mmu->private_root_hpa = INVALID_PAGE; + + /* + * The shared root is allocated normally and is not moved from the src. + */ + shared_root_hpa = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu, false); + mmu->root.hpa = shared_root_hpa; + + kvm_mmu_load_pgd(vcpu); + static_call(kvm_x86_flush_tlb_current)(vcpu); + +out_unlock: + write_unlock(&vcpu->kvm->mmu_lock); + write_unlock(&src_vcpu->kvm->mmu_lock); + + return r; +} +EXPORT_SYMBOL(kvm_mmu_move_private_pages_from); + static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 327dee4f6170e..685528fdc0ad6 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -296,6 +296,23 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, trace_kvm_mmu_get_page(sp, true); } +static struct kvm_mmu_page * +kvm_tdp_mmu_get_vcpu_root_no_alloc(struct kvm_vcpu *vcpu, union kvm_mmu_page_role role) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_mmu_page *root; + + lockdep_assert_held_read(&kvm->mmu_lock); + + for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word == role.word && + kvm_tdp_mmu_get_root(root)) + return root; + } + + return NULL; +} + static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, bool private) { @@ -311,11 +328,9 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, */ if (private) kvm_mmu_page_role_set_private(&role); - for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { - if (root->role.word == role.word && - kvm_tdp_mmu_get_root(root)) - goto out; - } + root = kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!!root) + goto out; root = tdp_mmu_alloc_sp(vcpu, role); tdp_mmu_init_sp(root, NULL, 0); @@ -330,6 +345,58 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, return root; } +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + union kvm_mmu_page_role role = vcpu->arch.mmu->root_role; + struct kvm *kvm = vcpu->kvm; + struct kvm *src_kvm = src_vcpu->kvm; + struct kvm_mmu_page *private_root = NULL; + struct kvm_mmu_page *root; + s64 num_private_pages, old; + + lockdep_assert_held_write(&vcpu->kvm->mmu_lock); + lockdep_assert_held_write(&src_vcpu->kvm->mmu_lock); + + /* Find the private root of the source. */ + kvm_mmu_page_role_set_private(&role); + for_each_tdp_mmu_root(src_kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word == role.word) { + private_root = root; + break; + } + } + if (!private_root) + return INVALID_PAGE; + + /* Remove the private root from the src kvm and add it to dst kvm. */ + list_del_rcu(&private_root->link); + list_add_rcu(&private_root->link, &kvm->arch.tdp_mmu_roots); + + num_private_pages = atomic64_read(&src_kvm->arch.tdp_private_mmu_pages); + old = atomic64_cmpxchg(&kvm->arch.tdp_private_mmu_pages, 0, + num_private_pages); + /* The destination VM should have no private pages at this point. */ + WARN_ON(old); + atomic64_set(&src_kvm->arch.tdp_private_mmu_pages, 0); + + return __pa(private_root->spt); +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool private) +{ + struct kvm_mmu_page *root; + union kvm_mmu_page_role role = vcpu->arch.mmu->root_role; + + if (private) + kvm_mmu_page_role_set_private(&role); + root = kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!root) + return INVALID_PAGE; + + return __pa(root->spt); +} + hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private) { return __pa(kvm_tdp_mmu_get_vcpu_root(vcpu, private)->spt); diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3ae3c3b8642ac..0e9d38432673d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -11,6 +11,9 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm); void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm); hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool private); +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root) {