From patchwork Fri Apr 7 20:19:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81027 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp514388vqo; Fri, 7 Apr 2023 13:33:19 -0700 (PDT) X-Google-Smtp-Source: AKy350ZxSlNbWzk9ge5H5iv2rCv2i+NVg0/J1+sqgPzBbjxbX6EE/W8Pgbpw78UIpzfdmENY/Mvl X-Received: by 2002:a05:6402:2c3:b0:501:d18f:af83 with SMTP id b3-20020a05640202c300b00501d18faf83mr2887748edx.8.1680899599363; Fri, 07 Apr 2023 13:33:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899599; cv=none; d=google.com; s=arc-20160816; b=yC6kqXGiHQOVt3TCvGynReuUadtWnBE27dvqjpDgu7mo6MKLh243Xh/PCkZG272eSa xwSejJYszaMfOBEhQgTtojyHi5E2jEAfnNAMi7cMB8Obprpatj1BJorH7JeLNCDqK19G LQ5YlphfUT89abG6jnvWcVIpWnb03bBawdrgZWXjQMkvjfJlK6UlBQdN3mNyiTA4Cu4g nqwOvLsZDt5ArIDZkCKHY5J83e7Mzyts3kHc1tkhke++qLwkFiDGECVM8SM7QQL2uYwg 2CvB46znRwm4wrYygyjWFH6pEMiPSyCBxka2pztwNDen1RtvZMFfsKGFkxi4lStaWJYL CncQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=8b0y9dzGATppcj8q9R3EYlNwH8XwDGz6PntC1PTa0lM=; b=BgVLmumfse89Gz4Q2b71CX9ek/06HoLrK5T6keaN1KHZqhOYe/41111UlPxtb1XWaf 56L1o+hpuSfV8aLuKHxcrPCrNJkqYoBI2o0gRixCmB04oZ3c+s1yYN3vEirrY4jRDxuL Q2pq//yLTUOPubeR1Lr/wQvQ2vv9j2nr0ttLB2E0XUtgPvScJ9gV741eaQSP977r6cdV 5VKjuK2EVLhSSY3n8ALFhOeZoK4YNIUPst2adlUL6K2jiC4UkyJDdyuIeMa63hhftCxy zVPu+nMhkaHobVvOHiqFr5C9wHQg8zEtnVEYERrjdIjEKcV9vHit3VSNXktGrq2y7WAE MTlg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kVkihDPr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gb18-20020a170907961200b00948c33042fcsi5128663ejc.998.2023.04.07.13.32.55; Fri, 07 Apr 2023 13:33:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kVkihDPr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230210AbjDGUT7 (ORCPT + 99 others); Fri, 7 Apr 2023 16:19:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230234AbjDGUT4 (ORCPT ); Fri, 7 Apr 2023 16:19:56 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB5A3C65D for ; Fri, 7 Apr 2023 13:19:55 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id c8-20020a170902d48800b001a1e0fd4085so25154255plg.20 for ; Fri, 07 Apr 2023 13:19:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898795; x=1683490795; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8b0y9dzGATppcj8q9R3EYlNwH8XwDGz6PntC1PTa0lM=; b=kVkihDPr/2wcKyOVlHQk1B6rs/tWhOM8mww1inoytYh7iKm5x67D2NyRve2+XK9jnl y50/l1iKtKC5LzduH0aqkWUFALmZcItOqyfohGKWVmSf9aMu6zllI5ks8kPNysQtvZHS wJ1FosoGrMpU/qJwrubMmQRiXytK2DXFJhIQtsc39RDr1ACjSnKLorktmolfRB+5lluB LVCdw30oylwxXqslJbJDvRraCncjkV3Fsst1lK+eX2DoL1MnMtmILAZ5KLba1pQ499Sw sOBOa6f9Jeg5PHSFk5QBFh0xhUP4qdblmkBOiJFfqYNqeiJtU6Xopnp4rxVwAg21QdRI ebkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898795; x=1683490795; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8b0y9dzGATppcj8q9R3EYlNwH8XwDGz6PntC1PTa0lM=; b=IEm+Dm0eQeZEpmdUgqP78jtLB8wh5ynUYcKVdyPC3rHE7ClqfqDCFVeoYCfXVgGpkD I5JUSuOor1f3HclAR7VU+KtapqwlUO18waC7Yc8/rhZrM8KPIfJyYQQDaeoamJ4LJUVO rcvbVEQNEB2743CFdH6QCEdDUKUgfH2Rq4D6Mw1WLVdXutzKvZDAqYJRA4fkey6QBYWp CQOVK+fb2x8QwKoO753vUUiY6zpvjTUa+tl/QtpYlQgSmR7yM43ufACMnEH5GihUuYzp YFDKLztGuE7thzXSsu4AjjGQROAATWdpKlO3MDMyQxW592f8mUwiY3F1/epLvGjx+Gkl enDw== X-Gm-Message-State: AAQBX9fbqLhXsyGLwdDr7I6jBf6SW3QXwCBj9mK12x9BWKlFvyJ9EORo Io5UODkE0AL2Wakq/6KnDblaBbfbeg== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a63:24c3:0:b0:513:2523:1b5f with SMTP id k186-20020a6324c3000000b0051325231b5fmr725494pgk.3.1680898795205; Fri, 07 Apr 2023 13:19:55 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:17 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-2-sagis@google.com> Subject: [RFC PATCH 1/5] KVM: Split tdp_mmu_pages to private and shared lists From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762550977896144790?= X-GMAIL-MSGID: =?utf-8?q?1762550977896144790?= tdp_mmu_pages holds all the active pages used by the mmu. When we transfer the state during intra-host migration we need to transfer the private pages but not the shared ones. Keeping them in separate counters makes this transfer more efficient. Signed-off-by: Sagi Shahar --- arch/x86/include/asm/kvm_host.h | 5 ++++- arch/x86/kvm/mmu/tdp_mmu.c | 11 +++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index ae377eec81987..5ed70cd9d74bf 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1426,9 +1426,12 @@ struct kvm_arch { struct task_struct *nx_huge_page_recovery_thread; #ifdef CONFIG_X86_64 - /* The number of TDP MMU pages across all roots. */ + /* The number of non-private TDP MMU pages across all roots. */ atomic64_t tdp_mmu_pages; + /* Same as tdp_mmu_pages but only for private pages. */ + atomic64_t tdp_private_mmu_pages; + /* * List of struct kvm_mmu_pages being used as roots. * All struct kvm_mmu_pages in the list should have diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 58a236a69ec72..327dee4f6170e 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -44,6 +44,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) destroy_workqueue(kvm->arch.tdp_mmu_zap_wq); WARN_ON(atomic64_read(&kvm->arch.tdp_mmu_pages)); + WARN_ON(atomic64_read(&kvm->arch.tdp_private_mmu_pages)); WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); /* @@ -373,13 +374,19 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, +1); - atomic64_inc(&kvm->arch.tdp_mmu_pages); + if (is_private_sp(sp)) + atomic64_inc(&kvm->arch.tdp_private_mmu_pages); + else + atomic64_inc(&kvm->arch.tdp_mmu_pages); } static void tdp_unaccount_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, -1); - atomic64_dec(&kvm->arch.tdp_mmu_pages); + if (is_private_sp(sp)) + atomic64_dec(&kvm->arch.tdp_private_mmu_pages); + else + atomic64_dec(&kvm->arch.tdp_mmu_pages); } /** From patchwork Fri Apr 7 20:19:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp513878vqo; Fri, 7 Apr 2023 13:32:03 -0700 (PDT) X-Google-Smtp-Source: AKy350Zejxcl2rSc2AW9nXXReAXu80/najz/V0eRtYoMkoTJqKc2PoBCb066cJB63Xry0Uuhj2jz X-Received: by 2002:a17:906:eda9:b0:921:412b:d8c1 with SMTP id sa9-20020a170906eda900b00921412bd8c1mr553507ejb.71.1680899523346; Fri, 07 Apr 2023 13:32:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899523; cv=none; d=google.com; s=arc-20160816; b=jyIycTpecuSUNG4NXhI2GusKMBlFVpE1nQtHqjrqRbCdC67vwuaUgQ4UOCz6ImWP5B Xf2EqwrDnY7t8GWAwoTM1SKc5k0edi5C3ODghIESBDQQOMp3FGhiIXNslZmVxIvm5vyx kLdDlsTEqbweQFs1WNgDJebMuS+uqRLXRBVjlTCtj55IJAZOBhdlc5e9DdQP/h3hM9Bx DxStxPTkCXTQeESlTNhXmxo4xYoTeDRnMktLSbR6JyUg4UZlg1pfpgY3V5RkNPxA/gRI KN28cfbLIQLOY5qKGbjWQ4HJOtE4jrb7j1Tw9km+qDmaEW8m3uF3lA91P1OLMFIVPprx JZ/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=PEBmMXjBIn1CI5gPicrS6RIKVR0bvHgFijUJvWGbelY=; b=f/o1WDnkBXYOgg/JQB6sy90ojGnVKaQT0dzaynqB/8EJqQyZr2PpIEG3uXi/qcwAqP r9tWghWXaejGmkUn8Mzi2+4YejkQfC/A8KtbU/tvls95MzMXp29yxgGh7/pKC5PlHGO1 DZ7lduF3zbZDC7vdliGFsJQGs+r241AnxmDvMR/kHVdvcdmh9s96yLIO4Gi3554zppVN f5WXz8aNeyvJppP/PljTiznQYdc7I1vBLuGRCUhtlhMHYwFN6GoBasJMiT9IDGugwsH6 eXa3M80sS9w5oOPFy9FXRrPpF/nrcl77T94njkWwSIshHiSyuXxxTwXxDZKqqHs+37JN M8wg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=UNL2PK7t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kq16-20020a170906abd000b0092fd743c0e5si3809878ejb.184.2023.04.07.13.31.35; Fri, 07 Apr 2023 13:32:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=UNL2PK7t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230347AbjDGUUJ (ORCPT + 99 others); Fri, 7 Apr 2023 16:20:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230394AbjDGUUE (ORCPT ); Fri, 7 Apr 2023 16:20:04 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5F28C65C for ; Fri, 7 Apr 2023 13:20:01 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-54196bfcd5fso422923687b3.4 for ; Fri, 07 Apr 2023 13:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898801; x=1683490801; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PEBmMXjBIn1CI5gPicrS6RIKVR0bvHgFijUJvWGbelY=; b=UNL2PK7t+b3VGpTAoySNDTA+mhQvxrB+W6h/mWFGL0YyDEz7kTPWaIOvNmS+WVvWyS ivIZ9nIH8jHYb+kHJlxEkyXUjU1+enRo+mdeLY912hcyg00OuMBTaCSGzuzjoX5NssFe 79+sx6iktGmnKyPBoLLzL1kNQRZZljEl+8AoTkeMnvVtmXwTK7UJYy3kLEXtqTPbRJ5r eQqQbPAfzfNU/u6TfzAOYjyWVAewSxAy6MF20Isaa/0ATivpmpkVdCOHsJEPNt4Y/jRJ TtVcpT89P+DpOMKYz47RYrAaCTXlT347XRNyZPzRpHF1cy/TvBAT/UT1pBLIFEMshICg X3og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898801; x=1683490801; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PEBmMXjBIn1CI5gPicrS6RIKVR0bvHgFijUJvWGbelY=; b=ZoEH06Je+ya4q0NtuQF9QuC8i2V1VpC7COMZRDrpmNSni3i4lR9fgRfWrkCzmHh6P1 DaHaKz44/sY6UhwRI546CFoOBnQl+94Whg4PEv+FtOH+8zBmq20W13HApaj3gVd1jRKi +4/x136ei8s0qAHM1xMnvfOFnNutlj9NFiy00KReK0QYRRvfiGaQQlWw+nRuPzem+5GG f0WYl9n+3FGc9ucc15SnMsUFuP0jIFODalFYwF/rghPXiPzhV8/y1KiiMKEyJ8DIjSoj df3f4a+IA5BBpk9lNxagVQhlVOzS7FYMBmIauIyE2NhFXDwh5biBN32277ql7lSYXP7h pA6w== X-Gm-Message-State: AAQBX9dyJwvvTx2c7/+mk8oqi6uZh2YfcyW6PfBNITIXCNCs8P2GsRwB L8qN6Ow5tp7rQaGdSLPGQEN6jJFdeg== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a25:d6d6:0:b0:ad2:3839:f49 with SMTP id n205-20020a25d6d6000000b00ad238390f49mr1823667ybg.5.1680898800862; Fri, 07 Apr 2023 13:20:00 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:18 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-3-sagis@google.com> Subject: [RFC PATCH 2/5] KVM: SEV: Refactor common code out of sev_vm_move_enc_context_from From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762550898462826729?= X-GMAIL-MSGID: =?utf-8?q?1762550898462826729?= Both SEV and TDX are going to use similar flows for intra-host migration. This change moves some of the code which will be used by both architecture into shared code in x86.h Signed-off-by: Sagi Shahar --- arch/x86/kvm/svm/sev.c | 175 +++++------------------------------------ arch/x86/kvm/x86.c | 166 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.h | 16 ++++ 3 files changed, 201 insertions(+), 156 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index c25aeb550cd97..18831a0b7734e 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -1553,116 +1553,6 @@ static bool is_cmd_allowed_from_mirror(u32 cmd_id) return false; } -static int sev_lock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm) -{ - struct kvm_sev_info *dst_sev = &to_kvm_svm(dst_kvm)->sev_info; - struct kvm_sev_info *src_sev = &to_kvm_svm(src_kvm)->sev_info; - int r = -EBUSY; - - if (dst_kvm == src_kvm) - return -EINVAL; - - /* - * Bail if these VMs are already involved in a migration to avoid - * deadlock between two VMs trying to migrate to/from each other. - */ - if (atomic_cmpxchg_acquire(&dst_sev->migration_in_progress, 0, 1)) - return -EBUSY; - - if (atomic_cmpxchg_acquire(&src_sev->migration_in_progress, 0, 1)) - goto release_dst; - - r = -EINTR; - if (mutex_lock_killable(&dst_kvm->lock)) - goto release_src; - if (mutex_lock_killable_nested(&src_kvm->lock, SINGLE_DEPTH_NESTING)) - goto unlock_dst; - return 0; - -unlock_dst: - mutex_unlock(&dst_kvm->lock); -release_src: - atomic_set_release(&src_sev->migration_in_progress, 0); -release_dst: - atomic_set_release(&dst_sev->migration_in_progress, 0); - return r; -} - -static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm) -{ - struct kvm_sev_info *dst_sev = &to_kvm_svm(dst_kvm)->sev_info; - struct kvm_sev_info *src_sev = &to_kvm_svm(src_kvm)->sev_info; - - mutex_unlock(&dst_kvm->lock); - mutex_unlock(&src_kvm->lock); - atomic_set_release(&dst_sev->migration_in_progress, 0); - atomic_set_release(&src_sev->migration_in_progress, 0); -} - -/* vCPU mutex subclasses. */ -enum sev_migration_role { - SEV_MIGRATION_SOURCE = 0, - SEV_MIGRATION_TARGET, - SEV_NR_MIGRATION_ROLES, -}; - -static int sev_lock_vcpus_for_migration(struct kvm *kvm, - enum sev_migration_role role) -{ - struct kvm_vcpu *vcpu; - unsigned long i, j; - - kvm_for_each_vcpu(i, vcpu, kvm) { - if (mutex_lock_killable_nested(&vcpu->mutex, role)) - goto out_unlock; - -#ifdef CONFIG_PROVE_LOCKING - if (!i) - /* - * Reset the role to one that avoids colliding with - * the role used for the first vcpu mutex. - */ - role = SEV_NR_MIGRATION_ROLES; - else - mutex_release(&vcpu->mutex.dep_map, _THIS_IP_); -#endif - } - - return 0; - -out_unlock: - - kvm_for_each_vcpu(j, vcpu, kvm) { - if (i == j) - break; - -#ifdef CONFIG_PROVE_LOCKING - if (j) - mutex_acquire(&vcpu->mutex.dep_map, role, 0, _THIS_IP_); -#endif - - mutex_unlock(&vcpu->mutex); - } - return -EINTR; -} - -static void sev_unlock_vcpus_for_migration(struct kvm *kvm) -{ - struct kvm_vcpu *vcpu; - unsigned long i; - bool first = true; - - kvm_for_each_vcpu(i, vcpu, kvm) { - if (first) - first = false; - else - mutex_acquire(&vcpu->mutex.dep_map, - SEV_NR_MIGRATION_ROLES, 0, _THIS_IP_); - - mutex_unlock(&vcpu->mutex); - } -} - static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm) { struct kvm_sev_info *dst = &to_kvm_svm(dst_kvm)->sev_info; @@ -1744,25 +1634,6 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm) } } -static int sev_check_source_vcpus(struct kvm *dst, struct kvm *src) -{ - struct kvm_vcpu *src_vcpu; - unsigned long i; - - if (!sev_es_guest(src)) - return 0; - - if (atomic_read(&src->online_vcpus) != atomic_read(&dst->online_vcpus)) - return -EINVAL; - - kvm_for_each_vcpu(i, src_vcpu, src) { - if (!src_vcpu->arch.guest_state_protected) - return -EINVAL; - } - - return 0; -} - int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) { struct kvm_sev_info *dst_sev = &to_kvm_svm(kvm)->sev_info; @@ -1777,19 +1648,20 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) ret = -EBADF; goto out_fput; } - source_kvm = source_kvm_file->private_data; - ret = sev_lock_two_vms(kvm, source_kvm); + src_sev = &to_kvm_svm(source_kvm)->sev_info; + + ret = pre_move_enc_context_from(kvm, source_kvm, + &dst_sev->migration_in_progress, + &src_sev->migration_in_progress); if (ret) goto out_fput; - if (sev_guest(kvm) || !sev_guest(source_kvm)) { + if (sev_guest(kvm) || !sev_es_guest(source_kvm)) { ret = -EINVAL; - goto out_unlock; + goto out_post; } - src_sev = &to_kvm_svm(source_kvm)->sev_info; - dst_sev->misc_cg = get_current_misc_cg(); cg_cleanup_sev = dst_sev; if (dst_sev->misc_cg != src_sev->misc_cg) { @@ -1799,34 +1671,21 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) charged = true; } - ret = sev_lock_vcpus_for_migration(kvm, SEV_MIGRATION_SOURCE); - if (ret) - goto out_dst_cgroup; - ret = sev_lock_vcpus_for_migration(source_kvm, SEV_MIGRATION_TARGET); - if (ret) - goto out_dst_vcpu; - - ret = sev_check_source_vcpus(kvm, source_kvm); - if (ret) - goto out_source_vcpu; - sev_migrate_from(kvm, source_kvm); kvm_vm_dead(source_kvm); cg_cleanup_sev = src_sev; ret = 0; -out_source_vcpu: - sev_unlock_vcpus_for_migration(source_kvm); -out_dst_vcpu: - sev_unlock_vcpus_for_migration(kvm); out_dst_cgroup: /* Operates on the source on success, on the destination on failure. */ if (charged) sev_misc_cg_uncharge(cg_cleanup_sev); put_misc_cg(cg_cleanup_sev->misc_cg); cg_cleanup_sev->misc_cg = NULL; -out_unlock: - sev_unlock_two_vms(kvm, source_kvm); +out_post: + post_move_enc_context_from(kvm, source_kvm, + &dst_sev->migration_in_progress, + &src_sev->migration_in_progress); out_fput: if (source_kvm_file) fput(source_kvm_file); @@ -2058,7 +1917,11 @@ int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd) } source_kvm = source_kvm_file->private_data; - ret = sev_lock_two_vms(kvm, source_kvm); + source_sev = &to_kvm_svm(source_kvm)->sev_info; + mirror_sev = &to_kvm_svm(kvm)->sev_info; + ret = lock_two_vms_for_migration(kvm, source_kvm, + &mirror_sev->migration_in_progress, + &source_sev->migration_in_progress); if (ret) goto e_source_fput; @@ -2078,9 +1941,7 @@ int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd) * The mirror kvm holds an enc_context_owner ref so its asid can't * disappear until we're done with it */ - source_sev = &to_kvm_svm(source_kvm)->sev_info; kvm_get_kvm(source_kvm); - mirror_sev = &to_kvm_svm(kvm)->sev_info; list_add_tail(&mirror_sev->mirror_entry, &source_sev->mirror_vms); /* Set enc_context_owner and copy its encryption context over */ @@ -2101,7 +1962,9 @@ int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd) */ e_unlock: - sev_unlock_two_vms(kvm, source_kvm); + unlock_two_vms_for_migration(kvm, source_kvm, + &mirror_sev->migration_in_progress, + &source_sev->migration_in_progress); e_source_fput: if (source_kvm_file) fput(source_kvm_file); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 870041887ed91..865c434a94899 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -13596,6 +13596,172 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, } EXPORT_SYMBOL_GPL(kvm_sev_es_string_io); +/* vCPU mutex subclasses. */ +enum migration_role { + MIGRATION_SOURCE = 0, + MIGRATION_TARGET, + NR_MIGRATION_ROLES, +}; + +static int lock_vcpus_for_migration(struct kvm *kvm, enum migration_role role) +{ + struct kvm_vcpu *vcpu; + unsigned long i, j; + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (mutex_lock_killable_nested(&vcpu->mutex, role)) + goto out_unlock; + +#ifdef CONFIG_PROVE_LOCKING + if (!i) + /* + * Reset the role to one that avoids colliding with + * the role used for the first vcpu mutex. + */ + role = NR_MIGRATION_ROLES; + else + mutex_release(&vcpu->mutex.dep_map, _THIS_IP_); +#endif + } + + return 0; + +out_unlock: + + kvm_for_each_vcpu(j, vcpu, kvm) { + if (i == j) + break; + +#ifdef CONFIG_PROVE_LOCKING + if (j) + mutex_acquire(&vcpu->mutex.dep_map, role, 0, _THIS_IP_); +#endif + + mutex_unlock(&vcpu->mutex); + } + return -EINTR; +} + +static void unlock_vcpus_for_migration(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + bool first = true; + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (first) + first = false; + else + mutex_acquire(&vcpu->mutex.dep_map, NR_MIGRATION_ROLES, + 0, _THIS_IP_); + + mutex_unlock(&vcpu->mutex); + } +} + +int lock_two_vms_for_migration(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress) +{ + int r = -EBUSY; + + if (dst_kvm == src_kvm) + return -EINVAL; + + /* + * Bail if these VMs are already involved in a migration to avoid + * deadlock between two VMs trying to migrate to/from each other. + */ + if (atomic_cmpxchg_acquire(dst_migration_in_progress, 0, 1)) + return -EBUSY; + + if (atomic_cmpxchg_acquire(src_migration_in_progress, 0, 1)) + goto release_dst; + + r = -EINTR; + if (mutex_lock_killable(&dst_kvm->lock)) + goto release_src; + if (mutex_lock_killable_nested(&src_kvm->lock, SINGLE_DEPTH_NESTING)) + goto unlock_dst; + return 0; + +unlock_dst: + mutex_unlock(&dst_kvm->lock); +release_src: + atomic_set_release(src_migration_in_progress, 0); +release_dst: + atomic_set_release(dst_migration_in_progress, 0); + return r; +} +EXPORT_SYMBOL_GPL(lock_two_vms_for_migration); + +void unlock_two_vms_for_migration(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress) +{ + mutex_unlock(&dst_kvm->lock); + mutex_unlock(&src_kvm->lock); + atomic_set_release(dst_migration_in_progress, 0); + atomic_set_release(src_migration_in_progress, 0); +} +EXPORT_SYMBOL_GPL(unlock_two_vms_for_migration); + +int pre_move_enc_context_from(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress) +{ + struct kvm_vcpu *src_vcpu; + unsigned long i; + int ret = -EINVAL; + + ret = lock_two_vms_for_migration(dst_kvm, src_kvm, + dst_migration_in_progress, + src_migration_in_progress); + if (ret) + return ret; + + ret = lock_vcpus_for_migration(dst_kvm, MIGRATION_TARGET); + if (ret) + goto unlock_vms; + + ret = lock_vcpus_for_migration(src_kvm, MIGRATION_SOURCE); + if (ret) + goto unlock_dst_vcpu; + + if (atomic_read(&dst_kvm->online_vcpus) != + atomic_read(&src_kvm->online_vcpus)) + goto unlock_dst_vcpu; + + kvm_for_each_vcpu(i, src_vcpu, src_kvm) { + if (!src_vcpu->arch.guest_state_protected) + goto unlock_dst_vcpu; + } + + return 0; + +unlock_dst_vcpu: + unlock_vcpus_for_migration(dst_kvm); +unlock_vms: + unlock_two_vms_for_migration(dst_kvm, src_kvm, + dst_migration_in_progress, + src_migration_in_progress); + + return ret; +} +EXPORT_SYMBOL_GPL(pre_move_enc_context_from); + +void post_move_enc_context_from(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress) +{ + unlock_vcpus_for_migration(src_kvm); + unlock_vcpus_for_migration(dst_kvm); + unlock_two_vms_for_migration(dst_kvm, src_kvm, + dst_migration_in_progress, + src_migration_in_progress); +} +EXPORT_SYMBOL_GPL(post_move_enc_context_from); + bool kvm_arch_dirty_log_supported(struct kvm *kvm) { return kvm->arch.vm_type != KVM_X86_PROTECTED_VM; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 33a1a5341e788..554c797184994 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -502,4 +502,20 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, unsigned int port, void *data, unsigned int count, int in); +int lock_two_vms_for_migration(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress); + +void unlock_two_vms_for_migration(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress); + +int pre_move_enc_context_from(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress); + +void post_move_enc_context_from(struct kvm *dst_kvm, struct kvm *src_kvm, + atomic_t *dst_migration_in_progress, + atomic_t *src_migration_in_progress); + #endif From patchwork Fri Apr 7 20:19:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81025 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp514281vqo; Fri, 7 Apr 2023 13:33:04 -0700 (PDT) X-Google-Smtp-Source: AKy350bie7PF6Ka+N0rYRO24cZMljkqm8qlVxQkf5Ya58TbYq4ziI2s9iK9TQWBKGhQwVBEWTVNi X-Received: by 2002:a05:6402:7c3:b0:4fb:b09e:4c8b with SMTP id u3-20020a05640207c300b004fbb09e4c8bmr3301965edy.37.1680899583825; Fri, 07 Apr 2023 13:33:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899583; cv=none; d=google.com; s=arc-20160816; b=KwRZ9WMn9ju+XScRZKmCprQhudHTrTvFIptlkJrp9LvNw9uTEVwYyMnGNpxJHbX2kh tJZCZPcAg9vhyrO0SFObW8n5qwRG0xvnxuHZ2+NpI0w3GVdBe05FZByI9YSkd6lL+C3c RovVJYA5dMWh6F9GR0JgRfRn/MC1eR1jMckDAB13zDCjjC3qpprGuIEdQnpsEruKxU79 QKsLeAGQYw9cNxicwmv5du7fGDoZ4TbMBpr+8agQBs6SDsdHC4+8jaHVsXR/gWMHsTp+ WX6XkzabrsZcIvwezAnMZ387fLjc//exKP3D35RVee/WM1fJHrEr+Ak3Ues636pLq64C fZMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=GgRFXdpbsewXFN8nfe13N4oAoHTkIOTXjaNF6DLenHQ=; b=GqGUvlVbR0dVCzcQMWSm/Fy7fKj43v76Zu47NLiA/xIqs1NuvRHog617k2J3zITCwg lSlyM8kAdhb/vFeL3o4edCWIRjAhEasMobmKnlE2TbYhlKK0Yxx1KX8/SzzLKt2kkxL5 j1ulbR5eTSreulCJ5wX/Ao61lwHwY2h3GCrJ0k5cjBi6MN524Er0yrmBgrzos+TU5YHR 4YMZZgLQtvJWVfcLHSE6d2gdlnjK6X3vHyA4Xv0t/C3rUVVkBDQVMc/xNVBO6Vy+2wwX 7cfS/WDtp5OjCq/ZKq9XSGsBdWEWGQLBQtrW9KfdDXbg0lirgff2/Blxe/nR568hzZfu c5nw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=bUPKu4nh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u17-20020a17090626d100b00936255a28b0si3400920ejc.865.2023.04.07.13.32.39; Fri, 07 Apr 2023 13:33:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=bUPKu4nh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230445AbjDGUUO (ORCPT + 99 others); Fri, 7 Apr 2023 16:20:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230446AbjDGUUK (ORCPT ); Fri, 7 Apr 2023 16:20:10 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A190FC66B for ; Fri, 7 Apr 2023 13:20:04 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-632389f445cso1050b3a.0 for ; Fri, 07 Apr 2023 13:20:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898804; x=1683490804; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GgRFXdpbsewXFN8nfe13N4oAoHTkIOTXjaNF6DLenHQ=; b=bUPKu4nhvU6AIpMyXIiHt43tbAVl8CQ5zUEZSFuYtVCkd015VQ7DKPwcFXmkmH17As lVLq79uj/NQdWNkY26hN2EggIFXZpzKO7MNJr65yXQcHcpHpQeTfnrxw50qg9C4gx8RL iy6eCReaQSIdPkYWlJzdeM1yXiqFLrTJzBWQ+KyLOH2z0fuAXahHqDM3ShgTW6UQ8eiS 71szTjXJlV/MBpQ6pN2g2pLgUqRonwbez8iewZcmNKFaJDdvBEolIbXNt2Ioa/Qw1pE1 lRrvA7nz+YB0FqnzFEDqCWuFkMZ4mLINvwECc8AohsUCtkcFdtPYhDRkIHP8oomf5Drr OrxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898804; x=1683490804; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GgRFXdpbsewXFN8nfe13N4oAoHTkIOTXjaNF6DLenHQ=; b=ueetEMczomD0UuiVggyZQPxVo68R856gQTmBsBtQMaWpyKGumQKiXT2yJVYKeRTUAj W/CQKFg9UoEzeBvovagQrE5yKKkOr2puXs6PxB8HBJqzzNgtbKdpSAbRRP7CBBr2BoMW xtHayZUdTx8goELwswX9of0D4wLQN9yFzbnKDYC4YKhAc2npLL9qHPyElC1zWJ/3iTfI 6YlJhg2x6O+ZfyxQF8eJE3CWLfon0muvO9g9o95ONU06rmRjlaAOF+x47iURMupl2XvP 28r+S3C0q9ozl4i8f95aLDWLCerCWUI8zUiTNHIwyKQmuYJjUCXXj3J8R4sV/JVLRMGY xwag== X-Gm-Message-State: AAQBX9cl0rlbJXVfIxStr//f3QoBqz3J31SGRsLEzHCYOIPFKnM5xfCu KPjmlAhFfzbw8LzOoTCf7wAWOKaBQw== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a05:6a00:2d20:b0:627:d4fa:6a9c with SMTP id fa32-20020a056a002d2000b00627d4fa6a9cmr1663271pfb.6.1680898803826; Fri, 07 Apr 2023 13:20:03 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:19 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-4-sagis@google.com> Subject: [RFC PATCH 3/5] KVM: TDX: Add base implementation for tdx_vm_move_enc_context_from From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762550961834756924?= X-GMAIL-MSGID: =?utf-8?q?1762550961834756924?= This should mostly match the logic in sev_vm_move_enc_context_from. Signed-off-by: Sagi Shahar --- arch/x86/kvm/vmx/main.c | 10 +++++++ arch/x86/kvm/vmx/tdx.c | 56 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 ++ arch/x86/kvm/vmx/x86_ops.h | 5 ++++ 4 files changed, 73 insertions(+) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 5b64fe5404958..9d5d0ac465bf6 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -979,6 +979,14 @@ static int vt_vcpu_mem_enc_ioctl(struct kvm_vcpu *vcpu, void __user *argp) return tdx_vcpu_ioctl(vcpu, argp); } +static int vt_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) +{ + if (!is_td(kvm)) + return -ENOTTY; + + return tdx_vm_move_enc_context_from(kvm, source_fd); +} + #define VMX_REQUIRED_APICV_INHIBITS \ ( \ BIT(APICV_INHIBIT_REASON_DISABLE)| \ @@ -1141,6 +1149,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .dev_mem_enc_ioctl = tdx_dev_ioctl, .mem_enc_ioctl = vt_mem_enc_ioctl, .vcpu_mem_enc_ioctl = vt_vcpu_mem_enc_ioctl, + + .vm_move_enc_context_from = vt_move_enc_context_from, }; struct kvm_x86_init_ops vt_init_ops __initdata = { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 8af7e4e81c860..0999a6d827c99 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -2826,3 +2826,59 @@ int __init tdx_init(void) INIT_LIST_HEAD(&per_cpu(associated_tdvcpus, cpu)); return 0; } + +static __always_inline bool tdx_guest(struct kvm *kvm) +{ + struct kvm_tdx *tdx_kvm = to_kvm_tdx(kvm); + + return tdx_kvm->finalized; +} + +static int tdx_migrate_from(struct kvm *dst, struct kvm *src) +{ + return -EINVAL; +} + +int tdx_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) +{ + struct kvm_tdx *dst_tdx = to_kvm_tdx(kvm); + struct file *src_kvm_file; + struct kvm_tdx *src_tdx; + struct kvm *src_kvm; + int ret; + + src_kvm_file = fget(source_fd); + if (!file_is_kvm(src_kvm_file)) { + ret = -EBADF; + goto out_fput; + } + src_kvm = src_kvm_file->private_data; + src_tdx = to_kvm_tdx(src_kvm); + + ret = pre_move_enc_context_from(kvm, src_kvm, + &dst_tdx->migration_in_progress, + &src_tdx->migration_in_progress); + if (ret) + goto out_fput; + + if (tdx_guest(kvm) || !tdx_guest(src_kvm)) { + ret = -EINVAL; + goto out_post; + } + + ret = tdx_migrate_from(kvm, src_kvm); + if (ret) + goto out_post; + + kvm_vm_dead(src_kvm); + ret = 0; + +out_post: + post_move_enc_context_from(kvm, src_kvm, + &dst_tdx->migration_in_progress, + &src_tdx->migration_in_progress); +out_fput: + if (src_kvm_file) + fput(src_kvm_file); + return ret; +} diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 71818c5001862..21b7e710be1fd 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -24,6 +24,8 @@ struct kvm_tdx { atomic_t tdh_mem_track; u64 tsc_offset; + + atomic_t migration_in_progress; }; union tdx_exit_reason { diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index d049e0c72ed0c..275f5d75e9bf1 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -187,6 +187,8 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); void tdx_flush_tlb(struct kvm_vcpu *vcpu); int tdx_sept_tlb_remote_flush(struct kvm *kvm); void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); + +int tdx_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd); #else static inline int tdx_init(void) { return 0; }; static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return -ENOSYS; } @@ -241,6 +243,9 @@ static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { ret static inline void tdx_flush_tlb(struct kvm_vcpu *vcpu) {} static inline int tdx_sept_tlb_remote_flush(struct kvm *kvm) { return 0; } static inline void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) {} + +static inline int tdx_vm_move_enc_context_from(struct kvm *kvm, u + nsigned int source_fd) { return -EOPNOTSUPP; } #endif #if defined(CONFIG_INTEL_TDX_HOST) && defined(CONFIG_KVM_SMM) From patchwork Fri Apr 7 20:19:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp514035vqo; Fri, 7 Apr 2023 13:32:25 -0700 (PDT) X-Google-Smtp-Source: AKy350ZaqbqDPPzv+yjtZ3jGDuQ/W+HiBKi8tHgFtcyg0GMttYbz2jVVwO3KCR3nh3jenrk8xWjY X-Received: by 2002:aa7:d98e:0:b0:4fc:8642:ce56 with SMTP id u14-20020aa7d98e000000b004fc8642ce56mr2208830eds.25.1680899544863; Fri, 07 Apr 2023 13:32:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899544; cv=none; d=google.com; s=arc-20160816; b=ODk+F8s6jUvXJPkVZ1kjMxMSjrpO04J0aonpp/U8Q99/YzQaGtlXhpVC5PHYSCaEY6 9owA+K12fkFu4IVz+6LOaaaNsorwq2Llqu8lrMM4AU1mjP0uiO3hOAj3C6KHdld+3RRR ySBnGTuoV+1YZ1ZF6EjwzJT7N03gdqxK0HitdgyXby0xAd//KKHSzUvHVWLqjSkwbrPR FHjFXfvOgM9lTX2DK0Oip4TBXFd79RhgocEjiENEZ6McuU4Cquo0Z83zaEfWY5a15vgm OYYWp1X1VdStFlrWrHThuQMsK3e0q3aUvPMiXFCqAbjEjro69OkabQsYARSEDp0B2UXR DOtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=HNPF6cF55pSJrG4vKP61aGnpmnZAXtuohe9MtpkraqFMTMGEED+FtU733UX1Ka35cE O9oCoaJEBDENusrviKRak38f/C00G1nb0uaVmr6thgjqR/rqaS19LjTWfw/B1IfE2eJ7 G1NV10X38L/Znrx7AhBEzWAJI8EjCQJOKbmwUtjLC2wG24hY5SZH+aS2ldzpVc4T6zVv HbTlJ+fvRGwYSYvecgb/FaUufP19kMnb2QzmU7DbPVtlcPVFu7/9cIfRs4Z9UHODtvCM QcaYQradw+uJCVUhAP+w/FIoLbmw2jF88rl40/STjfpUrL5lJGLTJZwHAS3u3DSMWczj yAlg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=reXZKVgJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o12-20020a056402038c00b005028f35cbbesi3892874edv.200.2023.04.07.13.32.00; Fri, 07 Apr 2023 13:32:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=reXZKVgJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231126AbjDGUU2 (ORCPT + 99 others); Fri, 7 Apr 2023 16:20:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231231AbjDGUUS (ORCPT ); Fri, 7 Apr 2023 16:20:18 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D93FBC65F for ; Fri, 7 Apr 2023 13:20:07 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-54bfc4e0330so79098737b3.3 for ; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=reXZKVgJAcUhKBFqAhAfXZIMQul2LNsezKlEEoHtiLFTJvynT88LRPnZVM8H43my2O 0AdWQfw3AEOAoOIF7B7AWJxVFSFD/w3Q2pEfhk9TLSFsHjsMCNwDjt1nToC3Erq9lXvB g1LbxQEVzYJTdlQ5MYz5hDc9P8gRoRV3iaOFlg0OdPEkY2HKDJ8CNZ5ywg2KOO/iF5t6 lVplsBeH6uvx9ev0WzAiyQG3vF+n90o7T5CR+YbfiXxyMGTfkTZG4CFGpeaqYlO+IJif dO3yYbv3bKl2jMrlV2SrXMEB8JHo/vfYywtDCcW7wAKIKF7yN3CbU+qRkgehxk1sT0AX eQgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=7JnEeDrobtxT7XUksu4yoXsn8s+3qK9fTEQymgEIjbh3vbTWZvWX2Q9uihCi6oCBB8 oj5m7e9rU+aWgvxf2ThTUWE9Vwlda1XfO98XPUiBz5X489kG7lcsw7KYepgbWm1lOxW1 8AxMNnrxWf4Fdt9ljYWkvE2F143n8xHw/MnuSJRVEjK4aFhDIrhJfQozgFxt8gpr9zAP S7qa5KRkLAlDB+nGhIJLgKsfJ44E+zivoG6/zRVImGI9Nl6hiyADXqxox2xf0B9eGwqs DgnFmZ2wkHTRRzrdsyrkh7sNRLzYm592GlYoSZtPmBAbotBK789krz9u/kMGTBtMcBNS amEQ== X-Gm-Message-State: AAQBX9eaNlf3CYb4IAKUd3DnLvKeJdWBF8btMQxmnBD61pydccvdmeVf VoAxVi5czxKowqb+Lx6I1/OFjEGMkA== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a05:6902:909:b0:a27:3ecc:ffe7 with SMTP id bu9-20020a056902090900b00a273eccffe7mr5158014ybb.3.1680898807168; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:20 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-5-sagis@google.com> Subject: [RFC PATCH 4/5] KVM: TDX: Implement moving private pages between 2 TDs From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762550921073069923?= X-GMAIL-MSGID: =?utf-8?q?1762550921073069923?= Added functionality for moving the private EPT table from one TD to a new one. This function moves the root of the private EPT table and overwrite the root of the destination. Signed-off-by: Sagi Shahar --- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 77 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 3 ++ 4 files changed, 137 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index d10b08eeaefee..09bae7fe18a12 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -120,6 +120,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu); void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a35f2e7f9bc70..1acc9338323da 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3789,6 +3789,66 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm) return r; } +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_mmu *src_mmu = src_vcpu->arch.mmu; + gfn_t gfn_shared = kvm_gfn_shared_mask(vcpu->kvm); + hpa_t private_root_hpa, shared_root_hpa; + int r = -EINVAL; + + // Hold locks for both src and dst. Always take the src lock first. + write_lock(&src_vcpu->kvm->mmu_lock); + write_lock(&vcpu->kvm->mmu_lock); + + if (!gfn_shared) + goto out_unlock; + + WARN_ON_ONCE(!is_tdp_mmu_active(vcpu)); + WARN_ON_ONCE(!is_tdp_mmu_active(src_vcpu)); + + r = mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->root_role.direct); + if (r) + goto out_unlock; + + /* + * The private root is moved from the src to the dst and is marked as + * invalid in the src. + */ + private_root_hpa = kvm_tdp_mmu_move_private_pages_from(vcpu, src_vcpu); + if (private_root_hpa == INVALID_PAGE) { + /* + * This likely means that the private root was already moved by + * another vCPU. + */ + private_root_hpa = kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(vcpu, true); + if (private_root_hpa == INVALID_PAGE) { + r = -EINVAL; + goto out_unlock; + } + } + + mmu->private_root_hpa = private_root_hpa; + src_mmu->private_root_hpa = INVALID_PAGE; + + /* + * The shared root is allocated normally and is not moved from the src. + */ + shared_root_hpa = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu, false); + mmu->root.hpa = shared_root_hpa; + + kvm_mmu_load_pgd(vcpu); + static_call(kvm_x86_flush_tlb_current)(vcpu); + +out_unlock: + write_unlock(&vcpu->kvm->mmu_lock); + write_unlock(&src_vcpu->kvm->mmu_lock); + + return r; +} +EXPORT_SYMBOL(kvm_mmu_move_private_pages_from); + static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 327dee4f6170e..685528fdc0ad6 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -296,6 +296,23 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, trace_kvm_mmu_get_page(sp, true); } +static struct kvm_mmu_page * +kvm_tdp_mmu_get_vcpu_root_no_alloc(struct kvm_vcpu *vcpu, union kvm_mmu_page_role role) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_mmu_page *root; + + lockdep_assert_held_read(&kvm->mmu_lock); + + for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word == role.word && + kvm_tdp_mmu_get_root(root)) + return root; + } + + return NULL; +} + static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, bool private) { @@ -311,11 +328,9 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, */ if (private) kvm_mmu_page_role_set_private(&role); - for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { - if (root->role.word == role.word && - kvm_tdp_mmu_get_root(root)) - goto out; - } + root = kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!!root) + goto out; root = tdp_mmu_alloc_sp(vcpu, role); tdp_mmu_init_sp(root, NULL, 0); @@ -330,6 +345,58 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcpu, return root; } +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + union kvm_mmu_page_role role = vcpu->arch.mmu->root_role; + struct kvm *kvm = vcpu->kvm; + struct kvm *src_kvm = src_vcpu->kvm; + struct kvm_mmu_page *private_root = NULL; + struct kvm_mmu_page *root; + s64 num_private_pages, old; + + lockdep_assert_held_write(&vcpu->kvm->mmu_lock); + lockdep_assert_held_write(&src_vcpu->kvm->mmu_lock); + + /* Find the private root of the source. */ + kvm_mmu_page_role_set_private(&role); + for_each_tdp_mmu_root(src_kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word == role.word) { + private_root = root; + break; + } + } + if (!private_root) + return INVALID_PAGE; + + /* Remove the private root from the src kvm and add it to dst kvm. */ + list_del_rcu(&private_root->link); + list_add_rcu(&private_root->link, &kvm->arch.tdp_mmu_roots); + + num_private_pages = atomic64_read(&src_kvm->arch.tdp_private_mmu_pages); + old = atomic64_cmpxchg(&kvm->arch.tdp_private_mmu_pages, 0, + num_private_pages); + /* The destination VM should have no private pages at this point. */ + WARN_ON(old); + atomic64_set(&src_kvm->arch.tdp_private_mmu_pages, 0); + + return __pa(private_root->spt); +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool private) +{ + struct kvm_mmu_page *root; + union kvm_mmu_page_role role = vcpu->arch.mmu->root_role; + + if (private) + kvm_mmu_page_role_set_private(&role); + root = kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!root) + return INVALID_PAGE; + + return __pa(root->spt); +} + hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private) { return __pa(kvm_tdp_mmu_get_vcpu_root(vcpu, private)->spt); diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3ae3c3b8642ac..0e9d38432673d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -11,6 +11,9 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm); void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm); hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool private); +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root) { From patchwork Fri Apr 7 20:19:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Shahar X-Patchwork-Id: 81029 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp514617vqo; Fri, 7 Apr 2023 13:33:51 -0700 (PDT) X-Google-Smtp-Source: AKy350Yr3QP1Y4jVnwPNWm/HARYtJtqDuIA1zA69nPzQGT9JIqUEYzyRaXF/Nhn+6mKICsmO4BPV X-Received: by 2002:a17:907:78d6:b0:947:335f:5a0d with SMTP id kv22-20020a17090778d600b00947335f5a0dmr612298ejc.62.1680899630915; Fri, 07 Apr 2023 13:33:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680899630; cv=none; d=google.com; s=arc-20160816; b=ltMKkV+CvnK6NMkcip4aL6P/GdBgWJcWK6S9dm0XAkxJ/oGK7u/Ag8t5gesPhjzfPG sW2/1DDjtfD46FHiXjQ6YtbBHTyd88pYZKY9lYnrcPorg7FVhcAbU27t/gybl6nUvrkA zwKEKWhtVzdPzr0a1wXqn/kqh4ggC5uHHae1Q60b1XjiJMYi6ATm8FJrqw4mxR77xq27 EZ7GZ/CU1OpGmBoe9pKBgWTlPXn1M4eHmjU5NNk947Wo0gzuqSocDr01+ARFiPNmNqF0 IsGEo5xrDHKIGC97UgvifNjm3U+Tn/a+uEQ8sI9FvtKIhn896BT+4VMvvXIFEKYKc8Pv uMUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=88YN7x3qppPMVMra5pRzHHPVPBEL4gMuQXheETIYTNU=; b=v8sCpRMz9P/Up7is1XxHFQ+BHMzX6LXtHXSqmWvB4tgWX4Iwi1vc/t/E5H9VisrlDn z4osB9H12AUIWgsxhB0w7BUxsZEDWRfXy1lM2bi7RjemYis4RqrPw/VpzFlI6qd/51yV 1VWY49vS5wnDsR0EsY6MIg6iaYLc5iue//JbnrP5srO5QTk49n2KWD9UOet68vHXLMaH z3ygWK7covBQQlwaNnsHoRIC4f247RRnlGRynP2OMl/j4TJlVdXb9XJwWWskpIyyhmS5 X3fk0/Nad9g0yaHIaxQFssnCxelqFH2yU8JMOJWw9fxeRhfjYL36TWC4HmBm9HhfvpI0 ATFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=OMW9kzjN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e24-20020a170906249800b00948021bad39si3846186ejb.646.2023.04.07.13.33.27; Fri, 07 Apr 2023 13:33:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=OMW9kzjN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230013AbjDGUUn (ORCPT + 99 others); Fri, 7 Apr 2023 16:20:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231283AbjDGUU3 (ORCPT ); Fri, 7 Apr 2023 16:20:29 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF449CA1E for ; Fri, 7 Apr 2023 13:20:11 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 204-20020a250fd5000000b00b6d6655dc35so42735805ybp.6 for ; Fri, 07 Apr 2023 13:20:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898810; x=1683490810; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=88YN7x3qppPMVMra5pRzHHPVPBEL4gMuQXheETIYTNU=; b=OMW9kzjNG7JkuumKUv1gxM5ClRnzMIgI7kcKO2DxeN5LnLC/y/rEnRoSpywnKQs6hD xlQD/OsuGnJ5npkky5Zeo0idUTx/ZeUjWrN65bVSD+9YE94mwbexeVuduMYRcJXkdWR9 c5e4/E0ZMYZkZuNx3ir9D1lXfiqPJjj8W9x34KqqTmzX6auXI6JlOD5uUwQyobkTZHkX GCYUe1C5vg1gq2woLG6c4aqn7ii1vyB75r4qVRPb3gbnB8AF+iJW/hONV03AuDt//Ykl +JKBwB+Km4olaMCfJvrbGcPiR60/pmR36bZ7vdgt09knNdseMPjEeCZ5XJB2aV2wWMRD M2Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898810; x=1683490810; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=88YN7x3qppPMVMra5pRzHHPVPBEL4gMuQXheETIYTNU=; b=PqwT6TPWyaZCnjM5nJua/CMiBucQExiiWd3RbQaE60QBXvgbUz+HHZVcxzT0aSyZb0 1B+bMUfI4STuCQJqm8cGTdsxKGFuqx+l8AR0Y+8Gym3ZRM49C3Ejb/N2u0mf6oUurhyE 6YtOH5QSwF56wZM5SMBrD7IEh2h3UOTGTT2m2aCGPretjuWMEcLmFrlsqLOtXICfRbOn /8zhPMw0rMmG+Ao6VngTJRIQjAryn7zf5YAB/VHvwydKxMCiNIJ4qOh6O4w2b7JtNPr/ rzxWGF6O28N8rQaY8l68WK0qevvI0hBcmtlrusQIIStQpyuZ9H+SwsLEnXfMM3cjgzCp rVuw== X-Gm-Message-State: AAQBX9dzWiyNfaYe9/0FyfYAq+re0xnkNxgHb1C0HJbcZEpY4xmdFpgN LiirI6vGAnBUFN5hq/t7l4WeQYu/5A== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a25:d958:0:b0:b75:3fd4:1b31 with SMTP id q85-20020a25d958000000b00b753fd41b31mr2479838ybg.1.1680898810376; Fri, 07 Apr 2023 13:20:10 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:21 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-6-sagis@google.com> Subject: [RFC PATCH 5/5] KVM: TDX: Add core logic for TDX intra-host migration From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762551011471763384?= X-GMAIL-MSGID: =?utf-8?q?1762551011471763384?= Adds the core logic for transferring state between source and destination TDs during intra-host migration. Signed-off-by: Sagi Shahar --- arch/x86/kvm/vmx/tdx.c | 191 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 190 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0999a6d827c99..05b164a91437b 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -2834,9 +2834,198 @@ static __always_inline bool tdx_guest(struct kvm *kvm) return tdx_kvm->finalized; } +#define for_each_memslot_pair(memslots_1, memslots_2, memslot_iter_1, \ + memslot_iter_2) \ + for (memslot_iter_1 = rb_first(&memslots_1->gfn_tree), \ + memslot_iter_2 = rb_first(&memslots_2->gfn_tree); \ + memslot_iter_1 && memslot_iter_2; \ + memslot_iter_1 = rb_next(memslot_iter_1), \ + memslot_iter_2 = rb_next(memslot_iter_2)) + static int tdx_migrate_from(struct kvm *dst, struct kvm *src) { - return -EINVAL; + struct rb_node *src_memslot_iter, *dst_memslot_iter; + struct vcpu_tdx *dst_tdx_vcpu, *src_tdx_vcpu; + struct kvm_memslots *src_slots, *dst_slots; + struct kvm_vcpu *dst_vcpu, *src_vcpu; + struct kvm_tdx *src_tdx, *dst_tdx; + unsigned long i, j; + int ret; + + src_tdx = to_kvm_tdx(src); + dst_tdx = to_kvm_tdx(dst); + + src_slots = __kvm_memslots(src, 0); + dst_slots = __kvm_memslots(dst, 0); + + ret = -EINVAL; + + if (!src_tdx->finalized) { + pr_warn("Cannot migrate from a non finalized VM\n"); + goto abort; + } + + // Traverse both memslots in gfn order and compare them + for_each_memslot_pair(src_slots, dst_slots, src_memslot_iter, dst_memslot_iter) { + struct kvm_memory_slot *src_slot, *dst_slot; + + src_slot = + container_of(src_memslot_iter, struct kvm_memory_slot, + gfn_node[src_slots->node_idx]); + dst_slot = + container_of(src_memslot_iter, struct kvm_memory_slot, + gfn_node[dst_slots->node_idx]); + + if (src_slot->base_gfn != dst_slot->base_gfn || + src_slot->npages != dst_slot->npages) { + pr_warn("Cannot migrate between VMs with different memory slots configurations\n"); + goto abort; + } + + if (src_slot->flags != dst_slot->flags) { + pr_warn("Cannot migrate between VMs with different memory slots configurations\n"); + goto abort; + } + + if (src_slot->flags & KVM_MEM_PRIVATE) { + if (src_slot->restrictedmem.file->f_inode->i_ino != + dst_slot->restrictedmem.file->f_inode->i_ino) { + pr_warn("Private memslots points to different restricted files\n"); + goto abort; + } + + if (src_slot->restrictedmem.index != dst_slot->restrictedmem.index) { + pr_warn("Private memslots points to the restricted file at different offsets\n"); + goto abort; + } + } + } + + if (src_memslot_iter || dst_memslot_iter) { + pr_warn("Cannot migrate between VMs with different memory slots configurations\n"); + goto abort; + } + + dst_tdx->hkid = src_tdx->hkid; + dst_tdx->tdr_pa = src_tdx->tdr_pa; + + dst_tdx->tdcs_pa = kcalloc(tdx_info.nr_tdcs_pages, sizeof(*dst_tdx->tdcs_pa), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!dst_tdx->tdcs_pa) { + ret = -ENOMEM; + goto late_abort; + } + memcpy(dst_tdx->tdcs_pa, src_tdx->tdcs_pa, + tdx_info.nr_tdcs_pages * sizeof(*dst_tdx->tdcs_pa)); + + dst_tdx->tsc_offset = src_tdx->tsc_offset; + dst_tdx->attributes = src_tdx->attributes; + dst_tdx->xfam = src_tdx->xfam; + dst_tdx->kvm.arch.gfn_shared_mask = src_tdx->kvm.arch.gfn_shared_mask; + + kvm_for_each_vcpu(i, src_vcpu, src) + tdx_flush_vp_on_cpu(src_vcpu); + + /* Copy per-vCPU state */ + kvm_for_each_vcpu(i, src_vcpu, src) { + src_tdx_vcpu = to_tdx(src_vcpu); + dst_vcpu = kvm_get_vcpu(dst, i); + dst_tdx_vcpu = to_tdx(dst_vcpu); + + vcpu_load(dst_vcpu); + + memcpy(dst_vcpu->arch.regs, src_vcpu->arch.regs, + NR_VCPU_REGS * sizeof(src_vcpu->arch.regs[0])); + dst_vcpu->arch.regs_avail = src_vcpu->arch.regs_avail; + dst_vcpu->arch.regs_dirty = src_vcpu->arch.regs_dirty; + + dst_vcpu->arch.tsc_offset = dst_tdx->tsc_offset; + + dst_tdx_vcpu->interrupt_disabled_hlt = src_tdx_vcpu->interrupt_disabled_hlt; + dst_tdx_vcpu->buggy_hlt_workaround = src_tdx_vcpu->buggy_hlt_workaround; + + dst_tdx_vcpu->tdvpr_pa = src_tdx_vcpu->tdvpr_pa; + dst_tdx_vcpu->tdvpx_pa = kcalloc(tdx_info.nr_tdvpx_pages, + sizeof(*dst_tdx_vcpu->tdvpx_pa), + GFP_KERNEL_ACCOUNT); + if (!dst_tdx_vcpu->tdvpx_pa) { + ret = -ENOMEM; + vcpu_put(dst_vcpu); + goto late_abort; + } + memcpy(dst_tdx_vcpu->tdvpx_pa, src_tdx_vcpu->tdvpx_pa, + tdx_info.nr_tdvpx_pages * sizeof(*dst_tdx_vcpu->tdvpx_pa)); + + td_vmcs_write64(dst_tdx_vcpu, POSTED_INTR_DESC_ADDR, __pa(&dst_tdx_vcpu->pi_desc)); + + /* Copy private EPT tables */ + if (kvm_mmu_move_private_pages_from(dst_vcpu, src_vcpu)) { + ret = -EINVAL; + vcpu_put(dst_vcpu); + goto late_abort; + } + + for (j = 0; j < tdx_info.nr_tdvpx_pages; j++) + src_tdx_vcpu->tdvpx_pa[j] = 0; + + src_tdx_vcpu->tdvpr_pa = 0; + + vcpu_put(dst_vcpu); + } + + for_each_memslot_pair(src_slots, dst_slots, src_memslot_iter, + dst_memslot_iter) { + struct kvm_memory_slot *src_slot, *dst_slot; + + src_slot = container_of(src_memslot_iter, + struct kvm_memory_slot, + gfn_node[src_slots->node_idx]); + dst_slot = container_of(src_memslot_iter, + struct kvm_memory_slot, + gfn_node[dst_slots->node_idx]); + + for (i = 1; i < KVM_NR_PAGE_SIZES; ++i) { + unsigned long ugfn; + int level = i + 1; + + /* + * If the gfn and userspace address are not aligned wrt each other, then + * large page support should already be disabled at this level. + */ + ugfn = dst_slot->userspace_addr >> PAGE_SHIFT; + if ((dst_slot->base_gfn ^ ugfn) & (KVM_PAGES_PER_HPAGE(level) - 1)) + continue; + + dst_slot->arch.lpage_info[i - 1] = + src_slot->arch.lpage_info[i - 1]; + src_slot->arch.lpage_info[i - 1] = NULL; + } + } + + dst->mem_attr_array.xa_head = src->mem_attr_array.xa_head; + src->mem_attr_array.xa_head = NULL; + + dst_tdx->finalized = true; + + /* Clear source VM to avoid freeing the hkid and pages on VM put */ + src_tdx->hkid = -1; + src_tdx->tdr_pa = 0; + for (i = 0; i < tdx_info.nr_tdcs_pages; i++) + src_tdx->tdcs_pa[i] = 0; + + return 0; + +late_abort: + /* If we aborted after the state transfer already started, the src VM + * is no longer valid. + */ + kvm_vm_dead(src); + +abort: + dst_tdx->hkid = -1; + dst_tdx->tdr_pa = 0; + + return ret; } int tdx_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)