From patchwork Wed Oct 19 16:56:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 5758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp431803wrs; Wed, 19 Oct 2022 09:57:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM63HslZg93eABbPvKgdrYNYjQn4K1L/r+r17LJZAPWHcGYVdIT5oWCY8znWw0Q13Fimd38w X-Received: by 2002:a17:907:70a:b0:741:78ab:dce5 with SMTP id xb10-20020a170907070a00b0074178abdce5mr7684164ejb.527.1666198674423; Wed, 19 Oct 2022 09:57:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666198674; cv=none; d=google.com; s=arc-20160816; b=B8uwmjVEPn/ZgkCxgbJdsb+oqEF5omvMPPRuo324arFu/b8NvPjJEBy2ewqpMvfcR6 80QevFMs5+9EcEml7CZ2eefNNimXI/RWZb+wbnk5btQuEfqGvQ9m1UF1s+OFv14CfroT pz+R4miA95XmuR4gfjHW0tbygwGsRI+LkgaHBHeeHw5gyVjLKK7m0de+kLxGsQvadP1S 2BZu/1gCYguOQz4S1q5UTPVsnNm6UiNNGyMUT6XSuP780e2PXtEcVO61xlt+CRhv7VD3 kidOHWVtILzrFkrbefbfN7ifMMaPpCcT8JilF3MNtGNUwv7cXcqbd7+dCpNJxYNzvsEK sOyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=X3MIYjg97/CMOF++gdgmdL3lkYNVJQ+hwakprJNa1vw=; b=OrZKBK3DVwzSUpI/QWjRfBIyGOL+yqQBKyf6RfsPOtZd69fVceguIjDQXlqKVhOZKx ZczY5HoTFiIxE1JycpHkfqTHqWE8I3jszwIWKOeVKxYpADXE8CDcIF3N9ery7pbfcxuF JC+EMRocFlkOr2oBCrdv8ERTAAlhcvYvaW92SPZfLuN82Suc7bQ9VDtczrKPrp3Gk0nP BuJH7Z+Z/wus70hoW2bk+xwk8LeKI32z515h98YrNGpd21NKCrvKoFeJ7wsVb8yVSlFY ATGIMxw4UnJ1qpsTaTKbmDO1LnBVxcL2NMJLJr8rHru/NMO1mDY7+ll80W6EOjkdnv31 jvNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=El92XSKr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xg11-20020a170907320b00b00741a0c28f07si11374106ejb.943.2022.10.19.09.57.30; Wed, 19 Oct 2022 09:57:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=El92XSKr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231286AbiJSQ4p (ORCPT + 99 others); Wed, 19 Oct 2022 12:56:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230420AbiJSQ4c (ORCPT ); Wed, 19 Oct 2022 12:56:32 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 594531CFF34 for ; Wed, 19 Oct 2022 09:56:31 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id h2-20020a170902f54200b0018553a8b797so5713916plf.9 for ; Wed, 19 Oct 2022 09:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=X3MIYjg97/CMOF++gdgmdL3lkYNVJQ+hwakprJNa1vw=; b=El92XSKrFJmmBh4KfZzKzepItG1uCX4HbMX09QRxMZr1OWDatcXilIKUz56utPIpDH 1OxZNpUR++MRrh0EyHFZLWpCqhbyhL0eCulj4zUbw2Qe49Ye/yusT/x52WKkMy4warlf SH+JnEvkqTCXBrMdUg4s4BJF/osU6WmXcuTAEA164VP4NPZESPf74orLZ9SxmZnTJ85V ek8g/yjRi5XzVFgCkydkvOQbo+w2pK9kwl80Rm4WZ1lwFZG60i8gNNXkw7dVF1UO+e7O lKcl0X+QatGHPr7WVCn7oiK1D7n8TPwEOUCmAENidE0ZvrYjqLDxQPwtfAKAHppONXmT llAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=X3MIYjg97/CMOF++gdgmdL3lkYNVJQ+hwakprJNa1vw=; b=VbK5Q9JCeu79hl8eKPjh8nxLKYCRO3OF8XziHMvdqaya6q5/TcwEzOOAhV8Lc+fcpn ORteCJAuqMqNSUL3Tc/lbrnQwSK7fDrAe+sayPZ31H/+0UVJ1ZvFxQqP07SzevzjDSVN pXIfmX9Sj+qnDJ7w3u2rGYp83p2RbhyfbwOmK9PdacLnXF8TPKT1QCcN0otfc66bplyg GH84gLNq3vIp/rlUa38a5B5iWOyhzxav+jfTAULteowy8ugA34UCr0ccQnipXUXCwSQW UuaM6qyJfL/1bDPoRC2OchCjHWVfg/6/Pn/W/R6kgP7OAg58J6CUPQ0M+yTYHiek7llo hbxA== X-Gm-Message-State: ACrzQf2g4evJllMrGXF/k5pMcPmBh+X5xUoUzPiyeRB2lAPqpVyJO6Ul Cu3V50vTtDiLs7h/6Cw0p+ciMzwvqR0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:6907:b0:179:c9bc:dd73 with SMTP id j7-20020a170902690700b00179c9bcdd73mr9515508plk.159.1666198590558; Wed, 19 Oct 2022 09:56:30 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 19 Oct 2022 16:56:15 +0000 In-Reply-To: <20221019165618.927057-1-seanjc@google.com> Mime-Version: 1.0 References: <20221019165618.927057-1-seanjc@google.com> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog Message-ID: <20221019165618.927057-6-seanjc@google.com> Subject: [PATCH v6 5/8] KVM: x86/mmu: Track the number of TDP MMU pages, but not the actual pages From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Mingwei Zhang , David Matlack , Yan Zhao , Ben Gardon X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747135941015562529?= X-GMAIL-MSGID: =?utf-8?q?1747135941015562529?= Track the number of TDP MMU "shadow" pages instead of tracking the pages themselves. With the NX huge page list manipulation moved out of the common linking flow, elminating the list-based tracking means the happy path of adding a shadow page doesn't need to acquire a spinlock and can instead inc/dec an atomic. Keep the tracking as the WARN during TDP MMU teardown on leaked shadow pages is very, very useful for detecting KVM bugs. Tracking the number of pages will also make it trivial to expose the counter to userspace as a stat in the future, which may or may not be desirable. Note, the TDP MMU needs to use a separate counter (and stat if that ever comes to be) from the existing n_used_mmu_pages. The TDP MMU doesn't bother supporting the shrinker nor does it honor KVM_SET_NR_MMU_PAGES (because the TDP MMU consumes so few pages relative to shadow paging), and including TDP MMU pages in that counter would break both the shrinker and shadow MMUs, e.g. if a VM is using nested TDP. Cc: Yan Zhao Reviewed-by: Mingwei Zhang Reviewed-by: David Matlack Signed-off-by: Sean Christopherson Reviewed-by: Yan Zhao --- arch/x86/include/asm/kvm_host.h | 11 +++-------- arch/x86/kvm/mmu/tdp_mmu.c | 20 +++++++++----------- 2 files changed, 12 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 0333dbb8ec85..bbd2cecd34cb 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1290,6 +1290,9 @@ struct kvm_arch { */ bool tdp_mmu_enabled; + /* The number of TDP MMU pages across all roots. */ + atomic64_t tdp_mmu_pages; + /* * List of kvm_mmu_page structs being used as roots. * All kvm_mmu_page structs in the list should have @@ -1310,18 +1313,10 @@ struct kvm_arch { */ struct list_head tdp_mmu_roots; - /* - * List of kvm_mmu_page structs not being used as roots. - * All kvm_mmu_page structs in the list should have - * tdp_mmu_page set and a tdp_mmu_root_count of 0. - */ - struct list_head tdp_mmu_pages; - /* * Protects accesses to the following fields when the MMU lock * is held in read mode: * - tdp_mmu_roots (above) - * - tdp_mmu_pages (above) * - the link field of kvm_mmu_page structs used by the TDP MMU * - possible_nx_huge_pages; * - the possible_nx_huge_page_link field of kvm_mmu_page structs used diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 059231c82345..4e5b3ae824c1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -29,7 +29,6 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm) kvm->arch.tdp_mmu_enabled = true; INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots); spin_lock_init(&kvm->arch.tdp_mmu_pages_lock); - INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages); kvm->arch.tdp_mmu_zap_wq = wq; return 1; } @@ -54,7 +53,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) /* Also waits for any queued work items. */ destroy_workqueue(kvm->arch.tdp_mmu_zap_wq); - WARN_ON(!list_empty(&kvm->arch.tdp_mmu_pages)); + WARN_ON(atomic64_read(&kvm->arch.tdp_mmu_pages)); WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); /* @@ -377,11 +376,13 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, +1); + atomic64_inc(&kvm->arch.tdp_mmu_pages); } static void tdp_unaccount_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, -1); + atomic64_dec(&kvm->arch.tdp_mmu_pages); } /** @@ -397,17 +398,17 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, bool shared) { tdp_unaccount_mmu_page(kvm, sp); + + if (!sp->nx_huge_page_disallowed) + return; + if (shared) spin_lock(&kvm->arch.tdp_mmu_pages_lock); else lockdep_assert_held_write(&kvm->mmu_lock); - list_del(&sp->link); - - if (sp->nx_huge_page_disallowed) { - sp->nx_huge_page_disallowed = false; - untrack_possible_nx_huge_page(kvm, sp); - } + sp->nx_huge_page_disallowed = false; + untrack_possible_nx_huge_page(kvm, sp); if (shared) spin_unlock(&kvm->arch.tdp_mmu_pages_lock); @@ -1140,9 +1141,6 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, tdp_mmu_set_spte(kvm, iter, spte); } - spin_lock(&kvm->arch.tdp_mmu_pages_lock); - list_add(&sp->link, &kvm->arch.tdp_mmu_pages); - spin_unlock(&kvm->arch.tdp_mmu_pages_lock); tdp_account_mmu_page(kvm, sp); return 0;