From patchwork Fri Jul 14 06:50:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120296 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2341092vqm; Fri, 14 Jul 2023 00:54:49 -0700 (PDT) X-Google-Smtp-Source: APBJJlFAkEQq2Z+hc8nCRu26cuzgJKXhHn1hOYt5pn5lT54BE5hvoQFs2OBU7b89Rm20/H7m6YvE X-Received: by 2002:aa7:c3d8:0:b0:51f:f1a4:edc6 with SMTP id l24-20020aa7c3d8000000b0051ff1a4edc6mr3119464edr.37.1689321289286; Fri, 14 Jul 2023 00:54:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321289; cv=none; d=google.com; s=arc-20160816; b=NRuCI15IZ4dSMEIPgxEL91h4k7p5xTufO5SYeB3C1wJeNyxj8rqRlAap0bn+pYBIMw HYZ6zfFMthNudMQa6fXW7uXYcR1j92AZKTnBky/7t0YITX3lFrhlElA0raLF2DkBdgNL SBON1puhQuT+oF7+Vl5rgIpIZGEZ/vFO/gtpvuvY/i08XwaNtjD3SM0r8meXLvGZ0fHj DMww6XpNTeuTjip1gcFOUgMf2YmrHmnTBRApAcNiJGVyW/oXhLhTpiOAmJm6CDZT9DwE iQcyDSc/AT2RuV2YRJBd3U6nCOmJmIYkEfkTkFjxcu+jkMc3zRJ386K0Iie+QzPI9L6k 5tOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=tq5Q9F2gLxJLnmUdRuEyz1Rj9zYdg2CJUzYnXrBGmrg=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=L0YU/b2Tqo6yeZ+5XSkq5QNDQj7KKn455bGH8iWPsbIxTM+zZPHDLxAc6v5b2RuC3m 3KmcTRQ5T27zbV05ENUKd2i6xnZs/AYUe+kRwSS8krqFCeR2HeViNh3uB91qJ5IV8/px egl3Fa+2YtBeYNcAu6ZdVBNOI3SOVcl5m+E7EFrbKAOfFaW7fFcwoQaMojXvgImbYwpq urMz9T8a0oGhyy1LZHJoltU76CT+tMJwSut47xMPFroMxFldBzDsXd8/tn6AEJbcPuBX h5elrwTCmfXHKPeOcFMFK5DUMA74YPtJ5TuqzZMQuYNYXPeFw0HIZ0wE/tBpf9uzC2HY YCsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YndcM5mn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u18-20020aa7d892000000b0051e5cabfebdsi1867596edq.373.2023.07.14.00.54.24; Fri, 14 Jul 2023 00:54:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YndcM5mn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235247AbjGNHR7 (ORCPT + 99 others); Fri, 14 Jul 2023 03:17:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235221AbjGNHRu (ORCPT ); Fri, 14 Jul 2023 03:17:50 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9049126B3; Fri, 14 Jul 2023 00:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319069; x=1720855069; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ut2e3J98JDctUs6MMT7+OPH8GV0+N2ela+vuLlRy0YU=; b=YndcM5mn3lBZKgh+LzVexAQHPVUIrMy0yPzV8Y+ZJ984DB1+ZbgWnBeK CJmgltDgKzDYgSqh7vXXh/2vur7DpnZA3rZsFV6VO4WQR2Qwg8rML8oNK xOTu9J9VDNyvXaGTbbwQzNFruaTeHhVnCJtqyjisbZOfzOeuqG9AVcNIh 3VzCy7MsIJo8aWdWbHXCymuc7Om3npoXKonmgPmkBx5jaMZl+Y2LcxAte RUJDi+Kfe0ACzHnNOOlHoK/irI/OmnJW406nvdj3eSue4dcPduv9wO4BD r5JlEVwBFDVreMms+9dPal+LmaF1BCcpd/tP7ImwTwqnvH3urOUiXbN+A Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="396221566" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="396221566" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:16:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="1052955616" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="1052955616" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:16:46 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 01/12] KVM: x86/mmu: helpers to return if KVM honors guest MTRRs Date: Fri, 14 Jul 2023 14:50:06 +0800 Message-Id: <20230714065006.20201-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381760268571705 X-GMAIL-MSGID: 1771381760268571705 Added helpers to check if KVM honors guest MTRRs. The inner helper __kvm_mmu_honors_guest_mtrrs() is also provided to outside callers for the purpose of checking if guest MTRRs were honored before stopping non-coherent DMA. Suggested-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/kvm/mmu.h | 7 +++++++ arch/x86/kvm/mmu/mmu.c | 15 +++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 92d5a1924fc1..38bd449226f6 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -235,6 +235,13 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, return -(u32)fault & errcode; } +bool __kvm_mmu_honors_guest_mtrrs(struct kvm *kvm, bool vm_has_noncoherent_dma); + +static inline bool kvm_mmu_honors_guest_mtrrs(struct kvm *kvm) +{ + return __kvm_mmu_honors_guest_mtrrs(kvm, kvm_arch_has_noncoherent_dma(kvm)); +} + void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); int kvm_arch_write_log_dirty(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1e5db621241f..b4f89f015c37 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4516,6 +4516,21 @@ static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vcpu, } #endif +bool __kvm_mmu_honors_guest_mtrrs(struct kvm *kvm, bool vm_has_noncoherent_dma) +{ + /* + * If the TDP is enabled, the host MTRRs are ignored by TDP + * (shadow_memtype_mask is non-zero), and the VM has non-coherent DMA + * (DMA doesn't snoop CPU caches), KVM's ABI is to honor the memtype + * from the guest's MTRRs so that guest accesses to memory that is + * DMA'd aren't cached against the guest's wishes. + * + * Note, KVM may still ultimately ignore guest MTRRs for certain PFNs, + * e.g. KVM will force UC memtype for host MMIO. + */ + return vm_has_noncoherent_dma && tdp_enabled && shadow_memtype_mask; +} + int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { /* From patchwork Fri Jul 14 06:50:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2338079vqm; Fri, 14 Jul 2023 00:46:20 -0700 (PDT) X-Google-Smtp-Source: APBJJlFNGKXmCzg6Qnb3AN5Reu2p2JboS8c/UshaHkk+2KBP9XbBNlmQKjY28gSGEhWRbIzrd3nU X-Received: by 2002:a05:6358:2610:b0:134:d24d:86b with SMTP id l16-20020a056358261000b00134d24d086bmr6076943rwc.16.1689320780601; Fri, 14 Jul 2023 00:46:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689320780; cv=none; d=google.com; s=arc-20160816; b=quOwnGm8an9i2V5UZ31cnoVGLxjl5TBjMVuC+mYnUGFqITKNYhsI7xpM+kJILQ5JQk P//YX0ctAGUI0jQRfBHzkvdhNFoPMIrHlOuHd0wYj43oCmRo6A2If+8yFdo6rHX87BC0 bTyrXy61TCtVjFQoq5ImHRtRhfQtIQlJAVV0KJ7Ey3AEQO9sZaeoYNi6Tn0a97pfwOTY pFXlpLJ53MChMOlIlficVrkQXXLiu383IAR1iG+PxgBY1fowyqBhv9QSZIXt11dkMWjK 3UA4x8eF9wbT/m6a7LxC+r6no2gIy6QjG4buHlY/oJ8bF1djfd+pwiS5z5nsVn8mMER4 hnwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=U6ZrHyKCEoL2hYaOj4OX6561KOcphkx90hTStb8bNEY=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=nSkfsvbKVKRNiulTtWNmSsJ8paNSEM1ERwsULSSkJMDbCr6n/QqpO5t70XC1V6GgIv XyQ7qzDocjyxxsWN8qz+2zXiMumIjHsPTv45A3jE2JIUyeFXyYEAC3quD6wx8mCcMvT3 hN778OxLDYopnFpwo35KV+DcsigPHSQfx7QwPBxLE+VWzOlRKV3ZLuceRBkUWR870VlY oChR18o23eOwVUFrIqieMfFT8GXBpe/bl3jJzyTMQf/WO3GsfPfAfp3VPZYk48iR37VQ UgRm0FZo685k/PWTvNlStW1jZ9ajAeF9gWr6uzzl2gqXtbPIPqMxzUPvMI0LF7VQkDnc tiRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=U1raypuw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 2-20020a630b02000000b0055c7eb346e1si3148746pgl.757.2023.07.14.00.46.08; Fri, 14 Jul 2023 00:46:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=U1raypuw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235017AbjGNHS0 (ORCPT + 99 others); Fri, 14 Jul 2023 03:18:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234484AbjGNHSY (ORCPT ); Fri, 14 Jul 2023 03:18:24 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEA622722; Fri, 14 Jul 2023 00:18:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319104; x=1720855104; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=nOdi2KVC4OwTsVt3a1I9Bdhj8qUZvzNWXLp63uGaNfw=; b=U1raypuwcmokLsKtOEHtQPxn2QVjLS/EdsRal2C3weCpEe1h7bdyLci9 tvhElaVvLZ6xp+yOk3IRCUdIz6nzacF8MlzRg2PfQDaC0j4pquv2/IeWz +sOoeIbGP3XrInwv/4tqAEuke+jDDwVWAsTv7RQoUrDM1dqgdXm9ntXYB ul73TKny2q+bOG8ZRQ3CgRz8WzuHhiIN8N2ri2ocNDoXWAkTRgA89NeTJ U2HkH8v5ef9afjpm6NXq9VflEQ+BVeN06Fd+OZdokjuhYbaCQurVKg3VA hR4+coXpWO2/f/VoLXcsISAnAX4ZksWtjiuGkGgz7MBh2zt/En/IyIzGV A==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="396221903" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="396221903" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:17:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="751940812" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="751940812" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:17:16 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 02/12] KVM: x86/mmu: Use KVM honors guest MTRRs helper in kvm_tdp_page_fault() Date: Fri, 14 Jul 2023 14:50:43 +0800 Message-Id: <20230714065043.20258-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381227017376512 X-GMAIL-MSGID: 1771381227017376512 Let kvm_tdp_page_fault() use helper function kvm_mmu_honors_guest_mtrrs() to decide if it needs to consult guest MTRR to check GFN range consistency. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/kvm/mmu/mmu.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b4f89f015c37..7f52bbe013b3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4536,16 +4536,9 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) /* * If the guest's MTRRs may be used to compute the "real" memtype, * restrict the mapping level to ensure KVM uses a consistent memtype - * across the entire mapping. If the host MTRRs are ignored by TDP - * (shadow_memtype_mask is non-zero), and the VM has non-coherent DMA - * (DMA doesn't snoop CPU caches), KVM's ABI is to honor the memtype - * from the guest's MTRRs so that guest accesses to memory that is - * DMA'd aren't cached against the guest's wishes. - * - * Note, KVM may still ultimately ignore guest MTRRs for certain PFNs, - * e.g. KVM will force UC memtype for host MMIO. + * across the entire mapping. */ - if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) { + if (kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) { for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) { int page_num = KVM_PAGES_PER_HPAGE(fault->max_level); gfn_t base = gfn_round_for_level(fault->gfn, From patchwork Fri Jul 14 06:51:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120289 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2340626vqm; Fri, 14 Jul 2023 00:53:20 -0700 (PDT) X-Google-Smtp-Source: APBJJlEwI0S33u84ERceGvn6ZOluLo/b8N+nsslcSpNs86QeZ2MsU6Zc7ZLEPjiB79r+2V4WM4fT X-Received: by 2002:a05:6512:2315:b0:4f8:5cde:a44f with SMTP id o21-20020a056512231500b004f85cdea44fmr3451850lfu.10.1689321200529; Fri, 14 Jul 2023 00:53:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321200; cv=none; d=google.com; s=arc-20160816; b=Xyf02nuuQIGpWtENW2snQjju329/CaTy2lPMMxSmLIo7J7uDhIcBhy0F/5kFda186C OxIfGEgL+ONz23iUtCkUmrocvUcoer/mwbjCmdRPHbpCDnIx6RQy2SQwrP3OKwIDWduF AlILC4JYsGhnLy8X4PBRIkJpLMXVJUR1XwcSYaAiKGRN4hiRkVo85IUtqm5AB2ZEyLug C1tSb5KZF9KR2lPpfqebfwnggjD2AKFSgFhAQpm7PXZx3hDWKSYiJNQ2b0/WtLWtsGlb 6lruVTIe2AEhZjERKI52uZnEFQAiAdgsoer4W0+3OhYgOEyV4WRG9RTipT3UbCfcfYjx XaLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=lCuxgGddHFvD682Gsqif0iStIaVltt/taSh4YVbErkA=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=YCS28d8JPfw/DEfV8VY3Um3lxfoDjCGGsbK7GE8/BnWCZ6JfHV6zLrw4iv9l8lFKDs IsGrl/wxUj5+RcelNi6vv+TAUyF4qj0WishyY/cN0o+ogelleaykJsS/Dqbea6rpv9jp S4WIZ/XtuqFp7PHEVU1yYSpy5BwRTxU6s7lAgqKSL+lhQsM+xHPnFIGjorB2/2eLSn1S jEd2ZGitnf3+0R4VX8QgLMV8a0n1nhPoJ9pYD/Z8+2teqEGRlfrysEi4LILjbZWqMlXf q2/M7qOFIvScEX3Lccee3Qti7C76GAP/RHVYF5HcvX8xu0YVjbraIVVkyehev1eGdQNF OdQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UmZkIiTd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i26-20020aa7dd1a000000b0051e4797bcbfsi1742534edv.30.2023.07.14.00.52.50; Fri, 14 Jul 2023 00:53:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UmZkIiTd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235340AbjGNHSc (ORCPT + 99 others); Fri, 14 Jul 2023 03:18:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235321AbjGNHS2 (ORCPT ); Fri, 14 Jul 2023 03:18:28 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2481B271E; Fri, 14 Jul 2023 00:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319106; x=1720855106; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=039VwauHGCgL1mSJQW9bLXP44yBS4nRZJPzdsRVd4cc=; b=UmZkIiTdjpYoxgPlUhOZ3r7gc6A7OFZJYj0d2vr9Out7ctv30/BvILRW N+YwePwFCtoL3z/NKnSZMrM3G7iFFL2NvCV/Q8zYNFo2wwauMlOrnuN0u 28FIoPAYli67O2IssEpQcCY5hXiGvkQnObmV/PNtkXU201bi40XuD3J8m 9P20+md6b0YchQP2vvXgR9/3+pHrPCRNXoLlVFg/Ifrxe4hhyi+haHzbQ nDWvwRvOLSKNTcCeaen4yBxmJNNDxSytNW8PGYKkOan/nOCq3hlTOdc+C hFau40q/Rj+v5AcXvV0K4RX6IGafdycEETi038cuPRWgfQLxpddxbJtqh w==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="350283092" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="350283092" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:17:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="699577183" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="699577183" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:17:56 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 03/12] KVM: x86/mmu: Use KVM honors guest MTRRs helper when CR0.CD toggles Date: Fri, 14 Jul 2023 14:51:22 +0800 Message-Id: <20230714065122.20315-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381666752229275 X-GMAIL-MSGID: 1771381666752229275 Zap SPTEs when CR0.CD is toggled if and only if KVM's MMU is honoring guest MTRRs, which is the only time that KVM incorporates the guest's CR0.CD into the final memtype. Suggested-by: Chao Gao Signed-off-by: Yan Zhao --- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9e7186864542..6693daeb5686 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -942,7 +942,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon kvm_mmu_reset_context(vcpu); if (((cr0 ^ old_cr0) & X86_CR0_CD) && - kvm_arch_has_noncoherent_dma(vcpu->kvm) && + kvm_mmu_honors_guest_mtrrs(vcpu->kvm) && !kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) kvm_zap_gfn_range(vcpu->kvm, 0, ~0ULL); } From patchwork Fri Jul 14 06:51:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120285 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2340487vqm; Fri, 14 Jul 2023 00:52:58 -0700 (PDT) X-Google-Smtp-Source: APBJJlGvWjeRQRxALWbYEKXU4vpWuDm5215p+UN85r2bJz3jwtML6sCLsG6BhEQQL51KmU0i8TnE X-Received: by 2002:a17:906:9614:b0:965:d18b:f03a with SMTP id s20-20020a170906961400b00965d18bf03amr3600721ejx.58.1689321178442; Fri, 14 Jul 2023 00:52:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321178; cv=none; d=google.com; s=arc-20160816; b=IqJzwpTo93S3xegXF7qXnT+DeVgosm/E8kW3KgMEnas0fWuQSbZHycRco/vRD7jM45 vL2c0wiNd4jrnZISsfhAD7rKuiPw0KQckDF6u9WC+MR01sMRXA0o6mZO0GqNkt9J/iZy SWnXEReNeH3c44X7oJwqHS2udZ1b9edFLr8aPBrG1Hav2J890Wq/UhdpWX59Oir6DEgD X9M454g6rpvVx5MfvvTtd71morkYO2HQ31qVSmdcjqnN6pawjcx6G41qmYtCsvOkDgfj 881ZX0uTi5w4MHBWUgnr3QZG75yYCYYfTpMUymXxwBKt3x7RmM3d8E69RtNZWwXZmCPl adAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=Nkqna1rV7LB8tdJTBxE7FC2M2i0BbhE1nwIxhYlmZFs=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=kIQi41Tprd2NeLXc55nSH6GsV4M8xATRlavgEn4IcC/2u/vlzZhfFaHX1bquQWs5W2 04CBP2x4GD3mip/+RS+LpBF0R5b3thueiokKX3u6Y+bhXT6+tZiinLnK/eYwmcplERgv cwuW22SlaexU779RbDov8pByH3/H/dbQZ3b5ZtE956XSeb7URJh04tujGwIu0Lv7hvj4 PXsiiZ87je6I+EkP7dnBUFIcE70cOwvh7v71GWc2QcoraGtsig6fDE7VWvg9ZIFxIsmH /26ggyBrG6HOteqA3P+3Utu5ObtnnUTbHqdAGhXZYSoMxdzKEQjS8M6B06+mrwiykeCU n+2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gLQQifvB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n24-20020a170906119800b00992f7fe9c0bsi8803626eja.328.2023.07.14.00.52.34; Fri, 14 Jul 2023 00:52:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=gLQQifvB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235359AbjGNHS5 (ORCPT + 99 others); Fri, 14 Jul 2023 03:18:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235348AbjGNHSf (ORCPT ); Fri, 14 Jul 2023 03:18:35 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8CCF2D45; Fri, 14 Jul 2023 00:18:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319113; x=1720855113; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9Cr+l5teQ1mq0jHpBaqwW8YXlqIUC0zccTjsvm332ms=; b=gLQQifvBnx/FOLCByzBelYmIrWesD5bJZw00LgINXc2Bw9OrlB3LZzCj 8326Cen8srYzmRQTzgdeqq0drlp1yKGuv9EgcG5+K6mxHnTon63b2tqbP Y4RebUAqHUxfAy0AcE2DN9jvWzg9YuZn9tEJymiPwy36MJNWujzv2vVXp Bd37RW8a3K+eyzqrk7lX/f+8oUTzCmkT2hQvJ7I5o/h8KV42g6Pila6I8 s02ZnYqHqWKgRGEuAheEKHVdqBEwrbP3X4H2UQqPYSHrAYOSoD4SnVW25 dFzmcFBjUOOP/00IDODfs0/TL/nGL6dVVruxXkN2+hb3+1ALxdxXpha5s w==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="396222058" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="396222058" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:18:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="716244152" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="716244152" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:18:30 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 04/12] KVM: x86/mmu: Use KVM honors guest MTRRs helper when update mtrr Date: Fri, 14 Jul 2023 14:51:56 +0800 Message-Id: <20230714065156.20375-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381643700976620 X-GMAIL-MSGID: 1771381643700976620 When guest MTRRs are updated, zap SPTEs and do zap range calcluation if and only if KVM's MMU is honoring guest MTRRs, which is the only time that KVM incorporates the guest's MTRR type into the final memtype. Suggested-by: Chao Gao Suggested-by: Sean Christopherson Cc: Kai Huang Signed-off-by: Yan Zhao --- arch/x86/kvm/mtrr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 3eb6e7f47e96..a67c28a56417 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -320,7 +320,7 @@ static void update_mtrr(struct kvm_vcpu *vcpu, u32 msr) struct kvm_mtrr *mtrr_state = &vcpu->arch.mtrr_state; gfn_t start, end; - if (!tdp_enabled || !kvm_arch_has_noncoherent_dma(vcpu->kvm)) + if (!kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) return; if (!mtrr_is_enabled(mtrr_state) && msr != MSR_MTRRdefType) From patchwork Fri Jul 14 06:52:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120305 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2344932vqm; Fri, 14 Jul 2023 01:03:59 -0700 (PDT) X-Google-Smtp-Source: APBJJlGdyhE1dfYgsv3DBZk/8HR6FM8H5YxxmOdbyYVFST4/uXpGlU761WZ2G/ewNnRgC7FBEpqA X-Received: by 2002:a17:903:110d:b0:1b8:3936:7b64 with SMTP id n13-20020a170903110d00b001b839367b64mr2958354plh.1.1689321838776; Fri, 14 Jul 2023 01:03:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321838; cv=none; d=google.com; s=arc-20160816; b=szlb4wUREnS46UUSXPDyTriQrWNdrf1WbvDHjuwpQ+OcWpL4f0l4x5dZrQcl74TqXt 0GGpjmM6yM5OLWL4yEESSsucNUcOYDUpJp50GkDxts7xtoCeCJ/UA9VixFbbfsfDO39B ZeA9N0+hBNZVTW1o1TkkIKqVZdFy9oSiFcy1Fh+CPuCs24OnI+0/EfaMPCScWpeI4G1Z YF4PyYLhfqVkTfTa1EUb6hJpSXqQpu1YIx1ThvUCGayOO9ZH3UvJt3PVjZCuZxmnGhWQ xiGu3THNfF77v6F5XbjyL2XJ1nIaoGaDThgMObaKQPWVHVXdSS9mMTlVeEfNJJbxsF4E /QiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=R3n1GEot05nOcwcu+MAz9apcLO2phCCWui8K3Q36Nrw=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=qS6ONN8RSUKBJ7YmelcVsS1ieBX4isRwPqzCBcbsdA4L7JHJ4yBo2sG99tXdjtqb64 1Qy+Nf+HN7Sik8qHXy2Rpm4WUqVllclPCj4SdwDytCbEGN6yErEJ3seTItSkAnD+s5Sx LFapBt0bAgItd8apSVebGdFi6iBmK63G/6NYyC5RWnFqjQyfsMLW+E0Qg8vrv8Q9LrCu fFu+ppwAHpDyVFxNrbT4jklNL9Rin6bb7cWtMbCVvA1C2BQWFFrLwPBP5qKS1gCSc4ic YSu3cmN1MDXakMvQGX2MAVhxZhW+wtVg9xuT65XwpSvQ0UEbvTo8yzpJKXU1jWTEa+ov ZUHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=agEYkOCX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t9-20020a170902e84900b001b8a9ddf11asi7317104plg.438.2023.07.14.01.03.44; Fri, 14 Jul 2023 01:03:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=agEYkOCX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235402AbjGNHTa (ORCPT + 99 others); Fri, 14 Jul 2023 03:19:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235403AbjGNHTQ (ORCPT ); Fri, 14 Jul 2023 03:19:16 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9345B3C01; Fri, 14 Jul 2023 00:19:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319140; x=1720855140; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mQjyugDM9ae/+xsV/na3qkdiKTAXgzQ9oL683PGryPA=; b=agEYkOCX/ojDMWCLjs+RukhYKku+B/Z4+erbNDbf5ilm1si4H9+5iDij +Y1/DEN27kSmOG8V/6+Y6/dfgWXSkCGt2JKb5DIWT2Da/MEZgPtbEmoOR T+0enAG2KN3SIkrYrdoIZocWW5zfbaSoVBoucnj/+MSNmytIrk8MazQnD JAj1+QZFNdfc6pu7439ZLNe1dMD+SuJZuLdUa7OlvpKv9PYCMZ3fWD10B lxT6tt/YGk5giYb/4ZEQ795WleUiOXZjYvkneavw7H1bW91ctVCfzLoNR 8j2cFBOFIYzMryY22NxThVhlKgOLf6M5dBXmh6fkoIr8Dey3TWYAt1d2Z Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="355349578" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="355349578" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:19:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="757477345" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="757477345" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:18:57 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 05/12] KVM: x86/mmu: zap KVM TDP when noncoherent DMA assignment starts/stops Date: Fri, 14 Jul 2023 14:52:23 +0800 Message-Id: <20230714065223.20432-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771382336492992408 X-GMAIL-MSGID: 1771382336492992408 Zap KVM TDP when noncoherent DMA assignment starts (noncoherent dma count transitions from 0 to 1) or stops (noncoherent dma count transistions from 1 to 0). Before the zap, test if guest MTRR is to be honored after the assignment starts or was honored before the assignment stops. When there's no noncoherent DMA device, EPT memory type is ((MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT) When there're noncoherent DMA devices, EPT memory type needs to honor guest CR0.CD and MTRR settings. So, if noncoherent DMA count transitions between 0 and 1, EPT leaf entries need to be zapped to clear stale memory type. This issue might be hidden when the device is statically assigned with VFIO adding/removing MMIO regions of the noncoherent DMA devices for several times during guest boot, and current KVM MMU will call kvm_mmu_zap_all_fast() on the memslot removal. But if the device is hot-plugged, or if the guest has mmio_always_on for the device, the MMIO regions of it may only be added for once, then there's no path to do the EPT entries zapping to clear stale memory type. Therefore do the EPT zapping when noncoherent assignment starts/stops to ensure stale entries cleaned away. Signed-off-by: Yan Zhao --- arch/x86/kvm/x86.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6693daeb5686..ac9548efa76f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -13164,15 +13164,31 @@ bool noinstr kvm_arch_has_assigned_device(struct kvm *kvm) } EXPORT_SYMBOL_GPL(kvm_arch_has_assigned_device); +static void kvm_noncoherent_dma_assignment_start_or_stop(struct kvm *kvm) +{ + /* + * Non-coherent DMA assignement and de-assignment will affect + * whether KVM honors guest MTRRs and cause changes in memtypes + * in TDP. + * So, specify the second parameter as true here to indicate + * non-coherent DMAs are/were involved and TDP zap might be + * necessary. + */ + if (__kvm_mmu_honors_guest_mtrrs(kvm, true)) + kvm_zap_gfn_range(kvm, gpa_to_gfn(0), gpa_to_gfn(~0ULL)); +} + void kvm_arch_register_noncoherent_dma(struct kvm *kvm) { - atomic_inc(&kvm->arch.noncoherent_dma_count); + if (atomic_inc_return(&kvm->arch.noncoherent_dma_count) == 1) + kvm_noncoherent_dma_assignment_start_or_stop(kvm); } EXPORT_SYMBOL_GPL(kvm_arch_register_noncoherent_dma); void kvm_arch_unregister_noncoherent_dma(struct kvm *kvm) { - atomic_dec(&kvm->arch.noncoherent_dma_count); + if (!atomic_dec_return(&kvm->arch.noncoherent_dma_count)) + kvm_noncoherent_dma_assignment_start_or_stop(kvm); } EXPORT_SYMBOL_GPL(kvm_arch_unregister_noncoherent_dma); From patchwork Fri Jul 14 06:52:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120284 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2340469vqm; Fri, 14 Jul 2023 00:52:56 -0700 (PDT) X-Google-Smtp-Source: APBJJlHq5hYCc2cqCHOzcD3Cj20vjuyuEkVTRKgO3CglpI1irNEKRv37UMPEPIq/ks7aTXAsQk2f X-Received: by 2002:a17:906:2d0:b0:974:c32c:b485 with SMTP id 16-20020a17090602d000b00974c32cb485mr3503022ejk.45.1689321176157; Fri, 14 Jul 2023 00:52:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321176; cv=none; d=google.com; s=arc-20160816; b=MVJ3pKc2k8wNgVrhIKZ20dxmfAx5jY/wt/bvx2iAt2oWJ9hjLv4cf738/dBlOwPUjs s8AKc/ysSU/x/DfPfKH3SvWVSFG1fLbRLKHuYujpvHdqJcbkizOIef/HyVYstbzh4HMP T1Aml2g/OEr89FgsmTm1r3mAc/KywfI8HcIlNJHKufqIOazoI7ST7zDdYkxtfkhHjrN3 MXXwrzm/Efduv+wfPVUxuwryA3ubJgJbyikdLD6aEk/J4y+cUyREH86Riv2tYury3HWe /lVKoROfWT9yfzPVcUxkwmU5uGNjyH84CA9Ekh48K4RY4HM8j/wtMga5N+CWWxKXneyr T/Qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=64ZOjWn+3KcHUmU2AaxAHXDOw3E55jO3xSpLnppNF7g=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=orXC9vxvbBNkr/536+2ObYUFJdAX8k24ms+8mM/14hLNiu/R4i99NIIP/zy9rOdrWp qzhcLWdGJV8ezyKNPrGT8Nuok66aWuu1LGhTADmBF96O8sVdOFZ4+MzivdzNW1fFnyFr gYG/joevjtC1HCRlgPPROuvuu4eWB0W/pNmeLc2M2ydUARCoUWQ+jFuj9+CHbqFzN4/A C5zDykbKd7yAZ2nKXDgACtzPXeDxtP8vxM2DBZp9mvh8vcF+JS6n1/X7cVysNKWHmb2Q Trr4EN817Hty51F4WbM+rEUGaLpJ382sb0FjCV2b+yO75LHnzylbtp1mnTmuK9w+A+VK LsuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ho4VMUKq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n23-20020a170906119700b009932528281asi9063471eja.579.2023.07.14.00.52.31; Fri, 14 Jul 2023 00:52:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Ho4VMUKq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235374AbjGNHTv (ORCPT + 99 others); Fri, 14 Jul 2023 03:19:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235366AbjGNHTr (ORCPT ); Fri, 14 Jul 2023 03:19:47 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 696722733; Fri, 14 Jul 2023 00:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319173; x=1720855173; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rpvZF8/E2sO1ceRHFD/Pgi1BwdLKFTmV/M8xCwRXvyg=; b=Ho4VMUKqqZw9Cb/OYdvyxSTZsqmqhRnZxXuBzbAQYtNvrJz9cT18Y0mx A64CWUazmozlNr5X8Kb3nqC0UYSLFlRwXiBy73D+WdrAzlZ6PjFrvuEs9 bPUsj+Cv2+IzO9Jn4bX0K0Ff9bgpVDalXfc1zVw3gY7W2Etxi1q2M45Oc qWOe0zbdIJCm7LZsSCR1K78MHS2doSKLCyxoSK/5QhsAEAGSoVDyipNWf S2J4sc2ei7Eg1j9f7iBz09+epOlWV2Dyt6t+55oumJf6J9/GkOSVHcuSD pc5MCH7KJVOP3A1hEXIzHz7gpDIfBesl2nGOrdLbf/JpUWSkujl5mDV5Y w==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="350283288" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="350283288" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:19:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="968900802" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="968900802" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:19:30 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 06/12] KVM: x86/mmu: move TDP zaps from guest MTRRs update to CR0.CD toggling Date: Fri, 14 Jul 2023 14:52:56 +0800 Message-Id: <20230714065256.20492-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381641782577259 X-GMAIL-MSGID: 1771381641782577259 If guest MTRRs are honored, always zap TDP when CR0.CD toggles and don't do it if guest MTRRs are updated under CR0.CD=1. This is because CR0.CD=1 takes precedence over guest MTRRs to decide TDP memory types, TDP memtypes are not changed if guest MTRRs update under CR0.CD=1. Instead, always do the TDP zapping when CR0.CD toggles, because even with the quirk KVM_X86_QUIRK_CD_NW_CLEARED, TDP memory types may change after guest CR0.CD toggles. Suggested-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/kvm/mtrr.c | 3 +++ arch/x86/kvm/x86.c | 3 +-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index a67c28a56417..3ce58734ad22 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -323,6 +323,9 @@ static void update_mtrr(struct kvm_vcpu *vcpu, u32 msr) if (!kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) return; + if (kvm_is_cr0_bit_set(vcpu, X86_CR0_CD)) + return; + if (!mtrr_is_enabled(mtrr_state) && msr != MSR_MTRRdefType) return; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ac9548efa76f..32cc8bfaa5f1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -942,8 +942,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon kvm_mmu_reset_context(vcpu); if (((cr0 ^ old_cr0) & X86_CR0_CD) && - kvm_mmu_honors_guest_mtrrs(vcpu->kvm) && - !kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) + kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) kvm_zap_gfn_range(vcpu->kvm, 0, ~0ULL); } EXPORT_SYMBOL_GPL(kvm_post_set_cr0); From patchwork Fri Jul 14 06:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2335192vqm; Fri, 14 Jul 2023 00:38:12 -0700 (PDT) X-Google-Smtp-Source: APBJJlHPDpDPuwvXt2fRtbrg+ZrPo81Kut0cHn148eAuzljrHWuTCAt0OZOAcnsrWp7qVnNSiFno X-Received: by 2002:a05:620a:1917:b0:767:a33:a9e6 with SMTP id bj23-20020a05620a191700b007670a33a9e6mr4572252qkb.52.1689320291919; Fri, 14 Jul 2023 00:38:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689320291; cv=none; d=google.com; s=arc-20160816; b=kg0wb5JBUIUOZv6OSA0OkkooZV4FsnUMJG37Rt6KpRqj/M7f3yFao1XgAGuN4Zj95U zKyV1RulTqHX71WNvFNGWf0SNo9XDyoWHIFCcUEUqzcoo4smeytmkrQ+O2ztBrnWBD7k uuEI0Hd8vbyY5A4VN8yvGKP+0WjkCuMNnBJn+Z3VzP537Zj2qU1kDvno1Yqzm1y4MoKL 0FUSTuNZXqPo9M6aVQ/R4jDhZETlcqkzBphtZTgGzQWEsU6brX2ZGnpXcvWXNPY1MQIq l2cBxeY3Ehd0YyZFyGAm56Fa4vi1kv0cxvCyVJwi/dwKQuJ1zEfHeLRghmUDA7G/yi66 oHfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=ewKL26IdazRhDxza10qvawg6HyLSestYjMERa7wGjIE=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=pgHDVrcTg5D6V0ygO6TLdJEI3ljkMBG55M4VooAVbWF4kF4dM5N2/YTp+BRcu8brp1 OVZGjNvkqZGB9qHk8UITROFKKImW70SVfzSu6yG+JirGiOeedcGneGGXscb/o6KZg9vB JRUqf81gO+WzJzNklHijQMCjvLddMX9WiBGIEbWXJHDBHd8Qn65H69xWJ6GBVvmMX8F8 ru9avKAknMN+heE+a7/+tEojKbXnrjG7gMVdxSSe0pP/TRrg2dEs+Gf8ngt97dNK5FSz h6wRsnLRuAn2zSuwfWz7gZg9mTNL7NXBL0puM68+xN2qeJjqirk/ev2ieLEX0F3gbLNR Yosg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aDtHqKtC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cm18-20020a17090afa1200b00263f5f3b890si820596pjb.69.2023.07.14.00.37.57; Fri, 14 Jul 2023 00:38:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aDtHqKtC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235419AbjGNHU0 (ORCPT + 99 others); Fri, 14 Jul 2023 03:20:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235417AbjGNHUN (ORCPT ); Fri, 14 Jul 2023 03:20:13 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8852E3A98; Fri, 14 Jul 2023 00:20:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319201; x=1720855201; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=5CzqIi7Pagm76e5p+ceQcdp/nHSMn4lESsq6WuvF7BA=; b=aDtHqKtCZ5GOjyxTQ4MpugYAiV1ck3cvxF/BZlONoo0UkmnA8Y0iC8uG UmD4LtQ2xCzD5fgc9xc0kDaUcQfFOPJyI4X6CKHxV6Tk8I3+SKzqRR0vQ Ib40tCmnyemvwfAmXGGG+DrHdqYxMI3dbMpyoFqxceKcK+g1UTySqG+wU 0jTGdC8MqSPJv4CMmAZBkEgIiBnwBjI23bn2+zdGFKmso5ok3e0P5PxVp bBTw1cfu66/U5/mzwAEsDuk2QItkoaxXVknBsTONABSMTW/DAsiyFdxXN z0dYoZuTV1lSG3Hs2/eJciH+srQAoGxEvIo2aw2JYHkrXkFV8TxTqLk6c A==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="350283370" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="350283370" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:20:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="968900908" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="968900908" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:19:59 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 07/12] KVM: VMX: drop IPAT in memtype when CD=1 for KVM_X86_QUIRK_CD_NW_CLEARED Date: Fri, 14 Jul 2023 14:53:26 +0800 Message-Id: <20230714065326.20557-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771380714852924933 X-GMAIL-MSGID: 1771380714852924933 For KVM_X86_QUIRK_CD_NW_CLEARED is on, remove the IPAT (ignore PAT) bit in EPT memory types when cache is disabled and non-coherent DMA are present. To correctly emulate CR0.CD=1, UC + IPAT are required as memtype in EPT. However, as with commit fb279950ba02 ("KVM: vmx: obey KVM_QUIRK_CD_NW_CLEARED"), WB + IPAT are now returned to workaround a BIOS issue that guest MTRRs are enabled too late. Without this workaround, a super slow guest boot-up is expected during the pre-guest-MTRR-enabled period due to UC as the effective memory type for all guest memory. Absent emulating CR0.CD=1 with UC, it makes no sense to set IPAT when KVM is honoring the guest memtype. Removing the IPAT bit in this patch allows effective memory type to honor PAT values as well, as WB is the weakest memtype. It means if a guest explicitly claims UC as the memtype in PAT, the effective memory is UC instead of previous WB. If, for some unknown reason, a guest meets a slow boot-up issue with the removal of IPAT, it's desired to fix the blamed PAT in the guest. Besides, this patch is also a preparation patch for later fine-grained gfn zap when guest MTRRs are honored, because it allows zapping only non-WB ranges when CR0.CD toggles. BTW, returning guest MTRR type as if CR0.CD=0 is also not preferred because it still has to hardcode the MTRR type to WB during the pre-guest-MTRR-enabled period to workaround the slow guest boot-up issue (guest MTRR type when guest MTRRs are disabled is UC). In addition, it will make the quirk unnecessarily complexer . The change of removing IPAT has been verified with normal boot-up time on old OVMF of commit c9e5618f84b0cb54a9ac2d7604f7b7e7859b45a7 as well, dated back to Apr 14 2015. Suggested-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/kvm/vmx/vmx.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 0ecf4be2c6af..c1e93678cea4 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7548,8 +7548,6 @@ static int vmx_vm_init(struct kvm *kvm) static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { - u8 cache; - /* We wanted to honor guest CD/MTRR/PAT, but doing so could result in * memory aliases with conflicting memory types and sometimes MCEs. * We have to be careful as to what are honored and when. @@ -7576,11 +7574,10 @@ static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) if (kvm_read_cr0_bits(vcpu, X86_CR0_CD)) { if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) - cache = MTRR_TYPE_WRBACK; + return MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT; else - cache = MTRR_TYPE_UNCACHABLE; - - return (cache << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; + return (MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT) | + VMX_EPT_IPAT_BIT; } return kvm_mtrr_get_guest_memory_type(vcpu, gfn) << VMX_EPT_MT_EPTE_SHIFT; From patchwork Fri Jul 14 06:53:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120318 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2347261vqm; Fri, 14 Jul 2023 01:08:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlEl1Ga9lGJ4JnMXqFFFJnV84Lxzv5MsYMw4emynOsz35fKJLQHgKVh565bDKlvNXEFsCfEQ X-Received: by 2002:aa7:d499:0:b0:51e:472a:4fc with SMTP id b25-20020aa7d499000000b0051e472a04fcmr3032169edr.42.1689322120632; Fri, 14 Jul 2023 01:08:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689322120; cv=none; d=google.com; s=arc-20160816; b=mxUr3k+Pg2fpCK3AHPYPTKO+LLtQSJusGi8s1+skPjMqxLKm5oZQIGZbZQtoStq4nZ 676BoBTY1+VPMq/OEhF4WChKY0KlO57nbXzPWuFl5zrt9n0AM0UHJAFwoAXnG07tMGLJ jgFguixIfMeV6aKyPKxFimFh6s8RkhsrzqSBMzHiR+V7bCTbxZWP/T33LB+YB9oUD8hg dmgZvYnqYAr5geIVIUexiZar+dGCZi2OH+KbX3ZYvreRfQrMUYXJgm0FpIivmv/xTLw8 ItCnTfHmU5Ceo4NUfNOk1U1YzyM/Ztjx59OCOx+wYhmyktGoa5C9izAX/9JihGKx8bgS s0ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=F7IrtXdNao+EE52ZJt8IdpxStIYJFdwyXY4MYVnwZYA=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=GUYOlIQX1ytuY6P9O5xXFuYRS7K4ATegPbZSaCs52wriWgMIG4HJAQPAkJOWyBfI6Q WsfXnz2X1w1ngWAJyaCtSVtSVS9tkC8O2Toilqpgdwy+mEyePQV6qtsCEKs7cMjkIlcM 1ZilgFkvlANY2HcG9AATpp19snaw0d9cSxf/PEfBUBZZIdiHlLDeImpynPm9GigUonKR c1xd0Pz2BhQgmPTZo+Ps8NS8Ln7GyyU0xfVM7KE7xTfmIwF7mkvAEbImKFEULZ3WA7bM hUKdxD2ZoJRYLLHSV9XGonL0Hdtq8NUzGB5YF+TERHNL94fFmJa8Xsrz8J2Gdyx6tQJd dJrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bS3ASxC4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p13-20020a056402074d00b0051da08decbdsi9347964edy.636.2023.07.14.01.08.17; Fri, 14 Jul 2023 01:08:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bS3ASxC4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235429AbjGNHVE (ORCPT + 99 others); Fri, 14 Jul 2023 03:21:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235376AbjGNHUx (ORCPT ); Fri, 14 Jul 2023 03:20:53 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 828C33AA1; Fri, 14 Jul 2023 00:20:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319231; x=1720855231; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=1HwyvtI1AZxHBRdttaQhXcm1l9EvNjwuOZBpeJuX3yg=; b=bS3ASxC4++J+v0ACTGIQdW8A7KpwLmhnHudscuJA0FDsnkb8BbYMmNNH 8Cx/aX0vyBLlevN3cWx2jzn/OTYMWHlBvGx5oYmirH5PSAQnN6y8/taia LT2gKeSugQnz9cp0O+7FqXl4VTIotqnc/THtNgYkmL77s3tNAjlg4YYUE zpj/dizHdCV34ws2EyRdKoPc2NRM1TLQbbePuVyrFmFl+pPmA3VqFyAcf ks4FqbWfcRKksSuNboA57MrpSZwtX4jz/1MBL+5UDkcs4GZVY+fHRrfY2 KL+P3Pht2kKeHj/rM693iQ0ksmMhdl2GplmX+nqpTxwpGUa9O6xpko5UD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="396222339" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="396222339" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:20:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="1052956624" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="1052956624" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:20:29 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 08/12] KVM: x86: centralize code to get CD=1 memtype when guest MTRRs are honored Date: Fri, 14 Jul 2023 14:53:56 +0800 Message-Id: <20230714065356.20620-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771382631535149411 X-GMAIL-MSGID: 1771382631535149411 Centralize the code to get cache disabled memtype when guest MTRRs are honored. If a TDP honors guest MTRRs, it is required to call the provided API to get the memtype for CR0.CD=1. This is the preparation patch for later implementation of fine-grained gfn zap for CR0.CD toggles when guest MTRRs are honored. No functional change intended. Signed-off-by: Yan Zhao --- arch/x86/kvm/mtrr.c | 16 ++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 10 +++++----- arch/x86/kvm/x86.h | 2 ++ 3 files changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 3ce58734ad22..64c6daa659c8 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -721,3 +721,19 @@ bool kvm_mtrr_check_gfn_range_consistency(struct kvm_vcpu *vcpu, gfn_t gfn, return type == mtrr_default_type(mtrr_state); } + +/* + * this routine is supposed to be called when guest mtrrs are honored + */ +void kvm_honors_guest_mtrrs_get_cd_memtype(struct kvm_vcpu *vcpu, + u8 *type, bool *ipat) +{ + if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) { + *type = MTRR_TYPE_WRBACK; + *ipat = false; + } else { + *type = MTRR_TYPE_UNCACHABLE; + *ipat = true; + } +} +EXPORT_SYMBOL_GPL(kvm_honors_guest_mtrrs_get_cd_memtype); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c1e93678cea4..7fec1ee23b54 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7573,11 +7573,11 @@ static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; if (kvm_read_cr0_bits(vcpu, X86_CR0_CD)) { - if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_CD_NW_CLEARED)) - return MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT; - else - return (MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT) | - VMX_EPT_IPAT_BIT; + bool ipat; + u8 cache; + + kvm_honors_guest_mtrrs_get_cd_memtype(vcpu, &cache, &ipat); + return cache << VMX_EPT_MT_EPTE_SHIFT | (ipat ? VMX_EPT_IPAT_BIT : 0); } return kvm_mtrr_get_guest_memory_type(vcpu, gfn) << VMX_EPT_MT_EPTE_SHIFT; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 82e3dafc5453..e7733dc4dccc 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -313,6 +313,8 @@ int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); bool kvm_mtrr_check_gfn_range_consistency(struct kvm_vcpu *vcpu, gfn_t gfn, int page_num); +void kvm_honors_guest_mtrrs_get_cd_memtype(struct kvm_vcpu *vcpu, + u8 *type, bool *ipat); bool kvm_vector_hashing_enabled(void); void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code); int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type, From patchwork Fri Jul 14 06:54:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120290 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2340669vqm; Fri, 14 Jul 2023 00:53:28 -0700 (PDT) X-Google-Smtp-Source: APBJJlFYjws7moxNsWWzgvAUTDp1t1lAzmmTj87C115AcGVypeGq198SQK9qlZZEThLNeLOuP/j0 X-Received: by 2002:a19:500b:0:b0:4fb:774f:9a84 with SMTP id e11-20020a19500b000000b004fb774f9a84mr2584427lfb.13.1689321208107; Fri, 14 Jul 2023 00:53:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321208; cv=none; d=google.com; s=arc-20160816; b=yWH7nH4GezUCofLKJUP02uNmO0gHaZr6MhhH0bOm6iQUiQ88ByqPOurBXCWrfjusQd wnABYn+fRb4rRNl2GNoluHLEk1DpjhY25Gx9Q6EBeGH6mHnjpHh2mR+3F6hDspw/ln7e XOmRWRRRfFMiGuNj+6CWiq6j57pOfxXuHrZI3LEVrqUGpi/Xe6FM4AlFahYbgrns5mxV 603k3iy6N8bcPPGPaS91I+/dboYQqUa5P9Z72st8qvFkQHfoeBovSZdLQI19tjVv67DM eXGDPW32UbeRP0JrygNlGsMjl+lH2wOx5bCO/a1RoG7SW1mnA8Oi2Y0vhL0Br/jVJw6W 53aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=c8+4K2FVwSWBphLclr+CG9gvnO7WKBP6CiDFmQ55bUo=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=kwGWOXrRj3DyavxDwuajcShkv0OrQ0vW1oEu6lo4dgqB41yV4E5JEpQvuDxK3G+/sz pTiNE5swzztCm89tAqps5O4QHly4wUwImutayhGtd12ovATdfHccGyJxSjwf1fzh6Hk4 gJZ5Blk04RypABNgOl6N1KpHsOuyZPfjd2NLzNC41JIwrwdCPtlHmk/rgvIV9TyM6WCn QSJvetQlptYtU56PiLP/5RkWNdO5FZZAIfVX/HLWDYIGg34Evbv2pKcmcYE5UtthgwcZ Si+8rb1OznrPhjAeOvIQeG7HXr/QFcn/1EivPROrTcOPGjq1XTc2WDxUf3uRp84+kLKI dZvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FSqXVHHq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bo9-20020a0564020b2900b0051e1a616e32si8916375edb.89.2023.07.14.00.52.58; Fri, 14 Jul 2023 00:53:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FSqXVHHq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235131AbjGNHVy (ORCPT + 99 others); Fri, 14 Jul 2023 03:21:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235377AbjGNHVs (ORCPT ); Fri, 14 Jul 2023 03:21:48 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 095673586; Fri, 14 Jul 2023 00:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319295; x=1720855295; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9kHtvDYDM2RRR9tqpvSAXG8JNF/ruRm/IeraNi3+ZfI=; b=FSqXVHHqIicF/0FBBfP8z/885QPAMeZBVQoGCB8a3qGp+/J03lh8DBxp zmyr84gcjv40m4wZbLP1fHaqi3R5n5XyGq+kmbmT+yLBwdFCv3ZfXVwXg fyRVH4vnG+LptuuXUWymibJL3v4hMI9Rr0gazi7KeMaw5sf1wrbPHys2X GnI48UsR6+fIGaukhY85f9BeQNjnqwjrsgUMvdA5EWbUTqqDaVLJEJzpp 5I4E2b6MCy9thDLYmVzvTH5MDcQrp0NPVZIuNN3h4I8g2AWGGx471Ge84 YsEPNSXyEFtplk3YSTnwyC5qMqj62p11/DbQIjLhYHzUgIHRg8HLIsFpI Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="345727768" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="345727768" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:21:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="896316954" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="896316954" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:21:28 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 09/12] KVM: x86/mmu: serialize vCPUs to zap gfn when guest MTRRs are honored Date: Fri, 14 Jul 2023 14:54:54 +0800 Message-Id: <20230714065454.20688-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381674870736869 X-GMAIL-MSGID: 1771381674870736869 Serialize concurrent and repeated calls of kvm_zap_gfn_range() from every vCPU for CR0.CD toggles and MTRR updates when guest MTRRs are honored. During guest boot-up, if guest MTRRs are honored by TDP, TDP zaps are triggered several times by each vCPU for CR0.CD toggles and MTRRs updates. This will take unexpected longer CPU cycles because of the contention of kvm->mmu_lock. Therefore, introduce a mtrr_zap_list to remove duplicated zap and an atomic mtrr_zapping to allow only one vCPU to do the real zap work at one time. Cc: Yuan Yao Suggested-by: Sean Christopherson Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mtrr.c | 122 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 5 +- arch/x86/kvm/x86.h | 1 + 4 files changed, 130 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 28bd38303d70..8da1517a1513 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1444,6 +1444,10 @@ struct kvm_arch { */ #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1) struct kvm_mmu_memory_cache split_desc_cache; + + struct list_head mtrr_zap_list; + spinlock_t mtrr_zap_list_lock; + atomic_t mtrr_zapping; }; struct kvm_vm_stat { diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 64c6daa659c8..996a274cee40 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -25,6 +25,8 @@ #define IA32_MTRR_DEF_TYPE_FE (1ULL << 10) #define IA32_MTRR_DEF_TYPE_TYPE_MASK (0xff) +static void kvm_mtrr_zap_gfn_range(struct kvm_vcpu *vcpu, + gfn_t gfn_start, gfn_t gfn_end); static bool is_mtrr_base_msr(unsigned int msr) { /* MTRR base MSRs use even numbers, masks use odd numbers. */ @@ -341,7 +343,7 @@ static void update_mtrr(struct kvm_vcpu *vcpu, u32 msr) var_mtrr_range(var_mtrr_msr_to_range(vcpu, msr), &start, &end); } - kvm_zap_gfn_range(vcpu->kvm, gpa_to_gfn(start), gpa_to_gfn(end)); + kvm_mtrr_zap_gfn_range(vcpu, gpa_to_gfn(start), gpa_to_gfn(end)); } static bool var_mtrr_range_is_valid(struct kvm_mtrr_range *range) @@ -737,3 +739,121 @@ void kvm_honors_guest_mtrrs_get_cd_memtype(struct kvm_vcpu *vcpu, } } EXPORT_SYMBOL_GPL(kvm_honors_guest_mtrrs_get_cd_memtype); + +struct mtrr_zap_range { + gfn_t start; + /* end is exclusive */ + gfn_t end; + struct list_head node; +}; + +/* + * Add @range into kvm->arch.mtrr_zap_list and sort the list in + * "length" ascending + "start" descending order, so that + * ranges consuming more zap cycles can be dequeued later and their + * chances of being found duplicated are increased. + */ +static void kvm_add_mtrr_zap_list(struct kvm *kvm, struct mtrr_zap_range *range) +{ + struct list_head *head = &kvm->arch.mtrr_zap_list; + u64 len = range->end - range->start; + struct mtrr_zap_range *cur, *n; + bool added = false; + + spin_lock(&kvm->arch.mtrr_zap_list_lock); + + if (list_empty(head)) { + list_add(&range->node, head); + spin_unlock(&kvm->arch.mtrr_zap_list_lock); + return; + } + + list_for_each_entry_safe(cur, n, head, node) { + u64 cur_len = cur->end - cur->start; + + if (len < cur_len) + break; + + if (len > cur_len) + continue; + + if (range->start > cur->start) + break; + + if (range->start < cur->start) + continue; + + /* equal len & start, no need to add */ + added = true; + kfree(range); + break; + } + + if (!added) + list_add_tail(&range->node, &cur->node); + + spin_unlock(&kvm->arch.mtrr_zap_list_lock); +} + +static void kvm_zap_mtrr_zap_list(struct kvm *kvm) +{ + struct list_head *head = &kvm->arch.mtrr_zap_list; + struct mtrr_zap_range *cur = NULL; + + spin_lock(&kvm->arch.mtrr_zap_list_lock); + + while (!list_empty(head)) { + u64 start, end; + + cur = list_first_entry(head, typeof(*cur), node); + start = cur->start; + end = cur->end; + list_del(&cur->node); + kfree(cur); + spin_unlock(&kvm->arch.mtrr_zap_list_lock); + + kvm_zap_gfn_range(kvm, start, end); + + spin_lock(&kvm->arch.mtrr_zap_list_lock); + } + + spin_unlock(&kvm->arch.mtrr_zap_list_lock); +} + +static void kvm_zap_or_wait_mtrr_zap_list(struct kvm *kvm) +{ + if (atomic_cmpxchg_acquire(&kvm->arch.mtrr_zapping, 0, 1) == 0) { + kvm_zap_mtrr_zap_list(kvm); + atomic_set_release(&kvm->arch.mtrr_zapping, 0); + return; + } + + while (atomic_read(&kvm->arch.mtrr_zapping)) + cpu_relax(); +} + +static void kvm_mtrr_zap_gfn_range(struct kvm_vcpu *vcpu, + gfn_t gfn_start, gfn_t gfn_end) +{ + struct mtrr_zap_range *range; + + range = kmalloc(sizeof(*range), GFP_KERNEL_ACCOUNT); + if (!range) + goto fail; + + range->start = gfn_start; + range->end = gfn_end; + + kvm_add_mtrr_zap_list(vcpu->kvm, range); + + kvm_zap_or_wait_mtrr_zap_list(vcpu->kvm); + return; + +fail: + kvm_zap_gfn_range(vcpu->kvm, gfn_start, gfn_end); +} + +void kvm_honors_guest_mtrrs_zap_on_cd_toggle(struct kvm_vcpu *vcpu) +{ + return kvm_mtrr_zap_gfn_range(vcpu, gpa_to_gfn(0), gpa_to_gfn(~0ULL)); +} diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 32cc8bfaa5f1..bb79154cf465 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -943,7 +943,7 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon if (((cr0 ^ old_cr0) & X86_CR0_CD) && kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) - kvm_zap_gfn_range(vcpu->kvm, 0, ~0ULL); + kvm_honors_guest_mtrrs_zap_on_cd_toggle(vcpu); } EXPORT_SYMBOL_GPL(kvm_post_set_cr0); @@ -12310,6 +12310,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm->arch.guest_can_read_msr_platform_info = true; kvm->arch.enable_pmu = enable_pmu; + spin_lock_init(&kvm->arch.mtrr_zap_list_lock); + INIT_LIST_HEAD(&kvm->arch.mtrr_zap_list); + #if IS_ENABLED(CONFIG_HYPERV) spin_lock_init(&kvm->arch.hv_root_tdp_lock); kvm->arch.hv_root_tdp = INVALID_PAGE; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index e7733dc4dccc..56d8755b2560 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -315,6 +315,7 @@ bool kvm_mtrr_check_gfn_range_consistency(struct kvm_vcpu *vcpu, gfn_t gfn, int page_num); void kvm_honors_guest_mtrrs_get_cd_memtype(struct kvm_vcpu *vcpu, u8 *type, bool *ipat); +void kvm_honors_guest_mtrrs_zap_on_cd_toggle(struct kvm_vcpu *vcpu); bool kvm_vector_hashing_enabled(void); void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code); int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type, From patchwork Fri Jul 14 06:55:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120309 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2345661vqm; Fri, 14 Jul 2023 01:05:24 -0700 (PDT) X-Google-Smtp-Source: APBJJlERiRgxr3jb0c4Xgt0sHq4MWl8qoRDbO3s6C3qLQYAr8F0Z69NC7f8mlVGTXElg4F9GdT6U X-Received: by 2002:a05:6808:1808:b0:3a3:a8ce:c634 with SMTP id bh8-20020a056808180800b003a3a8cec634mr4703197oib.53.1689321923867; Fri, 14 Jul 2023 01:05:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321923; cv=none; d=google.com; s=arc-20160816; b=Xkx+6AJLKHC5Sc4IS9QC70Ik1HXjkRqVTnTdeCjoAWYGNZpt1d4l5Qw1tXaonsDDIk af7YOowr5bzyJILAWHp/VgjEYj60EpruEChb5jZw0j4dVPckj4AyWeVGIgV/FfsS+oPI nxqriNJszNWJeV6Nv/kcl0iMrQEDN3w/hVZGnOIUqOj+ahmf0UTMD/oJds7pEnOTWq/B o9dzv1jM4itxdyXZWY+TJumY1tfP3V9MPiu0Za9R9+mnx4eDRIKW28R674e9xw/RTPyj sJFsf5h5u19Z0UIA7tgagsewbdDpi+ueJSJ3LR3NZzysiQTV+6mL7X1XIMf47r120xE3 ZoOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=DABK1sQ7IXKhvQA0Sz3Wb3i8D+gjpjkMm6Lz7S+jmFs=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=lRP+V/Y4Ww/G7PnDcKkiyHQQ8l+tx79zsf6JMLfEfBTnTyI4tlymr1dPs9pmXeOvnR nyObe4/dCiUu88VAE5AzCTqyhbRfbyBx8x3kWXxcFVCwDznDe/y8sycxdmQrQLv9xo4Q uFeozj2hoL1qbjEROemknysdP2HvnogWeImiIjADKWPR1Qjikz5J7IMi8IuUnZOqVvnR 7Pqmp6EbiRxZQDgWqu1K6l31vTlrZX2rxv1AO+SwJoitSXC8vTbipVZ/zdKzW3vICE1J oFBsu8skGyVSapJd+rCFnVLF5tPvSCA9Dx/u4aR7qrLyX1g1NjZ0/OpmyvkYIETB/i1f io2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="HCu/19A1"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id iq12-20020a17090afb4c00b00262e93858f6si878605pjb.58.2023.07.14.01.05.10; Fri, 14 Jul 2023 01:05:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="HCu/19A1"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235377AbjGNHWS (ORCPT + 99 others); Fri, 14 Jul 2023 03:22:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235391AbjGNHWM (ORCPT ); Fri, 14 Jul 2023 03:22:12 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 773D635B0; Fri, 14 Jul 2023 00:22:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319326; x=1720855326; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JwujvIAhHDIhsRscSy/VyNy1PdDMfjp1hDs4yRNRsl0=; b=HCu/19A1wtQk9Wgv4lBPCu5JKD50QZSEdR6tNF9K/1pswnJ8zEXTDXaj 6UoEGtl8mSlajCUv32g3OOukapTw/2v8KAj89tVhbT7dqWfwI58l3bf78 DtIzMcQ13OHKsfhdJQwgja+ubBCL8l21pLliJuie8+8f2syXAqWyvDri3 HALcR2dh2veBGpVosNFYaNlADolEAHEmmhJa0YKSRq/4u6Bh2zLwOw+H5 EsqqnzjqrcB0o2RY0LQWcwTlQC2vxT+Ugyqhl1dQk9eq2pCPLXVQPlJQ8 ICKdzv/xpVzPfsR6IGTVBChrhkxpRH3Hci6R9AZShMnuZ8TNiZPG1Q+q/ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="362877539" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="362877539" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:22:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="835937381" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="835937381" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:22:03 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 10/12] KVM: x86/mmu: fine-grained gfn zap when guest MTRRs are honored Date: Fri, 14 Jul 2023 14:55:30 +0800 Message-Id: <20230714065530.20748-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771382425565446267 X-GMAIL-MSGID: 1771382425565446267 When guest MTRRs are honored and CR0.CD toggles, rather than blindly zap everything, find out fine-grained ranges to zap according to guest MTRRs. Fine-grained and precise zap ranges allow reduced traversal footprint during zap and increased chances for concurrent vCPUs to find and skip duplicated ranges to zap. Opportunistically fix a typo in a nearby comment. Suggested-by: Sean Christopherson Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yan Zhao --- arch/x86/kvm/mtrr.c | 164 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 162 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 996a274cee40..9fdbdbf874a8 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -179,7 +179,7 @@ static struct fixed_mtrr_segment fixed_seg_table[] = { { .start = 0xc0000, .end = 0x100000, - .range_shift = 12, /* 12K */ + .range_shift = 12, /* 4K */ .range_start = 24, } }; @@ -747,6 +747,19 @@ struct mtrr_zap_range { struct list_head node; }; +static void kvm_clear_mtrr_zap_list(struct kvm *kvm) +{ + struct list_head *head = &kvm->arch.mtrr_zap_list; + struct mtrr_zap_range *tmp, *n; + + spin_lock(&kvm->arch.mtrr_zap_list_lock); + list_for_each_entry_safe(tmp, n, head, node) { + list_del(&tmp->node); + kfree(tmp); + } + spin_unlock(&kvm->arch.mtrr_zap_list_lock); +} + /* * Add @range into kvm->arch.mtrr_zap_list and sort the list in * "length" ascending + "start" descending order, so that @@ -795,6 +808,67 @@ static void kvm_add_mtrr_zap_list(struct kvm *kvm, struct mtrr_zap_range *range) spin_unlock(&kvm->arch.mtrr_zap_list_lock); } +/* + * Fixed ranges are only 256 pages in total. + * After balancing between reducing overhead of zap multiple ranges + * and increasing chances of finding duplicated ranges, + * just add fixed mtrr ranges as a whole to the mtrr zap list + * if memory type of one of them is not the specified type. + */ +static int prepare_zaplist_fixed_mtrr_of_non_type(struct kvm_vcpu *vcpu, u8 type) +{ + struct kvm_mtrr *mtrr_state = &vcpu->arch.mtrr_state; + struct mtrr_zap_range *range; + int index, seg_end; + u8 mem_type; + + for (index = 0; index < KVM_NR_FIXED_MTRR_REGION; index++) { + mem_type = mtrr_state->fixed_ranges[index]; + + if (mem_type == type) + continue; + + range = kmalloc(sizeof(*range), GFP_KERNEL_ACCOUNT); + if (!range) + return -ENOMEM; + + seg_end = ARRAY_SIZE(fixed_seg_table) - 1; + range->start = gpa_to_gfn(fixed_seg_table[0].start); + range->end = gpa_to_gfn(fixed_seg_table[seg_end].end); + kvm_add_mtrr_zap_list(vcpu->kvm, range); + break; + } + return 0; +} + +/* + * Add var mtrr ranges to the mtrr zap list + * if its memory type does not equal to type + */ +static int prepare_zaplist_var_mtrr_of_non_type(struct kvm_vcpu *vcpu, u8 type) +{ + struct kvm_mtrr *mtrr_state = &vcpu->arch.mtrr_state; + struct mtrr_zap_range *range; + struct kvm_mtrr_range *tmp; + u8 mem_type; + + list_for_each_entry(tmp, &mtrr_state->head, node) { + mem_type = tmp->base & 0xff; + if (mem_type == type) + continue; + + range = kmalloc(sizeof(*range), GFP_KERNEL_ACCOUNT); + if (!range) + return -ENOMEM; + + var_mtrr_range(tmp, &range->start, &range->end); + range->start = gpa_to_gfn(range->start); + range->end = gpa_to_gfn(range->end); + kvm_add_mtrr_zap_list(vcpu->kvm, range); + } + return 0; +} + static void kvm_zap_mtrr_zap_list(struct kvm *kvm) { struct list_head *head = &kvm->arch.mtrr_zap_list; @@ -853,7 +927,93 @@ static void kvm_mtrr_zap_gfn_range(struct kvm_vcpu *vcpu, kvm_zap_gfn_range(vcpu->kvm, gfn_start, gfn_end); } +/* + * Zap SPTEs when guest MTRRs are honored and CR0.CD toggles + * in fine-grained way according to guest MTRRs. + * As guest MTRRs are per-vCPU, they are unchanged across this function. + * + * when CR0.CD=1, TDP memtype is WB or UC + IPAT; + * when CR0.CD=0, TDP memtype is determined by guest MTRRs. + * + * On CR0.CD toggles, as guest MTRRs remain unchanged, + * - if old memtype are new memtype are equal, nothing needs to do; + * - if guest default MTRR type equals to memtype in CR0.CD=1, + * only MTRR ranges of non-default-memtype are required to be zapped. + * - if guest default MTRR type !equals to memtype in CR0.CD=1, + * everything is zapped because memtypes for almost all guest memory + * are out-dated. + * _____________________________________________________________________ + *| quirk on | CD=1 to CD=0 | CD=0 to CD=1 | + *| | old memtype = WB | new memtype = WB | + *|----------------------|----------------------|-----------------------| + *| MTRR enabled | new memtype = | old memtype = | + *| | guest MTRR type | guest MTRR type | + *| ------------------|----------------------|-----------------------| + *| | if default MTRR | zap non-WB guest | zap non-WB guest | + *| | type == WB | MTRR ranges | MTRR ranges | + *| |-----------------|----------------------|-----------------------| + *| | if default MTRR | zap all | zap all | + *| | type != WB | as almost all guest MTRR ranges are non-WB | + *|----------------------|----------------------------------------------| + *| MTRR disabled | new memtype = UC | old memtype = UC | + *| (w/ FEATURE_MTRR) | zap all | zap all | + *|----------------------|----------------------|-----------------------| + *| MTRR disabled | new memtype = WB | old memtype = WB | + *| (w/o FEATURE_MTRR) | do nothing | do nothing | + *|______________________|______________________|_______________________| + * + * _____________________________________________________________________ + *| quirk off | CD=1 to CD=0 | CD=0 to CD=1 | + *| | old memtype = UC + IPAT | new memtype = UC + IPAT | + *|---------------|--------------------------|--------------------------| + *| MTRR enabled | new memtype = guest MTRR | old memtype = guest MTRR | + *| | type (!= UC + IPAT) | type (!= UC + IPAT) | + *| | zap all | zap all | + *|---------------|------------------------- |--------------------------| + *| MTRR disabled | new memtype = UC | old memtype = UC | + *| (w/ | (!= UC + IPAT) | (!= UC + IPAT) | + *| FEATURE_MTRR) | zap all | zap all | + *|---------------|--------------------------|--------------------------| + *| MTRR disabled | new memtype = WB | old memtype = WB | + *| (w/o | (!= UC + IPAT) | (!= UC + IPAT) | + *| FEATURE_MTRR) | zap all | zap all | + *|_______________|__________________________|__________________________| + * + */ void kvm_honors_guest_mtrrs_zap_on_cd_toggle(struct kvm_vcpu *vcpu) { - return kvm_mtrr_zap_gfn_range(vcpu, gpa_to_gfn(0), gpa_to_gfn(~0ULL)); + struct kvm_mtrr *mtrr_state = &vcpu->arch.mtrr_state; + bool mtrr_enabled = mtrr_is_enabled(mtrr_state); + u8 default_mtrr_type; + bool cd_ipat; + u8 cd_type; + + kvm_honors_guest_mtrrs_get_cd_memtype(vcpu, &cd_type, &cd_ipat); + + default_mtrr_type = mtrr_enabled ? mtrr_default_type(mtrr_state) : + mtrr_disabled_type(vcpu); + + if (cd_type != default_mtrr_type || cd_ipat) + return kvm_mtrr_zap_gfn_range(vcpu, gpa_to_gfn(0), gpa_to_gfn(~0ULL)); + + /* + * If mtrr is not enabled, it will go to zap all above if the default + * type does not equal to cd_type; + * Or it has no need to zap if the default type equals to cd_type. + */ + if (mtrr_enabled) { + if (prepare_zaplist_fixed_mtrr_of_non_type(vcpu, default_mtrr_type)) + goto fail; + + if (prepare_zaplist_var_mtrr_of_non_type(vcpu, default_mtrr_type)) + goto fail; + + kvm_zap_or_wait_mtrr_zap_list(vcpu->kvm); + } + return; +fail: + kvm_clear_mtrr_zap_list(vcpu->kvm); + /* resort to zapping all on failure*/ + kvm_zap_gfn_range(vcpu->kvm, gpa_to_gfn(0), gpa_to_gfn(~0ULL)); + return; } From patchwork Fri Jul 14 06:56:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120281 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2339556vqm; Fri, 14 Jul 2023 00:50:32 -0700 (PDT) X-Google-Smtp-Source: APBJJlHGYf/qbsX/qNMT/V8l/Hcrtg2oN0ZNL4whYiDJ+BgzlDVKqEkaGTmW0YYetnSTu2Ub1Xch X-Received: by 2002:aa7:c90f:0:b0:51e:f83:6de6 with SMTP id b15-20020aa7c90f000000b0051e0f836de6mr4225961edt.16.1689321031843; Fri, 14 Jul 2023 00:50:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321031; cv=none; d=google.com; s=arc-20160816; b=pXtzQhp36iZYfaVrL+KDsnCkaYR2MjWMJFNWZF52DQY/0vQWdATv8sccl7vUm4C/k2 D7D9XegNEBDYCb/fn24oms1OGcdrKZHkdvhQx42QcN6pPhKV8EeoEE/2nx7FfgqU5wKD T8gFF/Nerrq/7y+EJM9Ol2WeIh5C5xdwqEjn6idKloVIkIz0OlDBtsYGAryMgXdJPXCq LPz7Cb3e4yJyd+H7DEoDYRgc6M7dwxDxdgLBwZ4+tnvG5fPMxGrszsjsiXyWkPyZIA0e gksRmcbcpGHmNWszky8YjbnBch2vosxrNCbQprnuirUel3gQfeyNaqztwwaw/ZE+zvcf ROqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=qWAdaoGuameuODGLfZXmVLvKHCw/myGhCBz+sMLKGFI=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=fUdlXH2klrXVVxif4wDd9zrwEd09ucBrrFuKJAoLtCSxL2/8h9AMXMoPOoc1p8GknF TE2vWgerYvFX9rkE1lzu8b6mmUqu3pI79rgAJdZ6PLRMPgADsMk8Njyq4bi46Q05yfyJ C2PdsLWoQOvjqhZ/zvbKD6S628pTpuv3+1dV/xUSYXMEU+ifoWuJYnWqyJdQsiVdDVKk 02XmC76U5qNd80ZxMKE0naN2oFEeuTx2oecWXrtQuRyD2DzpV/CJoT5B/tzowd3Uqb/w m1/G06yqC4UG+nbZvhksBI4Pddp3aeXXqs0dNhWUyk660yhg2Tzk+UamvrvO88E+f56w CgqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=m6TyDK46; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w7-20020aa7da47000000b0051df3b287b3si1466512eds.49.2023.07.14.00.50.07; Fri, 14 Jul 2023 00:50:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=m6TyDK46; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234267AbjGNHW6 (ORCPT + 99 others); Fri, 14 Jul 2023 03:22:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235395AbjGNHWv (ORCPT ); Fri, 14 Jul 2023 03:22:51 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A7C0358C; Fri, 14 Jul 2023 00:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319359; x=1720855359; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=dXHUrVqTsBzck553w5GbptEVC8cgKTaiRMcUQpmhAWg=; b=m6TyDK4673WDnKHhPDlm+0HlRumfU4GdMCQAx//Vdxp2ZzK+G5w2nPCt /8uyqiYBRlbfuQblzkUvHFMj4vHxlmUC9Hts5kWb8D0xwR2U6InA/K4pX iOSHpd5OBCU7QanafFHAIf1cgvinTeiioU2IhW3SiA5Hh7nZcecZhyUoe yGlI53CaArmQJZi83hsNei5A88fBvqKKhkxXBf0deaBeK9xxQWg68mEPB 6HnsBihHhJ0KT8RcpaSxu9164+pAw/WXm1flVsn/8r/pLw+wTNhAHpAr/ BqpOrevwZAAb24QgPdXTR/C+Y4xlrauyg+HJ3yRUafoRuiFe+OevqFrCk g==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="345727883" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="345727883" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:22:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="896317393" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="896317393" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:22:36 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 11/12] KVM: x86/mmu: split a single gfn zap range when guest MTRRs are honored Date: Fri, 14 Jul 2023 14:56:02 +0800 Message-Id: <20230714065602.20805-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381490514379119 X-GMAIL-MSGID: 1771381490514379119 Split a single gfn zap range (specifially range [0, ~0UL)) to smaller ranges according to current memslot layout when guest MTRRs are honored. Though vCPUs have been serialized to perform kvm_zap_gfn_range() for MTRRs updates and CR0.CD toggles, contention caused rescheduling cost is still huge when there're concurrent page fault holding mmu_lock for read. Split a single huge zap range according to the actual memslot layout can reduce unnecessary transversal and yielding cost in tdp mmu. Also, it can increase the chances for larger ranges to find existing ranges to zap in zap list. Signed-off-by: Yan Zhao --- arch/x86/kvm/mtrr.c | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 9fdbdbf874a8..00e98dfc4b0d 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -909,21 +909,44 @@ static void kvm_zap_or_wait_mtrr_zap_list(struct kvm *kvm) static void kvm_mtrr_zap_gfn_range(struct kvm_vcpu *vcpu, gfn_t gfn_start, gfn_t gfn_end) { + int idx = srcu_read_lock(&vcpu->kvm->srcu); + const struct kvm_memory_slot *memslot; struct mtrr_zap_range *range; + struct kvm_memslot_iter iter; + struct kvm_memslots *slots; + gfn_t start, end; + int i; - range = kmalloc(sizeof(*range), GFP_KERNEL_ACCOUNT); - if (!range) - goto fail; - - range->start = gfn_start; - range->end = gfn_end; - - kvm_add_mtrr_zap_list(vcpu->kvm, range); + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + slots = __kvm_memslots(vcpu->kvm, i); + kvm_for_each_memslot_in_gfn_range(&iter, slots, gfn_start, gfn_end) { + memslot = iter.slot; + start = max(gfn_start, memslot->base_gfn); + end = min(gfn_end, memslot->base_gfn + memslot->npages); + if (WARN_ON_ONCE(start >= end)) + continue; + + range = kmalloc(sizeof(*range), GFP_KERNEL_ACCOUNT); + if (!range) + goto fail; + + range->start = start; + range->end = end; + + /* + * Redundent ranges in different address space will be + * removed in kvm_add_mtrr_zap_list(). + */ + kvm_add_mtrr_zap_list(vcpu->kvm, range); + } + } + srcu_read_unlock(&vcpu->kvm->srcu, idx); kvm_zap_or_wait_mtrr_zap_list(vcpu->kvm); return; fail: + srcu_read_unlock(&vcpu->kvm->srcu, idx); kvm_zap_gfn_range(vcpu->kvm, gfn_start, gfn_end); } From patchwork Fri Jul 14 06:56:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhao X-Patchwork-Id: 120287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp2340556vqm; Fri, 14 Jul 2023 00:53:06 -0700 (PDT) X-Google-Smtp-Source: APBJJlEnKbZgXhV9Z+68CO+lmlU7XDMTRgeP+5pgGHuXh9Htt4bYUcupV0ChsC6MRVFwjzW6ZgLv X-Received: by 2002:a17:907:ea6:b0:96a:2dd7:2ee0 with SMTP id ho38-20020a1709070ea600b0096a2dd72ee0mr2759641ejc.5.1689321186369; Fri, 14 Jul 2023 00:53:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689321186; cv=none; d=google.com; s=arc-20160816; b=wxM9VP0wpYTd4EhdUBdjXi/wqsNMXpGSNB0o39IVBbruKJbyD1TPLjIxFLSKQhGZuO OwmEOUWE1tUG2+ixZDHrXeVO7w9vYrVtbVNmJz+lylG3jbEiAn4wsdMuO55GZhozz+JP 1Ag50J5bxf6OvfKhBui47p7brG/ZKUhzW8IH95pfjc5+XI3NmYP2SXG0641GiPSsX+Hy 3Fi/BefhPCi7ZNORk6bDx01LT6GUdTeXPOwoUwQxx68j57Ycb8ia78wlz8gG62eHjgw9 u2yhQxbY8Y73hiIovkE8kDIwliiWRQ/VNaAiuY8hkpkvInQeZ1sersClN4FxObN1wYVu kz8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=kjQAVCleEFlUBs/DBr/mKj9A07LfCezhAgaysk7ARMQ=; fh=I6wrbN01dP2yWc/eoJu3Jm+A24tUYxebxIQloyExA+I=; b=Q2MuExMY7cSWObb+/Ga8QM8nd5GJuoHvCqGhgAP2cbiKU8XFwhjoLkh/2wHcoIEFul XmB1CEYjClvrn0S2vdO685sTaKpnUe+mcZI0X0XMig2XBnGp49LWAZotTJ52iHL7Gtrj YJMO/rH0wKv6V9qyiWQ4/8HNNK0+s6JQqF1jLhOz71jJWfB6eiK3+3hq46OnKFibIjmT XWZ77klR/0vyv7wWG6++B9jOTxES6ICOVzJS3mAxM4dekIREBHdnBKMxlHi4RyOuVUJJ 5lK3WS/n0RmoCpnTv0KAmqAIoX6B/wOIYGcH4ppBnwIRJkzF8uB4yNDd+NrWfaeBaxAb b5rw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RiGjdZwL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m18-20020a1709066d1200b00993860a6d3asi8485717ejr.510.2023.07.14.00.52.41; Fri, 14 Jul 2023 00:53:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RiGjdZwL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235102AbjGNHXQ (ORCPT + 99 others); Fri, 14 Jul 2023 03:23:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234757AbjGNHXJ (ORCPT ); Fri, 14 Jul 2023 03:23:09 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA3DB30E9; Fri, 14 Jul 2023 00:23:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689319387; x=1720855387; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=IHJc+z0SmVKAVyBpoFP/KQLvug9IHbTWEWnO52mZneQ=; b=RiGjdZwLFAAm8zo7LVTJL49npMPpVeiEDvf0i3OfD51MLdJsj0nBpGCF YXYu7530hu5ijYQE3WKwIHD3GQMAu9aZDcykRq9OxKqAEeB/Hw42Z+r2a WptAta0aBmfDVkKFCko2SdSFGEiYwUmYYteL4sAEII90wS5MQpGnjtlkX akaGsQ9gh4/wUshLHVawDzfL/oj9z6S41nTdNZ76SxovmRMxbEuoLF4pM A6oxUGRBtncmvA8b3Euiek5JLhEHHkVSAo4QAiyTH2jLTntqv7M2ermz0 oCiMgf4uL7A5oHI1UOl+v3MkrWzCVhYbetfHRUTXKaOTqWB+bznOQC7X+ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="345727983" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="345727983" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:23:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="896317519" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="896317519" Received: from yzhao56-desk.sh.intel.com ([10.239.159.62]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2023 00:23:04 -0700 From: Yan Zhao To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, seanjc@google.com, chao.gao@intel.com, kai.huang@intel.com, robert.hoo.linux@gmail.com, yuan.yao@linux.intel.com, Yan Zhao Subject: [PATCH v4 12/12] KVM: x86/mmu: convert kvm_zap_gfn_range() to use shared mmu_lock in TDP MMU Date: Fri, 14 Jul 2023 14:56:31 +0800 Message-Id: <20230714065631.20869-1-yan.y.zhao@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230714064656.20147-1-yan.y.zhao@intel.com> References: <20230714064656.20147-1-yan.y.zhao@intel.com> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771381652545977613 X-GMAIL-MSGID: 1771381652545977613 Convert kvm_zap_gfn_range() from holding mmu_lock for write to holding for read in TDP MMU and allow zapping of non-leaf SPTEs of level <= 1G. TLB flushes are executed/requested within tdp_mmu_zap_spte_atomic() guarded by RCU lock. GFN zap can be super slow if mmu_lock is held for write when there are contentions. In worst cases, huge cpu cycles are spent on yielding GFN by GFN, i.e. the loop of "check and flush tlb -> drop rcu lock -> drop mmu_lock -> cpu_relax() -> take mmu_lock -> take rcu lock" are entered for every GFN. Contentions can either from concurrent zaps holding mmu_lock for write or from tdp_mmu_map() holding mmu_lock for read. After converting to hold mmu_lock for read, there will be less contentions detected and retaking mmu_lock for read is also faster. There's no need to flush TLB before dropping mmu_lock when there're contentions as SPTEs have been zapped atomically and TLBs are flushed/flush requested immediately within RCU lock. In order to reduce TLB flush count, non-leaf SPTEs not greater than 1G level are allowed to be zapped if their ranges are fully covered in the gfn zap range. Signed-off-by: Yan Zhao --- arch/x86/kvm/mmu/mmu.c | 14 +++++++---- arch/x86/kvm/mmu/tdp_mmu.c | 50 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 1 + 3 files changed, 60 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 7f52bbe013b3..1fa2a0a3fc9b 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6310,15 +6310,19 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end); + if (flush) + kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start); + if (tdp_mmu_enabled) { + write_unlock(&kvm->mmu_lock); + read_lock(&kvm->mmu_lock); + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) - flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start, - gfn_end, true, flush); + kvm_tdp_mmu_zap_gfn_range(kvm, i, gfn_start, gfn_end); + read_unlock(&kvm->mmu_lock); + write_lock(&kvm->mmu_lock); } - if (flush) - kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start); - kvm_mmu_invalidate_end(kvm, 0, -1ul); write_unlock(&kvm->mmu_lock); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 512163d52194..2ad18275b643 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -888,6 +888,56 @@ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end, return flush; } +static void zap_gfn_range_atomic(struct kvm *kvm, struct kvm_mmu_page *root, + gfn_t start, gfn_t end) +{ + struct tdp_iter iter; + + end = min(end, tdp_mmu_max_gfn_exclusive()); + + lockdep_assert_held_read(&kvm->mmu_lock); + + rcu_read_lock(); + + for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { +retry: + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) + continue; + + if (!is_shadow_present_pte(iter.old_spte)) + continue; + + /* + * As also documented in tdp_mmu_zap_root(), + * KVM must be able to zap a 1gb shadow page without + * inducing a stall to allow in-place replacement with a 1gb hugepage. + */ + if (iter.gfn < start || + iter.gfn + KVM_PAGES_PER_HPAGE(iter.level) > end || + iter.level > KVM_MAX_HUGEPAGE_LEVEL) + continue; + + /* Note, a successful atomic zap also does a remote TLB flush. */ + if (tdp_mmu_zap_spte_atomic(kvm, &iter)) + goto retry; + } + + rcu_read_unlock(); +} + +/* + * Zap all SPTEs for the range of gfns, [start, end), for all roots with + * shared mmu lock in atomic way. + * TLB flushs are performed within the rcu lock. + */ +void kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start, gfn_t end) +{ + struct kvm_mmu_page *root; + + for_each_valid_tdp_mmu_root_yield_safe(kvm, root, as_id, true) + zap_gfn_range_atomic(kvm, root, start, end); +} + void kvm_tdp_mmu_zap_all(struct kvm *kvm) { struct kvm_mmu_page *root; diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 0a63b1afabd3..90856bd7a2fd 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -22,6 +22,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t end, bool can_yield, bool flush); +void kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start, gfn_t end); bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp); void kvm_tdp_mmu_zap_all(struct kvm *kvm); void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm);