From patchwork Fri Feb 3 19:28:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 52624 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1031851wrn; Fri, 3 Feb 2023 12:00:35 -0800 (PST) X-Google-Smtp-Source: AK7set/oJq58fKQvqcN7mqHAqM8uBFw8Vf7/XJcDEv+QnuAyh+Fv6NPrmcRdsP84cNVeeXGwXHr8 X-Received: by 2002:a17:902:e3d2:b0:196:33dc:98ab with SMTP id r18-20020a170902e3d200b0019633dc98abmr8985328ple.20.1675454434999; Fri, 03 Feb 2023 12:00:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675454434; cv=none; d=google.com; s=arc-20160816; b=l5to96gVZSENfKld14nu1ykIkmBjgc+IdmfUL+1vN1tSwc3OIzB1h2rTqc+2py6TTA rswQEbjss55VkXWZRviRGgldtPfKEHOYlrm3X8zXVqmHcilC5riWr+rPcs3TKaacFrB1 ZjdYCqvMke5ysNeW3wCk4HNJh0NrFRcmU2O06N4wcVF3QcNmTE3mQSnJgqRWL5sekjjG J7Rp1uHLZgZ/mqG6EUjAEYOy+Y7DwKd4x96Bnfnho2eqnUppubo5E6Uv1uI9NqacQykf KsI5RExbislQmi04WO1QIwJYtvNcI5TKIuGJutdrXHp3YVvWU0unEpqkHS2mNpXKJ0M0 dPcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=JnRekf2u9xlodJ6kWv+4sMoTYGQkhV0MHFbmGB0K31Y=; b=wrJrBroHqaT24UEb/+n147I3M2y7pXz9Szxuc5O2cZRqmzGXIN86XLlPvhr9yjbaXG nwJAMiJThBQwAnVn33KNMFn3DBpyoRa1VL7vwe6XPzxnQ8A1gMH+/Y5k5ra9SIP8A6Dq Ws8NT/jDM/1EH8Kyn69AItFas5lYY34PKpc6snjomGEcj08/wMK8OIz641bpBlSAQ3BO A2adpW89s6gV5M8Y0FjqqcaTFTG1l9J0T3lOs9TwU5P4O+U49kUxOG7LVZXSHPG1coqx 2vDYD33alcax61Ks6sTrdKNc/8HNwVQ7SbOaQ8VKqB2I8IZZpYD/Ll2p3sP7PgtmV0+e vk+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Zi6H4ARQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id be3-20020a170902aa0300b0019648e516dfsi3068487plb.352.2023.02.03.12.00.17; Fri, 03 Feb 2023 12:00:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Zi6H4ARQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233143AbjBCT2k (ORCPT + 99 others); Fri, 3 Feb 2023 14:28:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233294AbjBCT2g (ORCPT ); Fri, 3 Feb 2023 14:28:36 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4482DA913D for ; Fri, 3 Feb 2023 11:28:31 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id y9-20020a1709027c8900b00195e237dc8bso3009683pll.13 for ; Fri, 03 Feb 2023 11:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JnRekf2u9xlodJ6kWv+4sMoTYGQkhV0MHFbmGB0K31Y=; b=Zi6H4ARQow0EtXg0mUHG9UGNsiViGbeZO/GDNJqScT1LjoaHQG/12s0iu9hLuYReeH t9C1ND+dlehha7qxQlRIP3e3fttV6FiwBMPowHok+6ipPXZJMeqYAk3AS0hnbk34FRk4 GRrc3WZIrvEOJAtBfEjk3pbqjV7T/JZXj4XsXMpgodFXBxvGflCKR88pCDoLsqjU9Hx/ rfdTnpdHymBjyFU2lVEsP8hzxnLaRnQo2GLObrDxfiv76bQJlDJkKVrHOsVRhEQ1xoLC bwd49pgcBDx5UlmZRnmSn4IJ/BG8CEufoOwm2BZvpbAUTAPsciIep4wD/BTUu7bR74p4 2ASA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JnRekf2u9xlodJ6kWv+4sMoTYGQkhV0MHFbmGB0K31Y=; b=XO1iRNwgZzuz63VdqXG2q+jEjPuOLuoGrQ5SyW3KXWvhSHLPJqwBLIOtLb7cXK2Mv9 mV2fPC+45OWBZt+lLi7L7oYQDs0TXlECXcGzUVedyFTgLtywRcWYCxtKkIUKa4/NvSqA 9P/bwk0d77HPi3lDDn5U00YTl8cpgXrdye5Y27V2J6XUohPQqM4b50dee/ZR1WgA3jGc P2xigKpacApam955kPZLtP7cOdjYmJre/LFsskvzibAjZe1lq4l9T+r1JxyZfGktvodG 9i9/IA2oh7zaZOFGYh5t4K68I7OXWkvI4v1pkw16QF4tplxLmn0LyTBrchFLm1KuxlUP RziA== X-Gm-Message-State: AO0yUKV+HoCrUDGtv+VHNSX08bf8FgzkZ283gcl46d0dPMizA2z4mfwX wyB7yKsJ1uP3nQmqun2Pir9KBUnEUYdR X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a17:90a:b10f:b0:22c:6d9e:10f3 with SMTP id z15-20020a17090ab10f00b0022c6d9e10f3mr1496364pjq.57.1675452510817; Fri, 03 Feb 2023 11:28:30 -0800 (PST) Date: Fri, 3 Feb 2023 11:28:20 -0800 In-Reply-To: <20230203192822.106773-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230203192822.106773-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230203192822.106773-4-vipinsh@google.com> Subject: [Patch v2 3/5] KVM: x86/mmu: Optimize SPTE change for aging gfn range From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756841309513175160?= X-GMAIL-MSGID: =?utf-8?q?1756841309513175160?= No need to check all of the conditions in __handle_changed_spte(). Aging a gfn range implies resetting access bit or marking spte for access tracking. Use atomic operation to only reset those bits. This avoids checking many conditions in __handle_changed_spte() API. Also, clean up code by removing dead code and API parameters. Signed-off-by: Vipin Sharma --- arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++------------------------ 1 file changed, 25 insertions(+), 43 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 83f15052aa6c..18630a06fa1f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -697,7 +697,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, /* - * __tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping + * _tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping * @kvm: KVM instance * @as_id: Address space ID, i.e. regular vs. SMM * @sptep: Pointer to the SPTE @@ -705,18 +705,12 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, * @new_spte: The new value that will be set for the SPTE * @gfn: The base GFN that was (or will be) mapped by the SPTE * @level: The level _containing_ the SPTE (its parent PT's level) - * @record_acc_track: Notify the MM subsystem of changes to the accessed state - * of the page. Should be set unless handling an MMU - * notifier for access tracking. Leaving record_acc_track - * unset in that case prevents page accesses from being - * double counted. * * Returns the old SPTE value, which _may_ be different than @old_spte if the * SPTE had voldatile bits. */ -static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, - u64 old_spte, u64 new_spte, gfn_t gfn, int level, - bool record_acc_track) +static u64 _tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, + u64 old_spte, u64 new_spte, gfn_t gfn, int level) { lockdep_assert_held_write(&kvm->mmu_lock); @@ -732,37 +726,20 @@ static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); - - if (record_acc_track) - handle_changed_spte_acc_track(old_spte, new_spte, level); - + handle_changed_spte_acc_track(old_spte, new_spte, level); handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, new_spte, level); return old_spte; } -static inline void _tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, - u64 new_spte, bool record_acc_track) -{ - WARN_ON_ONCE(iter->yielded); - - iter->old_spte = __tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, - iter->old_spte, new_spte, - iter->gfn, iter->level, - record_acc_track); -} - static inline void tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - _tdp_mmu_set_spte(kvm, iter, new_spte, true); -} + WARN_ON_ONCE(iter->yielded); -static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm, - struct tdp_iter *iter, - u64 new_spte) -{ - _tdp_mmu_set_spte(kvm, iter, new_spte, false); + iter->old_spte = _tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, + iter->old_spte, new_spte, + iter->gfn, iter->level); } #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ @@ -911,8 +888,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) return false; - __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, - sp->gfn, sp->role.level + 1, true); + _tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, + sp->gfn, sp->role.level + 1); return true; } @@ -1251,32 +1228,37 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. + * + * No need to mark corresponding PFN as accessed as this call is coming from + * MMU notifier for that page via HVA. */ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - u64 new_spte = 0; + u64 new_spte; /* If we have a non-accessed entry we don't need to change the pte. */ if (!is_accessed_spte(iter->old_spte)) return false; - new_spte = iter->old_spte; - - if (spte_ad_enabled(new_spte)) { - new_spte &= ~shadow_accessed_mask; + if (spte_ad_enabled(iter->old_spte)) { + iter->old_spte = kvm_tdp_mmu_clear_spte_bit(iter, + shadow_accessed_mask); + new_spte = iter->old_spte & ~shadow_accessed_mask; } else { + new_spte = mark_spte_for_access_track(iter->old_spte); + iter->old_spte = kvm_tdp_mmu_write_spte(iter->sptep, iter->old_spte, + new_spte, iter->level); /* * Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ - if (is_writable_pte(new_spte)) - kvm_set_pfn_dirty(spte_to_pfn(new_spte)); - - new_spte = mark_spte_for_access_track(new_spte); + if (is_writable_pte(iter->old_spte)) + kvm_set_pfn_dirty(spte_to_pfn(iter->old_spte)); } - tdp_mmu_set_spte_no_acc_track(kvm, iter, new_spte); + trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, + iter->old_spte, new_spte); return true; }