From patchwork Mon Feb 26 08:29:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Isaku Yamahata X-Patchwork-Id: 206474 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1954619dyb; Mon, 26 Feb 2024 01:16:05 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV8xLjM/+LVw0LQYQfnPS5RJrR5jxTMjI1a8MVWR9V4jO1goOJ+sAlCEs/wOvSlR02/1djp3FnyuYYnPxzP7yuvQ5zJlg== X-Google-Smtp-Source: AGHT+IHbLaTBeEq3Bycswp81htW72aIXz6U233xVCoVEXg2r04YzZOJRrqIu1SrcLFYUOIFkwV1E X-Received: by 2002:a1f:ccc6:0:b0:4c8:e834:6ce2 with SMTP id c189-20020a1fccc6000000b004c8e8346ce2mr2865376vkg.5.1708938965532; Mon, 26 Feb 2024 01:16:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708938965; cv=pass; d=google.com; s=arc-20160816; b=RioKLrsGpOBMAJlMLOxXElv5NjngANVYkdKgTv6WvBOPfn/LP4Y/EL+dgiig/sHQbW aUADznrSjpIG4SG6mH1QDqzcm664Hkl9D3jR4rYrAVAR0tvxZanvF2DSowaqzKKZbtrI az1cIgnqOYYzf/L61fJz4hNoGGxivzbHBIYHESjwJmWZKWwZHcdKNPOZb0UCKhg2MD+t G6hEI25QNaEn77YlacwEyHZmWe+ODywIwRgyTYhxBLR3ZJfypZ8uYf0NLgDoxMD6JC5/ EHytH2L8+5isBojaP1Vn4yicbie8CKqaD+YT3lhbs7L+z2DW17b/vWk98cKBA1RIcSV2 ji7A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=pxyvOJjXCSvCHlCnTrJwYXeOXUKtG5NNwiVL8zKi07o=; fh=Itbyk7CEvizIrzGEESCqq3I2tZgG1kc/GkVOa3S7Hsg=; b=Rrk0MHuxjU6a5ooU9I9oprUED2Z3QwhcO9Sp6o1Tcaqv17npKB98/8HOGWoFu7Agem QU6sTTJ1sVSK30Bl2MuhREIBv2QyO4e5M/W71gYBYnDylK9ZTeCc8GveB4os96qJfAFE t9c3k5EffLqmrI1T0H3+CW/xuU7FpwHfy1i3CwQ1wlm08WqKZI4ENEyq6PEl+AxYX6Fl Tsf8ITRyL2hzSjRnjujsQPAvpZDurauLaqSlLzg4NLbr10r7U1DKSLNApNLtgBuEXRiA zEDwaFVaZw20y6R8IhyAYkmWy/vu6mhE18NRcjWUy2aYOzemAPaII9MTurTa1dWCFsKj 8j2Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="e/7M++5H"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-80909-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80909-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id 4-20020a0562140cc400b0068f51dc06f1si4846408qvx.450.2024.02.26.01.16.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 01:16:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80909-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="e/7M++5H"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-80909-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80909-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 2FDAD1C23381 for ; Mon, 26 Feb 2024 09:15:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 882231369BD; Mon, 26 Feb 2024 08:30:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="e/7M++5H" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54076134CF9; Mon, 26 Feb 2024 08:29:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708936197; cv=none; b=Ixwkaq+qnRrd/bsvo70YDGoLsCWWotPTZW9DV6wWjMfYhbimycqoDtQ9rWr1xdNvDPsW9yYMACsgEjBBFahuWf3ulthT3khb0UQiIQPbm4+1eUUjq1y+DktADS6Gwr8U1VADPI45hKBSN4voTj4wcktY7txBbmeJBWbASWgCb1c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708936197; c=relaxed/simple; bh=vAER6tofN9QA4BcfDZw3ooI9Hrt9ifz5SaXVfMtvuno=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=L+OazS2hU2IOTPnyePAp4b2DSO8j8+1r/eOJGyF4n/zoFATfiCGtO7KTCvwJMyWSzYOh29tp1z2gmIhza/l1TdX7qjOU4FcTwGrbeZmWvrlPaeyPwqv8elDXYhahMr2+vgRXfNWuoQUOmLZz2Q3HbyMxHbK0SzOCaDoX11vvnjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=e/7M++5H; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708936194; x=1740472194; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vAER6tofN9QA4BcfDZw3ooI9Hrt9ifz5SaXVfMtvuno=; b=e/7M++5HFJq76rr3qLy86biEXqxchkp7+m8gxhNMqXziwRmet0a+fsCy uwWhYYdjswOzd40Tp8VgmBw3E1r2bWYQisQ32uOumDnctDVaqtRoRV6ZO sW7P2lLcBM16Hrv/J4biValdE3BEBMUmI/fSZ10E+CLR8aiUDqCLitHFW 59bXFgf8c1qlIczL4mhDvUe3p2iOGZRDyNmHoTKgfwl8bpcDxEmUnSgrJ 0LTTn9CFo4oCUgEkJNzTS2ftoUHukJwA+Sy+KBbZhSOqBBDuCqC8SMkzx GMSboVilxO+NVtyv7bkNX9hos6xFcRU6xI2CqktZHYZt8kBy8uxSaOjO2 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10995"; a="14623332" X-IronPort-AV: E=Sophos;i="6.06,185,1705392000"; d="scan'208";a="14623332" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 00:29:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,185,1705392000"; d="scan'208";a="6519432" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 00:29:36 -0800 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , Kai Huang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v8 12/14] KVM: TDX: Implement merge pages into a large page Date: Mon, 26 Feb 2024 00:29:26 -0800 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791952385159220232 X-GMAIL-MSGID: 1791952385159220232 From: Isaku Yamahata Implement merge_private_stp callback. Signed-off-by: Isaku Yamahata --- v7: - Fix subject, x86/tdp_mmu => TDX - comment: use ulink instead of free for clarity v6: - repeat TDH.MEM.PAGE.PROMOTE() on TDX_INTERRUPTED_RESTARTABLE Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 74 ++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx_arch.h | 1 + arch/x86/kvm/vmx/tdx_errno.h | 2 + arch/x86/kvm/vmx/tdx_ops.h | 11 ++++++ 4 files changed, 88 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 88af64658a9c..5b4d94a6c6e2 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1674,6 +1674,51 @@ static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, return 0; } +static int tdx_sept_merge_private_spt(struct kvm *kvm, gfn_t gfn, + enum pg_level level, void *private_spt) +{ + int tdx_level = pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); + struct tdx_module_args out; + gpa_t gpa = gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level); + u64 err; + + /* See comment in tdx_sept_set_private_spte() */ + do { + err = tdh_mem_page_promote(kvm_tdx->tdr_pa, gpa, tdx_level, &out); + } while (err == TDX_INTERRUPTED_RESTARTABLE); + if (unlikely(err == TDX_ERROR_SEPT_BUSY)) + return -EAGAIN; + if (unlikely(err == (TDX_EPT_INVALID_PROMOTE_CONDITIONS | + TDX_OPERAND_ID_RCX))) + /* + * Some pages are accepted, some pending. Need to wait for TD + * to accept all pages. Tell it the caller. + */ + return -EAGAIN; + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_PAGE_PROMOTE, err, &out); + return -EIO; + } + WARN_ON_ONCE(out.rcx != __pa(private_spt)); + + /* + * TDH.MEM.PAGE.PROMOTE unlinks the Secure-EPT page for the lower level. + * Flush cache for reuse. + */ + do { + err = tdh_phymem_page_wbinvd(set_hkid_to_hpa(__pa(private_spt), + to_kvm_tdx(kvm)->hkid)); + } while (unlikely(err == (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX))); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err, NULL); + return -EIO; + } + + tdx_clear_page(__pa(private_spt), PAGE_SIZE); + return 0; +} + static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level) { @@ -1770,6 +1815,33 @@ static void tdx_track(struct kvm *kvm) } +static int tdx_sept_unzap_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level) +{ + int tdx_level = pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); + gpa_t gpa = gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level); + struct tdx_module_args out; + u64 err; + + do { + err = tdh_mem_range_unblock(kvm_tdx->tdr_pa, gpa, tdx_level, &out); + + /* + * tdh_mem_range_block() is accompanied with tdx_track() via kvm + * remote tlb flush. Wait for the caller of + * tdh_mem_range_block() to complete TDX track. + */ + } while (err == (TDX_TLB_TRACKING_NOT_DONE | TDX_OPERAND_ID_SEPT)); + if (unlikely(err == TDX_ERROR_SEPT_BUSY)) + return -EAGAIN; + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_RANGE_UNBLOCK, err, &out); + return -EIO; + } + return 0; +} + static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level, void *private_spt) { @@ -3331,9 +3403,11 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) x86_ops->link_private_spt = tdx_sept_link_private_spt; x86_ops->free_private_spt = tdx_sept_free_private_spt; x86_ops->split_private_spt = tdx_sept_split_private_spt; + x86_ops->merge_private_spt = tdx_sept_merge_private_spt; x86_ops->set_private_spte = tdx_sept_set_private_spte; x86_ops->remove_private_spte = tdx_sept_remove_private_spte; x86_ops->zap_private_spte = tdx_sept_zap_private_spte; + x86_ops->unzap_private_spte = tdx_sept_unzap_private_spte; return 0; diff --git a/arch/x86/kvm/vmx/tdx_arch.h b/arch/x86/kvm/vmx/tdx_arch.h index bb324f744bbf..a320f6d45731 100644 --- a/arch/x86/kvm/vmx/tdx_arch.h +++ b/arch/x86/kvm/vmx/tdx_arch.h @@ -29,6 +29,7 @@ #define TDH_MNG_KEY_FREEID 20 #define TDH_MNG_INIT 21 #define TDH_VP_INIT 22 +#define TDH_MEM_PAGE_PROMOTE 23 #define TDH_MEM_SEPT_RD 25 #define TDH_VP_RD 26 #define TDH_MNG_KEY_RECLAIMID 27 diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/kvm/vmx/tdx_errno.h index 416708e6cbb7..799d7166e69a 100644 --- a/arch/x86/kvm/vmx/tdx_errno.h +++ b/arch/x86/kvm/vmx/tdx_errno.h @@ -21,6 +21,8 @@ #define TDX_KEY_CONFIGURED 0x0000081500000000ULL #define TDX_NO_HKID_READY_TO_WBCACHE 0x0000082100000000ULL #define TDX_FLUSHVP_NOT_DONE 0x8000082400000000ULL +#define TDX_TLB_TRACKING_NOT_DONE 0xC0000B0800000000ULL +#define TDX_EPT_INVALID_PROMOTE_CONDITIONS 0xC0000B0900000000ULL #define TDX_EPT_ENTRY_STATE_INCORRECT 0xC0000B0D00000000ULL /* diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h index d8f0d9aa7439..bf660eefa9e0 100644 --- a/arch/x86/kvm/vmx/tdx_ops.h +++ b/arch/x86/kvm/vmx/tdx_ops.h @@ -254,6 +254,17 @@ static inline u64 tdh_mem_page_demote(hpa_t tdr, gpa_t gpa, int level, hpa_t pag return tdx_seamcall_sept(TDH_MEM_PAGE_DEMOTE, &in, out); } +static inline u64 tdh_mem_page_promote(hpa_t tdr, gpa_t gpa, int level, + struct tdx_module_args *out) +{ + struct tdx_module_args in = { + .rcx = gpa | level, + .rdx = tdr, + }; + + return tdx_seamcall_sept(TDH_MEM_PAGE_PROMOTE, &in, out); +} + static inline u64 tdh_mr_extend(hpa_t tdr, gpa_t gpa, struct tdx_module_args *out) {