From patchwork Fri Dec 22 23:52:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 182822 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp1395656dyi; Fri, 22 Dec 2023 15:58:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHkFpGZCExPrUaURJUEYV/CaXOjfNQMHCQxegoYkvv1/9smjDeB2JsliBgRfc4ElxRH++Kx X-Received: by 2002:a05:6a00:2d92:b0:6d8:2578:ef4a with SMTP id fb18-20020a056a002d9200b006d82578ef4amr2196579pfb.61.1703289483807; Fri, 22 Dec 2023 15:58:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703289483; cv=none; d=google.com; s=arc-20160816; b=PoXdAAZwSJbT02Oy8Gnwa9kpWCGB9PTKdo/gFtwl0p+EfrS9ygk5PZtszbzL0JzXt/ 0OBNIByj2BZwLas/eaPjek8IQwHgkh0ojjEzoz4+BDO0xjUK+uJel9vTKnkL2wgxvf9P aiF6GMQ1dblr1vTZgj0ypeynQJfHeUJYL9kcc57y7N+Ganf6onfC9wsEKGT0DDbMmQcV DrPoqBhphkPRqhVmHTUAP59UoEhMMbTR4hi+bsPGg8KbIe0r8PGbKb463LXo0G84sZub 3vurtTEwZhsa4/snGVQdlstGQw5RrowMvCaEHBwKWNzhtw/W0kVsdsC8QBiCpmq1VCJ5 0Mtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=SK7xEuA20z2XT4xn1qaMf/zRkn1oBx2NiJT9fNFJZVk=; fh=OvJRnOqsMTm9XoNmEwebcqh9Ud7yh1CTeKAP84ols98=; b=exyFpP+3i1YXa1mZuzz2gV7578Iy+gAFHHsK2/T1sj4dNWAqLKVu3W9I1TKFXHrx3E zgUJozsguOmaVnCL3PVTYJfD8FhQ7//YTP3NHrwoacacUU+9XUDIa5knEKl3Qy0gU8g4 u/3akojADwheQOldvSARvizrygdpAALphp5Wo00D5Gd/xu101e6QuGzZGw0Kp/KzL07i CD60Li7jI9qFM5wKrqb/4qw+yCkl703C6I6MncTT5EbQEvbRCBAhvDfllPZNlPFxKjbL lAtSaGHrUUhCj8kQ2NOzrFgDYDV9qbKoJzLN8LxSUZ5rUHuKBPdpeWsuO5LJ/vESgSbE RiRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=VbPuv1Xs; spf=pass (google.com: domain of linux-kernel+bounces-10137-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10137-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id e18-20020a656792000000b005cda184c282si781993pgr.210.2023.12.22.15.58.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Dec 2023 15:58:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10137-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=VbPuv1Xs; spf=pass (google.com: domain of linux-kernel+bounces-10137-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10137-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 460CBB24E20 for ; Fri, 22 Dec 2023 23:55:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B23964C62E; Fri, 22 Dec 2023 23:52:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VbPuv1Xs" X-Original-To: linux-kernel@vger.kernel.org Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B65E3D974 for ; Fri, 22 Dec 2023 23:52:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1703289150; x=1734825150; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CPNyfIjhkN66pkeS9MLVNslXAiONe+c3JXdzZ/y2Vvo=; b=VbPuv1XsrqWvlNdXpr/U8j4W3U5WNs9TZ+Wpbs65hWHphgBLKd847NJ8 TrXPNdKcO++u8RDlJvGJSe8EQ1s/F5iDfYxPIhBuXTFI6e6RQFrDiShEu 78d2qzrDFbKMaoXszV+FujGezhucoFOPmLnzSIy9qYC/cDcjI5MJ1ipRj qwnAs90lxfuw/EW+lguIVN1hMIst+46JgU+dlXIhjBhaKHAk0cVlP54YR ieS0Qhe4hBf0oYraTX3t/tVkplQPb/3N+YNo6t3cGi/hzeUNBX0Rxezlc 8E+5+j85Rvg+I2SGzEDALYEaTCQoemFFkGXBSxLkkP9FpoOAQL6UN3Hwv g==; X-IronPort-AV: E=McAfee;i="6600,9927,10932"; a="395063322" X-IronPort-AV: E=Sophos;i="6.04,297,1695711600"; d="scan'208";a="395063322" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Dec 2023 15:52:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10932"; a="900623009" X-IronPort-AV: E=Sophos;i="6.04,297,1695711600"; d="scan'208";a="900623009" Received: from jeroenke-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.249.35.180]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Dec 2023 15:52:22 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 43E9510A4DC; Sat, 23 Dec 2023 02:52:12 +0300 (+03) From: "Kirill A. Shutemov" To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "Rafael J. Wysocki" , Peter Zijlstra , Adrian Hunter , Kuppuswamy Sathyanarayanan , Elena Reshetova , Jun Nakajima , Rick Edgecombe , Tom Lendacky , "Kalra, Ashish" , Sean Christopherson , "Huang, Kai" , Baoquan He , kexec@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 10/16] x86/tdx: Convert shared memory back to private on kexec Date: Sat, 23 Dec 2023 02:52:02 +0300 Message-ID: <20231222235209.32143-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231222235209.32143-1-kirill.shutemov@linux.intel.com> References: <20231222235209.32143-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786028473866763039 X-GMAIL-MSGID: 1786028473866763039 TDX guests allocate shared buffers to perform I/O. It is done by allocating pages normally from the buddy allocator and converting them to shared with set_memory_decrypted(). The second kernel has no idea what memory is converted this way. It only sees E820_TYPE_RAM. Accessing shared memory via private mapping is fatal. It leads to unrecoverable TD exit. On kexec walk direct mapping and convert all shared memory back to private. It makes all RAM private again and second kernel may use it normally. The conversion occurs in two steps: stopping new conversions and unsharing all memory. In the case of normal kexec, the stopping of conversions takes place while scheduling is still functioning. This allows for waiting until any ongoing conversions are finished. The second step is carried out when all CPUs except one are inactive and interrupts are disabled. This prevents any conflicts with code that may access shared memory. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe --- arch/x86/coco/tdx/tdx.c | 119 +++++++++++++++++++++++++++++++- arch/x86/include/asm/x86_init.h | 2 + arch/x86/kernel/crash.c | 6 ++ arch/x86/kernel/reboot.c | 13 ++++ 4 files changed, 138 insertions(+), 2 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index 8a49484a2917..5c64db168edd 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -6,8 +6,10 @@ #include #include +#include #include #include +#include #include #include #include @@ -15,6 +17,7 @@ #include #include #include +#include /* MMIO direction */ #define EPT_READ 0 @@ -41,6 +44,9 @@ static atomic_long_t nr_shared; +static atomic_t conversions_in_progress; +static bool conversion_allowed = true; + static inline bool pte_decrypted(pte_t pte) { return cc_mkdec(pte_val(pte)) == pte_val(pte); @@ -726,6 +732,14 @@ static bool tdx_tlb_flush_required(bool private) static bool tdx_cache_flush_required(void) { + /* + * Avoid issuing CLFLUSH on set_memory_decrypted() if conversions + * stopped. Otherwise it can race with unshare_all_memory() and trigger + * implicit conversion to shared. + */ + if (!conversion_allowed) + return false; + /* * AMD SME/SEV can avoid cache flushing if HW enforces cache coherence. * TDX doesn't have such capability. @@ -809,12 +823,25 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) static int tdx_enc_status_change_prepare(unsigned long vaddr, int numpages, bool enc) { + atomic_inc(&conversions_in_progress); + + /* + * Check after bumping conversions_in_progress to serialize + * against tdx_shutdown(). + */ + if (!conversion_allowed) { + atomic_dec(&conversions_in_progress); + return -EBUSY; + } + /* * Only handle shared->private conversion here. * See the comment in tdx_early_init(). */ - if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } return 0; } @@ -826,17 +853,102 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages, * Only handle private->shared conversion here. * See the comment in tdx_early_init(). */ - if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } if (enc) atomic_long_sub(numpages, &nr_shared); else atomic_long_add(numpages, &nr_shared); + atomic_dec(&conversions_in_progress); + return 0; } +static void tdx_kexec_stop_conversion(bool crash) +{ + /* Stop new private<->shared conversions */ + conversion_allowed = false; + barrier(); + + /* + * Crash kernel reaches here with interrupts disabled: can't wait for + * conversions to finish. + * + * If race happened, just report and proceed. + */ + if (!crash) { + unsigned long timeout; + + /* + * Wait for in-flight conversions to complete. + * + * Do not wait more than 30 seconds. + */ + timeout = 30 * USEC_PER_SEC; + while (atomic_read(&conversions_in_progress) && timeout--) + udelay(1); + } + + if (atomic_read(&conversions_in_progress)) + pr_warn("Failed to finish shared<->private conversions\n"); +} + +static void tdx_kexec_unshare_mem(void) +{ + unsigned long addr, end; + long found = 0, shared; + + /* + * Walk direct mapping and convert all shared memory back to private, + */ + + addr = PAGE_OFFSET; + end = PAGE_OFFSET + get_max_mapped(); + + while (addr < end) { + unsigned long size; + unsigned int level; + pte_t *pte; + + pte = lookup_address(addr, &level); + size = page_level_size(level); + + if (pte && pte_decrypted(*pte)) { + int pages = size / PAGE_SIZE; + + /* + * Touching memory with shared bit set triggers implicit + * conversion to shared. + * + * Make sure nobody touches the shared range from + * now on. + */ + set_pte(pte, __pte(0)); + + if (!tdx_enc_status_changed(addr, pages, true)) { + pr_err("Failed to unshare range %#lx-%#lx\n", + addr, addr + size); + } + + found += pages; + } + + addr += size; + } + + __flush_tlb_all(); + + shared = atomic_long_read(&nr_shared); + if (shared != found) { + pr_err("shared page accounting is off\n"); + pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found); + } +} + void __init tdx_early_init(void) { struct tdx_module_args args = { @@ -896,6 +1008,9 @@ void __init tdx_early_init(void) x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = tdx_tlb_flush_required; + x86_platform.guest.enc_kexec_stop_conversion = tdx_kexec_stop_conversion; + x86_platform.guest.enc_kexec_unshare_mem = tdx_kexec_unshare_mem; + /* * TDX intercepts the RDMSR to read the X2APIC ID in the parallel * bringup low level code. That raises #VE which cannot be handled diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h index c9503fe2d13a..3196ff20a29e 100644 --- a/arch/x86/include/asm/x86_init.h +++ b/arch/x86/include/asm/x86_init.h @@ -154,6 +154,8 @@ struct x86_guest { int (*enc_status_change_finish)(unsigned long vaddr, int npages, bool enc); bool (*enc_tlb_flush_required)(bool enc); bool (*enc_cache_flush_required)(void); + void (*enc_kexec_stop_conversion)(bool crash); + void (*enc_kexec_unshare_mem)(void); }; /** diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c index c92d88680dbf..b99bd28ad22f 100644 --- a/arch/x86/kernel/crash.c +++ b/arch/x86/kernel/crash.c @@ -40,6 +40,7 @@ #include #include #include +#include /* Used while preparing memory map entries for second kernel */ struct crash_memmap_data { @@ -107,6 +108,11 @@ void native_machine_crash_shutdown(struct pt_regs *regs) crash_smp_send_stop(); + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { + x86_platform.guest.enc_kexec_stop_conversion(true); + x86_platform.guest.enc_kexec_unshare_mem(); + } + cpu_emergency_disable_virtualization(); /* diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c index 830425e6d38e..16dde83df49a 100644 --- a/arch/x86/kernel/reboot.c +++ b/arch/x86/kernel/reboot.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +32,7 @@ #include #include #include +#include /* * Power off function, if any @@ -716,6 +718,14 @@ static void native_machine_emergency_restart(void) void native_machine_shutdown(void) { + /* + * Call enc_kexec_stop_conversion() while all CPUs are still active and + * interrupts are enabled. This will allow all in-flight memory + * conversions to finish cleanly. + */ + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && kexec_in_progress) + x86_platform.guest.enc_kexec_stop_conversion(false); + /* Stop the cpus and apics */ #ifdef CONFIG_X86_IO_APIC /* @@ -752,6 +762,9 @@ void native_machine_shutdown(void) #ifdef CONFIG_X86_64 x86_platform.iommu_shutdown(); #endif + + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && kexec_in_progress) + x86_platform.guest.enc_kexec_unshare_mem(); } static void __machine_emergency_restart(int emergency)