From patchwork Tue Feb 27 21:24:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 207534 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp3035549dyb; Tue, 27 Feb 2024 15:56:37 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWeggMnRg0Uz+NLHVSs52fl6FXpvyyXhAaEyzrCTijXFkAG78RE8LG0p/0wC+TM4XT3NittxJTTJ745v+tuP62A4M0twA== X-Google-Smtp-Source: AGHT+IGXQ/RX/s6xHn++WMTNcodXbwq49kruxG90yBivd/I5bkfvpGrv9sfdHbxOfXxDK4SF8BuZ X-Received: by 2002:a17:906:d0d7:b0:a3f:c3f0:69bf with SMTP id bq23-20020a170906d0d700b00a3fc3f069bfmr787479ejb.13.1709078197529; Tue, 27 Feb 2024 15:56:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709078197; cv=pass; d=google.com; s=arc-20160816; b=r3gjscJcfcuh9tU0i8ShI8p1iti2ydsXDGTHpOj2XXIxRynqzjH+VgO6MRS8RPIVt7 YYZRNBZg8lQXq8Fg1dXiyuJIm71Vwe5yZ33Zk54ZAE+8iRGUiZOr82FCF0JNYBlTd5Qr ZZw2arxL/yz8GtkWzXPviHEXBilaJXk8+hHIvSI1WPxaji/QcvkAjHbiBU6GRzBtuoA2 Kq2RrSnjxknj8NXMUhsi0DrlaxukSVbAcP4bDnY58Z4dOJCMF8yiL5fG1pAygpRAUk+K I2Jkc0ce/xI0TRClPxGLQEZOXCj3lTR2gKFzPgyYk8hEQig48/XRdOU9JdUKw0tuKuTk KdjQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=1/LIepJ8YYgKUU9WHWoDA1pw2OQ6YA2Xd6cXKdtLQjA=; fh=OvJRnOqsMTm9XoNmEwebcqh9Ud7yh1CTeKAP84ols98=; b=zCk56R2eBfH7SkU2zKfYERyHkeV38PicnWeXhEdLqDI5tSosHQ38bdfTlzd90cuVdP TZzTYzP0PAki3jUNpstLBToP75iYSK+zqfepGYcRUx3I5AuKNppzkl1TlYTLfyJG3mq2 F6yPqYKJCfCxZ1llaRPVrDWaStW5ivEC3KGYd9gveJfuEA9IKg0x04M8Gq3+3XyMGVXr ViC7p91Q93z5X/MAEYrq4+qkP45fP4U8O7HwOlVcui3RKGvGMHmkZIt1lBH0EJq/raIC WQ6wnAv9KMmBYnd9V2b5UO+XZ5vb65MBkho0XD8nHElRnkC8IOpyGTf5rfkswKW+qHgj +Rcg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="nX/Mm1GC"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-84041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84041-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id e14-20020a170906044e00b00a42ed61a969si1156626eja.957.2024.02.27.15.56.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 15:56:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-84041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="nX/Mm1GC"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-84041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-84041-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 895671F2D47D for ; Tue, 27 Feb 2024 21:28:16 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8953815AABD; Tue, 27 Feb 2024 21:25:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nX/Mm1GC" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0C3E158D6A for ; Tue, 27 Feb 2024 21:25:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709069117; cv=none; b=J3DDG5k1W9hU/oCjX2kb+8WYa/Lf3E5TVPpWulzTp3j5OPEne5bp9/XCncy2BigQ24OVD4LK76rtarf4NhH0xK/5mQqICSMPkN+inz06EsMldrz4zme1NIINiyjvdVFpH2Zvll8hiz+CTin16+ZcdvHSYNJqleu+xL9evBAurE8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709069117; c=relaxed/simple; bh=V4Wv9ihxrfAerMNIMD+R4MX+Bks/1ZXzWa1IUxhgkSg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=apLA5Law478syBEWkHS66wn44q5P8W9wAY5u5toDl3LU+q+mzdksnouzoQHcMxDSDx1ks1S9+Im1cK8JHGZMaQrQ5P/hVAD1atxGFmhhP3Nru0+D8nDl9aN6or7ZvSSCvk1PwcCSKqcUyycEGNIEcjMa1SXxDnW71qHOxrCRxD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nX/Mm1GC; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709069115; x=1740605115; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V4Wv9ihxrfAerMNIMD+R4MX+Bks/1ZXzWa1IUxhgkSg=; b=nX/Mm1GC0lhYFwdiHMDK7DAMbmONHNWCA06JZMwZhqRdDUYQ1GAxblWr e1GQI3vCNFH2qSdg+sxxFBAWZBuGxMDEuOddF2GCx05YD6l6Kx1RMEy5Q ZipHLn3US4zcYkYE1TO8xHT0wL28YNbiweggRgnjob/KElYxB0E+MYo5v f+533V17fKBWdw962aeD61qxCGvwuBJ/qSnf0OV6Np8PhSzxGtG/2Si83 rr6oJ5KdqTucCXadpbdwvJYi0aGEufOZKuFp9YqEE/qU7WS7P8R/ej8ev Mt26VkvIGFPiEPImJg653VgyajxMC+KkVY9sMiqgzotDSUEfPxZthirTL w==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="3567056" X-IronPort-AV: E=Sophos;i="6.06,188,1705392000"; d="scan'208";a="3567056" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2024 13:25:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="937032913" X-IronPort-AV: E=Sophos;i="6.06,188,1705392000"; d="scan'208";a="937032913" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga001.fm.intel.com with ESMTP; 27 Feb 2024 13:25:02 -0800 Received: by black.fi.intel.com (Postfix, from userid 1000) id 6579A9DB; Tue, 27 Feb 2024 23:24:56 +0200 (EET) From: "Kirill A. Shutemov" To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "Rafael J. Wysocki" , Peter Zijlstra , Adrian Hunter , Kuppuswamy Sathyanarayanan , Elena Reshetova , Jun Nakajima , Rick Edgecombe , Tom Lendacky , "Kalra, Ashish" , Sean Christopherson , "Huang, Kai" , Baoquan He , kexec@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 10/17] x86/tdx: Convert shared memory back to private on kexec Date: Tue, 27 Feb 2024 23:24:45 +0200 Message-ID: <20240227212452.3228893-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227212452.3228893-1-kirill.shutemov@linux.intel.com> References: <20240227212452.3228893-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792096910349639877 X-GMAIL-MSGID: 1792098380019016081 TDX guests allocate shared buffers to perform I/O. It is done by allocating pages normally from the buddy allocator and converting them to shared with set_memory_decrypted(). The second kernel has no idea what memory is converted this way. It only sees E820_TYPE_RAM. Accessing shared memory via private mapping is fatal. It leads to unrecoverable TD exit. On kexec walk direct mapping and convert all shared memory back to private. It makes all RAM private again and second kernel may use it normally. The conversion occurs in two steps: stopping new conversions and unsharing all memory. In the case of normal kexec, the stopping of conversions takes place while scheduling is still functioning. This allows for waiting until any ongoing conversions are finished. The second step is carried out when all CPUs except one are inactive and interrupts are disabled. This prevents any conflicts with code that may access shared memory. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe --- arch/x86/coco/tdx/tdx.c | 72 +++++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable.h | 5 +++ arch/x86/include/asm/set_memory.h | 3 ++ arch/x86/mm/pat/set_memory.c | 35 +++++++++++++-- 4 files changed, 112 insertions(+), 3 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index 979891e97d83..cc67f7380055 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -14,6 +15,7 @@ #include #include #include +#include /* MMIO direction */ #define EPT_READ 0 @@ -831,6 +833,73 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages, return 0; } +/* Stop new private<->shared conversions */ +static void tdx_kexec_stop_conversion(bool crash) +{ + /* + * Crash kernel reaches here with interrupts disabled: can't wait for + * conversions to finish. + * + * If race happened, just report and proceed. + */ + bool wait_for_lock = !crash; + + if (!stop_memory_enc_conversion(wait_for_lock)) + pr_warn("Failed to finish shared<->private conversions\n"); +} + +static void tdx_kexec_unshare_mem(void) +{ + unsigned long addr, end; + long found = 0, shared; + + /* + * Walk direct mapping and convert all shared memory back to private, + */ + + addr = PAGE_OFFSET; + end = PAGE_OFFSET + get_max_mapped(); + + while (addr < end) { + unsigned long size; + unsigned int level; + pte_t *pte; + + pte = lookup_address(addr, &level); + size = page_level_size(level); + + if (pte && pte_decrypted(*pte)) { + int pages = size / PAGE_SIZE; + + /* + * Touching memory with shared bit set triggers implicit + * conversion to shared. + * + * Make sure nobody touches the shared range from + * now on. + */ + set_pte(pte, __pte(0)); + + if (!tdx_enc_status_changed(addr, pages, true)) { + pr_err("Failed to unshare range %#lx-%#lx\n", + addr, addr + size); + } + + found += pages; + } + + addr += size; + } + + __flush_tlb_all(); + + shared = atomic_long_read(&nr_shared); + if (shared != found) { + pr_err("shared page accounting is off\n"); + pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found); + } +} + void __init tdx_early_init(void) { struct tdx_module_args args = { @@ -890,6 +959,9 @@ void __init tdx_early_init(void) x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = tdx_tlb_flush_required; + x86_platform.guest.enc_kexec_stop_conversion = tdx_kexec_stop_conversion; + x86_platform.guest.enc_kexec_unshare_mem = tdx_kexec_unshare_mem; + /* * TDX intercepts the RDMSR to read the X2APIC ID in the parallel * bringup low level code. That raises #VE which cannot be handled diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index df0f7d4a96f3..87f18e4f9373 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -141,6 +141,11 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } +static inline bool pte_decrypted(pte_t pte) +{ + return cc_mkdec(pte_val(pte)) == pte_val(pte); +} + #define pmd_dirty pmd_dirty static inline bool pmd_dirty(pmd_t pmd) { diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index a5e89641bd2d..9d4a8e548820 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -48,8 +48,11 @@ int set_memory_wc(unsigned long addr, int numpages); int set_memory_wb(unsigned long addr, int numpages); int set_memory_np(unsigned long addr, int numpages); int set_memory_4k(unsigned long addr, int numpages); + +bool stop_memory_enc_conversion(bool wait); int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); + int set_memory_np_noalias(unsigned long addr, int numpages); int set_memory_nonglobal(unsigned long addr, int numpages); int set_memory_global(unsigned long addr, int numpages); diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 0d2267ad4e0e..a81bb24b80c6 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -2176,12 +2176,41 @@ static int __set_memory_enc_pgtable(unsigned long addr, int numpages, bool enc) return ret; } +static DECLARE_RWSEM(mem_enc_lock); + +/* + * Stop new private<->shared conversions. + * + * Taking the exclusive mem_enc_lock waits for in-flight conversions to complete. + * The lock is not released to prevent new conversions from being started. + * + * If sleep is not allowed, as in a crash scenario, try to take the lock. + * Failure indicates that there is a race with the conversion. + */ +bool stop_memory_enc_conversion(bool wait) +{ + if (!wait) + return down_write_trylock(&mem_enc_lock); + + down_write(&mem_enc_lock); + + return true; +} + static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc) { - if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) - return __set_memory_enc_pgtable(addr, numpages, enc); + int ret = 0; - return 0; + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + if (!down_read_trylock(&mem_enc_lock)) + return -EBUSY; + + ret =__set_memory_enc_pgtable(addr, numpages, enc); + + up_read(&mem_enc_lock); + } + + return ret; } int set_memory_encrypted(unsigned long addr, int numpages)