From patchwork Tue Feb 27 18:16:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 207397 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2922276dyb; Tue, 27 Feb 2024 11:35:47 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUcGYUjkUSdEWkJi6ft/JqmMm4lv6EWbLPX6X9myQO+xB/1+NoYFdU7MPz9mpKQI+NeFokvm0C3zLj29SHTLEpHGK6BAg== X-Google-Smtp-Source: AGHT+IEfatvOo0YAq74w2oUNkZbGmkFmiG16v+TepjSmt5scTXi2xnCLEUkuj0r+WVvhmX0wJVSf X-Received: by 2002:a9d:62c6:0:b0:6e4:87b0:bff8 with SMTP id z6-20020a9d62c6000000b006e487b0bff8mr12120327otk.11.1709062547610; Tue, 27 Feb 2024 11:35:47 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709062547; cv=pass; d=google.com; s=arc-20160816; b=ICeEDMqNFCORejS16lV9pCikAImcPqu4fqfNGdXAHU4TY3LvmmdAqzkwer3eGnukZE rGn1IpJtUgm7iJ3JXe4O3YPtWGQ9e3zqPpEiTvw/fJE7L8cXYzkLHxjD2niEXgA2GP1o 1797fRNZjrCIeC9uA8AoJ58fvjpUZ1OvzvT/9QFZig4W0bxk5t3f+L1qyMU+LIDoN5YE ztDDYaJw5gGxQh6joNLyEJb4+CRQ4yJ2Z/hEqKgExb6TYggUxqt6HRFyi9YX3/NhfYvv 3NrUBHkt9mdQIH0VcaRhsWn52hs7aQb8zTkULLrqestv6kvSOn9miVLtyq32ltiw8moc YmLw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=2lY/aNwLINnKOMbDQO5jsCiPZIA7GFZi3ZqNd3+Td/k=; fh=OvJRnOqsMTm9XoNmEwebcqh9Ud7yh1CTeKAP84ols98=; b=Ae0RchI34lLtVKzelGRSr9mLhgXbNdD/4vDqgIWbHbo6sWCB7+AJQaQZM77Btm2t7u X3UVUtexLeGxpTMb500iZS2NsC2+278EO2Qin3qLGZUzA83UXXzgPshkXF4vAkoL42Qv Fx/K+Y8jKgUBTlyEOqfw//6WIZsXV5nD/LdDjRRBmDr3B8Dp00/FlRA+21AC1L9yfzKm mpAoZ9rUGO7Mc/4Z4yhO7iUW73MoFxxoGQfABNdpqp5fGpJpTklSKG6oHYRFcJr+vOK9 b2CsZQ1hjQAsIIVOJTlddysjtZfPBWoO7D9qtP7+WZaGeN6Gb562f+EcJhTpEz3ItCum DRyA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ljmNej1b; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-83814-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83814-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id y22-20020aa78056000000b006e4c0cf47b2si5817834pfm.70.2024.02.27.11.35.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 11:35:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83814-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ljmNej1b; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-83814-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83814-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id DF8A228E429 for ; Tue, 27 Feb 2024 18:18:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F04A314F99C; Tue, 27 Feb 2024 18:17:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ljmNej1b" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE409149DEE for ; Tue, 27 Feb 2024 18:16:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.8 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709057822; cv=none; b=oCnm7roj7kdfVITPCdf6w/NEWXyItPfEy3OsQELfV7tjODUpGyyHrkgqZkqwLsed5QcjthqtsbySXg/yLUleGvVav04E24LV94U+df08ttIsOlctX1KNUC+aXnhF9wAhlMTCFh//2s4xUpdpz5TEv6dJplsWCB2HDDm7YUtdOMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709057822; c=relaxed/simple; bh=OpIDWzdlg0any9JAzqVnZ+xhu2rjxA6sBoybOHgswyk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GCdxbgEN/js9YFsW8JTPS/Chb6B4R9/sdHx6VqjC0Nzej6NcN93waGLS+GqeHqsNf14vJQ+EYBPpWQrm5CZL+lnNaAhOMKAU+O8a7EUqYOgeEEUeP9l6jIaYjn+O1fWqApLaAJHZ/GiDJWLhy2sXeQH5vj05BrQD2FrvZXrUBsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ljmNej1b; arc=none smtp.client-ip=192.198.163.8 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709057820; x=1740593820; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OpIDWzdlg0any9JAzqVnZ+xhu2rjxA6sBoybOHgswyk=; b=ljmNej1bg0IkKYUr1C+SB3Xcn7JNOceWIVvItsxxow6tBM4FzDSs6A1V kaHf8XvtGOCRHo2XB6QxYG7ZqEo+vC/4icEfzSu4o02rZst11MMV5Wm+l iHKOTsDsL158NS2HaiLmJwvwXwEcbgdJMSIy2pI5EZqGevWDGKtPQxBll EQSqSrCpfZV5+1rp3IH6njQAg3E8IwFdOq1SmsMSP1jM+ExJiLHNzZHj2 CjWuGm6eQhSbswzgmhepGL8oPEqYSALaVzNGQwavd5O53Rkr6FimwnvR9 kUj3kKW+7DsxQbjwN/kSWm7+6fCM8kh4WWIVITbxcWIw4DMKg1Ul5jpac g==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="20966105" X-IronPort-AV: E=Sophos;i="6.06,188,1705392000"; d="scan'208";a="20966105" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2024 10:16:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="937032727" X-IronPort-AV: E=Sophos;i="6.06,188,1705392000"; d="scan'208";a="937032727" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga001.fm.intel.com with ESMTP; 27 Feb 2024 10:16:26 -0800 Received: by black.fi.intel.com (Postfix, from userid 1000) id C1F679AD; Tue, 27 Feb 2024 20:16:19 +0200 (EET) From: "Kirill A. Shutemov" To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "Rafael J. Wysocki" , Peter Zijlstra , Adrian Hunter , Kuppuswamy Sathyanarayanan , Elena Reshetova , Jun Nakajima , Rick Edgecombe , Tom Lendacky , "Kalra, Ashish" , Sean Christopherson , "Huang, Kai" , Baoquan He , kexec@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 10/16] x86/tdx: Convert shared memory back to private on kexec Date: Tue, 27 Feb 2024 20:16:10 +0200 Message-ID: <20240227181616.3032032-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227181616.3032032-1-kirill.shutemov@linux.intel.com> References: <20240227181616.3032032-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792081969763969037 X-GMAIL-MSGID: 1792081969763969037 TDX guests allocate shared buffers to perform I/O. It is done by allocating pages normally from the buddy allocator and converting them to shared with set_memory_decrypted(). The second kernel has no idea what memory is converted this way. It only sees E820_TYPE_RAM. Accessing shared memory via private mapping is fatal. It leads to unrecoverable TD exit. On kexec walk direct mapping and convert all shared memory back to private. It makes all RAM private again and second kernel may use it normally. The conversion occurs in two steps: stopping new conversions and unsharing all memory. In the case of normal kexec, the stopping of conversions takes place while scheduling is still functioning. This allows for waiting until any ongoing conversions are finished. The second step is carried out when all CPUs except one are inactive and interrupts are disabled. This prevents any conflicts with code that may access shared memory. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe --- arch/x86/coco/tdx/tdx.c | 124 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 122 insertions(+), 2 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index fd212c9bad89..bb77a927a831 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -6,8 +6,10 @@ #include #include +#include #include #include +#include #include #include #include @@ -15,6 +17,7 @@ #include #include #include +#include /* MMIO direction */ #define EPT_READ 0 @@ -41,6 +44,9 @@ static atomic_long_t nr_shared; +static atomic_t conversions_in_progress; +static bool conversion_allowed = true; + static inline bool pte_decrypted(pte_t pte) { return cc_mkdec(pte_val(pte)) == pte_val(pte); @@ -726,6 +732,14 @@ static bool tdx_tlb_flush_required(bool private) static bool tdx_cache_flush_required(void) { + /* + * Avoid issuing CLFLUSH on set_memory_decrypted() if conversions + * stopped. Otherwise it can race with unshare_all_memory() and trigger + * implicit conversion to shared. + */ + if (!conversion_allowed) + return false; + /* * AMD SME/SEV can avoid cache flushing if HW enforces cache coherence. * TDX doesn't have such capability. @@ -809,12 +823,25 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) static int tdx_enc_status_change_prepare(unsigned long vaddr, int numpages, bool enc) { + atomic_inc(&conversions_in_progress); + + /* + * Check after bumping conversions_in_progress to serialize + * against tdx_kexec_stop_conversion(). + */ + if (!conversion_allowed) { + atomic_dec(&conversions_in_progress); + return -EBUSY; + } + /* * Only handle shared->private conversion here. * See the comment in tdx_early_init(). */ - if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } return 0; } @@ -826,17 +853,107 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages, * Only handle private->shared conversion here. * See the comment in tdx_early_init(). */ - if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } if (enc) atomic_long_sub(numpages, &nr_shared); else atomic_long_add(numpages, &nr_shared); + atomic_dec(&conversions_in_progress); + return 0; } +static void tdx_kexec_stop_conversion(bool crash) +{ + /* Stop new private<->shared conversions */ + conversion_allowed = false; + + /* + * Make sure conversion_allowed is cleared before checking + * conversions_in_progress. + */ + barrier(); + + /* + * Crash kernel reaches here with interrupts disabled: can't wait for + * conversions to finish. + * + * If race happened, just report and proceed. + */ + if (!crash) { + unsigned long timeout; + + /* + * Wait for in-flight conversions to complete. + * + * Do not wait more than 30 seconds. + */ + timeout = 30 * USEC_PER_SEC; + while (atomic_read(&conversions_in_progress) && timeout--) + udelay(1); + } + + if (atomic_read(&conversions_in_progress)) + pr_warn("Failed to finish shared<->private conversions\n"); +} + +static void tdx_kexec_unshare_mem(void) +{ + unsigned long addr, end; + long found = 0, shared; + + /* + * Walk direct mapping and convert all shared memory back to private, + */ + + addr = PAGE_OFFSET; + end = PAGE_OFFSET + get_max_mapped(); + + while (addr < end) { + unsigned long size; + unsigned int level; + pte_t *pte; + + pte = lookup_address(addr, &level); + size = page_level_size(level); + + if (pte && pte_decrypted(*pte)) { + int pages = size / PAGE_SIZE; + + /* + * Touching memory with shared bit set triggers implicit + * conversion to shared. + * + * Make sure nobody touches the shared range from + * now on. + */ + set_pte(pte, __pte(0)); + + if (!tdx_enc_status_changed(addr, pages, true)) { + pr_err("Failed to unshare range %#lx-%#lx\n", + addr, addr + size); + } + + found += pages; + } + + addr += size; + } + + __flush_tlb_all(); + + shared = atomic_long_read(&nr_shared); + if (shared != found) { + pr_err("shared page accounting is off\n"); + pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found); + } +} + void __init tdx_early_init(void) { struct tdx_module_args args = { @@ -896,6 +1013,9 @@ void __init tdx_early_init(void) x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = tdx_tlb_flush_required; + x86_platform.guest.enc_kexec_stop_conversion = tdx_kexec_stop_conversion; + x86_platform.guest.enc_kexec_unshare_mem = tdx_kexec_unshare_mem; + /* * TDX intercepts the RDMSR to read the X2APIC ID in the parallel * bringup low level code. That raises #VE which cannot be handled