From patchwork Wed Jan 24 12:55:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 191696 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2553:b0:103:945f:af90 with SMTP id p19csp1170064dyi; Wed, 24 Jan 2024 10:17:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IHtWydBrn74GSzrGapdWPkbeZ8G0gDDDGb4DJWtdkKnXQEA76uacmEg9AgDeSQNqXYHe5T/ X-Received: by 2002:a17:906:f8d4:b0:a2f:c34:1dd9 with SMTP id lh20-20020a170906f8d400b00a2f0c341dd9mr1080641ejb.49.1706120226573; Wed, 24 Jan 2024 10:17:06 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706120226; cv=pass; d=google.com; s=arc-20160816; b=fy/66r1lgiL4vX6AAMfqvui+9Ix0SkLqUImRS4zyAQSsxkN1cYuL1Qqet+IHVdkJvI UbitQk1FrkHa58Efcw8900DWghKD0z2JWpOyRtZi3CGFytY9b2r0kJJZgOGZczTwKAk3 Hr4aJInOu2r5FmdxGfhwYznMo1kJwebmhygEpeEnW02aJiHoJwt+uS7dsWyFKVOgGY3a nSo8mrBPEZFshfaF966pnP4UkJgfsWxD5E1brJJNiXnq6zUvUUGKpaKfybJampTOzDyy TG8ORBUOMBpVrJPvhGu48a3wkG6mrin8ziah2BRHKcBQ8oyqVGOR++C/U4N95SM5Jyzn t6Jg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=2lY/aNwLINnKOMbDQO5jsCiPZIA7GFZi3ZqNd3+Td/k=; fh=OvJRnOqsMTm9XoNmEwebcqh9Ud7yh1CTeKAP84ols98=; b=GZmfWp0aO1wrB6wnm+VlCWzRp+CGI/bc71ekFp9t+Rr/PplRqlDUYRRbFZXu4mFIcU 1VIw+PzTwki7wTXSqkhJmchW7tYPlj4CrYtcFMNS5jA6/WLSaIrcAfo9pi4JYkW8VL+3 MQbwqZD9hUztk7hYf9BKDDHOCyEFXHkjpw4I3SxSdM+ybR2DLgnvT76Ihkzv/pGYKSuZ K7CzwJsY/8GxPID1bq3eKW8gwCAfgDU50PJO/imBi6q3pHcXj+nn6Gg7C4BZ2TPNq+ze HM+assYFK1WKSU7Dd02up2FFwE2Wq9+9gZpYiV6le2i2vafPVbsaPixFB/Tb4R42O4VU bE1A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LO7ctXRm; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-37024-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-37024-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id d13-20020a1709067f0d00b00a28dba30f8csi115272ejr.769.2024.01.24.10.17.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jan 2024 10:17:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-37024-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LO7ctXRm; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-37024-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-37024-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C7AA71F27474 for ; Wed, 24 Jan 2024 12:57:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2AB3777F33; Wed, 24 Jan 2024 12:56:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LO7ctXRm" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20B747763C for ; Wed, 24 Jan 2024 12:56:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.31 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706101008; cv=none; b=bvmX45HDpfSRQ5buYhOQuR9enLT2sbBTC84fJH8d5LDrEQljGIdd2qmIR7f5oovOTPf+mWmbujFj3xre3R+bFTd/bAG6VzTWPiOs9ixI/hMU2Rye/BoNG+0SOd3EVvGnix7L2+ENtpgqG52T8o+d+9Bn1nyKZl2E3Nwfcw/9oc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706101008; c=relaxed/simple; bh=OpIDWzdlg0any9JAzqVnZ+xhu2rjxA6sBoybOHgswyk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dmeZ2DMEdHvKnfYvSkUcCOIzeZs10ANFjHRRnLHVJyUyuM1ip9fKJn3rtNDIOUMy3mYdcHDiH36SXZiSkx3GlGce1A+JmG98tTo6xdGA3Rz9v7oZV54QWYwv/K3IRkMEbeexDuWtVGzk0rg5PEG2WdAx2Z6Wb0PhPLJ8fVDAHGU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LO7ctXRm; arc=none smtp.client-ip=134.134.136.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706101007; x=1737637007; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OpIDWzdlg0any9JAzqVnZ+xhu2rjxA6sBoybOHgswyk=; b=LO7ctXRmjAf/wW1P6dQkl5WRM4Qsl8CesuSHvSuk4Wh855CxiET7teZS 3okUdpz5CKOQvuOp0feVYkDEZaz12igHn5dUmO56y5LK6DzN2O3fdaEXB +7riklKq+1ZHYD7wdPG0fbaN6MgGGcmUZ9HKNEE81MwdMVWkG0Ohbi2t3 PCjE0RszXBaTyzRNlPMOC0MnyeCP0Xuyi5OANinbEzNJw736M9hRIJsOU hjKsjUpzNOZb30aazyEtahpnB1BQiolzfivF3SEOvKEQ3GLmDLo2wt5A8 3Ci1gSrxmcY1Rfnw8QwC1+MbDjCd/MkdKEq7C8zI892OngzduV4WwcDus w==; X-IronPort-AV: E=McAfee;i="6600,9927,10962"; a="466110023" X-IronPort-AV: E=Sophos;i="6.05,216,1701158400"; d="scan'208";a="466110023" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2024 04:56:46 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10962"; a="735924052" X-IronPort-AV: E=Sophos;i="6.05,216,1701158400"; d="scan'208";a="735924052" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga003.jf.intel.com with ESMTP; 24 Jan 2024 04:56:40 -0800 Received: by black.fi.intel.com (Postfix, from userid 1000) id 954A689B; Wed, 24 Jan 2024 14:56:02 +0200 (EET) From: "Kirill A. Shutemov" To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "Rafael J. Wysocki" , Peter Zijlstra , Adrian Hunter , Kuppuswamy Sathyanarayanan , Elena Reshetova , Jun Nakajima , Rick Edgecombe , Tom Lendacky , "Kalra, Ashish" , Sean Christopherson , "Huang, Kai" , Baoquan He , kexec@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv6 10/16] x86/tdx: Convert shared memory back to private on kexec Date: Wed, 24 Jan 2024 14:55:51 +0200 Message-ID: <20240124125557.493675-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240124125557.493675-1-kirill.shutemov@linux.intel.com> References: <20240124125557.493675-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788996722912491534 X-GMAIL-MSGID: 1788996722912491534 TDX guests allocate shared buffers to perform I/O. It is done by allocating pages normally from the buddy allocator and converting them to shared with set_memory_decrypted(). The second kernel has no idea what memory is converted this way. It only sees E820_TYPE_RAM. Accessing shared memory via private mapping is fatal. It leads to unrecoverable TD exit. On kexec walk direct mapping and convert all shared memory back to private. It makes all RAM private again and second kernel may use it normally. The conversion occurs in two steps: stopping new conversions and unsharing all memory. In the case of normal kexec, the stopping of conversions takes place while scheduling is still functioning. This allows for waiting until any ongoing conversions are finished. The second step is carried out when all CPUs except one are inactive and interrupts are disabled. This prevents any conflicts with code that may access shared memory. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe --- arch/x86/coco/tdx/tdx.c | 124 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 122 insertions(+), 2 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index fd212c9bad89..bb77a927a831 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -6,8 +6,10 @@ #include #include +#include #include #include +#include #include #include #include @@ -15,6 +17,7 @@ #include #include #include +#include /* MMIO direction */ #define EPT_READ 0 @@ -41,6 +44,9 @@ static atomic_long_t nr_shared; +static atomic_t conversions_in_progress; +static bool conversion_allowed = true; + static inline bool pte_decrypted(pte_t pte) { return cc_mkdec(pte_val(pte)) == pte_val(pte); @@ -726,6 +732,14 @@ static bool tdx_tlb_flush_required(bool private) static bool tdx_cache_flush_required(void) { + /* + * Avoid issuing CLFLUSH on set_memory_decrypted() if conversions + * stopped. Otherwise it can race with unshare_all_memory() and trigger + * implicit conversion to shared. + */ + if (!conversion_allowed) + return false; + /* * AMD SME/SEV can avoid cache flushing if HW enforces cache coherence. * TDX doesn't have such capability. @@ -809,12 +823,25 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) static int tdx_enc_status_change_prepare(unsigned long vaddr, int numpages, bool enc) { + atomic_inc(&conversions_in_progress); + + /* + * Check after bumping conversions_in_progress to serialize + * against tdx_kexec_stop_conversion(). + */ + if (!conversion_allowed) { + atomic_dec(&conversions_in_progress); + return -EBUSY; + } + /* * Only handle shared->private conversion here. * See the comment in tdx_early_init(). */ - if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } return 0; } @@ -826,17 +853,107 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages, * Only handle private->shared conversion here. * See the comment in tdx_early_init(). */ - if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } if (enc) atomic_long_sub(numpages, &nr_shared); else atomic_long_add(numpages, &nr_shared); + atomic_dec(&conversions_in_progress); + return 0; } +static void tdx_kexec_stop_conversion(bool crash) +{ + /* Stop new private<->shared conversions */ + conversion_allowed = false; + + /* + * Make sure conversion_allowed is cleared before checking + * conversions_in_progress. + */ + barrier(); + + /* + * Crash kernel reaches here with interrupts disabled: can't wait for + * conversions to finish. + * + * If race happened, just report and proceed. + */ + if (!crash) { + unsigned long timeout; + + /* + * Wait for in-flight conversions to complete. + * + * Do not wait more than 30 seconds. + */ + timeout = 30 * USEC_PER_SEC; + while (atomic_read(&conversions_in_progress) && timeout--) + udelay(1); + } + + if (atomic_read(&conversions_in_progress)) + pr_warn("Failed to finish shared<->private conversions\n"); +} + +static void tdx_kexec_unshare_mem(void) +{ + unsigned long addr, end; + long found = 0, shared; + + /* + * Walk direct mapping and convert all shared memory back to private, + */ + + addr = PAGE_OFFSET; + end = PAGE_OFFSET + get_max_mapped(); + + while (addr < end) { + unsigned long size; + unsigned int level; + pte_t *pte; + + pte = lookup_address(addr, &level); + size = page_level_size(level); + + if (pte && pte_decrypted(*pte)) { + int pages = size / PAGE_SIZE; + + /* + * Touching memory with shared bit set triggers implicit + * conversion to shared. + * + * Make sure nobody touches the shared range from + * now on. + */ + set_pte(pte, __pte(0)); + + if (!tdx_enc_status_changed(addr, pages, true)) { + pr_err("Failed to unshare range %#lx-%#lx\n", + addr, addr + size); + } + + found += pages; + } + + addr += size; + } + + __flush_tlb_all(); + + shared = atomic_long_read(&nr_shared); + if (shared != found) { + pr_err("shared page accounting is off\n"); + pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found); + } +} + void __init tdx_early_init(void) { struct tdx_module_args args = { @@ -896,6 +1013,9 @@ void __init tdx_early_init(void) x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = tdx_tlb_flush_required; + x86_platform.guest.enc_kexec_stop_conversion = tdx_kexec_stop_conversion; + x86_platform.guest.enc_kexec_unshare_mem = tdx_kexec_unshare_mem; + /* * TDX intercepts the RDMSR to read the X2APIC ID in the parallel * bringup low level code. That raises #VE which cannot be handled