From patchwork Wed Dec 7 01:49:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30582 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3156735wrr; Tue, 6 Dec 2022 17:51:33 -0800 (PST) X-Google-Smtp-Source: AA0mqf4jCNSnZ+oWhq2h6ukIWZAgYreebfC1r9dqd8TR1spm5G2azC1rAbEmkQAQAfv5kJfZ7rnw X-Received: by 2002:a17:906:a259:b0:7ad:ccae:a30d with SMTP id bi25-20020a170906a25900b007adccaea30dmr62275913ejb.704.1670377892851; Tue, 06 Dec 2022 17:51:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377892; cv=none; d=google.com; s=arc-20160816; b=PKDR7UuaBrqDM5HbqBygcGEwhXlt4qyMDi8Mwq2DKgNoF122BU8Nce/gpDwVReNEV2 CXcnD22Psxgu25Yu0WgDwXV5cF5l1GdUSNR3e92XZxuYn1Pgmsly7eA9KBbYNIqcXFtf cHCcIQbGnWn44kRkeuHRvKgiB7D/FU6M/iPyeBo3KAdkI/Kv4Ncy5tdTTNQJ6m/xGYy4 NCGLSMZZKuXrkMydMsXLjApp5elRF1NxVmNZVL1xCQirrdoI7TmhVk8nDZNhLjvQmTkz /G2bpfBCw8xlx9JGVDA5QJeY4i8wVCECJ2YJNUQpFEktef7tC3lg7CpHRf+kg+nkQpM3 gZ6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8l9ZiSm/kaiuS9XyH4XFj4NlMRfh5ugtCJPZ6kCsnx0=; b=MQc+ufcLSBnQNqj0X2hwJYI3wzwvISkafxzUJDCUmBfE757TWWQldOHBwm21cboNwD 1/ZkI3Ak5Bc/DiKcmYynC5uwDtrJhleSePcWAWJb5rPlwwpXryyXOu9VylqyG6saerMv amJigIxaTHnUs96mqqADw3pYoNkcfs9z8NhlM6KEwWt0PV6PSQ6LO+aOvlLTRlG/mnZq yYyr6BA9+0zkBEGF0hdPqGxaBMuUNOb3ePW1x5g1czbu1l+EUF/bVBYojHdQ9VYsj6rz zs4KjcmAy6Y+6EZ5pCpvhiqh8N1E3q2ixdCzwjqGQ28mgXfgeG9HpAvCDTd4HGq7LUhc Mzog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="h+/TP1vQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s4-20020a17090699c400b007c0fa8181c4si5100137ejn.567.2022.12.06.17.51.09; Tue, 06 Dec 2022 17:51:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="h+/TP1vQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229944AbiLGBuD (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229704AbiLGBty (ORCPT ); Tue, 6 Dec 2022 20:49:54 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A9D15289F; Tue, 6 Dec 2022 17:49:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377793; x=1701913793; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wTnYWmM1Z8lnXpkf3CCIP9h4eff3UeEwYWr89BwyTuQ=; b=h+/TP1vQM72IKetOdmHNO7dLaaAB/ZfiTzIEWYJk64r18gktSYqvaotZ 8shMTYTWjEuz+yVV+Ar1Wb9/eJ4S4BFKWJR4NZ0lv8jonYmOvd7wlplf9 yxfR6ThOTOqlGHMeCJTD2idi3Q45N7pgWr5+XeEaAQgKnAdE41QdJB4ca GsvN3iOwewViuCfhCPFlNEXrDIiarPNQJiVwnNlvsqo6XZylHeDTCcKfO wo2YnV5DGCNvkWV8AnfFejtWlYEltRfgUqW7orOavzicpij6NOiId1Jtz kdc9X2hD2RZQSnfyWuySHY1j2DzIIIoAuTcGGAdGCG6jyHOMrQXAZ4kQu w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494468" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494468" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:51 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427602" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427602" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:41 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 35405109C85; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Mike Rapoport , Dave Hansen Subject: [PATCHv8 01/14] x86/boot: Centralize __pa()/__va() definitions Date: Wed, 7 Dec 2022 04:49:20 +0300 Message-Id: <20221207014933.8435-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518168911831569?= X-GMAIL-MSGID: =?utf-8?q?1751518168911831569?= Replace multiple __pa()/__va() definitions with a single one in misc.h. Signed-off-by: Kirill A. Shutemov Reviewed-by: David Hildenbrand Reviewed-by: Mike Rapoport Reviewed-by: Dave Hansen --- arch/x86/boot/compressed/ident_map_64.c | 8 -------- arch/x86/boot/compressed/misc.h | 9 +++++++++ arch/x86/boot/compressed/sev.c | 2 -- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c index d4a314cc50d6..56550f289118 100644 --- a/arch/x86/boot/compressed/ident_map_64.c +++ b/arch/x86/boot/compressed/ident_map_64.c @@ -8,14 +8,6 @@ * Copyright (C) 2016 Kees Cook */ -/* - * Since we're dealing with identity mappings, physical and virtual - * addresses are the same, so override these defines which are ultimately - * used by the headers in misc.h. - */ -#define __pa(x) ((unsigned long)(x)) -#define __va(x) ((void *)((unsigned long)(x))) - /* No PAGE_TABLE_ISOLATION support needed either: */ #undef CONFIG_PAGE_TABLE_ISOLATION diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index 62208ec04ca4..d37d612a6390 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -19,6 +19,15 @@ /* cpu_feature_enabled() cannot be used this early */ #define USE_EARLY_PGTABLE_L5 +/* + * Boot stub deals with identity mappings, physical and virtual addresses are + * the same, so override these defines. + * + * will not define them if they are already defined. + */ +#define __pa(x) ((unsigned long)(x)) +#define __va(x) ((void *)((unsigned long)(x))) + #include #include #include diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index c93930d5ccbd..e4d3a2da2eb9 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -104,9 +104,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, } #undef __init -#undef __pa #define __init -#define __pa(x) ((unsigned long)(x)) #define __BOOT_COMPRESSED From patchwork Wed Dec 7 01:49:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30583 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3156765wrr; Tue, 6 Dec 2022 17:51:39 -0800 (PST) X-Google-Smtp-Source: AA0mqf7aewtTmL5yytC0dsOmpDVN+uESaChg9mwN2xyv02F6X3XQ80y7XTD0OUnn3Of5zVo/dBc/ X-Received: by 2002:a17:906:9c8a:b0:7bf:6698:d444 with SMTP id fj10-20020a1709069c8a00b007bf6698d444mr36186740ejc.548.1670377898986; Tue, 06 Dec 2022 17:51:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377898; cv=none; d=google.com; s=arc-20160816; b=Y9Um9ar9ZipadP6jaEgifnWx5okmEaTDo+7DM7lbKd6OIPa4ryZ19qxvtOZNSgfdYo 9+lLmfGhDELCld+73pn/e+u04o6CYU3Hp6Wkw0SWWeJpdmWa26/BZ+P7F+p5CKpLlvBL a/BKoL0y/IhORiE2HM44X7buPlXiwGLyPGCwUVKmFinOUXdlI9xqsrvsnN8aRBIztZMu qY3XPErh4VK1KxlG4FZMZjuHRjwzMNcsLQ1b7seSImUh9VsoGMYR3tAJCpdBb6xIK9HC MgJfNoAVO9nyHcj5eoZjJADlWAx2PtbtoFAd7o6GHXik+jjnSsEhsyh2idlBc2Qf/7IN bxyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XU50Q7zEBx7sORtEqB9KRZC433UJdQp10cWahFIOmAs=; b=Eg/qcYHgO/51cWlkmhQ1QWvnaWpiJz/PLuz9uOxhkMGoLJClqKCZs9+gHltuaY7Dzx ZVsSgxSxGQYwq/LswPRASnqF+WXIfbSEykQ5YlbosnTOTn+ayTZbcKUxXq4QmuloUkes d2gxfxz9FsQQsb7/roIIJ6EHv8IITn7BOQP5CCTDEwYUgS+MoHpHPeW9UzSPBqqqvTty Ynr0zR9FkyqzO9epW84LjDSjNkb1Iv7cyo55EqybINwpftsacvjtvxXm8dpB32dos/1g nTeLO8i/9941dDJUqLFXKP76AvpiSGqYHWy6HE1BCd3VpP66hldOHXmsyRa6o7wnc2nS W/nA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QIpWqZZS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dp16-20020a170906c15000b007ae86742c39si8787765ejc.504.2022.12.06.17.51.15; Tue, 06 Dec 2022 17:51:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QIpWqZZS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229954AbiLGBuG (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229726AbiLGBty (ORCPT ); Tue, 6 Dec 2022 20:49:54 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F9BE528B0; Tue, 6 Dec 2022 17:49:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377792; x=1701913792; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=21DVTJoUt4qdmIcbuPAPUd5xHsbd/8yH6qj8tM2+hM0=; b=QIpWqZZSZLnPwLQSTy64HsRKNlHvDctJvfl/Bx451CR7TO2rFm7gYLoZ zVN3coa/uQ1tCD0nvt/V1yftAgR4u23XAzNIXmliWtpbrs9GuV6YBttjv WXV3eYAmHX5MerzJeHHdizBzHIPh29MK2XebVmjxnObeQZ8Y0lxvjyhB+ QOl7WEXTCNoXTzupaWtGXpdDxPNb70CTEcOv594++hj7gozhFM7HML9D9 MIeFYcJcWKMoqHvLcaqZpWLE+1jjMoUVdanRQ/TmLvTBCR2vDV9SA2Mjd GKidijuinMNn4OHa48rpDFOo0RgLf1dxvx+3iGjgqWkWaRBBZ31+fhOJu A==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494450" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494450" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427598" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427598" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:41 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 40917109C86; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Mike Rapoport Subject: [PATCHv8 02/14] mm: Add support for unaccepted memory Date: Wed, 7 Dec 2022 04:49:21 +0300 Message-Id: <20221207014933.8435-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518175435284489?= X-GMAIL-MSGID: =?utf-8?q?1751518175435284489?= UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform. There are several ways kernel can deal with unaccepted memory: 1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time. Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point. 2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time. On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted. 3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time. This approach cannot function on its own. It is an extension of #2: background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that. The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience. Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands. Support of unaccepted memory requires a few changes in core-mm code: - memblock has to accept memory on allocation; - page allocator has to accept memory on the first allocation of the page; Memblock change is trivial. The page allocator is modified to accept pages. New memory gets accepted before putting pages on free lists. It is done lazily: only accept new pages when we run out of already accepted memory. Architecture has to provide two helpers if it wants to support unaccepted memory: - accept_memory() makes a range of physical addresses accepted. - range_contains_unaccepted_memory() checks anything within the range of physical addresses requires acceptance. Signed-off-by: Kirill A. Shutemov Acked-by: Mike Rapoport # memblock --- include/linux/mmzone.h | 5 ++ include/linux/page-flags.h | 24 ++++++++ mm/internal.h | 12 ++++ mm/memblock.c | 9 +++ mm/page_alloc.c | 119 +++++++++++++++++++++++++++++++++++++ 5 files changed, 169 insertions(+) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5f74891556f3..da335381e63f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -822,6 +822,11 @@ struct zone { /* free areas of different sizes */ struct free_area free_area[MAX_ORDER]; +#ifdef CONFIG_UNACCEPTED_MEMORY + /* pages to be accepted */ + struct list_head unaccepted_pages; +#endif + /* zone flags, see below */ unsigned long flags; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 0b0ae5084e60..ce953be8fe10 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -941,6 +941,7 @@ static inline bool is_page_hwpoison(struct page *page) #define PG_offline 0x00000100 #define PG_table 0x00000200 #define PG_guard 0x00000400 +#define PG_unaccepted 0x00000800 #define PageType(page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) @@ -966,6 +967,18 @@ static __always_inline void __ClearPage##uname(struct page *page) \ page->page_type |= PG_##lname; \ } +#define PAGE_TYPE_OPS_FALSE(uname) \ +static __always_inline int Page##uname(struct page *page) \ +{ \ + return false; \ +} \ +static __always_inline void __SetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ClearPage##uname(struct page *page) \ +{ \ +} + /* * PageBuddy() indicates that the page is free and in the buddy system * (see mm/page_alloc.c). @@ -996,6 +1009,17 @@ PAGE_TYPE_OPS(Buddy, buddy) */ PAGE_TYPE_OPS(Offline, offline) +/* + * PageUnaccepted() indicates that the page has to be "accepted" before it can + * be read or written. The page allocator must call accept_page() before + * touching the page or returning it to the caller. + */ +#ifdef CONFIG_UNACCEPTED_MEMORY +PAGE_TYPE_OPS(Unaccepted, unaccepted) +#else +PAGE_TYPE_OPS_FALSE(Unaccepted) +#endif + extern void page_offline_freeze(void); extern void page_offline_thaw(void); extern void page_offline_begin(void); diff --git a/mm/internal.h b/mm/internal.h index 6b7ef495b56d..8ef4f88608ad 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -856,4 +856,16 @@ static inline bool vma_soft_dirty_enabled(struct vm_area_struct *vma) return !(vma->vm_flags & VM_SOFTDIRTY); } +#ifndef CONFIG_UNACCEPTED_MEMORY +static inline bool range_contains_unaccepted_memory(phys_addr_t start, + phys_addr_t end) +{ + return false; +} + +static inline void accept_memory(phys_addr_t start, phys_addr_t end) +{ +} +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/memblock.c b/mm/memblock.c index 511d4783dcf1..3bc404a5352a 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1423,6 +1423,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, */ kmemleak_alloc_phys(found, size, 0); + /* + * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, + * require memory to be accepted before it can be used by the + * guest. + * + * Accept the memory of the allocated buffer. + */ + accept_memory(found, found + size); + return found; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6e60657875d3..6d597e833a73 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -450,6 +450,11 @@ EXPORT_SYMBOL(nr_online_nodes); int page_group_by_mobility_disabled __read_mostly; +#ifdef CONFIG_UNACCEPTED_MEMORY +/* Counts number of zones with unaccepted pages. */ +static DEFINE_STATIC_KEY_FALSE(unaccepted_pages); +#endif + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * During boot we initialize deferred pages on-demand, as needed, but once @@ -1043,12 +1048,15 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, { struct free_area *area = &zone->free_area[order]; + VM_BUG_ON_PAGE(PageUnevictable(page), page); list_move_tail(&page->buddy_list, &area->free_list[migratetype]); } static inline void del_page_from_free_list(struct page *page, struct zone *zone, unsigned int order) { + VM_BUG_ON_PAGE(PageUnevictable(page), page); + /* clear reported state and update reported page count */ if (page_reported(page)) __ClearPageReported(page); @@ -1728,6 +1736,97 @@ static void __free_pages_ok(struct page *page, unsigned int order, __count_vm_events(PGFREE, 1 << order); } +static bool page_contains_unaccepted(struct page *page, unsigned int order) +{ + phys_addr_t start = page_to_phys(page); + phys_addr_t end = start + (PAGE_SIZE << order); + + return range_contains_unaccepted_memory(start, end); +} + +static void accept_page(struct page *page, unsigned int order) +{ + phys_addr_t start = page_to_phys(page); + + accept_memory(start, start + (PAGE_SIZE << order)); +} + +#ifdef CONFIG_UNACCEPTED_MEMORY + +static bool try_to_accept_memory(struct zone *zone) +{ + unsigned long flags, order; + struct page *page; + bool last = false; + int migratetype; + + if (!static_branch_unlikely(&unaccepted_pages)) + return false; + + spin_lock_irqsave(&zone->lock, flags); + page = list_first_entry_or_null(&zone->unaccepted_pages, + struct page, lru); + if (!page) { + spin_unlock_irqrestore(&zone->lock, flags); + return false; + } + + list_del(&page->lru); + last = list_empty(&zone->unaccepted_pages); + + order = page->private; + VM_BUG_ON(order > MAX_ORDER || order < pageblock_order); + + migratetype = get_pfnblock_migratetype(page, page_to_pfn(page)); + __mod_zone_freepage_state(zone, -1 << order, migratetype); + spin_unlock_irqrestore(&zone->lock, flags); + + if (last) + static_branch_dec(&unaccepted_pages); + + accept_page(page, order); + __ClearPageUnaccepted(page); + __free_pages_ok(page, order, FPI_TO_TAIL | FPI_SKIP_KASAN_POISON); + + return true; +} + +static void __free_unaccepted(struct page *page, unsigned int order) +{ + struct zone *zone = page_zone(page); + unsigned long flags; + int migratetype; + bool first = false; + + VM_BUG_ON(order > MAX_ORDER || order < pageblock_order); + __SetPageUnaccepted(page); + page->private = order; + + spin_lock_irqsave(&zone->lock, flags); + first = list_empty(&zone->unaccepted_pages); + migratetype = get_pfnblock_migratetype(page, page_to_pfn(page)); + list_add_tail(&page->lru, &zone->unaccepted_pages); + __mod_zone_freepage_state(zone, 1 << order, migratetype); + spin_unlock_irqrestore(&zone->lock, flags); + + if (first) + static_branch_inc(&unaccepted_pages); +} + +#else + +static bool try_to_accept_memory(struct zone *zone) +{ + return false; +} + +static void __free_unaccepted(struct page *page, unsigned int order) +{ + BUILD_BUG(); +} + +#endif /* CONFIG_UNACCEPTED_MEMORY */ + void __free_pages_core(struct page *page, unsigned int order) { unsigned int nr_pages = 1 << order; @@ -1750,6 +1849,13 @@ void __free_pages_core(struct page *page, unsigned int order) atomic_long_add(nr_pages, &page_zone(page)->managed_pages); + if (page_contains_unaccepted(page, order)) { + if (order >= pageblock_order) + return __free_unaccepted(page, order); + else + accept_page(page, order); + } + /* * Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. @@ -1910,6 +2016,9 @@ static void __init deferred_free_range(unsigned long pfn, return; } + /* Accept chunks smaller than page-block upfront */ + accept_memory(PFN_PHYS(pfn), PFN_PHYS(pfn + nr_pages)); + for (i = 0; i < nr_pages; i++, page++, pfn++) { if (pageblock_aligned(pfn)) set_pageblock_migratetype(page, MIGRATE_MOVABLE); @@ -4247,6 +4356,9 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (try_to_accept_memory(zone)) + goto try_this_zone; + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * Watermark failed for this zone, but see if we can @@ -4299,6 +4411,9 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, return page; } else { + if (try_to_accept_memory(zone)) + goto try_this_zone; + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ if (static_branch_unlikely(&deferred_pages)) { @@ -6935,6 +7050,10 @@ static void __meminit zone_init_free_lists(struct zone *zone) INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free = 0; } + +#ifdef CONFIG_UNACCEPTED_MEMORY + INIT_LIST_HEAD(&zone->unaccepted_pages); +#endif } /* From patchwork Wed Dec 7 01:49:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30588 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157281wrr; Tue, 6 Dec 2022 17:53:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf7egfCul29Ege5yhT8OJE0hSX52wTajVjg/FQAKcOx4OKSXDRERw2pLJZbVNs99jB+gbr4+ X-Received: by 2002:a17:906:a20f:b0:7bf:f0e9:1cde with SMTP id r15-20020a170906a20f00b007bff0e91cdemr34985939ejy.512.1670377988760; Tue, 06 Dec 2022 17:53:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377988; cv=none; d=google.com; s=arc-20160816; b=FP5JfmXMlDZua4o8fKrOL+N9NvCYdxnXZTUvkV8K/RaKJr75ByL5tX9SG48DVHc5Fq O9uH+hHrHqzHh910OH3YYPnPasUNHAhycfOncNCbMXnCjIi6zfbZhu7yk9ku5T235L/R ZDusPjC0D4e7fjeoNgSbJB3ZMzNk4Pf72s9q2FNvH5g8U42j3MhVMgocrWug/5CX2B8e obveZx008j4yFNVQGazSAKQRnqfH3mxf+64B0mXXKsi3LJEiQbJnPIfAz8//uz8dO5QN bnMFe5qeYtALHJIMm6uS2dNG3KsaL5kazOK0tjLq/Ua3VhrQBt/gAMNlKyGblHaGCQw5 Dq5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rd22VUVbf1U3LUqt73jzmwUFWREGLHRluRioAV2cuRc=; b=dIin0+hNq+peDymCRi0FfWh+uY3czIq5WKE7corIc7NW2XPOBwg8NjMwM129GoEYVy knepvo2zZl9XyAraWt+HX3XW+eFsYmIi1OONlW3t4Ag7SZMp6SbdUTGM11cpASfBuIHC fLTxKOWm+1Ez/tJa3pZ4sxpve8n4b0ixO2dcUxVySsqEpAkMruAfa60FtINjp2KaPsr7 KnkHUMlidHvf79iWz5deRT2DXIvfRvpcuE5zn+gv1fx2Qzc5+tXUr7p3OeRNF4HkYlQM N5B0n14GpcHF9p+kVuQzGIrZp378uQI9QRf6CPeqfx5LfuEU0EVB4p1YVqNVGm1gfYhk nOiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XVQJLrRa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b20-20020a056402279400b0046b58b0497asi3557912ede.314.2022.12.06.17.52.44; Tue, 06 Dec 2022 17:53:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XVQJLrRa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229939AbiLGBt7 (ORCPT + 99 others); Tue, 6 Dec 2022 20:49:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbiLGBtx (ORCPT ); Tue, 6 Dec 2022 20:49:53 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AD8D528AB; Tue, 6 Dec 2022 17:49:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377792; x=1701913792; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=u+bYwyFm4XjH9XBhSQUvtr3jzM1TLHyKc0CMjo+kLEU=; b=XVQJLrRaUd5neqrX2yVzjILOLK/KWr25vuIzqj6+2JP/dk1dr7loSULx XRo8aK9CwgqRNpdAz0ENxQvc44yvIeWtxlABOJrOXiWnK4Iutm8J7xwoM pFkYe1TwrT9RZ4+CxGfIWF32fmqd/49lFGOQuIn7kFM2SzAaviTcBzQqs MzuhzvUqM2qyvFLSjGNlUwHT2c8ay1WKkDRKXdhV5u87UFbNtytOjIIM9 1UminUgMO2lrDiHRUJOhJK9QEYKvMUW6gWnx/rTbT5QjS2ZxQsSEi/rGa yG74sdN2oU4gsSjG2NZNNrwwvUUt81hVgabDOadog6PAfKVPIT5ve08Ce g==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494434" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494434" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427593" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427593" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:41 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 4782B109C87; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 03/14] mm: Report unaccepted memory in meminfo Date: Wed, 7 Dec 2022 04:49:22 +0300 Message-Id: <20221207014933.8435-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518270005584871?= X-GMAIL-MSGID: =?utf-8?q?1751518270005584871?= Track amount of unaccepted memory and report it in /proc/meminfo and in node meminfo. Signed-off-by: Kirill A. Shutemov --- drivers/base/node.c | 7 +++++++ fs/proc/meminfo.c | 5 +++++ include/linux/mmzone.h | 3 +++ mm/page_alloc.c | 2 ++ mm/vmstat.c | 1 + 5 files changed, 18 insertions(+) diff --git a/drivers/base/node.c b/drivers/base/node.c index faf3597a96da..ca6f0590be21 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -448,6 +448,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d ShmemPmdMapped: %8lu kB\n" "Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n" +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + "Node %d Unaccepted: %8lu kB\n" #endif , nid, K(node_page_state(pgdat, NR_FILE_DIRTY)), @@ -477,6 +480,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid, K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED)), nid, K(node_page_state(pgdat, NR_FILE_THPS)), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED)) +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid, K(node_page_state(pgdat, NR_UNACCEPTED)) #endif ); len += hugetlb_report_node_meminfo(buf, len, nid); diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 440960110a42..789b77c7b6df 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -155,6 +155,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES)); #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + show_val_kb(m, "Unaccepted: ", + global_node_page_state(NR_UNACCEPTED)); +#endif + hugetlb_report_meminfo(m); arch_report_meminfo(m); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index da335381e63f..9c762e8175fc 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -198,6 +198,9 @@ enum node_stat_item { NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */ NR_KERNEL_STACK_KB, /* measured in KiB */ +#ifdef CONFIG_UNACCEPTED_MEMORY + NR_UNACCEPTED, +#endif #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */ #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6d597e833a73..e80e8d398863 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1779,6 +1779,7 @@ static bool try_to_accept_memory(struct zone *zone) migratetype = get_pfnblock_migratetype(page, page_to_pfn(page)); __mod_zone_freepage_state(zone, -1 << order, migratetype); + __mod_node_page_state(page_pgdat(page), NR_UNACCEPTED, -1 << order); spin_unlock_irqrestore(&zone->lock, flags); if (last) @@ -1807,6 +1808,7 @@ static void __free_unaccepted(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, page_to_pfn(page)); list_add_tail(&page->lru, &zone->unaccepted_pages); __mod_zone_freepage_state(zone, 1 << order, migratetype); + __mod_node_page_state(page_pgdat(page), NR_UNACCEPTED, 1 << order); spin_unlock_irqrestore(&zone->lock, flags); if (first) diff --git a/mm/vmstat.c b/mm/vmstat.c index b2371d745e00..fb15213be374 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1230,6 +1230,7 @@ const char * const vmstat_text[] = { "nr_foll_pin_acquired", "nr_foll_pin_released", "nr_kernel_stack", + "nr_unaccepted", #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack", #endif From patchwork Wed Dec 7 01:49:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30585 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3156847wrr; Tue, 6 Dec 2022 17:51:53 -0800 (PST) X-Google-Smtp-Source: AA0mqf6JROss/99joOcZHUYrQDk0Tn/T8l5bf5Tciru2nqNLZ5dMrkSeLBijNi8QsKAaNsgUCZMk X-Received: by 2002:a17:906:1e55:b0:7ad:a0cb:f79e with SMTP id i21-20020a1709061e5500b007ada0cbf79emr57246218ejj.458.1670377912918; Tue, 06 Dec 2022 17:51:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377912; cv=none; d=google.com; s=arc-20160816; b=lBYNRNftDP39i86D6dWvjiOfi4jKU3Sz4VsWsRSCFQi0ePjfAIPRp5FLWo2rW95Xor 7bq5b+jd8XhB0ZbnUuWH9mUYGsA58ETCQk/rOf6A8/o5yFoFp7k9HjRXI05g21sX3rrK llQogFJnYJORsvXvDc/m7NCZIdyfnhaoWhsmbQzSz9Wg6vuHjb908diJt2dPErQPN2u5 Up5IukNuojZDeJgVG2Ml577b8pGZVfO4AAANJzx+xx8K4ACSudMeWgAUQaf+zmT65gRy 0uEMiyiOGSvE4TwA5onLQVzmP+LHNjKd4pQ7XiozMBocD/S8E8vStsr4yzlX06JQC+c+ tH3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3RNkOBOmz8kpmMAsTlTOkOKe/jwUeV5spebiVYy9bgk=; b=YgfLnNAA8KOjqk23MgWY/w5DScu8vW4Wl+N2XMz/qipe6ojrj+JzaSN2TP6dVqRI5b 1lkQ+S6RWau7oVq0+tbTTEVDbKa9PfecJjyz0kM9S83Ml0xygVkQ+4A2C2JS9Hxod2Ex pPUWiGqRP37GrL+oy6q4kKiPbmMR9WSHmbqcDwvy/FqOrLkMMQYCzuo/onWYaZE9CVPh 1Ck/Ge8Nmv8wCF+E7uOxdlY0Gb9kZMBUMyjcSd6YXHsbVebxBVkLQORcypmBBkw8jfm7 m2quOftC8JYqhugVKxVtvcXRcifUbmhwz+62xWzxGCyP1nRlU9HoJmP9k8XW8ya5ERDk j9ig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IF5NgYFO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dt3-20020a170907728300b00783ddc8b643si16061710ejc.317.2022.12.06.17.51.30; Tue, 06 Dec 2022 17:51:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IF5NgYFO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229886AbiLGBt5 (ORCPT + 99 others); Tue, 6 Dec 2022 20:49:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229678AbiLGBtw (ORCPT ); Tue, 6 Dec 2022 20:49:52 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79133528A4; Tue, 6 Dec 2022 17:49:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377791; x=1701913791; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nVCMUIvJT1/LroShuz37RCPWW1tr+mbN1QZ4d2Ez7Bk=; b=IF5NgYFOEwrYXAsrXQkDrfdek7Krc+gGQTtWj4NN3f+dUGk4eTfndi+d VF4H42xiDAu/pnfwx9drc+f+gYyZTPALFxB33S6OmWo8FuHGvKued4oaq WufT5GPEKVxb/hRLWPS0Qn/hmRHZi1HoF8ULws1iZ2c5Mp7O2qn8S2wRN 3Xmg6rurNBPmZEtEdD+c3jO2KnIZQFd0/wP0op7qmNHp44guLN8QQA2ou A0wDtPL5dYNn7IM3bCBdu2/MS+7k1fLjo492+a9zWZzL4ElwCQrq+VEci /YLe2qb5PUQbL84YWfQunDVeyda2X86qgG7ipNpD8f7YKev94GdWtPdgB w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494412" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494412" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:49 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427591" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427591" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:41 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 524FF109C88; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Borislav Petkov Subject: [PATCHv8 04/14] efi/x86: Get full memory map in allocate_e820() Date: Wed, 7 Dec 2022 04:49:23 +0300 Message-Id: <20221207014933.8435-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518190743525752?= X-GMAIL-MSGID: =?utf-8?q?1751518190743525752?= Currently allocate_e820() is only interested in the size of map and size of memory descriptor to determine how many e820 entries the kernel needs. UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address. Modify allocate_e820() to get a full memory map. Signed-off-by: Kirill A. Shutemov Reviewed-by: Borislav Petkov --- drivers/firmware/efi/libstub/x86-stub.c | 26 +++++++++++-------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index a0bfd31358ba..fff81843169c 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -681,28 +681,24 @@ static efi_status_t allocate_e820(struct boot_params *params, struct setup_data **e820ext, u32 *e820ext_size) { - unsigned long map_size, desc_size, map_key; + struct efi_boot_memmap *map; efi_status_t status; - __u32 nr_desc, desc_version; + __u32 nr_desc; - /* Only need the size of the mem map and size of each mem descriptor */ - map_size = 0; - status = efi_bs_call(get_memory_map, &map_size, NULL, &map_key, - &desc_size, &desc_version); - if (status != EFI_BUFFER_TOO_SMALL) - return (status != EFI_SUCCESS) ? status : EFI_UNSUPPORTED; - - nr_desc = map_size / desc_size + EFI_MMAP_NR_SLACK_SLOTS; + status = efi_get_memory_map(&map, false); + if (status != EFI_SUCCESS) + return status; - if (nr_desc > ARRAY_SIZE(params->e820_table)) { - u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table); + nr_desc = map->map_size / map->desc_size; + if (nr_desc > ARRAY_SIZE(params->e820_table) - EFI_MMAP_NR_SLACK_SLOTS) { + u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table) + + EFI_MMAP_NR_SLACK_SLOTS; status = alloc_e820ext(nr_e820ext, e820ext, e820ext_size); - if (status != EFI_SUCCESS) - return status; } - return EFI_SUCCESS; + efi_bs_call(free_pool, map); + return status; } struct exit_boot_struct { From patchwork Wed Dec 7 01:49:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30590 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157500wrr; Tue, 6 Dec 2022 17:53:54 -0800 (PST) X-Google-Smtp-Source: AA0mqf6fH0ZfFCfjk1h/5nXL/GgT2iRPMMXGXtyC2FOOCGVZre9sfcuWgdhc88a0YIy8yFxinkU5 X-Received: by 2002:a17:906:c458:b0:7c0:e23f:17cd with SMTP id ck24-20020a170906c45800b007c0e23f17cdmr11446255ejb.491.1670378034613; Tue, 06 Dec 2022 17:53:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378034; cv=none; d=google.com; s=arc-20160816; b=v6hHChHYkBvd5gzJw8w8feM2QgJtSlVDghu95jizO8pPhbN7UbkdWckMYKTPbQMM+N oHJIJSLBFun8Ew3sAx0T//Kb8z6EBd71rupr0swRI3fRXBdIWwqWxbjz0O4BwmV3z8xg D4ZcpC2nfwqPQtk/EuGZAVWSZpAeyd31moVDEo+Opm5Ce5ZJ/GTdJSbqlD1RbnT3hGBl Dk8400Z5FR1Oh6cVZ1IL/gZK730a+Yu/PZr9PW69z3WDx4odbHlnNmhTX13Io5cnUyP3 gV/7lhXFTuOQeCyYM6WAp4uYkXPOx3DiX5MdR5sOBkwKETfHsRCCdEn43l3Opx+Acq0p UOdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L6Z8NDQJ3rw3pdP+BJgzgZNXIv09nNrj+n1AYp23yxA=; b=BY9uhPJ8nrGlJGT+Krv4lXmt9pEQyBOeC2DfdVNEcFHakJGIGkxBnoWx5jskF0vpaO DKHkWYA+inzE86X/ZgsIVOkgODd1YJRdBU0UhuB5f0QiUPaphth37eMRA3NlK7nF8syC HpQn2sPYvhhSW52H+6DTdH4jhklfyfebEewvplGOd5V0BHGRduukHoBX0+LOpqC4+j2i 0OlO3ZYgYw2O7SzmeteMg3AdAG6Pq55zQ1ILiq4SgpR5ORCU5WfQq9XxQfMhRvXStK/1 xVU4FSEsb2NcoUunmZxQyZ0UQuqXTcynbbsdgsUUhC+v98prN1mGladYsF53N20fXrXq 5Wog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=J9G8vONH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sc16-20020a1709078a1000b007ae83c51922si14426946ejc.45.2022.12.06.17.53.30; Tue, 06 Dec 2022 17:53:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=J9G8vONH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230088AbiLGBvG (ORCPT + 99 others); Tue, 6 Dec 2022 20:51:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229968AbiLGBui (ORCPT ); Tue, 6 Dec 2022 20:50:38 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76037537E7; Tue, 6 Dec 2022 17:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377804; x=1701913804; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sl5TbMqiilwCWCUX0aBF82Zck/XC1oO5+a1IlLBkAUw=; b=J9G8vONHCf5f8f1o3t/zUhJU/d+Z0BLJn3GMxdfhxswdjhtJjxbWmgb5 qqQgK94R8P0SZQdWE+pVVhcFpl/WRKwpppWpZEqvHw6DePkblAyM5gqbS Q+CX5QrUJ8s6zRj+odLbSc661Koj2aZ2ABsPG8gHx0T+RsvoDIo93PRQW MYGRyCvAsF/KWvrBGcFanMmTU/tGgMQ7QKHwMqrlK17ApbcjdgNil4v0j rymH5JvfJCKzCzS7j76HL3ygPwmBH6A9f4PHPrPtxhF0nPx1SfQ5JCChc Xp7FulMo2AkitZBQA6MtJQECK/7z+QxTaJyZdY3GAR5fhPQxGd8W/tmIi A==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494565" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494565" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427706" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427706" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:52 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 5DFB3109C89; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 05/14] x86/boot: Add infrastructure required for unaccepted memory support Date: Wed, 7 Dec 2022 04:49:24 +0300 Message-Id: <20221207014933.8435-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518317763830787?= X-GMAIL-MSGID: =?utf-8?q?1751518317763830787?= Pull functionality from the main kernel headers and lib/ that is required for unaccepted memory support. This is preparatory patch. The users for the functionality will come in following patches. Signed-off-by: Kirill A. Shutemov Reviewed-by: Borislav Petkov (AMD) --- arch/x86/boot/bitops.h | 40 ++++++++++++ arch/x86/boot/compressed/align.h | 14 +++++ arch/x86/boot/compressed/bitmap.c | 43 +++++++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++++++ arch/x86/boot/compressed/find.c | 54 ++++++++++++++++ arch/x86/boot/compressed/find.h | 79 ++++++++++++++++++++++++ arch/x86/boot/compressed/math.h | 37 +++++++++++ arch/x86/boot/compressed/minmax.h | 61 ++++++++++++++++++ arch/x86/boot/compressed/pgtable_types.h | 25 ++++++++ 10 files changed, 438 insertions(+) create mode 100644 arch/x86/boot/compressed/align.h create mode 100644 arch/x86/boot/compressed/bitmap.c create mode 100644 arch/x86/boot/compressed/bitmap.h create mode 100644 arch/x86/boot/compressed/bits.h create mode 100644 arch/x86/boot/compressed/find.c create mode 100644 arch/x86/boot/compressed/find.h create mode 100644 arch/x86/boot/compressed/math.h create mode 100644 arch/x86/boot/compressed/minmax.h create mode 100644 arch/x86/boot/compressed/pgtable_types.h diff --git a/arch/x86/boot/bitops.h b/arch/x86/boot/bitops.h index 8518ae214c9b..38badf028543 100644 --- a/arch/x86/boot/bitops.h +++ b/arch/x86/boot/bitops.h @@ -41,4 +41,44 @@ static inline void set_bit(int nr, void *addr) asm("btsl %1,%0" : "+m" (*(u32 *)addr) : "Ir" (nr)); } +static __always_inline void __set_bit(long nr, volatile unsigned long *addr) +{ + asm volatile(__ASM_SIZE(bts) " %1,%0" : : "m" (*(volatile long *) addr), + "Ir" (nr) : "memory"); +} + +static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) +{ + asm volatile(__ASM_SIZE(btr) " %1,%0" : : "m" (*(volatile long *) addr), + "Ir" (nr) : "memory"); +} + +/** + * __ffs - find first set bit in word + * @word: The word to search + * + * Undefined if no bit exists, so code should check against 0 first. + */ +static __always_inline unsigned long __ffs(unsigned long word) +{ + asm("rep; bsf %1,%0" + : "=r" (word) + : "rm" (word)); + return word; +} + +/** + * ffz - find first zero bit in word + * @word: The word to search + * + * Undefined if no zero exists, so code should check against ~0UL first. + */ +static __always_inline unsigned long ffz(unsigned long word) +{ + asm("rep; bsf %1,%0" + : "=r" (word) + : "r" (~word)); + return word; +} + #endif /* BOOT_BITOPS_H */ diff --git a/arch/x86/boot/compressed/align.h b/arch/x86/boot/compressed/align.h new file mode 100644 index 000000000000..7ccabbc5d1b8 --- /dev/null +++ b/arch/x86/boot/compressed/align.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_ALIGN_H +#define BOOT_ALIGN_H +#define _LINUX_ALIGN_H /* Inhibit inclusion of */ + +/* @a is a power of 2 value */ +#define ALIGN(x, a) __ALIGN_KERNEL((x), (a)) +#define ALIGN_DOWN(x, a) __ALIGN_KERNEL((x) - ((a) - 1), (a)) +#define __ALIGN_MASK(x, mask) __ALIGN_KERNEL_MASK((x), (mask)) +#define PTR_ALIGN(p, a) ((typeof(p))ALIGN((unsigned long)(p), (a))) +#define PTR_ALIGN_DOWN(p, a) ((typeof(p))ALIGN_DOWN((unsigned long)(p), (a))) +#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0) + +#endif diff --git a/arch/x86/boot/compressed/bitmap.c b/arch/x86/boot/compressed/bitmap.c new file mode 100644 index 000000000000..789ecadeb521 --- /dev/null +++ b/arch/x86/boot/compressed/bitmap.c @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include "bitmap.h" + +void __bitmap_set(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_set = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_set = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_set >= 0) { + *p |= mask_to_set; + len -= bits_to_set; + bits_to_set = BITS_PER_LONG; + mask_to_set = ~0UL; + p++; + } + if (len) { + mask_to_set &= BITMAP_LAST_WORD_MASK(size); + *p |= mask_to_set; + } +} + +void __bitmap_clear(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_clear = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_clear = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_clear >= 0) { + *p &= ~mask_to_clear; + len -= bits_to_clear; + bits_to_clear = BITS_PER_LONG; + mask_to_clear = ~0UL; + p++; + } + if (len) { + mask_to_clear &= BITMAP_LAST_WORD_MASK(size); + *p &= ~mask_to_clear; + } +} diff --git a/arch/x86/boot/compressed/bitmap.h b/arch/x86/boot/compressed/bitmap.h new file mode 100644 index 000000000000..35357f5feda2 --- /dev/null +++ b/arch/x86/boot/compressed/bitmap.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_BITMAP_H +#define BOOT_BITMAP_H +#define __LINUX_BITMAP_H /* Inhibit inclusion of */ + +#include "../bitops.h" +#include "../string.h" +#include "align.h" + +#define BITMAP_MEM_ALIGNMENT 8 +#define BITMAP_MEM_MASK (BITMAP_MEM_ALIGNMENT - 1) + +#define BITMAP_FIRST_WORD_MASK(start) (~0UL << ((start) & (BITS_PER_LONG - 1))) +#define BITMAP_LAST_WORD_MASK(nbits) (~0UL >> (-(nbits) & (BITS_PER_LONG - 1))) + +#define BIT_WORD(nr) ((nr) / BITS_PER_LONG) + +void __bitmap_set(unsigned long *map, unsigned int start, int len); +void __bitmap_clear(unsigned long *map, unsigned int start, int len); + +static __always_inline void bitmap_set(unsigned long *map, unsigned int start, + unsigned int nbits) +{ + if (__builtin_constant_p(nbits) && nbits == 1) + __set_bit(start, map); + else if (__builtin_constant_p(start & BITMAP_MEM_MASK) && + IS_ALIGNED(start, BITMAP_MEM_ALIGNMENT) && + __builtin_constant_p(nbits & BITMAP_MEM_MASK) && + IS_ALIGNED(nbits, BITMAP_MEM_ALIGNMENT)) + memset((char *)map + start / 8, 0xff, nbits / 8); + else + __bitmap_set(map, start, nbits); +} + +static __always_inline void bitmap_clear(unsigned long *map, unsigned int start, + unsigned int nbits) +{ + if (__builtin_constant_p(nbits) && nbits == 1) + __clear_bit(start, map); + else if (__builtin_constant_p(start & BITMAP_MEM_MASK) && + IS_ALIGNED(start, BITMAP_MEM_ALIGNMENT) && + __builtin_constant_p(nbits & BITMAP_MEM_MASK) && + IS_ALIGNED(nbits, BITMAP_MEM_ALIGNMENT)) + memset((char *)map + start / 8, 0, nbits / 8); + else + __bitmap_clear(map, start, nbits); +} + +#endif diff --git a/arch/x86/boot/compressed/bits.h b/arch/x86/boot/compressed/bits.h new file mode 100644 index 000000000000..b0ffa007ee19 --- /dev/null +++ b/arch/x86/boot/compressed/bits.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_BITS_H +#define BOOT_BITS_H +#define __LINUX_BITS_H /* Inhibit inclusion of */ + +#ifdef __ASSEMBLY__ +#define _AC(X,Y) X +#define _AT(T,X) X +#else +#define __AC(X,Y) (X##Y) +#define _AC(X,Y) __AC(X,Y) +#define _AT(T,X) ((T)(X)) +#endif + +#define _UL(x) (_AC(x, UL)) +#define _ULL(x) (_AC(x, ULL)) +#define UL(x) (_UL(x)) +#define ULL(x) (_ULL(x)) + +#define BIT(nr) (UL(1) << (nr)) +#define BIT_ULL(nr) (ULL(1) << (nr)) +#define BIT_MASK(nr) (UL(1) << ((nr) % BITS_PER_LONG)) +#define BIT_WORD(nr) ((nr) / BITS_PER_LONG) +#define BIT_ULL_MASK(nr) (ULL(1) << ((nr) % BITS_PER_LONG_LONG)) +#define BIT_ULL_WORD(nr) ((nr) / BITS_PER_LONG_LONG) +#define BITS_PER_BYTE 8 + +#define GENMASK(h, l) \ + (((~UL(0)) - (UL(1) << (l)) + 1) & \ + (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) + +#define GENMASK_ULL(h, l) \ + (((~ULL(0)) - (ULL(1) << (l)) + 1) & \ + (~ULL(0) >> (BITS_PER_LONG_LONG - 1 - (h)))) + +#endif diff --git a/arch/x86/boot/compressed/find.c b/arch/x86/boot/compressed/find.c new file mode 100644 index 000000000000..b97a9e7c8085 --- /dev/null +++ b/arch/x86/boot/compressed/find.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include "bitmap.h" +#include "find.h" +#include "math.h" +#include "minmax.h" + +static __always_inline unsigned long swab(const unsigned long y) +{ +#if __BITS_PER_LONG == 64 + return __builtin_bswap32(y); +#else /* __BITS_PER_LONG == 32 */ + return __builtin_bswap64(y); +#endif +} + +unsigned long _find_next_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long nbits, + unsigned long start, unsigned long invert, unsigned long le) +{ + unsigned long tmp, mask; + + if (start >= nbits) + return nbits; + + tmp = addr1[start / BITS_PER_LONG]; + if (addr2) + tmp &= addr2[start / BITS_PER_LONG]; + tmp ^= invert; + + /* Handle 1st word. */ + mask = BITMAP_FIRST_WORD_MASK(start); + if (le) + mask = swab(mask); + + tmp &= mask; + + start = round_down(start, BITS_PER_LONG); + + while (!tmp) { + start += BITS_PER_LONG; + if (start >= nbits) + return nbits; + + tmp = addr1[start / BITS_PER_LONG]; + if (addr2) + tmp &= addr2[start / BITS_PER_LONG]; + tmp ^= invert; + } + + if (le) + tmp = swab(tmp); + + return min(start + __ffs(tmp), nbits); +} diff --git a/arch/x86/boot/compressed/find.h b/arch/x86/boot/compressed/find.h new file mode 100644 index 000000000000..903574b9d57a --- /dev/null +++ b/arch/x86/boot/compressed/find.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_FIND_H +#define BOOT_FIND_H +#define __LINUX_FIND_H /* Inhibit inclusion of */ + +#include "../bitops.h" +#include "align.h" +#include "bits.h" + +unsigned long _find_next_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long nbits, + unsigned long start, unsigned long invert, unsigned long le); + +/** + * find_next_bit - find the next set bit in a memory region + * @addr: The address to base the search on + * @offset: The bitnumber to start searching at + * @size: The bitmap size in bits + * + * Returns the bit number for the next set bit + * If no bits are set, returns @size. + */ +static inline +unsigned long find_next_bit(const unsigned long *addr, unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (offset >= size) + return size; + + val = *addr & GENMASK(size - 1, offset); + return val ? __ffs(val) : size; + } + + return _find_next_bit(addr, NULL, size, offset, 0UL, 0); +} + +/** + * find_next_zero_bit - find the next cleared bit in a memory region + * @addr: The address to base the search on + * @offset: The bitnumber to start searching at + * @size: The bitmap size in bits + * + * Returns the bit number of the next zero bit + * If no bits are zero, returns @size. + */ +static inline +unsigned long find_next_zero_bit(const unsigned long *addr, unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (offset >= size) + return size; + + val = *addr | ~GENMASK(size - 1, offset); + return val == ~0UL ? size : ffz(val); + } + + return _find_next_bit(addr, NULL, size, offset, ~0UL, 0); +} + +/** + * for_each_set_bitrange_from - iterate over all set bit ranges [b; e) + * @b: bit offset of start of current bitrange (first set bit); must be initialized + * @e: bit offset of end of current bitrange (first unset bit) + * @addr: bitmap address to base the search on + * @size: bitmap size in number of bits + */ +#define for_each_set_bitrange_from(b, e, addr, size) \ + for ((b) = find_next_bit((addr), (size), (b)), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1); \ + (b) < (size); \ + (b) = find_next_bit((addr), (size), (e) + 1), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1)) +#endif diff --git a/arch/x86/boot/compressed/math.h b/arch/x86/boot/compressed/math.h new file mode 100644 index 000000000000..f7eede84bbc2 --- /dev/null +++ b/arch/x86/boot/compressed/math.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_MATH_H +#define BOOT_MATH_H +#define __LINUX_MATH_H /* Inhibit inclusion of */ + +/* + * + * This looks more complex than it should be. But we need to + * get the type for the ~ right in round_down (it needs to be + * as wide as the result!), and we want to evaluate the macro + * arguments just once each. + */ +#define __round_mask(x, y) ((__typeof__(x))((y)-1)) + +/** + * round_up - round up to next specified power of 2 + * @x: the value to round + * @y: multiple to round up to (must be a power of 2) + * + * Rounds @x up to next multiple of @y (which must be a power of 2). + * To perform arbitrary rounding up, use roundup() below. + */ +#define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1) + +/** + * round_down - round down to next specified power of 2 + * @x: the value to round + * @y: multiple to round down to (must be a power of 2) + * + * Rounds @x down to next multiple of @y (which must be a power of 2). + * To perform arbitrary rounding down, use rounddown() below. + */ +#define round_down(x, y) ((x) & ~__round_mask(x, y)) + +#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d)) + +#endif diff --git a/arch/x86/boot/compressed/minmax.h b/arch/x86/boot/compressed/minmax.h new file mode 100644 index 000000000000..4efd05673260 --- /dev/null +++ b/arch/x86/boot/compressed/minmax.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_MINMAX_H +#define BOOT_MINMAX_H +#define __LINUX_MINMAX_H /* Inhibit inclusion of */ + +/* + * This returns a constant expression while determining if an argument is + * a constant expression, most importantly without evaluating the argument. + * Glory to Martin Uecker + */ +#define __is_constexpr(x) \ + (sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8))) + +/* + * min()/max()/clamp() macros must accomplish three things: + * + * - avoid multiple evaluations of the arguments (so side-effects like + * "x++" happen only once) when non-constant. + * - perform strict type-checking (to generate warnings instead of + * nasty runtime surprises). See the "unnecessary" pointer comparison + * in __typecheck(). + * - retain result as a constant expressions when called with only + * constant expressions (to avoid tripping VLA warnings in stack + * allocation usage). + */ +#define __typecheck(x, y) \ + (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1))) + +#define __no_side_effects(x, y) \ + (__is_constexpr(x) && __is_constexpr(y)) + +#define __safe_cmp(x, y) \ + (__typecheck(x, y) && __no_side_effects(x, y)) + +#define __cmp(x, y, op) ((x) op (y) ? (x) : (y)) + +#define __cmp_once(x, y, unique_x, unique_y, op) ({ \ + typeof(x) unique_x = (x); \ + typeof(y) unique_y = (y); \ + __cmp(unique_x, unique_y, op); }) + +#define __careful_cmp(x, y, op) \ + __builtin_choose_expr(__safe_cmp(x, y), \ + __cmp(x, y, op), \ + __cmp_once(x, y, __UNIQUE_ID(__x), __UNIQUE_ID(__y), op)) + +/** + * min - return minimum of two values of the same or compatible types + * @x: first value + * @y: second value + */ +#define min(x, y) __careful_cmp(x, y, <) + +/** + * max - return maximum of two values of the same or compatible types + * @x: first value + * @y: second value + */ +#define max(x, y) __careful_cmp(x, y, >) + +#endif diff --git a/arch/x86/boot/compressed/pgtable_types.h b/arch/x86/boot/compressed/pgtable_types.h new file mode 100644 index 000000000000..8f1d87a69efc --- /dev/null +++ b/arch/x86/boot/compressed/pgtable_types.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef BOOT_COMPRESSED_PGTABLE_TYPES_H +#define BOOT_COMPRESSED_PGTABLE_TYPES_H +#define _ASM_X86_PGTABLE_DEFS_H /* Inhibit inclusion of */ + +#define PAGE_SHIFT 12 + +#ifdef CONFIG_X86_64 +#define PTE_SHIFT 9 +#elif defined CONFIG_X86_PAE +#define PTE_SHIFT 9 +#else /* 2-level */ +#define PTE_SHIFT 10 +#endif + +enum pg_level { + PG_LEVEL_NONE, + PG_LEVEL_4K, + PG_LEVEL_2M, + PG_LEVEL_1G, + PG_LEVEL_512G, + PG_LEVEL_NUM +}; + +#endif From patchwork Wed Dec 7 01:49:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30589 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157348wrr; Tue, 6 Dec 2022 17:53:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf6AAwNjgYukGzhElJu9+cNK3vsUK/dLfrdbipk4VG2SNpdRBpkfyU6GJBzVk7WsmHX6gJvT X-Received: by 2002:a17:906:9f10:b0:7ad:86f9:1d70 with SMTP id fy16-20020a1709069f1000b007ad86f91d70mr53596815ejc.179.1670378003836; Tue, 06 Dec 2022 17:53:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378003; cv=none; d=google.com; s=arc-20160816; b=V/Uf7f6MqFnCCV1xzbSEDRc3HysLYik0Py5JgJlBf17oCUUDL/laqJi4s0l74wYDuv pzNqUQJ2kskktSWY53YmJwj0LE58GmtHS66xQIBzU3HhngpF2ljiWiAS55zGlMvVu0HG iXiOoQ3I5SotKsT49ThXZAsGznH1CxI0nz5/plGt44U9j4YE2NSzsTBEcpqAHWfPj2m3 9JquZ6eRPMrH0zQmukCtFwThfJax6MUPZy3LwhhI0Ufv+M2bKMDrgF3htSBHhsWc1Kpu +2QTM2Y8k9oEDjj6BprxGCgL6EZ3BpzFgBfgnh7HaCv2cWukuRpUSftP8jnYTWPl+GEv qAAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pd77DOHLi4DvHN3XGAcCJihZfS9CMOtLB2gSFUdkwEQ=; b=nv0dblE4jQ88+cdWAPV6ZZ9s/t4veBfoLF1v/88vrIqS4RPNPQhdKEBO/PUwfjEBxb ps0foset2nwdBAofpLnhbpw0rn8szYSZgd7kmoq5z22JVX0nBTSsK2bxu8dvxETB2e3/ UkUzFFqTmV+vpX3uPWzihTnmJXaNBkn7AxbPfVcWbUnHh+98ZeHbgXmX+srxzuZt2r4f CgKt46a4qlXj9CFYNDOFq8E+CO/SIe3IF2TnHmi3PYqgO7IP35z5Hg5sJdq6iNp5qOrh Lgx816n05f1gcVYs+ShDgF9c/k9kRr0KXMeKbPz17ubxG70axjWB/V+QCBjSXCdZ33bk FOhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aiJPCHXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sg11-20020a170907a40b00b0078df3bd80b9si15557708ejc.46.2022.12.06.17.53.00; Tue, 06 Dec 2022 17:53:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=aiJPCHXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230055AbiLGBuw (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229845AbiLGBuU (ORCPT ); Tue, 6 Dec 2022 20:50:20 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AC2A537C9; Tue, 6 Dec 2022 17:50:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377803; x=1701913803; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SLFpRTCJa55e8jNeDSo+ACXNzeUqbaJWH6UnEfk1F10=; b=aiJPCHXPCI9NZSxutl+m+OGSjJpaZL5PlbyJVnxpOImdlVTQDzsO4SKL B2cqzVqirqLKUYk1hszAukKhuabQmeYsiK58jD7S5+8BQlv2dm5THojGR b8tq0ixSEk9q18NnqR3e+t1NcGN9uxhsxNuTzW3kiflY/To7g+j2M27OG 9NfPFARvGTJYNhkJJKTYgprPpWQXuuiTRoNKJ6RXW2/eKWWOR6g3Y/vDO WfrjodYyZ7nz5cggya2y79zQi2W4lFohGoIlO6junyU0XmJ7XutyOfV6+ mlaoKndJqfzV2+PG0uOiqmVJwFvo1Kt1MDdmCZzHQeXygPGwQcrmyMooW Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494517" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494517" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427690" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427690" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:52 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 6978B109C8A; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 06/14] efi/x86: Implement support for unaccepted memory Date: Wed, 7 Dec 2022 04:49:25 +0300 Message-Id: <20221207014933.8435-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518285505082507?= X-GMAIL-MSGID: =?utf-8?q?1751518285505082507?= UEFI Specification version 2.9 introduces the concept of memory acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform. Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead. The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory. Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space. In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space. Any unaccepted memory that is not aligned to 2M gets accepted upfront. The bitmap is allocated and constructed in the EFI stub and passed down to the kernel via boot_params. allocate_e820() allocates the bitmap if unaccepted memory is present, according to the maximum address in the memory map. The same boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented. The implementation requires some basic helpers in boot stub. They provided by linux/ includes in the main kernel image, but is not present in boot stub. Create copy of required functionality in the boot stub. Signed-off-by: Kirill A. Shutemov --- Documentation/x86/zero-page.rst | 1 + arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/mem.c | 73 ++++++++++++++++++++++++ arch/x86/include/asm/unaccepted_memory.h | 10 ++++ arch/x86/include/uapi/asm/bootparam.h | 2 +- drivers/firmware/efi/Kconfig | 14 +++++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/x86-stub.c | 68 ++++++++++++++++++++++ include/linux/efi.h | 3 +- 9 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h diff --git a/Documentation/x86/zero-page.rst b/Documentation/x86/zero-page.rst index 45aa9cceb4f1..f21905e61ade 100644 --- a/Documentation/x86/zero-page.rst +++ b/Documentation/x86/zero-page.rst @@ -20,6 +20,7 @@ Offset/Size Proto Name Meaning 060/010 ALL ist_info Intel SpeedStep (IST) BIOS support information (struct ist_info) 070/008 ALL acpi_rsdp_addr Physical address of ACPI RSDP table +078/008 ALL unaccepted_memory Bitmap of unaccepted memory (1bit == 2M) 080/010 ALL hd0_info hd0 disk parameter, OBSOLETE!! 090/010 ALL hd1_info hd1 disk parameter, OBSOLETE!! 0A0/010 ALL sys_desc_table System description table (struct sys_desc_table), diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 3dc5db651dd0..0ae221540dee 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -107,6 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o diff --git a/arch/x86/boot/compressed/mem.c b/arch/x86/boot/compressed/mem.c new file mode 100644 index 000000000000..a848119e4455 --- /dev/null +++ b/arch/x86/boot/compressed/mem.c @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include "../cpuflags.h" +#include "bitmap.h" +#include "error.h" +#include "math.h" + +#define PMD_SHIFT 21 +#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) +#define PMD_MASK (~(PMD_SIZE - 1)) + +static inline void __accept_memory(phys_addr_t start, phys_addr_t end) +{ + /* Platform-specific memory-acceptance call goes here */ + error("Cannot accept memory"); +} + +/* + * The accepted memory bitmap only works at PMD_SIZE granularity. This + * function takes unaligned start/end addresses and either: + * 1. Accepts the memory immediately and in its entirety + * 2. Accepts unaligned parts, and marks *some* aligned part unaccepted + * + * The function will never reach the bitmap_set() with zero bits to set. + */ +void process_unaccepted_memory(struct boot_params *params, u64 start, u64 end) +{ + /* + * Ensure that at least one bit will be set in the bitmap by + * immediately accepting all regions under 2*PMD_SIZE. This is + * imprecise and may immediately accept some areas that could + * have been represented in the bitmap. But, results in simpler + * code below + * + * Consider case like this: + * + * | 4k | 2044k | 2048k | + * ^ 0x0 ^ 2MB ^ 4MB + * + * Only the first 4k has been accepted. The 0MB->2MB region can not be + * represented in the bitmap. The 2MB->4MB region can be represented in + * the bitmap. But, the 0MB->4MB region is <2*PMD_SIZE and will be + * immediately accepted in its entirety. + */ + if (end - start < 2 * PMD_SIZE) { + __accept_memory(start, end); + return; + } + + /* + * No matter how the start and end are aligned, at least one unaccepted + * PMD_SIZE area will remain to be marked in the bitmap. + */ + + /* Immediately accept a unaccepted_memory, + start / PMD_SIZE, (end - start) / PMD_SIZE); +} diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h new file mode 100644 index 000000000000..df0736d32858 --- /dev/null +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2020 Intel Corporation */ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +struct boot_params; + +void process_unaccepted_memory(struct boot_params *params, u64 start, u64 num); + +#endif diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h index 01d19fc22346..630a54046af0 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -189,7 +189,7 @@ struct boot_params { __u64 tboot_addr; /* 0x058 */ struct ist_info ist_info; /* 0x060 */ __u64 acpi_rsdp_addr; /* 0x070 */ - __u8 _pad3[8]; /* 0x078 */ + __u64 unaccepted_memory; /* 0x078 */ __u8 hd0_info[16]; /* obsolete! */ /* 0x080 */ __u8 hd1_info[16]; /* obsolete! */ /* 0x090 */ struct sys_desc_table sys_desc_table; /* obsolete! */ /* 0x0a0 */ diff --git a/drivers/firmware/efi/Kconfig b/drivers/firmware/efi/Kconfig index 6787ed8dfacf..8aa8adf0bcb5 100644 --- a/drivers/firmware/efi/Kconfig +++ b/drivers/firmware/efi/Kconfig @@ -314,6 +314,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. + config EFI_EMBEDDED_FIRMWARE bool select CRYPTO_LIB_SHA256 diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index a46df5d1d094..f525144e22e4 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -777,6 +777,7 @@ static __initdata char memory_type_name[][13] = { "MMIO Port", "PAL Code", "Persistent", + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size, diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index fff81843169c..27b9eed5883b 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "efistub.h" @@ -613,6 +614,16 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type = E820_TYPE_PMEM; break; + case EFI_UNACCEPTED_MEMORY: + if (!IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) { + efi_warn_once( +"The system has unaccepted memory, but kernel does not support it\nConsider enabling CONFIG_UNACCEPTED_MEMORY\n"); + continue; + } + e820_type = E820_TYPE_RAM; + process_unaccepted_memory(params, d->phys_addr, + d->phys_addr + PAGE_SIZE * d->num_pages); + break; default: continue; } @@ -677,6 +688,60 @@ static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext, return status; } +static efi_status_t allocate_unaccepted_bitmap(struct boot_params *params, + __u32 nr_desc, + struct efi_boot_memmap *map) +{ + unsigned long *mem = NULL; + u64 size, max_addr = 0; + efi_status_t status; + bool found = false; + int i; + + /* Check if there's any unaccepted memory and find the max address */ + for (i = 0; i < nr_desc; i++) { + efi_memory_desc_t *d; + unsigned long m = (unsigned long)map->map; + + d = efi_early_memdesc_ptr(m, map->desc_size, i); + if (d->type == EFI_UNACCEPTED_MEMORY) + found = true; + if (d->phys_addr + d->num_pages * PAGE_SIZE > max_addr) + max_addr = d->phys_addr + d->num_pages * PAGE_SIZE; + } + + if (!found) { + params->unaccepted_memory = 0; + return EFI_SUCCESS; + } + + /* + * If unaccepted memory is present, allocate a bitmap to track what + * memory has to be accepted before access. + * + * One bit in the bitmap represents 2MiB in the address space: + * A 4k bitmap can track 64GiB of physical address space. + * + * In the worst case scenario -- a huge hole in the middle of the + * address space -- It needs 256MiB to handle 4PiB of the address + * space. + * + * TODO: handle situation if params->unaccepted_memory is already set. + * It's required to deal with kexec. + * + * The bitmap will be populated in setup_e820() according to the memory + * map after efi_exit_boot_services(). + */ + size = DIV_ROUND_UP(max_addr, PMD_SIZE * BITS_PER_BYTE); + status = efi_allocate_pages(size, (unsigned long *)&mem, ULONG_MAX); + if (status == EFI_SUCCESS) { + memset(mem, 0, size); + params->unaccepted_memory = (unsigned long)mem; + } + + return status; +} + static efi_status_t allocate_e820(struct boot_params *params, struct setup_data **e820ext, u32 *e820ext_size) @@ -697,6 +762,9 @@ static efi_status_t allocate_e820(struct boot_params *params, status = alloc_e820ext(nr_e820ext, e820ext, e820ext_size); } + if (IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) && status == EFI_SUCCESS) + status = allocate_unaccepted_bitmap(params, nr_desc, map); + efi_bs_call(free_pool, map); return status; } diff --git a/include/linux/efi.h b/include/linux/efi.h index 7603fc58c47c..cfdcc165071e 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -108,7 +108,8 @@ typedef struct { #define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12 #define EFI_PAL_CODE 13 #define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16 /* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL) /* uncached */ From patchwork Wed Dec 7 01:49:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30592 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157559wrr; Tue, 6 Dec 2022 17:54:07 -0800 (PST) X-Google-Smtp-Source: AA0mqf5tUL6oRyGZMmAUYz1bw/F9kNA5VN4lgGjdsACLqiVgcNDCq2YAhAOil6ANcPIm+QZ1RrwR X-Received: by 2002:a17:907:6daa:b0:7ba:e537:c64b with SMTP id sb42-20020a1709076daa00b007bae537c64bmr54150904ejc.180.1670378047088; Tue, 06 Dec 2022 17:54:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378047; cv=none; d=google.com; s=arc-20160816; b=04syRrM65huN3iWramlsCPYMD7XZjJbsggU7rQ2CCO5UCFmxAXfyHP8uoV1V/Lzk9g ewBp5wyQFL6zj4SMfla0CcWPbB548006CBS8/KD+55kxMZOhkQFLmh69korHmE4WurQd LLQK+K9nvWRAOuPnpsgELxzjl213kYDeH1Hq4c45x+BdFYZ4rPTBWH65KMTG15TgqLyz ApxD6l1/mezwuw0N3lBv+gey6YlmcY5HMHmJPX6flmEqv1/pL51aVkIctB3tYVcrtZ/s ay13EJyTMkYo9jreraJhgo0DIQuA75+DXZ77GAsrzsIVTFRU7rpt9eekHYyd/7orsbQP e+7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XGFTIpSJTtn+ulXtV3l+BXIe2hGAedtrtZf2coafzls=; b=KStkmiy+2lljs5CwVPXB4iwGZh1fgR3pxoXcEBU9Dv0gIvFh6Tmf0GcUF5Koxka3JI h60/Lo1duNlzbLzpT1kQH8Kaeway7PiXCcg9AKzq6euWk9gC/N60zNJN8v3xlBAKW3mF RLiDtHdz4aK92T3We+Za9EGOfj7Ar2OXMyfh2z+4dTrqlwCC31hDLYIqE8ZckLFhWR0W XxE6ni4vRDGVpidujWlpupIwZ1N7goLK6gD2RGUgcQz2q2gixUXR5JfQq7iu/tkTuibQ gwN29k1edirXt3bu2QyCHJxyKOn8VGzKuTmtkF99u3fEEAecZAKb5xnoYwo/Og4qlOL3 d7og== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LfBxCZN4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wz5-20020a170906fe4500b007c0fa609b09si5399719ejb.771.2022.12.06.17.53.44; Tue, 06 Dec 2022 17:54:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LfBxCZN4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbiLGBu2 (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229968AbiLGBuS (ORCPT ); Tue, 6 Dec 2022 20:50:18 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF4BA532D7; Tue, 6 Dec 2022 17:50:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377800; x=1701913800; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=U2s814f8P0ClrQ6x0eaK3HK3hMxUAqo4D0aUxrmJytE=; b=LfBxCZN4JqfFVat+FhFFUtCCTZGi0Q+YJ5ahnTb/mtgVdjGBlpyGBac2 /6Cs8DRn+kwBCIqboIi24sNSvC3sn4ll9IEguZDO7YJORTa93Zjljz2l3 fk99Zp9dKqOJot86VptWHHm9VpFmK4A/KRlk+vz8TKlzz58Djpg0kAlmz jIHjK19CFQI3jlE9921dKwlXB96m5J13dcL8oR+7LL+Mw0iNQd98557Mu MCZ2/OaCJ/o2guDrlV6Y0oC1fZsJ2Bg/pQ3L9xmOImu2MDYtz1Yzn2y7D lyA/2F+S35GTz5V62U+adEJlJPE98wzLIa5LqaTtID1ugv2Zva6v2zyKB w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="316794436" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="316794436" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:00 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="640082283" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="640082283" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:52 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 7568D109C8B; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 07/14] x86/boot/compressed: Handle unaccepted memory Date: Wed, 7 Dec 2022 04:49:26 +0300 Message-Id: <20221207014933.8435-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518331138267161?= X-GMAIL-MSGID: =?utf-8?q?1751518331138267161?= The firmware will pre-accept the memory used to run the stub. But, the stub is responsible for accepting the memory into which it decompresses the main kernel. Accept memory just before decompression starts. The stub is also responsible for choosing a physical address in which to place the decompressed kernel image. The KASLR mechanism will randomize this physical address. Since the unaccepted memory region is relatively small, KASLR would be quite ineffective if it only used the pre-accepted area (EFI_CONVENTIONAL_MEMORY). Ensure that KASLR randomizes among the entire physical address space by also including EFI_UNACCEPTED_MEMORY. Signed-off-by: Kirill A. Shutemov --- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/efi.h | 1 + arch/x86/boot/compressed/kaslr.c | 35 ++++++++++++++++-------- arch/x86/boot/compressed/mem.c | 18 ++++++++++++ arch/x86/boot/compressed/misc.c | 6 ++++ arch/x86/boot/compressed/misc.h | 6 ++++ arch/x86/include/asm/unaccepted_memory.h | 2 ++ 7 files changed, 57 insertions(+), 13 deletions(-) diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 0ae221540dee..a9937b50c37f 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -107,7 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o -vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/mem.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/find.o $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o diff --git a/arch/x86/boot/compressed/efi.h b/arch/x86/boot/compressed/efi.h index 7db2f41b54cd..cf475243b6d5 100644 --- a/arch/x86/boot/compressed/efi.h +++ b/arch/x86/boot/compressed/efi.h @@ -32,6 +32,7 @@ typedef struct { } efi_table_hdr_t; #define EFI_CONVENTIONAL_MEMORY 7 +#define EFI_UNACCEPTED_MEMORY 15 #define EFI_MEMORY_MORE_RELIABLE \ ((u64)0x0000000000010000ULL) /* higher reliability */ diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index 454757fbdfe5..749f0fe7e446 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -672,6 +672,28 @@ static bool process_mem_region(struct mem_vector *region, } #ifdef CONFIG_EFI + +/* + * Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if supported) are + * guaranteed to be free. + * + * It is more conservative in picking free memory than the EFI spec allows: + * + * According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also free memory + * and thus available to place the kernel image into, but in practice there's + * firmware where using that memory leads to crashes. + */ +static inline bool memory_type_is_free(efi_memory_desc_t *md) +{ + if (md->type == EFI_CONVENTIONAL_MEMORY) + return true; + + if (md->type == EFI_UNACCEPTED_MEMORY) + return IS_ENABLED(CONFIG_UNACCEPTED_MEMORY); + + return false; +} + /* * Returns true if we processed the EFI memmap, which we prefer over the E820 * table if it is available. @@ -716,18 +738,7 @@ process_efi_entries(unsigned long minimum, unsigned long image_size) for (i = 0; i < nr_desc; i++) { md = efi_early_memdesc_ptr(pmap, e->efi_memdesc_size, i); - /* - * Here we are more conservative in picking free memory than - * the EFI spec allows: - * - * According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also - * free memory and thus available to place the kernel image into, - * but in practice there's firmware where using that memory leads - * to crashes. - * - * Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. - */ - if (md->type != EFI_CONVENTIONAL_MEMORY) + if (!memory_type_is_free(md)) continue; if (efi_soft_reserve_enabled() && diff --git a/arch/x86/boot/compressed/mem.c b/arch/x86/boot/compressed/mem.c index a848119e4455..626e4f10ba2c 100644 --- a/arch/x86/boot/compressed/mem.c +++ b/arch/x86/boot/compressed/mem.c @@ -3,12 +3,15 @@ #include "../cpuflags.h" #include "bitmap.h" #include "error.h" +#include "find.h" #include "math.h" #define PMD_SHIFT 21 #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE - 1)) +extern struct boot_params *boot_params; + static inline void __accept_memory(phys_addr_t start, phys_addr_t end) { /* Platform-specific memory-acceptance call goes here */ @@ -71,3 +74,18 @@ void process_unaccepted_memory(struct boot_params *params, u64 start, u64 end) bitmap_set((unsigned long *)params->unaccepted_memory, start / PMD_SIZE, (end - start) / PMD_SIZE); } + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long range_start, range_end; + unsigned long *bitmap, bitmap_size; + + bitmap = (unsigned long *)boot_params->unaccepted_memory; + range_start = start / PMD_SIZE; + bitmap_size = DIV_ROUND_UP(end, PMD_SIZE); + + for_each_set_bitrange_from(range_start, range_end, bitmap, bitmap_size) { + __accept_memory(range_start * PMD_SIZE, range_end * PMD_SIZE); + bitmap_clear(bitmap, range_start, range_end - range_start); + } +} diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index cf690d8712f4..c41a87b0ec06 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -454,6 +454,12 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, #endif debug_putstr("\nDecompressing Linux... "); + + if (boot_params->unaccepted_memory) { + debug_putstr("Accepting memory... "); + accept_memory(__pa(output), __pa(output) + needed_size); + } + __decompress(input_data, input_len, NULL, NULL, output, output_len, NULL, error); parse_elf(output); diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index d37d612a6390..44337a1ac23c 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -245,4 +245,10 @@ static inline unsigned long efi_find_vendor_table(struct boot_params *bp, } #endif /* CONFIG_EFI */ +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_memory(phys_addr_t start, phys_addr_t end); +#else +static inline void accept_memory(phys_addr_t start, phys_addr_t end) {} +#endif + #endif /* BOOT_COMPRESSED_MISC_H */ diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h index df0736d32858..41fbfc798100 100644 --- a/arch/x86/include/asm/unaccepted_memory.h +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -7,4 +7,6 @@ struct boot_params; void process_unaccepted_memory(struct boot_params *params, u64 start, u64 num); +void accept_memory(phys_addr_t start, phys_addr_t end); + #endif From patchwork Wed Dec 7 01:49:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30584 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3156832wrr; Tue, 6 Dec 2022 17:51:49 -0800 (PST) X-Google-Smtp-Source: AA0mqf6nPakPzRFYQ3dlN4gFITSbpxdQ7AiFxFIvMZIksJJ2SDGzieC0TthqW/ipl5XvlSEy38Lr X-Received: by 2002:a17:906:1748:b0:7b2:7fa8:3d84 with SMTP id d8-20020a170906174800b007b27fa83d84mr32289816eje.650.1670377909091; Tue, 06 Dec 2022 17:51:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377909; cv=none; d=google.com; s=arc-20160816; b=Cw28oG0ai+LWd5HMgYFj9F8Ut3VlZKdjzAfJcCs+cusSLfIIH5bK9rjzSQZ7xT3yXJ UwMK9Fr/ASsUbRzV7SifBUcZMdl+ED55ka2AJxRmFWvZj+p+myVsms0LdrA0qEo5Tn9y Wbo0UiTBrAYf2BULtF5bNaDJDayz0xoP5m3wX6YvvTVpWHSWL0j0pwFuvahzx5oeU1aU yOQo/yZmJWvQoHjpw3xBu1Y2TTiKGsrGqvIJe1ZUBQq8sC0OIRf1w5C39LWLDrLF7/Qq HF+Ez9t8uBZ6m+CGAF7I89ngkMvP+I/OZPXLOR6oRRSJkNpext8SWR+N0qGx9rupz9nM C1Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AbEhTgrAuKDGYsZ6/Y4F9BBFnx+cJUwn1WPTrIBrLw0=; b=B3YryrXt0cnxG3Kmxe3t9Mv1MWxNBX/0IPV9v4OUpiN1ZP5/1vD2Vm1g3q8SrsoE1c xwPx86ik/CnFInpmTEqrGG5SQ9h3bvPsIDyvmdJWQHXVjpEFgs0OshY6U3KD2rCzNvCo f41tiduNUX/8XU7qjthr6R84Ln2oz/X97ZX2mosvNFm8NxRcRNxfuaNbXMsTYZ1MgQ+r 5ZLfTrBccm8oIBWtFccKWjxcp/OnRXFOatfWt/0AlOHH28Xw1X5XprtQjQVjSgrrhvwo DlBFWxwspVaEOM5hEDKet236STx8yZKDFOSufumafoTQ0P6S82d/g5UqCqikZreJ1eeu 1j9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dJ6Kf+p7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gs44-20020a1709072d2c00b007c11812c620si1741699ejc.440.2022.12.06.17.51.26; Tue, 06 Dec 2022 17:51:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=dJ6Kf+p7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229976AbiLGBuU (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229726AbiLGBuJ (ORCPT ); Tue, 6 Dec 2022 20:50:09 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1722532C6; Tue, 6 Dec 2022 17:49:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377799; x=1701913799; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oBiSIOxE3eoLPk40ExcpaFUWqTkEiCbV0j2bGnDI4n8=; b=dJ6Kf+p7pq7OVLXdCIMJbKhO9G4PToMylfvLhXRMcImC4dSLwgiQDvhd Z0seE+3buo/iSbVFIOo/gZeLrKziE4t137pa1bvvaiyO7P9x45Y0ONGUY 096PCQQJMYEgrstWbWbk0GKICCr0FclPRydatuMaUMKl8zLrlQjhY2xBD qy9TIy3BMD97sdAFUHwW1vejSQGKz+ooqfBJ+gscqzvoJRfMMxuoms13N NZJ0djf7YePuHBTCulncAjGncCgFVhDU0JRFHWQjjFAFHjUIEP+wLDFWh QUAlV4yXyz9Ag9d60Cjd/UY86bXGc8zIutJZtQisCM+36Q6ZFW88i4hhz Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="316794419" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="316794419" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:59 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="640082279" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="640082279" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:51 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 814A0109C8C; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Mike Rapoport Subject: [PATCHv8 08/14] x86/mm: Reserve unaccepted memory bitmap Date: Wed, 7 Dec 2022 04:49:27 +0300 Message-Id: <20221207014933.8435-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518186310062295?= X-GMAIL-MSGID: =?utf-8?q?1751518186310062295?= A given page of memory can only be accepted once. The kernel has to accept memory both in the early decompression stage and during normal runtime. A bitmap is used to communicate the acceptance state of each page between the decompression stage and normal runtime. boot_params is used to communicate location of the bitmap throughout the boot. The bitmap is allocated and initially populated in EFI stub. Decompression stage accepts pages required for kernel/initrd and marks these pages accordingly in the bitmap. The main kernel picks up the bitmap from the same boot_params and uses it to determine what has to be accepted on allocation. In the runtime kernel, reserve the bitmap's memory to ensure nothing overwrites it. The size of bitmap is determined with e820__end_of_ram_pfn() which relies on setup_e820() marking unaccepted memory as E820_TYPE_RAM. Signed-off-by: Kirill A. Shutemov Acked-by: Mike Rapoport --- arch/x86/kernel/e820.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c index 9dac24680ff8..62068956bb76 100644 --- a/arch/x86/kernel/e820.c +++ b/arch/x86/kernel/e820.c @@ -1316,6 +1316,23 @@ void __init e820__memblock_setup(void) int i; u64 end; + /* + * Mark unaccepted memory bitmap reserved. + * + * This kind of reservation usually done from early_reserve_memory(), + * but early_reserve_memory() called before e820__memory_setup(), so + * e820_table is not finalized and e820__end_of_ram_pfn() cannot be + * used to get correct RAM size. + */ + if (boot_params.unaccepted_memory) { + unsigned long size; + + /* One bit per 2MB */ + size = DIV_ROUND_UP(e820__end_of_ram_pfn() * PAGE_SIZE, + PMD_SIZE * BITS_PER_BYTE); + memblock_reserve(boot_params.unaccepted_memory, size); + } + /* * The bootstrap memblock region count maximum is 128 entries * (INIT_MEMBLOCK_REGIONS), but EFI might pass us more E820 entries From patchwork Wed Dec 7 01:49:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30591 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157542wrr; Tue, 6 Dec 2022 17:54:02 -0800 (PST) X-Google-Smtp-Source: AA0mqf733KOijJGaCyCiag4zBjhnNoAzqjV5RwNdE/TZOTaZBzMwD6rMWFurk/9QO9do/CV4uxgl X-Received: by 2002:a05:6402:2926:b0:46b:b45d:4431 with SMTP id ee38-20020a056402292600b0046bb45d4431mr25818542edb.4.1670378042699; Tue, 06 Dec 2022 17:54:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378042; cv=none; d=google.com; s=arc-20160816; b=XKEMwlgWqJ+RSBUMhwRGYz82PPLCeb8vtMoGkE7NoP2ukwF+c4PIZI/d3XRoFm1SAi J0Hp3seHG5HD8Hm4+1f0cCcnDOeMEcLMsxiNCcPjTt4RxWUEPXsvJnyA/9K+uCO5buZt oNGj6pPYeVExKuM53kCmFREfsFDpGw65BWht/3vbdnlF1K65xb188wprcQFmb4k9AoKt PiJgRCl20Q49OxkeuwqKpehUqU67nixJwcWKXYg75gzR84YTsRh9C3VQydIMCeQVrGft 7Ymd1k0aNU+q8qVU/K9YNJjhTGYOmafdoupQPa37v6aWa+z1AUwV1EfIoC9Uejb5+Zgh 3DXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UBR5pBCbvwFu4O7MkWueqfBA2y89H4B1TrkONvPTWTY=; b=ZR7cbUyTerkTbgc7m2U2RXlFgvoigTebCqzpecTpB/XpFTc/2Y/wesOxXXEaHvNLIG SgS5ux7h9RVTaklsSCQrndX3+XZo/OddhWjitNEBm/T9cA3Px19Z6gWsliQH2eAGosKB phtX5h8TNeF2iJ0Jk7BDiqT9PAQGxPL0p7zfkmQL5K+6dy02ULo84OcXq79UnSMEYY07 gyVuRZf5CbdDrOvVp/1B9GiyyHnrP+rAlBB+R4XLsMYxYPpuBIeqo+H5W2gSnwQ3vZcE 42rytATWSFTzWCvvRbWLZUiTx8HleIHEL2/QdHYylnrNitbf+TxCUooKYuLBtno86WbF 3S4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nC0xE1Hv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x14-20020a05640226ce00b0046d385913f3si1359570edd.122.2022.12.06.17.53.39; Tue, 06 Dec 2022 17:54:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nC0xE1Hv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230104AbiLGBvJ (ORCPT + 99 others); Tue, 6 Dec 2022 20:51:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230009AbiLGBuj (ORCPT ); Tue, 6 Dec 2022 20:50:39 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 010DB537DC; Tue, 6 Dec 2022 17:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377805; x=1701913805; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DsLSYB0ldaIutEa0HLWA8q6UCSqPp2KzK7qJuCNXS/4=; b=nC0xE1HvoJtQLyCgd49iSjmGQmoCS3RC7x4pFVRLE2zx1eCVFDdQXbfE oZMxd5uyFUKyCdj5bjomAbL51i9Ckh08AMu0sIz79QLoqrprXX9TG63KW +Ba/qGT23keHGxDJNp6YW6nYzwgd7ELw3gDutObVttCFMFYfqNnm1nl4K Ss9/VNJHJ+kCCsgRU5XD24297nhUAPXVcF5Gx5iSMkf2IYq+7XGkj3CDW FGqO/I+eOHFK4C/rj0ijZiHxc8w/8y2g5fLysWwjXAvlnbWGBW0OTPwIo LcxLMHoyjMCZaP2PPcSgGc2LhyK1QFDdWjMN8jVk9VYcIn1uusvXALV9v w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494594" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494594" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427717" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427717" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 8CDBF109C8D; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 09/14] x86/mm: Provide helpers for unaccepted memory Date: Wed, 7 Dec 2022 04:49:28 +0300 Message-Id: <20221207014933.8435-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518326573767442?= X-GMAIL-MSGID: =?utf-8?q?1751518326573767442?= Core-mm requires few helpers to support unaccepted memory: - accept_memory() checks the range of addresses against the bitmap and accept memory if needed. - range_contains_unaccepted_memory() checks if anything within the range requires acceptance. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/page.h | 3 ++ arch/x86/include/asm/unaccepted_memory.h | 4 ++ arch/x86/mm/Makefile | 2 + arch/x86/mm/unaccepted_memory.c | 61 ++++++++++++++++++++++++ 4 files changed, 70 insertions(+) create mode 100644 arch/x86/mm/unaccepted_memory.c diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 9cc82f305f4b..df4ec3a988dc 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -19,6 +19,9 @@ struct page; #include + +#include + extern struct range pfn_mapped[]; extern int nr_pfn_mapped; diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h index 41fbfc798100..89fc91c61560 100644 --- a/arch/x86/include/asm/unaccepted_memory.h +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -7,6 +7,10 @@ struct boot_params; void process_unaccepted_memory(struct boot_params *params, u64 start, u64 num); +#ifdef CONFIG_UNACCEPTED_MEMORY + void accept_memory(phys_addr_t start, phys_addr_t end); +bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end); #endif +#endif diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index c80febc44cd2..b0ef1755e5c8 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -67,3 +67,5 @@ obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c new file mode 100644 index 000000000000..1df918b21469 --- /dev/null +++ b/arch/x86/mm/unaccepted_memory.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include + +#include +#include +#include + +/* Protects unaccepted memory bitmap */ +static DEFINE_SPINLOCK(unaccepted_memory_lock); + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long range_start, range_end; + unsigned long *bitmap; + unsigned long flags; + + if (!boot_params.unaccepted_memory) + return; + + bitmap = __va(boot_params.unaccepted_memory); + range_start = start / PMD_SIZE; + + spin_lock_irqsave(&unaccepted_memory_lock, flags); + for_each_set_bitrange_from(range_start, range_end, bitmap, + DIV_ROUND_UP(end, PMD_SIZE)) { + unsigned long len = range_end - range_start; + + /* Platform-specific memory-acceptance call goes here */ + panic("Cannot accept memory: unknown platform\n"); + bitmap_clear(bitmap, range_start, len); + } + spin_unlock_irqrestore(&unaccepted_memory_lock, flags); +} + +bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long *bitmap; + unsigned long flags; + bool ret = false; + + if (!boot_params.unaccepted_memory) + return 0; + + bitmap = __va(boot_params.unaccepted_memory); + + spin_lock_irqsave(&unaccepted_memory_lock, flags); + while (start < end) { + if (test_bit(start / PMD_SIZE, bitmap)) { + ret = true; + break; + } + + start += PMD_SIZE; + } + spin_unlock_irqrestore(&unaccepted_memory_lock, flags); + + return ret; +} From patchwork Wed Dec 7 01:49:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30587 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157205wrr; Tue, 6 Dec 2022 17:52:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf73jCN7T8CUTN7C2XRoyyEwa1U7YqN/+A7vEhz2Lsf+GUM+Ml/qSYQw3XO+3i40wil6kTMI X-Received: by 2002:aa7:cfc4:0:b0:46b:d83e:984c with SMTP id r4-20020aa7cfc4000000b0046bd83e984cmr24112549edy.60.1670377970932; Tue, 06 Dec 2022 17:52:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377970; cv=none; d=google.com; s=arc-20160816; b=mM1J3Iwhx/11gyycFUSHwNgfdq2CoFe5ffhnNfucmzGzNRsr1qszXAnaSKavJHwDD8 /9MmOa0KZyJKztmy5o47exYuykR/v3AVkwmCwf2zyDsWNhwjSTvOiGwjCae46j+yDqhA GqNvPiXv//t3vNfTftH4R2oxNWYjSEr4D9cWlIr4DKgLvV9qG8ZspRWHCnmjqTgEzP6I j/IhK8O//Ij8XKmUe6DQTS3q6kHSlk9D2P62h2LsVXzbFIdaWtGaRJEknc7eV1BB/mbd pvvb8LFtKhaAtUqRftsiAExOTsvAtEh4G1vxCnTAI0NTp/5aDjVdstS4wzUwjdxtEkkq ExKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VXeJx15b0x9FOojJzVSBusJ7ni62omarhCf6lRF7Rxs=; b=h3bG9R+voxJhf45AZLw0wzxo8ehEMz7MJH5CIrPjIVYuD61hTfTcUv/c3PsAeQ1fpL r0LqAWD8JLISwn03efKUKptVEZcLeKaRVD/QP7lU/IdeWnpAhTNOpiDT7UFbkYV+7dhu oDh7prm+etNtgyJa6nwenbczYtB9e2fNyVyLO784NbJRiYKrmqQOPR1BE8ywvP3RFoIk HdFGI/ubN9M3CYv+ezWne56XtDqrQdVzPr7U2msbMr/ehqbZaoD8oh0iRi3FqKqkkb61 YMt0bCbXPupcarvvOd+SEc60jlh+tzDGEfqoMF5vI5e1Aq5gJsHnnkDu1ohYv+AgrPBd 5KqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=H+8gSLsy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n14-20020a05640205ce00b0046b7d1b9d79si3483243edx.328.2022.12.06.17.52.28; Tue, 06 Dec 2022 17:52:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=H+8gSLsy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230052AbiLGBur (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229983AbiLGBuU (ORCPT ); Tue, 6 Dec 2022 20:50:20 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57215537D7; Tue, 6 Dec 2022 17:50:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377803; x=1701913803; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tWBdt2o2/am+QkNoUeL/v5kYgnfgBeMa0WLAghCR9t0=; b=H+8gSLsyYq6uQQIsuLtB+dg3i+JIX5N6kCsdgRNPIkcd1vuosWBBGeor dWIP9yxCa7LEp2hvVqMQ5JZ+5+ZuxNdRYeKnlnN+vYFZ1HeVICrHVha7L jyc15OdYLBGsVlbP84s/IrlkfLVEVWJjdzKAb3xOzWDP0vSsB88ad7wvc ZVASh4+oY2PnKJA15PCmipMw8j2fmv+UhwSClCJrvakTX8Fyqq8Q4ApIN 237z5170G62bYlGYQtgehDK54xyTdAoyShGKSIPJS91QCi3EkfvsV6ME2 /AsCvwju1cXHvnZFgI/ag5xQE+/CKcSXAO7rjfgBe+opBgxJhBZkPX17e w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494551" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494551" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427704" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427704" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:52 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 97FD3109C8E; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv8 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory Date: Wed, 7 Dec 2022 04:49:29 +0300 Message-Id: <20221207014933.8435-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518251465579826?= X-GMAIL-MSGID: =?utf-8?q?1751518251465579826?= load_unaligned_zeropad() can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory. load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads. But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest. There are three parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra "guard" page is accepted in addition to the memory that needs to be used. 1. Implicitly extend the range_contains_unaccepted_memory(start, end) checks up to end+2M if 'end' is aligned on a 2M boundary. It may require checking 2M chunk beyond end of RAM. The bitmap allocation is modified to accommodate this. 2. Implicitly extend accept_memory(start, end) to end+2M if 'end' is aligned on a 2M boundary. 3. Set PageUnaccepted() on both memory that itself needs to be accepted *and* memory where the next page needs to be accepted. Essentially, make PageUnaccepted(page) a marker for whether work needs to be done to make 'page' usable. That work might include accepting pages in addition to 'page' itself. Side note: This leads to something strange. Pages which were accepted at boot, marked by the firmware as accepted and will never _need_ to be accepted might have PageUnaccepted() set on them. PageUnaccepted(page) is a cue to ensure that the next page is accepted before 'page' can be used. This is an actual, real-world problem which was discovered during TDX testing. Signed-off-by: Kirill A. Shutemov Reviewed-by: Dave Hansen --- arch/x86/mm/unaccepted_memory.c | 39 +++++++++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 7 +++++ 2 files changed, 46 insertions(+) diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c index 1df918b21469..a0a58486eb74 100644 --- a/arch/x86/mm/unaccepted_memory.c +++ b/arch/x86/mm/unaccepted_memory.c @@ -23,6 +23,38 @@ void accept_memory(phys_addr_t start, phys_addr_t end) bitmap = __va(boot_params.unaccepted_memory); range_start = start / PMD_SIZE; + /* + * load_unaligned_zeropad() can lead to unwanted loads across page + * boundaries. The unwanted loads are typically harmless. But, they + * might be made to totally unrelated or even unmapped memory. + * load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now + * #VE) to recover from these unwanted loads. + * + * But, this approach does not work for unaccepted memory. For TDX, a + * load from unaccepted memory will not lead to a recoverable exception + * within the guest. The guest will exit to the VMM where the only + * recourse is to terminate the guest. + * + * There are three parts to fix this issue and comprehensively avoid + * access to unaccepted memory. Together these ensure that an extra + * "guard" page is accepted in addition to the memory that needs to be + * used: + * + * 1. Implicitly extend the range_contains_unaccepted_memory(start, end) + * checks up to end+2M if 'end' is aligned on a 2M boundary. + * + * 2. Implicitly extend accept_memory(start, end) to end+2M if 'end' is + * aligned on a 2M boundary. (immediately following this comment) + * + * 3. Set PageUnaccepted() on both memory that itself needs to be + * accepted *and* memory where the next page needs to be accepted. + * Essentially, make PageUnaccepted(page) a marker for whether work + * needs to be done to make 'page' usable. That work might include + * accepting pages in addition to 'page' itself. + */ + if (!(end % PMD_SIZE)) + end += PMD_SIZE; + spin_lock_irqsave(&unaccepted_memory_lock, flags); for_each_set_bitrange_from(range_start, range_end, bitmap, DIV_ROUND_UP(end, PMD_SIZE)) { @@ -46,6 +78,13 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) bitmap = __va(boot_params.unaccepted_memory); + /* + * Also consider the unaccepted state of the *next* page. See fix #1 in + * the comment on load_unaligned_zeropad() in accept_memory(). + */ + if (!(end % PMD_SIZE)) + end += PMD_SIZE; + spin_lock_irqsave(&unaccepted_memory_lock, flags); while (start < end) { if (test_bit(start / PMD_SIZE, bitmap)) { diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 27b9eed5883b..f375ab784c78 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -715,6 +715,13 @@ static efi_status_t allocate_unaccepted_bitmap(struct boot_params *params, return EFI_SUCCESS; } + /* + * range_contains_unaccepted_memory() may need to check one 2M chunk + * beyond the end of RAM to deal with load_unaligned_zeropad(). Make + * sure that the bitmap is large enough handle it. + */ + max_addr += PMD_SIZE; + /* * If unaccepted memory is present, allocate a bitmap to track what * memory has to be accepted before access. From patchwork Wed Dec 7 01:49:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30593 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157561wrr; Tue, 6 Dec 2022 17:54:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf4J/U1OqsNTnqCktiqg48OPMJ3X1YFTGy7CKItlxVp8WsEUh+Qheu6uNxvpCNmmi2tShz+W X-Received: by 2002:a17:906:f108:b0:7c0:fd18:ebe1 with SMTP id gv8-20020a170906f10800b007c0fd18ebe1mr7358829ejb.592.1670378047889; Tue, 06 Dec 2022 17:54:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378047; cv=none; d=google.com; s=arc-20160816; b=OQbqM+XSc5bnu/02mB4cS/n4TvbULuYb521oJ0iCjVBIFbGf+oxeq0Iwo2ZEftN6+G Sf2ozrOIMpCD6pz9gSx/qHJ8GlUdcHqOuSgU9m7TjUSNsHBe/qR3eRlXYnqgZ+bIS2Od 5ljV8Dbi8l/4i1i/C42A9ALdQSsjdQuD8CmAkDZFpf40l8bWr14Gdm2J4GZ3ZIe/q/SS P2NalgyHOlY1tsciisbU6gpJvp7XBxLwXpw4vDYzIdGyN7DRwizhmKJ5Gw/ypr4tyT60 0/q6xGBGx21QD51nKmH7n6Cjx1IL0orxoVm4QGH/hn2gwlBcoQeMtaIa3ti+LhyC3CKj KkVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wEx2e5BFTypQruktgveotPG4xgTAKVo5RG9TIHHowI4=; b=Qs6C1/tdktFVw0W4ephCQLmQEWKSNOC9BLCxaQNoSRVaBFhlB4zz11lLlDBDfGYqRZ f4UYmNs+EaOFs49lPCJ+asmvP7VoaeAZ+QR6KAXFAS9IB2/woyIk13+9htOtbFXYp8FO uSb/dhKbk76kcJhgUS8zXraX2z9gyXccgC7s21MfXJjV5XeOMkYU4HkRnPVCmAd9LKjQ 6IFxLpyNqqot/TON/lZlqzPmp2l1w8CaXnCZre/+bVjl5aexnVHxUrdmV54FhktWa54Z jPMhTAX74kBs77qU8Nhr2J91iOzsLAaULPgNbXBhW5uAMGe2yOhwxItPZvoC+JVQD0H/ 6azQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=b+tzc5iW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f14-20020a0564021e8e00b00469274348b5si3653659edf.263.2022.12.06.17.53.44; Tue, 06 Dec 2022 17:54:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=b+tzc5iW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229997AbiLGBuX (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229967AbiLGBuS (ORCPT ); Tue, 6 Dec 2022 20:50:18 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 847B4532C9; Tue, 6 Dec 2022 17:50:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377801; x=1701913801; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4ZWHE8KNs9LD2gK29L+GSS8ovmdLBSodlS7K4yGoZwg=; b=b+tzc5iWH+w2K98w/pM61E3YYVoOZb81+1vo73SQNiZ2yMGzTmfuHtd/ Pn14Ce8FY1LUUvp8GRDS0BdlZ7pDYz5bnTvb5LQ54kmiowrpC2Zf4seQE jCAFW+E7m44jf3j3ejGHBn1WLC6IsTd8qGHJhUqHUfnB/F/XpjsoaFaZG 8f9reMzXbNLKAS5ZxzCI+UPaJMVokJYRQr4GbdSNE7M3poobqJtE14+bv 7lQBfa2WwubxGmfH8Uc7aB4JxCJnn2rfdzCHdaf+yqnFv7MyfP/kHEAiP apo2GaSaOhALmu6Ppeud5in7vrCLyHGXK1HSeAfWCb8PtJrijWnYa6U5r Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="316794452" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="316794452" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="640082289" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="640082289" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id A268A109C8F; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 11/14] x86: Disable kexec if system has unaccepted memory Date: Wed, 7 Dec 2022 04:49:30 +0300 Message-Id: <20221207014933.8435-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518331657366666?= X-GMAIL-MSGID: =?utf-8?q?1751518331657366666?= On kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used. boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented. Disable kexec on machines with unaccepted memory for now. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/kexec.h | 5 +++++ arch/x86/mm/unaccepted_memory.c | 16 ++++++++++++++++ include/linux/kexec.h | 7 +++++++ kernel/kexec.c | 4 ++++ kernel/kexec_file.c | 4 ++++ 5 files changed, 36 insertions(+) diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index a3760ca796aa..87abab578154 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -189,6 +189,11 @@ extern void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages); void arch_kexec_protect_crashkres(void); #define arch_kexec_protect_crashkres arch_kexec_protect_crashkres +#ifdef CONFIG_UNACCEPTED_MEMORY +int arch_kexec_load(void); +#define arch_kexec_load arch_kexec_load +#endif + void arch_kexec_unprotect_crashkres(void); #define arch_kexec_unprotect_crashkres arch_kexec_unprotect_crashkres diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c index a0a58486eb74..1745e6a65024 100644 --- a/arch/x86/mm/unaccepted_memory.c +++ b/arch/x86/mm/unaccepted_memory.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0-only +#include #include #include #include @@ -98,3 +99,18 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) return ret; } + +#ifdef CONFIG_KEXEC_CORE +int arch_kexec_load(void) +{ + if (!boot_params.unaccepted_memory) + return 0; + + /* + * TODO: Information on memory acceptance status has to be communicated + * between kernel. + */ + pr_warn_once("Disable kexec: not yet supported on systems with unaccepted memory\n"); + return -EOPNOTSUPP; +} +#endif diff --git a/include/linux/kexec.h b/include/linux/kexec.h index 41a686996aaa..6b75051d5271 100644 --- a/include/linux/kexec.h +++ b/include/linux/kexec.h @@ -444,6 +444,13 @@ static inline void arch_kexec_protect_crashkres(void) { } static inline void arch_kexec_unprotect_crashkres(void) { } #endif +#ifndef arch_kexec_load +static inline int arch_kexec_load(void) +{ + return 0; +} +#endif + #ifndef page_to_boot_pfn static inline unsigned long page_to_boot_pfn(struct page *page) { diff --git a/kernel/kexec.c b/kernel/kexec.c index cb8e6e6f983c..65dff44b487f 100644 --- a/kernel/kexec.c +++ b/kernel/kexec.c @@ -192,6 +192,10 @@ static inline int kexec_load_check(unsigned long nr_segments, { int result; + result = arch_kexec_load(); + if (result) + return result; + /* We only trust the superuser with rebooting the system. */ if (!capable(CAP_SYS_BOOT) || kexec_load_disabled) return -EPERM; diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c index 45637511e0de..8f1454c3776a 100644 --- a/kernel/kexec_file.c +++ b/kernel/kexec_file.c @@ -329,6 +329,10 @@ SYSCALL_DEFINE5(kexec_file_load, int, kernel_fd, int, initrd_fd, int ret = 0, i; struct kimage **dest_image, *image; + ret = arch_kexec_load(); + if (ret) + return ret; + /* We only trust the superuser with rebooting the system. */ if (!capable(CAP_SYS_BOOT) || kexec_load_disabled) return -EPERM; From patchwork Wed Dec 7 01:49:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30586 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157142wrr; Tue, 6 Dec 2022 17:52:41 -0800 (PST) X-Google-Smtp-Source: AA0mqf5z0PYOFYowVneDDbPPCGtbq8sJIKNKMi/W6oqhYVrwFEqzTYDI/wZjuDwGkCtuljriOnOB X-Received: by 2002:a17:906:2b81:b0:7c0:f716:e395 with SMTP id m1-20020a1709062b8100b007c0f716e395mr9105005ejg.160.1670377960923; Tue, 06 Dec 2022 17:52:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670377960; cv=none; d=google.com; s=arc-20160816; b=rPvw5EjueICXQi2qc4tvyjsDXe7auJiK5kdW0dlO9vVSPcR4GFAlka92TnAB1NeTqi H32w/R8wOXY72zPwfRU95S1K8qbvWVZ7itkK/Y20K+RWd+1hBgZnRRsTqCMQd05bKqdj o1ZeIznXGwVmCrZ0MY1yRtKa4rfquBQxn+1Rm5eBFrFDf6db7eqYGh/lJW+o6ieYOdDL L6gMkw5TmFqwLzyTA+kssvzYov3x/JbMkHbK0v2iz1L1H8khAwC9JhGD44X3BaFd7C+G i1h3iLNOXSE7oKcLQVbbFong11M17xznpzi5Q3XrfTb0sdAWSYe0Tn//bS5odv2bDJSH SZeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cSxfTqcJ8/wxc8OG/gJd56WJ+yhLlqOBVhLlG4LXL/I=; b=m1bn2xgH/K71BzhdBq82XXKmaMsnswdU6HZ3q0xrFQOq9tqZEzrFC4CamhunJI+utF ZdOIdmxyNE/D1Awrut9I8WBTd94lSRuIcXMDsESgrJhKQ8s+RL3g4cSo3QmgwcmAjIqy Z78cIwanWq+1ClWhxHHPClaMPeKfhOGliReDgPU2Iu2WeZdCWULxifOv80TYhN9wufYE 5aY+BSSf++YM7q5YR7tI3Lw/ZMkmit01rTt5EC0eZVPsTLWo42h8CmBaHaAoWIvoAan7 K0wZvM7zebD3+34E3uIDRfCrwHE0S3yTovwbgWLV/OLiLuPYrV364EpWybW2dNMriqdr Mqdg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="B6oBsEh/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca15-20020aa7cd6f000000b00463b0df4604si2865947edb.488.2022.12.06.17.52.18; Tue, 06 Dec 2022 17:52:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="B6oBsEh/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229985AbiLGBuo (ORCPT + 99 others); Tue, 6 Dec 2022 20:50:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229979AbiLGBuU (ORCPT ); Tue, 6 Dec 2022 20:50:20 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AD44537CE; Tue, 6 Dec 2022 17:50:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377803; x=1701913803; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sQNT2Cy2EmGfD2ueukiP5unO91F0G3BJKGWeML9T+YI=; b=B6oBsEh/LUWps9lKt/VqFNquuf7NbuwOtn1zPf+zvd0goLAENaYY1HuC hE5iP65lNQqFCkx6v0pZay4ds/r1fpBYNlsnYF5W7k1/JrhYi4z6bq2EF F1zV/sY3UUaJZs1I1Egw5QPw7pq5y31PGN47KQhuXGiflgpHIAB4P57Oo E5xW6bv2tQNsNVZo0ZuloOeL3LeRN368omdmFFOxNHBraKgQw5qOLIieu 2sM/4XoaDAChwOc5SusBMbUIhrZT/9Fxa051fHtI3DqVoyXmoNVORLRMi XXFi+tlXl1sy24U429RDfSq7brHDEsvJW3/DJWiGHxyIylO4H+exe4pIv Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="316794469" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="316794469" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="640082297" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="640082297" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id ADFAB109C90; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv8 12/14] x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub Date: Wed, 7 Dec 2022 04:49:31 +0300 Message-Id: <20221207014933.8435-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518240753946104?= X-GMAIL-MSGID: =?utf-8?q?1751518240753946104?= Memory acceptance requires a hypercall and one or multiple module calls. Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed. Signed-off-by: Kirill A. Shutemov Reviewed-by: Dave Hansen --- arch/x86/coco/tdx/tdx.c | 27 ------------------ arch/x86/include/asm/shared/tdx.h | 46 +++++++++++++++++++++++++++++++ arch/x86/include/asm/tdx.h | 19 ------------- 3 files changed, 46 insertions(+), 46 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index cfd4c95b9f04..12c14affa5f2 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -14,15 +14,6 @@ #include #include -/* TDX module Call Leaf IDs */ -#define TDX_GET_INFO 1 -#define TDX_GET_VEINFO 3 -#define TDX_GET_REPORT 4 -#define TDX_ACCEPT_PAGE 6 - -/* TDX hypercall Leaf IDs */ -#define TDVMCALL_MAP_GPA 0x10001 - /* MMIO direction */ #define EPT_READ 0 #define EPT_WRITE 1 @@ -45,24 +36,6 @@ #define TDREPORT_SUBTYPE_0 0 -/* - * Wrapper for standard use of __tdx_hypercall with no output aside from - * return code. - */ -static inline u64 _tdx_hypercall(u64 fn, u64 r12, u64 r13, u64 r14, u64 r15) -{ - struct tdx_hypercall_args args = { - .r10 = TDX_HYPERCALL_STANDARD, - .r11 = fn, - .r12 = r12, - .r13 = r13, - .r14 = r14, - .r15 = r15, - }; - - return __tdx_hypercall(&args, 0); -} - /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed(void) { diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h index e53f26228fbb..c5f12b90ef70 100644 --- a/arch/x86/include/asm/shared/tdx.h +++ b/arch/x86/include/asm/shared/tdx.h @@ -13,6 +13,15 @@ #define TDX_CPUID_LEAF_ID 0x21 #define TDX_IDENT "IntelTDX " +/* TDX module Call Leaf IDs */ +#define TDX_GET_INFO 1 +#define TDX_GET_VEINFO 3 +#define TDX_GET_REPORT 5 +#define TDX_ACCEPT_PAGE 6 + +/* TDX hypercall Leaf IDs */ +#define TDVMCALL_MAP_GPA 0x10001 + #ifndef __ASSEMBLY__ /* @@ -33,8 +42,45 @@ struct tdx_hypercall_args { /* Used to request services from the VMM */ u64 __tdx_hypercall(struct tdx_hypercall_args *args, unsigned long flags); +/* + * Wrapper for standard use of __tdx_hypercall with no output aside from + * return code. + */ +static inline u64 _tdx_hypercall(u64 fn, u64 r12, u64 r13, u64 r14, u64 r15) +{ + struct tdx_hypercall_args args = { + .r10 = TDX_HYPERCALL_STANDARD, + .r11 = fn, + .r12 = r12, + .r13 = r13, + .r14 = r14, + .r15 = r15, + }; + + return __tdx_hypercall(&args, 0); +} + + /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed(void); +/* + * Used in __tdx_module_call() to gather the output registers' values of the + * TDCALL instruction when requesting services from the TDX module. This is a + * software only structure and not part of the TDX module/VMM ABI + */ +struct tdx_module_output { + u64 rcx; + u64 rdx; + u64 r8; + u64 r9; + u64 r10; + u64 r11; +}; + +/* Used to communicate with the TDX module */ +u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, + struct tdx_module_output *out); + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */ diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 28d889c9aa16..234197ec17e4 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -20,21 +20,6 @@ #ifndef __ASSEMBLY__ -/* - * Used to gather the output registers values of the TDCALL and SEAMCALL - * instructions when requesting services from the TDX module. - * - * This is a software only structure and not part of the TDX module/VMM ABI. - */ -struct tdx_module_output { - u64 rcx; - u64 rdx; - u64 r8; - u64 r9; - u64 r10; - u64 r11; -}; - /* * Used by the #VE exception handler to gather the #VE exception * info from the TDX module. This is a software only structure @@ -55,10 +40,6 @@ struct ve_info { void __init tdx_early_init(void); -/* Used to communicate with the TDX module */ -u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, - struct tdx_module_output *out); - void tdx_get_ve_info(struct ve_info *ve); bool tdx_handle_virt_exception(struct pt_regs *regs, struct ve_info *ve); From patchwork Wed Dec 7 01:49:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30595 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157799wrr; Tue, 6 Dec 2022 17:54:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf6ItXwu2rVAJsVAal87xiefxVl51dRL5V3NDg3N14rJb1f2H9L6EwffOJebfOP056nlHnEQ X-Received: by 2002:aa7:dbca:0:b0:458:3f65:4e42 with SMTP id v10-20020aa7dbca000000b004583f654e42mr80027418edt.39.1670378091669; Tue, 06 Dec 2022 17:54:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378091; cv=none; d=google.com; s=arc-20160816; b=rXPeKpSqSI03+eAgqT3acusv3kFt2PevcPTMD3HrBs8c2s/oBhtJ6MsaV8OU6SsrPA e6cAc+PiUnlB8ZN9ufnVRnPX4tdZTDnlPKFi7qvZmC3bPLoCecKUt9pYs9+6RTGCmLGg B0f+eYDv2l0uddJXYEdTARavAQwBUK4bXQp8OVpFru0GtRyOE9LlITqrEqJGHVeoO67X ggHk9E3x9oIHWlfHmDrKdBbmEQqm1SQYdLTs/zB9Nzbp/LbDi5LMFqicChF+bDrIqPSO wkGy6PLpfNoHKd41q971zs5+TQnP6upRFmjIdfd5mrAUhwh1uIoHg4Olpa7IFbwNgNAj wQRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=u5eXy8raN7nY7Im1G/xDbIuFNC7MzRo5Pkm1h411bjY=; b=uTpd+tRppD+YkfnQXnV17shOdUu8RwVle+ep0SzPderPmD+h+vWC9V11RgsuyOI/Oz oJzY7txcpPObukwJaZm/Od3e+KIURdrMHNwmdJdh4Yg/GBwcAftNpY0wHyUTKEWw+HC8 e4DNqeJbVxEn9uLe6ngo0GK7xpDcPqgYBwmJ0JIjyV+nOl4flBSzlLy/g/ioq2UcZ7c7 s02N5PJH9r/K7UzhqtQp+0H2MKXQaABlnr3LxOA6XKA9FFwIhcwn5+kvIhGyt+kTJDIi +lvXfJT4KWiUoP0H31/x9glXjkicEvl03aK4N5Qfu7MPi9INqT9zU2iTaK+ppYwpjkiF hmzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=cKBynRfv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qq14-20020a17090720ce00b007bea4cb8356si12255431ejb.635.2022.12.06.17.54.28; Tue, 06 Dec 2022 17:54:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=cKBynRfv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230117AbiLGBv0 (ORCPT + 99 others); Tue, 6 Dec 2022 20:51:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230036AbiLGBum (ORCPT ); Tue, 6 Dec 2022 20:50:42 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AEC0537F4; Tue, 6 Dec 2022 17:50:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377809; x=1701913809; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SfxCo98L571pSepCQHUd/DhlAMN+hwpTgH7KtlWdFyw=; b=cKBynRfvF+15QWSzbGOp+v7F/JLuA0Beqk0zwYWtIBQzXT/M7zrNeWiJ S9J1AbQOnKeHSG+JVRiR5q9wE8B1FRFxwYSGzTVzkfqBdDSvNYKXC3bIN ypfXdKsRdFyJCIeqtfexIq1/nN1fjhSZ87DPcvkivvNsyZ7OEeX5lXBED qOx6qTIwYcyWgykdMllOAbE0Zu71oveEqxDuYJHQo57glFwzc3uJalaOK pRYCxh2shatQpqn250AR1dcfQAOWc8M5vgfP6L3Vpkjl056fLwNCQVy6O L5mzEqJ8YKkHGLkiQS2wu/jCYuORaVOKr/jl/wXxFmr+OQtMR3QJ14tNY w==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494588" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494588" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427711" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427711" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id B99B6109C91; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv8 13/14] x86/tdx: Refactor try_accept_one() Date: Wed, 7 Dec 2022 04:49:32 +0300 Message-Id: <20221207014933.8435-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518377726085138?= X-GMAIL-MSGID: =?utf-8?q?1751518377726085138?= Rework try_accept_one() to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argument and streamlines code on the caller side. Signed-off-by: Kirill A. Shutemov Suggested-by: Borislav Petkov Reviewed-by: Dave Hansen --- arch/x86/coco/tdx/tdx.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index 12c14affa5f2..cf6d9a0968d8 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -674,18 +674,18 @@ static bool tdx_cache_flush_required(void) return true; } -static bool try_accept_one(phys_addr_t *start, unsigned long len, - enum pg_level pg_level) +static unsigned long try_accept_one(phys_addr_t start, unsigned long len, + enum pg_level pg_level) { unsigned long accept_size = page_level_size(pg_level); u64 tdcall_rcx; u8 page_size; - if (!IS_ALIGNED(*start, accept_size)) - return false; + if (!IS_ALIGNED(start, accept_size)) + return 0; if (len < accept_size) - return false; + return 0; /* * Pass the page physical address to the TDX module to accept the @@ -704,15 +704,14 @@ static bool try_accept_one(phys_addr_t *start, unsigned long len, page_size = 2; break; default: - return false; + return 0; } - tdcall_rcx = *start | page_size; + tdcall_rcx = start | page_size; if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) - return false; + return 0; - *start += accept_size; - return true; + return accept_size; } /* @@ -749,21 +748,22 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) */ while (start < end) { unsigned long len = end - start; + unsigned long accept_size; /* * Try larger accepts first. It gives chance to VMM to keep - * 1G/2M SEPT entries where possible and speeds up process by - * cutting number of hypercalls (if successful). + * 1G/2M Secure EPT entries where possible and speeds up + * process by cutting number of hypercalls (if successful). */ - if (try_accept_one(&start, len, PG_LEVEL_1G)) - continue; - - if (try_accept_one(&start, len, PG_LEVEL_2M)) - continue; - - if (!try_accept_one(&start, len, PG_LEVEL_4K)) + accept_size = try_accept_one(start, len, PG_LEVEL_1G); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_2M); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_4K); + if (!accept_size) return false; + start += accept_size; } return true; From patchwork Wed Dec 7 01:49:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 30594 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3157716wrr; Tue, 6 Dec 2022 17:54:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf7l+eMvX3SqOGF2eKZcOfG3qOoU79z97qewZQm3j6doZOhcZr9cX1EK/WjPYJZR8/qspqlX X-Received: by 2002:aa7:db19:0:b0:46b:fc0d:ddc0 with SMTP id t25-20020aa7db19000000b0046bfc0dddc0mr22219243eds.214.1670378077507; Tue, 06 Dec 2022 17:54:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670378077; cv=none; d=google.com; s=arc-20160816; b=O4Gq+wL3JfUQy+SDMsVrJ/vFxqZPTVvxWK9oGsy0m5TglDmTxGTqhGETxJ19QZLEXA +l3AxJSH6CuBdItw5V4sDya7TdVdn4N/SmqtBonXrtG5O18o6ZSmEQf9RxSPIk1kvQzi 5lqJyYEsX7th6shMA2GjfpPQCGJg58AnVDDFiPunAY8jo5cDBRyNo7sSYsUQ8L2GR6aK iAOm3yYQDtc5pnw0rOKJulsEVMjPx6m7EsaYkHB41hf96i6Z52FyHkf/xTAQDViey6gT MxiRGVehWaLVQcjQLOlmdhzSKi8INtastWOXK7Ldmi5YbOi5Z+/XgxFp5JQyyqbP17Q+ OpUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZWDCWTy3ohQPGoxQ8BIZwGZ1Eh1/b+RdDV4ttehW9CE=; b=tTuhNTha9HJgVhpHihnwTUcg+g1S6gLeL+Kjf9oltKzttgDmowlpkujkb+kZb5REEt 7U9iYr/y6KDb764XLMx5Np/dD3tWo+YiC8P2vPrFwc3muZzG1TaGihr6+iRDV5nrzI5k RmtndgL5VNrOserWbQ1TRxmlsxCOUNhmENUJoNVEfuduhKj9ays2kVyrztGd82DNhfcy oBkKsP0cyhR1C8WfZu3qANxDibbhBXcYhpYFzcNciOTfPMujkkOD+2MQZJoKHuCrEhF6 YP8KiiqZhqBYDzN9pcTI/gDACtAjM7jklhIobwx4J+7ga83J9q/4pDtThij1BZCNqqiH Af0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=irXLGx9g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t5-20020a50d705000000b0046b8be29281si3093315edi.138.2022.12.06.17.54.13; Tue, 06 Dec 2022 17:54:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=irXLGx9g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230107AbiLGBvN (ORCPT + 99 others); Tue, 6 Dec 2022 20:51:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229962AbiLGBuj (ORCPT ); Tue, 6 Dec 2022 20:50:39 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3B0E537CA; Tue, 6 Dec 2022 17:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1670377804; x=1701913804; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l/GAlLVQNZ1o8mZSLHtW1AKFSz7H9NKdXs1f6iRZxW4=; b=irXLGx9gmwb1STNh5DToL8J28+CImlZM7mjBC/CYgDFT0m8DEkz47ND5 FMomLAs06UYy/9WHJVdLVKBadiMe2G7JnlUJSWyvcsUCwPKbBxVN4577S 849n1gOPW8/SO5NWGth++M4ky9gfc//jy9BQGodS20vD3gHDzyfuWH0HJ d75AzOfffXlCbGOYXDkXB513qMRARlEMckJa4fE1iwEPaMcFeWKFDQs8U t2mCfLLUww+d2n4C43MBj8iz9W921HduIjFx/YN8dVuwkabP4lQDT4Ejb KVoRocNwDP6IuJZhGS8M9B7u4j7EMaszf4Ly/uc1B4CXGZ39Bf1ScHQgS Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="315494535" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="315494535" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:50:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10553"; a="646427686" X-IronPort-AV: E=Sophos;i="5.96,223,1665471600"; d="scan'208";a="646427686" Received: from puneets1-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.38.123]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Dec 2022 17:49:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id C5AE0109C92; Wed, 7 Dec 2022 04:49:39 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Dave Hansen , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv8 14/14] x86/tdx: Add unaccepted memory support Date: Wed, 7 Dec 2022 04:49:33 +0300 Message-Id: <20221207014933.8435-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> References: <20221207014933.8435-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751518363076424689?= X-GMAIL-MSGID: =?utf-8?q?1751518363076424689?= Hookup TDX-specific code to accept memory. Accepting the memory is the same process as converting memory from shared to private: kernel notifies VMM with MAP_GPA hypercall and then accept pages with ACCEPT_PAGE module call. The implementation in core kernel uses tdx_enc_status_changed(). It already used for converting memory to shared and back for I/O transactions. Boot stub provides own implementation of tdx_accept_memory(). It is similar in structure to tdx_enc_status_changed(), but only cares about converting memory to private. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 2 + arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/error.c | 19 ++++++ arch/x86/boot/compressed/error.h | 1 + arch/x86/boot/compressed/mem.c | 33 +++++++++- arch/x86/boot/compressed/tdx-shared.c | 2 + arch/x86/boot/compressed/tdx.c | 39 +++++++++++ arch/x86/coco/tdx/Makefile | 2 +- arch/x86/coco/tdx/tdx-shared.c | 95 +++++++++++++++++++++++++++ arch/x86/coco/tdx/tdx.c | 86 +----------------------- arch/x86/include/asm/shared/tdx.h | 2 + arch/x86/include/asm/tdx.h | 2 + arch/x86/mm/unaccepted_memory.c | 9 ++- 13 files changed, 206 insertions(+), 88 deletions(-) create mode 100644 arch/x86/boot/compressed/tdx-shared.c create mode 100644 arch/x86/coco/tdx/tdx-shared.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 947937d327c6..bc031fd1d95e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -888,6 +888,8 @@ config INTEL_TDX_GUEST select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT select X86_MCE + select UNACCEPTED_MEMORY + select EFI_STUB help Support running as a guest under Intel TDX. Without this support, the guest kernel can not boot or run under TDX. diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index a9937b50c37f..38c42c2b4349 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -106,7 +106,7 @@ ifdef CONFIG_X86_64 endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o -vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdx-shared.o $(obj)/tdcall.o vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/find.o $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o diff --git a/arch/x86/boot/compressed/error.c b/arch/x86/boot/compressed/error.c index c881878e56d3..5313c5cb2b80 100644 --- a/arch/x86/boot/compressed/error.c +++ b/arch/x86/boot/compressed/error.c @@ -22,3 +22,22 @@ void error(char *m) while (1) asm("hlt"); } + +/* EFI libstub provides vsnprintf() */ +#ifdef CONFIG_EFI_STUB +void panic(const char *fmt, ...) +{ + static char buf[1024]; + va_list args; + int len; + + va_start(args, fmt); + len = vsnprintf(buf, sizeof(buf), fmt, args); + va_end(args); + + if (len && buf[len - 1] == '\n') + buf[len - 1] = '\0'; + + error(buf); +} +#endif diff --git a/arch/x86/boot/compressed/error.h b/arch/x86/boot/compressed/error.h index 1de5821184f1..86fe33b93715 100644 --- a/arch/x86/boot/compressed/error.h +++ b/arch/x86/boot/compressed/error.h @@ -6,5 +6,6 @@ void warn(char *m); void error(char *m) __noreturn; +void panic(const char *fmt, ...) __noreturn __cold; #endif /* BOOT_COMPRESSED_ERROR_H */ diff --git a/arch/x86/boot/compressed/mem.c b/arch/x86/boot/compressed/mem.c index 626e4f10ba2c..23a84c46aa9b 100644 --- a/arch/x86/boot/compressed/mem.c +++ b/arch/x86/boot/compressed/mem.c @@ -5,6 +5,8 @@ #include "error.h" #include "find.h" #include "math.h" +#include "tdx.h" +#include #define PMD_SHIFT 21 #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) @@ -12,10 +14,39 @@ extern struct boot_params *boot_params; +/* + * accept_memory() and process_unaccepted_memory() called from EFI stub which + * runs before decompresser and its early_tdx_detect(). + * + * Enumerate TDX directly from the early users. + */ +static bool early_is_tdx_guest(void) +{ + static bool once; + static bool is_tdx; + + if (!IS_ENABLED(CONFIG_INTEL_TDX_GUEST)) + return false; + + if (!once) { + u32 eax, sig[3]; + + cpuid_count(TDX_CPUID_LEAF_ID, 0, &eax, + &sig[0], &sig[2], &sig[1]); + is_tdx = !memcmp(TDX_IDENT, sig, sizeof(sig)); + once = true; + } + + return is_tdx; +} + static inline void __accept_memory(phys_addr_t start, phys_addr_t end) { /* Platform-specific memory-acceptance call goes here */ - error("Cannot accept memory"); + if (early_is_tdx_guest()) + tdx_accept_memory(start, end); + else + error("Cannot accept memory: unknown platform\n"); } /* diff --git a/arch/x86/boot/compressed/tdx-shared.c b/arch/x86/boot/compressed/tdx-shared.c new file mode 100644 index 000000000000..5ac43762fe13 --- /dev/null +++ b/arch/x86/boot/compressed/tdx-shared.c @@ -0,0 +1,2 @@ +#include "error.h" +#include "../../coco/tdx/tdx-shared.c" diff --git a/arch/x86/boot/compressed/tdx.c b/arch/x86/boot/compressed/tdx.c index 918a7606f53c..de1d4a87418d 100644 --- a/arch/x86/boot/compressed/tdx.c +++ b/arch/x86/boot/compressed/tdx.c @@ -3,12 +3,17 @@ #include "../cpuflags.h" #include "../string.h" #include "../io.h" +#include "align.h" #include "error.h" +#include "pgtable_types.h" #include #include #include +#include + +static u64 cc_mask; /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed(void) @@ -16,6 +21,38 @@ void __tdx_hypercall_failed(void) error("TDVMCALL failed. TDX module bug?"); } +static u64 get_cc_mask(void) +{ + struct tdx_module_output out; + unsigned int gpa_width; + + /* + * TDINFO TDX module call is used to get the TD execution environment + * information like GPA width, number of available vcpus, debug mode + * information, etc. More details about the ABI can be found in TDX + * Guest-Host-Communication Interface (GHCI), section 2.4.2 TDCALL + * [TDG.VP.INFO]. + * + * The GPA width that comes out of this call is critical. TDX guests + * can not meaningfully run without it. + */ + if (__tdx_module_call(TDX_GET_INFO, 0, 0, 0, 0, &out)) + error("TDCALL GET_INFO failed (Buggy TDX module!)\n"); + + gpa_width = out.rcx & GENMASK(5, 0); + + /* + * The highest bit of a guest physical address is the "sharing" bit. + * Set it for shared pages and clear it for private pages. + */ + return BIT_ULL(gpa_width - 1); +} + +u64 cc_mkdec(u64 val) +{ + return val & ~cc_mask; +} + static inline unsigned int tdx_io_in(int size, u16 port) { struct tdx_hypercall_args args = { @@ -70,6 +107,8 @@ void early_tdx_detect(void) if (memcmp(TDX_IDENT, sig, sizeof(sig))) return; + cc_mask = get_cc_mask(); + /* Use hypercalls instead of I/O instructions */ pio_ops.f_inb = tdx_inb; pio_ops.f_outb = tdx_outb; diff --git a/arch/x86/coco/tdx/Makefile b/arch/x86/coco/tdx/Makefile index 46c55998557d..2c7dcbf1458b 100644 --- a/arch/x86/coco/tdx/Makefile +++ b/arch/x86/coco/tdx/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += tdx.o tdcall.o +obj-y += tdx.o tdx-shared.o tdcall.o diff --git a/arch/x86/coco/tdx/tdx-shared.c b/arch/x86/coco/tdx/tdx-shared.c new file mode 100644 index 000000000000..ee74f7bbe806 --- /dev/null +++ b/arch/x86/coco/tdx/tdx-shared.c @@ -0,0 +1,95 @@ +#include +#include + +static unsigned long try_accept_one(phys_addr_t start, unsigned long len, + enum pg_level pg_level) +{ + unsigned long accept_size = page_level_size(pg_level); + u64 tdcall_rcx; + u8 page_size; + + if (!IS_ALIGNED(start, accept_size)) + return 0; + + if (len < accept_size) + return 0; + + /* + * Pass the page physical address to the TDX module to accept the + * pending, private page. + * + * Bits 2:0 of RCX encode page size: 0 - 4K, 1 - 2M, 2 - 1G. + */ + switch (pg_level) { + case PG_LEVEL_4K: + page_size = 0; + break; + case PG_LEVEL_2M: + page_size = 1; + break; + case PG_LEVEL_1G: + page_size = 2; + break; + default: + return 0; + } + + tdcall_rcx = start | page_size; + if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) + return 0; + + return accept_size; +} + +bool tdx_enc_status_changed_phys(phys_addr_t start, phys_addr_t end, bool enc) +{ + if (!enc) { + /* Set the shared (decrypted) bits: */ + start |= cc_mkdec(0); + end |= cc_mkdec(0); + } + + /* + * Notify the VMM about page mapping conversion. More info about ABI + * can be found in TDX Guest-Host-Communication Interface (GHCI), + * section "TDG.VP.VMCALL" + */ + if (_tdx_hypercall(TDVMCALL_MAP_GPA, start, end - start, 0, 0)) + return false; + + /* private->shared conversion requires only MapGPA call */ + if (!enc) + return true; + + /* + * For shared->private conversion, accept the page using + * TDX_ACCEPT_PAGE TDX module call. + */ + while (start < end) { + unsigned long len = end - start; + unsigned long accept_size; + + /* + * Try larger accepts first. It gives chance to VMM to keep + * 1G/2M Secure EPT entries where possible and speeds up + * process by cutting number of hypercalls (if successful). + */ + + accept_size = try_accept_one(start, len, PG_LEVEL_1G); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_2M); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_4K); + if (!accept_size) + return false; + start += accept_size; + } + + return true; +} + +void tdx_accept_memory(phys_addr_t start, phys_addr_t end) +{ + if (!tdx_enc_status_changed_phys(start, end, true)) + panic("Accepting memory failed: %#llx-%#llx\n", start, end); +} diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index cf6d9a0968d8..0aa95fe29045 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -674,46 +674,6 @@ static bool tdx_cache_flush_required(void) return true; } -static unsigned long try_accept_one(phys_addr_t start, unsigned long len, - enum pg_level pg_level) -{ - unsigned long accept_size = page_level_size(pg_level); - u64 tdcall_rcx; - u8 page_size; - - if (!IS_ALIGNED(start, accept_size)) - return 0; - - if (len < accept_size) - return 0; - - /* - * Pass the page physical address to the TDX module to accept the - * pending, private page. - * - * Bits 2:0 of RCX encode page size: 0 - 4K, 1 - 2M, 2 - 1G. - */ - switch (pg_level) { - case PG_LEVEL_4K: - page_size = 0; - break; - case PG_LEVEL_2M: - page_size = 1; - break; - case PG_LEVEL_1G: - page_size = 2; - break; - default: - return 0; - } - - tdcall_rcx = start | page_size; - if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) - return 0; - - return accept_size; -} - /* * Inform the VMM of the guest's intent for this physical page: shared with * the VMM or private to the guest. The VMM is expected to change its mapping @@ -722,51 +682,9 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len, static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) { phys_addr_t start = __pa(vaddr); - phys_addr_t end = __pa(vaddr + numpages * PAGE_SIZE); - - if (!enc) { - /* Set the shared (decrypted) bits: */ - start |= cc_mkdec(0); - end |= cc_mkdec(0); - } - - /* - * Notify the VMM about page mapping conversion. More info about ABI - * can be found in TDX Guest-Host-Communication Interface (GHCI), - * section "TDG.VP.VMCALL" - */ - if (_tdx_hypercall(TDVMCALL_MAP_GPA, start, end - start, 0, 0)) - return false; - - /* private->shared conversion requires only MapGPA call */ - if (!enc) - return true; + phys_addr_t end = __pa(vaddr + numpages * PAGE_SIZE); - /* - * For shared->private conversion, accept the page using - * TDX_ACCEPT_PAGE TDX module call. - */ - while (start < end) { - unsigned long len = end - start; - unsigned long accept_size; - - /* - * Try larger accepts first. It gives chance to VMM to keep - * 1G/2M Secure EPT entries where possible and speeds up - * process by cutting number of hypercalls (if successful). - */ - - accept_size = try_accept_one(start, len, PG_LEVEL_1G); - if (!accept_size) - accept_size = try_accept_one(start, len, PG_LEVEL_2M); - if (!accept_size) - accept_size = try_accept_one(start, len, PG_LEVEL_4K); - if (!accept_size) - return false; - start += accept_size; - } - - return true; + return tdx_enc_status_changed_phys(start, end, enc); } void __init tdx_early_init(void) diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h index c5f12b90ef70..a21685fd7c4f 100644 --- a/arch/x86/include/asm/shared/tdx.h +++ b/arch/x86/include/asm/shared/tdx.h @@ -82,5 +82,7 @@ struct tdx_module_output { u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out); +void tdx_accept_memory(phys_addr_t start, phys_addr_t end); + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */ diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 234197ec17e4..3a7340ad9a4b 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -50,6 +50,8 @@ bool tdx_early_handle_ve(struct pt_regs *regs); int tdx_mcall_get_report0(u8 *reportdata, u8 *tdreport); +bool tdx_enc_status_changed_phys(phys_addr_t start, phys_addr_t end, bool enc); + #else static inline void tdx_early_init(void) { }; diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c index 1745e6a65024..45706c684ca5 100644 --- a/arch/x86/mm/unaccepted_memory.c +++ b/arch/x86/mm/unaccepted_memory.c @@ -7,6 +7,7 @@ #include #include +#include #include /* Protects unaccepted memory bitmap */ @@ -62,7 +63,13 @@ void accept_memory(phys_addr_t start, phys_addr_t end) unsigned long len = range_end - range_start; /* Platform-specific memory-acceptance call goes here */ - panic("Cannot accept memory: unknown platform\n"); + if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { + tdx_accept_memory(range_start * PMD_SIZE, + range_end * PMD_SIZE); + } else { + panic("Cannot accept memory: unknown platform\n"); + } + bitmap_clear(bitmap, range_start, len); } spin_unlock_irqrestore(&unaccepted_memory_lock, flags);