From patchwork Fri Nov 11 18:35:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kristen Carlson Accardi X-Patchwork-Id: 1526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp904778wru; Fri, 11 Nov 2022 10:36:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf6bC5xWuMzpytvndrM6Hn8HVMMnEZ1a4p5UbKTRC2gZfzcynbOrEC0g4yhGSMbER7iyE9DZ X-Received: by 2002:aa7:9e46:0:b0:56c:34af:a597 with SMTP id z6-20020aa79e46000000b0056c34afa597mr3903525pfq.28.1668191803817; Fri, 11 Nov 2022 10:36:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668191803; cv=none; d=google.com; s=arc-20160816; b=Yud/1EFLuXDFjUzVp+BPqsYChjjlRPNM8kszw4f1Rw0no6fdHaVaNHhu88Q/I7VI2h I1CiqpFG9A47bccy+4Nzeqpc3yKUtOEJoO7DjLNlP20evG6Hi6DOBQfNXnJ+aHG2a4Gh 8ZjqsE4ag4rMVpur1C0lRQabpA0WjIfUBo+8z4VdOzJLJZlXayfguvWEIotP8gqflRSZ svAFs1XEjF6SSSfd+kSWI1Y0f88bN4iQ4hlcqSsMoOcqzVChsjRKavTFmleXmT1LZNHz vVCGoonYoJ5T5Fv4348g2gVj1Q+RWCR1537YOJVEs5cf3Uu1GdpFM1OO7JMm/MumoEUd IDBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=XlyUxGMtRYjJ5jVlVkxtDJ1Th4AFxmPRFl9N4w9sUBM=; b=WDnNDImDt6To68G6sZPL1jI2Io8LjDwLmiI6fTAeQ2IsNcsG47jf2hpit7EyCTLI7R 6dkL9mWn8Gc5ViZU5KF9mFElm28bK0HrPmTyU4VSQr1NfxBNxk8HGn5opaNTRGeSfH1/ joGYwLTmpU78rA+RNQmqZMTMe+FQkRL5SySkG+qM1RcmjzyJlFAib+/B9QBg+LUZDyyW HWH0yZSktZg1s9XlrhekbQxzGqY+rHIbVmTCONB7Yc4GIawmu7icwIATnXkRSWn+bSIR Ei4eeGThsMJnUHpzhMhrlVt2YI1IOHaMV+5xi0iPdww36G0nl2keFd9lgjc5F5DroZK3 E3ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=a3maXEy2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u14-20020a056a00124e00b005615ef4cfccsi3270014pfi.185.2022.11.11.10.36.28; Fri, 11 Nov 2022 10:36:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=a3maXEy2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233962AbiKKSgG (ORCPT + 99 others); Fri, 11 Nov 2022 13:36:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234140AbiKKSf4 (ORCPT ); Fri, 11 Nov 2022 13:35:56 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23CF583692; Fri, 11 Nov 2022 10:35:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1668191739; x=1699727739; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=zkMqL68m/5zo4IQcHhhj9rd1XQDhqh5jgd3Tu0wSwho=; b=a3maXEy2Kf14SUJXdiryH8v/8W9m1yBW5azsFEreBJra7juFsZRfq5iC k4qBIlbchCBOjQa+e/HrU7Cjjt7lt6Szai2UKDulwMZdVsLgRkB0BR5pj iR9j7MEDjHOHE7ErYoltMlGX7noP2ykYqOCx7b5FTBlBznc1Nei+OKcJU us6oLoUK7jlAuemHcArQ9aAxcu7dcc9VOEtfuvCABUX9s9XV0nTZtPNnx yA83GSWanNtwNa6OlDKkKvd62qsqROBKn839afnL4z8jiixwMqKyjGaNW m9WDYllLn4kMnA5ff0v9aXnjBU32uKPPTEGh4yZkh1VZiXySNox+6x4bn g==; X-IronPort-AV: E=McAfee;i="6500,9779,10528"; a="292050285" X-IronPort-AV: E=Sophos;i="5.96,157,1665471600"; d="scan'208";a="292050285" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2022 10:35:38 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10528"; a="640089155" X-IronPort-AV: E=Sophos;i="5.96,157,1665471600"; d="scan'208";a="640089155" Received: from hermesli-mobl.amr.corp.intel.com (HELO kcaccard-desk.amr.corp.intel.com) ([10.212.218.5]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2022 10:35:38 -0800 From: Kristen Carlson Accardi To: jarkko@kernel.org, dave.hansen@linux.kernel.org, tj@kernel.org, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, cgroups@vger.kernel.org Cc: zhiquan1.li@intel.com Subject: [PATCH 00/26] Add Cgroup support for SGX EPC memory Date: Fri, 11 Nov 2022 10:35:05 -0800 Message-Id: <20221111183532.3676646-1-kristen@linux.intel.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749225888933779701?= X-GMAIL-MSGID: =?utf-8?q?1749225888933779701?= Utilize the Miscellaneous cgroup controller to regulate the distribution of SGX EPC memory, which is a subset of system RAM that is used to provide SGX-enabled applications with protected memory, and is otherwise inaccessible. SGX EPC memory allocations are separate from normal RAM allocations, and is managed solely by the SGX subsystem. The existing cgroup memory controller cannot be used to limit or account for SGX EPC memory. This patchset implements the support for sgx_epc memory within the misc cgroup controller, and then utilizes the misc cgroup controller to provide support for setting the total system capacity, max limit per cgroup, and events. This work was originally authored by Sean Christopherson a few years ago, and was modified to work with more recent kernels, and to utilize the misc cgroup controller rather than a custom controller. It is currently based on top of the MCA patches. Here's the MCA patchset for reference. https://lore.kernel.org/linux-sgx/2d52c8c4-8ed0-6df2-2911-da5b9fcc9ae4@intel.com/T/#t The patchset adds support for multiple LRUs to track both reclaimable EPC pages (i.e. pages the reclaimer knows about), as well as unreclaimable EPC pages (i.e. pages which the reclaimer isn't aware of, such as va pages). These pages are assigned to an LRU, as well as an enclave, so that an enclave's full EPC usage can be tracked, and limited to a max value. During OOM events, an enclave can be have its memory zapped, and all the EPC pages not tracked by the reclaimer can be freed. I appreciate your comments and feedback. Kristen Carlson Accardi (13): x86/sgx: Add 'struct sgx_epc_lru' to encapsulate lru list(s) x86/sgx: Use sgx_epc_lru for existing active page list x86/sgx: Track epc pages on reclaimable or unreclaimable lists cgroup/misc: Add notifier block list support for css events cgroup/misc: Expose root_misc cgroup/misc: Expose parent_misc() cgroup/misc: allow users of misc cgroup to read specific cgroup usage cgroup/misc: allow misc cgroup consumers to read the max value cgroup/misc: Add private per cgroup data to struct misc_cg cgroup/misc: Add tryget functionality for misc controller cgroup/misc: Add SGX EPC resource type x86/sgx: Add support for misc cgroup controller Docs/x86/sgx: Add description for cgroup support Sean Christopherson (13): x86/sgx: Call cond_resched() at the end of sgx_reclaim_pages() x86/sgx: Store struct sgx_encl when allocating new va pages x86/sgx: Introduce RECLAIM_IN_PROGRESS flag for EPC pages x86/sgx: Use a list to track to-be-reclaimed pages during reclaim x86/sgx: Add EPC page flags to identify type of page x86/sgx: Allow reclaiming up to 32 pages, but scan 16 by default x86/sgx: Return the number of EPC pages that were successfully reclaimed x86/sgx: Add option to ignore age of page during EPC reclaim x86/sgx: Add helper to retrieve SGX EPC LRU given an EPC page x86/sgx: Prepare for multiple LRUs x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgroup x86/sgx: Add helper to grab pages from an arbitrary EPC LRU x86/sgx: Add EPC OOM path to forcefully reclaim EPC Documentation/x86/sgx.rst | 77 ++++ arch/x86/Kconfig | 13 + arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/encl.c | 89 ++++- arch/x86/kernel/cpu/sgx/encl.h | 4 +- arch/x86/kernel/cpu/sgx/epc_cgroup.c | 561 +++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/epc_cgroup.h | 59 +++ arch/x86/kernel/cpu/sgx/ioctl.c | 13 +- arch/x86/kernel/cpu/sgx/main.c | 405 +++++++++++++++---- arch/x86/kernel/cpu/sgx/sgx.h | 96 ++++- arch/x86/kernel/cpu/sgx/virt.c | 28 +- include/linux/misc_cgroup.h | 71 ++++ kernel/cgroup/misc.c | 145 ++++++- 13 files changed, 1446 insertions(+), 116 deletions(-) create mode 100644 arch/x86/kernel/cpu/sgx/epc_cgroup.c create mode 100644 arch/x86/kernel/cpu/sgx/epc_cgroup.h