From patchwork Mon Feb 26 19:09:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206869 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281225dyb; Mon, 26 Feb 2024 11:13:39 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV8+ByoDFwenrY7oUroEJ7A7QspinpsT03kHChnfy80r4oXRN3p5l2W+r381Su70YNOP+U6pAr/nYMz8PcNUbv2fl/Jug== X-Google-Smtp-Source: AGHT+IHMnDdIkzf1tB8tHtOxDvrHxEfs+g14EsZb97XVP6tCOcF7gHSewe2fHjsADqyI9Cox0CA/ X-Received: by 2002:a67:ec53:0:b0:470:494d:64a0 with SMTP id z19-20020a67ec53000000b00470494d64a0mr5751873vso.5.1708974819644; Mon, 26 Feb 2024 11:13:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974819; cv=pass; d=google.com; s=arc-20160816; b=k20J+fywhb3Fx8iZe8W4Llla6dJ1VFQdwoCBLxy7SfgIJ5h55CjFJD5HtZt8mLll07 lCSO4K3zrMC+I2W7Efdi7ddzF67k27skkTOGksCRGQ+TkicxYvsNcCpNliqfJwX6auVw /JnyYrHxv7m7owDanRW90ZAlbpK57gPHm7sSakqcc9xkqFVYG3tP9YFjoAmaGZ7l2OpM rcgsrUFtjb59A9a5VKOzKSug0s21dBc7Veuusw/kT3u3kop830gCGWVf2dFEKp2pYvQK dDKzrAi1Ehl9DLgS5432e2tY1dW8zzzZPpr9Cmhi06nrlrQZ0tnzJGo5737uGxXvDqCO jvDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=w9uxx0p4IqyzsB+rWsNbj/igvIL8fkPgh5yQf8P8jHg=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=QaTMZ741tEuoM7G1lEvvxgYBeajvTKKj1xZMxijdvXqscasU6NEQUix/NkTP7eHZfI roFR4PUn0kqbKFT/fdntwh7sLYCKiDLKLELOSym5SF1HhzjprQG4VgkMLPBj7qTT1yQb T4Uv7iuWvYwvXrHOpIEiNzeAmPJLQHzHtM4k2BWT7iKIxODACbbugbCE1A96OsBOPtrK xpbVPH4z2GD/W0jNQlgExNX5LzFLiiDjjJllHUZKF+95ihqkfooMIIHO8YLrxGRdUc/q LQFsYTNywB5zmTEfzL7FlfVoHKM7/IdB6+SdXuDcUJu7/bvPq7zU9oD4E/5PbehevgAh 174g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SQNLRYbW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id ay18-20020a05622a229200b0042e2ad75644si6342960qtb.686.2024.02.26.11.13.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:13:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SQNLRYbW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 5FCAE1C2825A for ; Mon, 26 Feb 2024 19:13:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BCDCF132C25; Mon, 26 Feb 2024 19:10:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SQNLRYbW" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1590512FF87 for ; Mon, 26 Feb 2024 19:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974602; cv=none; b=h2earKiOX6rc+gfSmt5Qmby7evS1vytg8/KybE0iNxG74DCj73HjOqZiQiTHSFHXtOYC7MZ/Dsii9aqOH8crKX2tPpfO48eYiISBVxqX1k/8b40+BdUr0Wz3r/rh2UI8WOEy5R147m8aEhmScj5YM2s9AyupkwPTXAda++C0vVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974602; c=relaxed/simple; bh=P5XlQHAG5P20JDtEHex7uiurCNEBN1oum2HZdu24mXg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=PkTj2D33u48de/2hOzEucGLwFMaTtEcs74qWRwYTRqHtDDRTxWGLugbJkbpASUDT3BS5NAWEp+/fUcIDmOCaiGmdM+gkYf9BxcFGI3j1a7CZ0s8B6nPXiTqYhlstY1B6IU4vQnE1lE5S92Fkp0w0AE/q51061cdWn1RpZlvNOg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SQNLRYbW; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974601; x=1740510601; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P5XlQHAG5P20JDtEHex7uiurCNEBN1oum2HZdu24mXg=; b=SQNLRYbWkiFCbFapC0Jlw12doMrenHQeKQ8Y6XoiyNgmlR9yqYaLW5OR tEzJCSidkU5cpvnP+BYQAkGtEknOLYrlHxRfO4et1X9tPgp0FF3frUHvX UIxaZQ/AmQIKep3uPvhsJKY1YNaWY1NcY/DIQenorb1i6tJfZ6bp6dnhQ +llE4A9jNFm9naQOD+rFKwTXvF3BKJI+KOCY/l7kCbsIMQ/pZ2HfD0s7/ 0LZ7VQCtzAMB+qVfDuX3Cj6HG8bXx5/Dp3Umoer0q7s6CDPckBvuFs6AR n6xOVhzX8NDMZadIfiY6WluYDqAlHdzF3igF3MdL5/O0TcG1yuRwLcX+d Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721366" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721366" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911452" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:59 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 4/9] thp: Add thp_get_unmapped_area_vmflags() Date: Mon, 26 Feb 2024 11:09:46 -0800 Message-Id: <20240226190951.3240433-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791989980200435257 X-GMAIL-MSGID: 1791989980200435257 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled. Signed-off-by: Rick Edgecombe --- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ef7251dfd9f9 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -139,6 +139,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags); void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); @@ -286,6 +289,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {} #define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return 0; +} + static inline bool can_split_folio(struct folio *folio, int *pextra_pins) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e9ef43a719a5..f235f6d3ff62 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -628,7 +628,8 @@ static inline bool is_transparent_hugepage(struct folio *folio) static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, - loff_t off, unsigned long flags, unsigned long size) + loff_t off, unsigned long flags, unsigned long size, + vm_flags_t vm_flags) { loff_t off_end = off + len; loff_t off_align = round_up(off, size); @@ -641,8 +642,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area_vmflags(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags, vm_flags); /* * The failure might be due to length padding. The caller will retry @@ -662,17 +663,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, return ret; } -unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) { unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE, vm_flags); if (ret) return ret; - return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); + return mm_get_unmapped_area_vmflags(current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return thp_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index ac7601d05e89..e02bb17fef5b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1857,16 +1857,18 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, */ pgoff = 0; get_area = shmem_get_unmapped_area; - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; } - if (get_area) + if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - else + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + /* Ensures that larger anonymous mappings are THP aligned. */ + addr = thp_get_unmapped_area_vmflags(file, addr, len, + pgoff, flags, vm_flags); + } else { addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, pgoff, flags, vm_flags); + } if (IS_ERR_VALUE(addr)) return addr;