From patchwork Mon Feb 26 19:09:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206872 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281534dyb; Mon, 26 Feb 2024 11:14:12 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW395JQ/TLl3l3wyYdxF/CfFsIb6C45YApMRZ3+/7Yab//nUSQcTG/VVztb6K7ElBs9yr4kSwC3i9x6xqm2lZGMeoWCLw== X-Google-Smtp-Source: AGHT+IFSzrdYcqfIAOjvRp0bVTW+0POl2l9mYy5FoUe161/4D13C3irK/JtknxPtR3Hv03DNzTpd X-Received: by 2002:a05:6a20:d90f:b0:1a0:ef2c:eeec with SMTP id jd15-20020a056a20d90f00b001a0ef2ceeecmr69946pzb.40.1708974851889; Mon, 26 Feb 2024 11:14:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974851; cv=pass; d=google.com; s=arc-20160816; b=kS68yjPG0HEsiX9A/G9ntODZvFNpqB60OTHIMqqDH4BBgG9L4RXy6Bmlw3CkB0CToE +obbDUF2CZbJ5lgK9C1b3Ssoe0DGjqgsI0Xfe5Ut/K5XjHyCFp1QG4eyqmJ3fmVuprjo iZposimgEQYWImvJFRtrZ6AJ4yK2am/WImfbfL+SmR4dPhnch6pUYECS9oglq59emFpq A7xBYHWK5ULmcfdWxzt31sjANA8vEMcsAV6XJk8SYE0YerkZeHF49Ic+PSACE7+tggtA fe3E6EknfE3JbAGjCk9hAlXVYwuBGoUNfBoQaB1B4Yp9d+97Lcsem8XNDnrb8JKw2Re4 Ux+g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=bL1SZY389qbnApMbft1l+2IEAzO4i9QutCx173tdjdc=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=xr6fHIWotpLIK4HWfU5NpYTMM/PyA6HjKr+GxDPOfD9tUuP7RRjkdX4QOgj0IW+qy0 R8Keq7dbe0G93BgacAgC2uFzlTyMOeaBItkuqcu2bhBoUtsJMjfQCkyaHyyy5ABkxodn BEcPAdMf0Rumq6gQZTd90syLTBdlWPz6aC+PJiYXkhsLb5vPT04vrxMO3qGNymIPFrWu 4VsVsgGoPRNyJ6CxXS19ONkusxlYgXnaSzmUYj57mTzzXParzQp9TzAEMgX6UpQ/jb7V uKplMSpKmhEHNaX0CngYAmNM68uAm98fJiZ4Ej26zsH9l6i8LF+eR5pW7cPcs55zjdCU GwCg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="hr8A/QW6"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82204-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id n129-20020a632787000000b005e43cb39729si4142794pgn.870.2024.02.26.11.14.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="hr8A/QW6"; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82204-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id A9AFA292569 for ; Mon, 26 Feb 2024 19:14:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 560A41332AB; Mon, 26 Feb 2024 19:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="hr8A/QW6" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEE2912FB3E for ; Mon, 26 Feb 2024 19:09:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974600; cv=none; b=bqyjr5HpBltpIoTXmEhxLhxzSGlHkbOUMuiVKnb/Y2BgLBPmsyvVING+kfXHsKWxfUtfdCGS6z392R13g232Ip5WjBDEtexEdb3PIf8fesKzYoWTBqLbU2JWhwqhH9VmV2/WhcWHggGDwVQDAwVy+AxRlXUWvZnUhsEQYMZckVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974600; c=relaxed/simple; bh=g4JChMg3I76Dgv8nBf8Y40nYEQC6u5audbIsXOb5O2c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OienkmOUkZsYvl9n5Blm7Lu/B+Gnfvx5tTxokZLD964lSVL6bmQX7adpGyaKdU8jzmF7aTWlL3wtzlKOBXw0FUKr4GxP1LK+wsSWwN7kD0ZYkR121xJuV+m3URAzm1mCzhIkTiUKBBtbV3VnJ02N6w97SGFYrJlm3PzA39tFNag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=hr8A/QW6; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974599; x=1740510599; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=g4JChMg3I76Dgv8nBf8Y40nYEQC6u5audbIsXOb5O2c=; b=hr8A/QW6sahH/WkOLVvk9UD7bEakOspbTcARDTepUozhnyJdgzxAvC+y Ypw/iaWxSvdwarTrgnU2ukyOdw5Xv1M3tWgie9oDEdPExO6dfY8SbNeeD 9vcd5KAQ5Tw+/xhvwKfbyVXpJ38cpHej2BWpk32cS7ziNjR1yopnWgYaX B2Zla/RWXRc3YAvCV/uoBVei+QSKcioRr6MFV0wVQ1A73KysWOLta5Hmg AiiP797xC+8o9AhnLoNpwWBRDUyxJAN06luhLqPdr0RCzaBnLFSJs4JCM sbnb4l1ZQ2hIO3O6VBwvKZY6TZSYrMY8bpNvxb3gLjVqSKtMsaZnrs6fj Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721326" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721326" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911442" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:57 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 1/9] mm: Switch mm->get_unmapped_area() to a flag Date: Mon, 26 Feb 2024 11:09:43 -0800 Message-Id: <20240226190951.3240433-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990014435335613 X-GMAIL-MSGID: 1791990014435335613 The mm_struct contains a function pointer *get_unmapped_area(), which is set to either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() during the initialization of the mm. Since the function pointer only ever points to two functions that are named the same across all arch's, a function pointer is not really required. In addition future changes will want to add versions of the functions that take additional arguments. So to save a pointers worth of bytes in mm_struct, and prevent adding additional function pointers to mm_struct in future changes, remove it and keep the information about which get_unmapped_area() to use in a flag. Add the new flag to MMF_INIT_MASK so it doesn't get clobbered on fork by mmf_init_flags(). Most MM flags get clobbered on fork. In the pre-existing behavior mm->get_unmapped_area() would get copied to the new mm in dup_mm(), so not clobbering the flag preserves the existing behavior around inherriting the topdown-ness. Introduce a helper, mm_get_unmapped_area(), to easily convert code that refers to the old function pointer to instead select and call either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() based on the flag. Then drop the mm->get_unmapped_area() function pointer. Leave the get_unmapped_area() pointer in struct file_operations alone. The main purpose of this change is to reorganize in preparation for future changes, but it also converts the calls of mm->get_unmapped_area() from indirect branches into a direct ones. The stress-ng bigheap benchmark calls realloc a lot, which calls through get_unmapped_area() in the kernel. On x86, the change yielded a ~1% improvement there on a retpoline config. In testing a few x86 configs, removing the pointer unfortunately didn't result in any actual size reductions in the compiled layout of mm_struct. But depending on compiler or arch alignment requirements, the change could shrink the size of mm_struct. Signed-off-by: Rick Edgecombe Acked-by: Dave Hansen Acked-by: Liam R. Howlett Reviewed-by: Kirill A. Shutemov --- v2: - Fix comment on MMF_TOPDOWN (Kirill, rppt) - Move MMF_TOPDOWN to actually unused bit - Add MMF_TOPDOWN to MMF_INIT_MASK so it doesn't get clobbered on fork, and result in the children using the search up path. - New lower performance results after above bug fix - Add Reviews and Acks --- arch/s390/mm/hugetlbpage.c | 2 +- arch/s390/mm/mmap.c | 4 ++-- arch/sparc/kernel/sys_sparc_64.c | 15 ++++++--------- arch/sparc/mm/hugetlbpage.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/mm/hugetlbpage.c | 2 +- arch/x86/mm/mmap.c | 4 ++-- drivers/char/mem.c | 2 +- drivers/dax/device.c | 6 +++--- fs/hugetlbfs/inode.c | 2 +- fs/proc/inode.c | 15 ++++++++------- fs/ramfs/file-mmu.c | 2 +- include/linux/mm_types.h | 6 +----- include/linux/sched/coredump.h | 5 ++++- include/linux/sched/mm.h | 5 +++++ io_uring/io_uring.c | 2 +- mm/debug.c | 6 ------ mm/huge_memory.c | 6 +++--- mm/mmap.c | 21 ++++++++++++++++++--- mm/shmem.c | 11 +++++------ mm/util.c | 6 +++--- 21 files changed, 68 insertions(+), 58 deletions(-) diff --git a/arch/s390/mm/hugetlbpage.c b/arch/s390/mm/hugetlbpage.c index 297a6d897d5a..c2d2850ec8d5 100644 --- a/arch/s390/mm/hugetlbpage.c +++ b/arch/s390/mm/hugetlbpage.c @@ -328,7 +328,7 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, goto check_asce_limit; } - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) addr = hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/s390/mm/mmap.c b/arch/s390/mm/mmap.c index fc9a7dc26c5e..cd52d72b59cf 100644 --- a/arch/s390/mm/mmap.c +++ b/arch/s390/mm/mmap.c @@ -182,10 +182,10 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) */ if (mmap_is_legacy(rlim_stack)) { mm->mmap_base = mmap_base_legacy(random_factor); - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { mm->mmap_base = mmap_base(random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } diff --git a/arch/sparc/kernel/sys_sparc_64.c b/arch/sparc/kernel/sys_sparc_64.c index 1e9a9e016237..1dbf7211666e 100644 --- a/arch/sparc/kernel/sys_sparc_64.c +++ b/arch/sparc/kernel/sys_sparc_64.c @@ -218,14 +218,10 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, unsigned long len, unsigned long pgoff, unsigned long flags) { unsigned long align_goal, addr = -ENOMEM; - unsigned long (*get_area)(struct file *, unsigned long, - unsigned long, unsigned long, unsigned long); - - get_area = current->mm->get_unmapped_area; if (flags & MAP_FIXED) { /* Ok, don't mess with it. */ - return get_area(NULL, orig_addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags); } flags &= ~MAP_SHARED; @@ -238,7 +234,8 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u align_goal = (64UL * 1024); do { - addr = get_area(NULL, orig_addr, len + (align_goal - PAGE_SIZE), pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, + len + (align_goal - PAGE_SIZE), pgoff, flags); if (!(addr & ~PAGE_MASK)) { addr = (addr + (align_goal - 1UL)) & ~(align_goal - 1UL); break; @@ -256,7 +253,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u * be obtained. */ if (addr & ~PAGE_MASK) - addr = get_area(NULL, orig_addr, len, pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, len, pgoff, flags); return addr; } @@ -292,7 +289,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap == RLIM_INFINITY || sysctl_legacy_va_layout) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { /* We know it's 32-bit */ unsigned long task_size = STACK_TOP32; @@ -303,7 +300,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap = (task_size / 6 * 5); mm->mmap_base = PAGE_ALIGN(task_size - gap - random_factor); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c index b432500c13a5..38a1bef47efb 100644 --- a/arch/sparc/mm/hugetlbpage.c +++ b/arch/sparc/mm/hugetlbpage.c @@ -123,7 +123,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, (!vma || addr + len <= vm_start_gap(vma))) return addr; } - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/x86/kernel/cpu/sgx/driver.c b/arch/x86/kernel/cpu/sgx/driver.c index 262f5fb18d74..22b65a5f5ec6 100644 --- a/arch/x86/kernel/cpu/sgx/driver.c +++ b/arch/x86/kernel/cpu/sgx/driver.c @@ -113,7 +113,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file, if (flags & MAP_FIXED) return addr; - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } #ifdef CONFIG_COMPAT diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 5804bbae4f01..6d77c0039617 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -141,7 +141,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, } get_unmapped_area: - if (mm->get_unmapped_area == arch_get_unmapped_area) + if (!test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_bottomup(file, addr, len, pgoff, flags); else diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c index c90c20904a60..a2cabb1c81e1 100644 --- a/arch/x86/mm/mmap.c +++ b/arch/x86/mm/mmap.c @@ -129,9 +129,9 @@ static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base, void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) { if (mmap_is_legacy()) - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); else - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base, arch_rnd(mmap64_rnd_bits), task_size_64bit(0), diff --git a/drivers/char/mem.c b/drivers/char/mem.c index 3c6670cf905f..9b80e622ae80 100644 --- a/drivers/char/mem.c +++ b/drivers/char/mem.c @@ -544,7 +544,7 @@ static unsigned long get_unmapped_area_zero(struct file *file, } /* Otherwise flags & MAP_PRIVATE: with no shmem object beneath it */ - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); #else return -ENOSYS; #endif diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 93ebedc5ec8c..47c126d37b59 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -329,14 +329,14 @@ static unsigned long dax_get_unmapped_area(struct file *filp, if ((off + len_align) < off) goto out; - addr_align = current->mm->get_unmapped_area(filp, addr, len_align, - pgoff, flags); + addr_align = mm_get_unmapped_area(current->mm, filp, addr, len_align, + pgoff, flags); if (!IS_ERR_VALUE(addr_align)) { addr_align += (off - addr_align) & (align - 1); return addr_align; } out: - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } static const struct address_space_operations dev_dax_aops = { diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index f757d4f7ad98..a63d2eee086f 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -242,7 +242,7 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * If architectures have special needs, they should define their own * version of hugetlb_get_unmapped_area. */ - if (mm->get_unmapped_area == arch_get_unmapped_area_topdown) + if (test_bit(MMF_TOPDOWN, &mm->flags)) return hugetlb_get_unmapped_area_topdown(file, addr, len, pgoff, flags); return hugetlb_get_unmapped_area_bottomup(file, addr, len, diff --git a/fs/proc/inode.c b/fs/proc/inode.c index b33e490e3fd9..6f4c2e21e68f 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -454,15 +454,16 @@ pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo unsigned long len, unsigned long pgoff, unsigned long flags) { - typeof_member(struct proc_ops, proc_get_unmapped_area) get_area; - - get_area = pde->proc_ops->proc_get_unmapped_area; + if (pde->proc_ops->proc_get_unmapped_area) + return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, + len, pgoff, + flags); #ifdef CONFIG_MMU - if (!get_area) - get_area = current->mm->get_unmapped_area; + else + return mm_get_unmapped_area(current->mm, file, orig_addr, + len, pgoff, flags); #endif - if (get_area) - return get_area(file, orig_addr, len, pgoff, flags); + return orig_addr; } diff --git a/fs/ramfs/file-mmu.c b/fs/ramfs/file-mmu.c index c7a1aa3c882b..b45c7edc3225 100644 --- a/fs/ramfs/file-mmu.c +++ b/fs/ramfs/file-mmu.c @@ -35,7 +35,7 @@ static unsigned long ramfs_mmu_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } const struct file_operations ramfs_file_operations = { diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..f01c01b4a4fc 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -745,11 +745,7 @@ struct mm_struct { } ____cacheline_aligned_in_smp; struct maple_tree mm_mt; -#ifdef CONFIG_MMU - unsigned long (*get_unmapped_area) (struct file *filp, - unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags); -#endif + unsigned long mmap_base; /* base of mmap area */ unsigned long mmap_legacy_base; /* base of mmap area in bottom-up allocations */ #ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES diff --git a/include/linux/sched/coredump.h b/include/linux/sched/coredump.h index 02f5090ffea2..e62ff805cfc9 100644 --- a/include/linux/sched/coredump.h +++ b/include/linux/sched/coredump.h @@ -92,9 +92,12 @@ static inline int get_dumpable(struct mm_struct *mm) #define MMF_VM_MERGE_ANY 30 #define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY) +#define MMF_TOPDOWN 31 /* mm searches top down by default */ +#define MMF_TOPDOWN_MASK (1 << MMF_TOPDOWN) + #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ - MMF_VM_MERGE_ANY_MASK) + MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK) static inline unsigned long mmf_init_flags(unsigned long flags) { diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 9a19f1b42f64..cde946e926d8 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -8,6 +8,7 @@ #include #include #include +#include /* * Routines for handling mm_structs @@ -186,6 +187,10 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9626a363f121..b3c4ec115989 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3569,7 +3569,7 @@ static unsigned long io_uring_mmu_get_unmapped_area(struct file *filp, #else addr = 0UL; #endif - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } #else /* !CONFIG_MMU */ diff --git a/mm/debug.c b/mm/debug.c index ee533a5ceb79..32db5de8e1e7 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -162,9 +162,6 @@ EXPORT_SYMBOL(dump_vma); void dump_mm(const struct mm_struct *mm) { pr_emerg("mm %px task_size %lu\n" -#ifdef CONFIG_MMU - "get_unmapped_area %px\n" -#endif "mmap_base %lu mmap_legacy_base %lu\n" "pgd %px mm_users %d mm_count %d pgtables_bytes %lu map_count %d\n" "hiwater_rss %lx hiwater_vm %lx total_vm %lx locked_vm %lx\n" @@ -190,9 +187,6 @@ void dump_mm(const struct mm_struct *mm) "def_flags: %#lx(%pGv)\n", mm, mm->task_size, -#ifdef CONFIG_MMU - mm->get_unmapped_area, -#endif mm->mmap_base, mm->mmap_legacy_base, mm->pgd, atomic_read(&mm->mm_users), atomic_read(&mm->mm_count), diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 86ee29b5c39c..e9ef43a719a5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -641,8 +641,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = current->mm->get_unmapped_area(filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags); /* * The failure might be due to length padding. The caller will retry @@ -672,7 +672,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, if (ret) return ret; - return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index aa82eec17489..b61bc515c729 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1807,7 +1807,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { unsigned long (*get_area)(struct file *, unsigned long, - unsigned long, unsigned long, unsigned long); + unsigned long, unsigned long, unsigned long) + = NULL; unsigned long error = arch_mmap_check(addr, len, flags); if (error) @@ -1817,7 +1818,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (len > TASK_SIZE) return -ENOMEM; - get_area = current->mm->get_unmapped_area; if (file) { if (file->f_op->get_unmapped_area) get_area = file->f_op->get_unmapped_area; @@ -1834,7 +1834,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, get_area = thp_get_unmapped_area; } - addr = get_area(file, addr, len, pgoff, flags); + if (get_area) + addr = get_area(file, addr, len, pgoff, flags); + else + addr = mm_get_unmapped_area(current->mm, file, addr, len, + pgoff, flags); if (IS_ERR_VALUE(addr)) return addr; @@ -1849,6 +1853,17 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); +unsigned long +mm_get_unmapped_area(struct mm_struct *mm, struct file *file, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags)) + return arch_get_unmapped_area_topdown(file, addr, len, pgoff, flags); + return arch_get_unmapped_area(file, addr, len, pgoff, flags); +} +EXPORT_SYMBOL(mm_get_unmapped_area); + /** * find_vma_intersection() - Look up the first VMA which intersects the interval * @mm: The process address space. diff --git a/mm/shmem.c b/mm/shmem.c index 0d1ce70bce38..a8cac92dff88 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2242,8 +2242,6 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long uaddr, unsigned long len, unsigned long pgoff, unsigned long flags) { - unsigned long (*get_area)(struct file *, - unsigned long, unsigned long, unsigned long, unsigned long); unsigned long addr; unsigned long offset; unsigned long inflated_len; @@ -2253,8 +2251,8 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (len > TASK_SIZE) return -ENOMEM; - get_area = current->mm->get_unmapped_area; - addr = get_area(file, uaddr, len, pgoff, flags); + addr = mm_get_unmapped_area(current->mm, file, uaddr, len, pgoff, + flags); if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) return addr; @@ -2311,7 +2309,8 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (inflated_len < len) return addr; - inflated_addr = get_area(NULL, uaddr, inflated_len, 0, flags); + inflated_addr = mm_get_unmapped_area(current->mm, NULL, uaddr, + inflated_len, 0, flags); if (IS_ERR_VALUE(inflated_addr)) return addr; if (inflated_addr & ~PAGE_MASK) @@ -4757,7 +4756,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } #endif diff --git a/mm/util.c b/mm/util.c index 744b4d7e3fae..10c836a75e66 100644 --- a/mm/util.c +++ b/mm/util.c @@ -452,17 +452,17 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) if (mmap_is_legacy(rlim_stack)) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else { mm->mmap_base = mmap_base(random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; + set_bit(MMF_TOPDOWN, &mm->flags); } } #elif defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT) void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) { mm->mmap_base = TASK_UNMAPPED_BASE; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } #endif From patchwork Mon Feb 26 19:09:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206890 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2293891dyb; Mon, 26 Feb 2024 11:39:51 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVCTr/K5VgmGOnJvl2Ow6K+z8jZrGmIpZYYg/kmlC6dyT0UOCj2vgt7we/i1iZM/6Dqot1WaRelSqRFG9USzE/HwiJMdA== X-Google-Smtp-Source: AGHT+IEyK7VYlTqzoUbzBpmG0Yxm6ToB/qWe/zWzHQOSeX7u7mqaGwinHKZI5NA/uMz1gKWGZ4hf X-Received: by 2002:a17:903:2347:b0:1d9:a4a5:7a6c with SMTP id c7-20020a170903234700b001d9a4a57a6cmr11275802plh.57.1708976391122; Mon, 26 Feb 2024 11:39:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708976391; cv=pass; d=google.com; s=arc-20160816; b=vrJHky0XWXF6xFggDjbiqTKRknyzDjcKeGcL/LQ4+/Gh24ORSRZKxIdwAPswJxBEs/ AFncoAnvLRzRLP9wlEKVHLuuwv1bnw9eFPHdD7Im/K82Jnta5Ltfq5w1/LhAsikAm4fO IGVt3T3AFnelzpqM6voLpnAN30HmS8lKNXPg3dN52Roivtuvm3M0rnhDsrE0DN6SwPj7 /UYTi76EjXfZgejJllu7iK/nqhs07o8Miw2mFdekqYGvLuSpE+lWWqxal6zGyZ9hCEKQ tAZ1RuKu+RieG/duVru4nucOp+1YSDAEOTr+XKXh/3Bqr1o8mFMm7J6ahIhNCi7UkIKs uIYw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=GJ3WV1MyMnvFy0laLvbLjyXhu5lENNpTuhfkEsO4ulk=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=OlkpHRpwLtGxBdD4o2dLCD6PNLTF3Xs0elkqW0Nq+Uca9hJ6ILcM76NXYEHTe5u6D4 88TT+EXx9ZqGhg7+xX3xAUMd6MOk71XCAHmsnBMNFi+IUotanQZU8yx45CZ9dA/x5YRk GYPnkc2OrG0g7EWLpavNmaV48cqSuI8fbgffGl5Zqew3H0kIMZjZ38iO2byvSEtmcTJf s9UHr0c6MPczkScgvGfLTIAG1Besxi1rsWSVnm2Nd+YDdjoQkXI4c18Pvt5Wv0MQ1vqH Mz2RLkEzyBHFM146yB3KicvqR312urwiGmYiqfA3mJCdVCFAdDQeL8O2Nci5h7TQnhib u7dQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BMTozr32; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82205-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82205-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id t1-20020a170902e84100b001d95f009fd0si80633plg.367.2024.02.26.11.39.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:39:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82205-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BMTozr32; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82205-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82205-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id A2B63B23E7A for ; Mon, 26 Feb 2024 19:13:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2CDBB13247C; Mon, 26 Feb 2024 19:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BMTozr32" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8A9C12FF64 for ; Mon, 26 Feb 2024 19:09:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974601; cv=none; b=LJfQYh5jO0cNY3awX1TEG1RYpkjAUQ7eZx1EPALaG4t3bp9VADOK4j/Bb9VHk3K2J6+jnzlbdDMRvgsC9UkkAZMDkTxr9lprH6kCVXomHzKAl/4MgNvztxT6lSXXIORucyhiBkQcvs0wbx+EJYUrqKaLfdYRwOIxTc3Wipk9v6s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974601; c=relaxed/simple; bh=NmgdWGUKdUuPH56dpQmxlcj6RB+edyhe9JvhgeXosrc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=bUErzU2Q/Flq9DD56fVflW0ExcQFnol2VHzNMXE4mxwF+QKluSu8073pn3z9hTj6iubOeEH3dfU9MqA4i5GjEPqCRUl7DVQmcjvB6To/eD7MBl6FosVlE7/AP1UHc5iHLVXKTlPgyig+oF9sYnQEjAHIFEfo9C1lq3Y57uBX+5A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BMTozr32; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974600; x=1740510600; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NmgdWGUKdUuPH56dpQmxlcj6RB+edyhe9JvhgeXosrc=; b=BMTozr32Ybu9fPRNdf/0/sV47vpUR3CZFP6n72QeyduUXY6pfhIXWbHN L3zlhki4cDQbB3a4UmCJQAniwEvQB176rKdCelJpaN7CSdtjmkG5lw99t Dya6cJ3uxr+mlgyd8+xU/tB8WpPF6PjWfW//PIEvvL6dCAOk4h3VuLezU n9sCOAKcCOwgBJ67JdTeAwmoy+7rDHx/em5MxBL+JDeMgMuWm0I3/aU6U sE2prZnCnOWJpB0gCrJZtbJZ2L4FImz6632Lc4pwf6UDV0bmJ06akqAsC c+RgoH7wPyL2b+Qgc2E0OC8f2Y82cBNySMYqBy0WuQtnWLY36L5waEMXc g==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721338" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721338" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911445" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:58 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 2/9] mm: Introduce arch_get_unmapped_area_vmflags() Date: Mon, 26 Feb 2024 11:09:44 -0800 Message-Id: <20240226190951.3240433-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791991628137197042 X-GMAIL-MSGID: 1791991628137197042 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. In order to take the start gap into account, the maple tree search needs to know the size of start gap the new mapping will need. The call chain from do_mmap() to the actual maple tree search looks like this: do_mmap(size, vm_flags, map_flags, ..) mm/mmap.c:get_unmapped_area(size, map_flags, ...) arch_get_unmapped_area(size, map_flags, ...) vm_unmapped_area(struct vm_unmapped_area_info) One option would be to add another MAP_ flag to mean a one page start gap (as is for shadow stack), but this consumes a flag unnecessarily. Another option could be to simply increase the size passed in do_mmap() by the start gap size, and adjust after the fact, but this will interfere with the alignment requirements passed in struct vm_unmapped_area_info, and unknown to mmap.c. Instead, introduce variants of arch_get_unmapped_area/_topdown() that take vm_flags. In future changes, these variants can be used in mmap.c:get_unmapped_area() to allow the vm_flags to be passed through to vm_unmapped_area(), while preserving the normal arch_get_unmapped_area/_topdown() for the existing callers. Signed-off-by: Rick Edgecombe --- include/linux/sched/mm.h | 17 +++++++++++++++++ mm/mmap.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index cde946e926d8..7b44441865c5 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -191,6 +191,23 @@ unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +extern unsigned long +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags); +extern unsigned long +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t); + +unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, + struct file *filp, + unsigned long addr, + unsigned long len, + unsigned long pgoff, + unsigned long flags, + vm_flags_t vm_flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, diff --git a/mm/mmap.c b/mm/mmap.c index b61bc515c729..2021bc040e81 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1802,6 +1802,34 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, } #endif +#ifndef HAVE_ARCH_UNMAPPED_AREA_VMFLAGS +extern unsigned long +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) +{ + return arch_get_unmapped_area(filp, addr, len, pgoff, flags); +} + +extern unsigned long +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return arch_get_unmapped_area_topdown(filp, addr, len, pgoff, flags); +} +#endif + +unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *filp, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags)) + return arch_get_unmapped_area_topdown_vmflags(filp, addr, len, pgoff, + flags, vm_flags); + return arch_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, vm_flags); +} + unsigned long get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) From patchwork Mon Feb 26 19:09:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206868 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281001dyb; Mon, 26 Feb 2024 11:13:15 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVF7zCp66oC3d4TVXP9iwGTs3uD0hkPrbXz0sBTIh619G7F/izBxs5QWVWHXPWXm/zgVQ1rfsdl0rqKoA+Ig98YDI3Nsw== X-Google-Smtp-Source: AGHT+IG4CitSSW9i/zVYDNhehcl2y/Py7Ku52xBF31ZyecZzHk6Q/eE2BAwsqbe4OskKRuFn3ScZ X-Received: by 2002:a05:6402:358f:b0:565:a5cf:badb with SMTP id y15-20020a056402358f00b00565a5cfbadbmr5022056edc.33.1708974794832; Mon, 26 Feb 2024 11:13:14 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974794; cv=pass; d=google.com; s=arc-20160816; b=IPC225blKbSFXEMqMSt624vM0sUHJoP1vMDgDwLa7G4SpHhW2m93rjXNdIhoFEzxr8 jlxu9w/a/jG2Z8B61cS2jZeNNCCpQ0IWNQQXplzmVyNsFjJmSCii9f2+UTujXWks2fLV 3tFGBi20Ctiv4bjq1HyTbAPUSmHpiy218fO6UcpSMzJNvw5rHDg4Ey7oJHjqHTB9Wibv IfgWOnl2TGc2qboc4ppYOG72/Hp2UNqzFHNDxt9Bjc1UkrQqxqIzU/z7TPE64516INxw PRH/vMrvCRJvAqxPX7iiQL31fbQd7WGYGk2gK/CI737wxcL355CuDY/wOS/iS6R5IKvb RySw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=5NbTiyI+8TCmJc9FYOl/qFJ5qCvsFXa+XT0LadSrGrk=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=OpyLAEt17GwjjE1fIUlwOrHwg/BuPCAGgMjrNmwg1cKZ7KCbYAAxj70jRCU0ZKNg8G Ln7hnAWB16qD61GnuTzO3U/kKTe55epPBKlWEktDM3A8D+B7lYkb9mj1V4UthPulKSGQ /65g1G6dN1xGhRGKTCO9OtAxWDX52RL0CSqo9TJ20KFGIfk4Uy1HOKfgvuI8iEj7aODA t8vojsbNcSQfuThbb0eECEihi2aDLFWMu3xfPptNP1y/clE9QtLVLW9guX+vBMtR9jnt Dkdqth/fXdFlXXTIckEW8amDnlBS7NXkt5GTSfAH/YMMTXJHdr0W9A+qaxyQeVAzivEG cwUQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bb7Gj0zQ; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82206-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id s26-20020a05640217da00b005643e3f1a8asi25536edy.504.2024.02.26.11.13.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:13:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bb7Gj0zQ; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82206-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 42D8E1F2706E for ; Mon, 26 Feb 2024 19:13:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 767ED132495; Mon, 26 Feb 2024 19:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Bb7Gj0zQ" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DF3612FF6E for ; Mon, 26 Feb 2024 19:10:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974601; cv=none; b=fC0dIb0Lc+LYUA1Ua+iVxIzGjrS5Dx/Ut8l7+M5trI7wO1qPmt+iV+C9DyCTdcbExYHqYuolKYxMubqSeaVXIxBHDhv1VWmvAameyRG6G+fF0Vv5iWZzNTou1OwcQlD1yCWhRA2VWRqxX0W8KZfyIp/mLx/hdnlpFmX9DSZkwfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974601; c=relaxed/simple; bh=UpMkpaIresTqtixl68LlSa8NKJ54JsLLMvQyZqHlQM8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=XD8BKFeelfWWRSX6LY7gu0lk1T3/6WRt6jfNEchnF/O8Hi6QdjSy6UhiKYm9dRPBaqmvvocC9koR+K+0+Eq+GY7Nwy1mbIS48eif6uhuIInCdCt2Cozp8UelxGeMUpQwxrMB6K3kpGrRvUbGDvZflHgo7c95a90TLkyAVYO/RA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Bb7Gj0zQ; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974600; x=1740510600; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UpMkpaIresTqtixl68LlSa8NKJ54JsLLMvQyZqHlQM8=; b=Bb7Gj0zQnHHADrg/RL4WxdAJVTz38K88WKxl3HbWea/3PFo9frinNTqF 6RaVJ8UzgMQOOW1+dv9Ev4YrAGAcnd+HS6ozgYVOf7cvmtf9GGUbRqe5D yR2rffwqB4iHmn6EBdvH1fdUMenu78rrtWo2Ia5JjQDA27YJ4bCoI5YU4 auv2fkQNl0aMEPQk1ObHVUUb8fg2ejRF2Gg0FqDCLnwArtB8X3sWlqgJX 9Izn0Uyko4v03xWcguqQJDSV1C6t/nvFPwkiyCPw4LhzGrgk+XSwh5PP4 x9QZvPwXIrqyy1NyUhW/wd0kSFPErkOjAOgNZiI07bJc0fK8p3AfN/bn3 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721352" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721352" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911448" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:58 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 3/9] mm: Use get_unmapped_area_vmflags() Date: Mon, 26 Feb 2024 11:09:45 -0800 Message-Id: <20240226190951.3240433-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791989954741588265 X-GMAIL-MSGID: 1791989954741588265 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Use mm_get_unmapped_area_vmflags() in the do_mmap() so future changes can cause shadow stack mappings to be placed with a guard gap. Also use the THP variant that takes vm_flags, such that THP shadow stack can get the same treatment. Adjust the vm_flags calculation to happen earlier so that the vm_flags can be passed into __get_unmapped_area(). Signed-off-by: Rick Edgecombe --- v2: - Make get_unmapped_area() a static inline (Kirill) --- include/linux/mm.h | 11 ++++++++++- mm/mmap.c | 34 ++++++++++++++++------------------ 2 files changed, 26 insertions(+), 19 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..a091181ef65a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3351,7 +3351,16 @@ extern int install_special_mapping(struct mm_struct *mm, unsigned long randomize_stack_top(unsigned long stack_top); unsigned long randomize_page(unsigned long start, unsigned long range); -extern unsigned long get_unmapped_area(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); +unsigned long +__get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags); + +static inline unsigned long +get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + return __get_unmapped_area(file, addr, len, pgoff, flags, 0); +} extern unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long len, vm_flags_t vm_flags, unsigned long pgoff, diff --git a/mm/mmap.c b/mm/mmap.c index 2021bc040e81..ac7601d05e89 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1249,18 +1249,6 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (mm->map_count > sysctl_max_map_count) return -ENOMEM; - /* Obtain the address to map to. we verify (or select) it and ensure - * that it represents a valid section of the address space. - */ - addr = get_unmapped_area(file, addr, len, pgoff, flags); - if (IS_ERR_VALUE(addr)) - return addr; - - if (flags & MAP_FIXED_NOREPLACE) { - if (find_vma_intersection(mm, addr, addr + len)) - return -EEXIST; - } - if (prot == PROT_EXEC) { pkey = execute_only_pkey(mm); if (pkey < 0) @@ -1274,6 +1262,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; + /* Obtain the address to map to. we verify (or select) it and ensure + * that it represents a valid section of the address space. + */ + addr = __get_unmapped_area(file, addr, len, pgoff, flags, vm_flags); + if (IS_ERR_VALUE(addr)) + return addr; + + if (flags & MAP_FIXED_NOREPLACE) { + if (find_vma_intersection(mm, addr, addr + len)) + return -EEXIST; + } + if (flags & MAP_LOCKED) if (!can_do_mlock()) return -EPERM; @@ -1831,8 +1831,8 @@ unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *fi } unsigned long -get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, - unsigned long pgoff, unsigned long flags) +__get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) { unsigned long (*get_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long) @@ -1865,8 +1865,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (get_area) addr = get_area(file, addr, len, pgoff, flags); else - addr = mm_get_unmapped_area(current->mm, file, addr, len, - pgoff, flags); + addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, + pgoff, flags, vm_flags); if (IS_ERR_VALUE(addr)) return addr; @@ -1879,8 +1879,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, return error ? error : addr; } -EXPORT_SYMBOL(get_unmapped_area); - unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *file, unsigned long addr, unsigned long len, From patchwork Mon Feb 26 19:09:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206869 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281225dyb; Mon, 26 Feb 2024 11:13:39 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV8+ByoDFwenrY7oUroEJ7A7QspinpsT03kHChnfy80r4oXRN3p5l2W+r381Su70YNOP+U6pAr/nYMz8PcNUbv2fl/Jug== X-Google-Smtp-Source: AGHT+IHMnDdIkzf1tB8tHtOxDvrHxEfs+g14EsZb97XVP6tCOcF7gHSewe2fHjsADqyI9Cox0CA/ X-Received: by 2002:a67:ec53:0:b0:470:494d:64a0 with SMTP id z19-20020a67ec53000000b00470494d64a0mr5751873vso.5.1708974819644; Mon, 26 Feb 2024 11:13:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974819; cv=pass; d=google.com; s=arc-20160816; b=k20J+fywhb3Fx8iZe8W4Llla6dJ1VFQdwoCBLxy7SfgIJ5h55CjFJD5HtZt8mLll07 lCSO4K3zrMC+I2W7Efdi7ddzF67k27skkTOGksCRGQ+TkicxYvsNcCpNliqfJwX6auVw /JnyYrHxv7m7owDanRW90ZAlbpK57gPHm7sSakqcc9xkqFVYG3tP9YFjoAmaGZ7l2OpM rcgsrUFtjb59A9a5VKOzKSug0s21dBc7Veuusw/kT3u3kop830gCGWVf2dFEKp2pYvQK dDKzrAi1Ehl9DLgS5432e2tY1dW8zzzZPpr9Cmhi06nrlrQZ0tnzJGo5737uGxXvDqCO jvDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=w9uxx0p4IqyzsB+rWsNbj/igvIL8fkPgh5yQf8P8jHg=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=QaTMZ741tEuoM7G1lEvvxgYBeajvTKKj1xZMxijdvXqscasU6NEQUix/NkTP7eHZfI roFR4PUn0kqbKFT/fdntwh7sLYCKiDLKLELOSym5SF1HhzjprQG4VgkMLPBj7qTT1yQb T4Uv7iuWvYwvXrHOpIEiNzeAmPJLQHzHtM4k2BWT7iKIxODACbbugbCE1A96OsBOPtrK xpbVPH4z2GD/W0jNQlgExNX5LzFLiiDjjJllHUZKF+95ihqkfooMIIHO8YLrxGRdUc/q LQFsYTNywB5zmTEfzL7FlfVoHKM7/IdB6+SdXuDcUJu7/bvPq7zU9oD4E/5PbehevgAh 174g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SQNLRYbW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id ay18-20020a05622a229200b0042e2ad75644si6342960qtb.686.2024.02.26.11.13.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:13:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SQNLRYbW; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 5FCAE1C2825A for ; Mon, 26 Feb 2024 19:13:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BCDCF132C25; Mon, 26 Feb 2024 19:10:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SQNLRYbW" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1590512FF87 for ; Mon, 26 Feb 2024 19:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974602; cv=none; b=h2earKiOX6rc+gfSmt5Qmby7evS1vytg8/KybE0iNxG74DCj73HjOqZiQiTHSFHXtOYC7MZ/Dsii9aqOH8crKX2tPpfO48eYiISBVxqX1k/8b40+BdUr0Wz3r/rh2UI8WOEy5R147m8aEhmScj5YM2s9AyupkwPTXAda++C0vVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974602; c=relaxed/simple; bh=P5XlQHAG5P20JDtEHex7uiurCNEBN1oum2HZdu24mXg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=PkTj2D33u48de/2hOzEucGLwFMaTtEcs74qWRwYTRqHtDDRTxWGLugbJkbpASUDT3BS5NAWEp+/fUcIDmOCaiGmdM+gkYf9BxcFGI3j1a7CZ0s8B6nPXiTqYhlstY1B6IU4vQnE1lE5S92Fkp0w0AE/q51061cdWn1RpZlvNOg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SQNLRYbW; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974601; x=1740510601; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P5XlQHAG5P20JDtEHex7uiurCNEBN1oum2HZdu24mXg=; b=SQNLRYbWkiFCbFapC0Jlw12doMrenHQeKQ8Y6XoiyNgmlR9yqYaLW5OR tEzJCSidkU5cpvnP+BYQAkGtEknOLYrlHxRfO4et1X9tPgp0FF3frUHvX UIxaZQ/AmQIKep3uPvhsJKY1YNaWY1NcY/DIQenorb1i6tJfZ6bp6dnhQ +llE4A9jNFm9naQOD+rFKwTXvF3BKJI+KOCY/l7kCbsIMQ/pZ2HfD0s7/ 0LZ7VQCtzAMB+qVfDuX3Cj6HG8bXx5/Dp3Umoer0q7s6CDPckBvuFs6AR n6xOVhzX8NDMZadIfiY6WluYDqAlHdzF3igF3MdL5/O0TcG1yuRwLcX+d Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721366" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721366" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911452" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:09:59 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 4/9] thp: Add thp_get_unmapped_area_vmflags() Date: Mon, 26 Feb 2024 11:09:46 -0800 Message-Id: <20240226190951.3240433-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791989980200435257 X-GMAIL-MSGID: 1791989980200435257 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled. Signed-off-by: Rick Edgecombe --- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ef7251dfd9f9 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -139,6 +139,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags); void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); @@ -286,6 +289,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {} #define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return 0; +} + static inline bool can_split_folio(struct folio *folio, int *pextra_pins) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e9ef43a719a5..f235f6d3ff62 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -628,7 +628,8 @@ static inline bool is_transparent_hugepage(struct folio *folio) static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, - loff_t off, unsigned long flags, unsigned long size) + loff_t off, unsigned long flags, unsigned long size, + vm_flags_t vm_flags) { loff_t off_end = off + len; loff_t off_align = round_up(off, size); @@ -641,8 +642,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area_vmflags(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags, vm_flags); /* * The failure might be due to length padding. The caller will retry @@ -662,17 +663,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, return ret; } -unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) { unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE, vm_flags); if (ret) return ret; - return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); + return mm_get_unmapped_area_vmflags(current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return thp_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index ac7601d05e89..e02bb17fef5b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1857,16 +1857,18 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, */ pgoff = 0; get_area = shmem_get_unmapped_area; - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; } - if (get_area) + if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - else + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + /* Ensures that larger anonymous mappings are THP aligned. */ + addr = thp_get_unmapped_area_vmflags(file, addr, len, + pgoff, flags, vm_flags); + } else { addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, pgoff, flags, vm_flags); + } if (IS_ERR_VALUE(addr)) return addr; From patchwork Mon Feb 26 19:09:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206876 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281808dyb; Mon, 26 Feb 2024 11:14:45 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWUJMeDzzMxI2hAPQ9aC9GVVc6xru1K6Lj7J8k+txh90zMhPI1v4HJNWaKZMEP7ZtfYrjutbLFcdGc14NXrIuvxj1t1aw== X-Google-Smtp-Source: AGHT+IF3KXZoR0p2CWeGOE8mRTO/Stlxnv3X9bwB2vYbe091Zf2B3MDU/kJBrQyUgQupucP0n59o X-Received: by 2002:aa7:d349:0:b0:565:be3f:bc4e with SMTP id m9-20020aa7d349000000b00565be3fbc4emr3717578edr.29.1708974885368; Mon, 26 Feb 2024 11:14:45 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974885; cv=pass; d=google.com; s=arc-20160816; b=Hrr6Kxwf+5N5sR+3bIbltlOQjypzIJBP6uSjphwhel7j9EBtOnPeOi9b2cgKOok1iP o/QFOr6mKU/Z+lmGItjgahlVsqfyGZ1fUtbrAbfVyWJpf+oViVgpwsNOAHpRYcitdFX3 q1XXubsL8TWLMuukmrcDNiTUrQEccK6FJdxtIl2MEZmUfOA1/1U+FMDD8MG16WBX1Mpp j/s/bns9nGSXcjCGQjGs81dlFktXnlP/OpMS102vzEWocuK3TWj4FYb9NBMjZXkgq9+b 1XaJw9C8YAH+YMFM+HwONsCDqzKIGdtfzIOBWAPskSo2+Zy5SMn6I1dJ7bx7BbOEtb/0 EXtg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=6mHy37VmvlRmrDV3R0ptBwPC2WKBOKGgSYsH3isxCVo=; fh=bCWG1yUhxvnH/b3pPsFdUcu9hb4R3H1klOZqPFcUcfw=; b=dOOryu3ihyNL7IHtt+TlfRFk9H7EqEprIetT0pNAABaSy0757Kh3JDAU8jFwW/cr9C 0FPmUDC6WdSOWl6qlA6artqyhZT56+NTepCa/W3RqsjvYWo6bMkD3jiVOBZRWgrBiXiy +L3G3R+swUDhDurQ8Vl3I8N8GnYF9tno4AA72m9gn5qnBX4tM7GF9wge6+H7R+jqwdKZ Yi5oFkc8/TyUZpOcbK7D73STfMz2Hd9wR/IVQ+bxfFaTYWZ544uw+Ldd3D7Ggy2ojY0b LiWYajk9nvUKfikB0J/yxxatO4rslmzsK0CkP3tROW/calRa5yFuHYDGRKnQMrwCk+Ci R18Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FmAmElEE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82209-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82209-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id s3-20020a056402036300b00565b45566e0si29936edw.99.2024.02.26.11.14.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82209-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FmAmElEE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82209-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82209-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id CB7811F27B11 for ; Mon, 26 Feb 2024 19:14:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 10A6F13399A; Mon, 26 Feb 2024 19:10:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="FmAmElEE" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1828131E37; Mon, 26 Feb 2024 19:10:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974605; cv=none; b=gtrJTgX2T1j3tGNIWY2WEFi7XeENlhzWoD1zb70HAQx7uqzSui/0+ois/1RrFp8wsgDjN1XVfZO62WkgLeUtdDNGLx1htf42vm8hXlYLmaGRqXhpY9gkPaWsXM+VxODiFYDjL/BQbOduLMI0QT6MxeT7U2FmU+YrSINEaAzOZL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974605; c=relaxed/simple; bh=M4PemZaS0aqUDnC2dki1OuhIuHLGRlk1+X+hvrPVA+A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=N8iE2ocEDwqTtWa1fnPfxdDzy6iO/DL91ruAVjGuQzP4/Jej9fJsZTDLH2NGcFM9KUJaA6I9yhFtYsgBIeesz2moO1/WWR/zxf0+UIPYSUkMkmZYFDysnj/AIlkdHLkdF8RzAaJ6SUSjVuSuYHDaM8WL1HMbdJnQ44KYcqnkd9c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=FmAmElEE; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974603; x=1740510603; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=M4PemZaS0aqUDnC2dki1OuhIuHLGRlk1+X+hvrPVA+A=; b=FmAmElEE2j0GbdsLhEOnJFMYnobg+ZH8Wb4P+Wpnui51MRaU1GNBa/o2 zCR2kG4rZtkdKacBLnb/hcSR7FkrMUdiIq3T0jCL/H9wUWMnaA69/uq/M FiUNg8n7zHJoQvDTWg0abSvuGfjEEexUzlMFtKdtdFAjw0idVT2ErxcOJ d74EHGW9ZI+FTxk1gQu6/epQBaooAucnfubjhCYI+ZU9+vpO5OQ0bazQG MHYaNbG5lW6neKfPIdpSpVXFPbt6BxB6EqWplYq5MASwijPutyrY0ViaJ WtUqCa92Vp9/DWHV+1RCSL/2JNOvCNCjBS8aJtHpuYGU818oqj0K1lbi8 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721380" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721380" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911458" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:00 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org Subject: [PATCH v2 5/9] mm: Initialize struct vm_unmapped_area_info Date: Mon, 26 Feb 2024 11:09:47 -0800 Message-Id: <20240226190951.3240433-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990049334831744 X-GMAIL-MSGID: 1791990049334831744 Future changes will need to add a field to struct vm_unmapped_area_info. This would cause trouble for any archs that don't initialize the struct. Currently every user sets each field, so if new fields are added, the core code parsing the struct will see garbage in the new field. It could be possible to initialize the new field for each arch to 0, but instead simply inialize the field with a C99 struct inializing syntax. Cc: linux-mm@kvack.org Cc: linux-alpha@vger.kernel.org Cc: linux-snps-arc@lists.infradead.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-csky@vger.kernel.org Cc: loongarch@lists.linux.dev Cc: linux-mips@vger.kernel.org Cc: linux-parisc@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-s390@vger.kernel.org Cc: linux-sh@vger.kernel.org Cc: sparclinux@vger.kernel.org Cc: x86@kernel.org Suggested-by: Kirill A. Shutemov Signed-off-by: Rick Edgecombe Link: https://lore.kernel.org/lkml/3ynogxcgokc6i6xojbxzzwqectg472laes24u7jmtktlxcch5e@dfytra3ia3zc/#t --- Hi archs, For some context, this is part of a larger series to improve shadow stack guard gaps. It involves plumbing a new field via struct vm_unmapped_area_info. The first user is x86, but arm and riscv may likely use it as well. The change is compile tested only for non-x86 but seems like a relatively safe one. Thanks, Rick v2: - New patch --- arch/alpha/kernel/osf_sys.c | 2 +- arch/arc/mm/mmap.c | 2 +- arch/arm/mm/mmap.c | 4 ++-- arch/csky/abiv1/mmap.c | 2 +- arch/loongarch/mm/mmap.c | 2 +- arch/mips/mm/mmap.c | 2 +- arch/parisc/kernel/sys_parisc.c | 2 +- arch/powerpc/mm/book3s64/slice.c | 4 ++-- arch/s390/mm/hugetlbpage.c | 4 ++-- arch/s390/mm/mmap.c | 4 ++-- arch/sh/mm/mmap.c | 4 ++-- arch/sparc/kernel/sys_sparc_32.c | 2 +- arch/sparc/kernel/sys_sparc_64.c | 4 ++-- arch/sparc/mm/hugetlbpage.c | 4 ++-- arch/x86/kernel/sys_x86_64.c | 4 ++-- arch/x86/mm/hugetlbpage.c | 4 ++-- fs/hugetlbfs/inode.c | 4 ++-- mm/mmap.c | 4 ++-- 18 files changed, 29 insertions(+), 29 deletions(-) diff --git a/arch/alpha/kernel/osf_sys.c b/arch/alpha/kernel/osf_sys.c index 5db88b627439..dd6801bb9240 100644 --- a/arch/alpha/kernel/osf_sys.c +++ b/arch/alpha/kernel/osf_sys.c @@ -1218,7 +1218,7 @@ static unsigned long arch_get_unmapped_area_1(unsigned long addr, unsigned long len, unsigned long limit) { - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = 0; info.length = len; diff --git a/arch/arc/mm/mmap.c b/arch/arc/mm/mmap.c index 3c1c7ae73292..6549b3375f54 100644 --- a/arch/arc/mm/mmap.c +++ b/arch/arc/mm/mmap.c @@ -27,7 +27,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* * We enforce the MAP_FIXED case. diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index a0f8a0ca0788..525795578c29 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -34,7 +34,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, struct vm_area_struct *vma; int do_align = 0; int aliasing = cache_is_vipt_aliasing(); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* * We only need to do colour alignment if either the I or D @@ -87,7 +87,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, unsigned long addr = addr0; int do_align = 0; int aliasing = cache_is_vipt_aliasing(); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* * We only need to do colour alignment if either the I or D diff --git a/arch/csky/abiv1/mmap.c b/arch/csky/abiv1/mmap.c index 6792aca49999..726659d41fa9 100644 --- a/arch/csky/abiv1/mmap.c +++ b/arch/csky/abiv1/mmap.c @@ -28,7 +28,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, struct mm_struct *mm = current->mm; struct vm_area_struct *vma; int do_align = 0; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* * We only need to do colour alignment if either the I or D diff --git a/arch/loongarch/mm/mmap.c b/arch/loongarch/mm/mmap.c index a9630a81b38a..664bf4abfdcf 100644 --- a/arch/loongarch/mm/mmap.c +++ b/arch/loongarch/mm/mmap.c @@ -24,7 +24,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp, struct vm_area_struct *vma; unsigned long addr = addr0; int do_color_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (unlikely(len > TASK_SIZE)) return -ENOMEM; diff --git a/arch/mips/mm/mmap.c b/arch/mips/mm/mmap.c index 00fe90c6db3e..6321b53dc995 100644 --- a/arch/mips/mm/mmap.c +++ b/arch/mips/mm/mmap.c @@ -34,7 +34,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp, struct vm_area_struct *vma; unsigned long addr = addr0; int do_color_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (unlikely(len > TASK_SIZE)) return -ENOMEM; diff --git a/arch/parisc/kernel/sys_parisc.c b/arch/parisc/kernel/sys_parisc.c index 98af719d5f85..e87c0e325abf 100644 --- a/arch/parisc/kernel/sys_parisc.c +++ b/arch/parisc/kernel/sys_parisc.c @@ -104,7 +104,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp, struct vm_area_struct *vma, *prev; unsigned long filp_pgoff; int do_color_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (unlikely(len > TASK_SIZE)) return -ENOMEM; diff --git a/arch/powerpc/mm/book3s64/slice.c b/arch/powerpc/mm/book3s64/slice.c index c0b58afb9a47..5884f384866f 100644 --- a/arch/powerpc/mm/book3s64/slice.c +++ b/arch/powerpc/mm/book3s64/slice.c @@ -282,7 +282,7 @@ static unsigned long slice_find_area_bottomup(struct mm_struct *mm, { int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT); unsigned long found, next_end; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = 0; info.length = len; @@ -326,7 +326,7 @@ static unsigned long slice_find_area_topdown(struct mm_struct *mm, { int pshift = max_t(int, mmu_psize_defs[psize].shift, PAGE_SHIFT); unsigned long found, prev; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; unsigned long min_addr = max(PAGE_SIZE, mmap_min_addr); info.flags = VM_UNMAPPED_AREA_TOPDOWN; diff --git a/arch/s390/mm/hugetlbpage.c b/arch/s390/mm/hugetlbpage.c index c2d2850ec8d5..7f68485feea0 100644 --- a/arch/s390/mm/hugetlbpage.c +++ b/arch/s390/mm/hugetlbpage.c @@ -258,7 +258,7 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = 0; info.length = len; @@ -274,7 +274,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; unsigned long addr; info.flags = VM_UNMAPPED_AREA_TOPDOWN; diff --git a/arch/s390/mm/mmap.c b/arch/s390/mm/mmap.c index cd52d72b59cf..df88496e2903 100644 --- a/arch/s390/mm/mmap.c +++ b/arch/s390/mm/mmap.c @@ -77,7 +77,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (len > TASK_SIZE - mmap_min_addr) return -ENOMEM; @@ -116,7 +116,7 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp, unsigned long ad { struct vm_area_struct *vma; struct mm_struct *mm = current->mm; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* requested length too big for entire address space */ if (len > TASK_SIZE - mmap_min_addr) diff --git a/arch/sh/mm/mmap.c b/arch/sh/mm/mmap.c index b82199878b45..6aee5f761e08 100644 --- a/arch/sh/mm/mmap.c +++ b/arch/sh/mm/mmap.c @@ -57,7 +57,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, struct mm_struct *mm = current->mm; struct vm_area_struct *vma; int do_colour_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (flags & MAP_FIXED) { /* We do not accept a shared mapping if it would violate @@ -106,7 +106,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, struct mm_struct *mm = current->mm; unsigned long addr = addr0; int do_colour_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (flags & MAP_FIXED) { /* We do not accept a shared mapping if it would violate diff --git a/arch/sparc/kernel/sys_sparc_32.c b/arch/sparc/kernel/sys_sparc_32.c index 082a551897ed..7e781dbfd052 100644 --- a/arch/sparc/kernel/sys_sparc_32.c +++ b/arch/sparc/kernel/sys_sparc_32.c @@ -41,7 +41,7 @@ SYSCALL_DEFINE0(getpagesize) unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (flags & MAP_FIXED) { /* We do not accept a shared mapping if it would violate diff --git a/arch/sparc/kernel/sys_sparc_64.c b/arch/sparc/kernel/sys_sparc_64.c index 1dbf7211666e..fc48ab3f83af 100644 --- a/arch/sparc/kernel/sys_sparc_64.c +++ b/arch/sparc/kernel/sys_sparc_64.c @@ -93,7 +93,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsi struct vm_area_struct * vma; unsigned long task_size = TASK_SIZE; int do_color_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (flags & MAP_FIXED) { /* We do not accept a shared mapping if it would violate @@ -154,7 +154,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, unsigned long task_size = STACK_TOP32; unsigned long addr = addr0; int do_color_align; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* This should only ever run for 32-bit processes. */ BUG_ON(!test_thread_flag(TIF_32BIT)); diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c index 38a1bef47efb..614e2c46d781 100644 --- a/arch/sparc/mm/hugetlbpage.c +++ b/arch/sparc/mm/hugetlbpage.c @@ -31,7 +31,7 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *filp, { struct hstate *h = hstate_file(filp); unsigned long task_size = TASK_SIZE; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; if (test_thread_flag(TIF_32BIT)) task_size = STACK_TOP32; @@ -63,7 +63,7 @@ hugetlb_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, struct hstate *h = hstate_file(filp); struct mm_struct *mm = current->mm; unsigned long addr = addr0; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* This should only ever run for 32-bit processes. */ BUG_ON(!test_thread_flag(TIF_32BIT)); diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index c783aeb37dce..6e5d4fa5fc42 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -125,7 +125,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; unsigned long begin, end; if (flags & MAP_FIXED) @@ -165,7 +165,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, struct vm_area_struct *vma; struct mm_struct *mm = current->mm; unsigned long addr = addr0; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; /* requested length too big for entire address space */ if (len > TASK_SIZE) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 6d77c0039617..88726bd1f72d 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -51,7 +51,7 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = 0; info.length = len; @@ -74,7 +74,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = VM_UNMAPPED_AREA_TOPDOWN; info.length = len; diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index a63d2eee086f..848b2239a215 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -165,7 +165,7 @@ hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = 0; info.length = len; @@ -181,7 +181,7 @@ hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; info.flags = VM_UNMAPPED_AREA_TOPDOWN; info.length = len; diff --git a/mm/mmap.c b/mm/mmap.c index e02bb17fef5b..33af683a643f 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1699,7 +1699,7 @@ generic_get_unmapped_area(struct file *filp, unsigned long addr, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma, *prev; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; const unsigned long mmap_end = arch_get_mmap_end(addr, len, flags); if (len > mmap_end - mmap_min_addr) @@ -1747,7 +1747,7 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, { struct vm_area_struct *vma, *prev; struct mm_struct *mm = current->mm; - struct vm_unmapped_area_info info; + struct vm_unmapped_area_info info = {}; const unsigned long mmap_end = arch_get_mmap_end(addr, len, flags); /* requested length too big for entire address space */ From patchwork Mon Feb 26 19:09:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206871 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281453dyb; Mon, 26 Feb 2024 11:14:03 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWJHNxA/U4F36VLNjXFBnby/jp3YztP7dpC77GlR+yy3CZwHRfrwQ6Zw8vqXTGVge8RaQn/xdQRn14+uIEMeNzXZqRq0g== X-Google-Smtp-Source: AGHT+IF5PveVT0f1rwypVYEpcm9M71Bvob3HV7GU+0qJcO/s9kskWCP+69JRSSkFF2hmpwOfePK9 X-Received: by 2002:a05:6a20:c706:b0:1a0:e1ac:c4a9 with SMTP id hi6-20020a056a20c70600b001a0e1acc4a9mr27656pzb.60.1708974843527; Mon, 26 Feb 2024 11:14:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974843; cv=pass; d=google.com; s=arc-20160816; b=P4zQIYqRibJDfrUhRWJ+ruSwt5IvOcocsRTNVoKQHuyd5Vv2k2sWetwgYQ/YpysUdc 9UP5pJEVOR9pQ1m7CMR/6PN4sdD/+JAKpfetqI/cXBt8qcFbYCTj7Ac5AKCvNSdqevf/ 2+MuN5KmF3Z6TFse2qGKdwIFMcRFSl1r6SxQc7bwWqnU83GHwEXSWSwnABdr+T6OhCbc ix0qifOAcsu9bpQmMbvPQsqprsXFgcAJklbi6fJUy/1vqcRlnkQWPn8NaZcpv5tU2Myr ow10x/X5TnykKvqbJ7ycNNq8/xy2FDbQh0zcG4bQVVwwOTZ1wZx6lM20nm1jQ0dbSxRI rAtg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=xPLva11Xq7XC8tYhVSCbr9K0IrR4H1dbybdarN6i4iY=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=h0p+X8X+tVpmzNiKgrHqM37Be2gYdqfyaSxkcv5l8Iz1aBGzB160A1NzY9XpuAkaWa yUb9dQ6bBrtV9i3QyrNcaJyDjyijxBELzb4X68uMK5dZ81O89D8C/N1E/ZovxGQn6tLo sefmN/nJBk3rcVtW4+0Y7Pq8vHGdiI4N1sodDfx/8YUUX/BYIeOGJ05eJlvaRdp9kick WQ5NeBETNZ8We/GFhygTQVpP/1K3ZGyJorpE7Rl20SJPu7/0NDuLY1WHHjZbJC1MGhUo ryrTrfWs7km/XEndHhUzuk5u/I3jh47WbzmHh4fOxv4gbe+sQcjvpdPDO7zMtlTHVtvv C+Fw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jZr3WyDE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82208-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82208-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id 9-20020a630c49000000b005e44ef60e43si4158790pgm.165.2024.02.26.11.14.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82208-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jZr3WyDE; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82208-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82208-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 59070292703 for ; Mon, 26 Feb 2024 19:14:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 05B1613329A; Mon, 26 Feb 2024 19:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="jZr3WyDE" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C32C131E3C for ; Mon, 26 Feb 2024 19:10:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974604; cv=none; b=CA5SIDSE8QVMHQ/cEpGGm1kJdiV/+gKalx/ryv3Z6ETpq/ssyiMU24xfcPZAbvoYGdxLm82yM6ydff5aBBpesA2aEmtuygXH4dSjDrCzPYYy+PD7X5gj3MB+hWiLW3ABn+nevVW9zZ1lp6L92uYlGVssElu6szwucPsdB7he+Bs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974604; c=relaxed/simple; bh=U3F9hHxdfZ7A6mCry7GoiZtnuJCP/wXQbUGWHcLnyx0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=kF9ODpnBxB+lQljNPn29vva66zZBvKBlYf2hBEK0HZEzbaaP+x2UhF09SfK9vSj3pxRkJZvb32B4sM+OYgBbMj0/X3CzG0buVkXH2tNPmksfsWEppjMrH0QKAqYjRDFCaRqCZM/9TkOeSrlpG6sf5rGqYdKCOnmguCa/z0dkLoQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=jZr3WyDE; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974603; x=1740510603; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=U3F9hHxdfZ7A6mCry7GoiZtnuJCP/wXQbUGWHcLnyx0=; b=jZr3WyDERrAyY/xBfwcsOm7udIKEXcLZQIXVvXpkOb+fXQ2tduZ8yLCP Jy/l5vfDm5tHia8AOgqu/Dwn9paTGvM/51ToDFboZEizHA/XFCTAdOj41 IMZzZLWSH2kvJ540UgkQOmcjcgOcfE5mp2NFbF/NUZVyH5TZi7UuXHRO1 3Ih8A04OF8r/Nb/wUeTRr796EUUL2KYxWEtJQ66i3urYTEuMh3JrXSGLJ wT8IxCTuNyS7Is+tdgRFKpjb16edsm7+wXPiDU1rbvY0gWvm1w2SdG+4t Dd+Tl1aR5jqQyhUcaZpAVPpI+WPactiXygL7fC8fBphG0JA7Ba1VWcesj w==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721395" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721395" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911463" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:02 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 6/9] mm: Take placement mappings gap into account Date: Mon, 26 Feb 2024 11:09:48 -0800 Message-Id: <20240226190951.3240433-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990005604211504 X-GMAIL-MSGID: 1791990005604211504 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. For MAP_GROWSDOWN/VM_GROWSDOWN and MAP_GROWSUP/VM_GROWSUP this has not been a problem in practice because applications place these kinds of mappings very early, when there is not many mappings to find a space between. But for shadow stacks, they may be placed throughout the lifetime of the application. Use the start_gap field to find a space that includes the guard gap for the new mapping. Take care to not interfere with the alignment. Signed-off-by: Rick Edgecombe --- v2: - Remove VM_UNMAPPED_START_GAP_SET and have struct vm_unmapped_area_info initialized with zeros (in another patch). (Kirill) - Drop unrelated space change (Kirill) - Add comment around interactions of alignment and start gap step (Kirill) --- include/linux/mm.h | 1 + mm/mmap.c | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a091181ef65a..19fc1eb86b9a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3406,6 +3406,7 @@ struct vm_unmapped_area_info { unsigned long high_limit; unsigned long align_mask; unsigned long align_offset; + unsigned long start_gap; }; extern unsigned long vm_unmapped_area(struct vm_unmapped_area_info *info); diff --git a/mm/mmap.c b/mm/mmap.c index 33af683a643f..3d7642eb84ea 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1574,7 +1574,7 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); /* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap; if (length < info->length) return -ENOMEM; @@ -1586,7 +1586,13 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) if (mas_empty_area(&mas, low_limit, high_limit - 1, length)) return -ENOMEM; - gap = mas.index; + /* + * Adjust for the gap first so it doesn't interfere with the + * later alignment. The first step is the minimum needed to + * fufill the start gap, the next steps is the minimum to align + * that. It is the minimum needed to fufill both. + */ + gap = mas.index + info->start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp = mas_next(&mas, ULONG_MAX); if (tmp && (tmp->vm_flags & VM_STARTGAP_FLAGS)) { /* Avoid prev check if possible */ @@ -1625,7 +1631,7 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); /* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap; if (length < info->length) return -ENOMEM; From patchwork Mon Feb 26 19:09:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206873 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281631dyb; Mon, 26 Feb 2024 11:14:25 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWgJByVEvlHXGo2+VxiGBXxha9E3GRvlp0SxRkijE680fQV7i2hDyXAWf3lVkdENHcmZfzfJiJC0xxAShx/cUHw4jhanw== X-Google-Smtp-Source: AGHT+IGdKKtniunt9nTYVjqqRyInSOa68n24Nur4OqHTy1b8qxzIKCgrKjdweRRf41ctcTJ3p6db X-Received: by 2002:a05:620a:ec4:b0:787:c4bd:2460 with SMTP id x4-20020a05620a0ec400b00787c4bd2460mr23591qkm.36.1708974865142; Mon, 26 Feb 2024 11:14:25 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974865; cv=pass; d=google.com; s=arc-20160816; b=0Zo/8IUhtXE1XvPHo5F7FrDT2Et+Su4ym6wj8YVziHugI81u9dez65ZsHzIAy+lNwF 8PY0tdw/BbdQeyqp8o4Qgq54P3xilyedrFFjTSnQgc+1PCl+Ib+4nPEroqHT/bg4SU47 wxoEde5HNIqL91XxNbvaMwPy8lw/QPIAK1TEdDz5dAzuBIY9JBEUt9rAljYnRFXx+BzG 4ZpyY6j4wjFqK2/k3egEbSX6DImczZmWAyWho4dNuVdy5FiKODJHQ6YazCkPnkCVCyTw QxmILvCMQCogdmZIqzuNXc17pqG+Opk6KTHSU7t3tOwHTidnEsxF8Ykmc6dLMGsyCFlB n6Mw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=QpIDopAa6aJWE/FyfJUqodq2X8Jt1lCOE+G5EQZEW2I=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=wXEPTNRBqm5GUupLf5pZ1iHRl6/2zBG31ZQcNDkQ93P1k8Mc83ruAWNvrts3jJM7ik cW+y6Bo2jw9rhZIZZFKUJXGa/jHtBEQHrsiAg2Prcd+d4pOfM18NIk88VM8udrtODypE u8e2Q+FKoKpxPNX8qZePBNeDf0VrGyU8HJq0Cxf1zLcJFYEzRDP7gvyLpiAnvdTKz7ST k1rzSJ4mmwIT7tK0POMgi4ZH5W9if8BJE2j390LjH3LysVe74T9Oibrl7f3UX31ZOudq UNLhNKC7X/Pc8wDJTr4y3nepycfENEVHid9FFnpRiL0LQxMAYEx31UReRJASmBdcp+NL PDCg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=k1A88vYc; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82210-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82210-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id xy16-20020a05620a5dd000b00787dd0e4adfsi1156554qkn.682.2024.02.26.11.14.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82210-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=k1A88vYc; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82210-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82210-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E5B9E1C28AA4 for ; Mon, 26 Feb 2024 19:14:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 328EE13341D; Mon, 26 Feb 2024 19:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="k1A88vYc" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50DDD132462 for ; Mon, 26 Feb 2024 19:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974606; cv=none; b=T7NpxK4W4b865yWAiRr8mgbMhPLgf58/2gAn9nKrwj6nVhk7LqZ2ynjAhqZeHnwirPfR5lDxctjiKUhTgZ+Q0qoOBu5SS774rpLV4vc6xicOSTKEqR/F6SNrg3Wkyj3/oWyDxfsBiQnPDKZ4LRX2jQNocgcXPsjITq0Zglos0LI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974606; c=relaxed/simple; bh=K/25qnwR+QY0dPYdq2kDmPj6NmWSjdLBsEBSug4d6Ew=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=MGzDuy8hLI91SxhXW1moCZXku+kcdnIZtZc5XCBYzm+a+FX5jI5/P1Oq1Z25olTYmMuAgB6OidcprExdvt4XCiDu+Q/659IeUfluU/nFnDq83IbyK/T33vGNveZdz5yI5yLhvKCPX78JdWZ5lazTdtKpm8x7pQ536Uva7NMLo7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=k1A88vYc; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974605; x=1740510605; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=K/25qnwR+QY0dPYdq2kDmPj6NmWSjdLBsEBSug4d6Ew=; b=k1A88vYcyjw6WJXT1de92wDZar6loS1AxV8YLoIv16Pp514N5h4ks4gV VupC4kF1yOrreFkc5xSd2JyW9hGvOqCqbIL6raTD5K1B+lkiJkjby3RVM hW6utWLD5isoT0WGW5no3u3kcuJHpDUOQQl6/LDB61M7jIf83MIgq/oGE 2bOFVi219AITYKBhv41YqYBpqO7ZBVQENU2CaRzwoAgc1FcOZdGwdNWfU 2bbCNiCalVwmhcApvu/wqD5It1gwu+LqRN6wTx+miXELWz8f4XuunxiiJ cOVoCbQPlgK2wHQVoZjkeXxIW0zwzY1tnySUSJdAx7sxCjEKI1IJ68SrB A==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721407" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721407" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:04 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911467" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:02 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 7/9] x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS Date: Mon, 26 Feb 2024 11:09:49 -0800 Message-Id: <20240226190951.3240433-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990028150768440 X-GMAIL-MSGID: 1791990028150768440 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add x86 arch implementations of arch_get_unmapped_area_vmflags/_topdown() so future changes can allow the the guard gap of type of vma being placed to be taken into account. This will be used for shadow stack memory. Signed-off-by: Rick Edgecombe --- v2: - Remove unnecessary added extern --- arch/x86/include/asm/pgtable_64.h | 1 + arch/x86/kernel/sys_x86_64.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h index a629b1b9f65a..eb09a11621ad 100644 --- a/arch/x86/include/asm/pgtable_64.h +++ b/arch/x86/include/asm/pgtable_64.h @@ -244,6 +244,7 @@ extern void cleanup_highmap(void); #define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN +#define HAVE_ARCH_UNMAPPED_AREA_VMFLAGS #define PAGE_AGP PAGE_KERNEL_NOCACHE #define HAVE_PAGE_AGP 1 diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 6e5d4fa5fc42..95cb9efc47fb 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -120,8 +120,8 @@ static void find_start_end(unsigned long addr, unsigned long flags, } unsigned long -arch_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; @@ -158,9 +158,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, } unsigned long -arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, - const unsigned long len, const unsigned long pgoff, - const unsigned long flags) +arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) { struct vm_area_struct *vma; struct mm_struct *mm = current->mm; @@ -230,3 +230,18 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, */ return arch_get_unmapped_area(filp, addr0, len, pgoff, flags); } + +unsigned long +arch_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return arch_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); +} + +unsigned long +arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr, + const unsigned long len, const unsigned long pgoff, + const unsigned long flags) +{ + return arch_get_unmapped_area_topdown_vmflags(filp, addr, len, pgoff, flags, 0); +} From patchwork Mon Feb 26 19:09:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206874 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281667dyb; Mon, 26 Feb 2024 11:14:30 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVjNGJoFjeK6JUnyhezlDoWuc41MIcsxBkx3Dr5VgYjzIRZUS2HY5gZu27zEJzLZgFq7PTQBjI1U3M0YBs+fqKFIcZYxQ== X-Google-Smtp-Source: AGHT+IF0G3hHLhbScGQTvWqsBUXQ/uOS1JXv/qWaGSZ34SyVm8tnrE2NA7m4pqpIa/0nT16FThfb X-Received: by 2002:a05:6102:661:b0:470:528b:33b9 with SMTP id z1-20020a056102066100b00470528b33b9mr4066740vsf.13.1708974870423; Mon, 26 Feb 2024 11:14:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974870; cv=pass; d=google.com; s=arc-20160816; b=Bx8RllPDyRN6GsWUfBfqMn2jSRar1aKvQ6aSVV6owMBP6i3k+7M2wAQAVb0Drh9DeP oGDXsUyMaCfx6Sc8YEiTThzHLjuValOw50iD/0fXNL11PBcGYybcBwHnbzrOD8rMbfyW tKt9WoCkyIDfs7QItQqY5sEvdfyh7VaAiv9Msim9XD/mss8Nqe4i2mXu2R2RudkuD58h Bsnu4O5Am6xl6jNdRJa2vJf14tVbHxliRsoSf0SeiylsjDyLF1zA6DEn8JXExpIDFsml m1KG3gIjusG22bxvUWfZ28K34Ke+SglFEV5mmzUjgGwUjjn/OPvkdnEWdEAOMrOKEhiF 438w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=iPNFB5Ox9id13YpJe9qz5rG991kAdQLlrvip6tYfvmU=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=Yh0GxLA8wHtqJBPc41MNWPBR5mL3Q20t33rjpmIIpvhKLk/v1/9i0tW03Dl3oWTjLO 9o7Vd2B++Eq3+Mj+p7on9YsymbliE6t/l4gOUjqImMtS2PKXq34GxJ6Z5w7mz5mibj8y DcBBuIL1XzXy5EQfEb8PRDqxbkYqeYRnJLgZF42iQcpeaoDF9BzvNdclbeoIkMEnKcUY ev2KRGAmfmnDPIv9VWl+BTmmg6a3Vy6o21WUHtBmzKnUewZqi6BamZ6u4N/RmYRavyPv yfKGmFaS2mN6KgXD9rOfQOEg3raxHXbD/6FsVMJ4rirc0fjZsaTNIWKv+XbeAJpeELmv F1fw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=U7dIZYyv; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82211-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82211-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id c14-20020ac853ce000000b0042dee8b1173si5674718qtq.13.2024.02.26.11.14.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82211-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=U7dIZYyv; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82211-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82211-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 39BBF1C28AE2 for ; Mon, 26 Feb 2024 19:14:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9E659133433; Mon, 26 Feb 2024 19:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="U7dIZYyv" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8ADB132471 for ; Mon, 26 Feb 2024 19:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974606; cv=none; b=UQr+I1zK560AZXV9jMPVyqYnCkLTw02J9Y8S+ZNsrJMBQEe3O3ij0g+GTXylnv4iHTVerNKKrICsqRKAMwzRl0JIU015mfznH18tB1WjR/XyGUAXDzyhEdSDYuhefDmxz6+CskIJu9sp2lP4/Y9vzfiMinUOs6iSbjryOw0BIaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974606; c=relaxed/simple; bh=PSY3c+TQ2hmITdUufmc9vn7rLXvDTbUQjAaHGocBj3w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=DNd0cij0p/U+jMygqLr7PbIVdjPvzMU+puju+5FnanPAOebfXHfEB5QKEZTkLbC298WyKkg0S+n6So92ZFAJWBgPKQMnsgCGugXi/ws2sAs5PF+kSiqJcsaZKHNE8VsesiKIBZHFpmlwpZ75O4aaQ6aZWjVaZpuByrzcTbJxatc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=U7dIZYyv; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974605; x=1740510605; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PSY3c+TQ2hmITdUufmc9vn7rLXvDTbUQjAaHGocBj3w=; b=U7dIZYyvQs6leiB/kfmBlgVEpXV2Jm6XErpvbeC7L41jFPfx6PS9QnIR fLIMP6pKIfwD0lduKPDGrYOtzg87LMKciaLhFaybzLaAC9jxHgTFV10O+ oeEkbx2bUaKoaIaVoOToYOu6tm7HH3d2ea9uRMzcIaFXbhdam7t09oJhz CVBomgBoDW20o3xaukFmNJZAJT1VpPoTCTBcnUXosoH3jH9ocgg6fasGf OXAHK97VOoXHx37HVlaWl5zNEBkBD2l7gaSe4pDCaQO2TgGCmIZXEeq1l 05+iWI99ns2M++fLqfnqA657ygey2ftUwKpdHz2NGET+8qyl46ngwXtNE w==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721420" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721420" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911471" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:03 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 8/9] x86/mm: Care about shadow stack guard gap during placement Date: Mon, 26 Feb 2024 11:09:50 -0800 Message-Id: <20240226190951.3240433-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990033359163344 X-GMAIL-MSGID: 1791990033359163344 When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Now that the vm_flags is passed into the arch get_unmapped_area()'s, and vm_unmapped_area() is ready to consider it, have VM_SHADOW_STACK's get guard gap consideration for scenario 2. Signed-off-by: Rick Edgecombe --- arch/x86/kernel/sys_x86_64.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 95cb9efc47fb..f44afb3345db 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -119,6 +119,14 @@ static void find_start_end(unsigned long addr, unsigned long flags, *end = task_size_64bit(addr > DEFAULT_MAP_WINDOW); } +static inline unsigned long stack_guard_placement(vm_flags_t vm_flags) +{ + if (vm_flags & VM_SHADOW_STACK) + return PAGE_SIZE; + + return 0; +} + unsigned long arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags) @@ -150,6 +158,7 @@ arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned l info.high_limit = end; info.align_mask = 0; info.align_offset = pgoff << PAGE_SHIFT; + info.start_gap = stack_guard_placement(vm_flags); if (filp) { info.align_mask = get_align_mask(); info.align_offset += get_align_bits(); @@ -199,6 +208,7 @@ arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, info.low_limit = PAGE_SIZE; info.high_limit = get_mmap_base(0); + info.start_gap = stack_guard_placement(vm_flags); /* * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area From patchwork Mon Feb 26 19:09:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 206875 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2281755dyb; Mon, 26 Feb 2024 11:14:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXvIihsrhFL45QMNb6SYKdT24afclX8Qem5l4RlcPs1I//YXQbyMyoaPxjZ9p/a8wwfN1i4ZViOvrEk5GL3/pRLqI84wQ== X-Google-Smtp-Source: AGHT+IGkzbmYAHo/JLZuG4z5iQj+V6MWJQItWQd2GjBB2hIcHZ3w+N8hXg0Ki5njQLTA1TXlInpA X-Received: by 2002:a05:6808:20a5:b0:3c1:784b:d67e with SMTP id s37-20020a05680820a500b003c1784bd67emr92962oiw.10.1708974879881; Mon, 26 Feb 2024 11:14:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708974879; cv=pass; d=google.com; s=arc-20160816; b=m9T3g91YjRswcig2g0YXgScRlkP6W8NxjUSpP78G40PmZDD1Kto1vo3aR5TlK8PFzU loxumDh/TTOiFe1HwQ6fnZr6I7Qb+19yiBnjicRSHA5Tdkgg2uwPcco5FxJRXq7ZBizW x1dOFBKRdVFKgbra3xL97yL4CCQF3m1sIduFs/0oByR37R2LFUEhQnB7GmYKQxXql5xd RetRjcBtfTrx/Dts2Pw4pWq1bDq/O9mqVQFifpf1TcKxOZNYV6rtX+lVduus9+KXHu/Y oJ4Vc4JRnUy/Siwnej0c+LhVsoqWLPfhDuq6fFdwgha5IdGXGEAh6KM7jA0SGW9J2bRG ogrQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=NFCSdAhrunIzMrgwpPIPHm0AfcuiyWMeHA0mu0m/29Q=; fh=2Dp8by8JJdNpTt06R8k7jaI1v/L8CiIOfGced6Kg60c=; b=d+Ui+PS9Wl/2fgNbzwKf6/Lpe9aH99Czp5Gw4rPHW+WfHoNRIL82etoYHDjWBufpHx Nh4zHjmyBEIQ86XUC5BiMrZkmeXS3tG5IDTzsMCRj/Np+iLqF66VuAJubhSzJMatzp6a JwywcXHDV8IdF36vTCDZ4dbMOb/AkTnJghhKDovUVSlU3KG2VmdHaQ7xqRLDIzDo49Uq 23BwNl4IjJEstSFaU4ydoolN4n9QVAYcA5dIWc7IwJfpXqpMNK8paqiPo1Tm3LiBZwiY wCwbu9j9EwFENEHHxmgxKfYavKagIgk3d9686YER1oq5kLKlxdO3ZEC5itffIVhhvVF6 7RPQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jOT4lAGg; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82212-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id wm8-20020a05620a580800b00787adf4ceccsi5690125qkn.574.2024.02.26.11.14.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:14:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jOT4lAGg; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-82212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82212-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id A66CF1C28A6C for ; Mon, 26 Feb 2024 19:14:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 658F513398D; Mon, 26 Feb 2024 19:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="jOT4lAGg" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E05F132496 for ; Mon, 26 Feb 2024 19:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974607; cv=none; b=Bv9GsQsXNQOJWG6Y7mtcxJ4Q6juubHsE4xaX/TWiLDWIgUWG9jcGdAv/oyYgyfsu457LF74ENZJxbBZZvTpSyHsLC9UNLlP4qR3hmAC3btcM0RS29sdBZJYvfDSvOOi1++4Aq2Bax8KkTrMSrEd5hNOqpKHxGLw4TBX/Mqg81xE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708974607; c=relaxed/simple; bh=9Kz1B1g9YBWVQVBA7u5n8sBfRhBddgPHEWEy/INdrdY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Dikgc8YGooM1yMflyIaCpxnHE22e4TczU+1MHhu0GWoGr6afsQOj730HXxP37nYg3T9o65q2YasiNmU7qORD0fBtatmk2bSD1GrtRXDHoaHVjXFiFM1PKoI8Npv9TKVUXcfnXJpn6iaVJljgRI5GuEI4USpM5YxXti8A3+tvQbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=jOT4lAGg; arc=none smtp.client-ip=198.175.65.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708974606; x=1740510606; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9Kz1B1g9YBWVQVBA7u5n8sBfRhBddgPHEWEy/INdrdY=; b=jOT4lAGgGPJSBIobtFG6DfkeDJDgCH9S1EcXTIpJpCTFNDri0NJf14+l oPC5sVR5E5RLLYfdmLBOiYsmoyATTvnh96egcmJn1rs4CtG+HqLkL9LoI Ew0vQAn5R+LW7AZiyn3e+uS9d4p7Bodw01OY/SLZuZpC8ri/rTx2b/UQw CFhhdD6iMng2Q3j8kVxc3owxbgRLoZAv0834YLBEKOyz6Jbb74JZd5e4X uZRyrmcC/iC5QP/JwpQwR6pArw9vHT0u5acncF0VipYu1mBU4qGSLnMmr YNfkH2UkDydNgcScAwHoZLJRs2P5DKVzMu115buy2CjIZWaBV75v5NJJ7 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="14721432" X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="14721432" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:06 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,186,1705392000"; d="scan'208";a="6911475" Received: from bdmirand-mobl.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.251.3.213]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2024 11:10:04 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [PATCH v2 9/9] selftests/x86: Add placement guard gap test for shstk Date: Mon, 26 Feb 2024 11:09:51 -0800 Message-Id: <20240226190951.3240433-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> References: <20240226190951.3240433-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791990043869824815 X-GMAIL-MSGID: 1791990043869824815 The existing shadow stack test for guard gaps just checks that new mappings are not placed in an existing mapping's guard gap. Add one that checks that new mappings are not placed such that preexisting mappings are in the new mappings guard gap. Signed-off-by: Rick Edgecombe --- .../testing/selftests/x86/test_shadow_stack.c | 67 +++++++++++++++++-- 1 file changed, 63 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/test_shadow_stack.c b/tools/testing/selftests/x86/test_shadow_stack.c index 757e6527f67e..ee909a7927f9 100644 --- a/tools/testing/selftests/x86/test_shadow_stack.c +++ b/tools/testing/selftests/x86/test_shadow_stack.c @@ -556,7 +556,7 @@ struct node { * looked at the shadow stack gaps. * 5. See if it landed in the gap. */ -int test_guard_gap(void) +int test_guard_gap_other_gaps(void) { void *free_area, *shstk, *test_map = (void *)0xFFFFFFFFFFFFFFFF; struct node *head = NULL, *cur; @@ -593,11 +593,64 @@ int test_guard_gap(void) if (shstk - test_map - PAGE_SIZE != PAGE_SIZE) return 1; - printf("[OK]\tGuard gap test\n"); + printf("[OK]\tGuard gap test, other mapping's gaps\n"); return 0; } +/* Tests respecting the guard gap of the mapping getting placed */ +int test_guard_gap_new_mappings_gaps(void) +{ + void *free_area, *shstk_start, *test_map = (void *)0xFFFFFFFFFFFFFFFF; + struct node *head = NULL, *cur; + int ret = 0; + + free_area = mmap(0, PAGE_SIZE * 4, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + munmap(free_area, PAGE_SIZE * 4); + + /* Test letting map_shadow_stack find a free space */ + shstk_start = mmap(free_area, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (shstk_start == MAP_FAILED || shstk_start != free_area) + return 1; + + while (test_map > shstk_start) { + test_map = (void *)syscall(__NR_map_shadow_stack, 0, PAGE_SIZE, 0); + if (test_map == MAP_FAILED) { + printf("[INFO]\tmap_shadow_stack MAP_FAILED\n"); + ret = 1; + break; + } + + cur = malloc(sizeof(*cur)); + cur->mapping = test_map; + + cur->next = head; + head = cur; + + if (test_map == free_area + PAGE_SIZE) { + printf("[INFO]\tNew mapping has other mapping in guard gap!\n"); + ret = 1; + break; + } + } + + while (head) { + cur = head; + head = cur->next; + munmap(cur->mapping, PAGE_SIZE); + free(cur); + } + + munmap(shstk_start, PAGE_SIZE); + + if (!ret) + printf("[OK]\tGuard gap test, placement mapping's gaps\n"); + + return ret; +} + /* * Too complicated to pull it out of the 32 bit header, but also get the * 64 bit one needed above. Just define a copy here. @@ -850,9 +903,15 @@ int main(int argc, char *argv[]) goto out; } - if (test_guard_gap()) { + if (test_guard_gap_other_gaps()) { ret = 1; - printf("[FAIL]\tGuard gap test\n"); + printf("[FAIL]\tGuard gap test, other mappings' gaps\n"); + goto out; + } + + if (test_guard_gap_new_mappings_gaps()) { + ret = 1; + printf("[FAIL]\tGuard gap test, placement mapping's gaps\n"); goto out; }