From patchwork Fri Jul 7 16:52:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 117216 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3415189vqx; Fri, 7 Jul 2023 10:11:31 -0700 (PDT) X-Google-Smtp-Source: APBJJlGR0NP4EUTeLD2wECRVZdntIwABRdH9Zm7eX+laKnX2thPa39TnUv4z9NVgLtE1igwXXSCv X-Received: by 2002:a05:6512:2513:b0:4fb:82e5:55f4 with SMTP id be19-20020a056512251300b004fb82e555f4mr5629275lfb.1.1688749891634; Fri, 07 Jul 2023 10:11:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688749891; cv=none; d=google.com; s=arc-20160816; b=FDAXAY/LO20/WbCpCooDjy2xfCKL+jGoQV/Q8SgRErGuzePEdJUMlvBQ3SYcb77RXx O/lt+EnLCcI/nUTSCxuQIf43JNrTooToXFCisOJ2fcvDSghnzZVjqt9ITeuK1IHzT3S9 eHABBnUwRda+OuZOwTjijI3YSteU1lBACJ1m+p5J32s1L5pDTvx00ASv9L06KgTF6sIk lJm6oSqhwbylE81tnu21kqKqP7ccyflfoxwCsVNdqv8TVE4HcDnvwLPAuO+U8lYfcJFJ BzRhqQ/EGkIXjmWJeKvDW33OJPYtYk/zwUPr89/WAZJ0r0c3onA1uA43YEk8zdOF5l3o CTSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5IdTAm/mVq4hBk7ISMJ8Drz/LzCia5XtyQmyhwvy/ik=; fh=LCH763cqklLaGvVeU5MrvwAvEnSDPWcEneXIj6B58gE=; b=q9TDhAGOuBe1aMFJiFC3tA+zKnwlzXobwVumFO5uj3srsOFO76HFaNNC9E4I+SsbIX NVAJw0r2WadRZM5rvMYqg+bBKUA0jOsqkRI9+LBjOBe6LLbT2gF4Rt0OxFEi8V2Nejrb JstSTF7+Pt0Mrmx4tY0IsKQie2Sb3RmKfacAmTnDhgLvmvTF3+liK2R09sUhOpaH4Qvo ySIna309XU9a+YxynWIsFLzMyHOMs7NIw0oB6cwXGpzymmoNKaVlHDXvgrnpsiCq+rYW auFGO63Kay5899tQTm18a2MUJV8YH3BkGn0cECZVO+3qYRJCxcgUfnArufE80I+WdBh/ 5VPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="d/94MWC/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v5-20020aa7d805000000b0051e472983d1si111486edq.524.2023.07.07.10.11.07; Fri, 07 Jul 2023 10:11:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="d/94MWC/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233029AbjGGQxJ (ORCPT + 99 others); Fri, 7 Jul 2023 12:53:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233007AbjGGQxG (ORCPT ); Fri, 7 Jul 2023 12:53:06 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB08F2694 for ; Fri, 7 Jul 2023 09:52:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748756; x=1720284756; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9AZoEU1dBDfQPpclAW3AxhTYvcE6HGN/KVZD4GlGeoo=; b=d/94MWC/wkbSzhSnmtqy5gZ/71whsbZAxyYKwWuqvq42/epCEH7Ifmjb O6vAkzc2/R1uguu4CnTsodWb4+Yd5TsewLztAqMLlXiUY9wrQbTtYm7Ub YsTBfTzMyvPSrsrzT7TDfs4jgwuZURehrbUT+Ql6YzOVyteHSTW0QSDXl 0Mp+Zv9PAnlVYwI9LJcATgMQFdqXHRSG9bm9WCe5E3PJWUWNER6NoFPcM 1QeEHWL7lsz0IgkXIiOmJan0WKA3NfZNcGOGu9dEoTIpgRUeZvbmgpfG1 TmvUiu8YszLz2WU72jqNep4sxYVXCN+og6g9GO1mPuD9t0WIE0fYVtAZc g==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="353776318" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="353776318" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:52:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="697290444" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="697290444" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga006.jf.intel.com with ESMTP; 07 Jul 2023 09:52:33 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 1/3] mm: add function folio_in_range() Date: Sat, 8 Jul 2023 00:52:19 +0800 Message-Id: <20230707165221.4076590-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770782606450215865?= X-GMAIL-MSGID: =?utf-8?q?1770782606450215865?= It will be used to check whether the folio is mapped to specific VMA and whether the mapping address of folio is in the range. Signed-off-by: Yin Fengwei --- mm/internal.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index f1276d90484ad..66117523d7d71 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -585,6 +585,32 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, bool write, int *locked); extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, unsigned long bytes); + +static inline bool +folio_in_range(struct folio *folio, struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + pgoff_t pgoff, addr; + unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + + VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio); + if (start < vma->vm_start) + start = vma->vm_start; + + if (end > vma->vm_end) + end = vma->vm_end; + + pgoff = folio_pgoff(folio); + + /* if folio start address is not in vma range */ + if (pgoff < vma->vm_pgoff || pgoff > vma->vm_pgoff + vma_pglen) + return false; + + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + + return ((addr >= start) && (addr + folio_size(folio) <= end)); +} + /* * mlock_vma_folio() and munlock_vma_folio(): * should be called with vma's mmap_lock held for read or write, From patchwork Fri Jul 7 16:52:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 117214 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3414997vqx; Fri, 7 Jul 2023 10:11:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlEGhQGG8Ou4gGvC15aEx4IeqzvQpn6FoQvkifdWIjNMvR617bBGDHOj8TghNGRcTlB85mdY X-Received: by 2002:a05:6402:406:b0:51d:d1c0:63f7 with SMTP id q6-20020a056402040600b0051dd1c063f7mr5763014edv.4.1688749877030; Fri, 07 Jul 2023 10:11:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688749877; cv=none; d=google.com; s=arc-20160816; b=TCsOgwKP0ZBlsQDNaa05aqqZEk1MeZZOeeAm3vFqdaYeyrC6VTI27vEFYJ79/8X/0L BfYSoXJICY9h1g8a99cDVCUoejCbfqoTmOdapS/NRdXveCe62JJ/R7dB/x8b65FP4xOf Zo5nGzeolKGGlaBFQy5IWmlVdmsqrE+bpZx1NQSitLDXA8Or60EG6k/yTHj1SpHOvuzN JYnqVUFtw+gb6qq8hGRuSj/K/K6dUTYCVilSizghYF4XrlY7t9J/5AJ6YWFprDqN1GI2 zMc4M7gwVNny0SVY9tSpCYijYhcMpWQf/Cw529k/DBiW8iz+w/pSVFN6dujC3CyGGmcC aPgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n1MYy2yizNW+bwRlKUX2YbX2UWhaWjP2YNbEtM8Q7D8=; fh=LCH763cqklLaGvVeU5MrvwAvEnSDPWcEneXIj6B58gE=; b=DQvndO1C7GF0swAkv1onHh6rEeW3WOyO8wpbLWxD2wknL8FNscnibvPSo1cOyDikto lk7j/E5T2gYXcSR4xNG1qOzO0551qxS9BEPFLsJvg07MSHFI5AO85To6pE2JKkNiYXjc 7WCpQLruka3k5+mKYqYfwDH6A/gcyD6E/oPJu4PxbAHqhCeBpXiHSjM+K06UyL74UsP1 9wWeY1xhJTWWdLqEZEBlxifrYuDAPp1tidLW1LUsRmWUpP8NV9HH0mJ+tOkThRqNt/iJ J3X5AmQK8aPojkrfveQMZ2f+5O2QfIh1Qj290dbOMc40fQJhVV/x/721lz4/azDetdGu ksQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CojC+sZY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u5-20020aa7d0c5000000b0051e3cb37258si1103636edo.178.2023.07.07.10.10.53; Fri, 07 Jul 2023 10:11:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CojC+sZY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233026AbjGGQx0 (ORCPT + 99 others); Fri, 7 Jul 2023 12:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232745AbjGGQxZ (ORCPT ); Fri, 7 Jul 2023 12:53:25 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C35441999 for ; Fri, 7 Jul 2023 09:53:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748794; x=1720284794; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hMAoCmMSZm9wJK0rzWzlbRvlWfE6c9KZJN6MJC4MnjE=; b=CojC+sZYhmdBz9pGBf1Io4qGyE8KDj03MdnKaefDYGPb+BdV0ouzfVe2 Lw0uhrREbrfPNzcmjkyUFmocHb2GAWrB011GjhCAk49F+leiaaiPCauvF fz2knFeShT9GhGbbdL8Q0k2E2J6m/wc4ob6Z8PPQ2bVxFnUtiqzi5yzqY FY0y3t0tHl6TPZjSRyr7dMHipy67XDR8zgvSO9cL3TGpaaVVvp0f5Vmv2 KitVVlA0HH0UBm+Qqtpp4eB6qDqySv73XjhRWve6Myaiv0hnbp34VcNV7 jtQdh+QogtKLM56ceFI6SlP1R7n3nAzTQ9Q/BsahSiDyO1s0MnUG/Pj67 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="367422312" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="367422312" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:52:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="790040971" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="790040971" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga004.fm.intel.com with ESMTP; 07 Jul 2023 09:52:46 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 2/3] mm: handle large folio when large folio in VM_LOCKED VMA range Date: Sat, 8 Jul 2023 00:52:20 +0800 Message-Id: <20230707165221.4076590-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770782590834554003?= X-GMAIL-MSGID: =?utf-8?q?1770782590834554003?= If large folio is in the range of VM_LOCKED VMA, it should be mlocked to avoid being picked by page reclaim. Which may split the large folio and then mlock each pages again. Mlock this kind of large folio to prevent them being picked by page reclaim. For the large folio which cross the boundary of VM_LOCKED VMA, we'd better not to mlock it. So if the system is under memory pressure, this kind of large folio will be split and the pages ouf of VM_LOCKED VMA can be reclaimed. Signed-off-by: Yin Fengwei --- mm/internal.h | 11 ++++++++--- mm/rmap.c | 3 ++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 66117523d7d71..c7b8f0b008d81 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -637,7 +637,8 @@ static inline void mlock_vma_folio(struct folio *folio, * still be set while VM_SPECIAL bits are added: so ignore it then. */ if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) && - (compound || !folio_test_large(folio))) + (compound || !folio_test_large(folio) || + folio_in_range(folio, vma, vma->vm_start, vma->vm_end))) mlock_folio(folio); } @@ -645,8 +646,12 @@ void munlock_folio(struct folio *folio); static inline void munlock_vma_folio(struct folio *folio, struct vm_area_struct *vma, bool compound) { - if (unlikely(vma->vm_flags & VM_LOCKED) && - (compound || !folio_test_large(folio))) + /* + * To handle the case that a mlocked large folio is unmapped from VMA + * piece by piece, allow munlock the large folio which is partially + * mapped to VMA. + */ + if (unlikely(vma->vm_flags & VM_LOCKED)) munlock_folio(folio); } diff --git a/mm/rmap.c b/mm/rmap.c index 2668f5ea35342..7d6547d1bd096 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -817,7 +817,8 @@ static bool folio_referenced_one(struct folio *folio, address = pvmw.address; if ((vma->vm_flags & VM_LOCKED) && - (!folio_test_large(folio) || !pvmw.pte)) { + (!folio_test_large(folio) || !pvmw.pte || + folio_in_range(folio, vma, vma->vm_start, vma->vm_end))) { /* Restore the mlock which got missed */ mlock_vma_folio(folio, vma, !pvmw.pte); page_vma_mapped_walk_done(&pvmw); From patchwork Fri Jul 7 16:52:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 117217 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3415241vqx; Fri, 7 Jul 2023 10:11:36 -0700 (PDT) X-Google-Smtp-Source: APBJJlFpsBEpnWoKCsi+F54Dh9W+ILSgm8RgFe1TOlGeakHDzrG7sktH803A9d0FgTVHAA3eobeO X-Received: by 2002:a17:906:1d2:b0:993:da47:363f with SMTP id 18-20020a17090601d200b00993da47363fmr739414ejj.5.1688749895840; Fri, 07 Jul 2023 10:11:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688749895; cv=none; d=google.com; s=arc-20160816; b=rhOvlDLo5IwWmDv8VEBtnysyxKJXFBnkgIkTeyhc/Edb05KG3ym2m6+GX31bN21CcT juq0U1oWM/CtQKJPq4DQMG9gITETlPQkFgJKD4s86HGedSr7s1bgmwDL2NObvaLdWP/9 OTfcvgWZSH0p/Xdn+BAjQ0PQPOChw10CR0TP+F3gKU85QNJWm4Pq1WEo8oXO3PoKx6QJ G+qM0VULmdphamPY21wbFwyHSLWBeLz5q9sqm1d0iJP+tRD+BAAI+fD0TjYcOUc0MKn+ eGoveQysZpxIlDxnvVDyQNkt1hwQzPbr48fJ3V8m5BBh0z3yVwb8lOLAsWwZEiwcodEX 9xjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v6An4k3CwUFyb06/w5IRmp7Q0MPRAi6kDXl9CuGp118=; fh=LCH763cqklLaGvVeU5MrvwAvEnSDPWcEneXIj6B58gE=; b=0F1KItGlyxoOGJ8uGpfBWVyQczvxu1wAmvty9pd3Fy8283ixjTvY9sbFtMeoBq7fNW t9ANgXUzqhXph13yV3igtfAZyYUTXOnoo0t2IOWEV9jdld9kuJzNRMELutG/VNZV7pbd JsTzzFR+IGYfrsir5iM8MPRKEzLO/sa+sL6FybdiGiZ0o1U1/FwXjcJfHHrjL7ybtM0G 9Fbq81EFJFWwm/eZZzBh3l7tyGLK7qA2QcpcY+ztoop9Xhfw9v859I6SbfRhWjWiWkJF aZg1xgsBOCCee5+CMthMLYYrfnxs2nOdBpgLn9GKQ3srx1RMR79X/ZhukgzichmScW45 uWtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nZ29caYE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lh8-20020a170906f8c800b00992bbb918dfsi2221182ejb.173.2023.07.07.10.11.12; Fri, 07 Jul 2023 10:11:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nZ29caYE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233023AbjGGQxQ (ORCPT + 99 others); Fri, 7 Jul 2023 12:53:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232991AbjGGQxO (ORCPT ); Fri, 7 Jul 2023 12:53:14 -0400 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0EB5170C for ; Fri, 7 Jul 2023 09:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748785; x=1720284785; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LCsBu7dWqfVmLSb5jMD4R7baYuKbw8fnATRe6XbxLRw=; b=nZ29caYEJWijKG7lMoGUS+iyKtJZ3Sti+0oCGDOMnfiKD2bMVlYERpFQ avGQDM50/gt4XwvoWFzmtoQDESmO8tl/Drh+aIiPztCWdrCqpFlkpY6xW lfCrlzzr/5E+dWJ0FlkKxVpjMEOPANz7nZLyZsOm9m3COG/IisSyWYjDh ItPCCn3Xe8MHeGdZ5pqSUC03oX8kska4p/bsRGokfOKH5inQ/duNTJdWA gseykZPa/BnbZCrUXk4D8cT3QSt1CPknEXgOp3JwJYvPpi1Ru+Ph0+2qs 1PmiB9wpaLOPJmHCwg/RUOBgsLt9ejDZq+oWfaVY6skE3QHStdITYW365 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="427619015" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="427619015" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:53:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="966700777" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="966700777" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 07 Jul 2023 09:52:59 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Sat, 8 Jul 2023 00:52:21 +0800 Message-Id: <20230707165221.4076590-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770782610862948093?= X-GMAIL-MSGID: =?utf-8?q?1770782610862948093?= Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range - If there is cow folio, mlock the cow folio as cow folio is also in VM_LOCKED VMA range. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. The last rule is used to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio and this large folio become cross VMA boundary. Signed-off-by: Yin Fengwei --- mm/mlock.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 8 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index d7db94519884d..e09a481062ef5 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,64 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +void mlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + mlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + mlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + +void munlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + munlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + munlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +372,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1, nr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -329,24 +388,52 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, goto out; } - start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + pte = start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (!start_pte) { walk->action = ACTION_AGAIN; return 0; } - for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { + + for (; addr != end; pte += step, addr += (step << PAGE_SHIFT)) { + step = 1; ptent = ptep_get(pte); if (!pte_present(ptent)) continue; folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; - if (vma->vm_flags & VM_LOCKED) - mlock_folio(folio); - else - munlock_folio(folio); + + folio_get(folio); + nr = folio_nr_pages(folio) + folio_pfn(folio) - pte_pfn(ptent); + nr = min_t(unsigned int, nr, (end - addr) >> PAGE_SHIFT); + + if (vma->vm_flags & VM_LOCKED) { + /* + * Only mlock the 4K folio or large folio + * in VMA range + */ + if (folio_test_large(folio) && + !folio_in_range(folio, vma, + vma->vm_start, vma->vm_end)) { + folio_put(folio); + continue; + } + mlock_folio_range(folio, vma, pte, addr, nr); + } else { + /* + * Allow munlock large folio which is partially mapped + * to VMA. As it's possible that large folio is mlocked + * and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + munlock_folio_range(folio, vma, pte, addr, nr); + } + + step = nr; + folio_put(folio); } pte_unmap(start_pte); out: