From patchwork Mon Sep 18 07:33:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 141328 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp2536888vqi; Mon, 18 Sep 2023 02:50:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEy2NE81aD0ZypBC2Nv7igzNEIemb1V653oSpTD+pi3LDxeHNI864FH6lKJffQlo/ox54sn X-Received: by 2002:a05:6358:5295:b0:142:dc9c:11f4 with SMTP id g21-20020a056358529500b00142dc9c11f4mr11850981rwa.8.1695030636651; Mon, 18 Sep 2023 02:50:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695030636; cv=none; d=google.com; s=arc-20160816; b=tcYtL+9JumGX9VgpVRm4Sd+q1X1pKj8XGuJ7SGl5IVWudFfmR/fttXjUZrhSpEnS9x zjukxVzztm/cfEII9/0sMn7ZUM8fdtel+SgRLQg9hvd4O/D6mWtFR5uzCsZVV/qSt6Eh DXjXVb99r8pwmRxVeHfZXYmlXtHXyozmOlKasZc9z6W/HFQf1N31IJYfYNRbHr56uxuQ WKOQ0rc2oLST4IdkPYpatAin4emHchjGKjUk8SoaEQ3phcH5tQG49TJ7+tDcr+hHvJPA u/LshN2MhFF8R1+ylAlE9gb1TCm0xOyzOrQOQRk7m9lZGUAOoW63chZwfUzCsUSgvIfP GPTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=omkQZJY6GX4oLfOXjnuAwvtA5BrOk+Wt2K7yMw3C+oU=; fh=KwayAskJSS4ZGW1RnI6GoBsv+cfF2sa1gQNWUyxtw50=; b=nCYTtf12ArWSanm1//TVOPzk30FDARob1Knx/+B9zYCBO9igr/HUP+zitblc1HTJ7J oEGlrxhqD0HuSgNVhBKP4QNtJdCtvsabY1JR1hpbvETabB/nnopuAgBJ1lFakpgAMXo8 9MgOYlJGv0wYICzXyMVkniZ19ed1snOX6AgfjUg6+J55Ayu8ZW7n94X1TlZzBFk70LHL BUsL2JH8u87VFLO7pkCMLPX2xf0tqmhbA+VOYf+ZmRvijOsc01Q/dz3bbrrzAT02S4Om O7nxSig1yDHxoDR38wGlAvls1LdcPN0GN0DNLKeXi1UEetR2BpYCjE6HT7sb4TLkyvc0 w93g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QZSa3r5y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id o4-20020a6548c4000000b00577ae8b603fsi7661449pgs.314.2023.09.18.02.50.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 02:50:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QZSa3r5y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 496868077447; Mon, 18 Sep 2023 00:36:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238405AbjIRHfl (ORCPT + 27 others); Mon, 18 Sep 2023 03:35:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238432AbjIRHfJ (ORCPT ); Mon, 18 Sep 2023 03:35:09 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C41710C7 for ; Mon, 18 Sep 2023 00:34:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695022446; x=1726558446; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q8dtjLXEWzMrNVlVSld+SWqxhUjc/Zr37nPzybsHJrg=; b=QZSa3r5yWdMaUXZ1cRPNsLloPeAXwYAqgzN2XoCIURJICsqB8Jous8Ae h6zo7l6ikh9jka9cAkdTRL8f4XbjZtfIqNsbIUO11gKy4HBhlISRQgKHK JOQtUHPECF+OKMpf5HVHyIgWy1Baaqg9yBqJei7BVHxCq+aW/WtLPImgn 4drf8BDGK2q9Ku+eNXq1/f8t7BxAxUpwsXD6TlSRmjbZH8Sifg90HwvJW QBT8gtCdFTb2Sy3SGtXj1lblgDBuvADYwLuvWT4bsC8Pw//zwT+oFeCYa 1pALFCeR2L8dDUjpC7tlnd4BT91YXc4Q7oycUVbJc5nSkJfNPnnpVElcX g==; X-IronPort-AV: E=McAfee;i="6600,9927,10836"; a="364630974" X-IronPort-AV: E=Sophos;i="6.02,156,1688454000"; d="scan'208";a="364630974" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Sep 2023 00:34:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10836"; a="815915477" X-IronPort-AV: E=Sophos;i="6.02,156,1688454000"; d="scan'208";a="815915477" Received: from fyin-dev.sh.intel.com ([10.239.159.24]) by fmsmga004.fm.intel.com with ESMTP; 18 Sep 2023 00:34:02 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, hughd@google.com, yosryahmed@google.com, ryan.roberts@arm.com, david@redhat.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [PATCH v3 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Mon, 18 Sep 2023 15:33:18 +0800 Message-Id: <20230918073318.1181104-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230918073318.1181104-1-fengwei.yin@intel.com> References: <20230918073318.1181104-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 18 Sep 2023 00:36:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777368445142220983 X-GMAIL-MSGID: 1777368445142220983 Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range and fully mapped to page table. fully mapped folio is required as if folio is not fully mapped to a VM_LOCKED VMA, if system is in memory pressure, page reclaim is allowed to pick up this folio, split it and reclaim the pages which are not in VM_LOCKED VMA. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. This is required to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio. Signed-off-by: Yin Fengwei --- mm/mlock.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 06bdfab83b58..42b6865f8f82 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,58 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +static inline unsigned int folio_mlock_step(struct folio *folio, + pte_t *pte, unsigned long addr, unsigned long end) +{ + unsigned int count, i, nr = folio_nr_pages(folio); + unsigned long pfn = folio_pfn(folio); + pte_t ptent = ptep_get(pte); + + if (!folio_test_large(folio)) + return 1; + + count = pfn + nr - pte_pfn(ptent); + count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT); + + for (i = 0; i < count; i++, pte++) { + pte_t entry = ptep_get(pte); + + if (!pte_present(entry)) + break; + if (pte_pfn(entry) - pfn >= nr) + break; + } + + return i; +} + +static inline bool allow_mlock_munlock(struct folio *folio, + struct vm_area_struct *vma, unsigned long start, + unsigned long end, unsigned int step) +{ + /* + * For unlock, allow munlock large folio which is partially + * mapped to VMA. As it's possible that large folio is + * mlocked and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + if (!(vma->vm_flags & VM_LOCKED)) + return true; + + /* folio not in range [start, end), skip mlock */ + if (!folio_within_range(folio, vma, start, end)) + return false; + + /* folio is not fully mapped, skip mlock */ + if (step != folio_nr_pages(folio)) + return false; + + return true; +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +366,8 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1; + unsigned long start = addr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -334,6 +388,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } + for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { ptent = ptep_get(pte); if (!pte_present(ptent)) @@ -341,12 +396,19 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; + + step = folio_mlock_step(folio, pte, addr, end); + if (!allow_mlock_munlock(folio, vma, start, end, step)) + goto next_entry; + if (vma->vm_flags & VM_LOCKED) mlock_folio(folio); else munlock_folio(folio); + +next_entry: + pte += step - 1; + addr += (step - 1) << PAGE_SHIFT; } pte_unmap(start_pte); out: