From patchwork Mon Oct 16 20:05:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 153729 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3697925vqb; Mon, 16 Oct 2023 13:07:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFbtqPnMQ6gEMUo/l+B8O2psYPvZbjVzzfG0IfLIBd4TPbDhuw9aRAD63LoGQrA0HUp71Ww X-Received: by 2002:a05:6a20:8f28:b0:172:55c1:f1d4 with SMTP id b40-20020a056a208f2800b0017255c1f1d4mr74843pzk.4.1697486858400; Mon, 16 Oct 2023 13:07:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697486858; cv=none; d=google.com; s=arc-20160816; b=Vf5I47hMUjSjWfPn0FRL+L/I+oPwuiKauJ4AFHBV+XlcMNGsWeGFJkT1UJRnp8grdd hXe2Vp9dOlRO2mdKbHod5BHGhmz5IZiJ/AFD6grEDm8EH0Q/esGKUegeXZHA50QbtK/0 dbE8uqXTEGrX3wjCn15Ay3oo4EK7LFQkEW9WtwPfZdy9yVcz1pZEkIahXTWC38+5xvl3 5IjmMoY95KkCxnIsXOegwQFIyvwe6xyswUobnlpyGRvwAnjIsUKT8dUAXJsHVlJLBDeH ZAMoa0c5TEXcL/+mW2rBMOMf30iUkLTeK/JybtmYaqNT5ZuXU/kBBGJtPzZ71ktxO5Iv PgUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; fh=nXG0fUFBjqUBQ2k9qwRYO0tAMkDdGcBg8Wuhlk7xoZM=; b=tuPkgyCPbmrsoBUb2KZRG3uDfTC0mi7XoCWeh5OiHe+kOX6rquCCxeU38Bv5S2UYAT /b9SH6uOBGwC5VUc3/JuMienKO0tTwNxr5aCplRl/9AzE1Qmw+15QvLlZ+jRxCs84tPO qPAMfp0j4VjJlq7laEFpsxWPVIc8zLiedGtHNbkAJjP2comm9sva65EG/kY2vLoDChRs L7wVNI8kFV69YVxS3WbgaVjF5vzlPgKMsROR+OiePw0CNflYNXXwxa/RFfhIQiXp/lIr XNjBl2GX8ENWG4uM8SFXr1crePDiVdcam9yjCQiSOEizHkcE3rp8egz5GhgF056HdZ+d hPSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ZymhdnWb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id q8-20020a170902dac800b001c434b3659bsi53562plx.354.2023.10.16.13.07.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:07:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=ZymhdnWb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 632378020DB4; Mon, 16 Oct 2023 13:07:34 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234007AbjJPUHL (ORCPT + 18 others); Mon, 16 Oct 2023 16:07:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233871AbjJPUHC (ORCPT ); Mon, 16 Oct 2023 16:07:02 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D85B83 for ; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-694ed84c981so3824033b3a.3 for ; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486816; x=1698091616; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; b=ZymhdnWbKZ87bAuTMygb33npqOkU+9YCKJ2mM6s5lGVETQDZ61iTSfN8/jYIdBBTKM LMx1xyr+ioWpD9ltXJ9jKUaoj/dh848kIsAo3h32EzgAwGVlFJr8RvtzH7Fk5a0ghwH0 KMfLC0vufvHB9bLbIoBniuc6yCNr8lifeFlvXrSfwNwyxSUp3PJkvvkfRy4Ky0JX3iNl ItDAhFqocp/2tekjOuZNa6YWBG/sSiKwigT8RxslQEQwysD40QBsEQ83xeITFANwZWuG fC0hZAuNz7kUwnXMGSmDs/Thoame69WTv9M/Ro9kb79JZb2UxrMwph8/9gy3zRN9wTQ3 sonw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486816; x=1698091616; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dowlsYISY/BEh4ltWtb6fy0PpeiHEk3DajdxWZY/uS8=; b=s0Qy1ePX/F1XFg9oRIFe7K+iQ/5IWaUHjKunJS1UO6dychdwlmC4inmEfBsIFPWtmL An3EuzhW5S2pH90IiDbszgTr5vU2QXbXadUN5M2Z3AEeQl1wjSQG3dMiyN7lX0OW7XBl cF2hr5e7Fodfkp2OlvwGzvPz42O3MttZ5qYpNYVsQ50Jc+jkkJxSUMdgdkPcWLVsd2H5 nhwPa3nZUg3C/mPDXfhNsdjQrBCNMrwR+HrWcrB2AXwjbpNFR8bZlDe5gSeOxFuGmUwP Fihk/kMxP2gUAPWRYo2SEYwhVWvUSqLLLvurRnBsRK9WCnFahs+ivbOvpWuNkCxWg7NU DWZw== X-Gm-Message-State: AOJu0YwtlQdbQlMNYT8N6Rhr807001aRIKBTHeetxy4TW6h39Yeu9WuK gV33kb2of5lyrakdbiPCDILvh3CP1iI= X-Received: by 2002:a05:6300:8003:b0:12c:b10d:693d with SMTP id an3-20020a056300800300b0012cb10d693dmr97230pzc.6.1697486816530; Mon, 16 Oct 2023 13:06:56 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:56 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" , Matthew Wilcox Subject: [PATCH 1/5] mm/khugepaged: Convert __collapse_huge_page_isolate() to use folios Date: Mon, 16 Oct 2023 13:05:06 -0700 Message-Id: <20231016200510.7387-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:07:34 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779943980336925839 X-GMAIL-MSGID: 1779943980336925839 Replaces 11 calls to compound_head() with 1, and removes 1375 bytes of kernel text. Previously, to determine if any pte was shared, the page mapcount corresponding exactly to the pte was checked. This gave us a precise number of shared ptes. Using folio_estimated_sharers() instead uses the mapcount of the head page, giving us an estimate for tail page ptes. This means if a tail page's mapcount is greater than its head page's mapcount, folio_estimated_sharers() would be underestimating the number of shared ptes, and vice versa. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 51 ++++++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88433cc25d8a..7a552fe16c92 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -541,7 +541,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, struct collapse_control *cc, struct list_head *compound_pagelist) { - struct page *page = NULL; + struct folio *folio = NULL; pte_t *_pte; int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; bool writable = false; @@ -570,15 +570,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_PTE_UFFD_WP; goto out; } - page = vm_normal_page(vma, address, pteval); - if (unlikely(!page) || unlikely(is_zone_device_page(page))) { + folio = vm_normal_folio(vma, address, pteval); + if (unlikely(!folio) || unlikely(folio_is_zone_device(folio))) { result = SCAN_PAGE_NULL; goto out; } - VM_BUG_ON_PAGE(!PageAnon(page), page); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); - if (page_mapcount(page) > 1) { + if (folio_estimated_sharers(folio) > 1) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -588,16 +588,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } } - if (PageCompound(page)) { - struct page *p; - page = compound_head(page); + if (folio_test_large(folio)) { + struct folio *f; /* * Check if we have dealt with the compound page * already */ - list_for_each_entry(p, compound_pagelist, lru) { - if (page == p) + list_for_each_entry(f, compound_pagelist, lru) { + if (folio == f) goto next; } } @@ -608,7 +607,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * is needed to serialize against split_huge_page * when invoked from the VM. */ - if (!trylock_page(page)) { + if (!folio_trylock(folio)) { result = SCAN_PAGE_LOCK; goto out; } @@ -624,8 +623,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(page)) { - unlock_page(page); + if (!is_refcount_suitable(&folio->page)) { + folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; } @@ -634,27 +633,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * Isolate the page to avoid collapsing an hugepage * currently in use by the VM. */ - if (!isolate_lru_page(page)) { - unlock_page(page); + if (!folio_isolate_lru(folio)) { + folio_unlock(folio); result = SCAN_DEL_PAGE_LRU; goto out; } - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - compound_nr(page)); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageLRU(page), page); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - if (PageCompound(page)) - list_add_tail(&page->lru, compound_pagelist); + if (folio_test_large(folio)) + list_add_tail(&folio->lru, compound_pagelist); next: /* * If collapse was initiated by khugepaged, check that there is * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; @@ -668,13 +667,13 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } out: release_pte_pages(pte, _pte, compound_pagelist); - trace_mm_collapse_huge_page_isolate(page, none_or_zero, + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, referenced, writable, result); return result; } From patchwork Mon Oct 16 20:05:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 153730 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3697943vqb; Mon, 16 Oct 2023 13:07:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHsGQAeVQodzpobWGGX6JYBJPzMtVZWPwM2tBdw8d/Vnr8aPlZ3Rs17NNSvpBDwIO7nmcm5 X-Received: by 2002:a05:6a00:278c:b0:6b5:523e:9e9 with SMTP id bd12-20020a056a00278c00b006b5523e09e9mr185971pfb.3.1697486860697; Mon, 16 Oct 2023 13:07:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697486860; cv=none; d=google.com; s=arc-20160816; b=w9XekYSSAOj753/av6J2UT4NkRsfaYfENEH2P6z5GaDbxF3+yYWQ3RnSOSt17y5zku vZcC6CaWraVQx1bpkNp9hLsNguMHvr62/dflJrr7ZU4FHdgctkJAE9L90zMrdFCP2YOJ nkZ5gYZaYzbJ+U9odl2D/Nu7GjVIewB4npR7pQRXvfPTHhsNyrSWsthJ0l7lG1cH9G3s 0wff7lePOAh5k34vwJ/Ws21Mn/yM6socE5Sjg5BccWIdi6D1ePNlTUeFBpUe02s4ajIX XB0NqT+4vsNsAyYevchhskHeFZER+ydi9+vRfXfGxn650U+V6sLJiFgMgIz2ZrTNnjSm BTcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; fh=8QxY+O3ixhfYmsFkOjVWPUpalTwBFBIvZrV0f3Brsxg=; b=YzYVMaEca34MsYMLn6RhBn9q3uenaq0RpuDD9GpVxq/FalYxkHt+AO7uDIlo8nXYZm iuKATXki1gken3K0cF1XUrEivyDxPO2ea24ZXq7gRcuRqVBgWH0tFUXxDOSEdRQrcNVa xzxn9IOHVcKb+/CQg3iRVfPBbAgPPTND8cAUOgbkl5ufTq2xSzSxGscQC4/+TD9zkNLt YHQVHjzRYw0cT8liI56BBT1NmHT75IUrxh5o5V5UMO8cVcUnlNRyNdV2vu4sKJgjyuM2 fqIl/+kNbD8cJh+v8O7zkjhzjkM5prLMWXwT5l5H1LODIXCrjaAXwlbbUvqP8ngyUDG3 tLig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Dzj5XIv8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id c8-20020a056a00248800b006bdaa24308fsi538813pfv.105.2023.10.16.13.07.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:07:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Dzj5XIv8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 15D8C8020DAA; Mon, 16 Oct 2023 13:07:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233871AbjJPUHO (ORCPT + 18 others); Mon, 16 Oct 2023 16:07:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233924AbjJPUHC (ORCPT ); Mon, 16 Oct 2023 16:07:02 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0503BFA for ; Mon, 16 Oct 2023 13:06:58 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-27d18475ed4so3662601a91.0 for ; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486817; x=1698091617; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; b=Dzj5XIv8vdJdQHYaNnsOumoGm/nf1e2caHaM9fSBZi8iPbI1L6/WqXdb9va6k7c9ir qlbBEup8DLp63//3WVhhnoeg8GODyn6XLvo2d7aaoDUs671QjAWxRbfnnQ9jAUoNgPJc YvvmfNkLb3q04feTqLtOOmYy7GBu0aHrP9fdKjIUMknbfWoytWAao7emkagad6V8s6dP 0m0raf+GxbiaOkUdikHFSncKuzbyXSVarJh7MLkTSRRjO0bXSTj4lD7H9nwdQXBVCkQ+ IJl56ixYcW2ySqnmVn9cPPmY3Us7+rzZr0o7G17WIVE+9ZlFqilJ/8U8nn/zx+M7Isp+ aqzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486817; x=1698091617; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mMefHNBDnWRPz8cjT6IRO34+spfB9/Lnaff9PAvwfj8=; b=Vt98qR5CkSvUzL1+4o0jcX/ccwnhP8EuKBs6dkn+8i9boCnZmCoMEqu+CJMjlGiFDM luRcSygpnCYW//5SukAtH/34pSSBF82aZB14AMfSN4Tm8EiCRf+gksKAlP+IHcMuX2vg 8iRc5LxjbRd7W+QhlzS5kLiDG8skAba84R7OMqo2XJVRsBsmHafTS2MMRjmBOciZUIDt h7FQVBgfb0mF5SA7WjXuT2pH4VKNOsHZH4dNhF5WJ7EQHLaqznGQZcrnyUR29hjewcZv aY6QJ6nJpRLQWPcuw/q3Z36odGPqNhdhECkHWNR94EZK5+qPkfcVPym3LvxViKcIeqiO bWAg== X-Gm-Message-State: AOJu0Yz590taUqgQuuF2reGU9uJAFylP4AJ8C/roayi7QSzLtaBnUKUa GJFBjx5hvUyBhtwAzbyh53k= X-Received: by 2002:a17:90a:1959:b0:27d:10ab:2325 with SMTP id 25-20020a17090a195900b0027d10ab2325mr161116pjh.27.1697486817436; Mon, 16 Oct 2023 13:06:57 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:57 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 2/5] mm/khugepaged: Convert hpage_collapse_scan_pmd() to use folios Date: Mon, 16 Oct 2023 13:05:07 -0700 Message-Id: <20231016200510.7387-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:07:38 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779943982014443645 X-GMAIL-MSGID: 1779943982014443645 Replaces 5 calls to compound_head(), and removes 1466 bytes of kernel text. Previously, to determine if any pte was shared, the page mapcount corresponding exactly to the pte was checked. This gave us a precise number of shared ptes. Using folio_estimated_sharers() instead uses the mapcount of the head page, giving us an estimate for tail page ptes. This means if a tail page's mapcount is greater than its head page's mapcount, folio_estimated_sharers() would be underestimating the number of shared ptes, and vice versa. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7a552fe16c92..67aac53b31c8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1245,7 +1245,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, pte_t *pte, *_pte; int result = SCAN_FAIL, referenced = 0; int none_or_zero = 0, shared = 0; - struct page *page = NULL; + struct folio *folio = NULL; unsigned long _address; spinlock_t *ptl; int node = NUMA_NO_NODE, unmapped = 0; @@ -1316,13 +1316,13 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, if (pte_write(pteval)) writable = true; - page = vm_normal_page(vma, _address, pteval); - if (unlikely(!page) || unlikely(is_zone_device_page(page))) { + folio = vm_normal_folio(vma, _address, pteval); + if (unlikely(!folio) || unlikely(folio_is_zone_device(folio))) { result = SCAN_PAGE_NULL; goto out_unmap; } - if (page_mapcount(page) > 1) { + if (folio_estimated_sharers(folio) > 1) { ++shared; if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) { @@ -1332,29 +1332,27 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, } } - page = compound_head(page); - /* * Record which node the original page is from and save this * information to cc->node_load[]. * Khugepaged will allocate hugepage from the node has the max * hit record. */ - node = page_to_nid(page); + node = folio_nid(folio); if (hpage_collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; goto out_unmap; } cc->node_load[node]++; - if (!PageLRU(page)) { + if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; goto out_unmap; } - if (PageLocked(page)) { + if (folio_test_locked(folio)) { result = SCAN_PAGE_LOCK; goto out_unmap; } - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { result = SCAN_PAGE_ANON; goto out_unmap; } @@ -1369,7 +1367,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(page)) { + if (!is_refcount_suitable(&folio->page)) { result = SCAN_PAGE_COUNT; goto out_unmap; } @@ -1379,8 +1377,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * enough young pte to justify collapsing the page */ if (cc->is_khugepaged && - (pte_young(pteval) || page_is_young(page) || - PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + (pte_young(pteval) || folio_test_young(folio) || + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, address))) referenced++; } @@ -1402,7 +1400,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, *mmap_locked = false; } out: - trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, + trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, none_or_zero, result, unmapped); return result; } From patchwork Mon Oct 16 20:05:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 153728 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3697872vqb; Mon, 16 Oct 2023 13:07:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF6sPtQIT/vU2TXEwXoXpB9DhB3wPenS6JxcucPwXuIkpm7rjhuzizhHnC8KRkVyHJIVIma X-Received: by 2002:a05:6602:3e97:b0:79f:922b:3809 with SMTP id el23-20020a0566023e9700b0079f922b3809mr213977iob.1.1697486852540; Mon, 16 Oct 2023 13:07:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697486852; cv=none; d=google.com; s=arc-20160816; b=CRoWRin9uYR9++1fQpecX0k34pKMeC3sHVkbJQJdjI+kOowyreU2Gu7lJKyIyAbrkU YM4UPzVsrDOkggFgj3dzMYz8a9fq836tAQQLOkBT6NBu1tCrwCz80rDFhkI5Xnod3WyQ LnxuNklIzO/PSwGomZC9THl1uEc7RM/XExMkTWCUqQQFhEMwdVgIMz0f1PfznzTnwXnC 3BSvbGJMISdsBKUD/vFgA+dWr1I8cqcyXuu9xx9My3Rwm8mOEbe487ahzPozWS+ypUnZ LKFqLUX3weLQIU0KRTLK/KIkbliYycjmJTtQoWpt6ipOYCg7SRtuilBGkaf9T5Tzcu5K 2i1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; fh=8QxY+O3ixhfYmsFkOjVWPUpalTwBFBIvZrV0f3Brsxg=; b=cuNlt+sKY+FkJIZ/yT0qm3oAUiaa50xdtidVDjnxbtocKTXa2NfEDfW3+W5hdfMIZ/ /uhRE/8V2/kTE/m+O8cCpHDTg9UM/nmwf/7ZbhjKjZryO3pAAWXdksSKNRNSM/FuGO1c rPyP5D2D1/JpYnrcvd7pW0U0EHrpEUDTYaEGb20NDfyQrGqB6nQ3VwIaa2Y0nWXNtjhT Q831LM0ExI1LKF3u+efDbMUOeVhf1gP3UHq7NYHO+mDSTpyFbSvlxO2yD4S0TzaK0QES i8PJ5i75uIbGHwzDOWCErbM9bx3bhWoF18tjhM3hnpBDLZ+kTgo1jUamzs/3bM9lDfUB enog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=RAElPC84; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id q29-20020aa7983d000000b006b7884dcd6csi477195pfl.14.2023.10.16.13.07.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:07:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=RAElPC84; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 282E5802174A; Mon, 16 Oct 2023 13:07:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233449AbjJPUHF (ORCPT + 18 others); Mon, 16 Oct 2023 16:07:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231221AbjJPUHD (ORCPT ); Mon, 16 Oct 2023 16:07:03 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11E8D101 for ; Mon, 16 Oct 2023 13:06:59 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-27d1a03f540so3655732a91.1 for ; Mon, 16 Oct 2023 13:06:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486818; x=1698091618; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; b=RAElPC84O1IzaJtthHCnuhBPzH6pbi9XT9ZC8lp8Az361wv/rfbfiLh6ftiE1Xv3lr VLXbibwaRidhp5PDxrHRvCs5t18/ZyS52BA2xhzx0Ki+wjjObjWydsI265Spyv51Whrv xxoPeZ55koA8ygP6n0zXX6DbXUYoUc2DH0kQ/5UIk/DhywmKaT6jL7/e/LwBtta5hHcK thasI3Cg9OBZsuRrfG4BN4bMlISO7kz1UtvND2Mu+2iOrZkWLs6InBM+Ua8fcjPYsm21 yq3g5B7oDs1OZDCTiN8mgN4xNUhjOTwyryhiYCKfGe5q7DV64yWHdzAPf25LHZtB0Vmv jL8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486818; x=1698091618; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vjFYTTptetyjp2cWLw2h5u7A70TW4Bkl9MqEU9zvv1g=; b=kMqjt5uz1im3lnfwEcHk/8sY2HtE9VOU3slx9voqWU8SARG+pOhOYi/pEq6Nf8H1KS UXB7yxZtTAh99spLR/BOdAxBs7tiYMSpvIk5YCHlObyJlOhIBBQZ8mwIadW/Qc9btHqv YiCQDRpfVg0z0g5Iy1hDdjSCSjekMh3QsOd5E3V0VDb6969vqeNg7UVqFSenjKSu85fq rZOqc4/ZP+0pTOrtszlcgES/3iHhge7eoLBx0YjsD6ziQOtqnKEFB05NKbLj+D3weVxr rAQP1gNAuuLytGK9yaXTHYhchL7OzAVh6HaM4mD0aBJ30DV7HooAcTYRFi1OgsX6ug6W S61Q== X-Gm-Message-State: AOJu0Yyd5W5LCTz5YdNJShTGgHa8aJdGLEguXmqgdN61NWQ0bjsnRpQV RygkqOyP66QAKypC37KZsCvmJg9N2H4= X-Received: by 2002:a17:90a:eb09:b0:27c:fc09:401d with SMTP id j9-20020a17090aeb0900b0027cfc09401dmr179801pjz.38.1697486818298; Mon, 16 Oct 2023 13:06:58 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 3/5] mm/khugepaged: Convert is_refcount_suitable() to use folios Date: Mon, 16 Oct 2023 13:05:08 -0700 Message-Id: <20231016200510.7387-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:07:25 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779943973773903096 X-GMAIL-MSGID: 1779943973773903096 Both callers of is_refcount_suitable() have been converted to use folios, so convert it to take in a folio. Both callers only operate on head pages of folios so mapcount/refcount conversions here are trivial. Removes 3 calls to compound head, and removes 315 bytes of kernel text. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 67aac53b31c8..fa21a53ce0c0 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } -static bool is_refcount_suitable(struct page *page) +static bool is_refcount_suitable(struct folio *folio) { int expected_refcount; - expected_refcount = total_mapcount(page); - if (PageSwapCache(page)) - expected_refcount += compound_nr(page); + expected_refcount = folio_mapcount(folio); + if (folio_test_swapcache(folio)) + expected_refcount += folio_nr_pages(folio); - return page_count(page) == expected_refcount; + return folio_ref_count(folio) == expected_refcount; } static int __collapse_huge_page_isolate(struct vm_area_struct *vma, @@ -623,7 +623,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, * but not from this process. The other process cannot write to * the page, only trigger CoW. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { folio_unlock(folio); result = SCAN_PAGE_COUNT; goto out; @@ -1367,7 +1367,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, * has excessive GUP pins (i.e. 512). Anyway the same check * will be done again later the risk seems low. */ - if (!is_refcount_suitable(&folio->page)) { + if (!is_refcount_suitable(folio)) { result = SCAN_PAGE_COUNT; goto out_unmap; } From patchwork Mon Oct 16 20:05:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 153732 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3698083vqb; Mon, 16 Oct 2023 13:07:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFYgLywqIauw1MMTYpPXlmMWN5MIg2+ZH8ye7kDyfKt3m+1GHWQpSFH1JSM8BYjWq8675yh X-Received: by 2002:a17:90b:4f88:b0:274:99ed:a80c with SMTP id qe8-20020a17090b4f8800b0027499eda80cmr187954pjb.3.1697486879564; Mon, 16 Oct 2023 13:07:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697486879; cv=none; d=google.com; s=arc-20160816; b=sKIcT/sh4EKRimS+oRXrYid8FgxQji3/aeQ5wNkGnw50nxKJjNgCTTfu5fBX5OA3iG g5qZjI4zMRNCYdc04CRyKMqL1RmLuY7jRMJkxEYFx9ReRg52MxtLHCzeFLTGKdVYi4ZO 52vc336Yy5aJnglMkIKlXmc/v6stEAp69h05wHVBr4kYgYNs4rdaTNlriplTG+0Glq4C 5tDxdk+FC6FYXG+PLe/xmxKHX2ydbAF/SOGYrFllWWL3z6N1mBpCpHVg3ytSrVow6UD/ hpMY9DcpI66k5U9F+OmDV21n7bkqeZTRMiE8UZOewI/6j1BB1K7Cr9CvLgcdW7RaDOEC 1aPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; fh=8QxY+O3ixhfYmsFkOjVWPUpalTwBFBIvZrV0f3Brsxg=; b=Lz39oqSnpJZFSLGLWQ9Ct3KGuHxYDvO4O5a9oChQninsnMpjH1in/iXNp4ZFM5VpmP m2/2ozheRq98lVilFChNI7oCVGYia9k6q2/LqjRuFm1NHqDaPpgQh8dO3BhmI4nnDser OnWWELq+FIekTIfRxmCnhL3e8/fl0h85GpMvtoKrBYyCIFUbw2xHulNIDti4zKC3nYjq Rl5dskjIWp1WRqVGg7yhdPrCqRU4TrjPKG6HgkssWzuJRSlEsEBdRYz2I8LLJ3SVYPil Ujq+q0y2z7L1DB3uT/RQIafcBsAEb+o/cz5oIAtRjZuQfsOcGc7QsROoMj4y95JR7m3s Q5ZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=DIDrAM59; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id r201-20020a632bd2000000b005ab190f9504si47091pgr.602.2023.10.16.13.07.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:07:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=DIDrAM59; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 351CA808A907; Mon, 16 Oct 2023 13:07:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234101AbjJPUHQ (ORCPT + 18 others); Mon, 16 Oct 2023 16:07:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233191AbjJPUHD (ORCPT ); Mon, 16 Oct 2023 16:07:03 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2172C103 for ; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-517ab9a4a13so3676691a12.1 for ; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486819; x=1698091619; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; b=DIDrAM59k6y4oW+uDak6ZadlF9PGYiZzi1Pk22E1PYPAhi9/VcS6qJa3aW9tRdPHbc DEGgOfxXfJIWEWUtYF3hfigUYpsL/MVoHQm4rIf/VihJ1iuvkk668hKTIL60dwj4nhDl B6fD+Sn2NKaXZpgJ7YPviLXgWQ4vQ3QLbyPnZgvVJsH2zbl6e18Hm1fv/2Q/UjKGLcmf KlN0aovvejO4cZy7smeAV/3brDh1ts2MYsRl4V8mRPI5oP0zYpC0y7eNfwhk/l5/GpZg pgA9Lyx2ZYCaFCQytF6bLj6WCiU1XNvlBbHT45/9ylGZ05Ha7JJBBH7HXlmLWTY8TdML 1dMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486819; x=1698091619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mwAjlmpq6KKWWorM+tiqR60ij8VY2bcU8cxZ+UqJifo=; b=DnWIHCCJKPyYRTVBqTtqWgL6PeDZyGAPnL3MTVpCjY86ijvcGuhgmncVBE8p/FkSH0 LNcikHrROZHQYeDR1OdpmxVQFRodZPT2rU5e4XeXfEiESJIonlUDb20i4k/I6uuAM5k0 M5EmN4IFEY1n9yWCNwt7Gy1PMEastmhrLxME8+YTX8YHCvYAy2c0h2dCg3qenNkNplJ9 1ft4MtcjdjvGbfyrrUfakdWZZude1FHK7jySc2/66lThSGGvxsdUMDWlGroWs3mg14ne /2q/I1TLpTPnGU3wv+fZM6yVQ5DhcSbp645wPsdalzc90RO7ap8o6v7c9+XiTCp7U60S ee5A== X-Gm-Message-State: AOJu0Yy3wmT3zGUFJ/SzEYGgCNnSNGnem1cXLeJgH0CwXcShIbmDjfKm wQN2vQ8pGvTAIGppHwFKdvaoF4QKw2U= X-Received: by 2002:a17:90b:245:b0:27d:4259:b7ef with SMTP id fz5-20020a17090b024500b0027d4259b7efmr190516pjb.23.1697486819203; Mon, 16 Oct 2023 13:06:59 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 4/5] mm/khugepaged: Convert alloc_charge_hpage() to use folios Date: Mon, 16 Oct 2023 13:05:09 -0700 Message-Id: <20231016200510.7387-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:07:39 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779944002145182324 X-GMAIL-MSGID: 1779944002145182324 Also convert hpage_collapse_alloc_page() to hpage_collapse_alloc_folio(). This removes 1 call to compound_head() and helps convert khugepaged to use folios throughout. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index fa21a53ce0c0..70bba8ddea13 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -886,16 +886,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) } #endif -static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node, +static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node, nodemask_t *nmask) { - *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); - if (unlikely(!*hpage)) { + *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask); + + if (unlikely(!*folio)) { count_vm_event(THP_COLLAPSE_ALLOC_FAILED); return false; } - folio_prep_large_rmappable((struct folio *)*hpage); count_vm_event(THP_COLLAPSE_ALLOC); return true; } @@ -1062,15 +1062,16 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, int node = hpage_collapse_find_target_node(cc); struct folio *folio; - if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) + if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) return SCAN_ALLOC_HUGE_PAGE_FAIL; - folio = page_folio(*hpage); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *hpage = NULL; return SCAN_CGROUP_CHARGE_FAIL; } + + *hpage = folio_page(folio, 0); count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC); return SCAN_SUCCEED; From patchwork Mon Oct 16 20:05:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 153731 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3698070vqb; Mon, 16 Oct 2023 13:07:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHdKf6as09qzE+h6XouH5uBsebjVgjlqAXiHN+Y2jR/M3gTy9xjZMDD+52OOIO/L5Rt5PGW X-Received: by 2002:a05:6358:72a6:b0:166:d9c9:dbe with SMTP id w38-20020a05635872a600b00166d9c90dbemr229108rwf.3.1697486878756; Mon, 16 Oct 2023 13:07:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697486878; cv=none; d=google.com; s=arc-20160816; b=erwKjRHwf8BjXwduMTT7xkvGsu2qrqyIDrC3BZh62yqYMQ+ya5vJRZbxaMS3ngJWji N/nLNUaEQIlfBf0Zj/ELVVX6ecf2inBiy80PhlJ4WN/jhGFaUC+hJl2081+O7wW7MaYX 6ni4a7cOX+GD/cnfsNWeijAhTFk+GdMMhVIm4weMUAFBMRR6v+ASdexvqa99AEnDBie1 tBSYpZZy4iR+Nc/ExWbSjS2I4EXztWrtI57k1CQqXgIECrZ46eaW7PocOoKuPiMipY+B 89JmV9QyxIiUXxTYLNJ+Osb2eycHAggj58ATN3DFT647YA1f4hTeEuDOwr0iqlC4naSj Iy4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; fh=8QxY+O3ixhfYmsFkOjVWPUpalTwBFBIvZrV0f3Brsxg=; b=PwhvyEh6WjYyeo9S/AscLUGAHFR+mwi/waLVNsukb2h5L3waEZir544b1sm2HHI+3V LvRjnLgMIO37uaKRjDU1yg1/JAZ5ZNGcTRMZlzVPLuTQeZ4cfoXApe20264Au1qBLu8V i5AWsLANC1lUeTBRJuvBMTb1tYF+cgtelg32+iHo2T8o5185Kckz2vI48HW2likodKXX sBSo+8bek7VLhdE2CouQ7rQDkyhb3cqAl2TJoghHlR+VjpKRalFh8EUDcyGbG1VlRJb5 kc90PZtGU9LLZrGy4pPjfy3HxkX7FQVsYxhCKebaYAfx+6q1xZQCXi+zaSQWii17hXUD l62w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=aERMXq8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id k186-20020a6384c3000000b005ab05858e70si49458pgd.782.2023.10.16.13.07.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:07:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=aERMXq8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 8FD5F802176C; Mon, 16 Oct 2023 13:07:56 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233769AbjJPUHU (ORCPT + 18 others); Mon, 16 Oct 2023 16:07:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233991AbjJPUHD (ORCPT ); Mon, 16 Oct 2023 16:07:03 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8F45EE for ; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-5a1d91a29a0so3670495a12.1 for ; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486820; x=1698091620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=aERMXq8DV73yjLippLpiuRtwzFcFw/WYuvqnlRbHHHfjn25nRadm3YMgWO+1fvmLSx 148Kk/6Ks/w2NmbKpcJNf3srriXyzfQLBAzLns7MnPq1Gcgb8uM1G91NNv5+cx8nuu3g Dy5pvdCTKsq9lMFuKCkrLohcsygvLqvKtvQ7FZdokG5AwRHw0YCGKQUXzp+eyrFZZf7Q a0ToPBFaoEtnRFfBEKfsLoz/r80av0d3BWaxzzdIiC/ra2PdNovzBWGb1JxAwpwQCFlC 5fM03Fn3X+GAqPeV23BOZNKMWjipUIYNY4ryxydn7dC8+nebZDZhvUmwXLOwujZ2IWMz JPLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486820; x=1698091620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=bgXX9plPnK4RB+j7RJaYl8LMevRm18WLMteDEDuZ0zY7/YGufd4SPEPDebvFmZRobC rA5ysTg8bSAjAFb3LaW/2dX6avNQzu1JXSPWtBUjqo3hJSjf9aY6y786uQKacznbXAVW wEMYLhIk7tlgLq0bp4PsWp7phYo1/zDRwWPpfswszQvn1+XXpDyb9qGhWLEaPfgEdWgB ql8sDTOPJiOCkc/sqBGJP47jW3I1hLh9JC7WLkYUJKqIkur4iPkwhWFtTrx3FOaPpy9p fzT2CzlQxouijmVsmgnjXATd6uRdlBvjGFIliZjL9XWes89qvu9Sft5h7N1ewG7j2Zyp fDKg== X-Gm-Message-State: AOJu0YwW9NpTMRRZ4dQTgFrJLChYOQSoJAPPPWoa+WuXETbVZmZWGEWe pnT/NPXYOWEUtuMtSL9/5qd+zJe/HPE= X-Received: by 2002:a17:90b:368d:b0:27d:f85:9505 with SMTP id mj13-20020a17090b368d00b0027d0f859505mr159453pjb.24.1697486820185; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:59 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Mon, 16 Oct 2023 13:05:10 -0700 Message-Id: <20231016200510.7387-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:07:56 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779944001021120304 X-GMAIL-MSGID: 1779944001021120304 This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we can get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 70bba8ddea13..99c437979848 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1471,7 +1471,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1504,19 +1504,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (!folio) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1530,13 +1525,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1561,7 +1556,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1576,7 +1571,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1600,7 +1595,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1609,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1628,8 +1623,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1653,14 +1648,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1668,9 +1663,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }