From patchwork Tue Sep 26 00:52:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144653 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1589791vqu; Mon, 25 Sep 2023 17:54:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGcHuQo9FC3RdYccC+W98Mu4VMm2PipQDEwUOEuax9TDEfRbJ5nXD8wi10MIfVGY8VMj22S X-Received: by 2002:a05:6a20:320f:b0:15d:e68d:a855 with SMTP id hl15-20020a056a20320f00b0015de68da855mr5992761pzc.29.1695689646390; Mon, 25 Sep 2023 17:54:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689646; cv=none; d=google.com; s=arc-20160816; b=MlTMzri7944TexmYJcX3S+/+oQuQY2eBPFstE1CoP8OOPnDvX04NGuAmalnI1VspVc AHmkfbn9q/s6qZx508/JNUSfATT0/3oGExbbsrnQSE73nxz8J2qJnpx9MJKlnwlzGyGO ze3h9hw+tEgaO5fRg3oZaXSLUtRFUjfRZiHc1enNOfNv1AGlzBQlBeOTO5SGEz0hZmRw Y+KAy8QIqL2CzU7sOGD8Tq5n7yKjQVcz9N0yAOMQ9VnXiG38yo6wB+adMM9U4VQKUfoj JUwqU0c7FabDiheUNmgbv6HF5JL44sNfcQcLQYg3LagytI1sCqBVHeZWN9xnoLdz5Vvi qVsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=LSYNIqiX5/dP1JwV7wGTtypK5JC/Iahb15M57nxl0fM=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=dyJ2CpNoUGBrMK3rygNBveRrPMobBPJZYstjYaRWVk1gYE46Ww6m3DrajrpDxaUCqh YXNvpTjeTOBoJUQ30S9UEYUB2BLru8jad2sdxgCBGooRaiy1kP3WKKi8bZS59w5rvLx4 +rPr/nGSVAOY69NY8YhhL9TncqZrLzSWgpNAoEJ6XYm7DGwIqJrNczqUwWg7WCa4VGkH ZLtN7RO+3Lp9/jy1nsCIsV0MLbrilY2MMXkjnOg2dhRdEXEg2g3Qb3X1ZQzdG6wF2bAE qxTUKEvEqh1HJOHrWCu8UUf41A0tN+ZuUCSV5iSVlQ4sfe0wQrBTkBV1Rg4nw2UIsuZJ Bc9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id c1-20020a170902d48100b001b7eb771d5esi12272220plg.527.2023.09.25.17.54.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:54:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id BA05D819D70B; Mon, 25 Sep 2023 17:53:41 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232712AbjIZAx0 (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232166AbjIZAxS (ORCPT ); Mon, 25 Sep 2023 20:53:18 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C00C9109 for ; Mon, 25 Sep 2023 17:53:10 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Rvh3Z2N7HzVlCM; Tue, 26 Sep 2023 08:50:02 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:08 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 1/9] mm_types: add _last_cpupid into folio Date: Tue, 26 Sep 2023 08:52:46 +0800 Message-ID: <20230926005254.2861577-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:41 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059466677478185 X-GMAIL-MSGID: 1778059466677478185 At present, only arc/sparc/m68k define WANT_PAGE_VIRTUAL, both of them don't support numa balancing, and the page struct is aligned to _struct_page_alignment, it is safe to move _last_cpupid before 'virtual' in page, meanwhile, add it into folio, which make us to use folio->_last_cpupid directly. Signed-off-by: Kefeng Wang --- include/linux/mm_types.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5a995089cbf5..2fdfddd8264a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -183,6 +183,9 @@ struct page { #ifdef CONFIG_MEMCG unsigned long memcg_data; #endif +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS + int _last_cpupid; +#endif /* * On machines where all RAM is mapped into kernel address space, @@ -210,10 +213,6 @@ struct page { struct page *kmsan_shadow; struct page *kmsan_origin; #endif - -#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS - int _last_cpupid; -#endif } _struct_page_alignment; /* @@ -328,6 +327,9 @@ struct folio { atomic_t _refcount; #ifdef CONFIG_MEMCG unsigned long memcg_data; +#endif +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS + int _last_cpupid; #endif /* private: the union with struct page is transitional */ }; @@ -384,6 +386,9 @@ FOLIO_MATCH(_refcount, _refcount); #ifdef CONFIG_MEMCG FOLIO_MATCH(memcg_data, memcg_data); #endif +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS +FOLIO_MATCH(_last_cpupid, _last_cpupid); +#endif #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ From patchwork Tue Sep 26 00:52:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144657 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590480vqu; Mon, 25 Sep 2023 17:56:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFnkEwT+QJU5hV6h9pD2ci06hCzUOx7r/+vtQgBZRSzl5zxMGNatGIsfM44K7t4V9FguYwh X-Received: by 2002:a17:902:d502:b0:1c3:1c74:5d0a with SMTP id b2-20020a170902d50200b001c31c745d0amr7666756plg.34.1695689770247; Mon, 25 Sep 2023 17:56:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689770; cv=none; d=google.com; s=arc-20160816; b=Rmyn6i/vj9aCxXIPhrws67JXhWFzhLsTdfqUUzBqXrDnIjyX01401WrKpb721nDh7i OeU/wia6mpjqpXffwxdFoWB0eKkOBfa6/7UNR8hfOgASodDvAt1j6eP1wWsySeE4RxSu CpWHWAR6QejWOMaoJhCfd/oOa0xO9tO9qnmgnP2zvgLbGLZlRoS6Be9BGfuvblRlBqxW Ju/QwHR+C5+0tS8g2csUQEAB+5FsubFNj1PrteAqDDPA5N1pL8uh78E8ytNpesE6zokM ynrVNk+nbMsNOnfl2099hPTzTs50Aet2+dJcPAMMcAQKpCFlCFuamYq+SZAF6pibninC 66tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=fVfwzDbo5Kc4i4rlTbkYOzLkB4fLvg8axYuIJPcNuUQ=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=jhqobwLNikgjL5KSWZ58YmoxctpwXmH6O6TDJy+GyzIuBr/hM0tJAc8QV6LVGKx7jM jNQtUh2o8uMnUjsOdKr2hNbl9uXd/iT8qXHeh+kAZ+qBfoWwJtemYc9aHLF0zQ6t+Slw 8EVhQyh2Qmt0rGg2VmR7lOOydqmPvKz4eZrwL5iNuAoOHF+9xUip4aOAOx/++4WiCtYJ KwSMRbvJg4VTmbngLCU0rIF5Nuae2/U7LjTwbmq6fLAch6d40WRAEoFnznpF4V6LV/cu qx4y/9hL9IV2F2ut3Zm6P2HUXVd5pYYXw+CQZ2Qgw/POfKWZwF2LeMgEY4MjOicMdMqZ SDhg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id e9-20020a170902b78900b001c3f58d15aesi10871811pls.223.2023.09.25.17.56.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:56:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 63AD4825171E; Mon, 25 Sep 2023 17:53:27 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232609AbjIZAxX (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231228AbjIZAxS (ORCPT ); Mon, 25 Sep 2023 20:53:18 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EB8410A for ; Mon, 25 Sep 2023 17:53:11 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.57]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Rvh286hNcztT42; Tue, 26 Sep 2023 08:48:48 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:08 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 2/9] mm: mprotect: use a folio in change_pte_range() Date: Tue, 26 Sep 2023 08:52:47 +0800 Message-ID: <20230926005254.2861577-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:27 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059596479813907 X-GMAIL-MSGID: 1778059596479813907 Use a folio in change_pte_range() to save three compound_head() calls. Signed-off-by: Kefeng Wang --- mm/mprotect.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index b94fbb45d5c7..459daa987131 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -114,7 +114,7 @@ static long change_pte_range(struct mmu_gather *tlb, * pages. See similar comment in change_huge_pmd. */ if (prot_numa) { - struct page *page; + struct folio *folio; int nid; bool toptier; @@ -122,13 +122,14 @@ static long change_pte_range(struct mmu_gather *tlb, if (pte_protnone(oldpte)) continue; - page = vm_normal_page(vma, addr, oldpte); - if (!page || is_zone_device_page(page) || PageKsm(page)) + folio = vm_normal_folio(vma, addr, oldpte); + if (!folio || folio_is_zone_device(folio) || + folio_test_ksm(folio)) continue; /* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && - page_count(page) != 1) + folio_ref_count(folio) != 1) continue; /* @@ -136,14 +137,15 @@ static long change_pte_range(struct mmu_gather *tlb, * it cannot move them all from MIGRATE_ASYNC * context. */ - if (page_is_file_lru(page) && PageDirty(page)) + if (folio_is_file_lru(folio) && + folio_test_dirty(folio)) continue; /* * Don't mess with PTEs if page is already on the node * a single-threaded process is running on. */ - nid = page_to_nid(page); + nid = folio_nid(folio); if (target_node == nid) continue; toptier = node_is_toptier(nid); @@ -157,7 +159,7 @@ static long change_pte_range(struct mmu_gather *tlb, continue; if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !toptier) - xchg_page_access_time(page, + xchg_page_access_time(&folio->page, jiffies_to_msecs(jiffies)); } From patchwork Tue Sep 26 00:52:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144661 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1592180vqu; Mon, 25 Sep 2023 18:00:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDG6w6tDxuAy7ZPIIjzk9/DT1sc43HA9MgN2tjXxaWkt+vIdCIvEl0N0oZLf2sMt4Ec7Hn X-Received: by 2002:a05:6a20:1602:b0:15a:4c23:870b with SMTP id l2-20020a056a20160200b0015a4c23870bmr1619302pzj.23.1695690046626; Mon, 25 Sep 2023 18:00:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695690046; cv=none; d=google.com; s=arc-20160816; b=kevt/faCU2s9U6FjXayHgXMP7clp5k9+DE3P964Yv48bop/mFjV45mif2JrPdxZUW8 vhnLP5BEHPfqaL6n+yypl4xAInYkBs0AyAAB1XzHohcElWpQsLo0dqedpve6LXa15QLn zNEOB+owgLrwWO0ZTH05eDG86kayCODDrgTwQpS0dPCWrOhPUyvDwS8DFW8DYXd9Xx1r X0DHHslETqeDZtelhyq8XOV+HNscmLKCdMaAX5eXzWV+NdHFfUX9NZ6muyLcFOPO6UX9 RGv1m3rj7SdeK3HK5hKzt/P7jtWtFXgmOVimctTxgKoK2zA3V1RYLLtbBM+gKY5VdGAI onRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=xLDhMjCrVSLoYbaa4gzJ4uRL0hqY2r45hKznONkN8B8=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=UTVuFKXocmeF+NctqeEYN1GPuH4GTBmIOMgdW8nUG05JDK2UcAqTZgd7WBdi18jWnF yFOEuhVF637+Tz08VBNUdranIZIyTJ876KVmpEvajTVIzeEdqepovvWVivKaqSaVJKOw 3G+MlWNCpAtbQZvg1IdYrPLrWgRO8bIkdo6nW0ltpDZOovWC1GWbY7SbPmh/89pX9nHK nK1IcJMkxBGpouui8n1+LhswqSYU9FL7xcvUDLRyDYaU0LpA7mE0CiYoT8AZ24vNe9sa ayyFZRZOwr4FPhNf9omUSI/biZFJ7i1nQFuaI/THPdzJAtW8nD0e5MLrzzLqaC3ddA9y fuXA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id r200-20020a632bd1000000b00578afd89baasi11124817pgr.744.2023.09.25.18.00.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 18:00:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 85B4C81FDB9B; Mon, 25 Sep 2023 17:53:36 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232166AbjIZAx3 (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232332AbjIZAxS (ORCPT ); Mon, 25 Sep 2023 20:53:18 -0400 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDDF010E for ; Mon, 25 Sep 2023 17:53:11 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.56]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4Rvh4c4Mp8z15Mq5; Tue, 26 Sep 2023 08:50:56 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:09 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 3/9] mm: huge_memory: use a folio in change_huge_pmd() Date: Tue, 26 Sep 2023 08:52:48 +0800 Message-ID: <20230926005254.2861577-4-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:36 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059886172032331 X-GMAIL-MSGID: 1778059886172032331 Use a folio in change_huge_pmd(), this is in preparation for xchg_page_access_time() to folio conversion. Signed-off-by: Kefeng Wang --- mm/huge_memory.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0f93a73115f7..c7efa214add8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1849,7 +1849,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION if (is_swap_pmd(*pmd)) { swp_entry_t entry = pmd_to_swp_entry(*pmd); - struct page *page = pfn_swap_entry_to_page(entry); + struct folio *folio = page_folio(pfn_swap_entry_to_page(entry)); pmd_t newpmd; VM_BUG_ON(!is_pmd_migration_entry(*pmd)); @@ -1858,7 +1858,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * A protection check is difficult so * just be safe and disable write */ - if (PageAnon(page)) + if (folio_test_anon(folio)) entry = make_readable_exclusive_migration_entry(swp_offset(entry)); else entry = make_readable_migration_entry(swp_offset(entry)); @@ -1880,7 +1880,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, #endif if (prot_numa) { - struct page *page; + struct folio *folio; bool toptier; /* * Avoid trapping faults against the zero page. The read-only @@ -1893,8 +1893,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_protnone(*pmd)) goto unlock; - page = pmd_page(*pmd); - toptier = node_is_toptier(page_to_nid(page)); + folio = page_folio(pmd_page(*pmd)); + toptier = node_is_toptier(folio_nid(folio)); /* * Skip scanning top tier node if normal numa * balancing is disabled @@ -1905,7 +1905,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !toptier) - xchg_page_access_time(page, jiffies_to_msecs(jiffies)); + xchg_page_access_time(&folio->page, + jiffies_to_msecs(jiffies)); } /* * In case prot_numa, we are under mmap_read_lock(mm). It's critical From patchwork Tue Sep 26 00:52:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144658 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590534vqu; Mon, 25 Sep 2023 17:56:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFkkm7E6EVsJeRCCcnWPSPnIcIrLVKVBYpjW87RuD/Qutq2xfFUTrNCyrrDA5M2aKcW75O3 X-Received: by 2002:a05:6a20:7d96:b0:15d:6fd3:8e8c with SMTP id v22-20020a056a207d9600b0015d6fd38e8cmr6726842pzj.14.1695689781278; Mon, 25 Sep 2023 17:56:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689781; cv=none; d=google.com; s=arc-20160816; b=uOBQNFCtgv4AxOiSBouY5/Qjcof3yMjM7gX7w1rSBVJu/XzyixrPnDcQkKKkns4ckA ooWt6gvadpEBvYLDp0lIJdEpMnsVLJg9VC4WMKwkNzqJWUjONWkI6scboFPFBkOS3Gnf xFH+IaygD8oaVss9Qz5yLIn7eCD/Cu35kZwtfThn8W++DWoWEuaRyePrvQ0cI2dPAUty +ARcw21k5ioShLAaUSndJbPUQk/8UVmv2rLSaq1ad7zPWJ9+ChtyzFMH3W79mCFKW554 koCXWrtLg4lJvag6f5x9D2JGKQq7nSztzHs5sisn3JnZ0QT2v8xIz155jZu4pn1aFbsG 9c+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=1NMXgpblLezaPosE82Jei/cXHdbr/hhXwa/WKnoyZJg=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=RPZkPq2RIAV1gmtHRzb2aTzg7VRL4YizvJTbLXrm3yU2VwuSOCCIbWa8yTKFiDnbOq ZTvYoQppnfZh83MjRKR2s2DMB5JP7Pi9vFIE5j/j3HAlJBCOy6Ni1tAOIbUjE4YpCHco bqvayIpaxmfvcqpCkzRhoTqPNaflNRgOhsH3W5EoY8hbe7R2AbdJBkVExD9MbhivL8ze QVrZWfv3WWd9PupuMkp5BbYX2nhLozngckzt0lG82GSIZcGiOAXwRM1I0ZPLaurun71c BWWHa3TydS1sV6LkSI+Q6rvRJpaAJkZOsinVJo37FoLWW+AfchcU/UBcZgLH9Dn8PYU6 tYCQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id u7-20020a170902e5c700b001c6189eaadesi4149658plf.132.2023.09.25.17.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:56:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id CE86A8251703; Mon, 25 Sep 2023 17:53:48 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233001AbjIZAxc (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229495AbjIZAxS (ORCPT ); Mon, 25 Sep 2023 20:53:18 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15B7E116 for ; Mon, 25 Sep 2023 17:53:12 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.57]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Rvh2n28SpzNnv8; Tue, 26 Sep 2023 08:49:21 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:09 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 4/9] mm: convert xchg_page_access_time to xchg_folio_access_time() Date: Tue, 26 Sep 2023 08:52:49 +0800 Message-ID: <20230926005254.2861577-5-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:48 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059608478227570 X-GMAIL-MSGID: 1778059608478227570 Make xchg_page_access_time to take a folio, and rename it to xchg_folio_access_time() since all callers with a folio. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 7 ++++--- kernel/sched/fair.c | 2 +- mm/huge_memory.c | 4 ++-- mm/mprotect.c | 2 +- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a1d0c82ac9a7..49b9fa383e7d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1711,11 +1711,12 @@ static inline void page_cpupid_reset_last(struct page *page) } #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ -static inline int xchg_page_access_time(struct page *page, int time) +static inline int xchg_folio_access_time(struct folio *folio, int time) { int last_time; - last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS); + last_time = page_cpupid_xchg_last(&folio->page, + time >> PAGE_ACCESS_TIME_BUCKETS); return last_time << PAGE_ACCESS_TIME_BUCKETS; } @@ -1734,7 +1735,7 @@ static inline int page_cpupid_xchg_last(struct page *page, int cpupid) return page_to_nid(page); /* XXX */ } -static inline int xchg_page_access_time(struct page *page, int time) +static inline int xchg_folio_access_time(struct folio *folio, int time) { return 0; } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b507ec29e1e1..afb9dc98a8ee 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1714,7 +1714,7 @@ static int numa_hint_fault_latency(struct folio *folio) int last_time, time; time = jiffies_to_msecs(jiffies); - last_time = xchg_page_access_time(&folio->page, time); + last_time = xchg_folio_access_time(folio, time); return (time - last_time) & PAGE_ACCESS_TIME_MASK; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c7efa214add8..c4f4951615fd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1905,8 +1905,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !toptier) - xchg_page_access_time(&folio->page, - jiffies_to_msecs(jiffies)); + xchg_folio_access_time(folio, + jiffies_to_msecs(jiffies)); } /* * In case prot_numa, we are under mmap_read_lock(mm). It's critical diff --git a/mm/mprotect.c b/mm/mprotect.c index 459daa987131..1c556651888a 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -159,7 +159,7 @@ static long change_pte_range(struct mmu_gather *tlb, continue; if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !toptier) - xchg_page_access_time(&folio->page, + xchg_folio_access_time(folio, jiffies_to_msecs(jiffies)); } From patchwork Tue Sep 26 00:52:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144662 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1592243vqu; Mon, 25 Sep 2023 18:00:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFCOQtrh4FdZ27LC+e/UKg6sraPOtfId2IbxDo9lPLZQx/fqIkyYn/izLzPxmHsfNI44z0 X-Received: by 2002:a05:6a20:144b:b0:15d:9d8e:ad8b with SMTP id a11-20020a056a20144b00b0015d9d8ead8bmr6854177pzi.32.1695690053283; Mon, 25 Sep 2023 18:00:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695690053; cv=none; d=google.com; s=arc-20160816; b=xCSRK5CTybHc2rB6DL8IvgSbCulzRvWvyvUQcB6TGK/TX/HT59+B0YZBO8zJLBI40O YVUTLSQOTsYZFgSkjI89tajY7Q8jiFruv5mgQae0xgAy/CE4KDBnE+awjib4aNrEZUAC YvM5w+pTAiBMi6YqAnbDp5Y+tJeGA3EWQWR7a9vucW7FLjLUfc2ehaCgGdahLfpNMBvD rkdTCrzdrSD3gWfezcLKL4jmXPlm8lsL98rchQvFgJQi4KEFZEMhUPx13nR066kdhzJo aoP6wkPYeB0HaG9UQA9xGrIeVRKpCF2FVnvZSshTqZLIfrXNlONufvS51aEW97VZl2WX zqtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=a0yIiMoZ+BRAnMLaOS2qTLnjbKRujtjPqddXRR5KgqA=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=xRFICOtP7oUF8klf2r4+cbC7yMzMChxx6EtpzP3HEarJtRMRKleMdzXgKz+pRMMMdQ jOro7Pgjc0ejZovPJwWQaLX5krM49FW0kO0l2VUEaLanzVU5AhbNLdB8vkO6UglB9Op2 YRmBFDPKhlweQfQRJCgCyhb+f773ziDUBKRSPKLzONbBmsU2L+HEoNT6gd4jrrONYA1Y Q82MgNR88AnQTHlg7oHlM+YcvkAk+PY4Wsj5h0Wi3F61kFzZJZvwkBLPiANtTNpHbMat D37kgAMxYouealxG2+rRhGqplEHcLbKQaWVbd5LWyYLpj2024oCGlWRF1vubQiSxCt36 m9Mg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id j6-20020a170902da8600b001c1eb8b79afsi12288913plx.360.2023.09.25.18.00.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 18:00:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id D557381FDB90; Mon, 25 Sep 2023 17:54:01 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233363AbjIZAxg (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232511AbjIZAxU (ORCPT ); Mon, 25 Sep 2023 20:53:20 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91A4111D for ; Mon, 25 Sep 2023 17:53:12 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.53]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Rvh4c62kfzrSw6; Tue, 26 Sep 2023 08:50:56 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:10 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 5/9] mm: convert page_cpupid_last() to folio_cpupid_last() Date: Tue, 26 Sep 2023 08:52:50 +0800 Message-ID: <20230926005254.2861577-6-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:54:01 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059893457772669 X-GMAIL-MSGID: 1778059893457772669 Make page_cpupid_last() to take a folio, and rename it to folio_cpupid_last() since all callers with a folio. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 12 ++++++------ mm/huge_memory.c | 4 ++-- mm/memory.c | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 49b9fa383e7d..aa7fdda1b56c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1689,18 +1689,18 @@ static inline int page_cpupid_xchg_last(struct page *page, int cpupid) return xchg(&page->_last_cpupid, cpupid & LAST_CPUPID_MASK); } -static inline int page_cpupid_last(struct page *page) +static inline int folio_cpupid_last(struct folio *folio) { - return page->_last_cpupid; + return folio->_last_cpupid; } static inline void page_cpupid_reset_last(struct page *page) { page->_last_cpupid = -1 & LAST_CPUPID_MASK; } #else -static inline int page_cpupid_last(struct page *page) +static inline int folio_cpupid_last(struct folio *folio) { - return (page->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; + return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; } extern int page_cpupid_xchg_last(struct page *page, int cpupid); @@ -1740,9 +1740,9 @@ static inline int xchg_folio_access_time(struct folio *folio, int time) return 0; } -static inline int page_cpupid_last(struct page *page) +static inline int folio_cpupid_last(struct folio *folio) { - return page_to_nid(page); /* XXX */ + return folio_nid(folio); /* XXX */ } static inline int cpupid_to_nid(int cpupid) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c4f4951615fd..93981a759daf 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1555,7 +1555,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * to record page access time. So use default value. */ if (node_is_toptier(nid)) - last_cpupid = page_cpupid_last(&folio->page); + last_cpupid = folio_cpupid_last(folio); target_nid = numa_migrate_prep(folio, vma, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) { folio_put(folio); @@ -2508,7 +2508,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, if (page_is_idle(head)) set_page_idle(page_tail); - page_cpupid_xchg_last(page_tail, page_cpupid_last(head)); + page_cpupid_xchg_last(page_tail, folio_cpupid_last(folio)); /* * always add to the tail because some iterators expect new diff --git a/mm/memory.c b/mm/memory.c index 29c5618c91e5..5ab6e8d45a7d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4814,7 +4814,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) !node_is_toptier(nid)) last_cpupid = (-1 & LAST_CPUPID_MASK); else - last_cpupid = page_cpupid_last(&folio->page); + last_cpupid = folio_cpupid_last(folio); target_nid = numa_migrate_prep(folio, vma, vmf->address, nid, &flags); if (target_nid == NUMA_NO_NODE) { folio_put(folio); From patchwork Tue Sep 26 00:52:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144659 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590538vqu; Mon, 25 Sep 2023 17:56:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHy+L4/Lb5KCdykZbxOi4/Et2lSa1kPKwXHJUMlItGixHTEe+ysB8+4WWCGaMQeEoDtvrVp X-Received: by 2002:a17:903:24d:b0:1c3:4b24:d89d with SMTP id j13-20020a170903024d00b001c34b24d89dmr11776076plh.40.1695689781912; Mon, 25 Sep 2023 17:56:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689781; cv=none; d=google.com; s=arc-20160816; b=AvCxliUeVYrm2fg+TTxJCjpp68NnuGoD31f/OpIwczFYoGo4P2/0J0I2VMpJQv4C3e YzbM4SH5U5bVTka5XnE8nHctmds+Si/fTk5HDK9yosT4uwcmH+Iqb6Bxp4NolCWtRKJd 32y31qFjdJ21LbIHgyAVNM6l7oeWKpkabCsnCqANcwJ2gFfZPb+BzJGzwAGXCDF9iw2I z4ejGVD+4dz8jUJm56OEcYAFF6XzmSmtsLzYtjfXjTZG/32L308OD1VOHHbuGq9qDM6z 4HEuORbwOVJfEPAcxowrc1TNKrpuMI14ikNqsdNuAqOEN6cEvyN3uX+1gKfT8nJq2KIc ozNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=DXDkkBvU79MWZKI+0442dVxorWxgJcWylyLHViP7wEs=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=i4L1UtQA09L2RGqSDcCC++CLYlKe5nCjtZOe50xkHo+MkmiXSfPb8X7ODaryB3UZVN yEhl3R8tW6XBjq6zVJaRbaNm34GeqbnhoaPUFmDEfjLWDc/xAaXqO5xv3mFN7jQB67UG qamEV1Kd+6wRin/QQC1OHY1krGlGatGuTXdNa1dhk1+bJZvabTPE/j6Q4bxJWaLSnx8p MdFRZWlUAOsBsF9FIyiV/XWfJIzptFzpuWrmdkB/GGYf9oE/Wg1pI/vfjCH7307I0sFo KhPQbjrq/mohGGfKcgJDcc6VhRWyudwa8UcnntG9+xZNy8HIASbuYozyX8PE+EE6DN+S yswQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id f9-20020a17090274c900b001bf5753e0ccsi11462067plt.119.2023.09.25.17.56.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:56:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 413C28251732; Mon, 25 Sep 2023 17:53:51 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233320AbjIZAxe (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232372AbjIZAxT (ORCPT ); Mon, 25 Sep 2023 20:53:19 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1470CD6 for ; Mon, 25 Sep 2023 17:53:13 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.53]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Rvh2B6SqZztT0Z; Tue, 26 Sep 2023 08:48:50 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:10 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 6/9] mm: make wp_page_reuse() and finish_mkwrite_fault() to take a folio Date: Tue, 26 Sep 2023 08:52:51 +0800 Message-ID: <20230926005254.2861577-7-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:51 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059608753634459 X-GMAIL-MSGID: 1778059608753634459 Make finish_mkwrite_fault() to a static function, and convert wp_page_reuse() and finish_mkwrite_fault() to take a folio in preparation for page_cpupid_xchg_last() to folio conversion. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 1 - mm/memory.c | 37 ++++++++++++++++++++----------------- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index aa7fdda1b56c..9933f6345e66 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1335,7 +1335,6 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, struct page *page, unsigned int nr, unsigned long addr); vm_fault_t finish_fault(struct vm_fault *vmf); -vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); #endif /* diff --git a/mm/memory.c b/mm/memory.c index 5ab6e8d45a7d..119c40e4465e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3014,23 +3014,24 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) * case, all we need to do here is to mark the page as writable and update * any related book-keeping. */ -static inline void wp_page_reuse(struct vm_fault *vmf) +static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio) __releases(vmf->ptl) { struct vm_area_struct *vma = vmf->vma; - struct page *page = vmf->page; pte_t entry; VM_BUG_ON(!(vmf->flags & FAULT_FLAG_WRITE)); - VM_BUG_ON(page && PageAnon(page) && !PageAnonExclusive(page)); + if (folio) { + VM_BUG_ON(folio_test_anon(folio) && + !PageAnonExclusive(vmf->page)); - /* - * Clear the pages cpupid information as the existing - * information potentially belongs to a now completely - * unrelated process. - */ - if (page) - page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1); + /* + * Clear the pages cpupid information as the existing + * information potentially belongs to a now completely + * unrelated process. + */ + page_cpupid_xchg_last(vmf->page, (1 << LAST_CPUPID_SHIFT) - 1); + } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); @@ -3223,6 +3224,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * writeable once the page is prepared * * @vmf: structure describing the fault + * @folio: the folio of vmf->page * * This function handles all that is needed to finish a write page fault in a * shared mapping due to PTE being read-only once the mapped page is prepared. @@ -3234,7 +3236,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before * we acquired PTE lock. */ -vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) +static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, + struct folio *folio) { WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, @@ -3250,7 +3253,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return VM_FAULT_NOPAGE; } - wp_page_reuse(vmf); + wp_page_reuse(vmf, folio); return 0; } @@ -3275,9 +3278,9 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) ret = vma->vm_ops->pfn_mkwrite(vmf); if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)) return ret; - return finish_mkwrite_fault(vmf); + return finish_mkwrite_fault(vmf, NULL); } - wp_page_reuse(vmf); + wp_page_reuse(vmf, NULL); return 0; } @@ -3305,14 +3308,14 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) folio_put(folio); return tmp; } - tmp = finish_mkwrite_fault(vmf); + tmp = finish_mkwrite_fault(vmf, folio); if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) { folio_unlock(folio); folio_put(folio); return tmp; } } else { - wp_page_reuse(vmf); + wp_page_reuse(vmf, folio); folio_lock(folio); } ret |= fault_dirty_shared_page(vmf); @@ -3436,7 +3439,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; } - wp_page_reuse(vmf); + wp_page_reuse(vmf, folio); return 0; } copy: From patchwork Tue Sep 26 00:52:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144655 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590356vqu; Mon, 25 Sep 2023 17:55:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGo8n454U605bB2Tzqo0rhr0CtJ3VToXu+Tfq3pBFRQkt4A709B03uNTEJ5TNTuunsdlxU1 X-Received: by 2002:a17:902:ea0f:b0:1c3:749f:6a5c with SMTP id s15-20020a170902ea0f00b001c3749f6a5cmr7388114plg.12.1695689746246; Mon, 25 Sep 2023 17:55:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689746; cv=none; d=google.com; s=arc-20160816; b=rg/xPIVZOMP22PonyqkUFeka/sAo6Gcq/Vf7ssAeqBDHpnZBr+4AnnZVGkWvhYfPjF f+9PHhJO+4NbkeGfwrM/KIbD+Qj72oSnfQ8wGzl2+j1cvWOk5WpuKqJ4e8QuDBYk1UBq hKmU17fooqO60KzFNeaPSduWOorNJIP6p1IIRcVFpgRoT/y1BipoI1yaRd5/aSKQV2G/ 2n3Bg9LuOmUGRzxYJ3AHSrNgogtHl3iHcw8rwu4PWZtyifxf5N9BJNnzFYmKQb6z/2G7 B8NV5LAXXCPRKE/YUkjemuCtZtWOXasdRF0bwCPueByinbyT6oOC+tfNIKdHF6Yd1wj5 Fe1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=iGP1uZCLGGtg16MJd/pOyCwveUPyD+uTEMShphh8Lhg=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=x76gSN2k7i1FBZbrv4UGFqox0DRTKnv9Zqnbm7quD8B/9Nrs7DmyRQ8lFkJ2gMwDwC 5xphPl4+KBDmGku8cMegRMUK9t3wxIsPpSQHQLnsRJOvyEpgg1W49hqalHBWS9zHI+w3 6cVztg/JvdcYuwG2W9aq6F6fix0SJDXxccRaTp8CZE1qpiamuAIDYG+Fv5/o7N8Bb2Jo 0WNrnVcOdcIq38x5ciUY/wG6ES31CzInZfzML1OEPqLaFLeVLyL7p/xtvZJ49+6f1qIK sM7ZOya2uBFM6R6SVvTJpEJdEa9pEAVPKL8//CfDhyhKtQiduQBs/JZmhSWlJlbBa2Vz Cgrw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id ij9-20020a170902ab4900b001c5e1816251si6052615plb.497.2023.09.25.17.55.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:55:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 215368339641; Mon, 25 Sep 2023 17:53:42 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233386AbjIZAxi (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232536AbjIZAxU (ORCPT ); Mon, 25 Sep 2023 20:53:20 -0400 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8913DD9 for ; Mon, 25 Sep 2023 17:53:13 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.57]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4Rvh4f4Jj7z15NQw; Tue, 26 Sep 2023 08:50:58 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:11 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 7/9] mm: convert page_cpupid_xchg_last() to folio_cpupid_xchg_last() Date: Tue, 26 Sep 2023 08:52:52 +0800 Message-ID: <20230926005254.2861577-8-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:53:42 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059571300200087 X-GMAIL-MSGID: 1778059571300200087 Make page_cpupid_xchg_last() to take a folio, and rename it to olio_cpupid_xchg_last() since all callers with a folio. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 14 +++++++------- kernel/sched/fair.c | 2 +- mm/huge_memory.c | 2 +- mm/memory.c | 2 +- mm/migrate.c | 4 ++-- mm/mmzone.c | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9933f6345e66..a6f4b55bf469 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1683,9 +1683,9 @@ static inline bool __cpupid_match_pid(pid_t task_pid, int cpupid) #define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid) #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS -static inline int page_cpupid_xchg_last(struct page *page, int cpupid) +static inline int folio_cpupid_xchg_last(struct folio *folio, int cpupid) { - return xchg(&page->_last_cpupid, cpupid & LAST_CPUPID_MASK); + return xchg(&folio->_last_cpupid, cpupid & LAST_CPUPID_MASK); } static inline int folio_cpupid_last(struct folio *folio) @@ -1702,7 +1702,7 @@ static inline int folio_cpupid_last(struct folio *folio) return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; } -extern int page_cpupid_xchg_last(struct page *page, int cpupid); +extern int folio_cpupid_xchg_last(struct folio *folio, int cpupid); static inline void page_cpupid_reset_last(struct page *page) { @@ -1714,8 +1714,8 @@ static inline int xchg_folio_access_time(struct folio *folio, int time) { int last_time; - last_time = page_cpupid_xchg_last(&folio->page, - time >> PAGE_ACCESS_TIME_BUCKETS); + last_time = folio_cpupid_xchg_last(folio, + time >> PAGE_ACCESS_TIME_BUCKETS); return last_time << PAGE_ACCESS_TIME_BUCKETS; } @@ -1729,9 +1729,9 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) } } #else /* !CONFIG_NUMA_BALANCING */ -static inline int page_cpupid_xchg_last(struct page *page, int cpupid) +static inline int folio_cpupid_xchg_last(struct folio *folio, int cpupid) { - return page_to_nid(page); /* XXX */ + return folio_nid(folio); /* XXX */ } static inline int xchg_folio_access_time(struct folio *folio, int time) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index afb9dc98a8ee..dca1546aa9c1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1810,7 +1810,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, } this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid); - last_cpupid = page_cpupid_xchg_last(&folio->page, this_cpupid); + last_cpupid = folio_cpupid_xchg_last(folio, this_cpupid); if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 93981a759daf..89e65ff46ad4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2508,7 +2508,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, if (page_is_idle(head)) set_page_idle(page_tail); - page_cpupid_xchg_last(page_tail, folio_cpupid_last(folio)); + folio_cpupid_xchg_last(new_folio, folio_cpupid_last(folio)); /* * always add to the tail because some iterators expect new diff --git a/mm/memory.c b/mm/memory.c index 119c40e4465e..bf07ebdc24a0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3030,7 +3030,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio) * information potentially belongs to a now completely * unrelated process. */ - page_cpupid_xchg_last(vmf->page, (1 << LAST_CPUPID_SHIFT) - 1); + folio_cpupid_xchg_last(folio, (1 << LAST_CPUPID_SHIFT) - 1); } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); diff --git a/mm/migrate.c b/mm/migrate.c index 7d1804c4a5d9..d41139ccbd3f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -588,7 +588,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) * Copy NUMA information to the new page, to prevent over-eager * future migrations of this same page. */ - cpupid = page_cpupid_xchg_last(&folio->page, -1); + cpupid = folio_cpupid_xchg_last(folio, -1); /* * For memory tiering mode, when migrate between slow and fast * memory node, reset cpupid, because that is used to record @@ -601,7 +601,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) if (f_toptier != t_toptier) cpupid = -1; } - page_cpupid_xchg_last(&newfolio->page, cpupid); + folio_cpupid_xchg_last(newfolio, cpupid); folio_migrate_ksm(newfolio, folio); /* diff --git a/mm/mmzone.c b/mm/mmzone.c index 68e1511be12d..cd473f82b647 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -93,19 +93,19 @@ void lruvec_init(struct lruvec *lruvec) } #if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) -int page_cpupid_xchg_last(struct page *page, int cpupid) +int folio_cpupid_xchg_last(struct folio *folio, int cpupid) { unsigned long old_flags, flags; int last_cpupid; - old_flags = READ_ONCE(page->flags); + old_flags = READ_ONCE(folio->flags); do { flags = old_flags; last_cpupid = (flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; flags &= ~(LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT); flags |= (cpupid & LAST_CPUPID_MASK) << LAST_CPUPID_PGSHIFT; - } while (unlikely(!try_cmpxchg(&page->flags, &old_flags, flags))); + } while (unlikely(!try_cmpxchg(&folio->flags, &old_flags, flags))); return last_cpupid; } From patchwork Tue Sep 26 00:52:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144656 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590394vqu; Mon, 25 Sep 2023 17:55:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHTQJnBV/K0Va0w2X+nBmhPOTgMyixaMJO3v1KuTp2+msfxs8EN71J3R332ZcaqMzmPrAp7 X-Received: by 2002:a05:6808:2a5b:b0:3a8:1db8:456b with SMTP id fa27-20020a0568082a5b00b003a81db8456bmr7046513oib.35.1695689752424; Mon, 25 Sep 2023 17:55:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689752; cv=none; d=google.com; s=arc-20160816; b=U2GDPZEZyHmHqCfUqalX6R2CMOG/IEgS7V5ST1Eh0eNJ7snFjDeENSPKxt1sGpbZaA Is2P5fuab3prF0QfPcnRCVmslC3+E8scJSUCNBwiPFgXSTQlzgcXgsa1sNTLWkN0dWqU GGhKj8NyeX7cMBKpu/YrJMv1nlf/8DzSFmqhT9HzEBRwx+io09ZMbbEb/jg5akqSvYNS GshqqZS9TbxLhBUH+BTdfdIMm1jFfpQu0CE87b0PWrbMHLuNun21N8uZ9y9uwcetrmL4 peRgKTfj9MXLiz3qwq+dh1sZifGxR9ZQZBAGNvjZuDzurxoj/RmA9hcMjnLOl3NGuYeQ 4n3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Z4qWbMQVyNDvnH/0/qkU92K86v2t3NC3Tu9QW6Z7yRg=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=a13417A/6DD+X6gqYTCfVo/irxmT9krPqz70WcRhNoVTR5awRXE84ptPrSPOhqm+0b yiMRDb/IvSfW0OtgDXcwWRzW98a99BscWCMvzFHfjQI9fb6BLZV6OvZFwgXEzWCb0mIA QNC8dleBShYRXptaeeIzKD3BxCa+3SJvfk+7LCiY3EixgTjjh4uidU/Ryu31JO1I47pU Q2lymc9Q8csF72OIWkSdZ8mnbCP27JqpiVnLNT6b3SYxEWc+s+M1sNYBiyb07tHs9LXL 2Kt13OjLTp5vv0sQqCJDT+HrBl9CgIqNdvF1x9FJE2gCOtMCOaUGRnGZG2kfcVGd+TCj PABA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id k24-20020a6568d8000000b0056569ee3ae7si11365673pgt.540.2023.09.25.17.55.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:55:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A239283348BD; Mon, 25 Sep 2023 17:54:04 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233454AbjIZAxk (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbjIZAxU (ORCPT ); Mon, 25 Sep 2023 20:53:20 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18705109 for ; Mon, 25 Sep 2023 17:53:14 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Rvh2C6FzGztT3x; Tue, 26 Sep 2023 08:48:51 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:11 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 8/9] mm: page_alloc: use a folio in free_pages_prepare() Date: Tue, 26 Sep 2023 08:52:53 +0800 Message-ID: <20230926005254.2861577-9-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:54:04 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059577900065505 X-GMAIL-MSGID: 1778059577900065505 The page should not a tail page in free_pages_prepare(), let's use a folio in free_pages_prepare() to save several compound_head() calls. Signed-off-by: Kefeng Wang --- mm/page_alloc.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 06be8821d833..a888b9d57751 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1070,6 +1070,7 @@ static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, fpi_t fpi_flags) { int bad = 0; + struct folio *folio = page_folio(page); bool skip_kasan_poison = should_skip_kasan_poison(page, fpi_flags); bool init = want_init_on_free(); @@ -1078,12 +1079,12 @@ static __always_inline bool free_pages_prepare(struct page *page, trace_mm_page_free(page, order); kmsan_free_page(page, order); - if (unlikely(PageHWPoison(page)) && !order) { + if (unlikely(folio_test_hwpoison(folio)) && !order) { /* * Do not let hwpoison pages hit pcplists/buddy * Untie memcg state and reset page's owner */ - if (memcg_kmem_online() && PageMemcgKmem(page)) + if (memcg_kmem_online() && folio_memcg_kmem(folio)) __memcg_kmem_uncharge_page(page, order); reset_page_owner(page, order); page_table_check_free(page, order); @@ -1095,10 +1096,10 @@ static __always_inline bool free_pages_prepare(struct page *page, * avoid checking PageCompound for order-0 pages. */ if (unlikely(order)) { - bool compound = PageCompound(page); + bool compound = folio_test_large(folio); int i; - VM_BUG_ON_PAGE(compound && compound_order(page) != order, page); + VM_BUG_ON_FOLIO(compound && folio_order(folio) != order, folio); if (compound) page[1].flags &= ~PAGE_FLAGS_SECOND; @@ -1114,9 +1115,9 @@ static __always_inline bool free_pages_prepare(struct page *page, (page + i)->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; } } - if (PageMappingFlags(page)) + if (folio_mapping_flags(folio)) page->mapping = NULL; - if (memcg_kmem_online() && PageMemcgKmem(page)) + if (memcg_kmem_online() && folio_memcg_kmem(folio)) __memcg_kmem_uncharge_page(page, order); if (is_check_pages_enabled()) { if (free_page_is_bad(page)) @@ -1130,7 +1131,7 @@ static __always_inline bool free_pages_prepare(struct page *page, reset_page_owner(page, order); page_table_check_free(page, order); - if (!PageHighMem(page)) { + if (!folio_test_highmem(folio)) { debug_check_no_locks_freed(page_address(page), PAGE_SIZE << order); debug_check_no_obj_freed(page_address(page), From patchwork Tue Sep 26 00:52:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 144660 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1590555vqu; Mon, 25 Sep 2023 17:56:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHPpyRw/+RLQM2TCJxNGHf693DbyN+RZuh48nmN3QlDb39k5uy1/GCqTpHMO2qvOlmZSlXi X-Received: by 2002:a17:902:ecd2:b0:1bc:2c79:c6b6 with SMTP id a18-20020a170902ecd200b001bc2c79c6b6mr6803550plh.28.1695689785470; Mon, 25 Sep 2023 17:56:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695689785; cv=none; d=google.com; s=arc-20160816; b=PK9EemtP3ieAjVAeFDsbkFHNftd8Eww1I6xW5v+rVhNup5LfVI/XjWSiFmv2MX+0su J8Y8Siz+kOh2aimRMeDkQ1YuLNtuU/98L+5iTZX2frw4reX9Ri3ioNI1YBe23BSIwKy5 4mCill6u4npeMT/e/9KxQLdY8ZsRglITKn24NT0QYm+3aXF2kybTwz/DfcHWj30m79aI uJfqon5ruJWZ2zVnWZHMeykRsp+iC+gviFT8o7f5UIrS/IvA3EcpW5tMnGa4/acz+P3D uRKhK/4UiFrvtQo+4dFzylsh3ZYR/oLU4+QDRLR2TjG8UH56RZOw2J2GlE8zf473e6WC /Y7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=0mBqhTUBjVNOhn3gad4QEbyJ4wj4oLKXxijp0qqMZg4=; fh=FiTqiBStz9H28Qh/9rW0myzOFac7M9VOPFJTunt8oTE=; b=cZhBONclj4OiLaDQU1/yLTcOE+L0gF9a30pcxUOohLQ/BS0eGjv9em7rBDlPPpcwBU m+zHzOO/3/bswEnVy2TRCebTh2uSWEmwSHV0nZ/IdNxjj/1A50DEucgtHL1PmslSUwOe CBqDZ37GqnVhTepMxZRMycwDTucjxeSD4GCfnd9DhPtuvUVHZOJNusPH2aYGKH5JQnXh oexDypacmMW8Yq55miC71eH09ROHkDQ2zOZwa1wNl2SDCsSPdUhWZ1YgfWatN3Bsrxqr i8tzPicURQ3pDc6Q7OIj3MZtPSq+B50kpR679jCKCTMddxALiYSM5URwKWcEnM199Faj 7n9Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id b14-20020a170903228e00b001bb3406a5f7si11339815plh.11.2023.09.25.17.56.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 17:56:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 1FD01825173A; Mon, 25 Sep 2023 17:54:01 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233494AbjIZAxl (ORCPT + 27 others); Mon, 25 Sep 2023 20:53:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232582AbjIZAxV (ORCPT ); Mon, 25 Sep 2023 20:53:21 -0400 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16A0010A for ; Mon, 25 Sep 2023 17:53:14 -0700 (PDT) Received: from dggpemm100001.china.huawei.com (unknown [172.30.72.53]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4Rvh2z45FGzMlpj; Tue, 26 Sep 2023 08:49:31 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Tue, 26 Sep 2023 08:53:12 +0800 From: Kefeng Wang To: Andrew Morton CC: Mike Rapoport , Matthew Wilcox , David Hildenbrand , , , , Zi Yan , Kefeng Wang Subject: [PATCH -next 9/9] mm: convert page_cpupid_reset_last() to folio_cpupid_reset_last() Date: Tue, 26 Sep 2023 08:52:54 +0800 Message-ID: <20230926005254.2861577-10-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> References: <20230926005254.2861577-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm100001.china.huawei.com (7.185.36.93) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 25 Sep 2023 17:54:01 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778059612587229213 X-GMAIL-MSGID: 1778059612587229213 It isn't need to fill the default cpupid value for all the struct page, since cpupid is only used for numa balancing, and the pages for numa balancing are all from buddy, page_cpupid_reset_last() is already called by free_pages_prepare() to initialize it, so let's drop the page_cpupid_reset_last() in __init_single_page(), then make page_cpupid_reset_last() to take a folio and rename it to folio_cpupid_reset_last(). Signed-off-by: Kefeng Wang --- include/linux/mm.h | 10 +++++----- mm/mm_init.c | 1 - mm/page_alloc.c | 2 +- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6f4b55bf469..ca66a05eb2ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1692,9 +1692,9 @@ static inline int folio_cpupid_last(struct folio *folio) { return folio->_last_cpupid; } -static inline void page_cpupid_reset_last(struct page *page) +static inline void folio_cpupid_reset_last(struct folio *folio) { - page->_last_cpupid = -1 & LAST_CPUPID_MASK; + folio->_last_cpupid = -1 & LAST_CPUPID_MASK; } #else static inline int folio_cpupid_last(struct folio *folio) @@ -1704,9 +1704,9 @@ static inline int folio_cpupid_last(struct folio *folio) extern int folio_cpupid_xchg_last(struct folio *folio, int cpupid); -static inline void page_cpupid_reset_last(struct page *page) +static inline void folio_cpupid_reset_last(struct folio *folio) { - page->flags |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT; + folio->flags |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT; } #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ @@ -1769,7 +1769,7 @@ static inline bool cpupid_pid_unset(int cpupid) return true; } -static inline void page_cpupid_reset_last(struct page *page) +static inline void folio_cpupid_reset_last(struct folio *folio) { } diff --git a/mm/mm_init.c b/mm/mm_init.c index 06a72c223bce..74c0dc27fbf1 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -563,7 +563,6 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, set_page_links(page, zone, nid, pfn); init_page_count(page); page_mapcount_reset(page); - page_cpupid_reset_last(page); page_kasan_tag_reset(page); INIT_LIST_HEAD(&page->lru); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a888b9d57751..852fc78ddb34 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1126,7 +1126,7 @@ static __always_inline bool free_pages_prepare(struct page *page, return false; } - page_cpupid_reset_last(page); + folio_cpupid_reset_last(folio); page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order);