From patchwork Wed May 3 01:36:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 89538 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1018385vqo; Tue, 2 May 2023 18:57:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7kGvyQ7UudoI/xlKTeNYaEAda5MIqbDB2TZw214rSTcaUcMq0h3RJ3wbZyJyNb0YJEi65R X-Received: by 2002:a05:6a20:2d0a:b0:ef:3e60:3f46 with SMTP id g10-20020a056a202d0a00b000ef3e603f46mr20202484pzl.31.1683079074512; Tue, 02 May 2023 18:57:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683079074; cv=none; d=google.com; s=arc-20160816; b=gPLVfmWsxlc0w6rKxA1K19wU91a9qN8+WMdRBML6nVoV5NgU3YOaiNwwyCq1aWEE9O IiHNaY3YAT/VfaNZymUJIG+YVt3gHvTigK9cHEmTXY9m3Ao8EbkzFYlaAvw+i+J7rh7p wKzO4i2FB4+0DvaQTIWT4m6Zl+MZJhzznYyjHbkrXZ+2UkcuN6p+MeI+tN+5/8SJVH7Q 0bC1ao4K/ju8nkvTk8w2EV4egTiq1G4nwTdOTtDOjKAp1LVtXEOO0XSDTiGOJTNOaLzp YcQuRpzuqr7jODDq/PDAn6UgZtN59VE91qZgh6IPXeY6bBJQDNGNSKoN8jQho87Xd8gq tgbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MM6teyjPlmkc2NzJ+Re62gPlYr2XnVPXuXbH/2DQ358=; b=LYm2uFrNmodRG/duf8p6MdFNTcm1S5V7nnP4EpaWRjzAj/2exD9QATGUetGYgQiict g/cSoP/6d24hdmcC+e7QvfHoGR8NhjyB3WfDXwL89jU7+JtHYWm532SSfr2aq5I28+41 c+gBFAOTcy8GAtxIC0Bv9XgJ07CZ4SAdK+Z0Uw7DYRfl31IFJD97Bk7A3uoV1DnOMhdQ oRZ+ztYFtoE0ed4JKBBDoVWzsHbh0yGSVQpA6oHxQStlydKfnqLOs2lyr75s9yDZji4Q aq97L/KzPMx1RBmfxiq952YDbz2McpOOsirwoVRbFE0/QN0uzdMnFcxlQ58swYQV6BFm Aheg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=p1yGAarx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oc3-20020a17090b1c0300b0023e066d4e2dsi13798272pjb.184.2023.05.02.18.57.39; Tue, 02 May 2023 18:57:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=p1yGAarx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229598AbjECBgR (ORCPT + 99 others); Tue, 2 May 2023 21:36:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229572AbjECBgN (ORCPT ); Tue, 2 May 2023 21:36:13 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 227962D59; Tue, 2 May 2023 18:36:11 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1ab1b79d3a7so2047045ad.3; Tue, 02 May 2023 18:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683077770; x=1685669770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MM6teyjPlmkc2NzJ+Re62gPlYr2XnVPXuXbH/2DQ358=; b=p1yGAarxO09PbCP0WqCQvBy9dOXFLCU/jbfRSJ7zl4ThLhzF/99QvitdmQfZATQO4Z mXlwX+1OeWixhf944Nm9DoVYa9IDRBve633yEpMykaSkvvo59OuWKmJgtdc6/gptOx8s UL4KBFpStrCCxH/3gvuENWUj9c9GgqS6fePf/GrtaFzZ7R+ksFtFSeskgRNC8Zl2rLoT +MIUwMFeXCQt8bM4pJT7RtTujk1eDFrAPwNCVxm/PwYxVrz+Uvgyhq0lc/sUIa+cNTrk SreR49cxb2n2mslRvCsTVykUqNuqjYWQulou+sbVshMOmztKLKnqRq0fRP7HQ5YEpspv YCNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683077770; x=1685669770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MM6teyjPlmkc2NzJ+Re62gPlYr2XnVPXuXbH/2DQ358=; b=I7vwQKigM92pSiwtKq8fymPjJ0zddQGHcVv0w3Uq4p9c68ujxpsgUVv09aO33vjCE6 c9rTdWBSpjiwsEv21zo3iJ54OZso/3eoGtldqDdiVkZdoLkNx3tA6ggRKwq7dZb2rawR LGJrgBviJXKAbJm9PeBTmDf9QXioqZhrEy/7Fp4N6E5OvgZ2RjXic20cTPQIqvxvRC0P lHqsCKA0OG7WD/vnW52Rw3Ia6nVGyGAAThVGlK9pbw3G3rGiveL5gLGMG6Vm/IY2J5+j O+2KrsXTxVMrl2Ii21VLQJpZU3SHvMW3KJ7EKNDK0VEMV4upvwSZ1i7IOB1mk4mYdFqc Ckcg== X-Gm-Message-State: AC+VfDyo3TnxE449KCnolVA4/lHvUsW9LVCW4DhUk+cin/I3XR9e1uuo ZE0LPv0makTqis8AOZ0vBXs= X-Received: by 2002:a17:902:ecc3:b0:1aa:f173:2892 with SMTP id a3-20020a170902ecc300b001aaf1732892mr478420plh.57.1683077770360; Tue, 02 May 2023 18:36:10 -0700 (PDT) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id g22-20020a170902869600b001a67759f9f8sm20301684plo.106.2023.05.02.18.36.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 May 2023 18:36:10 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, linux-api@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v13 1/3] workingset: refactor LRU refault to expose refault recency check Date: Tue, 2 May 2023 18:36:06 -0700 Message-Id: <20230503013608.2431726-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230503013608.2431726-1-nphamcs@gmail.com> References: <20230503013608.2431726-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764836323730785663?= X-GMAIL-MSGID: =?utf-8?q?1764836323730785663?= In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. Signed-off-by: Nhat Pham Acked-by: Johannes Weiner --- include/linux/swap.h | 1 + mm/workingset.c | 150 +++++++++++++++++++++++++++++-------------- 2 files changed, 103 insertions(+), 48 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 3c69cb653cb9..b2128df5edea 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -368,6 +368,7 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) } /* linux/mm/workingset.c */ +bool workingset_test_recent(void *shadow, bool file, bool *workingset); void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); void workingset_refault(struct folio *folio, void *shadow); diff --git a/mm/workingset.c b/mm/workingset.c index 817758951886..d81f9dafc9f1 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -255,6 +255,29 @@ static void *lru_gen_eviction(struct folio *folio) return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); } +/* + * Tests if the shadow entry is for a folio that was recently evicted. + * Fills in @memcgid, @pglist_data, @token, @workingset with the values + * unpacked from shadow. + */ +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + struct mem_cgroup *eviction_memcg; + struct lruvec *lruvec; + struct lru_gen_folio *lrugen; + unsigned long min_seq; + + unpack_shadow(shadow, memcgid, pgdat, token, workingset); + eviction_memcg = mem_cgroup_from_id(*memcgid); + + lruvec = mem_cgroup_lruvec(eviction_memcg, *pgdat); + lrugen = &lruvec->lrugen; + + min_seq = READ_ONCE(lrugen->min_seq[file]); + return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); +} + static void lru_gen_refault(struct folio *folio, void *shadow) { int hist, tier, refs; @@ -269,23 +292,22 @@ static void lru_gen_refault(struct folio *folio, void *shadow) int type = folio_is_file_lru(folio); int delta = folio_nr_pages(folio); - unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); - - if (pgdat != folio_pgdat(folio)) - return; - rcu_read_lock(); + if (!lru_gen_test_recent(shadow, type, &memcg_id, &pgdat, &token, + &workingset)) + goto unlock; + memcg = folio_memcg_rcu(folio); if (memcg_id != mem_cgroup_id(memcg)) goto unlock; + if (pgdat != folio_pgdat(folio)) + return; + lruvec = mem_cgroup_lruvec(memcg, pgdat); lrugen = &lruvec->lrugen; - min_seq = READ_ONCE(lrugen->min_seq[type]); - if ((token >> LRU_REFS_WIDTH) != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))) - goto unlock; hist = lru_hist_from_seq(min_seq); /* see the comment in folio_lru_refs() */ @@ -317,6 +339,12 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + return false; +} + static void lru_gen_refault(struct folio *folio, void *shadow) { } @@ -385,42 +413,34 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) } /** - * workingset_refault - Evaluate the refault of a previously evicted folio. - * @folio: The freshly allocated replacement folio. - * @shadow: Shadow entry of the evicted folio. - * - * Calculates and evaluates the refault distance of the previously - * evicted folio in the context of the node and the memcg whose memory - * pressure caused the eviction. + * workingset_test_recent - tests if the shadow entry is for a folio that was + * recently evicted. Also fills in @workingset with the value unpacked from + * shadow. + * @shadow: the shadow entry to be tested. + * @file: whether the corresponding folio is from the file lru. + * @workingset: where the workingset value unpacked from shadow should + * be stored. + * + * Return: true if the shadow is for a recently evicted folio; false otherwise. */ -void workingset_refault(struct folio *folio, void *shadow) +bool workingset_test_recent(void *shadow, bool file, bool *workingset) { - bool file = folio_is_file_lru(folio); struct mem_cgroup *eviction_memcg; struct lruvec *eviction_lruvec; unsigned long refault_distance; unsigned long workingset_size; - struct pglist_data *pgdat; - struct mem_cgroup *memcg; - unsigned long eviction; - struct lruvec *lruvec; unsigned long refault; - bool workingset; int memcgid; - long nr; + struct pglist_data *pgdat; + unsigned long eviction; - if (lru_gen_enabled()) { - lru_gen_refault(folio, shadow); - return; - } + if (lru_gen_enabled()) + return lru_gen_test_recent(shadow, file, &memcgid, &pgdat, &eviction, + workingset); - unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might * have been deleted since the folio's eviction. @@ -439,7 +459,8 @@ void workingset_refault(struct folio *folio, void *shadow) */ eviction_memcg = mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !eviction_memcg) - goto out; + return false; + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -461,20 +482,6 @@ void workingset_refault(struct folio *folio, void *shadow) */ refault_distance = (refault - eviction) & EVICTION_MASK; - /* - * The activation decision for this folio is made at the level - * where the eviction occurred, as that is where the LRU order - * during folio reclaim is being determined. - * - * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. - */ - nr = folio_nr_pages(folio); - memcg = folio_memcg(folio); - pgdat = folio_pgdat(folio); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if @@ -495,7 +502,54 @@ void workingset_refault(struct folio *folio, void *shadow) NR_INACTIVE_ANON); } } - if (refault_distance > workingset_size) + + return refault_distance <= workingset_size; +} + +/** + * workingset_refault - Evaluate the refault of a previously evicted folio. + * @folio: The freshly allocated replacement folio. + * @shadow: Shadow entry of the evicted folio. + * + * Calculates and evaluates the refault distance of the previously + * evicted folio in the context of the node and the memcg whose memory + * pressure caused the eviction. + */ +void workingset_refault(struct folio *folio, void *shadow) +{ + bool file = folio_is_file_lru(folio); + struct pglist_data *pgdat; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + bool workingset; + long nr; + + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + + rcu_read_lock(); + + /* + * The activation decision for this folio is made at the level + * where the eviction occurred, as that is where the LRU order + * during folio reclaim is being determined. + * + * However, the cgroup that will own the folio is the one that + * is actually experiencing the refault event. + */ + nr = folio_nr_pages(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); + + if (!workingset_test_recent(shadow, file, &workingset)) goto out; folio_set_active(folio); From patchwork Wed May 3 01:36:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 89535 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1011932vqo; Tue, 2 May 2023 18:38:43 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6GwtBC/LnCnbR0sXQBizNjQznPTCNifcBG0LJQDhDAp1vpVFtRJX71Upim1X67ECaWyzbf X-Received: by 2002:a17:902:7c8d:b0:1a9:2c70:e1eb with SMTP id y13-20020a1709027c8d00b001a92c70e1ebmr435302pll.36.1683077923223; Tue, 02 May 2023 18:38:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683077923; cv=none; d=google.com; s=arc-20160816; b=q4KkK55FlCuTinXo6hEfg9Hrv3Jhfuv0U3o2WxcfPyrSbjyvw99ugyPRknr7oHIH0W 1mUg/UV3GoymtI23P81JBWuVqKEsbt9njzfNDVd1HV4o6R34et0EkHLQoFljjpj7I2JD PwX7iJPMeQm759XgbxICiORLjH4zC4ebexSoQGv4LGN++ASligxpcjxPxeB41GQ12McS 8jYpu9bhdnbsfI3meesEE3JQOql3eKnq7qM2rhQxaZ4m6k5HPxE8KX5U7953VAFKNl8y NW+NhuBu7PpFHJRpFXXKYBNuseWPw+Nh0pEXvev/gJsd/D3vrCBnNkLWxxWDUyZ1eblA bC9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9VX1P6p4S44WrbFadgyxcCHiNhnuoZy6JRGXDkBCmYw=; b=c+7SzZVwomcv597xkHkzvE/QShXsxOh77NhcyEhfzVw5bHy3tEP3l/97LFegMZ5vhv McmUNd91Gmv7/YBlCO/1VcIwj+cWCEF0am7p7jE/o+cnTHe/mnQtmli8RnY0kJSgkYd3 AtQ4OOwA9YBnY7UrwER9WcN5ZV5blpmz+VOkwRL30CHtqa/Id3mn7FA+ZzhDhDDRnOwO 0QYQiEq7mFkz2/AryGitlWneDugHhkP0WnsPxrpxVIak7lvMUMyzMhl/EcDwo0uEdCiP 2j3KJU5LgxzgF7sPxWx4WncOQqCoH0ZQLr9RgAfsaD842TGkExAQvyPg+RcoIFOsJk/U RpnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=ZUi5CHiS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p18-20020a170902ebd200b001a6cf4b4d6dsi27506082plg.289.2023.05.02.18.38.30; Tue, 02 May 2023 18:38:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=ZUi5CHiS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229606AbjECBgU (ORCPT + 99 others); Tue, 2 May 2023 21:36:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229461AbjECBgO (ORCPT ); Tue, 2 May 2023 21:36:14 -0400 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C7352D5E; Tue, 2 May 2023 18:36:12 -0700 (PDT) Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-24dfc3c662eso2147447a91.3; Tue, 02 May 2023 18:36:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683077772; x=1685669772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9VX1P6p4S44WrbFadgyxcCHiNhnuoZy6JRGXDkBCmYw=; b=ZUi5CHiSpNlwcO0k6OK5+buqnIzWlJEj+oOp4gzxGMfDwuYeEQCHHYTs5skf8QdGwT HbW9SlITQhs1GRxpbGgQDKekbP1G9s1Dzb3JCUmsYxtozfUyKeIDSB/wnq8zRyP+/a7W 1L33O5u5TbSy1rNUS6AGQaqie/J3MG+dloAXIMmEt96VXPC4/c4Gae4kXTZL5pLuaMMw mgv2h+N719a3+9DAI0U5kzthZGbSJkDNxknp4u9VTpua0oJ6EZ4YOhWXTSicqKI9vfjx J/RmIUMSPakSUOHSJlgP1jMV4B7B7vpK/7VMKF41+DKctXGlmUF68t79aE7067QWo2Qz by1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683077772; x=1685669772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9VX1P6p4S44WrbFadgyxcCHiNhnuoZy6JRGXDkBCmYw=; b=TldcMS+mmHK1jeUG8zrgHGLB/cvPcUN7jrlva86iOe0/n3awlMWF+bntoom8cXaSmI Nu6/zyA9HTLJpp+NhrAV6mXljS5z174ZGUu4+0MbJWbwXHQW5lpufnkUUy0H4kEGjdXe Vn6kCUrE3Vq186Xql1353LugZ5sxPaMXdU2IjdoSxDBgfbeoOtNLZ7V65ltWplk1Qysa lt1DlF26DMxJQGknn2AIQQO8n5iXBoUbj3zFrXC7dZfqkwdDxwCPwqGceqzqVyAmulSW 84F6j5WhpK2Hlmjz6sDFTCXrb/IChHhMDilmydtGkWnUTlHL7G3zqdtAHSRM1TmCN5tI 5Afw== X-Gm-Message-State: AC+VfDyF3Sp65DsPO5EdIbJhV6BLTUzZnMb9+U/16Eoby8eN08czVNca PmPuOiyL1qwN4/ENb2Kc1zFWMd1xMDI= X-Received: by 2002:a17:90b:33cc:b0:24d:ee34:57b6 with SMTP id lk12-20020a17090b33cc00b0024dee3457b6mr11070768pjb.41.1683077771422; Tue, 02 May 2023 18:36:11 -0700 (PDT) Received: from localhost (fwdproxy-prn-008.fbsv.net. [2a03:2880:ff:8::face:b00c]) by smtp.gmail.com with ESMTPSA id co1-20020a17090afe8100b0024de5227d1fsm48108pjb.40.2023.05.02.18.36.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 May 2023 18:36:11 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, linux-api@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v13 2/3] cachestat: implement cachestat syscall Date: Tue, 2 May 2023 18:36:07 -0700 Message-Id: <20230503013608.2431726-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230503013608.2431726-1-nphamcs@gmail.com> References: <20230503013608.2431726-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764835116377539308?= X-GMAIL-MSGID: =?utf-8?q?1764835116377539308?= There is currently no good way to query the page cache state of large file sets and directory trees. There is mincore(), but it scales poorly: the kernel writes out a lot of bitmap data that userspace has to aggregate, when the user really doesn not care about per-page information in that case. The user also needs to mmap and unmap each file as it goes along, which can be quite slow as well. Some use cases where this information could come in handy: * Allowing database to decide whether to perform an index scan or direct table queries based on the in-memory cache state of the index. * Visibility into the writeback algorithm, for performance issues diagnostic. * Workload-aware writeback pacing: estimating IO fulfilled by page cache (and IO to be done) within a range of a file, allowing for more frequent syncing when and where there is IO capacity, and batching when there is not. * Computing memory usage of large files/directory trees, analogous to the du tool for disk usage. More information about these use cases could be found in the following thread: https://lore.kernel.org/lkml/20230315170934.GA97793@cmpxchg.org/ This patch implements a new syscall that queries cache state of a file and summarizes the number of cached pages, number of dirty pages, number of pages marked for writeback, number of (recently) evicted pages, etc. in a given range. Currently, the syscall is only wired in for x86 architecture. NAME cachestat - query the page cache statistics of a file. SYNOPSIS #include struct cachestat_range { __u64 off; __u64 len; }; struct cachestat { __u64 nr_cache; __u64 nr_dirty; __u64 nr_writeback; __u64 nr_evicted; __u64 nr_recently_evicted; }; int cachestat(unsigned int fd, struct cachestat_range *cstat_range, struct cachestat *cstat, unsigned int flags); DESCRIPTION cachestat() queries the number of cached pages, number of dirty pages, number of pages marked for writeback, number of evicted pages, number of recently evicted pages, in the bytes range given by `off` and `len`. An evicted page is a page that is previously in the page cache but has been evicted since. A page is recently evicted if its last eviction was recent enough that its reentry to the cache would indicate that it is actively being used by the system, and that there is memory pressure on the system. These values are returned in a cachestat struct, whose address is given by the `cstat` argument. The `off` and `len` arguments must be non-negative integers. If `len` > 0, the queried range is [`off`, `off` + `len`]. If `len` == 0, we will query in the range from `off` to the end of the file. The `flags` argument is unused for now, but is included for future extensibility. User should pass 0 (i.e no flag specified). Currently, hugetlbfs is not supported. Because the status of a page can change after cachestat() checks it but before it returns to the application, the returned values may contain stale information. RETURN VALUE On success, cachestat returns 0. On error, -1 is returned, and errno is set to indicate the error. ERRORS EFAULT cstat or cstat_args points to an invalid address. EINVAL invalid flags. EBADF invalid file descriptor. EOPNOTSUPP file descriptor is of a hugetlbfs file Signed-off-by: Nhat Pham Acked-by: Johannes Weiner --- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 5 + include/uapi/asm-generic/unistd.h | 5 +- include/uapi/linux/mman.h | 14 ++ init/Kconfig | 10 ++ kernel/sys_ni.c | 1 + mm/filemap.c | 172 +++++++++++++++++++++++++ 8 files changed, 208 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl index 320480a8db4f..bc0a3c941b35 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -455,3 +455,4 @@ 448 i386 process_mrelease sys_process_mrelease 449 i386 futex_waitv sys_futex_waitv 450 i386 set_mempolicy_home_node sys_set_mempolicy_home_node +451 i386 cachestat sys_cachestat diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index c84d12608cd2..227538b0ce80 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 common cachestat sys_cachestat # # Due to a historical design error, certain syscalls are numbered differently diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index 33a0ee3bcb2e..6648c07c4381 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -72,6 +72,8 @@ struct open_how; struct mount_attr; struct landlock_ruleset_attr; enum landlock_rule_type; +struct cachestat_range; +struct cachestat; #include #include @@ -1058,6 +1060,9 @@ asmlinkage long sys_memfd_secret(unsigned int flags); asmlinkage long sys_set_mempolicy_home_node(unsigned long start, unsigned long len, unsigned long home_node, unsigned long flags); +asmlinkage long sys_cachestat(unsigned int fd, + struct cachestat_range __user *cstat_range, + struct cachestat __user *cstat, unsigned int flags); /* * Architecture-specific system calls diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 45fa180cc56a..cd639fae9086 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -886,8 +886,11 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) #define __NR_set_mempolicy_home_node 450 __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) +#define __NR_cachestat 451 +__SYSCALL(__NR_cachestat, sys_cachestat) + #undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452 /* * 32 bit systems traditionally used different diff --git a/include/uapi/linux/mman.h b/include/uapi/linux/mman.h index f55bc680b5b0..a246e11988d5 100644 --- a/include/uapi/linux/mman.h +++ b/include/uapi/linux/mman.h @@ -4,6 +4,7 @@ #include #include +#include #define MREMAP_MAYMOVE 1 #define MREMAP_FIXED 2 @@ -41,4 +42,17 @@ #define MAP_HUGE_2GB HUGETLB_FLAG_ENCODE_2GB #define MAP_HUGE_16GB HUGETLB_FLAG_ENCODE_16GB +struct cachestat_range { + __u64 off; + __u64 len; +}; + +struct cachestat { + __u64 nr_cache; + __u64 nr_dirty; + __u64 nr_writeback; + __u64 nr_evicted; + __u64 nr_recently_evicted; +}; + #endif /* _UAPI_LINUX_MMAN_H */ diff --git a/init/Kconfig b/init/Kconfig index 32c24950c4ce..f7f65af4ee12 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1771,6 +1771,16 @@ config RSEQ If unsure, say Y. +config CACHESTAT_SYSCALL + bool "Enable cachestat() system call" if EXPERT + default y + help + Enable the cachestat system call, which queries the page cache + statistics of a file (number of cached pages, dirty pages, + pages marked for writeback, (recently) evicted pages). + + If unsure say Y here. + config DEBUG_RSEQ default n bool "Enabled debugging of rseq() system call" if EXPERT diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 860b2dcf3ac4..04bfb1e4d377 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -299,6 +299,7 @@ COND_SYSCALL(set_mempolicy); COND_SYSCALL(migrate_pages); COND_SYSCALL(move_pages); COND_SYSCALL(set_mempolicy_home_node); +COND_SYSCALL(cachestat); COND_SYSCALL(perf_event_open); COND_SYSCALL(accept4); diff --git a/mm/filemap.c b/mm/filemap.c index a34abfe8c654..73b043240cef 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -58,6 +59,8 @@ #include +#include "swap.h" + /* * Shared mappings implemented 30.11.1994. It's not fully working yet, * though. @@ -4119,3 +4122,172 @@ bool filemap_release_folio(struct folio *folio, gfp_t gfp) return try_to_free_buffers(folio); } EXPORT_SYMBOL(filemap_release_folio); + +#ifdef CONFIG_CACHESTAT_SYSCALL +/** + * filemap_cachestat() - compute the page cache statistics of a mapping + * @mapping: The mapping to compute the statistics for. + * @first_index: The starting page cache index. + * @last_index: The final page index (inclusive). + * @cs: the cachestat struct to write the result to. + * + * This will query the page cache statistics of a mapping in the + * page range of [first_index, last_index] (inclusive). The statistics + * queried include: number of dirty pages, number of pages marked for + * writeback, and the number of (recently) evicted pages. + */ +static void filemap_cachestat(struct address_space *mapping, + pgoff_t first_index, pgoff_t last_index, struct cachestat *cs) +{ + XA_STATE(xas, &mapping->i_pages, first_index); + struct folio *folio; + + rcu_read_lock(); + xas_for_each(&xas, folio, last_index) { + unsigned long nr_pages; + pgoff_t folio_first_index, folio_last_index; + + if (xas_retry(&xas, folio)) + continue; + + if (xa_is_value(folio)) { + /* page is evicted */ + void *shadow = (void *)folio; + bool workingset; /* not used */ + int order = xa_get_order(xas.xa, xas.xa_index); + + nr_pages = 1 << order; + /* rounds down to the nearest multiple of 2^order */ + folio_first_index = xas.xa_index >> order << order; + folio_last_index = folio_first_index + nr_pages - 1; + + /* Folios might straddle the range boundaries, only count covered pages */ + if (folio_first_index < first_index) + nr_pages -= first_index - folio_first_index; + + if (folio_last_index > last_index) + nr_pages -= folio_last_index - last_index; + + cs->nr_evicted += nr_pages; + +#ifdef CONFIG_SWAP /* implies CONFIG_MMU */ + if (shmem_mapping(mapping)) { + /* shmem file - in swap cache */ + swp_entry_t swp = radix_to_swp_entry(folio); + + shadow = get_shadow_from_swap_cache(swp); + } +#endif + if (workingset_test_recent(shadow, true, &workingset)) + cs->nr_recently_evicted += nr_pages; + + goto resched; + } + + nr_pages = folio_nr_pages(folio); + folio_first_index = folio_pgoff(folio); + folio_last_index = folio_first_index + nr_pages - 1; + + /* Folios might straddle the range boundaries, only count covered pages */ + if (folio_first_index < first_index) + nr_pages -= first_index - folio_first_index; + + if (folio_last_index > last_index) + nr_pages -= folio_last_index - last_index; + + /* page is in cache */ + cs->nr_cache += nr_pages; + + if (folio_test_dirty(folio)) + cs->nr_dirty += nr_pages; + + if (folio_test_writeback(folio)) + cs->nr_writeback += nr_pages; + +resched: + if (need_resched()) { + xas_pause(&xas); + cond_resched_rcu(); + } + } + rcu_read_unlock(); +} + +/* + * The cachestat(2) system call. + * + * cachestat() returns the page cache statistics of a file in the + * bytes range specified by `off` and `len`: number of cached pages, + * number of dirty pages, number of pages marked for writeback, + * number of evicted pages, and number of recently evicted pages. + * + * An evicted page is a page that is previously in the page cache + * but has been evicted since. A page is recently evicted if its last + * eviction was recent enough that its reentry to the cache would + * indicate that it is actively being used by the system, and that + * there is memory pressure on the system. + * + * `off` and `len` must be non-negative integers. If `len` > 0, + * the queried range is [`off`, `off` + `len`]. If `len` == 0, + * we will query in the range from `off` to the end of the file. + * + * The `flags` argument is unused for now, but is included for future + * extensibility. User should pass 0 (i.e no flag specified). + * + * Currently, hugetlbfs is not supported. + * + * Because the status of a page can change after cachestat() checks it + * but before it returns to the application, the returned values may + * contain stale information. + * + * return values: + * zero - success + * -EFAULT - cstat or cstat_range points to an illegal address + * -EINVAL - invalid flags + * -EBADF - invalid file descriptor + * -EOPNOTSUPP - file descriptor is of a hugetlbfs file + */ +SYSCALL_DEFINE4(cachestat, unsigned int, fd, + struct cachestat_range __user *, cstat_range, + struct cachestat __user *, cstat, unsigned int, flags) +{ + struct fd f = fdget(fd); + struct address_space *mapping; + struct cachestat_range csr; + struct cachestat cs; + pgoff_t first_index, last_index; + + if (!f.file) + return -EBADF; + + if (copy_from_user(&csr, cstat_range, + sizeof(struct cachestat_range))) { + fdput(f); + return -EFAULT; + } + + /* hugetlbfs is not supported */ + if (is_file_hugepages(f.file)) { + fdput(f); + return -EOPNOTSUPP; + } + + if (flags != 0) { + fdput(f); + return -EINVAL; + } + + first_index = csr.off >> PAGE_SHIFT; + last_index = + csr.len == 0 ? ULONG_MAX : (csr.off + csr.len - 1) >> PAGE_SHIFT; + memset(&cs, 0, sizeof(struct cachestat)); + mapping = f.file->f_mapping; + filemap_cachestat(mapping, first_index, last_index, &cs); + fdput(f); + + if (copy_to_user(cstat, &cs, sizeof(struct cachestat))) + return -EFAULT; + + return 0; +} +#endif /* CONFIG_CACHESTAT_SYSCALL */ From patchwork Wed May 3 01:36:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 89536 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1017023vqo; Tue, 2 May 2023 18:54:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7yVbvXfL0DzbmoJ4dt1Rhsek3O/0+PNczKrfY8dd0oDg/yXhIGqz4RBBXRJ4OMlavjJlLA X-Received: by 2002:a17:90a:a502:b0:23f:7666:c8a1 with SMTP id a2-20020a17090aa50200b0023f7666c8a1mr20387014pjq.18.1683078852444; Tue, 02 May 2023 18:54:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683078852; cv=none; d=google.com; s=arc-20160816; b=MggDGpuN0sVfGhkB1tD14enIFRdloSg8O2/SC+MQORaNU4LI9m1eHpLmq7nBHITNgH TvzdHDWqbAxcuwU0/FcY5aoMYuClLE9pthefUEdI4xVuwz9X4CB5iA7fM4djgUhHlqdZ yeQx2eGSpMj6QcsHNkRRrYTuEcEHXIXsKRNTrG/XoYAbbfDhPGuYWPGj+qEcMsKWwg7R 0HBovrCw08Ru+nk9ThTxxeMGlSSVN2TGAM2braO+dPwgdEYUpPSewV3n3SjLgV8fgj0Y 1TOieUaG6n4BmeLXRqtCjMtpcZFnmQyQLafl9+d+Gwh7k5yPWUf9TQU0HKqYQM7SyV7e QmPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=smcp8zMjIvenU/5ttiTpkkwH8ja2gF/kjyxXZVmyP2o=; b=uOHp1W6egWYajmSBonnfs6uUm9bZiEnvspmtaGjsnodQebTllJUNl3zBXk+u/tcC3m IyTu+cAySYU1/BDh4KVp33EMlC1Z76v5+R2KWFwmo3QUWur7akVswfOSetwZPJtKaEXj YRKUtr7exGpcgaFQ0jwQ9sTWC+1SA1otDKkpI4zjI3r7y52Lt4ZCTXNUak6ru8qvTzKS K8NBv6tNUi8CJqPCnOoGzAvsF5yufzUP77V9r8Lz69HUe7UKbaufsWUxdF0qS/20U7Mr tF3Z7E51iadOjBL6wJ+xAX72il7RWiGBd3nwMLGbxMNkiIN/WXjjHmxO0lb64rp34r0v y9+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=OjfScoX4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oc3-20020a17090b1c0300b0023e066d4e2dsi13798272pjb.184.2023.05.02.18.53.47; Tue, 02 May 2023 18:54:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=OjfScoX4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229632AbjECBgX (ORCPT + 99 others); Tue, 2 May 2023 21:36:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjECBgP (ORCPT ); Tue, 2 May 2023 21:36:15 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 797FF273A; Tue, 2 May 2023 18:36:13 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-52c3ffc8d13so1709402a12.2; Tue, 02 May 2023 18:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683077772; x=1685669772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=smcp8zMjIvenU/5ttiTpkkwH8ja2gF/kjyxXZVmyP2o=; b=OjfScoX4IsdshgVkTAAiqStkhaB2dp21ScM4MVEUB5zwEv4pSpSusB6ivbbOac6+Tr 5xbiwomvIJyRKtVNPKwkqT2ORZGpPdoitC0bxtNvglisHxHr1sWP1Y2aFQhF0bRxNEdi 9ud+8QrdtXdhYdqPlsZ52O3MDPe6lOFA34bgjgBlhb8/31xaNbOt8Km/w9QmmDi1Yk/D HZyMSkmx2y183cohr4Bc6Vs9wZyVbQHOlIghjn/bD/iL4XmtCb3BEpGaJs96QV98MAVC /CaJWbvbXETp6+5AAWXpXiz8yti04DeRzMAlvbCdPOuH+K6LYYEzvFBU+mL48cK81qlH ihsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683077772; x=1685669772; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=smcp8zMjIvenU/5ttiTpkkwH8ja2gF/kjyxXZVmyP2o=; b=i6mes8dl0whRCVtsjAHD2Y+LpIHeh6SUMXFJkla8SmOTJ6IJ6lMIrWKAOSpiy/5LIN vCxVIiey3bAP6mqOt2MBH2y2y9wmzKkkyMpB1trAKvm+MIS7Q/ieR7mgqP3HwC1ZGtuM 8GQhjFosC+CzsMcJvWZ+BBrod2hXFP/GxicOmCPi07j+t8CIVAwG96f6rUfwtem8MxwY qfJ++JX84jN+0+h4obSEznWrM8tBBY45bH66CrrurNV+ZbNwCS6axwqDl21cSZy8Ifse PT27xRK4QXoGeVOy1RBGMQCRZvLEp5ggY4IlmlytIsFEdsxFwyewrIM+N2ogvOKIHulC UISw== X-Gm-Message-State: AC+VfDzKPNaBlIiRRM9TXKHvpys8vNphDlaOqQTvsWSqj0GBNjR91b1M ZTifCu25P1QhdCwnTqJPD5HvbiIgNVk= X-Received: by 2002:a17:90a:9204:b0:247:90d8:41fd with SMTP id m4-20020a17090a920400b0024790d841fdmr21084489pjo.26.1683077772331; Tue, 02 May 2023 18:36:12 -0700 (PDT) Received: from localhost (fwdproxy-prn-013.fbsv.net. [2a03:2880:ff:d::face:b00c]) by smtp.gmail.com with ESMTPSA id o15-20020a17090ac08f00b0024b79a69361sm134622pjs.32.2023.05.02.18.36.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 May 2023 18:36:12 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, linux-api@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v13 3/3] selftests: Add selftests for cachestat Date: Tue, 2 May 2023 18:36:08 -0700 Message-Id: <20230503013608.2431726-4-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230503013608.2431726-1-nphamcs@gmail.com> References: <20230503013608.2431726-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764836090441112759?= X-GMAIL-MSGID: =?utf-8?q?1764836090441112759?= Test cachestat on a newly created file, /dev/ files, and /proc/ files. Also test on a shmem file (which can also be tested with huge pages since tmpfs supports huge pages). Signed-off-by: Nhat Pham Acked-by: Johannes Weiner Acked-by: Nhat Pham --- MAINTAINERS | 7 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cachestat/.gitignore | 2 + tools/testing/selftests/cachestat/Makefile | 8 + .../selftests/cachestat/test_cachestat.c | 258 ++++++++++++++++++ 5 files changed, 276 insertions(+) create mode 100644 tools/testing/selftests/cachestat/.gitignore create mode 100644 tools/testing/selftests/cachestat/Makefile create mode 100644 tools/testing/selftests/cachestat/test_cachestat.c diff --git a/MAINTAINERS b/MAINTAINERS index 32772c383ab7..a02946fdd680 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4510,6 +4510,13 @@ S: Supported F: Documentation/filesystems/caching/cachefiles.rst F: fs/cachefiles/ +CACHESTAT: PAGE CACHE STATS FOR A FILE +M: Nhat Pham +M: Johannes Weiner +L: linux-mm@kvack.org +S: Maintained +F: tools/testing/selftests/cachestat/test_cachestat.c + CADENCE MIPI-CSI2 BRIDGES M: Maxime Ripard L: linux-media@vger.kernel.org diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 97dcdaa656f6..5994e56e1214 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -4,6 +4,7 @@ TARGETS += amd-pstate TARGETS += arm64 TARGETS += bpf TARGETS += breakpoints +TARGETS += cachestat TARGETS += capabilities TARGETS += cgroup TARGETS += clone3 diff --git a/tools/testing/selftests/cachestat/.gitignore b/tools/testing/selftests/cachestat/.gitignore new file mode 100644 index 000000000000..d6c30b43a4bb --- /dev/null +++ b/tools/testing/selftests/cachestat/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +test_cachestat diff --git a/tools/testing/selftests/cachestat/Makefile b/tools/testing/selftests/cachestat/Makefile new file mode 100644 index 000000000000..fca73aaa7d14 --- /dev/null +++ b/tools/testing/selftests/cachestat/Makefile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_GEN_PROGS := test_cachestat + +CFLAGS += $(KHDR_INCLUDES) +CFLAGS += -Wall +CFLAGS += -lrt + +include ../lib.mk diff --git a/tools/testing/selftests/cachestat/test_cachestat.c b/tools/testing/selftests/cachestat/test_cachestat.c new file mode 100644 index 000000000000..c3823b809c25 --- /dev/null +++ b/tools/testing/selftests/cachestat/test_cachestat.c @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +static const char * const dev_files[] = { + "/dev/zero", "/dev/null", "/dev/urandom", + "/proc/version", "/proc" +}; +static const int cachestat_nr = 451; + +void print_cachestat(struct cachestat *cs) +{ + ksft_print_msg( + "Using cachestat: Cached: %lu, Dirty: %lu, Writeback: %lu, Evicted: %lu, Recently Evicted: %lu\n", + cs->nr_cache, cs->nr_dirty, cs->nr_writeback, + cs->nr_evicted, cs->nr_recently_evicted); +} + +bool write_exactly(int fd, size_t filesize) +{ + char data[filesize]; + bool ret = true; + int random_fd = open("/dev/urandom", O_RDONLY); + + if (random_fd < 0) { + ksft_print_msg("Unable to access urandom.\n"); + ret = false; + goto out; + } else { + int remained = filesize; + char *cursor = data; + + while (remained) { + ssize_t read_len = read(random_fd, cursor, remained); + + if (read_len <= 0) { + ksft_print_msg("Unable to read from urandom.\n"); + ret = false; + goto close_random_fd; + } + + remained -= read_len; + cursor += read_len; + } + + /* write random data to fd */ + remained = filesize; + cursor = data; + while (remained) { + ssize_t write_len = write(fd, cursor, remained); + + if (write_len <= 0) { + ksft_print_msg("Unable write random data to file.\n"); + ret = false; + goto close_random_fd; + } + + remained -= write_len; + cursor += write_len; + } + } + +close_random_fd: + close(random_fd); +out: + return ret; +} + +/* + * Open/create the file at filename, (optionally) write random data to it + * (exactly num_pages), then test the cachestat syscall on this file. + * + * If test_fsync == true, fsync the file, then check the number of dirty + * pages. + */ +bool test_cachestat(const char *filename, bool write_random, bool create, + bool test_fsync, unsigned long num_pages, int open_flags, + mode_t open_mode) +{ + size_t PS = sysconf(_SC_PAGESIZE); + int filesize = num_pages * PS; + bool ret = true; + long syscall_ret; + struct cachestat cs; + struct cachestat_range cs_range = { 0, filesize }; + + int fd = open(filename, open_flags, open_mode); + + if (fd == -1) { + ksft_print_msg("Unable to create/open file.\n"); + ret = false; + goto out; + } else { + ksft_print_msg("Create/open %s\n", filename); + } + + if (write_random) { + if (!write_exactly(fd, filesize)) { + ksft_print_msg("Unable to access urandom.\n"); + ret = false; + goto out1; + } + } + + syscall_ret = syscall(cachestat_nr, fd, &cs_range, &cs, 0); + + ksft_print_msg("Cachestat call returned %ld\n", syscall_ret); + + if (syscall_ret) { + ksft_print_msg("Cachestat returned non-zero.\n"); + ret = false; + goto out1; + + } else { + print_cachestat(&cs); + + if (write_random) { + if (cs.nr_cache + cs.nr_evicted != num_pages) { + ksft_print_msg( + "Total number of cached and evicted pages is off.\n"); + ret = false; + } + } + } + + if (test_fsync) { + if (fsync(fd)) { + ksft_print_msg("fsync fails.\n"); + ret = false; + } else { + syscall_ret = syscall(cachestat_nr, fd, &cs_range, &cs, 0); + + ksft_print_msg("Cachestat call (after fsync) returned %ld\n", + syscall_ret); + + if (!syscall_ret) { + print_cachestat(&cs); + + if (cs.nr_dirty) { + ret = false; + ksft_print_msg( + "Number of dirty should be zero after fsync.\n"); + } + } else { + ksft_print_msg("Cachestat (after fsync) returned non-zero.\n"); + ret = false; + goto out1; + } + } + } + +out1: + close(fd); + + if (create) + remove(filename); +out: + return ret; +} + +bool test_cachestat_shmem(void) +{ + size_t PS = sysconf(_SC_PAGESIZE); + size_t filesize = PS * 512 * 2; /* 2 2MB huge pages */ + int syscall_ret; + size_t compute_len = PS * 512; + struct cachestat_range cs_range = { PS, compute_len }; + char *filename = "tmpshmcstat"; + struct cachestat cs; + bool ret = true; + unsigned long num_pages = compute_len / PS; + int fd = shm_open(filename, O_CREAT | O_RDWR, 0600); + + if (fd < 0) { + ksft_print_msg("Unable to create shmem file.\n"); + ret = false; + goto out; + } + + if (ftruncate(fd, filesize)) { + ksft_print_msg("Unable to trucate shmem file.\n"); + ret = false; + goto close_fd; + } + + if (!write_exactly(fd, filesize)) { + ksft_print_msg("Unable to write to shmem file.\n"); + ret = false; + goto close_fd; + } + + syscall_ret = syscall(cachestat_nr, fd, &cs_range, &cs, 0); + + if (syscall_ret) { + ksft_print_msg("Cachestat returned non-zero.\n"); + ret = false; + goto close_fd; + } else { + print_cachestat(&cs); + if (cs.nr_cache + cs.nr_evicted != num_pages) { + ksft_print_msg( + "Total number of cached and evicted pages is off.\n"); + ret = false; + } + } + +close_fd: + shm_unlink(filename); +out: + return ret; +} + +int main(void) +{ + int ret = 0; + + for (int i = 0; i < 5; i++) { + const char *dev_filename = dev_files[i]; + + if (test_cachestat(dev_filename, false, false, false, + 4, O_RDONLY, 0400)) + ksft_test_result_pass("cachestat works with %s\n", dev_filename); + else { + ksft_test_result_fail("cachestat fails with %s\n", dev_filename); + ret = 1; + } + } + + if (test_cachestat("tmpfilecachestat", true, true, + true, 4, O_CREAT | O_RDWR, 0400 | 0600)) + ksft_test_result_pass("cachestat works with a normal file\n"); + else { + ksft_test_result_fail("cachestat fails with normal file\n"); + ret = 1; + } + + if (test_cachestat_shmem()) + ksft_test_result_pass("cachestat works with a shmem file\n"); + else { + ksft_test_result_fail("cachestat fails with a shmem file\n"); + ret = 1; + } + + return ret; +}