From patchwork Tue Sep 12 17:52:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 138356 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp590601vqx; Tue, 12 Sep 2023 11:10:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHgGFoO2TZSUd/2LcB7L/PRp31s+RrGu/0/FZiFqtUaogj0aNi6nnWsSO/khmUbD0MOynV1 X-Received: by 2002:a17:902:9007:b0:1b8:400a:48f2 with SMTP id a7-20020a170902900700b001b8400a48f2mr506979plp.62.1694542226211; Tue, 12 Sep 2023 11:10:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694542226; cv=none; d=google.com; s=arc-20160816; b=XqxiPBqGBpAaFGu0f724UPuOJIWbbA37rff6VshVkQ6L75iR7SosnlhHyzVe1o5wG6 GR69Ba3LPG0L2X7d5OMzXVC3wc2GhVokdWdDjnpW3Ci7wQz+PitVfOVgZ8z1DD059S/u X778JVd5Zj72eMUfQ7kXL2baVw778KhmzAcvTtxidwcdUMog8CowjSgvfn3+votnOUo/ lel/UjpJNjhVD/GdeTuAJtG7oh/n1hoqN8mSLsgOTZaXuIl/fi8K0O2BwHm4eh48u/PK HtfA7Hj/hyn+cfI5S5dfz8TLm5b3spNzwZlOs6DZduByJA+tjxJQoXGOYMxFGDaw8Imw ftfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=4cyzr/0K5Dw6Q79NtyCfj6eo018Xmxirx96tjFXharo=; fh=sS+J4OyOC0EcVLWJpS3mBHGeO+0+dYZJ+ImCUfzsrH4=; b=AQ15Hm/s4DNmMfkSyYI2a1YzUkPKvh2u2zSLrAIgNrKzb0KXUAJLDN25Nj2WZoU/HM gb5vuyP7sJVR7Q2Aho3eVONc6Kf7cuGAfKXtN3GNrO8xdE8YI4TdzVvxPJvjr8fh52tG 9Le+AfcysW0z40j/8J+orAGiIUYW4YWR99rEXAPy6mxmt0yg+PSCbdyyHnDWOg2DUJ72 G0sTJNQsuYWhkiJ8p2w96QULx4/eNZrUbrl0p73eu6jB0lYI8uekxAkCME8JrbTcbRuT HophXfU6nSZKFuJIZfn5mfbhVZp5C1hwfwjad/5vXspW9XoXAd5XSjjKqT/bGJSwb3Fy V//Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id im23-20020a170902bb1700b001b3bd85f54bsi8095854plb.35.2023.09.12.11.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 11:10:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 305348211222; Tue, 12 Sep 2023 10:54:01 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237436AbjILRxm (ORCPT + 36 others); Tue, 12 Sep 2023 13:53:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbjILRxW (ORCPT ); Tue, 12 Sep 2023 13:53:22 -0400 Received: from 66-220-144-179.mail-mxout.facebook.com (66-220-144-179.mail-mxout.facebook.com [66.220.144.179]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FDBF1FE3 for ; Tue, 12 Sep 2023 10:52:53 -0700 (PDT) Received: by devbig1114.prn1.facebook.com (Postfix, from userid 425415) id DF124BCD1629; Tue, 12 Sep 2023 10:52:38 -0700 (PDT) From: Stefan Roesch To: kernel-team@fb.com Cc: shr@devkernel.io, akpm@linux-foundation.org, david@redhat.com, hannes@cmpxchg.org, riel@surriel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 1/4] mm/ksm: add "smart" page scanning mode Date: Tue, 12 Sep 2023 10:52:25 -0700 Message-Id: <20230912175228.952039-2-shr@devkernel.io> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230912175228.952039-1-shr@devkernel.io> References: <20230912175228.952039-1-shr@devkernel.io> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 12 Sep 2023 10:54:01 -0700 (PDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776856309198706670 X-GMAIL-MSGID: 1776856309198706670 This change adds a "smart" page scanning mode for KSM. So far all the candidate pages are continuously scanned to find candidates for de-duplication. There are a considerably number of pages that cannot be de-duplicated. This is costly in terms of CPU. By using smart scanning considerable CPU savings can be achieved. This change takes the history of scanning pages into account and skips the page scanning of certain pages for a while if de-deduplication for this page has not been successful in the past. To do this it introduces two new fields in the ksm_rmap_item structure: age and skip_age. age, is the KSM age and skip_page is the age for how long page scanning of this page is skipped. The age field is incremented each time the page is scanned and the page cannot be de-duplicated. How often a page is skipped is dependent how often de-duplication has been tried so far and the number of skips is currently limited to 8. This value has shown to be effective with different workloads. The feature is currently disable by default and can be enabled with the new smart_scan knob. The feature has shown to be very effective: upt to 25% of the page scans can be eliminated; the pages_to_scan rate can be reduced by 40 - 50% and a similar de-duplication rate can be maintained. Signed-off-by: Stefan Roesch --- mm/ksm.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/mm/ksm.c b/mm/ksm.c index 981af9c72e7a..bfd5087c7d5a 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -56,6 +56,8 @@ #define DO_NUMA(x) do { } while (0) #endif +typedef u8 rmap_age_t; + /** * DOC: Overview * @@ -193,6 +195,8 @@ struct ksm_stable_node { * @node: rb node of this rmap_item in the unstable tree * @head: pointer to stable_node heading this list in the stable tree * @hlist: link into hlist of rmap_items hanging off that stable_node + * @age: number of scan iterations since creation + * @skip_age: skip rmap item until age reaches skip_age */ struct ksm_rmap_item { struct ksm_rmap_item *rmap_list; @@ -212,6 +216,8 @@ struct ksm_rmap_item { struct hlist_node hlist; }; }; + rmap_age_t age; + rmap_age_t skip_age; }; #define SEQNR_MASK 0x0ff /* low bits of unstable tree seqnr */ @@ -281,6 +287,9 @@ static unsigned int zero_checksum __read_mostly; /* Whether to merge empty (zeroed) pages with actual zero pages */ static bool ksm_use_zero_pages __read_mostly; +/* Skip pages that couldn't be de-duplicated previously */ +static bool ksm_smart_scan; + /* The number of zero pages which is placed by KSM */ unsigned long ksm_zero_pages; @@ -2305,6 +2314,45 @@ static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, return rmap_item; } +static unsigned int inc_skip_age(rmap_age_t age) +{ + if (age <= 3) + return 1; + if (age <= 5) + return 2; + if (age <= 8) + return 4; + + return 8; +} + +static bool skip_rmap_item(struct page *page, struct ksm_rmap_item *rmap_item) +{ + rmap_age_t age; + + if (!ksm_smart_scan) + return false; + + if (PageKsm(page)) + return false; + + age = rmap_item->age++; + if (age < 3) + return false; + + if (rmap_item->skip_age == age) { + rmap_item->skip_age = 0; + return false; + } + + if (rmap_item->skip_age == 0) { + rmap_item->skip_age = age + inc_skip_age(age); + remove_rmap_item_from_tree(rmap_item); + } + + return true; +} + static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) { struct mm_struct *mm; @@ -2409,6 +2457,10 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) if (rmap_item) { ksm_scan.rmap_list = &rmap_item->rmap_list; + + if (skip_rmap_item(*page, rmap_item)) + goto next_page; + ksm_scan.address += PAGE_SIZE; } else put_page(*page); @@ -3449,6 +3501,28 @@ static ssize_t full_scans_show(struct kobject *kobj, } KSM_ATTR_RO(full_scans); +static ssize_t smart_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", ksm_smart_scan); +} + +static ssize_t smart_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err; + bool value; + + err = kstrtobool(buf, &value); + if (err) + return -EINVAL; + + ksm_smart_scan = value; + return count; +} +KSM_ATTR(smart_scan); + static struct attribute *ksm_attrs[] = { &sleep_millisecs_attr.attr, &pages_to_scan_attr.attr, @@ -3469,6 +3543,7 @@ static struct attribute *ksm_attrs[] = { &stable_node_chains_prune_millisecs_attr.attr, &use_zero_pages_attr.attr, &general_profit_attr.attr, + &smart_scan_attr.attr, NULL, };