From patchwork Fri Nov 3 16:03:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 161357 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp1142999vqu; Fri, 3 Nov 2023 09:04:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHAcr/5NmwharUdMwgjNvkySbwi6p/RhXjTiKDD9NTZicz0Ktaz4iSLyYedWTqVzVQRMUju X-Received: by 2002:a17:90a:f998:b0:280:152b:ed7e with SMTP id cq24-20020a17090af99800b00280152bed7emr18217517pjb.18.1699027457297; Fri, 03 Nov 2023 09:04:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699027457; cv=none; d=google.com; s=arc-20160816; b=zMgeMPU1YmL/qurlj1/y0E3agUIVwvEabH4emXIXm/5qCcR5T+utRQ70VljaaPs/Pw s4lTdW/DrD/Vd5llcvgt9Z/M9X9LQProcmkzJL9m1v0YIAwqwi8aahDGJ3EzyzQ78opt bE6NVQSKdH+0LCBU2qsZD9FUdLL4BrLYP2l6Uap1GZdoWU34sZ79lPWw23uLNMmQQo8c LO+ICKwhNBTiJaA6kKsZVHpP0CV1Zgcgm6nEv89eHBHodGKyxSlwoB5i6pD0WDScfBkq oXpv5PBd7FKdNPOGNMPOm1nOcSSXYzd5y3R17B17LsILhzEzaNY7TXcauntQ9ZFS5Y1n lJqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=kakEavv2Bx4kC7ui8vSjgo++eh6M+uQZM7WF3AfKH1U=; fh=bwsiIJU4ZySHCYgdbJrawkDYkx7XsqbTjYxIEnch01E=; b=oIeclnzJFZXTuAyXUzrDLYktrKRv5BfEjibT2/jpIiGmzvjvjebsb5a4M7YKrmiicr eTPvlFjvO1e+sCUXPDMmP02n7EdJJjFTSkNekrOPOKheI7aFaleASDB+gNaWTjPvlyA7 xxGZVLkt9BJ5p0nq87Dw/EOZxF0mL741BgMeDqnhHv/CfDCGfkRxYmj1HsKOe91gpscr UviTa7isL2++7LfHHRtsT/KX9JcXOgLFduqCBPCzxA5drZbJkjsV3JfhMgnBb1Or3SgQ NpFX8iyK+qiRYcAKkmR7+uZnd629gMpbNjB6WvQiweN6Vqe03nWJMHn7g45FetJrSJHt rF2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=KHTIeOvi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id i15-20020a17090adc0f00b0027d0bf1eef1si1770516pjv.98.2023.11.03.09.04.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 09:04:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=KHTIeOvi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 21A6E80260C5; Fri, 3 Nov 2023 09:04:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345483AbjKCQEH (ORCPT + 35 others); Fri, 3 Nov 2023 12:04:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345695AbjKCQD4 (ORCPT ); Fri, 3 Nov 2023 12:03:56 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB07DD75 for ; Fri, 3 Nov 2023 09:03:48 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d9cb4de3bf0so2677694276.0 for ; Fri, 03 Nov 2023 09:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1699027428; x=1699632228; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kakEavv2Bx4kC7ui8vSjgo++eh6M+uQZM7WF3AfKH1U=; b=KHTIeOviOjzGd7htZ/U2VaxaMZg5B6gkjXM8170aY89bNhBhPON1wB1PEHELMqFBGt 0YiCXjcxQ8nI2qtejl/VIjHGPZdL+2Nv55TtEeJuO7e9DMTvWlGWsMVSIHtjOz9Y22XT PSLVKcUHD0ajFFGS8mLQVRpWUkmcW9JI/eyasWtHuMvCqlSzw3pN15jjeO4EK9Bhd9Jd TV0S+I/o7RdCo4XdPxvTrShUmr7XGC+Hsws6Un1M1A3t2rJHD6nR0R+BwJK0X0C5dxBu dZzRqtrrXVGB0/J4tZPncMz70WlhVAuf0Oo04XwdEqPe+4e9MycU8jKbw1pawRpry7pI bArw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699027428; x=1699632228; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kakEavv2Bx4kC7ui8vSjgo++eh6M+uQZM7WF3AfKH1U=; b=X+hq87MjL7w0u3kM/MZCwoeLSP9X781aBUwC+OEMffxCBtUGTmqPpHJuqnNlFBwCBN 44O+Yg6IXxHKFmXmuJXcbJI4cF94NNuJzMhGt7xz3T6UOV8BldDgpbuDtH69MEdOupbW /7LMqts1ChsZ+aYy+LROmAkovdhwkar/Bw1PY9LpshHSKu/B4dDSDsvHqWMtsT4H2pNQ i/Xta7xqFg/skApLTH/OI9b1m2ieOk8S29kh6OIBE8Gkr0wO+RVJ0gH0Eh3+L0Vf9ldP YOatdW5IuSI1N0J0+lO4RBTqP7wZSD1aNrrDo2L8MxZ+TzicmL97n1TttWMxbn1C55bZ dK9g== X-Gm-Message-State: AOJu0YxzScH0b3jjdGmzNoKyrQXbSSjtPPd51QxGCV8t0qbGf2WHqjGK iTrwqvh4dJFIftax7kCbDmyJRL7gH74= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:74c:1f8e:4661:7aaa]) (user=glider job=sendgmr) by 2002:a25:770f:0:b0:da0:73c2:db78 with SMTP id s15-20020a25770f000000b00da073c2db78mr452259ybc.9.1699027428174; Fri, 03 Nov 2023 09:03:48 -0700 (PDT) Date: Fri, 3 Nov 2023 17:03:35 +0100 In-Reply-To: <20231103160335.2464561-1-glider@google.com> Mime-Version: 1.0 References: <20231103160335.2464561-1-glider@google.com> X-Mailer: git-send-email 2.42.0.869.gea05f2083d-goog Message-ID: <20231103160335.2464561-4-glider@google.com> Subject: [PATCH v8 3/3] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, aleksander.lobakin@intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com, alexandru.elisei@arm.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 03 Nov 2023 09:04:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781559414994288894 X-GMAIL-MSGID: 1781559414994288894 Update mteswap.c to perform inline compression of memory tags when possible. If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attemt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space. Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap: # cat /sys/kernel/debug/mteswap/stats 8 bytes: 102496 allocations, 67302 deallocations 128 bytes: 212234 allocations, 178278 deallocations uncompressed tag storage size: 8851200 compressed tag storage size: 4346368 Signed-off-by: Alexander Potapenko --- v8: - adapt to the new compression API, abandon mteswap_{no,}comp.c - move stats collection to mteswap.c v5: - drop a dead variable from _mte_free_saved_tags() in mteswap_comp.c - ensure MTE compression works with arbitrary page sizes - update patch description v4: - minor code simplifications suggested by Andy Shevchenko, added missing header dependencies - changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov) v3: - Addressed comments by Andy Shevchenko in another patch: - fixed includes order - replaced u64 with unsigned long - added MODULE_IMPORT_NS(MTECOMP) --- arch/arm64/mm/mteswap.c | 88 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 84 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc54..0f558942d88b8 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -1,28 +1,48 @@ // SPDX-License-Identifier: GPL-2.0-only +#include #include #include #include #include #include #include +#include +#include "mtecomp.h" + +enum mteswap_counters { + MTESWAP_CTR_INLINE = 0, + MTESWAP_CTR_NOINLINE, + MTESWAP_CTR_SIZE +}; +static atomic_long_t alloc_counters[MTESWAP_CTR_SIZE]; +static atomic_long_t dealloc_counters[MTESWAP_CTR_SIZE]; static DEFINE_XARRAY(mte_pages); void *mte_allocate_tag_storage(void) { - /* tags granule is 16 bytes, 2 tags stored per byte */ - return kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + void *ret; + + ret = kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + if (ret) + atomic_long_inc(&alloc_counters[MTESWAP_CTR_NOINLINE]); + return ret; } void mte_free_tag_storage(char *storage) { - kfree(storage); + if (!mte_is_compressed(storage)) { + kfree(storage); + atomic_long_dec(&alloc_counters[MTESWAP_CTR_NOINLINE]); + } else { + atomic_long_dec(&alloc_counters[MTESWAP_CTR_INLINE]); + } } int mte_save_tags(struct page *page) { - void *tag_storage, *ret; + void *tag_storage, *ret, *compressed; if (!page_mte_tagged(page)) return 0; @@ -32,6 +52,12 @@ int mte_save_tags(struct page *page) return -ENOMEM; mte_save_page_tags(page_address(page), tag_storage); + compressed = mte_compress(tag_storage); + if (compressed) { + mte_free_tag_storage(tag_storage); + tag_storage = (void *)compressed; + atomic_long_inc(&alloc_counters[MTESWAP_CTR_INLINE]); + } /* lookup the swap entry.val from the page */ ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage, @@ -50,13 +76,20 @@ int mte_save_tags(struct page *page) void mte_restore_tags(swp_entry_t entry, struct page *page) { void *tags = xa_load(&mte_pages, entry.val); + void *tag_storage = NULL; if (!tags) return; if (try_page_mte_tagging(page)) { + if (mte_is_compressed(tags)) { + tag_storage = mte_allocate_tag_storage(); + mte_decompress(tags, tag_storage); + tags = tag_storage; + } mte_restore_page_tags(page_address(page), tags); set_page_mte_tagged(page); + mte_free_tag_storage(tag_storage); } } @@ -83,3 +116,50 @@ void mte_invalidate_tags_area(int type) } xa_unlock(&mte_pages); } + +/* DebugFS interface. */ +static int stats_show(struct seq_file *seq, void *v) +{ + unsigned long total_mem_alloc = 0, total_mem_dealloc = 0; + unsigned long total_num_alloc = 0, total_num_dealloc = 0; + unsigned long sizes[2] = { 8, MTE_PAGE_TAG_STORAGE }; + long alloc, dealloc; + unsigned long size; + int i; + + for (i = 0; i < MTESWAP_CTR_SIZE; i++) { + alloc = atomic_long_read(&alloc_counters[i]); + dealloc = atomic_long_read(&dealloc_counters[i]); + total_num_alloc += alloc; + total_num_dealloc += dealloc; + size = sizes[i]; + /* + * Do not count 8-byte buffers towards compressed tag storage + * size. + */ + if (i) { + total_mem_alloc += (size * alloc); + total_mem_dealloc += (size * dealloc); + } + seq_printf(seq, + "%lu bytes: %lu allocations, %lu deallocations\n", + size, alloc, dealloc); + } + seq_printf(seq, "uncompressed tag storage size: %lu\n", + (total_num_alloc - total_num_dealloc) * + MTE_PAGE_TAG_STORAGE); + seq_printf(seq, "compressed tag storage size: %lu\n", + total_mem_alloc - total_mem_dealloc); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(stats); + +static int mteswap_init(void) +{ + struct dentry *mteswap_dir; + + mteswap_dir = debugfs_create_dir("mteswap", NULL); + debugfs_create_file("stats", 0444, mteswap_dir, NULL, &stats_fops); + return 0; +} +module_init(mteswap_init);