From patchwork Thu Jul 13 12:57:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 119862 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp1807962vqm; Thu, 13 Jul 2023 06:04:20 -0700 (PDT) X-Google-Smtp-Source: APBJJlHkKlAVCa2T4zZM9gunwAysDsF9Aoj6vqkpw7PmA1Ny1Sx78TV4yQBd7/OvgoGDkLnqdDK7 X-Received: by 2002:a05:6358:99a5:b0:129:c6d6:ce40 with SMTP id j37-20020a05635899a500b00129c6d6ce40mr2569512rwb.15.1689253459946; Thu, 13 Jul 2023 06:04:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689253459; cv=none; d=google.com; s=arc-20160816; b=R59wcyodqYPLHD2xQuI552FavgkQsGQET9oL5duCl3UTFU2uZxdaMmnLToYoXI2pFw kQzYBxlcJ897jIIZpeeewkLL1ZxRjpnN1VbQ0kEoqFuFDXZ7IngV52Adpeq9tO5tn5nm mSKwQYgM/QWpnCcs9CB6N9WtvdB4HnJenrJMHhDW9ztQoMaSIsSjYY12L51i/2OfWBaX VHYZp9lJFnrtqC0USnxq/hUbZbc9T3STwzh+maCffCG/TJ1lilEYh8tXxtIO5rPE0jIn a1vueYTpUNzW8CkCX1E7mFUxC9PIXORco19R91RdNKLTfcyLD9h9lrUWPO2v+8yKFv4O PLvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=GOfdDj4t/4l0qN/MzMyGSYQ+xos/zeOUfaRubiCXYXI=; fh=w2imRfrBx7xgrqExMhNQwXd/bCFo2iqBwf34S5/AxrI=; b=wGNo613j4cMypcaz9JdEU6EXPRUP+63PvP7Nl7lqp/vfQ9tD15rwjfihoZf4cvSpMh Oxync+xIon7JW5tKn8D4TREkFotxv3adnTLQEsYjOoySgrvAF3E8AT1hISw3Wlo0tJ0i W3Lr1pbZ8nTn8jVkAy5gcYX7djU4SoRBix13QK+rxLzyGMGmgwpGe0tvcgtrXWApaQo6 iVGhG2UqcKIGTfX/7RQ/7ie+agTesmgLclSXNWFs40blT2AdyZktct8vj2KifxNT2+t5 7SdCJTePnsVn9M0geD4Xnwmjazwu+bXa6RSucRe2glYn/RSrLyTeQxBBINhYEbI0EFMR MAJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=fkNAaakh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g4-20020a635204000000b0055b5f13dd64si5154573pgb.92.2023.07.13.06.04.04; Thu, 13 Jul 2023 06:04:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=fkNAaakh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234417AbjGMM6H (ORCPT + 99 others); Thu, 13 Jul 2023 08:58:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234891AbjGMM5p (ORCPT ); Thu, 13 Jul 2023 08:57:45 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FCE42702 for ; Thu, 13 Jul 2023 05:57:41 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-579e9b95b86so5507107b3.1 for ; Thu, 13 Jul 2023 05:57:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689253060; x=1691845060; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GOfdDj4t/4l0qN/MzMyGSYQ+xos/zeOUfaRubiCXYXI=; b=fkNAaakhJhj14joU9vjShVN1J78/4ao4j4r/p0M09hqWWrtvYQsGmFffwTbIc+z21k OzoCWRoZ9KXixeBMf8N4Ri/coa/+8/xhR9EKxTNP8sfg3qpJat0FAxgDfcso3YLSRxuh tY3+AZS3TUi6H9CqRMIO6SNQKVGT6MR1mFawBy7pnitaqt3ZGBxlgndqmv2HJtMKJA43 U3PMN16S19MICAm06IGvBgzUo76/EVqARHIYt3KZurXU3HFFjt3/XW6jXM66vj5+t1kA 5Fj1MEGzbzeQH4Mmxx1jNONJ0i9fxqfaD+AdEi66+pcUHkQ3G3yvhO7pNJTb2WABE3b5 KE4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689253060; x=1691845060; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GOfdDj4t/4l0qN/MzMyGSYQ+xos/zeOUfaRubiCXYXI=; b=QsKjDzfRn9yzjcgStmpJJ65+l4S4uYgZV2OuIGQW8kL6rQB/BnmHvPHX+/VidsPD9O LxyvsJla2oCDNtdHc7unkc6zONId2wk4/y89fqRepaOTxM4CzzM4MNG+6Sf7gSDz8V2a DRqhtfB/Azi58ZxPKuVybetbW8Hj5roOE4+3AjFKQP7m+r2i3c0Dpdb0k+Hf/K7g5K8t ejnYffAyY8NY9UVNlvISNyGjooTApw4SO7SVxyo9piksH11JTQ8nNxHHUBKLmrJbxcJd /HSAkzuU4qcwtoYLpCIhU/sLUaSEEv4v1z2B7lNvR0b2q+cGZGavKAgdRo2Ufg3wK6K4 Kbjw== X-Gm-Message-State: ABy/qLY9IQOeGJd6PH7qRcIoBPsOrt3ljQ/8r3uhLBf2moeJy6CSvr+o Fouzi3+32DFdeBBERN7R/SBaxclkDAw= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:7a88:66b7:31e5:7d85]) (user=glider job=sendgmr) by 2002:a81:8b48:0:b0:570:b1:ca37 with SMTP id e8-20020a818b48000000b0057000b1ca37mr13111ywk.5.1689253060736; Thu, 13 Jul 2023 05:57:40 -0700 (PDT) Date: Thu, 13 Jul 2023 14:57:05 +0200 In-Reply-To: <20230713125706.2884502-1-glider@google.com> Mime-Version: 1.0 References: <20230713125706.2884502-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230713125706.2884502-6-glider@google.com> Subject: [v2 5/5] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771310635867318589 X-GMAIL-MSGID: 1771310635867318589 Define the internal mteswap.h interface: - _mte_alloc_and_save_tags() - _mte_free_saved_tags() - _mte_restore_tags() , that encapsulates saving tags for a struct page (together with memory allocation), restoring tags, and deleting the storage allocated for them. These functions accept opaque pointers, which may point to 128-byte tag buffers, as well as smaller buffers containing compressed tags, or have compressed tags stored directly in them. The existing code from mteswap.c operating with uncompressed tags is split away into mteswap_nocomp.c, and the newly introduced mteswap_comp.c provides compression with the EA0 algorithm. The latter implementation is picked if CONFIG_ARM64_MTE_COMP=y. Soon after booting Android, tag compression saves ~2.5x memory previously spent by mteswap.c on tag allocations. With the growing uptime, the savings reach 20x and even more. Signed-off-by: Alexander Potapenko --- arch/arm64/mm/Makefile | 5 ++++ arch/arm64/mm/mteswap.c | 19 ++++++------- arch/arm64/mm/mteswap.h | 12 ++++++++ arch/arm64/mm/mteswap_comp.c | 50 ++++++++++++++++++++++++++++++++++ arch/arm64/mm/mteswap_nocomp.c | 37 +++++++++++++++++++++++++ 5 files changed, 112 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 170dc62b010b9..46a798e2b67cb 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,11 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) += mtecomp.o +ifdef CONFIG_ARM64_MTE_COMP +obj-$(CONFIG_ARM64_MTE) += mteswap_comp.o +else +obj-$(CONFIG_ARM64_MTE) += mteswap_nocomp.o +endif obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) += test_mtecomp.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1b..e072c1209fbb6 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,6 +7,8 @@ #include #include +#include "mteswap.h" + static DEFINE_XARRAY(mte_pages); void *mte_allocate_tag_storage(void) @@ -27,20 +29,18 @@ int mte_save_tags(struct page *page) if (!page_mte_tagged(page)) return 0; - tag_storage = mte_allocate_tag_storage(); + tag_storage = _mte_alloc_and_save_tags(page); if (!tag_storage) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); - /* page_private contains the swap entry.val set in do_swap_page */ ret = xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + _mte_free_saved_tags(tag_storage); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + _mte_free_saved_tags(ret); } return 0; @@ -53,10 +53,7 @@ void mte_restore_tags(swp_entry_t entry, struct page *page) if (!tags) return; - if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); - set_page_mte_tagged(page); - } + _mte_restore_tags(tags, page); } void mte_invalidate_tags(int type, pgoff_t offset) @@ -64,7 +61,7 @@ void mte_invalidate_tags(int type, pgoff_t offset) swp_entry_t entry = swp_entry(type, offset); void *tags = xa_erase(&mte_pages, entry.val); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } void mte_invalidate_tags_area(int type) @@ -78,7 +75,7 @@ void mte_invalidate_tags_area(int type) xa_lock(&mte_pages); xas_for_each(&xa_state, tags, last_entry.val - 1) { __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } xa_unlock(&mte_pages); } diff --git a/arch/arm64/mm/mteswap.h b/arch/arm64/mm/mteswap.h new file mode 100644 index 0000000000000..bf25f2b3e75a4 --- /dev/null +++ b/arch/arm64/mm/mteswap.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_ARM64_MM_MTESWAP_H_ +#define ARCH_ARM64_MM_MTESWAP_H_ + +#include + +void *_mte_alloc_and_save_tags(struct page *page); +void _mte_free_saved_tags(void *tags); +void _mte_restore_tags(void *tags, struct page *page); + +#endif // ARCH_ARM64_MM_MTESWAP_H_ diff --git a/arch/arm64/mm/mteswap_comp.c b/arch/arm64/mm/mteswap_comp.c new file mode 100644 index 0000000000000..bc2591f8d4f35 --- /dev/null +++ b/arch/arm64/mm/mteswap_comp.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management with EA0 compression. */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + u8 tags[128]; + u64 handle; + + mte_save_page_tags(page_address(page), tags); + handle = ea0_compress(tags); + return xa_mk_value(handle); +} + +void _mte_free_saved_tags(void *storage) +{ + unsigned long handle = xa_to_value(storage); + int size; + + if (!handle) + return; + size = ea0_storage_size(handle); + ea0_release_handle(handle); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + u64 handle = xa_to_value(tags); + u8 tags_decomp[128]; + + if (!handle) + return; + + if (try_page_mte_tagging(page)) { + if (!ea0_decompress(handle, tags_decomp)) + return; + mte_restore_page_tags(page_address(page), tags_decomp); + set_page_mte_tagged(page); + } +} diff --git a/arch/arm64/mm/mteswap_nocomp.c b/arch/arm64/mm/mteswap_nocomp.c new file mode 100644 index 0000000000000..efcdac88b342d --- /dev/null +++ b/arch/arm64/mm/mteswap_nocomp.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management without compression support. */ + +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + void *storage; + + storage = mte_allocate_tag_storage(); + if (!storage) + return NULL; + + mte_save_page_tags(page_address(page), storage); + return storage; +} + +void _mte_free_saved_tags(void *storage) +{ + mte_free_tag_storage(storage); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + if (try_page_mte_tagging(page)) { + mte_restore_page_tags(page_address(page), tags); + set_page_mte_tagged(page); + } +}