From patchwork Tue Jul 11 14:33:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 118592 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp535286vqm; Tue, 11 Jul 2023 07:51:48 -0700 (PDT) X-Google-Smtp-Source: APBJJlE382p5EUn2kSmYoTgz0muzeafTJzzRxrK99MgGhazdGQ2nub0UTOGocDPvrgNs4mULGwxf X-Received: by 2002:a17:902:758b:b0:1b8:5aa2:d17a with SMTP id j11-20020a170902758b00b001b85aa2d17amr12028332pll.7.1689087107898; Tue, 11 Jul 2023 07:51:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689087107; cv=none; d=google.com; s=arc-20160816; b=qUoH5qU0Eavc9onZxisnLbKa+/1T5YZ4eSc33wg/IUTuVsrHUt5QlgXa0CiOgLRjNC uKoni5AQqkRUHxa2oBg+EP54y9zmfAjwJV0aC8POnDUPY3fodG9ZH4rH4HdTkc3SFqfN nFLKIizEnQ1KzMlfAe8i4w+FIa8qYuOcL2k5eLH1iJCYK4V6nWACLxKbTYb0Jr1xKiaO AZJSVRUEbsLDg22qUKe+l4emK1qvlT73vjlLE838zZ1Wssrkf9N1AtaM2qXczyDqOFvY u1aXZqUUSZZVuJzXcwg/7+0a+v4QzVMvrLB8m75QHNPIbsWTZaYaK0Xw8qjyZUKVQy2p WU2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; fh=iAJec5y5uW60OtAwJ48IXZG7gbwwd7b6kg4lfH240Po=; b=hwmL4ORcDWCxhayctWOorVEsYI0uaP3paTDNwIqZFWLBpJIBz41IUxaizFAAZdJodD C4HRHRlIKE5FjtVJUZw/TN2asz2dxJIzjAkv18dYg9ZReVayCDx0A/+Gm/Gk8hUmgikJ j50ICvQ8t1058RUrvVfNL7bEEuXCeG8FsjTcuo9QC80+4inEb6AeGiTZ9XbnwdQadX4y BVo9Os7nEQk/M89Sn/9KnpTz0fjVD5AVyHQyvsIgn5ECQOsfXBwThud2DXs4jZs0PUHr KtNyQFLOyD9GYsnbStu/XefBDMEO+MxRqgooB8prGdzqtgJI8Hfwgg6rT/XjCHMuB/o2 bIuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=iXh0XHR0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j13-20020a170903024d00b001b9ea592e75si1484557plh.265.2023.07.11.07.51.34; Tue, 11 Jul 2023 07:51:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=iXh0XHR0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233603AbjGKOen (ORCPT + 99 others); Tue, 11 Jul 2023 10:34:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233626AbjGKOe2 (ORCPT ); Tue, 11 Jul 2023 10:34:28 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12B9F1BCD for ; Tue, 11 Jul 2023 07:34:09 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id a640c23a62f3a-98df34aa83aso799947766b.1 for ; Tue, 11 Jul 2023 07:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689086048; x=1691678048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=iXh0XHR0rIGogedmzgelpLOrRb0uIzpw8b9s1nFY8Nb6/qdqQKgAhgySorTb4eXpEX trkwn7CjRdl9LZ3qFzgWwhfwnKIr2mEP82qOfkac24bPvfdLP0bgW5XO+eYDfwAWSeG0 G54d2KUpEVu/yW50J26RCv4jyu2/MU28uXVp06zOyK6gMOZJYzyNyhzUbhAyT65fSoog hB9wUuJd5c3gUEzs7IVlS8Z/hPscKxNwwLMh1ilhVTBAq4wLucqv06Uk2GX4YcdLhzRx HHMBxA4OWKMNLVqPEqoqRlWqd4tqwj9mhWiWw3emVbCItRbvukjzLC/Cy6ia+3qUH7K8 +fzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689086048; x=1691678048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j5KQ14CSTc8NpsFet5DkO7+Eg5yvGp+eL0uzYXahxFI=; b=E/P52C5rJKu40ZwdODvDkVCJuWB0kQY6+5NytnYvMm6Nf5PeISpyi89cOqigWPk20F 9QLWbhjqBD7gc8OxOvgTQwoUHO5mp3at0PLgatXIJG4w2vZ3KXLhSUuZpZCLpQJOETlv JW/PJ37WSMB8JN5E3qOx5kmrmp3sJJHQQrC72bIMAKS0w51K7S2gLt/a2RTQT5KYQDvk +gJY08vdJB0IVrOqEDHBmEB6q0FJIcQZkli4i+orbVtA/eL2bKzlLpXW1xlQk+4wrUW/ 0OZ/0ououJz0CGocYhJNJS/DmOIVYkluNi+4Wj+VqP3S7dPLlpm7O8LcCztmsBtzQzAe tShQ== X-Gm-Message-State: ABy/qLauxQoTCYmeUMkvdFUxQeLzTYp3ImJE0M2C/7ImGFGG1rX9QICy kXsSsfb/9vXxURXPKp2EHMWNTk69CDo= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:564d:3aaa:6b5f:4419]) (user=glider job=sendgmr) by 2002:a17:907:75f4:b0:991:b9b5:4027 with SMTP id jz20-20020a17090775f400b00991b9b54027mr87513ejc.5.1689086048546; Tue, 11 Jul 2023 07:34:08 -0700 (PDT) Date: Tue, 11 Jul 2023 16:33:35 +0200 In-Reply-To: <20230711143337.3086664-1-glider@google.com> Mime-Version: 1.0 References: <20230711143337.3086664-1-glider@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230711143337.3086664-9-glider@google.com> Subject: [PATCH 4/5] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, yury.norov@gmail.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771135683719192956 X-GMAIL-MSGID: 1771136202936112199 Define the internal mteswap.h interface: - _mte_alloc_and_save_tags() - _mte_free_saved_tags() - _mte_restore_tags() , that encapsulates saving tags for a struct page (together with memory allocation), restoring tags, and deleting the storage allocated for them. These functions accept opaque pointers, which may point to 128-byte tag buffers, as well as smaller buffers containing compressed tags, or have compressed tags stored directly in them. The existing code from mteswap.c operating with uncompressed tags is split away into mteswap_nocomp.c, and the newly introduced mteswap_comp.c provides compression with the EA0 algorithm. The latter implementation is picked if CONFIG_ARM64_MTE_COMP=y. Soon after booting Android, tag compression saves ~2.5x memory previously spent by mteswap.c on tag allocations. With the growing uptime, the savings reach 20x and even more. Signed-off-by: Alexander Potapenko --- arch/arm64/mm/Makefile | 5 ++++ arch/arm64/mm/mteswap.c | 20 ++++++-------- arch/arm64/mm/mteswap.h | 12 ++++++++ arch/arm64/mm/mteswap_comp.c | 50 ++++++++++++++++++++++++++++++++++ arch/arm64/mm/mteswap_nocomp.c | 37 +++++++++++++++++++++++++ 5 files changed, 113 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 170dc62b010b9..46a798e2b67cb 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,11 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) += mtecomp.o +ifdef CONFIG_ARM64_MTE_COMP +obj-$(CONFIG_ARM64_MTE) += mteswap_comp.o +else +obj-$(CONFIG_ARM64_MTE) += mteswap_nocomp.o +endif obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) += test_mtecomp.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1b..7e8ee43e0d389 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,6 +7,8 @@ #include #include +#include "mteswap.h" + static DEFINE_XARRAY(mte_pages); void *mte_allocate_tag_storage(void) @@ -27,20 +29,19 @@ int mte_save_tags(struct page *page) if (!page_mte_tagged(page)) return 0; - tag_storage = mte_allocate_tag_storage(); + tag_storage = _mte_alloc_and_save_tags(page); + pr_err("HERE: %s:%d\n", __FILE__, __LINE__); // TODO if (!tag_storage) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); - /* page_private contains the swap entry.val set in do_swap_page */ ret = xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + _mte_free_saved_tags(tag_storage); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + _mte_free_saved_tags(ret); } return 0; @@ -53,10 +54,7 @@ void mte_restore_tags(swp_entry_t entry, struct page *page) if (!tags) return; - if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); - set_page_mte_tagged(page); - } + _mte_restore_tags(tags, page); } void mte_invalidate_tags(int type, pgoff_t offset) @@ -64,7 +62,7 @@ void mte_invalidate_tags(int type, pgoff_t offset) swp_entry_t entry = swp_entry(type, offset); void *tags = xa_erase(&mte_pages, entry.val); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } void mte_invalidate_tags_area(int type) @@ -78,7 +76,7 @@ void mte_invalidate_tags_area(int type) xa_lock(&mte_pages); xas_for_each(&xa_state, tags, last_entry.val - 1) { __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } xa_unlock(&mte_pages); } diff --git a/arch/arm64/mm/mteswap.h b/arch/arm64/mm/mteswap.h new file mode 100644 index 0000000000000..bf25f2b3e75a4 --- /dev/null +++ b/arch/arm64/mm/mteswap.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_ARM64_MM_MTESWAP_H_ +#define ARCH_ARM64_MM_MTESWAP_H_ + +#include + +void *_mte_alloc_and_save_tags(struct page *page); +void _mte_free_saved_tags(void *tags); +void _mte_restore_tags(void *tags, struct page *page); + +#endif // ARCH_ARM64_MM_MTESWAP_H_ diff --git a/arch/arm64/mm/mteswap_comp.c b/arch/arm64/mm/mteswap_comp.c new file mode 100644 index 0000000000000..bc2591f8d4f35 --- /dev/null +++ b/arch/arm64/mm/mteswap_comp.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management with EA0 compression. */ + +#include +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + u8 tags[128]; + u64 handle; + + mte_save_page_tags(page_address(page), tags); + handle = ea0_compress(tags); + return xa_mk_value(handle); +} + +void _mte_free_saved_tags(void *storage) +{ + unsigned long handle = xa_to_value(storage); + int size; + + if (!handle) + return; + size = ea0_storage_size(handle); + ea0_release_handle(handle); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + u64 handle = xa_to_value(tags); + u8 tags_decomp[128]; + + if (!handle) + return; + + if (try_page_mte_tagging(page)) { + if (!ea0_decompress(handle, tags_decomp)) + return; + mte_restore_page_tags(page_address(page), tags_decomp); + set_page_mte_tagged(page); + } +} diff --git a/arch/arm64/mm/mteswap_nocomp.c b/arch/arm64/mm/mteswap_nocomp.c new file mode 100644 index 0000000000000..efcdac88b342d --- /dev/null +++ b/arch/arm64/mm/mteswap_nocomp.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management without compression support. */ + +#include +#include +#include +#include +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + void *storage; + + storage = mte_allocate_tag_storage(); + if (!storage) + return NULL; + + mte_save_page_tags(page_address(page), storage); + return storage; +} + +void _mte_free_saved_tags(void *storage) +{ + mte_free_tag_storage(storage); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + if (try_page_mte_tagging(page)) { + mte_restore_page_tags(page_address(page), tags); + set_page_mte_tagged(page); + } +}