From patchwork Mon Nov 13 10:52:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 164404 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp1120366vqg; Mon, 13 Nov 2023 02:53:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IEVQRedqC3DofN4MjxMOZI/dat64N9w78LcR85JGoCXyS8syQA9CuE3tbQbBx3na7yPLO4U X-Received: by 2002:a17:903:22c6:b0:1cc:e66:372e with SMTP id y6-20020a17090322c600b001cc0e66372emr8321087plg.20.1699872799684; Mon, 13 Nov 2023 02:53:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699872799; cv=none; d=google.com; s=arc-20160816; b=NXCl5hY1H6cQFXp5mt3r+OXwLEX8q6W0YYpDLDx4uCllqnHQTaHOQRtiYK0OcVz5JZ /hNiv4VQyEPNF5FnqZDX/gYYU/J5Q+6yEe5KSlQ2BnNIxmeenROkOahXJtXagLhb5hK0 fsbbwr4/fiMqN/zt0v0AfSHyn6CgB82N/a2aooLnGsqaBL9EatrC4dIa1QaQxrfj2T7e L/N+CC8F/JefsFO8Aq9uovKHhsqu1eH12pM7wbFLUecE7FEMY1JmwA4S8No/bVvn2Ls0 CmADc9pFGAvefL6Xevw8cL5xL2vNawJsjZjKWIOElOCNw87uP8gj1xqXNzmNBe41MrFe A5KA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Zy+PlU+FeLCVAtPuXK15UDcGql8UQVRw/kwgKDnnM+M=; fh=bwsiIJU4ZySHCYgdbJrawkDYkx7XsqbTjYxIEnch01E=; b=HFN8fHyEfHhUaRJ8pKmn1x9uvLpxLBW0SFVAGu0HCZbSIDPtJ8KJO08ezlJ6WO0Rs4 4zHM0Mfix8wQGwQvaHqdgbAMsbLATjWkSscFCuqbP+yT1NNIj2AnoV5KIQIGj0j3zeVL XeqY/WTBaaAS+Vl1QuEq8sJPO+h2c4dpDy5xAo0PyNUctNCdo2O65KR6vmFoQ8EKe13P 0UAqFgu5CRpflG3FlCX0dW6w+jw+DJGsBh3cYCbe3gTcJS+3newjufoPwkKf4p+6ZDOJ oNQa/aSgmpwz07/F43C8rBUZU37RlXZBS77X3ujAm8NE3izRUn3auud8IkR7ZCRudAvK 2+QQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jcXleIbK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id o16-20020a170902d4d000b001c3411c9b83si5754043plg.454.2023.11.13.02.53.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Nov 2023 02:53:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jcXleIbK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id BF253804754A; Mon, 13 Nov 2023 02:53:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230029AbjKMKxD (ORCPT + 29 others); Mon, 13 Nov 2023 05:53:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229470AbjKMKwt (ORCPT ); Mon, 13 Nov 2023 05:52:49 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CFDD10D3 for ; Mon, 13 Nov 2023 02:52:46 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-dae71322ed4so5177375276.1 for ; Mon, 13 Nov 2023 02:52:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1699872765; x=1700477565; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Zy+PlU+FeLCVAtPuXK15UDcGql8UQVRw/kwgKDnnM+M=; b=jcXleIbKF8S2B4SR0ReqNqFTWLAyAr6Vp+FcbJIZPwfNWTEP9W19erEHYA5h9hVE4j 07N9i77xyKYsEB/sIJYsii5BSb+aHV8eJlr8ZW0HvxI73J4id7q40mkP9e60FHdkWPfd PVXJbo2J1PJcPcgZWhWdMENyL8g00IhFENV+o0PvMihBJVrA4W1LwVLM51YdeZ6hh9zA LypJHGaP139EFoL9cNvsS9r8Zv4u891zjw7L3ETaFAKRRNAE7x7EUCctn3N8ik4ZIUm8 vyMts8MvZYMTu0Ah0W42ux4A9F03kSCPtLRgAN14dmWnEAjwMO1brWNMRKOg8uBeQ6I9 CoSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699872765; x=1700477565; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zy+PlU+FeLCVAtPuXK15UDcGql8UQVRw/kwgKDnnM+M=; b=ZoHnvk9/QgJ83Hi447/et47lZYoYGWSOOqemz30bE582gJsIPaaB1bOtfQ4coWU6Ug F3KFJQMCP4aaR7SDNnK4vN0XJwzVZEkS1193VoGuNmz7lJ24QwJe98jd9tRoqtSkCkDb epV71J2i4O9CRSV5eYn1XoA9jzBJa6dxeSRhlnpwwS8g+iDSqG7ogPRKwDrg0lTVb1/g hU1jrYvOZGfdMRm+bfLIV5sIjk8QE6jldYe+Y3q5//0j6QrsKNyU/w/Jvf9WsMFTjwJG woy6iQ+1H80EWW/ZsrhkFyYTwgoQ6w7GsXxp5R5xTnTtHvmvXDorqNFkH+JZdhXimpwb z0Vg== X-Gm-Message-State: AOJu0YzNX09Ie7PpbCs1zVJj1aa5iMfii9DgM5NIDq4TT2KGrz7eRqDg p5oW4dOPE1F0C/yfSOz1S01gyWdghxY= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:be07:845d:4221:b15e]) (user=glider job=sendgmr) by 2002:a25:2e4b:0:b0:dae:e8dc:c026 with SMTP id b11-20020a252e4b000000b00daee8dcc026mr153989ybn.13.1699872765517; Mon, 13 Nov 2023 02:52:45 -0800 (PST) Date: Mon, 13 Nov 2023 11:52:32 +0100 In-Reply-To: <20231113105234.32058-1-glider@google.com> Mime-Version: 1.0 References: <20231113105234.32058-1-glider@google.com> X-Mailer: git-send-email 2.42.0.869.gea05f2083d-goog Message-ID: <20231113105234.32058-4-glider@google.com> Subject: [PATCH v9 3/4] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, aleksander.lobakin@intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com, alexandru.elisei@arm.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 13 Nov 2023 02:53:18 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782445820793238620 X-GMAIL-MSGID: 1782445820793238620 Update mteswap.c to perform inline compression of memory tags when possible. If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attempt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space. Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap: # cat /sys/kernel/debug/mteswap/stats 8 bytes: 102496 allocations, 67302 deallocations 128 bytes: 212234 allocations, 178278 deallocations uncompressed tag storage size: 8851200 compressed tag storage size: 4346368 (statistics collection is introduced in the following patch) Signed-off-by: Alexander Potapenko Reviewed-by: Catalin Marinas --- v9: - as requested by Yury Norov, split off statistics collection into a separate patch - minor fixes v8: - adapt to the new compression API, abandon mteswap_{no,}comp.c - move stats collection to mteswap.c v5: - drop a dead variable from _mte_free_saved_tags() in mteswap_comp.c - ensure MTE compression works with arbitrary page sizes - update patch description v4: - minor code simplifications suggested by Andy Shevchenko, added missing header dependencies - changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov) v3: - Addressed comments by Andy Shevchenko in another patch: - fixed includes order - replaced u64 with unsigned long - added MODULE_IMPORT_NS(MTECOMP) --- arch/arm64/mm/mteswap.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc54..70f5c8ecd640d 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -6,6 +6,8 @@ #include #include #include +#include +#include "mtecomp.h" static DEFINE_XARRAY(mte_pages); @@ -17,12 +19,13 @@ void *mte_allocate_tag_storage(void) void mte_free_tag_storage(char *storage) { - kfree(storage); + if (!mte_is_compressed(storage)) + kfree(storage); } int mte_save_tags(struct page *page) { - void *tag_storage, *ret; + void *tag_storage, *compressed_storage, *ret; if (!page_mte_tagged(page)) return 0; @@ -32,6 +35,11 @@ int mte_save_tags(struct page *page) return -ENOMEM; mte_save_page_tags(page_address(page), tag_storage); + compressed_storage = mte_compress(tag_storage); + if (compressed_storage) { + mte_free_tag_storage(tag_storage); + tag_storage = compressed_storage; + } /* lookup the swap entry.val from the page */ ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage, @@ -50,13 +58,20 @@ int mte_save_tags(struct page *page) void mte_restore_tags(swp_entry_t entry, struct page *page) { void *tags = xa_load(&mte_pages, entry.val); + void *tag_storage = NULL; if (!tags) return; if (try_page_mte_tagging(page)) { + if (mte_is_compressed(tags)) { + tag_storage = mte_allocate_tag_storage(); + mte_decompress(tags, tag_storage); + tags = tag_storage; + } mte_restore_page_tags(page_address(page), tags); set_page_mte_tagged(page); + mte_free_tag_storage(tag_storage); } }