From patchwork Wed Oct 11 17:28:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 15231 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp700391vqb; Wed, 11 Oct 2023 10:29:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEvIupqRaqClBh7981lubBbfcNSK3H/1O3cNpJqPRP161ns0GQA6c3WXaB7TNQAw4XnoGIq X-Received: by 2002:a05:6871:8788:b0:1e9:8a7e:5893 with SMTP id td8-20020a056871878800b001e98a7e5893mr3084816oab.5.1697045346717; Wed, 11 Oct 2023 10:29:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697045346; cv=none; d=google.com; s=arc-20160816; b=LgBoBKTRmwXPsHWVEV8cne/7dmfepZ/g6PXQ6pvxN3qDKK1fJYJNK39WzyWz0NcoQF B59ZEn6MGfRfZmHDfGsN/u8ThRnBozRkSTCwExbmqVflLEqZ+c42c2eepEdnia0iviYn fG9GKyhaBrf2mI32S6LNtvJ2/8JQM/JIVSe6WmtB8WJ2Fx4Plylh71gtiXtgtCN1MHWK 80x9xHNkRTNCUXP/UMRV+bgrwtKcJ73cnM10OvP/S+uTOxfN8rc+UmZB7DH87woVDC1k yBU2wPl8me9MGANG/kw8/cOE5DKRpDriq6jtuuFAoRrEfLEVeeVkn7dFE5+O8/gWzA/x 7s5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=LwuwjmnVQtXjn+X5MfT/OJgDNhPxUAgxvbWJ6WO5//o=; fh=bwsiIJU4ZySHCYgdbJrawkDYkx7XsqbTjYxIEnch01E=; b=dr8+DuImztMgdMFbseSBJoyAVQOeDpcsMqyTi67FDCWx3TK6R6NPDnWRYYanx/d+vL A0NXezDuWou6N3sE1m9a9/1ZFnzCQBo9se6F6I80hqPUSalO/CtinMKYHZ2unRnq32Iu EUtclafhGklTSjQ5+pWeC1S9VaOzWbAysUy8l6bfaTfj7AoK3Wj4AOG5S6rEv9H8K6Wq FkK7lBozYUMaC38AR9U/vK6KsCPPG2mpF3KsPyq7V7tYtTYlv23vmSWlxDh46bh40J2E kuk50ttsCA6YRw68NdKUggiJEQiei15z6XIOFApDmyUb6sV7TPkgsmZ5Gq7Lj8FcSGL6 v6UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=IXT8uKcl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id by22-20020a056a02059600b00577549e67b9si255637pgb.589.2023.10.11.10.29.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Oct 2023 10:29:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=IXT8uKcl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id B9A858111761; Wed, 11 Oct 2023 10:28:59 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231924AbjJKR2p (ORCPT + 18 others); Wed, 11 Oct 2023 13:28:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230012AbjJKR2o (ORCPT ); Wed, 11 Oct 2023 13:28:44 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5258698 for ; Wed, 11 Oct 2023 10:28:42 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5a7be940fe1so1099547b3.2 for ; Wed, 11 Oct 2023 10:28:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697045321; x=1697650121; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=LwuwjmnVQtXjn+X5MfT/OJgDNhPxUAgxvbWJ6WO5//o=; b=IXT8uKclxUzwcWT+thJ/UvkoT1GFIWMNl7FNEJIHHUH7bOrDotgNyOcEg9/dt5duHN JVh4l18b32Su5y71VGUwP2lN1G1949xhHN7FxCqSiC6Kh4h1FqTQP1vx063qdv/uc16k XUIJmDYmBPXEkwRH0F90h8Pi96Nc5UYPYMpcNR/4eVSKRm/w2SrBh0kkb9keNcW8L4Zh DI5NnH8MQZxa95v9c3G+pBKbZA8AYUR8p8/KY/yEmor9uz71NkZ+nJo7W1uAzhLQypNH 3dHRIM0ZNzGEVGAYBChpV9QIVrhWeSOcnRl5PcpYpZ0rXaHPFJU+4AdI22P+OFqVBHsT ooag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697045321; x=1697650121; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LwuwjmnVQtXjn+X5MfT/OJgDNhPxUAgxvbWJ6WO5//o=; b=vczjecMVi5GxjSnQQIX4xlw1XS2wenvesquxGadh/I8CsfJ9nv774FijXkpqTMk5lb JtPMT7/oWIJBwr/bFBaz3GgDqPrPnVsKLoZcC2TSXyIZHpsmcwEWBKlOPtFtvRcEJukh ZxCKLDkKIgkYQEdbmLDgGo/IUYEjvB8s0slpfEMNjJbNYmJ1qyQnPb+gM0ZKxr1wNEmY FdUCxpKULvlaIbtgCfdlf9Kbhax4820DgjfVnTbNxT7S5Y/8MmA0BPbpj1I9xMB+0caj bvVkKMONmPVR8LY7CWVbY7hMsc0ALyFS/J90LRe5BOUNUlGGnq0zAbGBJsSEWAO4kyg5 sjpg== X-Gm-Message-State: AOJu0Yztf8G+NEIMlGycjKi/tLj23wGnd9yQHqgI5uczhvEzu/uozcpA 0fjCKSVnrr73jUj0v+sMvFfZ/VaN2l8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:fe58:8292:66f9:d221]) (user=glider job=sendgmr) by 2002:a25:d4d2:0:b0:d9a:38e4:78b5 with SMTP id m201-20020a25d4d2000000b00d9a38e478b5mr140459ybf.5.1697045321463; Wed, 11 Oct 2023 10:28:41 -0700 (PDT) Date: Wed, 11 Oct 2023 19:28:30 +0200 Mime-Version: 1.0 X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231011172836.2579017-1-glider@google.com> Subject: [PATCH v7 0/5] Implement MTE tag compression for swapped pages From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, aleksander.lobakin@intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com, alexandru.elisei@arm.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-4.8 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 11 Oct 2023 10:28:59 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779481021409626504 X-GMAIL-MSGID: 1779481021409626504 Currently, when MTE pages are swapped out, the tags are kept in the memory, occupying PAGE_SIZE/32 bytes per page. This is especially problematic for devices that use zram-backed in-memory swap, because tags stored uncompressed in the heap effectively reduce the available amount of swap memory. The RLE-based algorithm suggested by Evgenii Stepanov and implemented in this patch series is able to efficiently compress fixed-size tag buffers, resulting in practical compression ratio between 2.5x and 4x. In most cases it is possible to store the compressed data in 63-bit Xarray values, resulting in no extra memory allocations. Our measurements show that the proposed algorithm provides better compression than existing kernel compression algorithms (LZ4, LZO, LZ4HC, ZSTD) can offer. To implement compression/decompression, we also extend with methods to read/write bit values at arbitrary places in the map. We refactor arch/arm64/mm/mteswap.c to support both the compressed (CONFIG_ARM64_MTE_COMP) and non-compressed case. For the former, in addition to tag compression, we move tag allocation from kmalloc() to separate kmem caches, providing greater locality and relaxing the alignment requirements. v7: - fixed comments by Yury Norov, Andy Shevchenko, Rasmus Villemoes - added perf tests for bitmap_read()/bitmap_write() - more efficient bitmap_write() implementation (meant to be sent in v5) v6: - fixed comments by Yury Norov - fixed handling of sizes divisible by MTE_GRANULES_PER_PAGE / 2 (caught while testing on a real device) v5: - fixed comments by Andy Shevchenko, Catalin Marinas, and Yury Norov - added support for 16K- and 64K pages - more efficient bitmap_write() implementation v4: - fixed a bunch of comments by Andy Shevchenko and Yury Norov - added Documentation/arch/arm64/mte-tag-compression.rst v3: - as suggested by Andy Shevchenko, use bitmap_get_value()/bitmap_set_value() written by Syed Nayyar Waris - switched to unsigned long to reduce typecasts - simplified the compression code v2: - as suggested by Yuri Norov, replace the poorly implemented struct bitq with Alexander Potapenko (4): lib/test_bitmap: add tests for bitmap_{read,write}() arm64: mte: implement CONFIG_ARM64_MTE_COMP arm64: mte: add a test for MTE tags compression arm64: mte: add compression support to mteswap.c Syed Nayyar Waris (1): lib/bitmap: add bitmap_{read,write}() Documentation/arch/arm64/index.rst | 1 + .../arch/arm64/mte-tag-compression.rst | 266 +++++++++ arch/arm64/Kconfig | 21 + arch/arm64/include/asm/mtecomp.h | 13 + arch/arm64/mm/Makefile | 7 + arch/arm64/mm/mtecomp.c | 524 ++++++++++++++++++ arch/arm64/mm/mtecomp.h | 12 + arch/arm64/mm/mteswap.c | 20 +- arch/arm64/mm/mteswap.h | 12 + arch/arm64/mm/mteswap_comp.c | 60 ++ arch/arm64/mm/mteswap_nocomp.c | 38 ++ arch/arm64/mm/test_mtecomp.c | 377 +++++++++++++ include/linux/bitmap.h | 75 +++ lib/test_bitmap.c | 166 ++++++ 14 files changed, 1581 insertions(+), 11 deletions(-) create mode 100644 Documentation/arch/arm64/mte-tag-compression.rst create mode 100644 arch/arm64/include/asm/mtecomp.h create mode 100644 arch/arm64/mm/mtecomp.c create mode 100644 arch/arm64/mm/mtecomp.h create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c create mode 100644 arch/arm64/mm/test_mtecomp.c